[AMDGPU] Iterative scan implementation for atomic optimizer.

[pravinjagtap]

Summary

This patch provides an alternative implementation to DPP for Scan Computations.

An alternative implementation iterates over all active lanes of Wavefront
using llvm.cttz and performs the following steps:

1. Read the value that needs to be atomically incremented using llvm.amdgcn.readlane intrinsic
2. Accumulate the result.
3. Update the scan result using llvm.amdgcn.writelane intrinsic if intermediate scan results are needed later in the kernel.

[cdevadas]

Shouldn't this new lowering get enabled for device functions too?

[foad]

This is the same as D147303?

[pravinjagtap]

In D147408#4239911, @foad wrote:

This is the same as D147303?

Yes Jay. D147303 was created for testing purpose. I have abandoned it. Sorry for confusion.

[pravinjagtap]

Addressed @cdevadas comment. Used isGraphics to guard graphic shaders DPP implementation against this new iterative approach using readlane and writelane

[cdevadas]

Need a test for device functions.

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

673–698

I'm not sure if this should get enabled for all graphics CCs. @foad can you confirm?

llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

285

You're turning this flag on by default and it is going to change the default behavior for shaders. Can you run the shader tests too?

[ruiling]

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

518

Why do we choose to unroll the loop over wave-front-size? I think this makes the sp3 assembly hard to read. Shouldn't a loop over active lanes just work?

[pravinjagtap]

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

518

Hello @ruiling,

One of the considerations for selecting this approach is its simplicity and efforts required for implementation. We know that the most optimized implementation for the scan is DPP with WWM. In the future, this iterative approach will become redundant when concerns related to WWM robustness are addressed. If you and everyone else think that loop over active lanes is the right thing to do, I will start implementing it.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

673–698

I think part of the point of doing this is to stop special casing graphics usage. Semantically the shaderiness shouldn't matter. A strategy switch would be a separate control if we wanted such a thing

llvm/test/CodeGen/AMDGPU/llc-pipeline.ll

265–268 ↗

(On Diff #510505)

Does this need to be rebased? I wouldn't expect pass changes

[b-sumner]

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

518

Scalar branches may be the most expensive aspect of this algorithm . If the loop is fully unrolled, we still end up with 64 in wave64. If we didn't unroll the active-lane approach loop, then if the wave is full, then couldn't we end up with 126 scalar branches?

[foad]

Scalar branches may be the most expensive aspect of this algorithm

If not-taken conditional branches are cheap then we could do something like this. It only has one taken branch, when we have finished handling all the active lanes.

  // Inclusive plus-scan v0 into v1. Also leaves the result of the plus-reduction in s3.
  s_mov s0, exec
  s_mov s3, 0 // accumulator
// repeat this section 32 or 64 times:
  s_ff1 s1, s0 // find lowest remaining active lane
  s_cmp_eq s1, -1
  s_cbranch_scc1 end
  s_bitset0 s0, s1
  v_readlane s2, v0, s1
  s_add s3, s2
  v_writelane v1, s3, s1
// end of repeated section
end:

[b-sumner]

In D147408#4243403, @foad wrote:

Scalar branches may be the most expensive aspect of this algorithm

If not-taken conditional branches are cheap then we could do something like this. It only has one taken branch, when we have finished handling all the active lanes.

  // Inclusive plus-scan v0 into v1. Also leaves the result of the plus-reduction in s3.
  s_mov s0, exec
  s_mov s3, 0 // accumulator
// repeat this section 32 or 64 times:
  s_ff1 s1, s0 // find lowest remaining active lane
  s_cmp_eq s1, -1
  s_cbranch_scc1 end
  s_bitset0 s0, s1
  v_readlane s2, v0, s1
  s_add s3, s2
  v_writelane v1, s3, s1
// end of repeated section
end:

Yes, that looks like what we want. The challenge will be creating IR that will lower to that.

[arsenm]

In D147408#4243427, @b-sumner wrote:

In D147408#4243403, @foad wrote:

Scalar branches may be the most expensive aspect of this algorithm

If not-taken conditional branches are cheap then we could do something like this. It only has one taken branch, when we have finished handling all the active lanes.

  // Inclusive plus-scan v0 into v1. Also leaves the result of the plus-reduction in s3.
  s_mov s0, exec
  s_mov s3, 0 // accumulator
// repeat this section 32 or 64 times:
  s_ff1 s1, s0 // find lowest remaining active lane
  s_cmp_eq s1, -1
  s_cbranch_scc1 end
  s_bitset0 s0, s1
  v_readlane s2, v0, s1
  s_add s3, s2
  v_writelane v1, s3, s1
// end of repeated section
end:

Yes, that looks like what we want. The challenge will be creating IR that will lower to that.

I increasingly think we should just have intrinsics for reduction ops and move all this into codegen

[nhaehnle]

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

673–698

Let's be clear:

• Using the loop is bound to be slower in almost all cases, often significantly so.
• The fast path is currently always used in graphics.
• We cannot cause such significant performance regressions for graphics.

I agree that if we do have two different paths here, it doesn't make sense to make them "graphics" vs. "compute", but to instead have a dedicated switch. The important part is that that switch defaults to the existing, fast path for graphics.

[ruiling]

If not-taken conditional branches are cheap then we could do something like this. It only has one taken branch, when we have finished handling all the active lanes.

  // Inclusive plus-scan v0 into v1. Also leaves the result of the plus-reduction in s3.
  s_mov s0, exec
  s_mov s3, 0 // accumulator
// repeat this section 32 or 64 times:
  s_ff1 s1, s0 // find lowest remaining active lane
  s_cmp_eq s1, -1
  s_cbranch_scc1 end
  s_bitset0 s0, s1
  v_readlane s2, v0, s1
  s_add s3, s2
  v_writelane v1, s3, s1
// end of repeated section
end:

The LLVM IR that can do this:

bb0:
  %value = ... 
  %ballot = call i32 @llvm.amdgcn.ballot.i32(i1 1)
  br label %bb1

bb1:
  %accum = phi i32 [ 0, %entry ], [ %new_accum, %bb1 ]
  %old_value_phi = phi i32 [ poison, %entry ], [ %old_value, %bb1 ]
  %active_bits = phi i32 [ %ballot, %entry ], [ %new_active_bits, %bb1 ]
  %ff1 = call i32 @llvm.cttz.i32(i32 %active_bits, i1 true)

  %lane_value = call i32 @llvm.amdgcn.readlane(i32 %value, i32 %ff1)
  %old_value = call i32 @llvm.amdgcn.writelane(i32 %accum, i32 %ff1, i32 %old_value_phi)
  %new_accum = add i32 %accum, %lane_value

  %mask = shl i32 1, %ff1
  %inverse_mask = xor i32 %mask, -1
  %new_active_bits = and i32 %active_bits, %inverse_mask
  %is_end = icmp eq i32 %new_active_bits, 0
  br i1 %is_end, label %bb2, label %bb1

bb2:

[pravinjagtap]

Implemented @ruiling suggestions. In this approach, we iterate over only active lanes of a wavefront using llvm.cttz to precompute an exclusive scan scan.

I have attempted the unrolled version of this loop to avoid the conditional cost of taken branch, but, unfortunately compile time cost increases proportionally as we need to create 64 basic blocks for one atomic operation (one for each active lane). So, compile time becomes the function of no of atomic ops into wavefront size.

TODO:

• Not finalized the dedicated switch between graphics vs compute. I am not sure about how this can be addressed. If we default to DDP then users of compute need to explicitly set the flag for selecting this iterative approach for compute.
• Device function test.

[pravinjagtap]

llvm/test/CodeGen/AMDGPU/llc-pipeline.ll

265–268 ↗

(On Diff #510505)

Code is rebased correctly. The AMDGPUAtomicOptimizer pass has dependency on UniformityInfoWrapperPass. Enabling atomic optimizer pass is scheduling Cycle Info and Uniformity analysis.

[pravinjagtap]

In D147408#4239304, @cdevadas wrote:

Shouldn't this new lowering get enabled for device functions too?

Hello @cdevadas, The current visitor of AtomicRMWInst considers only AMDGPUAS::GLOBAL_ADDRESS and AMDGPUAS::LOCAL_ADDRESS as potential candidates for atomic optimizations and *NOT* the AMDGPUAS::FLAT_ADDRESS. In cases of device functions, I am observing that input argument (if device function is doing atomic add then we need to pass the address to device function) are addrSpaceCasted to AMDGPUAS::FLAT_ADDRESS in the caller (i.e global function) before passing it to device function. Thats the reason why this lowering is not getting enabled for device functions. Will talk to @b-sumner and @arsenm about handling of this.

[pravinjagtap]

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

499

Hello @ruiling.

Your suggestions i.e., loop based iterative approach have been implemented to perform scan operation. Now, we iterate over only active lanes using @llvm.cttz and clear the associated bit when processed so that for the next iteration we will be branching out to next active lane.

[ruiling]

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

499

This part LGTM.

[pravinjagtap]

Introduced new command line flag amdgpu-atomic-optimizer-use-dpp which selects the scan implementation(DPP/Iterative).

Following is the plan for submission:

1. Submit this change with -amdgpu-atomic-optimizer-use-dpp=on and -amdgpu-atomic-optimizations=off
2. Downstream clients can opt in using -amdgpu-atomic-optimizer-use-dpp=on and -amdgpu-atomic-optimizations=on
3. Change default -amdgpu-atomic-optimizations to on and -amdgpu-atomic-optimizer-use-dpp to off so that compute uses iterative approach

In the lit tests you could see that iterative approach is being selected when we turned off -amdgpu-atomic-optimizer-use-dpp

[pravinjagtap]

Rebased & Ping

[foad]

llvm/lib/Target/AMDGPU/AMDGPU.h

89

Just "UseDpp"?

245

No reason for this to be public.

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

664

Remove llvm::, use getFunction

670–671

Sink this down to line 700, where you use them?

llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

277–280

Description seems a bit misleading to me since this option doesn't enable the whole atomic optimizer pass. I would suggest changing it to "Enable use of DPP in the atomic optimizer" or just "Use DPP in the atomic optimizer"

[foad]

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

512–513

Don't need these. They will always be the same as WaveTy and WaveFrontSize.

526

I think you can use an undef value here instead of V.

532–533

Use B.CreateIntrinsic, then you don't need M.

540

Typo "perform", "intermediate"

544–547

Use B.CreateIntrinsic

564

Maybe change "Compute" to "ComputeLoop" everywhere, since it is the body of the loop?

747–753

I don't think you need to change any of this. The original way of doing the icmp should work in all cases.

762

Why do you need to clone? Could you just do Terminator->removeFromParent then B.Insert(Terminator)?

[foad]

llvm/test/CodeGen/AMDGPU/atomic_optimizations_buffer.ll

459–462

Not your fault, but we really ought to be able to select s_ff1_i32_b64 here.

468

Not your fault, but we really ought to be able to select s_bitset0_b64 here.

[pravinjagtap]

Thank you @foad for comments. Addressed most of them.

[pravinjagtap]

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

670–671

Sink this down to line 700, where you use them?

ComputeEnd is required at line 766 & 770 after if ValDivergent loop. Thats why it is hoisted here.

747–753

Actually No. In the WWM, only the 0th lane (its always the case) will update the final value in a wavefront whereas in the iterative approach first active lane will update the final value (first active lane will not be 0th always in iterative approach).

[pravinjagtap]

llvm/test/CodeGen/AMDGPU/atomic_optimizations_buffer.ll

459–462

Not your fault, but we really ought to be able to select s_ff1_i32_b64 here.

I am not sure how to address this. May be, we need to teach ISel this specific pattern.

[foad]

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

670–671

I am suggesting to put these two lines immediately before the call to buildScanIteratively (line 695 now).

747–753

No, even in the DPP case, the atomic is executed by the first active lane, not lane 0. This happens after exiting the WWM section.

[pravinjagtap]

Addressed review comments

[pravinjagtap]

Rebased.

[pravinjagtap]

llvm/test/CodeGen/AMDGPU/atomic_optimizations_buffer.ll

459–462

Hello @foad,
Can we consider generating s_ff1_i32_b64 and s_bitset0_b64 as independent task (future enhancement) and unblock this to move forward since we need to submit this in stages ?

[foad]

llvm/test/CodeGen/AMDGPU/atomic_optimizations_buffer.ll

459–462

Can we consider generating s_ff1_i32_b64 and s_bitset0_b64 as independent task (future enhancement)

Yes of course, it does not need to block this patch.

[pravinjagtap]

Rebased. I think, work is in good shape now. Please let me know if there are any other concerns, if not, we can move forward.

Note: This work needs to be submitted in stages, LLPC needs to adapt to the new flag (-amdgpu-atomic-optimizer-use-dpp). By default this flag is turned ON now. Once LLPC adapts to this change, compute can turned OFF this flag.

[pravinjagtap]

Ping & Rebase

[cdevadas]

LGTM

llvm/test/CodeGen/AMDGPU/GlobalISel/atomic_optimizations_mul_one.ll

3

-verify-machineinstrs won't do anything here.

[ruiling]

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

760

I think you also need to update Dominator tree through DTU as we are inserting two extra blocks. And the branch setup process sounds messy. We insert a branch to ComputeLoop in the middle of the entry block. And then we further split the entry block for inserting the single-lane block. It might be more clear to first split the entry block before inserting the branch to ComputeLoop.

[arsenm]

Should have some checks on the IR output

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

54

Should probably add header comment explaining the different strategies

503

east const throughout

504

ST->getWavefrontSize()

523

Use poison for missing values

552

You already have WaveTy above

553

You already have WaveTy above

558

You already have WaveTy above

665

F->getContext()

701

Use consistent naming style for the blocks? other places used snake case

llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

277–285

I would expect this to be one cl::enum flag for the optimizer strategy. It would also still be better if this was a parsable pass parameter

llvm/test/CodeGen/AMDGPU/GlobalISel/atomic_optimizations_mul_one.ll

3

-verify-machineinstrs should be removed

[pravinjagtap]

Addressed review comments of @ruiling and @arsenm

[ruiling]

Thanks for the new version, looks good to me.

[pravinjagtap]

Added parsable cl::enum flag for the optimizer strategy as suggested by @arsenm.

We need to enable atomic optimizer using -amdgpu-atomic-optimizations=true and then
we can use -amdgpu-atomic-optimizer-strategy=DPP/Iterative flag to select the strategy for scan (defaulted to DPP for now and will be changed to Iterative for compute pipeline once LLPC adapts to this change).

This will allow LLPC to adjust to new change and then we can enable and default to Iterative strategy using only one flag -amdgpu-atomic-optimizer-strategy=DPP/Iterative
and deprecate -amdgpu-atomic-optimizations

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

503

These weren't done, const is still weirdly placed

759

Won't the regular post-pass verifier catch this?

784–788

Could use initializer list with all of these instead of push_back x 3. Also can use std::array

llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

277–285

You shouldn't nede to come up with your own parsing with clEnumVal. For an example see the recent amdgpu-lower-module-lds-strategy

[pravinjagtap]

Addressed review comments of @arsenm

[pravinjagtap]

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

759

Won't the regular post-pass verifier catch this?

Yes, During my experimentation I found that -verify-dom-info is not catching the issues related to updates required in dominator tree (both with opt and llc). Although, assert here is giving desired behavior.

[pravinjagtap]

Rebased.

[arsenm]

LGTM with nit

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

560

{OldValue, NewAccumulator}

782

You don't need std::initializer_list, you can just use std::array

[arsenm]

llvm/test/CodeGen/AMDGPU/global_atomics_iterative_scan.ll

101

Use named values

120

Can drop most attributes

[pravinjagtap]

Addressed review comments.

[foad]

We need to enable atomic optimizer using -amdgpu-atomic-optimizations=true and then
we can use -amdgpu-atomic-optimizer-strategy=DPP/Iterative flag to select the strategy for scan (defaulted to DPP for now and will be changed to Iterative for compute pipeline once LLPC adapts to this change).

This will allow LLPC to adjust to new change and then we can enable and default to Iterative strategy using only one flag -amdgpu-atomic-optimizer-strategy=DPP/Iterative
and deprecate -amdgpu-atomic-optimizations

Here's the LLPC patch to set -amdgpu-atomic-optimizer-strategy=DPP: https://github.com/GPUOpen-Drivers/llpc/pull/2506

llvm/lib/Target/AMDGPU/AMDGPU.h

 void initializeAMDGPUAlwaysInlinePass(PassRegistry&);
 
 Pass *createAMDGPUAnnotateKernelFeaturesPass();
 Pass *createAMDGPUAttributorPass();
 void initializeAMDGPUAttributorPass(PassRegistry &);
 void initializeAMDGPUAnnotateKernelFeaturesPass(PassRegistry &);
 extern char &AMDGPUAnnotateKernelFeaturesID;
 
-FunctionPass *createAMDGPUAtomicOptimizerPass();
+enum class ScanOptions : bool { DPP, Iterative };
+FunctionPass *createAMDGPUAtomicOptimizerPass(bool UseDpp);
 void initializeAMDGPUAtomicOptimizerPass(PassRegistry &);
 extern char &AMDGPUAtomicOptimizerID;
 
 ModulePass *createAMDGPULowerIntrinsicsPass();
 void initializeAMDGPULowerIntrinsicsPass(PassRegistry &);
 extern char &AMDGPULowerIntrinsicsID;
 
 ModulePass *createAMDGPUCtorDtorLoweringLegacyPass();
struct AMDGPUPromoteAllocaToVectorPass
   AMDGPUPromoteAllocaToVectorPass(TargetMachine &TM) : TM(TM) {}
   PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
 
 private:
   TargetMachine &TM;
 };
 
 struct AMDGPUAtomicOptimizerPass : PassInfoMixin<AMDGPUAtomicOptimizerPass> {
-  AMDGPUAtomicOptimizerPass(TargetMachine &TM) : TM(TM) {}
+  AMDGPUAtomicOptimizerPass(TargetMachine &TM, ScanOptions ScanImpl)
+      : TM(TM), ScanImpl(ScanImpl) {}
   PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
 
 private:
   TargetMachine &TM;
+  ScanOptions ScanImpl;
 };
 
 Pass *createAMDGPUStructurizeCFGPass();
 FunctionPass *createAMDGPUISelDag(TargetMachine &TM,
                                   CodeGenOpt::Level OptLevel);
 ModulePass *createAMDGPUAlwaysInlinePass(bool GlobalOpt = true);
 
 struct AMDGPUAlwaysInlinePass : PassInfoMixin<AMDGPUAlwaysInlinePass> {
   AMDGPUAlwaysInlinePass(bool GlobalOpt = true) : GlobalOpt(GlobalOpt) {}

llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

 //===-- AMDGPUAtomicOptimizer.cpp -----------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 /// \file
 /// This pass optimizes atomic operations by using a single lane of a wavefront
 /// to perform the atomic operation, thus reducing contention on that memory
 /// location.
-//
+/// Atomic optimizer uses following strategies to compute scan and reduced
+/// values
+/// 1. DPP -
+///   This is the most efficient implementation for scan. DPP uses Whole Wave
+///   Mode (WWM)
+/// 2. Iterative -
+//    An alternative implementation iterates over all active lanes
+///   of Wavefront using llvm.cttz and performs scan  using readlane & writelane
+///   intrinsics
 //===----------------------------------------------------------------------===//
 
 #include "AMDGPU.h"
 #include "GCNSubtarget.h"
 #include "llvm/Analysis/DomTreeUpdater.h"
 #include "llvm/Analysis/UniformityAnalysis.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/IR/IRBuilder.h"
struct ReplacementInfo {
   AtomicRMWInst::BinOp Op;
   unsigned ValIdx;
   bool ValDivergent;
 };
 
 class AMDGPUAtomicOptimizer : public FunctionPass {
 public:
   static char ID;
-
-  AMDGPUAtomicOptimizer() : FunctionPass(ID) {}
+  ScanOptions ScanImpl;
+  AMDGPUAtomicOptimizer(ScanOptions ScanImpl)
+      : FunctionPass(ID), ScanImpl(ScanImpl) {}
 
   bool runOnFunction(Function &F) override;
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addPreserved<DominatorTreeWrapperPass>();
     AU.addRequired<UniformityInfoWrapperPass>();
     AU.addRequired<TargetPassConfig>();
   }
 };
 
 class AMDGPUAtomicOptimizerImpl
     : public InstVisitor<AMDGPUAtomicOptimizerImpl> {
 private:
   SmallVector<ReplacementInfo, 8> ToReplace;
   const UniformityInfo *UA;
   const DataLayout *DL;
   DomTreeUpdater &DTU;
   const GCNSubtarget *ST;
   bool IsPixelShader;
+  ScanOptions ScanImpl;
 
   Value *buildReduction(IRBuilder<> &B, AtomicRMWInst::BinOp Op, Value *V,
                         Value *const Identity) const;
   Value *buildScan(IRBuilder<> &B, AtomicRMWInst::BinOp Op, Value *V,
                    Value *const Identity) const;
   Value *buildShiftRight(IRBuilder<> &B, Value *V, Value *const Identity) const;
 
+  std::pair<Value *, Value *>
+  buildScanIteratively(IRBuilder<> &B, AtomicRMWInst::BinOp Op,
+                       Value *const Identity, Value *V, Instruction &I,
+                       BasicBlock *ComputeLoop, BasicBlock *ComputeEnd) const;
+
   void optimizeAtomic(Instruction &I, AtomicRMWInst::BinOp Op, unsigned ValIdx,
                       bool ValDivergent) const;
 
 public:
   AMDGPUAtomicOptimizerImpl() = delete;
 
   AMDGPUAtomicOptimizerImpl(const UniformityInfo *UA, const DataLayout *DL,
                             DomTreeUpdater &DTU, const GCNSubtarget *ST,
-                            bool IsPixelShader)
-      : UA(UA), DL(DL), DTU(DTU), ST(ST), IsPixelShader(IsPixelShader) {}
+                            bool IsPixelShader, ScanOptions ScanImpl)
+      : UA(UA), DL(DL), DTU(DTU), ST(ST), IsPixelShader(IsPixelShader),
+        ScanImpl(ScanImpl) {}
 
   bool run(Function &F);
 
   void visitAtomicRMWInst(AtomicRMWInst &I);
   void visitIntrinsicInst(IntrinsicInst &I);
 };
 
 } // namespace
DomTreeUpdater DTU(DTW ? &DTW->getDomTree() : nullptr,
                      DomTreeUpdater::UpdateStrategy::Lazy);
 
   const TargetPassConfig &TPC = getAnalysis<TargetPassConfig>();
   const TargetMachine &TM = TPC.getTM<TargetMachine>();
   const GCNSubtarget *ST = &TM.getSubtarget<GCNSubtarget>(F);
 
   bool IsPixelShader = F.getCallingConv() == CallingConv::AMDGPU_PS;
 
-  return AMDGPUAtomicOptimizerImpl(UA, DL, DTU, ST, IsPixelShader).run(F);
+  return AMDGPUAtomicOptimizerImpl(UA, DL, DTU, ST, IsPixelShader, ScanImpl)
+      .run(F);
 }
 
 PreservedAnalyses AMDGPUAtomicOptimizerPass::run(Function &F,
                                                  FunctionAnalysisManager &AM) {
 
   const auto *UA = &AM.getResult<UniformityInfoAnalysis>(F);
   const DataLayout *DL = &F.getParent()->getDataLayout();
 
   DomTreeUpdater DTU(&AM.getResult<DominatorTreeAnalysis>(F),
                      DomTreeUpdater::UpdateStrategy::Lazy);
   const GCNSubtarget *ST = &TM.getSubtarget<GCNSubtarget>(F);
 
   bool IsPixelShader = F.getCallingConv() == CallingConv::AMDGPU_PS;
 
-  return AMDGPUAtomicOptimizerImpl(UA, DL, DTU, ST, IsPixelShader).run(F)
+  return AMDGPUAtomicOptimizerImpl(UA, DL, DTU, ST, IsPixelShader, ScanImpl)
+                 .run(F)
              ? PreservedAnalyses::none()
              : PreservedAnalyses::all();
 }
 
 bool AMDGPUAtomicOptimizerImpl::run(Function &F) {
 
   visit(F);
 
if (!ST->isWave32()) {
           WriteLane,
           {B.CreateCall(ReadLane, {Old, B.getInt32(47)}), B.getInt32(48), V});
     }
   }
 
   return V;
 }
 
+// Use the builder to create an exclusive scan and compute the final reduced
+// value using an iterative approach. This provides an alternative
+// implementation to DPP which uses WMM for scan computations. This API iterate
+// over active lanes to read, compute and update the value using
+// readlane and writelane intrinsics.
+std::pair<Value *, Value *> AMDGPUAtomicOptimizerImpl::buildScanIteratively(
+    IRBuilder<> &B, AtomicRMWInst::BinOp Op, Value *const Identity, Value *V,
+    Instruction &I, BasicBlock *ComputeLoop, BasicBlock *ComputeEnd) const {
+
+  Type *const Ty = I.getType();
+  const unsigned WaveFrontSize = ST->getWavefrontSize();
+  Type *const WaveTy = B.getIntNTy(WaveFrontSize);
+  BasicBlock *const EntryBB = I.getParent();
+  const bool NeedResult = !I.use_empty();
+
+  Value *const Ballot =
+      B.CreateIntrinsic(Intrinsic::amdgcn_ballot, WaveTy, B.getTrue());
+
+  // Start inserting instructions for ComputeLoop block
+  B.SetInsertPoint(ComputeLoop);
+  // Phi nodes for Accumulator, Scan results destination, and Active Lanes
+  PHINode *const Accumulator = B.CreatePHI(Ty, 2, "Accumulator");
+  Accumulator->addIncoming(Identity, EntryBB);
+  PHINode *OldValuePhi = nullptr;
+  if (NeedResult) {
+    OldValuePhi = B.CreatePHI(Ty, 2, "OldValuePhi");
+    OldValuePhi->addIncoming(PoisonValue::get(Ty), EntryBB);
+  }
+  PHINode *const ActiveBits = B.CreatePHI(WaveTy, 2, "ActiveBits");
+  ActiveBits->addIncoming(Ballot, EntryBB);
+
+  // Use llvm.cttz instrinsic to find the lowest remaining active lane.
+  Value *const FF1 =
+      B.CreateIntrinsic(Intrinsic::cttz, WaveTy, {ActiveBits, B.getTrue()});
+  Value *const LaneIdxInt = B.CreateTrunc(FF1, Ty);
+
+  // Get the value required for atomic operation
+  Value *const LaneValue =
+      B.CreateIntrinsic(Intrinsic::amdgcn_readlane, {}, {V, LaneIdxInt});
+
+  // Perform writelane if intermediate scan results are required later in the
+  // kernel computations
+  Value *OldValue = nullptr;
+  if (NeedResult) {
+    OldValue = B.CreateIntrinsic(Intrinsic::amdgcn_writelane, {},
+                                 {Accumulator, LaneIdxInt, OldValuePhi});
+    OldValuePhi->addIncoming(OldValue, ComputeLoop);
+  }
+
+  // Accumulate the results
+  Value *const NewAccumulator =
+      buildNonAtomicBinOp(B, Op, Accumulator, LaneValue);
+  Accumulator->addIncoming(NewAccumulator, ComputeLoop);
+
+  // Set bit to zero of current active lane so that for next iteration llvm.cttz
+  // return the next active lane
+  Value *const Mask = B.CreateShl(ConstantInt::get(WaveTy, 1), FF1);
+
+  Value *const InverseMask = B.CreateXor(Mask, ConstantInt::get(WaveTy, -1));
+  Value *const NewActiveBits = B.CreateAnd(ActiveBits, InverseMask);
+  ActiveBits->addIncoming(NewActiveBits, ComputeLoop);
+
+  // Branch out of the loop when all lanes are processed.
+  Value *const IsEnd =
+      B.CreateICmpEQ(NewActiveBits, ConstantInt::get(WaveTy, 0));
+  B.CreateCondBr(IsEnd, ComputeEnd, ComputeLoop);
+
+  B.SetInsertPoint(ComputeEnd);
+
+  return std::make_pair(OldValue, NewAccumulator);
+}
+
 static APInt getIdentityValueForAtomicOp(AtomicRMWInst::BinOp Op,
                                          unsigned BitWidth) {
   switch (Op) {
   default:
     llvm_unreachable("Unhandled atomic op");
   case AtomicRMWInst::Add:
   case AtomicRMWInst::Sub:
   case AtomicRMWInst::Or:
void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
 
   Value *const Identity = B.getInt(getIdentityValueForAtomicOp(Op, TyBitWidth));
 
   Value *ExclScan = nullptr;
   Value *NewV = nullptr;
 
   const bool NeedResult = !I.use_empty();
 
+  Function *F = I.getFunction();
+  LLVMContext &C = F->getContext();
+  BasicBlock *ComputeLoop = nullptr;
+  BasicBlock *ComputeEnd = nullptr;
   // If we have a divergent value in each lane, we need to combine the value
   // using DPP.
   if (ValDivergent) {
+    const AtomicRMWInst::BinOp ScanOp =
+        Op == AtomicRMWInst::Sub ? AtomicRMWInst::Add : Op;
+    if (ScanImpl == ScanOptions::DPP) {
       // First we need to set all inactive invocations to the identity value, so
       // that they can correctly contribute to the final result.
-    NewV = B.CreateIntrinsic(Intrinsic::amdgcn_set_inactive, Ty, {V, Identity});
-
+      NewV =
+          B.CreateIntrinsic(Intrinsic::amdgcn_set_inactive, Ty, {V, Identity});
       const AtomicRMWInst::BinOp ScanOp =
           Op == AtomicRMWInst::Sub ? AtomicRMWInst::Add : Op;
       if (!NeedResult && ST->hasPermLaneX16()) {
-      // On GFX10 the permlanex16 instruction helps us build a reduction without
-      // too many readlanes and writelanes, which are generally bad for
-      // performance.
+        // On GFX10 the permlanex16 instruction helps us build a reduction
+        // without too many readlanes and writelanes, which are generally bad
+        // for performance.
         NewV = buildReduction(B, ScanOp, NewV, Identity);
       } else {
         NewV = buildScan(B, ScanOp, NewV, Identity);
         if (NeedResult)
           ExclScan = buildShiftRight(B, NewV, Identity);
-
-      // Read the value from the last lane, which has accumulated the values of
-      // each active lane in the wavefront. This will be our new value which we
-      // will provide to the atomic operation.
+        // Read the value from the last lane, which has accumulated the values
+        // of each active lane in the wavefront. This will be our new value
+        // which we will provide to the atomic operation.
         Value *const LastLaneIdx = B.getInt32(ST->getWavefrontSize() - 1);
         assert(TyBitWidth == 32);
         NewV = B.CreateIntrinsic(Intrinsic::amdgcn_readlane, {},
                                  {NewV, LastLaneIdx});
       }
-
       // Finally mark the readlanes in the WWM section.
       NewV = B.CreateIntrinsic(Intrinsic::amdgcn_strict_wwm, Ty, NewV);
     } else {
+      // Alternative implementation for scan
+      ComputeLoop = BasicBlock::Create(C, "ComputeLoop", F);
+      ComputeEnd = BasicBlock::Create(C, "ComputeEnd", F);
+      std::tie(ExclScan, NewV) = buildScanIteratively(B, ScanOp, Identity, V, I,
+                                                      ComputeLoop, ComputeEnd);
+    }
+  } else {
     switch (Op) {
     default:
       llvm_unreachable("Unhandled atomic op");
 
     case AtomicRMWInst::Add:
     case AtomicRMWInst::Sub: {
       // The new value we will be contributing to the atomic operation is the
       // old value times the number of active lanes.
if (ValDivergent) {
     }
   }
 
   // We only want a single lane to enter our new control flow, and we do this
   // by checking if there are any active lanes below us. Only one lane will
   // have 0 active lanes below us, so that will be the only one to progress.
   Value *const Cond = B.CreateICmpEQ(Mbcnt, B.getIntN(TyBitWidth, 0));
 
   // Store I's original basic block before we split the block.
   BasicBlock *const EntryBB = I.getParent();
 
   // We need to introduce some new control flow to force a single lane to be
   // active. We do this by splitting I's basic block at I, and introducing the
   // new block such that:
   // entry --> single_lane -\
   //       \------------------> exit
   Instruction *const SingleLaneTerminator =
       SplitBlockAndInsertIfThen(Cond, &I, false, nullptr, &DTU, nullptr);
 
+  // Control flow is changed here after splitting I's block
+  assert(DTU.getDomTree().verify(DominatorTree::VerificationLevel::Fast));
+
+  // At this point, we have split the I's block to allow one lane in wavefront
+  // to update the precomputed reduced value. Also, completed the codegen for
+  // new control flow i.e. iterative loop which perform reduction and scan using
+  // ComputeLoop and ComputeEnd.
+  // For the new control flow, we need to move branch instruction i.e.
+  // terminator created during SplitBlockAndInsertIfThen from I's block to
+  // ComputeEnd block. We also need to set up predecessor to next block when
+  // single lane done updating the final reduced value.
+  BasicBlock *Predecessor = nullptr;
+  if (ValDivergent && ScanImpl == ScanOptions::Iterative) {
+    // Move terminator from I's block to ComputeEnd block.
+    Instruction *Terminator = EntryBB->getTerminator();
+    B.SetInsertPoint(ComputeEnd);
+    Terminator->removeFromParent();
+    B.Insert(Terminator);
+
+    // Branch to ComputeLoop Block unconditionally from the I's block for
+    // iterative approach.
+    B.SetInsertPoint(EntryBB);
+    B.CreateBr(ComputeLoop);
+
+    // Now control flow setup is complete for iterative solution.
+    // Therefore, update the dominator tree w.r.t new control flow.
+    SmallVector<DominatorTree::UpdateType, 3> DTUpdates;
+    DTUpdates.push_back({DominatorTree::Insert, EntryBB, ComputeLoop});
+    DTUpdates.push_back({DominatorTree::Insert, ComputeLoop, ComputeEnd});
+    DTUpdates.push_back(
+        {DominatorTree::Delete, EntryBB, SingleLaneTerminator->getParent()});
+    DTU.applyUpdates(DTUpdates);
+    assert(DTU.getDomTree().verify(DominatorTree::VerificationLevel::Fast));
+
+    Predecessor = ComputeEnd;
+  } else {
+    Predecessor = EntryBB;
+  }
   // Move the IR builder into single_lane next.
   B.SetInsertPoint(SingleLaneTerminator);
 
   // Clone the original atomic operation into single lane, replacing the
   // original value with our newly created one.
   Instruction *const NewI = I.clone();
   B.Insert(NewI);
   NewI->setOperand(ValIdx, NewV);
 
   // Move the IR builder into exit next, and start inserting just before the
   // original instruction.
   B.SetInsertPoint(&I);
 
   if (NeedResult) {
     // Create a PHI node to get our new atomic result into the exit block.
     PHINode *const PHI = B.CreatePHI(Ty, 2);
-    PHI->addIncoming(PoisonValue::get(Ty), EntryBB);
+    PHI->addIncoming(PoisonValue::get(Ty), Predecessor);
     PHI->addIncoming(NewI, SingleLaneTerminator->getParent());
 
     // We need to broadcast the value who was the lowest active lane (the first
     // lane) to all other lanes in the wavefront. We use an intrinsic for this,
     // but have to handle 64-bit broadcasts with two calls to this intrinsic.
     Value *BroadcastI = nullptr;
 
     if (TyBitWidth == 64) {
if (NeedResult) {
     }
 
     // Now that we have the result of our single atomic operation, we need to
     // get our individual lane's slice into the result. We use the lane offset
     // we previously calculated combined with the atomic result value we got
     // from the first lane, to get our lane's index into the atomic result.
     Value *LaneOffset = nullptr;
     if (ValDivergent) {
+      if (ScanImpl == ScanOptions::DPP) {
         LaneOffset =
             B.CreateIntrinsic(Intrinsic::amdgcn_strict_wwm, Ty, ExclScan);
       } else {
+        LaneOffset = ExclScan;
+      }
+    } else {
       switch (Op) {
       default:
         llvm_unreachable("Unhandled atomic op");
       case AtomicRMWInst::Add:
       case AtomicRMWInst::Sub:
         LaneOffset = buildMul(B, V, Mbcnt);
         break;
       case AtomicRMWInst::And:

 
 INITIALIZE_PASS_BEGIN(AMDGPUAtomicOptimizer, DEBUG_TYPE,
                       "AMDGPU atomic optimizations", false, false)
 INITIALIZE_PASS_DEPENDENCY(UniformityInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
 INITIALIZE_PASS_END(AMDGPUAtomicOptimizer, DEBUG_TYPE,
                     "AMDGPU atomic optimizations", false, false)
 
-FunctionPass *llvm::createAMDGPUAtomicOptimizerPass() {
-  return new AMDGPUAtomicOptimizer();
+FunctionPass *llvm::createAMDGPUAtomicOptimizerPass(bool IsUseDpp) {
+  return new AMDGPUAtomicOptimizer(IsUseDpp ? ScanOptions::DPP
+                                            : ScanOptions::Iterative);
 }

llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

static cl::opt<bool> EnableRegReassign(
   cl::init(true),
   cl::Hidden);
 
 static cl::opt<bool> OptVGPRLiveRange(
     "amdgpu-opt-vgpr-liverange",
     cl::desc("Enable VGPR liverange optimizations for if-else structure"),
     cl::init(true), cl::Hidden);
 
+static cl::opt<bool> EnableAtomicOptimizationsUsingDPP(
+    "amdgpu-atomic-optimizer-use-dpp",
+    cl::desc("Use DPP in the atomic optimizer"), cl::init(true), cl::Hidden);
+
 // Enable atomic optimization
-static cl::opt<bool> EnableAtomicOptimizations(
-  "amdgpu-atomic-optimizations",
+static cl::opt<bool>
+    EnableAtomicOptimizations("amdgpu-atomic-optimizations",
                               cl::desc("Enable atomic optimizations"),
-  cl::init(false),
-  cl::Hidden);
+                              cl::init(false), cl::Hidden);
 
 // Enable Mode register optimization
 static cl::opt<bool> EnableSIModeRegisterPass(
   "amdgpu-mode-register",
   cl::desc("Enable mode register pass"),
   cl::init(true),
   cl::Hidden);
 
PB.registerPipelineParsingCallback(
           PM.addPass(AMDGPUPromoteKernelArgumentsPass());
           return true;
         }
         if (PassName == "amdgpu-unify-divergent-exit-nodes") {
           PM.addPass(AMDGPUUnifyDivergentExitNodesPass());
           return true;
         }
         if (PassName == "amdgpu-atomic-optimizer") {
-          PM.addPass(AMDGPUAtomicOptimizerPass(*this));
+          PM.addPass(
+              AMDGPUAtomicOptimizerPass(*this, EnableAtomicOptimizationsUsingDPP
+                                                   ? ScanOptions::DPP
+                                                   : ScanOptions::Iterative));
           return true;
         }
         if (PassName == "amdgpu-codegenprepare") {
           PM.addPass(AMDGPUCodeGenPreparePass(*this));
           return true;
         }
         return false;
       });

 
 bool GCNPassConfig::addPreISel() {
   AMDGPUPassConfig::addPreISel();
 
   if (TM->getOptLevel() > CodeGenOpt::None)
     addPass(createAMDGPULateCodeGenPreparePass());
 
   if (isPassEnabled(EnableAtomicOptimizations, CodeGenOpt::Less)) {
-    addPass(createAMDGPUAtomicOptimizerPass());
+    addPass(createAMDGPUAtomicOptimizerPass(EnableAtomicOptimizationsUsingDPP));
   }
 
   if (TM->getOptLevel() > CodeGenOpt::None)
     addPass(createSinkingPass());
 
   // Merge divergent exit nodes. StructurizeCFG won't recognize the multi-exit
   // regions formed by them.
   addPass(&AMDGPUUnifyDivergentExitNodesID);

llvm/test/CodeGen/AMDGPU/GlobalISel/atomic_optimizations_mul_one.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: opt -S -mtriple=amdgcn-- -amdgpu-atomic-optimizer -verify-machineinstrs %s | FileCheck -check-prefix=IR %s
+; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-atomic-optimizer %s | FileCheck -check-prefix=IR %s
 ; RUN: llc -global-isel -mtriple=amdgcn-- -amdgpu-atomic-optimizations -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
 
 declare i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32, <4 x i32>, i32, i32, i32, i32 immarg)
 declare i32 @llvm.amdgcn.struct.buffer.atomic.sub.i32(i32, <4 x i32>, i32, i32, i32, i32 immarg)
 declare i32 @llvm.amdgcn.struct.buffer.atomic.xor.i32(i32, <4 x i32>, i32, i32, i32, i32 immarg)
 declare void @llvm.amdgcn.struct.buffer.store.format.v4i32(<4 x i32>, <4 x i32>, i32, i32, i32, i32 immarg)
 
 define amdgpu_cs void @atomic_add(<4 x i32> inreg %arg)  {

llvm/test/CodeGen/AMDGPU/atomic_optimizations_buffer.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -march=amdgcn -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX6 %s
-; RUN: llc -march=amdgcn -mcpu=tonga -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX8 %s
-; RUN: llc -march=amdgcn -mcpu=gfx900 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX9 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX10W64 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX10W32 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX11W64 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX11W32 %s
+; RUN: llc -march=amdgcn -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX6 %s
+; RUN: llc -march=amdgcn -mcpu=tonga -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX8 %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX9 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX10W64 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX10W32 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX11W64 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX11W32 %s
 
 declare i32 @llvm.amdgcn.workitem.id.x()
 declare i32 @llvm.amdgcn.raw.buffer.atomic.add(i32, <4 x i32>, i32, i32, i32 immarg)
 declare i32 @llvm.amdgcn.struct.buffer.atomic.add(i32, <4 x i32>, i32, i32, i32, i32 immarg)
 declare i32 @llvm.amdgcn.raw.buffer.atomic.sub(i32, <4 x i32>, i32, i32, i32 immarg)
 
 ; Show what the atomic optimization pass will do for raw buffers.
 

 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    s_waitcnt vmcnt(0)
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: add_i32_varying_vdata:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX8-NEXT:    v_mov_b32_e32 v2, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v2, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s4, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_add_i32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB2_2
-; GFX8-NEXT:  ; %bb.1:
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB2_4
+; GFX8-NEXT:  ; %bb.3:
 ; GFX8-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    buffer_atomic_add v0, off, s[8:11], 0 glc
-; GFX8-NEXT:  .LBB2_2:
+; GFX8-NEXT:  .LBB2_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v1
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s2, v0
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, s2, v1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    v_mov_b32_e32 v4, s1
-; GFX8-NEXT:    v_mov_b32_e32 v3, s0
-; GFX8-NEXT:    flat_store_dword v[3:4], v0
+; GFX8-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8-NEXT:    flat_store_dword v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: add_i32_varying_vdata:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s4, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_add_i32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB2_2
-; GFX9-NEXT:  ; %bb.1:
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB2_4
+; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    buffer_atomic_add v0, off, s[8:11], 0 glc
-; GFX9-NEXT:  .LBB2_2:
+; GFX9-NEXT:  .LBB2_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v2
-; GFX9-NEXT:    v_add_u32_e32 v0, s2, v0
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    v_add_u32_e32 v0, s2, v1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    global_store_dword v3, v0, s[0:1]
+; GFX9-NEXT:    global_store_dword v2, v0, s[0:1]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX10W64-LABEL: add_i32_varying_vdata:
 ; GFX10W64:       ; %bb.0: ; %entry
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX10W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX10W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX10W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX10W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX10W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX10W64-NEXT:    ; implicit-def: $vgpr4
+; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX10W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX10W64-NEXT:    s_mov_b32 s4, 0
+; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX10W64-NEXT:    ; implicit-def: $vgpr1
+; GFX10W64-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX10W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W64-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX10W64-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX10W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX10W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX10W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX10W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX10W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX10W64-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX10W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX10W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX10W64-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX10W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
+; GFX10W64-NEXT:    ; implicit-def: $vgpr0
 ; GFX10W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX10W64-NEXT:    s_cbranch_execz .LBB2_2
-; GFX10W64-NEXT:  ; %bb.1:
+; GFX10W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX10W64-NEXT:    s_cbranch_execz .LBB2_4
+; GFX10W64-NEXT:  ; %bb.3:
 ; GFX10W64-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    buffer_atomic_add v4, off, s[8:11], 0 glc
-; GFX10W64-NEXT:  .LBB2_2:
+; GFX10W64-NEXT:    buffer_atomic_add v0, off, s[8:11], 0 glc
+; GFX10W64-NEXT:  .LBB2_4:
 ; GFX10W64-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX10W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX10W64-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W64-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W64-NEXT:    v_add_nc_u32_e32 v1, s2, v1
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W64-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W64-NEXT:    s_endpgm
 ;
 ; GFX10W32-LABEL: add_i32_varying_vdata:
 ; GFX10W32:       ; %bb.0: ; %entry
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX10W32-NEXT:    ; implicit-def: $vgpr4
-; GFX10W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX10W32-NEXT:    s_cbranch_execz .LBB2_2
-; GFX10W32-NEXT:  ; %bb.1:
-; GFX10W32-NEXT:    s_mov_b32 s3, s4
+; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX10W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX10W32-NEXT:    s_mov_b32 s2, 0
+; GFX10W32-NEXT:    ; implicit-def: $vgpr1
+; GFX10W32-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX10W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W32-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX10W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX10W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX10W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX10W32-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX10W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX10W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX10W32-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX10W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
+; GFX10W32-NEXT:    ; implicit-def: $vgpr0
+; GFX10W32-NEXT:    s_and_saveexec_b32 s3, vcc_lo
+; GFX10W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX10W32-NEXT:    s_cbranch_execz .LBB2_4
+; GFX10W32-NEXT:  ; %bb.3:
 ; GFX10W32-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, s3
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    buffer_atomic_add v4, off, s[4:7], 0 glc
-; GFX10W32-NEXT:  .LBB2_2:
+; GFX10W32-NEXT:    buffer_atomic_add v0, off, s[4:7], 0 glc
+; GFX10W32-NEXT:  .LBB2_4:
 ; GFX10W32-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX10W32-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W32-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W32-NEXT:    v_add_nc_u32_e32 v1, s2, v1
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W32-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W32-NEXT:    s_endpgm
 ;
 ; GFX11W64-LABEL: add_i32_varying_vdata:
 ; GFX11W64:       ; %bb.0: ; %entry
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX11W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX11W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX11W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX11W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX11W64-NEXT:    ; implicit-def: $vgpr4
-; GFX11W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX11W64-NEXT:    s_cbranch_execz .LBB2_2
-; GFX11W64-NEXT:  ; %bb.1:
+; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    s_mov_b32 s4, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX11W64-NEXT:    ; implicit-def: $vgpr1
+; GFX11W64-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX11W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W64-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX11W64-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX11W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX11W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX11W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX11W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX11W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX11W64-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX11W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX11W64-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX11W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    ; implicit-def: $vgpr0
+; GFX11W64-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX11W64-NEXT:    s_cbranch_execz .LBB2_4
+; GFX11W64-NEXT:  ; %bb.3:
 ; GFX11W64-NEXT:    s_load_b128 s[8:11], s[0:1], 0x34
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    buffer_atomic_add_u32 v4, off, s[8:11], 0 glc
-; GFX11W64-NEXT:  .LBB2_2:
+; GFX11W64-NEXT:    buffer_atomic_add_u32 v0, off, s[8:11], 0 glc
+; GFX11W64-NEXT:  .LBB2_4:
 ; GFX11W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX11W64-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    v_add_nc_u32_e32 v1, s2, v1
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W64-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W64-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W64-NEXT:    s_endpgm
 ;
 ; GFX11W32-LABEL: add_i32_varying_vdata:
 ; GFX11W32:       ; %bb.0: ; %entry
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX11W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX11W32-NEXT:    ; implicit-def: $vgpr4
-; GFX11W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX11W32-NEXT:    s_cbranch_execz .LBB2_2
-; GFX11W32-NEXT:  ; %bb.1:
-; GFX11W32-NEXT:    s_mov_b32 s3, s4
+; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    s_mov_b32 s2, 0
+; GFX11W32-NEXT:    ; implicit-def: $vgpr1
+; GFX11W32-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX11W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W32-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX11W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX11W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX11W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX11W32-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX11W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX11W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX11W32-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX11W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    ; implicit-def: $vgpr0
+; GFX11W32-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX11W32-NEXT:    s_cbranch_execz .LBB2_4
+; GFX11W32-NEXT:  ; %bb.3:
 ; GFX11W32-NEXT:    s_load_b128 s[4:7], s[0:1], 0x34
-; GFX11W32-NEXT:    v_mov_b32_e32 v4, s3
+; GFX11W32-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    buffer_atomic_add_u32 v4, off, s[4:7], 0 glc
-; GFX11W32-NEXT:  .LBB2_2:
-; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX11W32-NEXT:    buffer_atomic_add_u32 v0, off, s[4:7], 0 glc
+; GFX11W32-NEXT:  .LBB2_4:
+; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX11W32-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W32-NEXT:    v_mov_b32_e32 v4, v3
+; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX11W32-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_add_nc_u32 v1, s2, v1
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W32-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W32-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W32-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = call i32 @llvm.amdgcn.raw.buffer.atomic.add(i32 %lane, <4 x i32> %inout, i32 0, i32 0, i32 0)
   store i32 %old, ptr addrspace(1) %out
   ret void
 }

 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    s_waitcnt vmcnt(0)
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: struct_add_i32_varying_vdata:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX8-NEXT:    v_mov_b32_e32 v2, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v2, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s4, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB3_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_add_i32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB3_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB3_2
-; GFX8-NEXT:  ; %bb.1:
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB3_4
+; GFX8-NEXT:  ; %bb.3:
 ; GFX8-NEXT:    s_load_dword s5, s[0:1], 0x44
 ; GFX8-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    v_mov_b32_e32 v3, s5
-; GFX8-NEXT:    buffer_atomic_add v0, v3, s[8:11], 0 idxen glc
-; GFX8-NEXT:  .LBB3_2:
+; GFX8-NEXT:    v_mov_b32_e32 v2, s5
+; GFX8-NEXT:    buffer_atomic_add v0, v2, s[8:11], 0 idxen glc
+; GFX8-NEXT:  .LBB3_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v1
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s2, v0
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, s2, v1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    v_mov_b32_e32 v4, s1
-; GFX8-NEXT:    v_mov_b32_e32 v3, s0
-; GFX8-NEXT:    flat_store_dword v[3:4], v0
+; GFX8-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8-NEXT:    flat_store_dword v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: struct_add_i32_varying_vdata:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s4, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB3_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_add_i32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB3_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB3_2
-; GFX9-NEXT:  ; %bb.1:
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB3_4
+; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    s_load_dword s5, s[0:1], 0x44
 ; GFX9-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    v_mov_b32_e32 v4, s5
-; GFX9-NEXT:    buffer_atomic_add v0, v4, s[8:11], 0 idxen glc
-; GFX9-NEXT:  .LBB3_2:
+; GFX9-NEXT:    v_mov_b32_e32 v2, s5
+; GFX9-NEXT:    buffer_atomic_add v0, v2, s[8:11], 0 idxen glc
+; GFX9-NEXT:  .LBB3_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v2
-; GFX9-NEXT:    v_add_u32_e32 v0, s2, v0
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    v_add_u32_e32 v0, s2, v1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    global_store_dword v3, v0, s[0:1]
+; GFX9-NEXT:    global_store_dword v2, v0, s[0:1]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX10W64-LABEL: struct_add_i32_varying_vdata:
 ; GFX10W64:       ; %bb.0: ; %entry
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX10W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX10W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX10W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX10W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX10W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX10W64-NEXT:    ; implicit-def: $vgpr4
+; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX10W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX10W64-NEXT:    s_mov_b32 s4, 0
+; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX10W64-NEXT:    ; implicit-def: $vgpr1
+; GFX10W64-NEXT:  .LBB3_1: ; %ComputeLoop
+; GFX10W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W64-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX10W64-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX10W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX10W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX10W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX10W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX10W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX10W64-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX10W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX10W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX10W64-NEXT:    s_cbranch_scc1 .LBB3_1
+; GFX10W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
+; GFX10W64-NEXT:    ; implicit-def: $vgpr0
 ; GFX10W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX10W64-NEXT:    s_cbranch_execz .LBB3_2
-; GFX10W64-NEXT:  ; %bb.1:
+; GFX10W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX10W64-NEXT:    s_cbranch_execz .LBB3_4
+; GFX10W64-NEXT:  ; %bb.3:
 ; GFX10W64-NEXT:    s_clause 0x1
 ; GFX10W64-NEXT:    s_load_dword s5, s[0:1], 0x44
 ; GFX10W64-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    v_mov_b32_e32 v5, s5
-; GFX10W64-NEXT:    buffer_atomic_add v4, v5, s[8:11], 0 idxen glc
-; GFX10W64-NEXT:  .LBB3_2:
+; GFX10W64-NEXT:    v_mov_b32_e32 v2, s5
+; GFX10W64-NEXT:    buffer_atomic_add v0, v2, s[8:11], 0 idxen glc
+; GFX10W64-NEXT:  .LBB3_4:
 ; GFX10W64-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX10W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX10W64-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W64-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W64-NEXT:    v_add_nc_u32_e32 v1, s2, v1
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W64-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W64-NEXT:    s_endpgm
 ;
 ; GFX10W32-LABEL: struct_add_i32_varying_vdata:
 ; GFX10W32:       ; %bb.0: ; %entry
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX10W32-NEXT:    ; implicit-def: $vgpr4
-; GFX10W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX10W32-NEXT:    s_cbranch_execz .LBB3_2
-; GFX10W32-NEXT:  ; %bb.1:
-; GFX10W32-NEXT:    s_mov_b32 s3, s4
+; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX10W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX10W32-NEXT:    s_mov_b32 s2, 0
+; GFX10W32-NEXT:    ; implicit-def: $vgpr1
+; GFX10W32-NEXT:  .LBB3_1: ; %ComputeLoop
+; GFX10W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W32-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX10W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX10W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX10W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX10W32-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX10W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX10W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX10W32-NEXT:    s_cbranch_scc1 .LBB3_1
+; GFX10W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
+; GFX10W32-NEXT:    ; implicit-def: $vgpr0
+; GFX10W32-NEXT:    s_and_saveexec_b32 s3, vcc_lo
+; GFX10W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX10W32-NEXT:    s_cbranch_execz .LBB3_4
+; GFX10W32-NEXT:  ; %bb.3:
 ; GFX10W32-NEXT:    s_clause 0x1
 ; GFX10W32-NEXT:    s_load_dword s8, s[0:1], 0x44
 ; GFX10W32-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, s3
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    v_mov_b32_e32 v5, s8
-; GFX10W32-NEXT:    buffer_atomic_add v4, v5, s[4:7], 0 idxen glc
-; GFX10W32-NEXT:  .LBB3_2:
+; GFX10W32-NEXT:    v_mov_b32_e32 v2, s8
+; GFX10W32-NEXT:    buffer_atomic_add v0, v2, s[4:7], 0 idxen glc
+; GFX10W32-NEXT:  .LBB3_4:
 ; GFX10W32-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX10W32-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W32-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W32-NEXT:    v_add_nc_u32_e32 v1, s2, v1
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W32-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W32-NEXT:    s_endpgm
 ;
 ; GFX11W64-LABEL: struct_add_i32_varying_vdata:
 ; GFX11W64:       ; %bb.0: ; %entry
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX11W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX11W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX11W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX11W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX11W64-NEXT:    ; implicit-def: $vgpr4
-; GFX11W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX11W64-NEXT:    s_cbranch_execz .LBB3_2
-; GFX11W64-NEXT:  ; %bb.1:
+; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    s_mov_b32 s4, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX11W64-NEXT:    ; implicit-def: $vgpr1
+; GFX11W64-NEXT:  .LBB3_1: ; %ComputeLoop
+; GFX11W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W64-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX11W64-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX11W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX11W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX11W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX11W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX11W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX11W64-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX11W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX11W64-NEXT:    s_cbranch_scc1 .LBB3_1
+; GFX11W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    ; implicit-def: $vgpr0
+; GFX11W64-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX11W64-NEXT:    s_cbranch_execz .LBB3_4
+; GFX11W64-NEXT:  ; %bb.3:
 ; GFX11W64-NEXT:    s_clause 0x1
 ; GFX11W64-NEXT:    s_load_b32 s5, s[0:1], 0x44
 ; GFX11W64-NEXT:    s_load_b128 s[8:11], s[0:1], 0x34
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    v_mov_b32_e32 v5, s5
-; GFX11W64-NEXT:    buffer_atomic_add_u32 v4, v5, s[8:11], 0 idxen glc
-; GFX11W64-NEXT:  .LBB3_2:
+; GFX11W64-NEXT:    v_mov_b32_e32 v2, s5
+; GFX11W64-NEXT:    buffer_atomic_add_u32 v0, v2, s[8:11], 0 idxen glc
+; GFX11W64-NEXT:  .LBB3_4:
 ; GFX11W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX11W64-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    v_add_nc_u32_e32 v1, s2, v1
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W64-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W64-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W64-NEXT:    s_endpgm
 ;
 ; GFX11W32-LABEL: struct_add_i32_varying_vdata:
 ; GFX11W32:       ; %bb.0: ; %entry
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX11W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX11W32-NEXT:    ; implicit-def: $vgpr4
-; GFX11W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX11W32-NEXT:    s_cbranch_execz .LBB3_2
-; GFX11W32-NEXT:  ; %bb.1:
-; GFX11W32-NEXT:    s_mov_b32 s3, s4
+; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    s_mov_b32 s2, 0
+; GFX11W32-NEXT:    ; implicit-def: $vgpr1
+; GFX11W32-NEXT:  .LBB3_1: ; %ComputeLoop
+; GFX11W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W32-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX11W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX11W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX11W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX11W32-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX11W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX11W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX11W32-NEXT:    s_cbranch_scc1 .LBB3_1
+; GFX11W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    ; implicit-def: $vgpr0
+; GFX11W32-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX11W32-NEXT:    s_cbranch_execz .LBB3_4
+; GFX11W32-NEXT:  ; %bb.3:
 ; GFX11W32-NEXT:    s_clause 0x1
 ; GFX11W32-NEXT:    s_load_b32 s8, s[0:1], 0x44
 ; GFX11W32-NEXT:    s_load_b128 s[4:7], s[0:1], 0x34
+; GFX11W32-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    v_dual_mov_b32 v4, s3 :: v_dual_mov_b32 v5, s8
-; GFX11W32-NEXT:    buffer_atomic_add_u32 v4, v5, s[4:7], 0 idxen glc
-; GFX11W32-NEXT:  .LBB3_2:
-; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX11W32-NEXT:    v_mov_b32_e32 v2, s8
+; GFX11W32-NEXT:    buffer_atomic_add_u32 v0, v2, s[4:7], 0 idxen glc
+; GFX11W32-NEXT:  .LBB3_4:
+; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX11W32-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W32-NEXT:    v_mov_b32_e32 v4, v3
+; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX11W32-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_add_nc_u32 v1, s2, v1
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W32-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W32-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W32-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = call i32 @llvm.amdgcn.struct.buffer.atomic.add(i32 %lane, <4 x i32> %inout, i32 %vindex, i32 0, i32 0, i32 0)
   store i32 %old, ptr addrspace(1) %out
   ret void
 }

 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    s_waitcnt vmcnt(0)
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: sub_i32_varying_vdata:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX8-NEXT:    v_mov_b32_e32 v2, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v2, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s4, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB7_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_add_i32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB7_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB7_2
-; GFX8-NEXT:  ; %bb.1:
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB7_4
+; GFX8-NEXT:  ; %bb.3:
 ; GFX8-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    buffer_atomic_sub v0, off, s[8:11], 0 glc
-; GFX8-NEXT:  .LBB7_2:
+; GFX8-NEXT:  .LBB7_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v1
-; GFX8-NEXT:    v_sub_u32_e32 v0, vcc, s2, v0
+; GFX8-NEXT:    v_sub_u32_e32 v2, vcc, s2, v1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    v_mov_b32_e32 v4, s1
-; GFX8-NEXT:    v_mov_b32_e32 v3, s0
-; GFX8-NEXT:    flat_store_dword v[3:4], v0
+; GFX8-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8-NEXT:    flat_store_dword v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: sub_i32_varying_vdata:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s4, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB7_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_add_i32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB7_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB7_2
-; GFX9-NEXT:  ; %bb.1:
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB7_4
+; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    buffer_atomic_sub v0, off, s[8:11], 0 glc
-; GFX9-NEXT:  .LBB7_2:
+; GFX9-NEXT:  .LBB7_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v2
-; GFX9-NEXT:    v_sub_u32_e32 v0, s2, v0
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    v_sub_u32_e32 v0, s2, v1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    global_store_dword v3, v0, s[0:1]
+; GFX9-NEXT:    global_store_dword v2, v0, s[0:1]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX10W64-LABEL: sub_i32_varying_vdata:
 ; GFX10W64:       ; %bb.0: ; %entry
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX10W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX10W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX10W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX10W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX10W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX10W64-NEXT:    ; implicit-def: $vgpr4
+; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX10W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX10W64-NEXT:    s_mov_b32 s4, 0
+; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX10W64-NEXT:    ; implicit-def: $vgpr1
+; GFX10W64-NEXT:  .LBB7_1: ; %ComputeLoop
+; GFX10W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W64-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX10W64-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX10W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX10W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX10W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX10W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX10W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX10W64-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX10W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX10W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX10W64-NEXT:    s_cbranch_scc1 .LBB7_1
+; GFX10W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
+; GFX10W64-NEXT:    ; implicit-def: $vgpr0
 ; GFX10W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX10W64-NEXT:    s_cbranch_execz .LBB7_2
-; GFX10W64-NEXT:  ; %bb.1:
+; GFX10W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX10W64-NEXT:    s_cbranch_execz .LBB7_4
+; GFX10W64-NEXT:  ; %bb.3:
 ; GFX10W64-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    buffer_atomic_sub v4, off, s[8:11], 0 glc
-; GFX10W64-NEXT:  .LBB7_2:
+; GFX10W64-NEXT:    buffer_atomic_sub v0, off, s[8:11], 0 glc
+; GFX10W64-NEXT:  .LBB7_4:
 ; GFX10W64-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX10W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX10W64-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W64-NEXT:    v_sub_nc_u32_e32 v4, s2, v4
+; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W64-NEXT:    v_sub_nc_u32_e32 v1, s2, v1
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W64-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W64-NEXT:    s_endpgm
 ;
 ; GFX10W32-LABEL: sub_i32_varying_vdata:
 ; GFX10W32:       ; %bb.0: ; %entry
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX10W32-NEXT:    ; implicit-def: $vgpr4
-; GFX10W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX10W32-NEXT:    s_cbranch_execz .LBB7_2
-; GFX10W32-NEXT:  ; %bb.1:
-; GFX10W32-NEXT:    s_mov_b32 s3, s4
+; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX10W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX10W32-NEXT:    s_mov_b32 s2, 0
+; GFX10W32-NEXT:    ; implicit-def: $vgpr1
+; GFX10W32-NEXT:  .LBB7_1: ; %ComputeLoop
+; GFX10W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W32-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX10W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX10W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX10W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX10W32-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX10W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX10W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX10W32-NEXT:    s_cbranch_scc1 .LBB7_1
+; GFX10W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
+; GFX10W32-NEXT:    ; implicit-def: $vgpr0
+; GFX10W32-NEXT:    s_and_saveexec_b32 s3, vcc_lo
+; GFX10W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX10W32-NEXT:    s_cbranch_execz .LBB7_4
+; GFX10W32-NEXT:  ; %bb.3:
 ; GFX10W32-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, s3
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    buffer_atomic_sub v4, off, s[4:7], 0 glc
-; GFX10W32-NEXT:  .LBB7_2:
+; GFX10W32-NEXT:    buffer_atomic_sub v0, off, s[4:7], 0 glc
+; GFX10W32-NEXT:  .LBB7_4:
 ; GFX10W32-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX10W32-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W32-NEXT:    v_sub_nc_u32_e32 v4, s2, v4
+; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W32-NEXT:    v_sub_nc_u32_e32 v1, s2, v1
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W32-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W32-NEXT:    s_endpgm
 ;
 ; GFX11W64-LABEL: sub_i32_varying_vdata:
 ; GFX11W64:       ; %bb.0: ; %entry
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX11W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX11W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX11W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX11W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX11W64-NEXT:    ; implicit-def: $vgpr4
-; GFX11W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX11W64-NEXT:    s_cbranch_execz .LBB7_2
-; GFX11W64-NEXT:  ; %bb.1:
+; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    s_mov_b32 s4, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX11W64-NEXT:    ; implicit-def: $vgpr1
+; GFX11W64-NEXT:  .LBB7_1: ; %ComputeLoop
+; GFX11W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W64-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX11W64-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX11W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX11W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX11W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX11W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX11W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX11W64-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX11W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX11W64-NEXT:    s_cbranch_scc1 .LBB7_1
+; GFX11W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    ; implicit-def: $vgpr0
+; GFX11W64-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX11W64-NEXT:    s_cbranch_execz .LBB7_4
+; GFX11W64-NEXT:  ; %bb.3:
 ; GFX11W64-NEXT:    s_load_b128 s[8:11], s[0:1], 0x34
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    buffer_atomic_sub_u32 v4, off, s[8:11], 0 glc
-; GFX11W64-NEXT:  .LBB7_2:
+; GFX11W64-NEXT:    buffer_atomic_sub_u32 v0, off, s[8:11], 0 glc
+; GFX11W64-NEXT:  .LBB7_4:
 ; GFX11W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX11W64-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W64-NEXT:    v_sub_nc_u32_e32 v4, s2, v4
+; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    v_sub_nc_u32_e32 v1, s2, v1
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W64-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W64-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W64-NEXT:    s_endpgm
 ;
 ; GFX11W32-LABEL: sub_i32_varying_vdata:
 ; GFX11W32:       ; %bb.0: ; %entry
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX11W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX11W32-NEXT:    ; implicit-def: $vgpr4
-; GFX11W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX11W32-NEXT:    s_cbranch_execz .LBB7_2
-; GFX11W32-NEXT:  ; %bb.1:
-; GFX11W32-NEXT:    s_mov_b32 s3, s4
+; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    s_mov_b32 s2, 0
+; GFX11W32-NEXT:    ; implicit-def: $vgpr1
+; GFX11W32-NEXT:  .LBB7_1: ; %ComputeLoop
+; GFX11W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W32-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX11W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX11W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX11W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX11W32-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX11W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX11W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX11W32-NEXT:    s_cbranch_scc1 .LBB7_1
+; GFX11W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    ; implicit-def: $vgpr0
+; GFX11W32-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX11W32-NEXT:    s_cbranch_execz .LBB7_4
+; GFX11W32-NEXT:  ; %bb.3:
 ; GFX11W32-NEXT:    s_load_b128 s[4:7], s[0:1], 0x34
-; GFX11W32-NEXT:    v_mov_b32_e32 v4, s3
+; GFX11W32-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    buffer_atomic_sub_u32 v4, off, s[4:7], 0 glc
-; GFX11W32-NEXT:  .LBB7_2:
-; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX11W32-NEXT:    buffer_atomic_sub_u32 v0, off, s[4:7], 0 glc
+; GFX11W32-NEXT:  .LBB7_4:
+; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX11W32-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W32-NEXT:    v_mov_b32_e32 v4, v3
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W32-NEXT:    v_sub_nc_u32_e32 v4, s2, v4
+; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX11W32-NEXT:    v_mov_b32_e32 v0, 0
+; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W32-NEXT:    v_sub_nc_u32_e32 v1, s2, v1
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W32-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W32-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W32-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = call i32 @llvm.amdgcn.raw.buffer.atomic.sub(i32 %lane, <4 x i32> %inout, i32 0, i32 0, i32 0)
   store i32 %old, ptr addrspace(1) %out
   ret void
 }

llvm/test/CodeGen/AMDGPU/atomic_optimizations_global_pointer.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -march=amdgcn -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=GFX7LESS %s
-; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX89,GFX8 %s
-; RUN: llc -march=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX89,GFX9 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX1064 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX1032 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=-wavefrontsize32,+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX1164 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=+wavefrontsize32,-wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX1132 %s
+; RUN: llc -march=amdgcn -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=GFX7LESS %s
+; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX89,GFX8 %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX89,GFX9 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX1064 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX1032 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=-wavefrontsize32,+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX1164 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=+wavefrontsize32,-wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX1132 %s
 
 declare i32 @llvm.amdgcn.workitem.id.x()
 
 ; Show what the atomic optimization pass will do for global pointers.
 
 define amdgpu_kernel void @add_i32_constant(ptr addrspace(1) %out, ptr addrspace(1) %inout) {
 ; GFX7LESS-LABEL: add_i32_constant:
 ; GFX7LESS:       ; %bb.0: ; %entry

 ; GFX7LESS-NEXT:    buffer_wbinvl1
 ; GFX7LESS-NEXT:    s_mov_b32 s4, s0
 ; GFX7LESS-NEXT:    s_mov_b32 s5, s1
 ; GFX7LESS-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX7LESS-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: add_i32_varying:
 ; GFX8:       ; %bb.0: ; %entry
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s6, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s2
+; GFX8-NEXT:    s_add_i32 s4, s4, 32
+; GFX8-NEXT:    s_min_u32 s7, s5, s4
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s7
+; GFX8-NEXT:    s_lshl_b64 s[4:5], 1, s7
+; GFX8-NEXT:    s_mov_b32 m0, s7
+; GFX8-NEXT:    v_writelane_b32 v1, s6, m0
+; GFX8-NEXT:    s_add_i32 s6, s6, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[4:5]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
 ; GFX8-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; GFX8-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX8-NEXT:    v_mov_b32_e32 v2, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s6, v2, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB2_2
-; GFX8-NEXT:  ; %bb.1:
+; GFX8-NEXT:    s_xor_b64 s[4:5], exec, s[4:5]
+; GFX8-NEXT:    s_cbranch_execz .LBB2_4
+; GFX8-NEXT:  ; %bb.3:
 ; GFX8-NEXT:    s_mov_b32 s11, 0xf000
 ; GFX8-NEXT:    s_mov_b32 s10, -1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_mov_b32 s8, s2
 ; GFX8-NEXT:    s_mov_b32 s9, s3
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s6
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    buffer_atomic_add v0, off, s[8:11], 0 glc
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    buffer_wbinvl1_vol
-; GFX8-NEXT:  .LBB2_2:
+; GFX8-NEXT:  .LBB2_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; GFX8-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX8-NEXT:    s_mov_b32 s2, -1
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s4, v0
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s4, v1
 ; GFX8-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: add_i32_varying:
 ; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s6, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s2
+; GFX9-NEXT:    s_add_i32 s4, s4, 32
+; GFX9-NEXT:    s_min_u32 s7, s5, s4
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s7
+; GFX9-NEXT:    s_lshl_b64 s[4:5], 1, s7
+; GFX9-NEXT:    s_mov_b32 m0, s7
+; GFX9-NEXT:    v_writelane_b32 v1, s6, m0
+; GFX9-NEXT:    s_add_i32 s6, s6, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[4:5]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; GFX9-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX9-NEXT:    v_mov_b32_e32 v2, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX9-NEXT:    v_add_u32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s6, v2, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB2_2
-; GFX9-NEXT:  ; %bb.1:
+; GFX9-NEXT:    s_xor_b64 s[4:5], exec, s[4:5]
+; GFX9-NEXT:    s_cbranch_execz .LBB2_4
+; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    s_mov_b32 s11, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s10, -1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_mov_b32 s8, s2
 ; GFX9-NEXT:    s_mov_b32 s9, s3
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s6
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    buffer_atomic_add v0, off, s[8:11], 0 glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    buffer_wbinvl1_vol
-; GFX9-NEXT:  .LBB2_2:
+; GFX9-NEXT:  .LBB2_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; GFX9-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s2, -1
-; GFX9-NEXT:    v_add_u32_e32 v0, s4, v0
+; GFX9-NEXT:    v_add_u32_e32 v0, s4, v1
 ; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: add_i32_varying:
 ; GFX1064:       ; %bb.0: ; %entry
-; GFX1064-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1064-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s6, v1, 15
-; GFX1064-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1064-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1064-NEXT:    s_mov_b32 s6, 0
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1064-NEXT:    ; implicit-def: $vgpr1
+; GFX1064-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX1064-NEXT:    s_ff1_i32_b32 s5, s2
+; GFX1064-NEXT:    s_add_i32 s4, s4, 32
+; GFX1064-NEXT:    s_min_u32 s7, s5, s4
+; GFX1064-NEXT:    v_readlane_b32 s8, v0, s7
+; GFX1064-NEXT:    s_lshl_b64 s[4:5], 1, s7
+; GFX1064-NEXT:    v_writelane_b32 v1, s6, s7
+; GFX1064-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[4:5]
+; GFX1064-NEXT:    s_add_i32 s6, s6, s8
+; GFX1064-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX1064-NEXT:  ; %bb.2: ; %ComputeEnd
 ; GFX1064-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT:    v_readlane_b32 s7, v1, 31
-; GFX1064-NEXT:    v_writelane_b32 v3, s6, 16
-; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT:    v_readlane_b32 s8, v1, 47
-; GFX1064-NEXT:    v_readlane_b32 s9, v1, 63
-; GFX1064-NEXT:    v_writelane_b32 v3, s7, 32
-; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[6:7], -1
-; GFX1064-NEXT:    s_mov_b32 s4, s9
-; GFX1064-NEXT:    v_writelane_b32 v3, s8, 48
-; GFX1064-NEXT:    s_mov_b64 exec, s[6:7]
-; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT:    s_mov_b32 s6, -1
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX1064-NEXT:    ; implicit-def: $vgpr0
-; GFX1064-NEXT:    s_and_saveexec_b64 s[8:9], vcc
-; GFX1064-NEXT:    s_cbranch_execz .LBB2_2
-; GFX1064-NEXT:  ; %bb.1:
-; GFX1064-NEXT:    v_mov_b32_e32 v0, s4
-; GFX1064-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX1064-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; GFX1064-NEXT:    s_xor_b64 s[4:5], exec, s[4:5]
+; GFX1064-NEXT:    s_cbranch_execz .LBB2_4
+; GFX1064-NEXT:  ; %bb.3:
+; GFX1064-NEXT:    v_mov_b32_e32 v0, s6
+; GFX1064-NEXT:    s_mov_b32 s11, 0x31016000
+; GFX1064-NEXT:    s_mov_b32 s10, -1
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT:    s_mov_b32 s4, s2
-; GFX1064-NEXT:    s_mov_b32 s5, s3
+; GFX1064-NEXT:    s_mov_b32 s8, s2
+; GFX1064-NEXT:    s_mov_b32 s9, s3
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT:    buffer_atomic_add v0, off, s[4:7], 0 glc
+; GFX1064-NEXT:    buffer_atomic_add v0, off, s[8:11], 0 glc
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1064-NEXT:    buffer_gl0_inv
 ; GFX1064-NEXT:    buffer_gl1_inv
-; GFX1064-NEXT:  .LBB2_2:
+; GFX1064-NEXT:  .LBB2_4:
 ; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1064-NEXT:    s_or_b64 exec, exec, s[8:9]
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v0, v3
 ; GFX1064-NEXT:    s_mov_b32 s3, 0x31016000
-; GFX1064-NEXT:    v_add_nc_u32_e32 v0, s2, v0
-; GFX1064-NEXT:    s_mov_b32 s2, s6
+; GFX1064-NEXT:    v_add_nc_u32_e32 v0, s2, v1
+; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1064-NEXT:    s_endpgm
 ;
 ; GFX1032-LABEL: add_i32_varying:
 ; GFX1032:       ; %bb.0: ; %entry
-; GFX1032-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1032-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1032-NEXT:    s_mov_b32 s2, exec_lo
+; GFX1032-NEXT:    s_mov_b32 s4, 0
+; GFX1032-NEXT:    ; implicit-def: $vgpr1
+; GFX1032-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_ff1_i32_b32 s3, s2
+; GFX1032-NEXT:    v_readlane_b32 s5, v0, s3
+; GFX1032-NEXT:    s_lshl_b32 s6, 1, s3
+; GFX1032-NEXT:    v_writelane_b32 v1, s4, s3
+; GFX1032-NEXT:    s_andn2_b32 s2, s2, s6
+; GFX1032-NEXT:    s_add_i32 s4, s4, s5
+; GFX1032-NEXT:    s_cmp_lg_u32 s2, 0
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX1032-NEXT:  ; %bb.2: ; %ComputeEnd
 ; GFX1032-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; GFX1032-NEXT:    s_or_saveexec_b32 s4, -1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1032-NEXT:    v_readlane_b32 s5, v1, 15
-; GFX1032-NEXT:    v_readlane_b32 s6, v1, 31
-; GFX1032-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s4
-; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-NEXT:    s_or_saveexec_b32 s4, -1
-; GFX1032-NEXT:    v_writelane_b32 v3, s5, 16
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s4
-; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT:    s_mov_b32 s4, s6
-; GFX1032-NEXT:    s_mov_b32 s6, -1
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
 ; GFX1032-NEXT:    ; implicit-def: $vgpr0
-; GFX1032-NEXT:    s_and_saveexec_b32 s8, vcc_lo
-; GFX1032-NEXT:    s_cbranch_execz .LBB2_2
-; GFX1032-NEXT:  ; %bb.1:
+; GFX1032-NEXT:    s_and_saveexec_b32 s5, vcc_lo
+; GFX1032-NEXT:    s_xor_b32 s5, exec_lo, s5
+; GFX1032-NEXT:    s_cbranch_execz .LBB2_4
+; GFX1032-NEXT:  ; %bb.3:
 ; GFX1032-NEXT:    v_mov_b32_e32 v0, s4
-; GFX1032-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX1032-NEXT:    s_mov_b32 s11, 0x31016000
+; GFX1032-NEXT:    s_mov_b32 s10, -1
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT:    s_mov_b32 s4, s2
-; GFX1032-NEXT:    s_mov_b32 s5, s3
+; GFX1032-NEXT:    s_mov_b32 s8, s2
+; GFX1032-NEXT:    s_mov_b32 s9, s3
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT:    buffer_atomic_add v0, off, s[4:7], 0 glc
+; GFX1032-NEXT:    buffer_atomic_add v0, off, s[8:11], 0 glc
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1032-NEXT:    buffer_gl0_inv
 ; GFX1032-NEXT:    buffer_gl1_inv
-; GFX1032-NEXT:  .LBB2_2:
+; GFX1032-NEXT:  .LBB2_4:
 ; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s8
+; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s5
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX1032-NEXT:    v_mov_b32_e32 v0, v3
 ; GFX1032-NEXT:    s_mov_b32 s3, 0x31016000
-; GFX1032-NEXT:    v_add_nc_u32_e32 v0, s2, v0
-; GFX1032-NEXT:    s_mov_b32 s2, s6
+; GFX1032-NEXT:    v_add_nc_u32_e32 v0, s2, v1
+; GFX1032-NEXT:    s_mov_b32 s2, -1
 ; GFX1032-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1164-LABEL: add_i32_varying:
 ; GFX1164:       ; %bb.0: ; %entry
-; GFX1164-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1164-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1164-NEXT:    s_mov_b32 s6, 0
 ; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1164-NEXT:    v_readlane_b32 s6, v1, 15
-; GFX1164-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
+; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1164-NEXT:    ; implicit-def: $vgpr1
+; GFX1164-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX1164-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1164-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX1164-NEXT:    s_ctz_i32_b32 s5, s2
+; GFX1164-NEXT:    s_add_i32 s4, s4, 32
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT:    s_min_u32 s7, s5, s4
+; GFX1164-NEXT:    v_readlane_b32 s8, v0, s7
+; GFX1164-NEXT:    s_lshl_b64 s[4:5], 1, s7
+; GFX1164-NEXT:    v_writelane_b32 v1, s6, s7
+; GFX1164-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[4:5]
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX1164-NEXT:    s_add_i32 s6, s6, s8
+; GFX1164-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1164-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX1164-NEXT:  ; %bb.2: ; %ComputeEnd
 ; GFX1164-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
-; GFX1164-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1164-NEXT:    v_readlane_b32 s7, v1, 31
-; GFX1164-NEXT:    v_writelane_b32 v3, s6, 16
-; GFX1164-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1164-NEXT:    v_readlane_b32 s8, v1, 47
-; GFX1164-NEXT:    v_readlane_b32 s9, v1, 63
-; GFX1164-NEXT:    v_writelane_b32 v3, s7, 32
-; GFX1164-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[6:7], -1
-; GFX1164-NEXT:    s_mov_b32 s4, s9
-; GFX1164-NEXT:    v_writelane_b32 v3, s8, 48
-; GFX1164-NEXT:    s_mov_b64 exec, s[6:7]
-; GFX1164-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1164-NEXT:    s_mov_b32 s6, -1
+; GFX1164-NEXT:    s_mov_b64 s[4:5], exec
 ; GFX1164-NEXT:    ; implicit-def: $vgpr0
-; GFX1164-NEXT:    s_and_saveexec_b64 s[8:9], vcc
-; GFX1164-NEXT:    s_cbranch_execz .LBB2_2
-; GFX1164-NEXT:  ; %bb.1:
-; GFX1164-NEXT:    v_mov_b32_e32 v0, s4
-; GFX1164-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1164-NEXT:    s_xor_b64 s[4:5], exec, s[4:5]
+; GFX1164-NEXT:    s_cbranch_execz .LBB2_4
+; GFX1164-NEXT:  ; %bb.3:
+; GFX1164-NEXT:    v_mov_b32_e32 v0, s6
+; GFX1164-NEXT:    s_mov_b32 s11, 0x31016000
+; GFX1164-NEXT:    s_mov_b32 s10, -1
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX1164-NEXT:    s_mov_b32 s4, s2
-; GFX1164-NEXT:    s_mov_b32 s5, s3
+; GFX1164-NEXT:    s_mov_b32 s8, s2
+; GFX1164-NEXT:    s_mov_b32 s9, s3
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1164-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1164-NEXT:    buffer_atomic_add_u32 v0, off, s[4:7], 0 glc
+; GFX1164-NEXT:    buffer_atomic_add_u32 v0, off, s[8:11], 0 glc
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1164-NEXT:    buffer_gl0_inv
 ; GFX1164-NEXT:    buffer_gl1_inv
-; GFX1164-NEXT:  .LBB2_2:
-; GFX1164-NEXT:    s_or_b64 exec, exec, s[8:9]
+; GFX1164-NEXT:  .LBB2_4:
+; GFX1164-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v0, v3
 ; GFX1164-NEXT:    s_mov_b32 s3, 0x31016000
 ; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_e32 v0, s2, v0
-; GFX1164-NEXT:    s_mov_b32 s2, s6
+; GFX1164-NEXT:    v_add_nc_u32_e32 v0, s2, v1
+; GFX1164-NEXT:    s_mov_b32 s2, -1
 ; GFX1164-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1164-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1164-NEXT:    s_endpgm
 ;
 ; GFX1132-LABEL: add_i32_varying:
 ; GFX1132:       ; %bb.0: ; %entry
-; GFX1132-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1132-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
+; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1132-NEXT:    s_mov_b32 s2, exec_lo
+; GFX1132-NEXT:    s_mov_b32 s4, 0
+; GFX1132-NEXT:    ; implicit-def: $vgpr1
+; GFX1132-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX1132-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1132-NEXT:    s_ctz_i32_b32 s3, s2
+; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX1132-NEXT:    v_readlane_b32 s5, v0, s3
+; GFX1132-NEXT:    s_lshl_b32 s6, 1, s3
+; GFX1132-NEXT:    v_writelane_b32 v1, s4, s3
+; GFX1132-NEXT:    s_and_not1_b32 s2, s2, s6
+; GFX1132-NEXT:    s_add_i32 s4, s4, s5
+; GFX1132-NEXT:    s_cmp_lg_u32 s2, 0
+; GFX1132-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX1132-NEXT:  ; %bb.2: ; %ComputeEnd
 ; GFX1132-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
-; GFX1132-NEXT:    s_or_saveexec_b32 s4, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1132-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1132-NEXT:    v_readlane_b32 s5, v1, 15
-; GFX1132-NEXT:    v_readlane_b32 s6, v1, 31
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_3) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
-; GFX1132-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s4
-; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1132-NEXT:    s_or_saveexec_b32 s4, -1
-; GFX1132-NEXT:    v_writelane_b32 v3, s5, 16
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s4
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_2)
-; GFX1132-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1132-NEXT:    s_mov_b32 s4, s6
-; GFX1132-NEXT:    s_mov_b32 s6, -1
+; GFX1132-NEXT:    s_mov_b32 s5, exec_lo
 ; GFX1132-NEXT:    ; implicit-def: $vgpr0
-; GFX1132-NEXT:    s_and_saveexec_b32 s8, vcc_lo
-; GFX1132-NEXT:    s_cbranch_execz .LBB2_2
-; GFX1132-NEXT:  ; %bb.1:
+; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1132-NEXT:    s_xor_b32 s5, exec_lo, s5
+; GFX1132-NEXT:    s_cbranch_execz .LBB2_4
+; GFX1132-NEXT:  ; %bb.3:
 ; GFX1132-NEXT:    v_mov_b32_e32 v0, s4
-; GFX1132-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX1132-NEXT:    s_mov_b32 s11, 0x31016000
+; GFX1132-NEXT:    s_mov_b32 s10, -1
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX1132-NEXT:    s_mov_b32 s4, s2
-; GFX1132-NEXT:    s_mov_b32 s5, s3
+; GFX1132-NEXT:    s_mov_b32 s8, s2
+; GFX1132-NEXT:    s_mov_b32 s9, s3
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1132-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1132-NEXT:    buffer_atomic_add_u32 v0, off, s[4:7], 0 glc
+; GFX1132-NEXT:    buffer_atomic_add_u32 v0, off, s[8:11], 0 glc
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1132-NEXT:    buffer_gl0_inv
 ; GFX1132-NEXT:    buffer_gl1_inv
-; GFX1132-NEXT:  .LBB2_2:
-; GFX1132-NEXT:    s_or_b32 exec_lo, exec_lo, s8
+; GFX1132-NEXT:  .LBB2_4:
+; GFX1132-NEXT:    s_or_b32 exec_lo, exec_lo, s5
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX1132-NEXT:    v_mov_b32_e32 v0, v3
 ; GFX1132-NEXT:    s_mov_b32 s3, 0x31016000
 ; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_e32 v0, s2, v0
-; GFX1132-NEXT:    s_mov_b32 s2, s6
+; GFX1132-NEXT:    v_add_nc_u32_e32 v0, s2, v1
+; GFX1132-NEXT:    s_mov_b32 s2, -1
 ; GFX1132-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1132-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1132-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = atomicrmw add ptr addrspace(1) %inout, i32 %lane acq_rel
   store i32 %old, ptr addrspace(1) %out
   ret void

 ; GFX7LESS-NEXT:    buffer_wbinvl1
 ; GFX7LESS-NEXT:    s_mov_b32 s4, s0
 ; GFX7LESS-NEXT:    s_mov_b32 s5, s1
 ; GFX7LESS-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX7LESS-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: sub_i32_varying:
 ; GFX8:       ; %bb.0: ; %entry
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s6, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB8_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s2
+; GFX8-NEXT:    s_add_i32 s4, s4, 32
+; GFX8-NEXT:    s_min_u32 s7, s5, s4
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s7
+; GFX8-NEXT:    s_lshl_b64 s[4:5], 1, s7
+; GFX8-NEXT:    s_mov_b32 m0, s7
+; GFX8-NEXT:    v_writelane_b32 v1, s6, m0
+; GFX8-NEXT:    s_add_i32 s6, s6, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[4:5]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB8_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
 ; GFX8-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; GFX8-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX8-NEXT:    v_mov_b32_e32 v2, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s6, v2, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB8_2
-; GFX8-NEXT:  ; %bb.1:
+; GFX8-NEXT:    s_xor_b64 s[4:5], exec, s[4:5]
+; GFX8-NEXT:    s_cbranch_execz .LBB8_4
+; GFX8-NEXT:  ; %bb.3:
 ; GFX8-NEXT:    s_mov_b32 s11, 0xf000
 ; GFX8-NEXT:    s_mov_b32 s10, -1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_mov_b32 s8, s2
 ; GFX8-NEXT:    s_mov_b32 s9, s3
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s6
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    buffer_atomic_sub v0, off, s[8:11], 0 glc
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    buffer_wbinvl1_vol
-; GFX8-NEXT:  .LBB8_2:
+; GFX8-NEXT:  .LBB8_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; GFX8-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX8-NEXT:    s_mov_b32 s2, -1
-; GFX8-NEXT:    v_sub_u32_e32 v0, vcc, s4, v0
+; GFX8-NEXT:    v_sub_u32_e32 v0, vcc, s4, v1
 ; GFX8-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: sub_i32_varying:
 ; GFX9:       ; %bb.0: ; %entry
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s6, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB8_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s2
+; GFX9-NEXT:    s_add_i32 s4, s4, 32
+; GFX9-NEXT:    s_min_u32 s7, s5, s4
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s7
+; GFX9-NEXT:    s_lshl_b64 s[4:5], 1, s7
+; GFX9-NEXT:    s_mov_b32 m0, s7
+; GFX9-NEXT:    v_writelane_b32 v1, s6, m0
+; GFX9-NEXT:    s_add_i32 s6, s6, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[4:5]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB8_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; GFX9-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX9-NEXT:    v_mov_b32_e32 v2, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX9-NEXT:    v_add_u32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s6, v2, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB8_2
-; GFX9-NEXT:  ; %bb.1:
+; GFX9-NEXT:    s_xor_b64 s[4:5], exec, s[4:5]
+; GFX9-NEXT:    s_cbranch_execz .LBB8_4
+; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    s_mov_b32 s11, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s10, -1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_mov_b32 s8, s2
 ; GFX9-NEXT:    s_mov_b32 s9, s3
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s6
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    buffer_atomic_sub v0, off, s[8:11], 0 glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    buffer_wbinvl1_vol
-; GFX9-NEXT:  .LBB8_2:
+; GFX9-NEXT:  .LBB8_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; GFX9-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s2, -1
-; GFX9-NEXT:    v_sub_u32_e32 v0, s4, v0
+; GFX9-NEXT:    v_sub_u32_e32 v0, s4, v1
 ; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: sub_i32_varying:
 ; GFX1064:       ; %bb.0: ; %entry
-; GFX1064-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1064-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s6, v1, 15
-; GFX1064-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1064-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1064-NEXT:    s_mov_b32 s6, 0
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1064-NEXT:    ; implicit-def: $vgpr1
+; GFX1064-NEXT:  .LBB8_1: ; %ComputeLoop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX1064-NEXT:    s_ff1_i32_b32 s5, s2
+; GFX1064-NEXT:    s_add_i32 s4, s4, 32
+; GFX1064-NEXT:    s_min_u32 s7, s5, s4
+; GFX1064-NEXT:    v_readlane_b32 s8, v0, s7
+; GFX1064-NEXT:    s_lshl_b64 s[4:5], 1, s7
+; GFX1064-NEXT:    v_writelane_b32 v1, s6, s7
+; GFX1064-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[4:5]
+; GFX1064-NEXT:    s_add_i32 s6, s6, s8
+; GFX1064-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB8_1
+; GFX1064-NEXT:  ; %bb.2: ; %ComputeEnd
 ; GFX1064-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT:    v_readlane_b32 s7, v1, 31
-; GFX1064-NEXT:    v_writelane_b32 v3, s6, 16
-; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT:    v_readlane_b32 s8, v1, 47
-; GFX1064-NEXT:    v_readlane_b32 s9, v1, 63
-; GFX1064-NEXT:    v_writelane_b32 v3, s7, 32
-; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[6:7], -1
-; GFX1064-NEXT:    s_mov_b32 s4, s9
-; GFX1064-NEXT:    v_writelane_b32 v3, s8, 48
-; GFX1064-NEXT:    s_mov_b64 exec, s[6:7]
-; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT:    s_mov_b32 s6, -1
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX1064-NEXT:    ; implicit-def: $vgpr0
-; GFX1064-NEXT:    s_and_saveexec_b64 s[8:9], vcc
-; GFX1064-NEXT:    s_cbranch_execz .LBB8_2
-; GFX1064-NEXT:  ; %bb.1:
-; GFX1064-NEXT:    v_mov_b32_e32 v0, s4
-; GFX1064-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX1064-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; GFX1064-NEXT:    s_xor_b64 s[4:5], exec, s[4:5]
+; GFX1064-NEXT:    s_cbranch_execz .LBB8_4
+; GFX1064-NEXT:  ; %bb.3:
+; GFX1064-NEXT:    v_mov_b32_e32 v0, s6
+; GFX1064-NEXT:    s_mov_b32 s11, 0x31016000
+; GFX1064-NEXT:    s_mov_b32 s10, -1
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT:    s_mov_b32 s4, s2
-; GFX1064-NEXT:    s_mov_b32 s5, s3
+; GFX1064-NEXT:    s_mov_b32 s8, s2
+; GFX1064-NEXT:    s_mov_b32 s9, s3
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT:    buffer_atomic_sub v0, off, s[4:7], 0 glc
+; GFX1064-NEXT:    buffer_atomic_sub v0, off, s[8:11], 0 glc
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1064-NEXT:    buffer_gl0_inv
 ; GFX1064-NEXT:    buffer_gl1_inv
-; GFX1064-NEXT:  .LBB8_2:
+; GFX1064-NEXT:  .LBB8_4:
 ; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1064-NEXT:    s_or_b64 exec, exec, s[8:9]
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v0, v3
 ; GFX1064-NEXT:    s_mov_b32 s3, 0x31016000
-; GFX1064-NEXT:    v_sub_nc_u32_e32 v0, s2, v0
-; GFX1064-NEXT:    s_mov_b32 s2, s6
+; GFX1064-NEXT:    v_sub_nc_u32_e32 v0, s2, v1
+; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1064-NEXT:    s_endpgm
 ;
 ; GFX1032-LABEL: sub_i32_varying:
 ; GFX1032:       ; %bb.0: ; %entry
-; GFX1032-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1032-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1032-NEXT:    s_mov_b32 s2, exec_lo
+; GFX1032-NEXT:    s_mov_b32 s4, 0
+; GFX1032-NEXT:    ; implicit-def: $vgpr1
+; GFX1032-NEXT:  .LBB8_1: ; %ComputeLoop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_ff1_i32_b32 s3, s2
+; GFX1032-NEXT:    v_readlane_b32 s5, v0, s3
+; GFX1032-NEXT:    s_lshl_b32 s6, 1, s3
+; GFX1032-NEXT:    v_writelane_b32 v1, s4, s3
+; GFX1032-NEXT:    s_andn2_b32 s2, s2, s6
+; GFX1032-NEXT:    s_add_i32 s4, s4, s5
+; GFX1032-NEXT:    s_cmp_lg_u32 s2, 0
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB8_1
+; GFX1032-NEXT:  ; %bb.2: ; %ComputeEnd
 ; GFX1032-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; GFX1032-NEXT:    s_or_saveexec_b32 s4, -1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1032-NEXT:    v_readlane_b32 s5, v1, 15
-; GFX1032-NEXT:    v_readlane_b32 s6, v1, 31
-; GFX1032-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s4
-; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-NEXT:    s_or_saveexec_b32 s4, -1
-; GFX1032-NEXT:    v_writelane_b32 v3, s5, 16
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s4
-; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT:    s_mov_b32 s4, s6
-; GFX1032-NEXT:    s_mov_b32 s6, -1
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
 ; GFX1032-NEXT:    ; implicit-def: $vgpr0
-; GFX1032-NEXT:    s_and_saveexec_b32 s8, vcc_lo
-; GFX1032-NEXT:    s_cbranch_execz .LBB8_2
-; GFX1032-NEXT:  ; %bb.1:
+; GFX1032-NEXT:    s_and_saveexec_b32 s5, vcc_lo
+; GFX1032-NEXT:    s_xor_b32 s5, exec_lo, s5
+; GFX1032-NEXT:    s_cbranch_execz .LBB8_4
+; GFX1032-NEXT:  ; %bb.3:
 ; GFX1032-NEXT:    v_mov_b32_e32 v0, s4
-; GFX1032-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX1032-NEXT:    s_mov_b32 s11, 0x31016000
+; GFX1032-NEXT:    s_mov_b32 s10, -1
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT:    s_mov_b32 s4, s2
-; GFX1032-NEXT:    s_mov_b32 s5, s3
+; GFX1032-NEXT:    s_mov_b32 s8, s2
+; GFX1032-NEXT:    s_mov_b32 s9, s3
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT:    buffer_atomic_sub v0, off, s[4:7], 0 glc
+; GFX1032-NEXT:    buffer_atomic_sub v0, off, s[8:11], 0 glc
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1032-NEXT:    buffer_gl0_inv
 ; GFX1032-NEXT:    buffer_gl1_inv
-; GFX1032-NEXT:  .LBB8_2:
+; GFX1032-NEXT:  .LBB8_4:
 ; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s8
+; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s5
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX1032-NEXT:    v_mov_b32_e32 v0, v3
 ; GFX1032-NEXT:    s_mov_b32 s3, 0x31016000
-; GFX1032-NEXT:    v_sub_nc_u32_e32 v0, s2, v0
-; GFX1032-NEXT:    s_mov_b32 s2, s6
+; GFX1032-NEXT:    v_sub_nc_u32_e32 v0, s2, v1
+; GFX1032-NEXT:    s_mov_b32 s2, -1
 ; GFX1032-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1164-LABEL: sub_i32_varying:
 ; GFX1164:       ; %bb.0: ; %entry
-; GFX1164-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1164-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1164-NEXT:    s_mov_b32 s6, 0
 ; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1164-NEXT:    v_readlane_b32 s6, v1, 15
-; GFX1164-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
+; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1164-NEXT:    ; implicit-def: $vgpr1
+; GFX1164-NEXT:  .LBB8_1: ; %ComputeLoop
+; GFX1164-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1164-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX1164-NEXT:    s_ctz_i32_b32 s5, s2
+; GFX1164-NEXT:    s_add_i32 s4, s4, 32
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT:    s_min_u32 s7, s5, s4
+; GFX1164-NEXT:    v_readlane_b32 s8, v0, s7
+; GFX1164-NEXT:    s_lshl_b64 s[4:5], 1, s7
+; GFX1164-NEXT:    v_writelane_b32 v1, s6, s7
+; GFX1164-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[4:5]
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX1164-NEXT:    s_add_i32 s6, s6, s8
+; GFX1164-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1164-NEXT:    s_cbranch_scc1 .LBB8_1
+; GFX1164-NEXT:  ; %bb.2: ; %ComputeEnd
 ; GFX1164-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
-; GFX1164-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1164-NEXT:    v_readlane_b32 s7, v1, 31
-; GFX1164-NEXT:    v_writelane_b32 v3, s6, 16
-; GFX1164-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1164-NEXT:    v_readlane_b32 s8, v1, 47
-; GFX1164-NEXT:    v_readlane_b32 s9, v1, 63
-; GFX1164-NEXT:    v_writelane_b32 v3, s7, 32
-; GFX1164-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[6:7], -1
-; GFX1164-NEXT:    s_mov_b32 s4, s9
-; GFX1164-NEXT:    v_writelane_b32 v3, s8, 48
-; GFX1164-NEXT:    s_mov_b64 exec, s[6:7]
-; GFX1164-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1164-NEXT:    s_mov_b32 s6, -1
+; GFX1164-NEXT:    s_mov_b64 s[4:5], exec
 ; GFX1164-NEXT:    ; implicit-def: $vgpr0
-; GFX1164-NEXT:    s_and_saveexec_b64 s[8:9], vcc
-; GFX1164-NEXT:    s_cbranch_execz .LBB8_2
-; GFX1164-NEXT:  ; %bb.1:
-; GFX1164-NEXT:    v_mov_b32_e32 v0, s4
-; GFX1164-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1164-NEXT:    s_xor_b64 s[4:5], exec, s[4:5]
+; GFX1164-NEXT:    s_cbranch_execz .LBB8_4
+; GFX1164-NEXT:  ; %bb.3:
+; GFX1164-NEXT:    v_mov_b32_e32 v0, s6
+; GFX1164-NEXT:    s_mov_b32 s11, 0x31016000
+; GFX1164-NEXT:    s_mov_b32 s10, -1
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX1164-NEXT:    s_mov_b32 s4, s2
-; GFX1164-NEXT:    s_mov_b32 s5, s3
+; GFX1164-NEXT:    s_mov_b32 s8, s2
+; GFX1164-NEXT:    s_mov_b32 s9, s3
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1164-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1164-NEXT:    buffer_atomic_sub_u32 v0, off, s[4:7], 0 glc
+; GFX1164-NEXT:    buffer_atomic_sub_u32 v0, off, s[8:11], 0 glc
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1164-NEXT:    buffer_gl0_inv
 ; GFX1164-NEXT:    buffer_gl1_inv
-; GFX1164-NEXT:  .LBB8_2:
-; GFX1164-NEXT:    s_or_b64 exec, exec, s[8:9]
+; GFX1164-NEXT:  .LBB8_4:
+; GFX1164-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v0, v3
 ; GFX1164-NEXT:    s_mov_b32 s3, 0x31016000
 ; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1164-NEXT:    v_sub_nc_u32_e32 v0, s2, v0
-; GFX1164-NEXT:    s_mov_b32 s2, s6
+; GFX1164-NEXT:    v_sub_nc_u32_e32 v0, s2, v1
+; GFX1164-NEXT:    s_mov_b32 s2, -1
 ; GFX1164-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1164-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1164-NEXT:    s_endpgm
 ;
 ; GFX1132-LABEL: sub_i32_varying:
 ; GFX1132:       ; %bb.0: ; %entry
-; GFX1132-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1132-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
+; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1132-NEXT:    s_mov_b32 s2, exec_lo
+; GFX1132-NEXT:    s_mov_b32 s4, 0
+; GFX1132-NEXT:    ; implicit-def: $vgpr1
+; GFX1132-NEXT:  .LBB8_1: ; %ComputeLoop
+; GFX1132-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1132-NEXT:    s_ctz_i32_b32 s3, s2
+; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX1132-NEXT:    v_readlane_b32 s5, v0, s3
+; GFX1132-NEXT:    s_lshl_b32 s6, 1, s3
+; GFX1132-NEXT:    v_writelane_b32 v1, s4, s3
+; GFX1132-NEXT:    s_and_not1_b32 s2, s2, s6
+; GFX1132-NEXT:    s_add_i32 s4, s4, s5
+; GFX1132-NEXT:    s_cmp_lg_u32 s2, 0
+; GFX1132-NEXT:    s_cbranch_scc1 .LBB8_1
+; GFX1132-NEXT:  ; %bb.2: ; %ComputeEnd
 ; GFX1132-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
-; GFX1132-NEXT:    s_or_saveexec_b32 s4, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1132-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1132-NEXT:    v_readlane_b32 s5, v1, 15
-; GFX1132-NEXT:    v_readlane_b32 s6, v1, 31
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_3) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
-; GFX1132-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s4
-; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1132-NEXT:    s_or_saveexec_b32 s4, -1
-; GFX1132-NEXT:    v_writelane_b32 v3, s5, 16
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s4
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_2)
-; GFX1132-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1132-NEXT:    s_mov_b32 s4, s6
-; GFX1132-NEXT:    s_mov_b32 s6, -1
+; GFX1132-NEXT:    s_mov_b32 s5, exec_lo
 ; GFX1132-NEXT:    ; implicit-def: $vgpr0
-; GFX1132-NEXT:    s_and_saveexec_b32 s8, vcc_lo
-; GFX1132-NEXT:    s_cbranch_execz .LBB8_2
-; GFX1132-NEXT:  ; %bb.1:
+; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1132-NEXT:    s_xor_b32 s5, exec_lo, s5
+; GFX1132-NEXT:    s_cbranch_execz .LBB8_4
+; GFX1132-NEXT:  ; %bb.3:
 ; GFX1132-NEXT:    v_mov_b32_e32 v0, s4
-; GFX1132-NEXT:    s_mov_b32 s7, 0x31016000
+; GFX1132-NEXT:    s_mov_b32 s11, 0x31016000
+; GFX1132-NEXT:    s_mov_b32 s10, -1
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX1132-NEXT:    s_mov_b32 s4, s2
-; GFX1132-NEXT:    s_mov_b32 s5, s3
+; GFX1132-NEXT:    s_mov_b32 s8, s2
+; GFX1132-NEXT:    s_mov_b32 s9, s3
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1132-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1132-NEXT:    buffer_atomic_sub_u32 v0, off, s[4:7], 0 glc
+; GFX1132-NEXT:    buffer_atomic_sub_u32 v0, off, s[8:11], 0 glc
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0)
 ; GFX1132-NEXT:    buffer_gl0_inv
 ; GFX1132-NEXT:    buffer_gl1_inv
-; GFX1132-NEXT:  .LBB8_2:
-; GFX1132-NEXT:    s_or_b32 exec_lo, exec_lo, s8
+; GFX1132-NEXT:  .LBB8_4:
+; GFX1132-NEXT:    s_or_b32 exec_lo, exec_lo, s5
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX1132-NEXT:    v_mov_b32_e32 v0, v3
 ; GFX1132-NEXT:    s_mov_b32 s3, 0x31016000
 ; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1132-NEXT:    v_sub_nc_u32_e32 v0, s2, v0
-; GFX1132-NEXT:    s_mov_b32 s2, s6
+; GFX1132-NEXT:    v_sub_nc_u32_e32 v0, s2, v1
+; GFX1132-NEXT:    s_mov_b32 s2, -1
 ; GFX1132-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1132-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1132-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = atomicrmw sub ptr addrspace(1) %inout, i32 %lane acq_rel
   store i32 %old, ptr addrspace(1) %out
   ret void

llvm/test/CodeGen/AMDGPU/atomic_optimizations_local_pointer.ll

-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --force-update
-; RUN: llc -march=amdgcn -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX7LESS %s
-; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX8 %s
-; RUN: llc -march=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX9 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX1064 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX1032 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=-wavefrontsize32,+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX1164 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=+wavefrontsize32,-wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX1132 %s
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -march=amdgcn -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX7LESS %s
+; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX8 %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX9 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX1064 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX1032 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=-wavefrontsize32,+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX1164 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=+wavefrontsize32,-wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX1132 %s
 
 declare i32 @llvm.amdgcn.workitem.id.x()
 
 @local_var32 = addrspace(3) global i32 undef, align 4
 @local_var64 = addrspace(3) global i64 undef, align 8
 
 ; Show what the atomic optimization pass will do for local pointers.
 

 ; GFX7LESS-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7LESS-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7LESS-NEXT:    s_mov_b32 s2, -1
 ; GFX7LESS-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX7LESS-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: add_i32_varying:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    v_mov_b32_e32 v3, 0
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX8-NEXT:    v_mov_b32_e32 v1, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s4, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_add_i32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB2_2
-; GFX8-NEXT:  ; %bb.1:
-; GFX8-NEXT:    v_mov_b32_e32 v0, s4
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB2_4
+; GFX8-NEXT:  ; %bb.3:
+; GFX8-NEXT:    v_mov_b32_e32 v0, 0
+; GFX8-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX8-NEXT:    s_mov_b32 m0, -1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    ds_add_rtn_u32 v0, v3, v0
+; GFX8-NEXT:    ds_add_rtn_u32 v0, v0, v2
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:  .LBB2_2:
+; GFX8-NEXT:  .LBB2_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v2
 ; GFX8-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX8-NEXT:    s_mov_b32 s2, -1
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s4, v0
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s4, v1
 ; GFX8-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: add_i32_varying:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s4, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_add_i32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB2_2
-; GFX9-NEXT:  ; %bb.1:
-; GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB2_4
+; GFX9-NEXT:  ; %bb.3:
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    ds_add_rtn_u32 v0, v3, v0
+; GFX9-NEXT:    ds_add_rtn_u32 v0, v0, v2
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:  .LBB2_2:
+; GFX9-NEXT:  .LBB2_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v2
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s2, -1
-; GFX9-NEXT:    v_add_u32_e32 v0, s4, v0
+; GFX9-NEXT:    v_add_u32_e32 v0, s4, v1
 ; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: add_i32_varying:
 ; GFX1064:       ; %bb.0: ; %entry
-; GFX1064-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1064-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1064-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1064-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1064-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1064-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT:    s_mov_b32 s2, -1
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1064-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1064-NEXT:    s_mov_b32 s4, 0
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1064-NEXT:    ; implicit-def: $vgpr1
+; GFX1064-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX1064-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX1064-NEXT:    s_add_i32 s5, s5, 32
+; GFX1064-NEXT:    s_min_u32 s5, s6, s5
+; GFX1064-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1064-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1064-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1064-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX1064-NEXT:    s_add_i32 s4, s4, s8
+; GFX1064-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX1064-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX1064-NEXT:    ; implicit-def: $vgpr0
-; GFX1064-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1064-NEXT:    s_cbranch_execz .LBB2_2
-; GFX1064-NEXT:  ; %bb.1:
-; GFX1064-NEXT:    v_mov_b32_e32 v0, s7
-; GFX1064-NEXT:    s_mov_b32 s3, s7
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1064-NEXT:    s_cbranch_execz .LBB2_4
+; GFX1064-NEXT:  ; %bb.3:
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT:    ds_add_rtn_u32 v0, v4, v0
+; GFX1064-NEXT:    ds_add_rtn_u32 v0, v0, v2
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_gl0_inv
-; GFX1064-NEXT:  .LBB2_2:
+; GFX1064-NEXT:  .LBB2_4:
 ; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1064-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1064-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1064-NEXT:    v_add_nc_u32_e32 v0, s3, v0
+; GFX1064-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1064-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT:    v_add_nc_u32_e32 v0, s2, v1
+; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1064-NEXT:    s_endpgm
 ;
 ; GFX1032-LABEL: add_i32_varying:
 ; GFX1032:       ; %bb.0: ; %entry
-; GFX1032-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1032-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1032-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1032-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT:    s_mov_b32 s2, -1
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1032-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1032-NEXT:    s_mov_b32 s2, 0
+; GFX1032-NEXT:    ; implicit-def: $vgpr1
+; GFX1032-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX1032-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1032-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1032-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1032-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX1032-NEXT:    s_add_i32 s2, s2, s5
+; GFX1032-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX1032-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
 ; GFX1032-NEXT:    ; implicit-def: $vgpr0
 ; GFX1032-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1032-NEXT:    s_cbranch_execz .LBB2_2
-; GFX1032-NEXT:  ; %bb.1:
-; GFX1032-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1032-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1032-NEXT:    s_cbranch_execz .LBB2_4
+; GFX1032-NEXT:  ; %bb.3:
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT:    ds_add_rtn_u32 v0, v4, v0
+; GFX1032-NEXT:    ds_add_rtn_u32 v0, v0, v2
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_gl0_inv
-; GFX1032-NEXT:  .LBB2_2:
+; GFX1032-NEXT:  .LBB2_4:
 ; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1032-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1032-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1032-NEXT:    v_add_nc_u32_e32 v0, s3, v0
+; GFX1032-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1032-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT:    v_add_nc_u32_e32 v0, s2, v1
+; GFX1032-NEXT:    s_mov_b32 s2, -1
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1164-LABEL: add_i32_varying:
 ; GFX1164:       ; %bb.0: ; %entry
-; GFX1164-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1164-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1164-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1164-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1164-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1164-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1164-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1164-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1164-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1164-NEXT:    s_mov_b32 s2, -1
+; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1164-NEXT:    s_mov_b32 s4, 0
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1164-NEXT:    ; implicit-def: $vgpr1
+; GFX1164-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX1164-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1164-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX1164-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX1164-NEXT:    s_add_i32 s5, s5, 32
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT:    s_min_u32 s5, s6, s5
+; GFX1164-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1164-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1164-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1164-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX1164-NEXT:    s_add_i32 s4, s4, s8
+; GFX1164-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1164-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX1164-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
 ; GFX1164-NEXT:    ; implicit-def: $vgpr0
-; GFX1164-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1164-NEXT:    s_cbranch_execz .LBB2_2
-; GFX1164-NEXT:  ; %bb.1:
-; GFX1164-NEXT:    v_mov_b32_e32 v0, s7
-; GFX1164-NEXT:    s_mov_b32 s3, s7
+; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1164-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1164-NEXT:    s_cbranch_execz .LBB2_4
+; GFX1164-NEXT:  ; %bb.3:
+; GFX1164-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1164-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1164-NEXT:    ds_add_rtn_u32 v0, v4, v0
+; GFX1164-NEXT:    ds_add_rtn_u32 v0, v0, v2
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_gl0_inv
-; GFX1164-NEXT:  .LBB2_2:
-; GFX1164-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1164-NEXT:  .LBB2_4:
+; GFX1164-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1164-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1164-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_e32 v0, s3, v0
+; GFX1164-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1164-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_add_nc_u32_e32 v0, s2, v1
+; GFX1164-NEXT:    s_mov_b32 s2, -1
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1164-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1164-NEXT:    s_endpgm
 ;
 ; GFX1132-LABEL: add_i32_varying:
 ; GFX1132:       ; %bb.0: ; %entry
-; GFX1132-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1132-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1132-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1132-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1132-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1132-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1132-NEXT:    s_mov_b32 s2, -1
+; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1132-NEXT:    s_mov_b32 s2, 0
+; GFX1132-NEXT:    ; implicit-def: $vgpr1
+; GFX1132-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX1132-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1132-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX1132-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1132-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1132-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1132-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX1132-NEXT:    s_add_i32 s2, s2, s5
+; GFX1132-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1132-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX1132-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
 ; GFX1132-NEXT:    ; implicit-def: $vgpr0
-; GFX1132-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1132-NEXT:    s_cbranch_execz .LBB2_2
-; GFX1132-NEXT:  ; %bb.1:
-; GFX1132-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1132-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1132-NEXT:    s_cbranch_execz .LBB2_4
+; GFX1132-NEXT:  ; %bb.3:
+; GFX1132-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1132-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1132-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1132-NEXT:    ds_add_rtn_u32 v0, v4, v0
+; GFX1132-NEXT:    ds_add_rtn_u32 v0, v0, v2
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_gl0_inv
-; GFX1132-NEXT:  .LBB2_2:
+; GFX1132-NEXT:  .LBB2_4:
 ; GFX1132-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1132-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1132-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1132-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_e32 v0, s3, v0
+; GFX1132-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1132-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT:    v_add_nc_u32_e32 v0, s2, v1
+; GFX1132-NEXT:    s_mov_b32 s2, -1
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1132-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1132-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
   store i32 %old, ptr addrspace(1) %out
   ret void
 }
 
 define amdgpu_kernel void @add_i32_varying_nouse() {
 ; GFX7LESS-LABEL: add_i32_varying_nouse:
 ; GFX7LESS:       ; %bb.0: ; %entry
 ; GFX7LESS-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7LESS-NEXT:    s_mov_b32 m0, -1
 ; GFX7LESS-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7LESS-NEXT:    ds_add_u32 v1, v0
 ; GFX7LESS-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7LESS-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: add_i32_varying_nouse:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX8-NEXT:    v_mov_b32_e32 v1, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[0:1], -1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s2, v1, 63
-; GFX8-NEXT:    s_mov_b64 exec, s[0:1]
-; GFX8-NEXT:    s_mov_b32 s0, s2
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
-; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB3_2
-; GFX8-NEXT:  ; %bb.1:
-; GFX8-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[0:1], exec
+; GFX8-NEXT:    s_mov_b32 s2, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v1, exec_hi, v1
+; GFX8-NEXT:  .LBB3_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s3, s1
+; GFX8-NEXT:    s_ff1_i32_b32 s4, s0
+; GFX8-NEXT:    s_add_i32 s3, s3, 32
+; GFX8-NEXT:    s_min_u32 s3, s4, s3
+; GFX8-NEXT:    v_readlane_b32 s6, v0, s3
+; GFX8-NEXT:    s_lshl_b64 s[4:5], 1, s3
+; GFX8-NEXT:    s_add_i32 s2, s2, s6
+; GFX8-NEXT:    s_andn2_b64 s[0:1], s[0:1], s[4:5]
+; GFX8-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB3_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
+; GFX8-NEXT:    s_and_saveexec_b64 s[0:1], vcc
+; GFX8-NEXT:    s_xor_b64 s[0:1], exec, s[0:1]
+; GFX8-NEXT:    s_cbranch_execz .LBB3_4
+; GFX8-NEXT:  ; %bb.3:
+; GFX8-NEXT:    v_mov_b32_e32 v0, 0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s2
 ; GFX8-NEXT:    s_mov_b32 m0, -1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    ds_add_u32 v2, v0
+; GFX8-NEXT:    ds_add_u32 v0, v1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:  .LBB3_2:
+; GFX8-NEXT:  .LBB3_4:
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: add_i32_varying_nouse:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[0:1], -1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s2, v1, 63
-; GFX9-NEXT:    s_mov_b64 exec, s[0:1]
-; GFX9-NEXT:    s_mov_b32 s0, s2
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
-; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB3_2
-; GFX9-NEXT:  ; %bb.1:
-; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[0:1], exec
+; GFX9-NEXT:    s_mov_b32 s2, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v1, exec_hi, v1
+; GFX9-NEXT:  .LBB3_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s3, s1
+; GFX9-NEXT:    s_ff1_i32_b32 s4, s0
+; GFX9-NEXT:    s_add_i32 s3, s3, 32
+; GFX9-NEXT:    s_min_u32 s3, s4, s3
+; GFX9-NEXT:    v_readlane_b32 s6, v0, s3
+; GFX9-NEXT:    s_lshl_b64 s[4:5], 1, s3
+; GFX9-NEXT:    s_add_i32 s2, s2, s6
+; GFX9-NEXT:    s_andn2_b64 s[0:1], s[0:1], s[4:5]
+; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB3_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
+; GFX9-NEXT:    s_and_saveexec_b64 s[0:1], vcc
+; GFX9-NEXT:    s_xor_b64 s[0:1], exec, s[0:1]
+; GFX9-NEXT:    s_cbranch_execz .LBB3_4
+; GFX9-NEXT:  ; %bb.3:
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s2
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    ds_add_u32 v2, v0
+; GFX9-NEXT:    ds_add_u32 v0, v1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:  .LBB3_2:
+; GFX9-NEXT:  .LBB3_4:
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: add_i32_varying_nouse:
 ; GFX1064:       ; %bb.0: ; %entry
-; GFX1064-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1064-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1064-NEXT:    v_add_nc_u32_e32 v1, v1, v2
-; GFX1064-NEXT:    s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT:    v_readlane_b32 s2, v1, 0
-; GFX1064-NEXT:    v_readlane_b32 s3, v1, 32
-; GFX1064-NEXT:    s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v0
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1064-NEXT:    s_mov_b64 s[0:1], exec
+; GFX1064-NEXT:    s_mov_b32 s2, 0
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v1, exec_hi, v1
+; GFX1064-NEXT:  .LBB3_1: ; %ComputeLoop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_ff1_i32_b32 s3, s1
+; GFX1064-NEXT:    s_ff1_i32_b32 s4, s0
+; GFX1064-NEXT:    s_add_i32 s3, s3, 32
+; GFX1064-NEXT:    s_min_u32 s3, s4, s3
+; GFX1064-NEXT:    v_readlane_b32 s6, v0, s3
+; GFX1064-NEXT:    s_lshl_b64 s[4:5], 1, s3
+; GFX1064-NEXT:    s_andn2_b64 s[0:1], s[0:1], s[4:5]
+; GFX1064-NEXT:    s_add_i32 s2, s2, s6
+; GFX1064-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB3_1
+; GFX1064-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
+; GFX1064-NEXT:    s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-NEXT:    s_xor_b64 s[0:1], exec, s[0:1]
+; GFX1064-NEXT:    s_cbranch_execz .LBB3_4
+; GFX1064-NEXT:  ; %bb.3:
 ; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1064-NEXT:    s_add_i32 s0, s2, s3
-; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
-; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX1064-NEXT:    s_cbranch_execz .LBB3_2
-; GFX1064-NEXT:  ; %bb.1:
-; GFX1064-NEXT:    v_mov_b32_e32 v3, s0
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s2
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT:    ds_add_u32 v0, v3
+; GFX1064-NEXT:    ds_add_u32 v0, v1
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_gl0_inv
-; GFX1064-NEXT:  .LBB3_2:
+; GFX1064-NEXT:  .LBB3_4:
 ; GFX1064-NEXT:    s_endpgm
 ;
 ; GFX1032-LABEL: add_i32_varying_nouse:
 ; GFX1032:       ; %bb.0: ; %entry
-; GFX1032-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    s_or_saveexec_b32 s0, -1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1032-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1032-NEXT:    v_add_nc_u32_e32 v1, v1, v2
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s0
-; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1032-NEXT:    s_mov_b32 s1, exec_lo
+; GFX1032-NEXT:    s_mov_b32 s0, 0
+; GFX1032-NEXT:  .LBB3_1: ; %ComputeLoop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_ff1_i32_b32 s2, s1
+; GFX1032-NEXT:    v_readlane_b32 s3, v0, s2
+; GFX1032-NEXT:    s_lshl_b32 s2, 1, s2
+; GFX1032-NEXT:    s_andn2_b32 s1, s1, s2
+; GFX1032-NEXT:    s_add_i32 s0, s0, s3
+; GFX1032-NEXT:    s_cmp_lg_u32 s1, 0
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB3_1
+; GFX1032-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v1
+; GFX1032-NEXT:    s_and_saveexec_b32 s1, vcc_lo
+; GFX1032-NEXT:    s_xor_b32 s1, exec_lo, s1
+; GFX1032-NEXT:    s_cbranch_execz .LBB3_4
+; GFX1032-NEXT:  ; %bb.3:
 ; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1032-NEXT:    v_mov_b32_e32 v3, v1
-; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX1032-NEXT:    s_and_saveexec_b32 s0, vcc_lo
-; GFX1032-NEXT:    s_cbranch_execz .LBB3_2
-; GFX1032-NEXT:  ; %bb.1:
+; GFX1032-NEXT:    v_mov_b32_e32 v1, s0
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT:    ds_add_u32 v0, v3
+; GFX1032-NEXT:    ds_add_u32 v0, v1
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_gl0_inv
-; GFX1032-NEXT:  .LBB3_2:
+; GFX1032-NEXT:  .LBB3_4:
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1164-LABEL: add_i32_varying_nouse:
 ; GFX1164:       ; %bb.0: ; %entry
-; GFX1164-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    s_or_saveexec_b64 s[0:1], -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1164-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_e32 v1, v1, v2
-; GFX1164-NEXT:    v_permlane64_b32 v2, v1
-; GFX1164-NEXT:    s_mov_b64 exec, s[0:1]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[0:1], -1
-; GFX1164-NEXT:    v_add_nc_u32_e32 v1, v1, v2
-; GFX1164-NEXT:    s_mov_b64 exec, s[0:1]
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instid1(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_3) | instskip(SKIP_1) | instid1(VALU_DEP_3)
-; GFX1164-NEXT:    v_mov_b32_e32 v3, v1
+; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
 ; GFX1164-NEXT:    s_mov_b64 s[0:1], exec
-; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v4
-; GFX1164-NEXT:    s_cbranch_execz .LBB3_2
-; GFX1164-NEXT:  ; %bb.1:
+; GFX1164-NEXT:    s_mov_b32 s2, 0
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v1, exec_hi, v1
+; GFX1164-NEXT:  .LBB3_1: ; %ComputeLoop
+; GFX1164-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1164-NEXT:    s_ctz_i32_b32 s3, s1
+; GFX1164-NEXT:    s_ctz_i32_b32 s4, s0
+; GFX1164-NEXT:    s_add_i32 s3, s3, 32
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT:    s_min_u32 s3, s4, s3
+; GFX1164-NEXT:    v_readlane_b32 s6, v0, s3
+; GFX1164-NEXT:    s_lshl_b64 s[4:5], 1, s3
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT:    s_and_not1_b64 s[0:1], s[0:1], s[4:5]
+; GFX1164-NEXT:    s_add_i32 s2, s2, s6
+; GFX1164-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX1164-NEXT:    s_cbranch_scc1 .LBB3_1
+; GFX1164-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1164-NEXT:    s_mov_b64 s[0:1], exec
+; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v1
+; GFX1164-NEXT:    s_xor_b64 s[0:1], exec, s[0:1]
+; GFX1164-NEXT:    s_cbranch_execz .LBB3_4
+; GFX1164-NEXT:  ; %bb.3:
+; GFX1164-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1164-NEXT:    v_mov_b32_e32 v1, s2
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1164-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1164-NEXT:    ds_add_u32 v0, v3
+; GFX1164-NEXT:    ds_add_u32 v0, v1
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_gl0_inv
-; GFX1164-NEXT:  .LBB3_2:
+; GFX1164-NEXT:  .LBB3_4:
 ; GFX1164-NEXT:    s_endpgm
 ;
 ; GFX1132-LABEL: add_i32_varying_nouse:
 ; GFX1132:       ; %bb.0: ; %entry
-; GFX1132-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    s_or_saveexec_b32 s0, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1132-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
-; GFX1132-NEXT:    v_add_nc_u32_e32 v1, v1, v2
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s0
-; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v3, v1
-; GFX1132-NEXT:    s_mov_b32 s0, exec_lo
-; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v4
-; GFX1132-NEXT:    s_cbranch_execz .LBB3_2
-; GFX1132-NEXT:  ; %bb.1:
+; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1132-NEXT:    s_mov_b32 s1, exec_lo
+; GFX1132-NEXT:    s_mov_b32 s0, 0
+; GFX1132-NEXT:  .LBB3_1: ; %ComputeLoop
+; GFX1132-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1132-NEXT:    s_ctz_i32_b32 s2, s1
+; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
+; GFX1132-NEXT:    v_readlane_b32 s3, v0, s2
+; GFX1132-NEXT:    s_lshl_b32 s2, 1, s2
+; GFX1132-NEXT:    s_and_not1_b32 s1, s1, s2
+; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT:    s_add_i32 s0, s0, s3
+; GFX1132-NEXT:    s_cmp_lg_u32 s1, 0
+; GFX1132-NEXT:    s_cbranch_scc1 .LBB3_1
+; GFX1132-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1132-NEXT:    s_mov_b32 s1, exec_lo
+; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v1
+; GFX1132-NEXT:    s_xor_b32 s1, exec_lo, s1
+; GFX1132-NEXT:    s_cbranch_execz .LBB3_4
+; GFX1132-NEXT:  ; %bb.3:
+; GFX1132-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s0
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1132-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1132-NEXT:    ds_add_u32 v0, v3
+; GFX1132-NEXT:    ds_add_u32 v0, v1
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_gl0_inv
-; GFX1132-NEXT:  .LBB3_2:
+; GFX1132-NEXT:  .LBB3_4:
 ; GFX1132-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
   ret void
 }
 
 define amdgpu_kernel void @add_i64_constant(ptr addrspace(1) %out) {

 ; GFX7LESS-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7LESS-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7LESS-NEXT:    s_mov_b32 s2, -1
 ; GFX7LESS-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX7LESS-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: sub_i32_varying:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    v_mov_b32_e32 v3, 0
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX8-NEXT:    v_mov_b32_e32 v1, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s4, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB9_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_add_i32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB9_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB9_2
-; GFX8-NEXT:  ; %bb.1:
-; GFX8-NEXT:    v_mov_b32_e32 v0, s4
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB9_4
+; GFX8-NEXT:  ; %bb.3:
+; GFX8-NEXT:    v_mov_b32_e32 v0, 0
+; GFX8-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX8-NEXT:    s_mov_b32 m0, -1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    ds_sub_rtn_u32 v0, v3, v0
+; GFX8-NEXT:    ds_sub_rtn_u32 v0, v0, v2
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:  .LBB9_2:
+; GFX8-NEXT:  .LBB9_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v2
 ; GFX8-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX8-NEXT:    s_mov_b32 s2, -1
-; GFX8-NEXT:    v_sub_u32_e32 v0, vcc, s4, v0
+; GFX8-NEXT:    v_sub_u32_e32 v0, vcc, s4, v1
 ; GFX8-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: sub_i32_varying:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s4, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB9_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_add_i32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB9_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB9_2
-; GFX9-NEXT:  ; %bb.1:
-; GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB9_4
+; GFX9-NEXT:  ; %bb.3:
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    ds_sub_rtn_u32 v0, v3, v0
+; GFX9-NEXT:    ds_sub_rtn_u32 v0, v0, v2
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:  .LBB9_2:
+; GFX9-NEXT:  .LBB9_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v2
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s2, -1
-; GFX9-NEXT:    v_sub_u32_e32 v0, s4, v0
+; GFX9-NEXT:    v_sub_u32_e32 v0, s4, v1
 ; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: sub_i32_varying:
 ; GFX1064:       ; %bb.0: ; %entry
-; GFX1064-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1064-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1064-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1064-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1064-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1064-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT:    s_mov_b32 s2, -1
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1064-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1064-NEXT:    s_mov_b32 s4, 0
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1064-NEXT:    ; implicit-def: $vgpr1
+; GFX1064-NEXT:  .LBB9_1: ; %ComputeLoop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX1064-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX1064-NEXT:    s_add_i32 s5, s5, 32
+; GFX1064-NEXT:    s_min_u32 s5, s6, s5
+; GFX1064-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1064-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1064-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1064-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX1064-NEXT:    s_add_i32 s4, s4, s8
+; GFX1064-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB9_1
+; GFX1064-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX1064-NEXT:    ; implicit-def: $vgpr0
-; GFX1064-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1064-NEXT:    s_cbranch_execz .LBB9_2
-; GFX1064-NEXT:  ; %bb.1:
-; GFX1064-NEXT:    v_mov_b32_e32 v0, s7
-; GFX1064-NEXT:    s_mov_b32 s3, s7
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1064-NEXT:    s_cbranch_execz .LBB9_4
+; GFX1064-NEXT:  ; %bb.3:
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT:    ds_sub_rtn_u32 v0, v4, v0
+; GFX1064-NEXT:    ds_sub_rtn_u32 v0, v0, v2
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_gl0_inv
-; GFX1064-NEXT:  .LBB9_2:
+; GFX1064-NEXT:  .LBB9_4:
 ; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1064-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1064-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1064-NEXT:    v_sub_nc_u32_e32 v0, s3, v0
+; GFX1064-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1064-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT:    v_sub_nc_u32_e32 v0, s2, v1
+; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1064-NEXT:    s_endpgm
 ;
 ; GFX1032-LABEL: sub_i32_varying:
 ; GFX1032:       ; %bb.0: ; %entry
-; GFX1032-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1032-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1032-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1032-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT:    s_mov_b32 s2, -1
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1032-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1032-NEXT:    s_mov_b32 s2, 0
+; GFX1032-NEXT:    ; implicit-def: $vgpr1
+; GFX1032-NEXT:  .LBB9_1: ; %ComputeLoop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX1032-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1032-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1032-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1032-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX1032-NEXT:    s_add_i32 s2, s2, s5
+; GFX1032-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB9_1
+; GFX1032-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
 ; GFX1032-NEXT:    ; implicit-def: $vgpr0
 ; GFX1032-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1032-NEXT:    s_cbranch_execz .LBB9_2
-; GFX1032-NEXT:  ; %bb.1:
-; GFX1032-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1032-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1032-NEXT:    s_cbranch_execz .LBB9_4
+; GFX1032-NEXT:  ; %bb.3:
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT:    ds_sub_rtn_u32 v0, v4, v0
+; GFX1032-NEXT:    ds_sub_rtn_u32 v0, v0, v2
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_gl0_inv
-; GFX1032-NEXT:  .LBB9_2:
+; GFX1032-NEXT:  .LBB9_4:
 ; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1032-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1032-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1032-NEXT:    v_sub_nc_u32_e32 v0, s3, v0
+; GFX1032-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1032-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT:    v_sub_nc_u32_e32 v0, s2, v1
+; GFX1032-NEXT:    s_mov_b32 s2, -1
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1164-LABEL: sub_i32_varying:
 ; GFX1164:       ; %bb.0: ; %entry
-; GFX1164-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1164-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1164-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1164-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1164-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1164-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1164-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1164-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1164-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1164-NEXT:    s_mov_b32 s2, -1
+; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1164-NEXT:    s_mov_b32 s4, 0
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1164-NEXT:    ; implicit-def: $vgpr1
+; GFX1164-NEXT:  .LBB9_1: ; %ComputeLoop
+; GFX1164-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1164-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX1164-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX1164-NEXT:    s_add_i32 s5, s5, 32
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT:    s_min_u32 s5, s6, s5
+; GFX1164-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1164-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1164-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1164-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX1164-NEXT:    s_add_i32 s4, s4, s8
+; GFX1164-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1164-NEXT:    s_cbranch_scc1 .LBB9_1
+; GFX1164-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
 ; GFX1164-NEXT:    ; implicit-def: $vgpr0
-; GFX1164-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1164-NEXT:    s_cbranch_execz .LBB9_2
-; GFX1164-NEXT:  ; %bb.1:
-; GFX1164-NEXT:    v_mov_b32_e32 v0, s7
-; GFX1164-NEXT:    s_mov_b32 s3, s7
+; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1164-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1164-NEXT:    s_cbranch_execz .LBB9_4
+; GFX1164-NEXT:  ; %bb.3:
+; GFX1164-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1164-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1164-NEXT:    ds_sub_rtn_u32 v0, v4, v0
+; GFX1164-NEXT:    ds_sub_rtn_u32 v0, v0, v2
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_gl0_inv
-; GFX1164-NEXT:  .LBB9_2:
-; GFX1164-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1164-NEXT:  .LBB9_4:
+; GFX1164-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1164-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1164-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1164-NEXT:    v_sub_nc_u32_e32 v0, s3, v0
+; GFX1164-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1164-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_sub_nc_u32_e32 v0, s2, v1
+; GFX1164-NEXT:    s_mov_b32 s2, -1
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1164-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1164-NEXT:    s_endpgm
 ;
 ; GFX1132-LABEL: sub_i32_varying:
 ; GFX1132:       ; %bb.0: ; %entry
-; GFX1132-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1132-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1132-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1132-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1132-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1132-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1132-NEXT:    s_mov_b32 s2, -1
+; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1132-NEXT:    s_mov_b32 s2, 0
+; GFX1132-NEXT:    ; implicit-def: $vgpr1
+; GFX1132-NEXT:  .LBB9_1: ; %ComputeLoop
+; GFX1132-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1132-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX1132-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1132-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1132-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1132-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX1132-NEXT:    s_add_i32 s2, s2, s5
+; GFX1132-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1132-NEXT:    s_cbranch_scc1 .LBB9_1
+; GFX1132-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
 ; GFX1132-NEXT:    ; implicit-def: $vgpr0
-; GFX1132-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1132-NEXT:    s_cbranch_execz .LBB9_2
-; GFX1132-NEXT:  ; %bb.1:
-; GFX1132-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1132-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1132-NEXT:    s_cbranch_execz .LBB9_4
+; GFX1132-NEXT:  ; %bb.3:
+; GFX1132-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1132-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1132-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1132-NEXT:    ds_sub_rtn_u32 v0, v4, v0
+; GFX1132-NEXT:    ds_sub_rtn_u32 v0, v0, v2
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_gl0_inv
-; GFX1132-NEXT:  .LBB9_2:
+; GFX1132-NEXT:  .LBB9_4:
 ; GFX1132-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1132-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1132-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1132-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1132-NEXT:    v_sub_nc_u32_e32 v0, s3, v0
+; GFX1132-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1132-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT:    v_sub_nc_u32_e32 v0, s2, v1
+; GFX1132-NEXT:    s_mov_b32 s2, -1
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1132-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1132-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = atomicrmw sub ptr addrspace(3) @local_var32, i32 %lane acq_rel
   store i32 %old, ptr addrspace(1) %out
   ret void
 }
 
 define amdgpu_kernel void @sub_i32_varying_nouse() {
 ; GFX7LESS-LABEL: sub_i32_varying_nouse:
 ; GFX7LESS:       ; %bb.0: ; %entry
 ; GFX7LESS-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7LESS-NEXT:    s_mov_b32 m0, -1
 ; GFX7LESS-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7LESS-NEXT:    ds_sub_u32 v1, v0
 ; GFX7LESS-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7LESS-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: sub_i32_varying_nouse:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX8-NEXT:    v_mov_b32_e32 v1, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[0:1], -1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s2, v1, 63
-; GFX8-NEXT:    s_mov_b64 exec, s[0:1]
-; GFX8-NEXT:    s_mov_b32 s0, s2
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
-; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB10_2
-; GFX8-NEXT:  ; %bb.1:
-; GFX8-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[0:1], exec
+; GFX8-NEXT:    s_mov_b32 s2, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v1, exec_hi, v1
+; GFX8-NEXT:  .LBB10_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s3, s1
+; GFX8-NEXT:    s_ff1_i32_b32 s4, s0
+; GFX8-NEXT:    s_add_i32 s3, s3, 32
+; GFX8-NEXT:    s_min_u32 s3, s4, s3
+; GFX8-NEXT:    v_readlane_b32 s6, v0, s3
+; GFX8-NEXT:    s_lshl_b64 s[4:5], 1, s3
+; GFX8-NEXT:    s_add_i32 s2, s2, s6
+; GFX8-NEXT:    s_andn2_b64 s[0:1], s[0:1], s[4:5]
+; GFX8-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB10_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
+; GFX8-NEXT:    s_and_saveexec_b64 s[0:1], vcc
+; GFX8-NEXT:    s_xor_b64 s[0:1], exec, s[0:1]
+; GFX8-NEXT:    s_cbranch_execz .LBB10_4
+; GFX8-NEXT:  ; %bb.3:
+; GFX8-NEXT:    v_mov_b32_e32 v0, 0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s2
 ; GFX8-NEXT:    s_mov_b32 m0, -1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    ds_sub_u32 v2, v0
+; GFX8-NEXT:    ds_sub_u32 v0, v1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:  .LBB10_2:
+; GFX8-NEXT:  .LBB10_4:
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: sub_i32_varying_nouse:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[0:1], -1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s2, v1, 63
-; GFX9-NEXT:    s_mov_b64 exec, s[0:1]
-; GFX9-NEXT:    s_mov_b32 s0, s2
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
-; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB10_2
-; GFX9-NEXT:  ; %bb.1:
-; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[0:1], exec
+; GFX9-NEXT:    s_mov_b32 s2, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v1, exec_hi, v1
+; GFX9-NEXT:  .LBB10_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s3, s1
+; GFX9-NEXT:    s_ff1_i32_b32 s4, s0
+; GFX9-NEXT:    s_add_i32 s3, s3, 32
+; GFX9-NEXT:    s_min_u32 s3, s4, s3
+; GFX9-NEXT:    v_readlane_b32 s6, v0, s3
+; GFX9-NEXT:    s_lshl_b64 s[4:5], 1, s3
+; GFX9-NEXT:    s_add_i32 s2, s2, s6
+; GFX9-NEXT:    s_andn2_b64 s[0:1], s[0:1], s[4:5]
+; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB10_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
+; GFX9-NEXT:    s_and_saveexec_b64 s[0:1], vcc
+; GFX9-NEXT:    s_xor_b64 s[0:1], exec, s[0:1]
+; GFX9-NEXT:    s_cbranch_execz .LBB10_4
+; GFX9-NEXT:  ; %bb.3:
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s2
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    ds_sub_u32 v2, v0
+; GFX9-NEXT:    ds_sub_u32 v0, v1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:  .LBB10_2:
+; GFX9-NEXT:  .LBB10_4:
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: sub_i32_varying_nouse:
 ; GFX1064:       ; %bb.0: ; %entry
-; GFX1064-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1064-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1064-NEXT:    v_add_nc_u32_e32 v1, v1, v2
-; GFX1064-NEXT:    s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT:    v_readlane_b32 s2, v1, 0
-; GFX1064-NEXT:    v_readlane_b32 s3, v1, 32
-; GFX1064-NEXT:    s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v0
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1064-NEXT:    s_mov_b64 s[0:1], exec
+; GFX1064-NEXT:    s_mov_b32 s2, 0
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v1, exec_hi, v1
+; GFX1064-NEXT:  .LBB10_1: ; %ComputeLoop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_ff1_i32_b32 s3, s1
+; GFX1064-NEXT:    s_ff1_i32_b32 s4, s0
+; GFX1064-NEXT:    s_add_i32 s3, s3, 32
+; GFX1064-NEXT:    s_min_u32 s3, s4, s3
+; GFX1064-NEXT:    v_readlane_b32 s6, v0, s3
+; GFX1064-NEXT:    s_lshl_b64 s[4:5], 1, s3
+; GFX1064-NEXT:    s_andn2_b64 s[0:1], s[0:1], s[4:5]
+; GFX1064-NEXT:    s_add_i32 s2, s2, s6
+; GFX1064-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB10_1
+; GFX1064-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
+; GFX1064-NEXT:    s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-NEXT:    s_xor_b64 s[0:1], exec, s[0:1]
+; GFX1064-NEXT:    s_cbranch_execz .LBB10_4
+; GFX1064-NEXT:  ; %bb.3:
 ; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1064-NEXT:    s_add_i32 s0, s2, s3
-; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
-; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX1064-NEXT:    s_cbranch_execz .LBB10_2
-; GFX1064-NEXT:  ; %bb.1:
-; GFX1064-NEXT:    v_mov_b32_e32 v3, s0
+; GFX1064-NEXT:    v_mov_b32_e32 v1, s2
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT:    ds_sub_u32 v0, v3
+; GFX1064-NEXT:    ds_sub_u32 v0, v1
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_gl0_inv
-; GFX1064-NEXT:  .LBB10_2:
+; GFX1064-NEXT:  .LBB10_4:
 ; GFX1064-NEXT:    s_endpgm
 ;
 ; GFX1032-LABEL: sub_i32_varying_nouse:
 ; GFX1032:       ; %bb.0: ; %entry
-; GFX1032-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    s_or_saveexec_b32 s0, -1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1032-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1032-NEXT:    v_add_nc_u32_e32 v1, v1, v2
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s0
-; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1032-NEXT:    s_mov_b32 s1, exec_lo
+; GFX1032-NEXT:    s_mov_b32 s0, 0
+; GFX1032-NEXT:  .LBB10_1: ; %ComputeLoop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_ff1_i32_b32 s2, s1
+; GFX1032-NEXT:    v_readlane_b32 s3, v0, s2
+; GFX1032-NEXT:    s_lshl_b32 s2, 1, s2
+; GFX1032-NEXT:    s_andn2_b32 s1, s1, s2
+; GFX1032-NEXT:    s_add_i32 s0, s0, s3
+; GFX1032-NEXT:    s_cmp_lg_u32 s1, 0
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB10_1
+; GFX1032-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v1
+; GFX1032-NEXT:    s_and_saveexec_b32 s1, vcc_lo
+; GFX1032-NEXT:    s_xor_b32 s1, exec_lo, s1
+; GFX1032-NEXT:    s_cbranch_execz .LBB10_4
+; GFX1032-NEXT:  ; %bb.3:
 ; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1032-NEXT:    v_mov_b32_e32 v3, v1
-; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX1032-NEXT:    s_and_saveexec_b32 s0, vcc_lo
-; GFX1032-NEXT:    s_cbranch_execz .LBB10_2
-; GFX1032-NEXT:  ; %bb.1:
+; GFX1032-NEXT:    v_mov_b32_e32 v1, s0
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT:    ds_sub_u32 v0, v3
+; GFX1032-NEXT:    ds_sub_u32 v0, v1
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_gl0_inv
-; GFX1032-NEXT:  .LBB10_2:
+; GFX1032-NEXT:  .LBB10_4:
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1164-LABEL: sub_i32_varying_nouse:
 ; GFX1164:       ; %bb.0: ; %entry
-; GFX1164-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    s_or_saveexec_b64 s[0:1], -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1164-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_add_nc_u32_e32 v1, v1, v2
-; GFX1164-NEXT:    v_permlane64_b32 v2, v1
-; GFX1164-NEXT:    s_mov_b64 exec, s[0:1]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[0:1], -1
-; GFX1164-NEXT:    v_add_nc_u32_e32 v1, v1, v2
-; GFX1164-NEXT:    s_mov_b64 exec, s[0:1]
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instid1(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_3) | instskip(SKIP_1) | instid1(VALU_DEP_3)
-; GFX1164-NEXT:    v_mov_b32_e32 v3, v1
+; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
 ; GFX1164-NEXT:    s_mov_b64 s[0:1], exec
-; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v4
-; GFX1164-NEXT:    s_cbranch_execz .LBB10_2
-; GFX1164-NEXT:  ; %bb.1:
+; GFX1164-NEXT:    s_mov_b32 s2, 0
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v1, exec_hi, v1
+; GFX1164-NEXT:  .LBB10_1: ; %ComputeLoop
+; GFX1164-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1164-NEXT:    s_ctz_i32_b32 s3, s1
+; GFX1164-NEXT:    s_ctz_i32_b32 s4, s0
+; GFX1164-NEXT:    s_add_i32 s3, s3, 32
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT:    s_min_u32 s3, s4, s3
+; GFX1164-NEXT:    v_readlane_b32 s6, v0, s3
+; GFX1164-NEXT:    s_lshl_b64 s[4:5], 1, s3
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT:    s_and_not1_b64 s[0:1], s[0:1], s[4:5]
+; GFX1164-NEXT:    s_add_i32 s2, s2, s6
+; GFX1164-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX1164-NEXT:    s_cbranch_scc1 .LBB10_1
+; GFX1164-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1164-NEXT:    s_mov_b64 s[0:1], exec
+; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v1
+; GFX1164-NEXT:    s_xor_b64 s[0:1], exec, s[0:1]
+; GFX1164-NEXT:    s_cbranch_execz .LBB10_4
+; GFX1164-NEXT:  ; %bb.3:
+; GFX1164-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1164-NEXT:    v_mov_b32_e32 v1, s2
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1164-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1164-NEXT:    ds_sub_u32 v0, v3
+; GFX1164-NEXT:    ds_sub_u32 v0, v1
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_gl0_inv
-; GFX1164-NEXT:  .LBB10_2:
+; GFX1164-NEXT:  .LBB10_4:
 ; GFX1164-NEXT:    s_endpgm
 ;
 ; GFX1132-LABEL: sub_i32_varying_nouse:
 ; GFX1132:       ; %bb.0: ; %entry
-; GFX1132-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    s_or_saveexec_b32 s0, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1132-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
-; GFX1132-NEXT:    v_add_nc_u32_e32 v1, v1, v2
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s0
-; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v3, v1
-; GFX1132-NEXT:    s_mov_b32 s0, exec_lo
-; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v4
-; GFX1132-NEXT:    s_cbranch_execz .LBB10_2
-; GFX1132-NEXT:  ; %bb.1:
+; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1132-NEXT:    s_mov_b32 s1, exec_lo
+; GFX1132-NEXT:    s_mov_b32 s0, 0
+; GFX1132-NEXT:  .LBB10_1: ; %ComputeLoop
+; GFX1132-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1132-NEXT:    s_ctz_i32_b32 s2, s1
+; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
+; GFX1132-NEXT:    v_readlane_b32 s3, v0, s2
+; GFX1132-NEXT:    s_lshl_b32 s2, 1, s2
+; GFX1132-NEXT:    s_and_not1_b32 s1, s1, s2
+; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT:    s_add_i32 s0, s0, s3
+; GFX1132-NEXT:    s_cmp_lg_u32 s1, 0
+; GFX1132-NEXT:    s_cbranch_scc1 .LBB10_1
+; GFX1132-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1132-NEXT:    s_mov_b32 s1, exec_lo
+; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v1
+; GFX1132-NEXT:    s_xor_b32 s1, exec_lo, s1
+; GFX1132-NEXT:    s_cbranch_execz .LBB10_4
+; GFX1132-NEXT:  ; %bb.3:
+; GFX1132-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s0
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1132-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1132-NEXT:    ds_sub_u32 v0, v3
+; GFX1132-NEXT:    ds_sub_u32 v0, v1
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_gl0_inv
-; GFX1132-NEXT:  .LBB10_2:
+; GFX1132-NEXT:  .LBB10_4:
 ; GFX1132-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = atomicrmw sub ptr addrspace(3) @local_var32, i32 %lane acq_rel
   ret void
 }
 
 define amdgpu_kernel void @sub_i64_constant(ptr addrspace(1) %out) {

 ; GFX7LESS-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7LESS-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7LESS-NEXT:    s_mov_b32 s2, -1
 ; GFX7LESS-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX7LESS-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: and_i32_varying:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_mov_b32_e32 v1, -1
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_mov_b32_e32 v2, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v2, -1
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_and_b32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_and_b32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_and_b32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_and_b32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_and_b32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_and_b32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v2, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    s_mov_b32 s4, -1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB14_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_and_b32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB14_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB14_2
-; GFX8-NEXT:  ; %bb.1:
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB14_4
+; GFX8-NEXT:  ; %bb.3:
 ; GFX8-NEXT:    v_mov_b32_e32 v0, 0
-; GFX8-NEXT:    v_mov_b32_e32 v3, s4
+; GFX8-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX8-NEXT:    s_mov_b32 m0, -1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    ds_and_rtn_b32 v0, v0, v3
+; GFX8-NEXT:    ds_and_rtn_b32 v0, v0, v2
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:  .LBB14_2:
+; GFX8-NEXT:  .LBB14_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v1
 ; GFX8-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX8-NEXT:    s_mov_b32 s2, -1
-; GFX8-NEXT:    v_and_b32_e32 v0, s4, v0
+; GFX8-NEXT:    v_and_b32_e32 v0, s4, v1
 ; GFX8-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: and_i32_varying:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_mov_b32_e32 v1, -1
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_mov_b32_e32 v2, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v2, -1
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_and_b32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_and_b32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_and_b32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_and_b32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_and_b32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_and_b32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v2, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    s_mov_b32 s4, -1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB14_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_and_b32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB14_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB14_2
-; GFX9-NEXT:  ; %bb.1:
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB14_4
+; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
-; GFX9-NEXT:    v_mov_b32_e32 v3, s4
+; GFX9-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    ds_and_rtn_b32 v0, v0, v3
+; GFX9-NEXT:    ds_and_rtn_b32 v0, v0, v2
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:  .LBB14_2:
+; GFX9-NEXT:  .LBB14_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v1
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s2, -1
-; GFX9-NEXT:    v_and_b32_e32 v0, s4, v0
+; GFX9-NEXT:    v_and_b32_e32 v0, s4, v1
 ; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: and_i32_varying:
 ; GFX1064:       ; %bb.0: ; %entry
-; GFX1064-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    v_mov_b32_e32 v1, -1
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_e32 v3, -1
-; GFX1064-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1064-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1064-NEXT:    v_and_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1064-NEXT:    v_and_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1064-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1064-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1064-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1064-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT:    s_mov_b32 s2, -1
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1064-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1064-NEXT:    s_mov_b32 s4, -1
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1064-NEXT:    ; implicit-def: $vgpr1
+; GFX1064-NEXT:  .LBB14_1: ; %ComputeLoop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX1064-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX1064-NEXT:    s_add_i32 s5, s5, 32
+; GFX1064-NEXT:    s_min_u32 s5, s6, s5
+; GFX1064-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1064-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1064-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1064-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX1064-NEXT:    s_and_b32 s4, s4, s8
+; GFX1064-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB14_1
+; GFX1064-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX1064-NEXT:    ; implicit-def: $vgpr0
-; GFX1064-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1064-NEXT:    s_cbranch_execz .LBB14_2
-; GFX1064-NEXT:  ; %bb.1:
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1064-NEXT:    s_cbranch_execz .LBB14_4
+; GFX1064-NEXT:  ; %bb.3:
 ; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1064-NEXT:    v_mov_b32_e32 v4, s7
-; GFX1064-NEXT:    s_mov_b32 s3, s7
+; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT:    ds_and_rtn_b32 v0, v0, v4
+; GFX1064-NEXT:    ds_and_rtn_b32 v0, v0, v2
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_gl0_inv
-; GFX1064-NEXT:  .LBB14_2:
+; GFX1064-NEXT:  .LBB14_4:
 ; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1064-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1064-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1064-NEXT:    v_and_b32_e32 v0, s3, v0
+; GFX1064-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1064-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT:    v_and_b32_e32 v0, s2, v1
+; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1064-NEXT:    s_endpgm
 ;
 ; GFX1032-LABEL: and_i32_varying:
 ; GFX1032:       ; %bb.0: ; %entry
-; GFX1032-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    v_mov_b32_e32 v1, -1
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_mov_b32_e32 v3, -1
-; GFX1032-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1032-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1032-NEXT:    v_and_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1032-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1032-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1032-NEXT:    s_mov_b32 s3, exec_lo
 ; GFX1032-NEXT:    s_mov_b32 s2, -1
+; GFX1032-NEXT:    ; implicit-def: $vgpr1
+; GFX1032-NEXT:  .LBB14_1: ; %ComputeLoop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX1032-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1032-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1032-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1032-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX1032-NEXT:    s_and_b32 s2, s2, s5
+; GFX1032-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB14_1
+; GFX1032-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
 ; GFX1032-NEXT:    ; implicit-def: $vgpr0
 ; GFX1032-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1032-NEXT:    s_cbranch_execz .LBB14_2
-; GFX1032-NEXT:  ; %bb.1:
+; GFX1032-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1032-NEXT:    s_cbranch_execz .LBB14_4
+; GFX1032-NEXT:  ; %bb.3:
 ; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1032-NEXT:    v_mov_b32_e32 v4, s4
+; GFX1032-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT:    ds_and_rtn_b32 v0, v0, v4
+; GFX1032-NEXT:    ds_and_rtn_b32 v0, v0, v2
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_gl0_inv
-; GFX1032-NEXT:  .LBB14_2:
+; GFX1032-NEXT:  .LBB14_4:
 ; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1032-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1032-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1032-NEXT:    v_and_b32_e32 v0, s3, v0
+; GFX1032-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1032-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT:    v_and_b32_e32 v0, s2, v1
+; GFX1032-NEXT:    s_mov_b32 s2, -1
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1164-LABEL: and_i32_varying:
 ; GFX1164:       ; %bb.0: ; %entry
-; GFX1164-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    v_mov_b32_e32 v1, -1
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_mov_b32_e32 v3, -1
-; GFX1164-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1164-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_and_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1164-NEXT:    v_and_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1164-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1164-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1164-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1164-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1164-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1164-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1164-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1164-NEXT:    s_mov_b32 s2, -1
+; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1164-NEXT:    s_mov_b32 s4, -1
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1164-NEXT:    ; implicit-def: $vgpr1
+; GFX1164-NEXT:  .LBB14_1: ; %ComputeLoop
+; GFX1164-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1164-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX1164-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX1164-NEXT:    s_add_i32 s5, s5, 32
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT:    s_min_u32 s5, s6, s5
+; GFX1164-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1164-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1164-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1164-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX1164-NEXT:    s_and_b32 s4, s4, s8
+; GFX1164-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1164-NEXT:    s_cbranch_scc1 .LBB14_1
+; GFX1164-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
 ; GFX1164-NEXT:    ; implicit-def: $vgpr0
-; GFX1164-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1164-NEXT:    s_cbranch_execz .LBB14_2
-; GFX1164-NEXT:  ; %bb.1:
+; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1164-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1164-NEXT:    s_cbranch_execz .LBB14_4
+; GFX1164-NEXT:  ; %bb.3:
 ; GFX1164-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1164-NEXT:    v_mov_b32_e32 v4, s7
-; GFX1164-NEXT:    s_mov_b32 s3, s7
+; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1164-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1164-NEXT:    ds_and_rtn_b32 v0, v0, v4
+; GFX1164-NEXT:    ds_and_rtn_b32 v0, v0, v2
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_gl0_inv
-; GFX1164-NEXT:  .LBB14_2:
-; GFX1164-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1164-NEXT:  .LBB14_4:
+; GFX1164-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1164-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1164-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1164-NEXT:    v_and_b32_e32 v0, s3, v0
+; GFX1164-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1164-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_and_b32_e32 v0, s2, v1
+; GFX1164-NEXT:    s_mov_b32 s2, -1
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1164-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1164-NEXT:    s_endpgm
 ;
 ; GFX1132-LABEL: and_i32_varying:
 ; GFX1132:       ; %bb.0: ; %entry
-; GFX1132-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    v_mov_b32_e32 v1, -1
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    v_mov_b32_e32 v3, -1
-; GFX1132-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    v_and_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1132-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_and_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1132-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1132-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1132-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
 ; GFX1132-NEXT:    s_mov_b32 s2, -1
+; GFX1132-NEXT:    ; implicit-def: $vgpr1
+; GFX1132-NEXT:  .LBB14_1: ; %ComputeLoop
+; GFX1132-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1132-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX1132-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1132-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1132-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1132-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX1132-NEXT:    s_and_b32 s2, s2, s5
+; GFX1132-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1132-NEXT:    s_cbranch_scc1 .LBB14_1
+; GFX1132-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
 ; GFX1132-NEXT:    ; implicit-def: $vgpr0
-; GFX1132-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1132-NEXT:    s_cbranch_execz .LBB14_2
-; GFX1132-NEXT:  ; %bb.1:
+; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1132-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1132-NEXT:    s_cbranch_execz .LBB14_4
+; GFX1132-NEXT:  ; %bb.3:
 ; GFX1132-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1132-NEXT:    v_mov_b32_e32 v4, s4
+; GFX1132-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1132-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1132-NEXT:    ds_and_rtn_b32 v0, v0, v4
+; GFX1132-NEXT:    ds_and_rtn_b32 v0, v0, v2
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_gl0_inv
-; GFX1132-NEXT:  .LBB14_2:
+; GFX1132-NEXT:  .LBB14_4:
 ; GFX1132-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1132-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1132-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1132-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1132-NEXT:    v_and_b32_e32 v0, s3, v0
+; GFX1132-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1132-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT:    v_and_b32_e32 v0, s2, v1
+; GFX1132-NEXT:    s_mov_b32 s2, -1
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1132-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1132-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = atomicrmw and ptr addrspace(3) @local_var32, i32 %lane acq_rel
   store i32 %old, ptr addrspace(1) %out

 ; GFX7LESS-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7LESS-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7LESS-NEXT:    s_mov_b32 s2, -1
 ; GFX7LESS-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX7LESS-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: or_i32_varying:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    v_mov_b32_e32 v3, 0
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX8-NEXT:    v_mov_b32_e32 v1, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_or_b32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_or_b32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s4, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB15_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_or_b32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB15_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB15_2
-; GFX8-NEXT:  ; %bb.1:
-; GFX8-NEXT:    v_mov_b32_e32 v0, s4
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB15_4
+; GFX8-NEXT:  ; %bb.3:
+; GFX8-NEXT:    v_mov_b32_e32 v0, 0
+; GFX8-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX8-NEXT:    s_mov_b32 m0, -1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    ds_or_rtn_b32 v0, v3, v0
+; GFX8-NEXT:    ds_or_rtn_b32 v0, v0, v2
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:  .LBB15_2:
+; GFX8-NEXT:  .LBB15_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v2
 ; GFX8-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX8-NEXT:    s_mov_b32 s2, -1
-; GFX8-NEXT:    v_or_b32_e32 v0, s4, v0
+; GFX8-NEXT:    v_or_b32_e32 v0, s4, v1
 ; GFX8-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: or_i32_varying:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_or_b32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_or_b32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s4, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB15_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_or_b32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB15_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB15_2
-; GFX9-NEXT:  ; %bb.1:
-; GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB15_4
+; GFX9-NEXT:  ; %bb.3:
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    ds_or_rtn_b32 v0, v3, v0
+; GFX9-NEXT:    ds_or_rtn_b32 v0, v0, v2
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:  .LBB15_2:
+; GFX9-NEXT:  .LBB15_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v2
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s2, -1
-; GFX9-NEXT:    v_or_b32_e32 v0, s4, v0
+; GFX9-NEXT:    v_or_b32_e32 v0, s4, v1
 ; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: or_i32_varying:
 ; GFX1064:       ; %bb.0: ; %entry
-; GFX1064-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1064-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1064-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1064-NEXT:    v_or_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1064-NEXT:    v_or_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1064-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1064-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1064-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1064-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT:    s_mov_b32 s2, -1
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1064-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1064-NEXT:    s_mov_b32 s4, 0
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1064-NEXT:    ; implicit-def: $vgpr1
+; GFX1064-NEXT:  .LBB15_1: ; %ComputeLoop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX1064-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX1064-NEXT:    s_add_i32 s5, s5, 32
+; GFX1064-NEXT:    s_min_u32 s5, s6, s5
+; GFX1064-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1064-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1064-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1064-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX1064-NEXT:    s_or_b32 s4, s4, s8
+; GFX1064-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB15_1
+; GFX1064-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX1064-NEXT:    ; implicit-def: $vgpr0
-; GFX1064-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1064-NEXT:    s_cbranch_execz .LBB15_2
-; GFX1064-NEXT:  ; %bb.1:
-; GFX1064-NEXT:    v_mov_b32_e32 v0, s7
-; GFX1064-NEXT:    s_mov_b32 s3, s7
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1064-NEXT:    s_cbranch_execz .LBB15_4
+; GFX1064-NEXT:  ; %bb.3:
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT:    ds_or_rtn_b32 v0, v4, v0
+; GFX1064-NEXT:    ds_or_rtn_b32 v0, v0, v2
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_gl0_inv
-; GFX1064-NEXT:  .LBB15_2:
+; GFX1064-NEXT:  .LBB15_4:
 ; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1064-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1064-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1064-NEXT:    v_or_b32_e32 v0, s3, v0
+; GFX1064-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1064-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT:    v_or_b32_e32 v0, s2, v1
+; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1064-NEXT:    s_endpgm
 ;
 ; GFX1032-LABEL: or_i32_varying:
 ; GFX1032:       ; %bb.0: ; %entry
-; GFX1032-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1032-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1032-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1032-NEXT:    v_or_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1032-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1032-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT:    s_mov_b32 s2, -1
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1032-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1032-NEXT:    s_mov_b32 s2, 0
+; GFX1032-NEXT:    ; implicit-def: $vgpr1
+; GFX1032-NEXT:  .LBB15_1: ; %ComputeLoop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX1032-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1032-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1032-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1032-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX1032-NEXT:    s_or_b32 s2, s2, s5
+; GFX1032-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB15_1
+; GFX1032-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
 ; GFX1032-NEXT:    ; implicit-def: $vgpr0
 ; GFX1032-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1032-NEXT:    s_cbranch_execz .LBB15_2
-; GFX1032-NEXT:  ; %bb.1:
-; GFX1032-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1032-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1032-NEXT:    s_cbranch_execz .LBB15_4
+; GFX1032-NEXT:  ; %bb.3:
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT:    ds_or_rtn_b32 v0, v4, v0
+; GFX1032-NEXT:    ds_or_rtn_b32 v0, v0, v2
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_gl0_inv
-; GFX1032-NEXT:  .LBB15_2:
+; GFX1032-NEXT:  .LBB15_4:
 ; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1032-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1032-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1032-NEXT:    v_or_b32_e32 v0, s3, v0
+; GFX1032-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1032-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT:    v_or_b32_e32 v0, s2, v1
+; GFX1032-NEXT:    s_mov_b32 s2, -1
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1164-LABEL: or_i32_varying:
 ; GFX1164:       ; %bb.0: ; %entry
-; GFX1164-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1164-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1164-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_or_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1164-NEXT:    v_or_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1164-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1164-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1164-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1164-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1164-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1164-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1164-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1164-NEXT:    s_mov_b32 s2, -1
+; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1164-NEXT:    s_mov_b32 s4, 0
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1164-NEXT:    ; implicit-def: $vgpr1
+; GFX1164-NEXT:  .LBB15_1: ; %ComputeLoop
+; GFX1164-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1164-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX1164-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX1164-NEXT:    s_add_i32 s5, s5, 32
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT:    s_min_u32 s5, s6, s5
+; GFX1164-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1164-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1164-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1164-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX1164-NEXT:    s_or_b32 s4, s4, s8
+; GFX1164-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1164-NEXT:    s_cbranch_scc1 .LBB15_1
+; GFX1164-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
 ; GFX1164-NEXT:    ; implicit-def: $vgpr0
-; GFX1164-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1164-NEXT:    s_cbranch_execz .LBB15_2
-; GFX1164-NEXT:  ; %bb.1:
-; GFX1164-NEXT:    v_mov_b32_e32 v0, s7
-; GFX1164-NEXT:    s_mov_b32 s3, s7
+; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1164-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1164-NEXT:    s_cbranch_execz .LBB15_4
+; GFX1164-NEXT:  ; %bb.3:
+; GFX1164-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1164-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1164-NEXT:    ds_or_rtn_b32 v0, v4, v0
+; GFX1164-NEXT:    ds_or_rtn_b32 v0, v0, v2
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_gl0_inv
-; GFX1164-NEXT:  .LBB15_2:
-; GFX1164-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1164-NEXT:  .LBB15_4:
+; GFX1164-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1164-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1164-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1164-NEXT:    v_or_b32_e32 v0, s3, v0
+; GFX1164-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1164-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_or_b32_e32 v0, s2, v1
+; GFX1164-NEXT:    s_mov_b32 s2, -1
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1164-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1164-NEXT:    s_endpgm
 ;
 ; GFX1132-LABEL: or_i32_varying:
 ; GFX1132:       ; %bb.0: ; %entry
-; GFX1132-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1132-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_or_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1132-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_or_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1132-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1132-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1132-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1132-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1132-NEXT:    s_mov_b32 s2, -1
+; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1132-NEXT:    s_mov_b32 s2, 0
+; GFX1132-NEXT:    ; implicit-def: $vgpr1
+; GFX1132-NEXT:  .LBB15_1: ; %ComputeLoop
+; GFX1132-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1132-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX1132-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1132-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1132-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1132-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX1132-NEXT:    s_or_b32 s2, s2, s5
+; GFX1132-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1132-NEXT:    s_cbranch_scc1 .LBB15_1
+; GFX1132-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
 ; GFX1132-NEXT:    ; implicit-def: $vgpr0
-; GFX1132-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1132-NEXT:    s_cbranch_execz .LBB15_2
-; GFX1132-NEXT:  ; %bb.1:
-; GFX1132-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1132-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1132-NEXT:    s_cbranch_execz .LBB15_4
+; GFX1132-NEXT:  ; %bb.3:
+; GFX1132-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1132-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1132-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1132-NEXT:    ds_or_rtn_b32 v0, v4, v0
+; GFX1132-NEXT:    ds_or_rtn_b32 v0, v0, v2
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_gl0_inv
-; GFX1132-NEXT:  .LBB15_2:
+; GFX1132-NEXT:  .LBB15_4:
 ; GFX1132-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1132-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1132-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1132-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1132-NEXT:    v_or_b32_e32 v0, s3, v0
+; GFX1132-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1132-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT:    v_or_b32_e32 v0, s2, v1
+; GFX1132-NEXT:    s_mov_b32 s2, -1
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1132-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1132-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = atomicrmw or ptr addrspace(3) @local_var32, i32 %lane acq_rel
   store i32 %old, ptr addrspace(1) %out

 ; GFX7LESS-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7LESS-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7LESS-NEXT:    s_mov_b32 s2, -1
 ; GFX7LESS-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX7LESS-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: xor_i32_varying:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    v_mov_b32_e32 v3, 0
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX8-NEXT:    v_mov_b32_e32 v1, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_xor_b32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_xor_b32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s4, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB16_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_xor_b32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB16_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB16_2
-; GFX8-NEXT:  ; %bb.1:
-; GFX8-NEXT:    v_mov_b32_e32 v0, s4
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB16_4
+; GFX8-NEXT:  ; %bb.3:
+; GFX8-NEXT:    v_mov_b32_e32 v0, 0
+; GFX8-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX8-NEXT:    s_mov_b32 m0, -1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    ds_xor_rtn_b32 v0, v3, v0
+; GFX8-NEXT:    ds_xor_rtn_b32 v0, v0, v2
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:  .LBB16_2:
+; GFX8-NEXT:  .LBB16_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v2
 ; GFX8-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX8-NEXT:    s_mov_b32 s2, -1
-; GFX8-NEXT:    v_xor_b32_e32 v0, s4, v0
+; GFX8-NEXT:    v_xor_b32_e32 v0, s4, v1
 ; GFX8-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: xor_i32_varying:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_xor_b32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_xor_b32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s4, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB16_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_xor_b32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB16_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB16_2
-; GFX9-NEXT:  ; %bb.1:
-; GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB16_4
+; GFX9-NEXT:  ; %bb.3:
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    ds_xor_rtn_b32 v0, v3, v0
+; GFX9-NEXT:    ds_xor_rtn_b32 v0, v0, v2
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:  .LBB16_2:
+; GFX9-NEXT:  .LBB16_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v2
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s2, -1
-; GFX9-NEXT:    v_xor_b32_e32 v0, s4, v0
+; GFX9-NEXT:    v_xor_b32_e32 v0, s4, v1
 ; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: xor_i32_varying:
 ; GFX1064:       ; %bb.0: ; %entry
-; GFX1064-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1064-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1064-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1064-NEXT:    v_xor_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1064-NEXT:    v_xor_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1064-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1064-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1064-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1064-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT:    s_mov_b32 s2, -1
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1064-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1064-NEXT:    s_mov_b32 s4, 0
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1064-NEXT:    ; implicit-def: $vgpr1
+; GFX1064-NEXT:  .LBB16_1: ; %ComputeLoop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX1064-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX1064-NEXT:    s_add_i32 s5, s5, 32
+; GFX1064-NEXT:    s_min_u32 s5, s6, s5
+; GFX1064-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1064-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1064-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1064-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX1064-NEXT:    s_xor_b32 s4, s4, s8
+; GFX1064-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB16_1
+; GFX1064-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX1064-NEXT:    ; implicit-def: $vgpr0
-; GFX1064-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1064-NEXT:    s_cbranch_execz .LBB16_2
-; GFX1064-NEXT:  ; %bb.1:
-; GFX1064-NEXT:    v_mov_b32_e32 v0, s7
-; GFX1064-NEXT:    s_mov_b32 s3, s7
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1064-NEXT:    s_cbranch_execz .LBB16_4
+; GFX1064-NEXT:  ; %bb.3:
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT:    ds_xor_rtn_b32 v0, v4, v0
+; GFX1064-NEXT:    ds_xor_rtn_b32 v0, v0, v2
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_gl0_inv
-; GFX1064-NEXT:  .LBB16_2:
+; GFX1064-NEXT:  .LBB16_4:
 ; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1064-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1064-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1064-NEXT:    v_xor_b32_e32 v0, s3, v0
+; GFX1064-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1064-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT:    v_xor_b32_e32 v0, s2, v1
+; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1064-NEXT:    s_endpgm
 ;
 ; GFX1032-LABEL: xor_i32_varying:
 ; GFX1032:       ; %bb.0: ; %entry
-; GFX1032-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1032-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1032-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1032-NEXT:    v_xor_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1032-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1032-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT:    s_mov_b32 s2, -1
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1032-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1032-NEXT:    s_mov_b32 s2, 0
+; GFX1032-NEXT:    ; implicit-def: $vgpr1
+; GFX1032-NEXT:  .LBB16_1: ; %ComputeLoop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX1032-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1032-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1032-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1032-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX1032-NEXT:    s_xor_b32 s2, s2, s5
+; GFX1032-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB16_1
+; GFX1032-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
 ; GFX1032-NEXT:    ; implicit-def: $vgpr0
 ; GFX1032-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1032-NEXT:    s_cbranch_execz .LBB16_2
-; GFX1032-NEXT:  ; %bb.1:
-; GFX1032-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1032-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1032-NEXT:    s_cbranch_execz .LBB16_4
+; GFX1032-NEXT:  ; %bb.3:
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT:    ds_xor_rtn_b32 v0, v4, v0
+; GFX1032-NEXT:    ds_xor_rtn_b32 v0, v0, v2
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_gl0_inv
-; GFX1032-NEXT:  .LBB16_2:
+; GFX1032-NEXT:  .LBB16_4:
 ; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1032-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1032-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1032-NEXT:    v_xor_b32_e32 v0, s3, v0
+; GFX1032-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1032-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT:    v_xor_b32_e32 v0, s2, v1
+; GFX1032-NEXT:    s_mov_b32 s2, -1
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1164-LABEL: xor_i32_varying:
 ; GFX1164:       ; %bb.0: ; %entry
-; GFX1164-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1164-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1164-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_xor_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1164-NEXT:    v_xor_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1164-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1164-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1164-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1164-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1164-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1164-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1164-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1164-NEXT:    s_mov_b32 s2, -1
+; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1164-NEXT:    s_mov_b32 s4, 0
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1164-NEXT:    ; implicit-def: $vgpr1
+; GFX1164-NEXT:  .LBB16_1: ; %ComputeLoop
+; GFX1164-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1164-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX1164-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX1164-NEXT:    s_add_i32 s5, s5, 32
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT:    s_min_u32 s5, s6, s5
+; GFX1164-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1164-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1164-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1164-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX1164-NEXT:    s_xor_b32 s4, s4, s8
+; GFX1164-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1164-NEXT:    s_cbranch_scc1 .LBB16_1
+; GFX1164-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
 ; GFX1164-NEXT:    ; implicit-def: $vgpr0
-; GFX1164-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1164-NEXT:    s_cbranch_execz .LBB16_2
-; GFX1164-NEXT:  ; %bb.1:
-; GFX1164-NEXT:    v_mov_b32_e32 v0, s7
-; GFX1164-NEXT:    s_mov_b32 s3, s7
+; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1164-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1164-NEXT:    s_cbranch_execz .LBB16_4
+; GFX1164-NEXT:  ; %bb.3:
+; GFX1164-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1164-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1164-NEXT:    ds_xor_rtn_b32 v0, v4, v0
+; GFX1164-NEXT:    ds_xor_rtn_b32 v0, v0, v2
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_gl0_inv
-; GFX1164-NEXT:  .LBB16_2:
-; GFX1164-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1164-NEXT:  .LBB16_4:
+; GFX1164-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1164-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1164-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1164-NEXT:    v_xor_b32_e32 v0, s3, v0
+; GFX1164-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1164-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_xor_b32_e32 v0, s2, v1
+; GFX1164-NEXT:    s_mov_b32 s2, -1
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1164-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1164-NEXT:    s_endpgm
 ;
 ; GFX1132-LABEL: xor_i32_varying:
 ; GFX1132:       ; %bb.0: ; %entry
-; GFX1132-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1132-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_xor_b32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1132-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_xor_b32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1132-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1132-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1132-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1132-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1132-NEXT:    s_mov_b32 s2, -1
+; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1132-NEXT:    s_mov_b32 s2, 0
+; GFX1132-NEXT:    ; implicit-def: $vgpr1
+; GFX1132-NEXT:  .LBB16_1: ; %ComputeLoop
+; GFX1132-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1132-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX1132-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1132-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1132-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1132-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX1132-NEXT:    s_xor_b32 s2, s2, s5
+; GFX1132-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1132-NEXT:    s_cbranch_scc1 .LBB16_1
+; GFX1132-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
 ; GFX1132-NEXT:    ; implicit-def: $vgpr0
-; GFX1132-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1132-NEXT:    s_cbranch_execz .LBB16_2
-; GFX1132-NEXT:  ; %bb.1:
-; GFX1132-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1132-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1132-NEXT:    s_cbranch_execz .LBB16_4
+; GFX1132-NEXT:  ; %bb.3:
+; GFX1132-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1132-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1132-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1132-NEXT:    ds_xor_rtn_b32 v0, v4, v0
+; GFX1132-NEXT:    ds_xor_rtn_b32 v0, v0, v2
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_gl0_inv
-; GFX1132-NEXT:  .LBB16_2:
+; GFX1132-NEXT:  .LBB16_4:
 ; GFX1132-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1132-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1132-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1132-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1132-NEXT:    v_xor_b32_e32 v0, s3, v0
+; GFX1132-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1132-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT:    v_xor_b32_e32 v0, s2, v1
+; GFX1132-NEXT:    s_mov_b32 s2, -1
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1132-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1132-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = atomicrmw xor ptr addrspace(3) @local_var32, i32 %lane acq_rel
   store i32 %old, ptr addrspace(1) %out

 ; GFX7LESS-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7LESS-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7LESS-NEXT:    s_mov_b32 s2, -1
 ; GFX7LESS-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX7LESS-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: max_i32_varying:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_bfrev_b32_e32 v1, 1
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_mov_b32_e32 v2, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_bfrev_b32_e32 v2, 1
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_max_i32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_max_i32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_max_i32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_max_i32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_max_i32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_max_i32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v2, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    s_brev_b32 s4, 1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB17_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_max_i32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB17_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB17_2
-; GFX8-NEXT:  ; %bb.1:
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB17_4
+; GFX8-NEXT:  ; %bb.3:
 ; GFX8-NEXT:    v_mov_b32_e32 v0, 0
-; GFX8-NEXT:    v_mov_b32_e32 v3, s4
+; GFX8-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX8-NEXT:    s_mov_b32 m0, -1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    ds_max_rtn_i32 v0, v0, v3
+; GFX8-NEXT:    ds_max_rtn_i32 v0, v0, v2
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:  .LBB17_2:
+; GFX8-NEXT:  .LBB17_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v1
 ; GFX8-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX8-NEXT:    s_mov_b32 s2, -1
-; GFX8-NEXT:    v_max_i32_e32 v0, s4, v0
+; GFX8-NEXT:    v_max_i32_e32 v0, s4, v1
 ; GFX8-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: max_i32_varying:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_bfrev_b32_e32 v1, 1
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_mov_b32_e32 v2, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_bfrev_b32_e32 v2, 1
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_max_i32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_max_i32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_max_i32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_max_i32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_max_i32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_max_i32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v2, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    s_brev_b32 s4, 1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB17_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_max_i32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB17_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB17_2
-; GFX9-NEXT:  ; %bb.1:
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB17_4
+; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
-; GFX9-NEXT:    v_mov_b32_e32 v3, s4
+; GFX9-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    ds_max_rtn_i32 v0, v0, v3
+; GFX9-NEXT:    ds_max_rtn_i32 v0, v0, v2
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:  .LBB17_2:
+; GFX9-NEXT:  .LBB17_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v1
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s2, -1
-; GFX9-NEXT:    v_max_i32_e32 v0, s4, v0
+; GFX9-NEXT:    v_max_i32_e32 v0, s4, v1
 ; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: max_i32_varying:
 ; GFX1064:       ; %bb.0: ; %entry
-; GFX1064-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    v_bfrev_b32_e32 v1, 1
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_bfrev_b32_e32 v3, 1
-; GFX1064-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1064-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1064-NEXT:    v_max_i32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1064-NEXT:    v_max_i32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1064-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1064-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1064-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1064-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT:    s_mov_b32 s2, -1
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1064-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1064-NEXT:    s_brev_b32 s4, 1
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1064-NEXT:    ; implicit-def: $vgpr1
+; GFX1064-NEXT:  .LBB17_1: ; %ComputeLoop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX1064-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX1064-NEXT:    s_add_i32 s5, s5, 32
+; GFX1064-NEXT:    s_min_u32 s5, s6, s5
+; GFX1064-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1064-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1064-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1064-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX1064-NEXT:    s_max_i32 s4, s4, s8
+; GFX1064-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB17_1
+; GFX1064-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX1064-NEXT:    ; implicit-def: $vgpr0
-; GFX1064-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1064-NEXT:    s_cbranch_execz .LBB17_2
-; GFX1064-NEXT:  ; %bb.1:
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1064-NEXT:    s_cbranch_execz .LBB17_4
+; GFX1064-NEXT:  ; %bb.3:
 ; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1064-NEXT:    v_mov_b32_e32 v4, s7
-; GFX1064-NEXT:    s_mov_b32 s3, s7
+; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT:    ds_max_rtn_i32 v0, v0, v4
+; GFX1064-NEXT:    ds_max_rtn_i32 v0, v0, v2
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_gl0_inv
-; GFX1064-NEXT:  .LBB17_2:
+; GFX1064-NEXT:  .LBB17_4:
 ; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1064-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1064-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1064-NEXT:    v_max_i32_e32 v0, s3, v0
+; GFX1064-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1064-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT:    v_max_i32_e32 v0, s2, v1
+; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1064-NEXT:    s_endpgm
 ;
 ; GFX1032-LABEL: max_i32_varying:
 ; GFX1032:       ; %bb.0: ; %entry
-; GFX1032-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    v_bfrev_b32_e32 v1, 1
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_bfrev_b32_e32 v3, 1
-; GFX1032-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1032-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1032-NEXT:    v_max_i32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1032-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1032-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT:    s_mov_b32 s2, -1
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1032-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1032-NEXT:    s_brev_b32 s2, 1
+; GFX1032-NEXT:    ; implicit-def: $vgpr1
+; GFX1032-NEXT:  .LBB17_1: ; %ComputeLoop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX1032-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1032-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1032-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1032-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX1032-NEXT:    s_max_i32 s2, s2, s5
+; GFX1032-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB17_1
+; GFX1032-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
 ; GFX1032-NEXT:    ; implicit-def: $vgpr0
 ; GFX1032-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1032-NEXT:    s_cbranch_execz .LBB17_2
-; GFX1032-NEXT:  ; %bb.1:
+; GFX1032-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1032-NEXT:    s_cbranch_execz .LBB17_4
+; GFX1032-NEXT:  ; %bb.3:
 ; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1032-NEXT:    v_mov_b32_e32 v4, s4
+; GFX1032-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT:    ds_max_rtn_i32 v0, v0, v4
+; GFX1032-NEXT:    ds_max_rtn_i32 v0, v0, v2
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_gl0_inv
-; GFX1032-NEXT:  .LBB17_2:
+; GFX1032-NEXT:  .LBB17_4:
 ; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1032-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1032-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1032-NEXT:    v_max_i32_e32 v0, s3, v0
+; GFX1032-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1032-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT:    v_max_i32_e32 v0, s2, v1
+; GFX1032-NEXT:    s_mov_b32 s2, -1
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1164-LABEL: max_i32_varying:
 ; GFX1164:       ; %bb.0: ; %entry
-; GFX1164-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    v_bfrev_b32_e32 v1, 1
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_bfrev_b32_e32 v3, 1
-; GFX1164-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1164-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_max_i32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1164-NEXT:    v_max_i32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1164-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1164-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1164-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1164-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1164-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1164-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1164-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1164-NEXT:    s_mov_b32 s2, -1
+; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1164-NEXT:    s_brev_b32 s4, 1
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1164-NEXT:    ; implicit-def: $vgpr1
+; GFX1164-NEXT:  .LBB17_1: ; %ComputeLoop
+; GFX1164-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1164-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX1164-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX1164-NEXT:    s_add_i32 s5, s5, 32
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT:    s_min_u32 s5, s6, s5
+; GFX1164-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1164-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1164-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1164-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX1164-NEXT:    s_max_i32 s4, s4, s8
+; GFX1164-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1164-NEXT:    s_cbranch_scc1 .LBB17_1
+; GFX1164-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
 ; GFX1164-NEXT:    ; implicit-def: $vgpr0
-; GFX1164-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1164-NEXT:    s_cbranch_execz .LBB17_2
-; GFX1164-NEXT:  ; %bb.1:
+; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1164-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1164-NEXT:    s_cbranch_execz .LBB17_4
+; GFX1164-NEXT:  ; %bb.3:
 ; GFX1164-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1164-NEXT:    v_mov_b32_e32 v4, s7
-; GFX1164-NEXT:    s_mov_b32 s3, s7
+; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1164-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1164-NEXT:    ds_max_rtn_i32 v0, v0, v4
+; GFX1164-NEXT:    ds_max_rtn_i32 v0, v0, v2
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_gl0_inv
-; GFX1164-NEXT:  .LBB17_2:
-; GFX1164-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1164-NEXT:  .LBB17_4:
+; GFX1164-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1164-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1164-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1164-NEXT:    v_max_i32_e32 v0, s3, v0
+; GFX1164-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1164-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_max_i32_e32 v0, s2, v1
+; GFX1164-NEXT:    s_mov_b32 s2, -1
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1164-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1164-NEXT:    s_endpgm
 ;
 ; GFX1132-LABEL: max_i32_varying:
 ; GFX1132:       ; %bb.0: ; %entry
-; GFX1132-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    v_bfrev_b32_e32 v1, 1
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    v_bfrev_b32_e32 v3, 1
-; GFX1132-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    v_max_i32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1132-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_max_i32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1132-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1132-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1132-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
+; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1132-NEXT:    s_brev_b32 s2, 1
+; GFX1132-NEXT:    ; implicit-def: $vgpr1
+; GFX1132-NEXT:  .LBB17_1: ; %ComputeLoop
+; GFX1132-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1132-NEXT:    s_ctz_i32_b32 s4, s3
 ; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1132-NEXT:    s_mov_b32 s2, -1
+; GFX1132-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1132-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1132-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1132-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX1132-NEXT:    s_max_i32 s2, s2, s5
+; GFX1132-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1132-NEXT:    s_cbranch_scc1 .LBB17_1
+; GFX1132-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
 ; GFX1132-NEXT:    ; implicit-def: $vgpr0
-; GFX1132-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1132-NEXT:    s_cbranch_execz .LBB17_2
-; GFX1132-NEXT:  ; %bb.1:
+; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1132-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1132-NEXT:    s_cbranch_execz .LBB17_4
+; GFX1132-NEXT:  ; %bb.3:
 ; GFX1132-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1132-NEXT:    v_mov_b32_e32 v4, s4
+; GFX1132-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1132-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1132-NEXT:    ds_max_rtn_i32 v0, v0, v4
+; GFX1132-NEXT:    ds_max_rtn_i32 v0, v0, v2
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_gl0_inv
-; GFX1132-NEXT:  .LBB17_2:
+; GFX1132-NEXT:  .LBB17_4:
 ; GFX1132-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1132-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1132-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1132-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1132-NEXT:    v_max_i32_e32 v0, s3, v0
+; GFX1132-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1132-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT:    v_max_i32_e32 v0, s2, v1
+; GFX1132-NEXT:    s_mov_b32 s2, -1
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1132-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1132-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = atomicrmw max ptr addrspace(3) @local_var32, i32 %lane acq_rel
   store i32 %old, ptr addrspace(1) %out

 ; GFX7LESS-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7LESS-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7LESS-NEXT:    s_mov_b32 s2, -1
 ; GFX7LESS-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX7LESS-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: min_i32_varying:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_bfrev_b32_e32 v1, -2
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_mov_b32_e32 v2, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_bfrev_b32_e32 v2, -2
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_min_i32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_min_i32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_min_i32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_min_i32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_min_i32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_min_i32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v2, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    s_brev_b32 s4, -2
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB19_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_min_i32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB19_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB19_2
-; GFX8-NEXT:  ; %bb.1:
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB19_4
+; GFX8-NEXT:  ; %bb.3:
 ; GFX8-NEXT:    v_mov_b32_e32 v0, 0
-; GFX8-NEXT:    v_mov_b32_e32 v3, s4
+; GFX8-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX8-NEXT:    s_mov_b32 m0, -1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    ds_min_rtn_i32 v0, v0, v3
+; GFX8-NEXT:    ds_min_rtn_i32 v0, v0, v2
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:  .LBB19_2:
+; GFX8-NEXT:  .LBB19_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v1
 ; GFX8-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX8-NEXT:    s_mov_b32 s2, -1
-; GFX8-NEXT:    v_min_i32_e32 v0, s4, v0
+; GFX8-NEXT:    v_min_i32_e32 v0, s4, v1
 ; GFX8-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: min_i32_varying:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_bfrev_b32_e32 v1, -2
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_mov_b32_e32 v2, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_bfrev_b32_e32 v2, -2
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_min_i32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_min_i32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_min_i32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_min_i32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_min_i32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_min_i32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v2, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    s_brev_b32 s4, -2
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB19_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_min_i32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB19_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB19_2
-; GFX9-NEXT:  ; %bb.1:
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB19_4
+; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
-; GFX9-NEXT:    v_mov_b32_e32 v3, s4
+; GFX9-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    ds_min_rtn_i32 v0, v0, v3
+; GFX9-NEXT:    ds_min_rtn_i32 v0, v0, v2
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:  .LBB19_2:
+; GFX9-NEXT:  .LBB19_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v1
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s2, -1
-; GFX9-NEXT:    v_min_i32_e32 v0, s4, v0
+; GFX9-NEXT:    v_min_i32_e32 v0, s4, v1
 ; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: min_i32_varying:
 ; GFX1064:       ; %bb.0: ; %entry
-; GFX1064-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    v_bfrev_b32_e32 v1, -2
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_bfrev_b32_e32 v3, -2
-; GFX1064-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1064-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1064-NEXT:    v_min_i32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1064-NEXT:    v_min_i32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1064-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1064-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1064-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1064-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT:    s_mov_b32 s2, -1
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1064-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1064-NEXT:    s_brev_b32 s4, -2
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1064-NEXT:    ; implicit-def: $vgpr1
+; GFX1064-NEXT:  .LBB19_1: ; %ComputeLoop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX1064-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX1064-NEXT:    s_add_i32 s5, s5, 32
+; GFX1064-NEXT:    s_min_u32 s5, s6, s5
+; GFX1064-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1064-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1064-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1064-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX1064-NEXT:    s_min_i32 s4, s4, s8
+; GFX1064-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB19_1
+; GFX1064-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX1064-NEXT:    ; implicit-def: $vgpr0
-; GFX1064-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1064-NEXT:    s_cbranch_execz .LBB19_2
-; GFX1064-NEXT:  ; %bb.1:
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1064-NEXT:    s_cbranch_execz .LBB19_4
+; GFX1064-NEXT:  ; %bb.3:
 ; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1064-NEXT:    v_mov_b32_e32 v4, s7
-; GFX1064-NEXT:    s_mov_b32 s3, s7
+; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT:    ds_min_rtn_i32 v0, v0, v4
+; GFX1064-NEXT:    ds_min_rtn_i32 v0, v0, v2
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_gl0_inv
-; GFX1064-NEXT:  .LBB19_2:
+; GFX1064-NEXT:  .LBB19_4:
 ; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1064-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1064-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1064-NEXT:    v_min_i32_e32 v0, s3, v0
+; GFX1064-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1064-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT:    v_min_i32_e32 v0, s2, v1
+; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1064-NEXT:    s_endpgm
 ;
 ; GFX1032-LABEL: min_i32_varying:
 ; GFX1032:       ; %bb.0: ; %entry
-; GFX1032-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    v_bfrev_b32_e32 v1, -2
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_bfrev_b32_e32 v3, -2
-; GFX1032-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1032-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1032-NEXT:    v_min_i32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1032-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1032-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT:    s_mov_b32 s2, -1
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1032-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1032-NEXT:    s_brev_b32 s2, -2
+; GFX1032-NEXT:    ; implicit-def: $vgpr1
+; GFX1032-NEXT:  .LBB19_1: ; %ComputeLoop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX1032-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1032-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1032-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1032-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX1032-NEXT:    s_min_i32 s2, s2, s5
+; GFX1032-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB19_1
+; GFX1032-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
 ; GFX1032-NEXT:    ; implicit-def: $vgpr0
 ; GFX1032-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1032-NEXT:    s_cbranch_execz .LBB19_2
-; GFX1032-NEXT:  ; %bb.1:
+; GFX1032-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1032-NEXT:    s_cbranch_execz .LBB19_4
+; GFX1032-NEXT:  ; %bb.3:
 ; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1032-NEXT:    v_mov_b32_e32 v4, s4
+; GFX1032-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT:    ds_min_rtn_i32 v0, v0, v4
+; GFX1032-NEXT:    ds_min_rtn_i32 v0, v0, v2
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_gl0_inv
-; GFX1032-NEXT:  .LBB19_2:
+; GFX1032-NEXT:  .LBB19_4:
 ; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1032-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1032-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1032-NEXT:    v_min_i32_e32 v0, s3, v0
+; GFX1032-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1032-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT:    v_min_i32_e32 v0, s2, v1
+; GFX1032-NEXT:    s_mov_b32 s2, -1
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1164-LABEL: min_i32_varying:
 ; GFX1164:       ; %bb.0: ; %entry
-; GFX1164-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    v_bfrev_b32_e32 v1, -2
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_bfrev_b32_e32 v3, -2
-; GFX1164-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1164-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_min_i32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1164-NEXT:    v_min_i32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1164-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1164-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1164-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1164-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1164-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1164-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1164-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1164-NEXT:    s_mov_b32 s2, -1
+; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1164-NEXT:    s_brev_b32 s4, -2
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1164-NEXT:    ; implicit-def: $vgpr1
+; GFX1164-NEXT:  .LBB19_1: ; %ComputeLoop
+; GFX1164-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1164-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX1164-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX1164-NEXT:    s_add_i32 s5, s5, 32
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT:    s_min_u32 s5, s6, s5
+; GFX1164-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1164-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1164-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1164-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX1164-NEXT:    s_min_i32 s4, s4, s8
+; GFX1164-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1164-NEXT:    s_cbranch_scc1 .LBB19_1
+; GFX1164-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
 ; GFX1164-NEXT:    ; implicit-def: $vgpr0
-; GFX1164-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1164-NEXT:    s_cbranch_execz .LBB19_2
-; GFX1164-NEXT:  ; %bb.1:
+; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1164-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1164-NEXT:    s_cbranch_execz .LBB19_4
+; GFX1164-NEXT:  ; %bb.3:
 ; GFX1164-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1164-NEXT:    v_mov_b32_e32 v4, s7
-; GFX1164-NEXT:    s_mov_b32 s3, s7
+; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1164-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1164-NEXT:    ds_min_rtn_i32 v0, v0, v4
+; GFX1164-NEXT:    ds_min_rtn_i32 v0, v0, v2
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_gl0_inv
-; GFX1164-NEXT:  .LBB19_2:
-; GFX1164-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1164-NEXT:  .LBB19_4:
+; GFX1164-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1164-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1164-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1164-NEXT:    v_min_i32_e32 v0, s3, v0
+; GFX1164-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1164-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_min_i32_e32 v0, s2, v1
+; GFX1164-NEXT:    s_mov_b32 s2, -1
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1164-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1164-NEXT:    s_endpgm
 ;
 ; GFX1132-LABEL: min_i32_varying:
 ; GFX1132:       ; %bb.0: ; %entry
-; GFX1132-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    v_bfrev_b32_e32 v1, -2
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    v_bfrev_b32_e32 v3, -2
-; GFX1132-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    v_min_i32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1132-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_min_i32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1132-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1132-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1132-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
+; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1132-NEXT:    s_brev_b32 s2, -2
+; GFX1132-NEXT:    ; implicit-def: $vgpr1
+; GFX1132-NEXT:  .LBB19_1: ; %ComputeLoop
+; GFX1132-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1132-NEXT:    s_ctz_i32_b32 s4, s3
 ; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1132-NEXT:    s_mov_b32 s2, -1
+; GFX1132-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1132-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1132-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1132-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX1132-NEXT:    s_min_i32 s2, s2, s5
+; GFX1132-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1132-NEXT:    s_cbranch_scc1 .LBB19_1
+; GFX1132-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
 ; GFX1132-NEXT:    ; implicit-def: $vgpr0
-; GFX1132-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1132-NEXT:    s_cbranch_execz .LBB19_2
-; GFX1132-NEXT:  ; %bb.1:
+; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1132-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1132-NEXT:    s_cbranch_execz .LBB19_4
+; GFX1132-NEXT:  ; %bb.3:
 ; GFX1132-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1132-NEXT:    v_mov_b32_e32 v4, s4
+; GFX1132-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1132-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1132-NEXT:    ds_min_rtn_i32 v0, v0, v4
+; GFX1132-NEXT:    ds_min_rtn_i32 v0, v0, v2
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_gl0_inv
-; GFX1132-NEXT:  .LBB19_2:
+; GFX1132-NEXT:  .LBB19_4:
 ; GFX1132-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1132-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1132-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1132-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1132-NEXT:    v_min_i32_e32 v0, s3, v0
+; GFX1132-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1132-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT:    v_min_i32_e32 v0, s2, v1
+; GFX1132-NEXT:    s_mov_b32 s2, -1
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1132-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1132-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = atomicrmw min ptr addrspace(3) @local_var32, i32 %lane acq_rel
   store i32 %old, ptr addrspace(1) %out

 ; GFX7LESS-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7LESS-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7LESS-NEXT:    s_mov_b32 s2, -1
 ; GFX7LESS-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX7LESS-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: umax_i32_varying:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    v_mov_b32_e32 v3, 0
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX8-NEXT:    v_mov_b32_e32 v1, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_max_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_max_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s4, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB21_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_max_u32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB21_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB21_2
-; GFX8-NEXT:  ; %bb.1:
-; GFX8-NEXT:    v_mov_b32_e32 v0, s4
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB21_4
+; GFX8-NEXT:  ; %bb.3:
+; GFX8-NEXT:    v_mov_b32_e32 v0, 0
+; GFX8-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX8-NEXT:    s_mov_b32 m0, -1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    ds_max_rtn_u32 v0, v3, v0
+; GFX8-NEXT:    ds_max_rtn_u32 v0, v0, v2
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:  .LBB21_2:
+; GFX8-NEXT:  .LBB21_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v2
 ; GFX8-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX8-NEXT:    s_mov_b32 s2, -1
-; GFX8-NEXT:    v_max_u32_e32 v0, s4, v0
+; GFX8-NEXT:    v_max_u32_e32 v0, s4, v1
 ; GFX8-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: umax_i32_varying:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_max_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_max_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s4, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB21_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_max_u32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB21_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB21_2
-; GFX9-NEXT:  ; %bb.1:
-; GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB21_4
+; GFX9-NEXT:  ; %bb.3:
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    ds_max_rtn_u32 v0, v3, v0
+; GFX9-NEXT:    ds_max_rtn_u32 v0, v0, v2
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:  .LBB21_2:
+; GFX9-NEXT:  .LBB21_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v2
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s2, -1
-; GFX9-NEXT:    v_max_u32_e32 v0, s4, v0
+; GFX9-NEXT:    v_max_u32_e32 v0, s4, v1
 ; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: umax_i32_varying:
 ; GFX1064:       ; %bb.0: ; %entry
-; GFX1064-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1064-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1064-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1064-NEXT:    v_max_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1064-NEXT:    v_max_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1064-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1064-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1064-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1064-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT:    s_mov_b32 s2, -1
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1064-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1064-NEXT:    s_mov_b32 s4, 0
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1064-NEXT:    ; implicit-def: $vgpr1
+; GFX1064-NEXT:  .LBB21_1: ; %ComputeLoop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX1064-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX1064-NEXT:    s_add_i32 s5, s5, 32
+; GFX1064-NEXT:    s_min_u32 s5, s6, s5
+; GFX1064-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1064-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1064-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1064-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX1064-NEXT:    s_max_u32 s4, s4, s8
+; GFX1064-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB21_1
+; GFX1064-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX1064-NEXT:    ; implicit-def: $vgpr0
-; GFX1064-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1064-NEXT:    s_cbranch_execz .LBB21_2
-; GFX1064-NEXT:  ; %bb.1:
-; GFX1064-NEXT:    v_mov_b32_e32 v0, s7
-; GFX1064-NEXT:    s_mov_b32 s3, s7
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1064-NEXT:    s_cbranch_execz .LBB21_4
+; GFX1064-NEXT:  ; %bb.3:
+; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT:    ds_max_rtn_u32 v0, v4, v0
+; GFX1064-NEXT:    ds_max_rtn_u32 v0, v0, v2
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_gl0_inv
-; GFX1064-NEXT:  .LBB21_2:
+; GFX1064-NEXT:  .LBB21_4:
 ; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1064-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1064-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1064-NEXT:    v_max_u32_e32 v0, s3, v0
+; GFX1064-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1064-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT:    v_max_u32_e32 v0, s2, v1
+; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1064-NEXT:    s_endpgm
 ;
 ; GFX1032-LABEL: umax_i32_varying:
 ; GFX1032:       ; %bb.0: ; %entry
-; GFX1032-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1032-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1032-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1032-NEXT:    v_max_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1032-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1032-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT:    s_mov_b32 s2, -1
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1032-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1032-NEXT:    s_mov_b32 s2, 0
+; GFX1032-NEXT:    ; implicit-def: $vgpr1
+; GFX1032-NEXT:  .LBB21_1: ; %ComputeLoop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX1032-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1032-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1032-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1032-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX1032-NEXT:    s_max_u32 s2, s2, s5
+; GFX1032-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB21_1
+; GFX1032-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
 ; GFX1032-NEXT:    ; implicit-def: $vgpr0
 ; GFX1032-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1032-NEXT:    s_cbranch_execz .LBB21_2
-; GFX1032-NEXT:  ; %bb.1:
-; GFX1032-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1032-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1032-NEXT:    s_cbranch_execz .LBB21_4
+; GFX1032-NEXT:  ; %bb.3:
+; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1032-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT:    ds_max_rtn_u32 v0, v4, v0
+; GFX1032-NEXT:    ds_max_rtn_u32 v0, v0, v2
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_gl0_inv
-; GFX1032-NEXT:  .LBB21_2:
+; GFX1032-NEXT:  .LBB21_4:
 ; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1032-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1032-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1032-NEXT:    v_max_u32_e32 v0, s3, v0
+; GFX1032-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1032-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT:    v_max_u32_e32 v0, s2, v1
+; GFX1032-NEXT:    s_mov_b32 s2, -1
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1164-LABEL: umax_i32_varying:
 ; GFX1164:       ; %bb.0: ; %entry
-; GFX1164-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1164-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1164-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_max_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1164-NEXT:    v_max_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1164-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1164-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1164-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1164-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1164-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1164-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1164-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1164-NEXT:    s_mov_b32 s2, -1
+; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1164-NEXT:    s_mov_b32 s4, 0
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1164-NEXT:    ; implicit-def: $vgpr1
+; GFX1164-NEXT:  .LBB21_1: ; %ComputeLoop
+; GFX1164-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1164-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX1164-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX1164-NEXT:    s_add_i32 s5, s5, 32
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT:    s_min_u32 s5, s6, s5
+; GFX1164-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1164-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1164-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1164-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX1164-NEXT:    s_max_u32 s4, s4, s8
+; GFX1164-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1164-NEXT:    s_cbranch_scc1 .LBB21_1
+; GFX1164-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
 ; GFX1164-NEXT:    ; implicit-def: $vgpr0
-; GFX1164-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1164-NEXT:    s_cbranch_execz .LBB21_2
-; GFX1164-NEXT:  ; %bb.1:
-; GFX1164-NEXT:    v_mov_b32_e32 v0, s7
-; GFX1164-NEXT:    s_mov_b32 s3, s7
+; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1164-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1164-NEXT:    s_cbranch_execz .LBB21_4
+; GFX1164-NEXT:  ; %bb.3:
+; GFX1164-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1164-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1164-NEXT:    ds_max_rtn_u32 v0, v4, v0
+; GFX1164-NEXT:    ds_max_rtn_u32 v0, v0, v2
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_gl0_inv
-; GFX1164-NEXT:  .LBB21_2:
-; GFX1164-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1164-NEXT:  .LBB21_4:
+; GFX1164-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1164-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1164-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1164-NEXT:    v_max_u32_e32 v0, s3, v0
+; GFX1164-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1164-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_max_u32_e32 v0, s2, v1
+; GFX1164-NEXT:    s_mov_b32 s2, -1
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1164-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1164-NEXT:    s_endpgm
 ;
 ; GFX1132-LABEL: umax_i32_varying:
 ; GFX1132:       ; %bb.0: ; %entry
-; GFX1132-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    v_mov_b32_e32 v1, 0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_mov_b32_e32 v3, 0
-; GFX1132-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    v_max_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1132-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_max_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1132-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1132-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1132-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1132-NEXT:    v_mov_b32_e32 v4, 0
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1132-NEXT:    s_mov_b32 s2, -1
+; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
+; GFX1132-NEXT:    s_mov_b32 s2, 0
+; GFX1132-NEXT:    ; implicit-def: $vgpr1
+; GFX1132-NEXT:  .LBB21_1: ; %ComputeLoop
+; GFX1132-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1132-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX1132-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1132-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1132-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1132-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX1132-NEXT:    s_max_u32 s2, s2, s5
+; GFX1132-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1132-NEXT:    s_cbranch_scc1 .LBB21_1
+; GFX1132-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
 ; GFX1132-NEXT:    ; implicit-def: $vgpr0
-; GFX1132-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1132-NEXT:    s_cbranch_execz .LBB21_2
-; GFX1132-NEXT:  ; %bb.1:
-; GFX1132-NEXT:    v_mov_b32_e32 v0, s4
+; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1132-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1132-NEXT:    s_cbranch_execz .LBB21_4
+; GFX1132-NEXT:  ; %bb.3:
+; GFX1132-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1132-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1132-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1132-NEXT:    ds_max_rtn_u32 v0, v4, v0
+; GFX1132-NEXT:    ds_max_rtn_u32 v0, v0, v2
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_gl0_inv
-; GFX1132-NEXT:  .LBB21_2:
+; GFX1132-NEXT:  .LBB21_4:
 ; GFX1132-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1132-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1132-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1132-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1132-NEXT:    v_max_u32_e32 v0, s3, v0
+; GFX1132-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1132-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT:    v_max_u32_e32 v0, s2, v1
+; GFX1132-NEXT:    s_mov_b32 s2, -1
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1132-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1132-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = atomicrmw umax ptr addrspace(3) @local_var32, i32 %lane acq_rel
   store i32 %old, ptr addrspace(1) %out

 ; GFX7LESS-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7LESS-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7LESS-NEXT:    s_mov_b32 s2, -1
 ; GFX7LESS-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX7LESS-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: umin_i32_varying:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_mov_b32_e32 v1, -1
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_mov_b32_e32 v2, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v2, -1
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_min_u32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_min_u32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_min_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_min_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_min_u32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_min_u32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v2, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    s_mov_b32 s4, -1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB23_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_min_u32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB23_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB23_2
-; GFX8-NEXT:  ; %bb.1:
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB23_4
+; GFX8-NEXT:  ; %bb.3:
 ; GFX8-NEXT:    v_mov_b32_e32 v0, 0
-; GFX8-NEXT:    v_mov_b32_e32 v3, s4
+; GFX8-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX8-NEXT:    s_mov_b32 m0, -1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    ds_min_rtn_u32 v0, v0, v3
+; GFX8-NEXT:    ds_min_rtn_u32 v0, v0, v2
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:  .LBB23_2:
+; GFX8-NEXT:  .LBB23_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v1
 ; GFX8-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX8-NEXT:    s_mov_b32 s2, -1
-; GFX8-NEXT:    v_min_u32_e32 v0, s4, v0
+; GFX8-NEXT:    v_min_u32_e32 v0, s4, v1
 ; GFX8-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: umin_i32_varying:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_mov_b32_e32 v1, -1
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_mov_b32_e32 v2, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v2, -1
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_min_u32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_min_u32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_min_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_min_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_min_u32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_min_u32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v2, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    s_mov_b32 s4, -1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB23_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_min_u32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB23_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB23_2
-; GFX9-NEXT:  ; %bb.1:
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB23_4
+; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
-; GFX9-NEXT:    v_mov_b32_e32 v3, s4
+; GFX9-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    ds_min_rtn_u32 v0, v0, v3
+; GFX9-NEXT:    ds_min_rtn_u32 v0, v0, v2
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:  .LBB23_2:
+; GFX9-NEXT:  .LBB23_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s4, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v1
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s2, -1
-; GFX9-NEXT:    v_min_u32_e32 v0, s4, v0
+; GFX9-NEXT:    v_min_u32_e32 v0, s4, v1
 ; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: umin_i32_varying:
 ; GFX1064:       ; %bb.0: ; %entry
-; GFX1064-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    v_mov_b32_e32 v1, -1
-; GFX1064-NEXT:    s_not_b64 exec, exec
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_e32 v3, -1
-; GFX1064-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1064-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1064-NEXT:    v_min_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1064-NEXT:    v_min_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1064-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1064-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1064-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1064-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1064-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1064-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT:    s_mov_b32 s2, -1
+; GFX1064-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1064-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1064-NEXT:    s_mov_b32 s4, -1
+; GFX1064-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1064-NEXT:    ; implicit-def: $vgpr1
+; GFX1064-NEXT:  .LBB23_1: ; %ComputeLoop
+; GFX1064-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1064-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX1064-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX1064-NEXT:    s_add_i32 s5, s5, 32
+; GFX1064-NEXT:    s_min_u32 s5, s6, s5
+; GFX1064-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1064-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1064-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1064-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX1064-NEXT:    s_min_u32 s4, s4, s8
+; GFX1064-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1064-NEXT:    s_cbranch_scc1 .LBB23_1
+; GFX1064-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX1064-NEXT:    ; implicit-def: $vgpr0
-; GFX1064-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1064-NEXT:    s_cbranch_execz .LBB23_2
-; GFX1064-NEXT:  ; %bb.1:
+; GFX1064-NEXT:    s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1064-NEXT:    s_cbranch_execz .LBB23_4
+; GFX1064-NEXT:  ; %bb.3:
 ; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1064-NEXT:    v_mov_b32_e32 v4, s7
-; GFX1064-NEXT:    s_mov_b32 s3, s7
+; GFX1064-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1064-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT:    ds_min_rtn_u32 v0, v0, v4
+; GFX1064-NEXT:    ds_min_rtn_u32 v0, v0, v2
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_gl0_inv
-; GFX1064-NEXT:  .LBB23_2:
+; GFX1064-NEXT:  .LBB23_4:
 ; GFX1064-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX1064-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1064-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1064-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1064-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1064-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1064-NEXT:    v_min_u32_e32 v0, s3, v0
+; GFX1064-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1064-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT:    v_min_u32_e32 v0, s2, v1
+; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1064-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1064-NEXT:    s_endpgm
 ;
 ; GFX1032-LABEL: umin_i32_varying:
 ; GFX1032:       ; %bb.0: ; %entry
-; GFX1032-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    v_mov_b32_e32 v1, -1
-; GFX1032-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_mov_b32_e32 v3, -1
-; GFX1032-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1032-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1032-NEXT:    v_min_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1032-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1032-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1032-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1032-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1032-NEXT:    s_mov_b32 s3, exec_lo
 ; GFX1032-NEXT:    s_mov_b32 s2, -1
+; GFX1032-NEXT:    ; implicit-def: $vgpr1
+; GFX1032-NEXT:  .LBB23_1: ; %ComputeLoop
+; GFX1032-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1032-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX1032-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1032-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1032-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1032-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX1032-NEXT:    s_min_u32 s2, s2, s5
+; GFX1032-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1032-NEXT:    s_cbranch_scc1 .LBB23_1
+; GFX1032-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
 ; GFX1032-NEXT:    ; implicit-def: $vgpr0
 ; GFX1032-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1032-NEXT:    s_cbranch_execz .LBB23_2
-; GFX1032-NEXT:  ; %bb.1:
+; GFX1032-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1032-NEXT:    s_cbranch_execz .LBB23_4
+; GFX1032-NEXT:  ; %bb.3:
 ; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1032-NEXT:    v_mov_b32_e32 v4, s4
+; GFX1032-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1032-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT:    ds_min_rtn_u32 v0, v0, v4
+; GFX1032-NEXT:    ds_min_rtn_u32 v0, v0, v2
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_gl0_inv
-; GFX1032-NEXT:  .LBB23_2:
+; GFX1032-NEXT:  .LBB23_4:
 ; GFX1032-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX1032-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1032-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; GFX1032-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1032-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1032-NEXT:    v_min_u32_e32 v0, s3, v0
+; GFX1032-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1032-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT:    v_min_u32_e32 v0, s2, v1
+; GFX1032-NEXT:    s_mov_b32 s2, -1
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1032-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1164-LABEL: umin_i32_varying:
 ; GFX1164:       ; %bb.0: ; %entry
-; GFX1164-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    v_mov_b32_e32 v1, -1
-; GFX1164-NEXT:    s_not_b64 exec, exec
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_mov_b32_e32 v3, -1
-; GFX1164-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1164-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_min_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
-; GFX1164-NEXT:    v_min_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1164-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX1164-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX1164-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX1164-NEXT:    v_readlane_b32 s7, v1, 63
-; GFX1164-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX1164-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX1164-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
-; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1164-NEXT:    s_or_saveexec_b64 s[4:5], -1
-; GFX1164-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX1164-NEXT:    s_mov_b64 exec, s[4:5]
-; GFX1164-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1164-NEXT:    s_mov_b32 s2, -1
+; GFX1164-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
+; GFX1164-NEXT:    s_mov_b32 s4, -1
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX1164-NEXT:    ; implicit-def: $vgpr1
+; GFX1164-NEXT:  .LBB23_1: ; %ComputeLoop
+; GFX1164-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1164-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX1164-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX1164-NEXT:    s_add_i32 s5, s5, 32
+; GFX1164-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT:    s_min_u32 s5, s6, s5
+; GFX1164-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX1164-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX1164-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX1164-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX1164-NEXT:    s_min_u32 s4, s4, s8
+; GFX1164-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX1164-NEXT:    s_cbranch_scc1 .LBB23_1
+; GFX1164-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1164-NEXT:    s_mov_b64 s[2:3], exec
 ; GFX1164-NEXT:    ; implicit-def: $vgpr0
-; GFX1164-NEXT:    s_and_saveexec_b64 s[4:5], vcc
-; GFX1164-NEXT:    s_cbranch_execz .LBB23_2
-; GFX1164-NEXT:  ; %bb.1:
+; GFX1164-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1164-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX1164-NEXT:    s_cbranch_execz .LBB23_4
+; GFX1164-NEXT:  ; %bb.3:
 ; GFX1164-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1164-NEXT:    v_mov_b32_e32 v4, s7
-; GFX1164-NEXT:    s_mov_b32 s3, s7
+; GFX1164-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX1164-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1164-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1164-NEXT:    ds_min_rtn_u32 v0, v0, v4
+; GFX1164-NEXT:    ds_min_rtn_u32 v0, v0, v2
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_gl0_inv
-; GFX1164-NEXT:  .LBB23_2:
-; GFX1164-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX1164-NEXT:  .LBB23_4:
+; GFX1164-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX1164-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1164-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1164-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1164-NEXT:    v_min_u32_e32 v0, s3, v0
+; GFX1164-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1164-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1164-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT:    v_min_u32_e32 v0, s2, v1
+; GFX1164-NEXT:    s_mov_b32 s2, -1
 ; GFX1164-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1164-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1164-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1164-NEXT:    s_endpgm
 ;
 ; GFX1132-LABEL: umin_i32_varying:
 ; GFX1132:       ; %bb.0: ; %entry
-; GFX1132-NEXT:    v_mov_b32_e32 v1, v0
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    v_mov_b32_e32 v1, -1
-; GFX1132-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    v_mov_b32_e32 v3, -1
-; GFX1132-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    v_min_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_mov_b32_e32 v2, v1
-; GFX1132-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1132-NEXT:    v_min_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1132-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX1132-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX1132-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
-; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1132-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX1132-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX1132-NEXT:    s_mov_b32 exec_lo, s2
-; GFX1132-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1132-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
 ; GFX1132-NEXT:    s_mov_b32 s2, -1
+; GFX1132-NEXT:    ; implicit-def: $vgpr1
+; GFX1132-NEXT:  .LBB23_1: ; %ComputeLoop
+; GFX1132-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1132-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX1132-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX1132-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX1132-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX1132-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX1132-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX1132-NEXT:    s_min_u32 s2, s2, s5
+; GFX1132-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX1132-NEXT:    s_cbranch_scc1 .LBB23_1
+; GFX1132-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX1132-NEXT:    s_mov_b32 s3, exec_lo
 ; GFX1132-NEXT:    ; implicit-def: $vgpr0
-; GFX1132-NEXT:    s_and_saveexec_b32 s3, vcc_lo
-; GFX1132-NEXT:    s_cbranch_execz .LBB23_2
-; GFX1132-NEXT:  ; %bb.1:
+; GFX1132-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX1132-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX1132-NEXT:    s_cbranch_execz .LBB23_4
+; GFX1132-NEXT:  ; %bb.3:
 ; GFX1132-NEXT:    v_mov_b32_e32 v0, 0
-; GFX1132-NEXT:    v_mov_b32_e32 v4, s4
+; GFX1132-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX1132-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX1132-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX1132-NEXT:    ds_min_rtn_u32 v0, v0, v4
+; GFX1132-NEXT:    ds_min_rtn_u32 v0, v0, v2
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_gl0_inv
-; GFX1132-NEXT:  .LBB23_2:
+; GFX1132-NEXT:  .LBB23_4:
 ; GFX1132-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX1132-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
-; GFX1132-NEXT:    v_readfirstlane_b32 s3, v0
-; GFX1132-NEXT:    v_mov_b32_e32 v0, v3
-; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX1132-NEXT:    v_min_u32_e32 v0, s3, v0
+; GFX1132-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX1132-NEXT:    s_mov_b32 s3, 0x31016000
+; GFX1132-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT:    v_min_u32_e32 v0, s2, v1
+; GFX1132-NEXT:    s_mov_b32 s2, -1
 ; GFX1132-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX1132-NEXT:    buffer_store_b32 v0, off, s[0:3], 0
 ; GFX1132-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX1132-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = atomicrmw umin ptr addrspace(3) @local_var32, i32 %lane acq_rel
   store i32 %old, ptr addrspace(1) %out

llvm/test/CodeGen/AMDGPU/atomic_optimizations_raw_buffer.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -march=amdgcn -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX6 %s
-; RUN: llc -march=amdgcn -mcpu=tonga -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX8 %s
-; RUN: llc -march=amdgcn -mcpu=gfx900 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX9 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX10W64 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX10W32 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX11W64 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX11W32 %s
+; RUN: llc -march=amdgcn -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX6 %s
+; RUN: llc -march=amdgcn -mcpu=tonga -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX8 %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX9 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX10W64 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX10W32 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX11W64 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX11W32 %s
 
 declare i32 @llvm.amdgcn.workitem.id.x()
 declare i32 @llvm.amdgcn.raw.buffer.atomic.add(i32, <4 x i32>, i32, i32, i32)
 declare i32 @llvm.amdgcn.raw.buffer.atomic.sub(i32, <4 x i32>, i32, i32, i32)
 
 ; Show what the atomic optimization pass will do for raw buffers.
 
 define amdgpu_kernel void @add_i32_constant(ptr addrspace(1) %out, <4 x i32> %inout) {

 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    s_waitcnt vmcnt(0)
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: add_i32_varying_vdata:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX8-NEXT:    v_mov_b32_e32 v2, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v2, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s4, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_add_i32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB2_2
-; GFX8-NEXT:  ; %bb.1:
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB2_4
+; GFX8-NEXT:  ; %bb.3:
 ; GFX8-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    buffer_atomic_add v0, off, s[8:11], 0 glc
-; GFX8-NEXT:  .LBB2_2:
+; GFX8-NEXT:  .LBB2_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v1
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s2, v0
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, s2, v1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    v_mov_b32_e32 v4, s1
-; GFX8-NEXT:    v_mov_b32_e32 v3, s0
-; GFX8-NEXT:    flat_store_dword v[3:4], v0
+; GFX8-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8-NEXT:    flat_store_dword v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: add_i32_varying_vdata:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s4, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_add_i32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB2_2
-; GFX9-NEXT:  ; %bb.1:
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB2_4
+; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    buffer_atomic_add v0, off, s[8:11], 0 glc
-; GFX9-NEXT:  .LBB2_2:
+; GFX9-NEXT:  .LBB2_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v2
-; GFX9-NEXT:    v_add_u32_e32 v0, s2, v0
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    v_add_u32_e32 v0, s2, v1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    global_store_dword v3, v0, s[0:1]
+; GFX9-NEXT:    global_store_dword v2, v0, s[0:1]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX10W64-LABEL: add_i32_varying_vdata:
 ; GFX10W64:       ; %bb.0: ; %entry
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX10W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX10W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX10W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX10W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX10W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX10W64-NEXT:    ; implicit-def: $vgpr4
+; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX10W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX10W64-NEXT:    s_mov_b32 s4, 0
+; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX10W64-NEXT:    ; implicit-def: $vgpr1
+; GFX10W64-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX10W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W64-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX10W64-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX10W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX10W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX10W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX10W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX10W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX10W64-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX10W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX10W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX10W64-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX10W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
+; GFX10W64-NEXT:    ; implicit-def: $vgpr0
 ; GFX10W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX10W64-NEXT:    s_cbranch_execz .LBB2_2
-; GFX10W64-NEXT:  ; %bb.1:
+; GFX10W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX10W64-NEXT:    s_cbranch_execz .LBB2_4
+; GFX10W64-NEXT:  ; %bb.3:
 ; GFX10W64-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    buffer_atomic_add v4, off, s[8:11], 0 glc
-; GFX10W64-NEXT:  .LBB2_2:
+; GFX10W64-NEXT:    buffer_atomic_add v0, off, s[8:11], 0 glc
+; GFX10W64-NEXT:  .LBB2_4:
 ; GFX10W64-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX10W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX10W64-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W64-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W64-NEXT:    v_add_nc_u32_e32 v1, s2, v1
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W64-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W64-NEXT:    s_endpgm
 ;
 ; GFX10W32-LABEL: add_i32_varying_vdata:
 ; GFX10W32:       ; %bb.0: ; %entry
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX10W32-NEXT:    ; implicit-def: $vgpr4
-; GFX10W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX10W32-NEXT:    s_cbranch_execz .LBB2_2
-; GFX10W32-NEXT:  ; %bb.1:
-; GFX10W32-NEXT:    s_mov_b32 s3, s4
+; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX10W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX10W32-NEXT:    s_mov_b32 s2, 0
+; GFX10W32-NEXT:    ; implicit-def: $vgpr1
+; GFX10W32-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX10W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W32-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX10W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX10W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX10W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX10W32-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX10W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX10W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX10W32-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX10W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
+; GFX10W32-NEXT:    ; implicit-def: $vgpr0
+; GFX10W32-NEXT:    s_and_saveexec_b32 s3, vcc_lo
+; GFX10W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX10W32-NEXT:    s_cbranch_execz .LBB2_4
+; GFX10W32-NEXT:  ; %bb.3:
 ; GFX10W32-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, s3
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    buffer_atomic_add v4, off, s[4:7], 0 glc
-; GFX10W32-NEXT:  .LBB2_2:
+; GFX10W32-NEXT:    buffer_atomic_add v0, off, s[4:7], 0 glc
+; GFX10W32-NEXT:  .LBB2_4:
 ; GFX10W32-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX10W32-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W32-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W32-NEXT:    v_add_nc_u32_e32 v1, s2, v1
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W32-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W32-NEXT:    s_endpgm
 ;
 ; GFX11W64-LABEL: add_i32_varying_vdata:
 ; GFX11W64:       ; %bb.0: ; %entry
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX11W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX11W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX11W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX11W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX11W64-NEXT:    ; implicit-def: $vgpr4
-; GFX11W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX11W64-NEXT:    s_cbranch_execz .LBB2_2
-; GFX11W64-NEXT:  ; %bb.1:
+; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    s_mov_b32 s4, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX11W64-NEXT:    ; implicit-def: $vgpr1
+; GFX11W64-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX11W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W64-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX11W64-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX11W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX11W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX11W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX11W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX11W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX11W64-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX11W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX11W64-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX11W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    ; implicit-def: $vgpr0
+; GFX11W64-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX11W64-NEXT:    s_cbranch_execz .LBB2_4
+; GFX11W64-NEXT:  ; %bb.3:
 ; GFX11W64-NEXT:    s_load_b128 s[8:11], s[0:1], 0x34
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    buffer_atomic_add_u32 v4, off, s[8:11], 0 glc
-; GFX11W64-NEXT:  .LBB2_2:
+; GFX11W64-NEXT:    buffer_atomic_add_u32 v0, off, s[8:11], 0 glc
+; GFX11W64-NEXT:  .LBB2_4:
 ; GFX11W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX11W64-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    v_add_nc_u32_e32 v1, s2, v1
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W64-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W64-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W64-NEXT:    s_endpgm
 ;
 ; GFX11W32-LABEL: add_i32_varying_vdata:
 ; GFX11W32:       ; %bb.0: ; %entry
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX11W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX11W32-NEXT:    ; implicit-def: $vgpr4
-; GFX11W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX11W32-NEXT:    s_cbranch_execz .LBB2_2
-; GFX11W32-NEXT:  ; %bb.1:
-; GFX11W32-NEXT:    s_mov_b32 s3, s4
+; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    s_mov_b32 s2, 0
+; GFX11W32-NEXT:    ; implicit-def: $vgpr1
+; GFX11W32-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX11W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W32-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX11W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX11W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX11W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX11W32-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX11W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX11W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX11W32-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX11W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    ; implicit-def: $vgpr0
+; GFX11W32-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX11W32-NEXT:    s_cbranch_execz .LBB2_4
+; GFX11W32-NEXT:  ; %bb.3:
 ; GFX11W32-NEXT:    s_load_b128 s[4:7], s[0:1], 0x34
-; GFX11W32-NEXT:    v_mov_b32_e32 v4, s3
+; GFX11W32-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    buffer_atomic_add_u32 v4, off, s[4:7], 0 glc
-; GFX11W32-NEXT:  .LBB2_2:
-; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX11W32-NEXT:    buffer_atomic_add_u32 v0, off, s[4:7], 0 glc
+; GFX11W32-NEXT:  .LBB2_4:
+; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX11W32-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W32-NEXT:    v_mov_b32_e32 v4, v3
+; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX11W32-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_add_nc_u32 v1, s2, v1
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W32-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W32-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W32-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = call i32 @llvm.amdgcn.raw.buffer.atomic.add(i32 %lane, <4 x i32> %inout, i32 0, i32 0, i32 0)
   store i32 %old, ptr addrspace(1) %out
   ret void
 }

 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    s_waitcnt vmcnt(0)
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: sub_i32_varying_vdata:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX8-NEXT:    v_mov_b32_e32 v2, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v2, vcc, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v2, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s4, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB6_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_add_i32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB6_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB6_2
-; GFX8-NEXT:  ; %bb.1:
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB6_4
+; GFX8-NEXT:  ; %bb.3:
 ; GFX8-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    buffer_atomic_sub v0, off, s[8:11], 0 glc
-; GFX8-NEXT:  .LBB6_2:
+; GFX8-NEXT:  .LBB6_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v1
-; GFX8-NEXT:    v_sub_u32_e32 v0, vcc, s2, v0
+; GFX8-NEXT:    v_sub_u32_e32 v2, vcc, s2, v1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    v_mov_b32_e32 v4, s1
-; GFX8-NEXT:    v_mov_b32_e32 v3, s0
-; GFX8-NEXT:    flat_store_dword v[3:4], v0
+; GFX8-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8-NEXT:    flat_store_dword v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: sub_i32_varying_vdata:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s4, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB6_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_add_i32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB6_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB6_2
-; GFX9-NEXT:  ; %bb.1:
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB6_4
+; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    buffer_atomic_sub v0, off, s[8:11], 0 glc
-; GFX9-NEXT:  .LBB6_2:
+; GFX9-NEXT:  .LBB6_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v2
-; GFX9-NEXT:    v_sub_u32_e32 v0, s2, v0
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    v_sub_u32_e32 v0, s2, v1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    global_store_dword v3, v0, s[0:1]
+; GFX9-NEXT:    global_store_dword v2, v0, s[0:1]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX10W64-LABEL: sub_i32_varying_vdata:
 ; GFX10W64:       ; %bb.0: ; %entry
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX10W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX10W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX10W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX10W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX10W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX10W64-NEXT:    ; implicit-def: $vgpr4
+; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX10W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX10W64-NEXT:    s_mov_b32 s4, 0
+; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX10W64-NEXT:    ; implicit-def: $vgpr1
+; GFX10W64-NEXT:  .LBB6_1: ; %ComputeLoop
+; GFX10W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W64-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX10W64-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX10W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX10W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX10W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX10W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX10W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX10W64-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX10W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX10W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX10W64-NEXT:    s_cbranch_scc1 .LBB6_1
+; GFX10W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
+; GFX10W64-NEXT:    ; implicit-def: $vgpr0
 ; GFX10W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX10W64-NEXT:    s_cbranch_execz .LBB6_2
-; GFX10W64-NEXT:  ; %bb.1:
+; GFX10W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX10W64-NEXT:    s_cbranch_execz .LBB6_4
+; GFX10W64-NEXT:  ; %bb.3:
 ; GFX10W64-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    buffer_atomic_sub v4, off, s[8:11], 0 glc
-; GFX10W64-NEXT:  .LBB6_2:
+; GFX10W64-NEXT:    buffer_atomic_sub v0, off, s[8:11], 0 glc
+; GFX10W64-NEXT:  .LBB6_4:
 ; GFX10W64-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX10W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX10W64-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W64-NEXT:    v_sub_nc_u32_e32 v4, s2, v4
+; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W64-NEXT:    v_sub_nc_u32_e32 v1, s2, v1
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W64-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W64-NEXT:    s_endpgm
 ;
 ; GFX10W32-LABEL: sub_i32_varying_vdata:
 ; GFX10W32:       ; %bb.0: ; %entry
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX10W32-NEXT:    ; implicit-def: $vgpr4
-; GFX10W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX10W32-NEXT:    s_cbranch_execz .LBB6_2
-; GFX10W32-NEXT:  ; %bb.1:
-; GFX10W32-NEXT:    s_mov_b32 s3, s4
+; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX10W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX10W32-NEXT:    s_mov_b32 s2, 0
+; GFX10W32-NEXT:    ; implicit-def: $vgpr1
+; GFX10W32-NEXT:  .LBB6_1: ; %ComputeLoop
+; GFX10W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W32-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX10W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX10W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX10W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX10W32-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX10W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX10W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX10W32-NEXT:    s_cbranch_scc1 .LBB6_1
+; GFX10W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
+; GFX10W32-NEXT:    ; implicit-def: $vgpr0
+; GFX10W32-NEXT:    s_and_saveexec_b32 s3, vcc_lo
+; GFX10W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX10W32-NEXT:    s_cbranch_execz .LBB6_4
+; GFX10W32-NEXT:  ; %bb.3:
 ; GFX10W32-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, s3
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    buffer_atomic_sub v4, off, s[4:7], 0 glc
-; GFX10W32-NEXT:  .LBB6_2:
+; GFX10W32-NEXT:    buffer_atomic_sub v0, off, s[4:7], 0 glc
+; GFX10W32-NEXT:  .LBB6_4:
 ; GFX10W32-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX10W32-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W32-NEXT:    v_sub_nc_u32_e32 v4, s2, v4
+; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W32-NEXT:    v_sub_nc_u32_e32 v1, s2, v1
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W32-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W32-NEXT:    s_endpgm
 ;
 ; GFX11W64-LABEL: sub_i32_varying_vdata:
 ; GFX11W64:       ; %bb.0: ; %entry
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX11W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX11W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX11W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX11W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX11W64-NEXT:    ; implicit-def: $vgpr4
-; GFX11W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX11W64-NEXT:    s_cbranch_execz .LBB6_2
-; GFX11W64-NEXT:  ; %bb.1:
+; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    s_mov_b32 s4, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX11W64-NEXT:    ; implicit-def: $vgpr1
+; GFX11W64-NEXT:  .LBB6_1: ; %ComputeLoop
+; GFX11W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W64-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX11W64-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX11W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX11W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX11W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX11W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX11W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX11W64-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX11W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX11W64-NEXT:    s_cbranch_scc1 .LBB6_1
+; GFX11W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    ; implicit-def: $vgpr0
+; GFX11W64-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX11W64-NEXT:    s_cbranch_execz .LBB6_4
+; GFX11W64-NEXT:  ; %bb.3:
 ; GFX11W64-NEXT:    s_load_b128 s[8:11], s[0:1], 0x34
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    buffer_atomic_sub_u32 v4, off, s[8:11], 0 glc
-; GFX11W64-NEXT:  .LBB6_2:
+; GFX11W64-NEXT:    buffer_atomic_sub_u32 v0, off, s[8:11], 0 glc
+; GFX11W64-NEXT:  .LBB6_4:
 ; GFX11W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX11W64-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W64-NEXT:    v_sub_nc_u32_e32 v4, s2, v4
+; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    v_sub_nc_u32_e32 v1, s2, v1
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W64-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W64-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W64-NEXT:    s_endpgm
 ;
 ; GFX11W32-LABEL: sub_i32_varying_vdata:
 ; GFX11W32:       ; %bb.0: ; %entry
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX11W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX11W32-NEXT:    ; implicit-def: $vgpr4
-; GFX11W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX11W32-NEXT:    s_cbranch_execz .LBB6_2
-; GFX11W32-NEXT:  ; %bb.1:
-; GFX11W32-NEXT:    s_mov_b32 s3, s4
+; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    s_mov_b32 s2, 0
+; GFX11W32-NEXT:    ; implicit-def: $vgpr1
+; GFX11W32-NEXT:  .LBB6_1: ; %ComputeLoop
+; GFX11W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W32-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX11W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX11W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX11W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX11W32-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX11W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX11W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX11W32-NEXT:    s_cbranch_scc1 .LBB6_1
+; GFX11W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    ; implicit-def: $vgpr0
+; GFX11W32-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX11W32-NEXT:    s_cbranch_execz .LBB6_4
+; GFX11W32-NEXT:  ; %bb.3:
 ; GFX11W32-NEXT:    s_load_b128 s[4:7], s[0:1], 0x34
-; GFX11W32-NEXT:    v_mov_b32_e32 v4, s3
+; GFX11W32-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    buffer_atomic_sub_u32 v4, off, s[4:7], 0 glc
-; GFX11W32-NEXT:  .LBB6_2:
-; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX11W32-NEXT:    buffer_atomic_sub_u32 v0, off, s[4:7], 0 glc
+; GFX11W32-NEXT:  .LBB6_4:
+; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX11W32-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W32-NEXT:    v_mov_b32_e32 v4, v3
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W32-NEXT:    v_sub_nc_u32_e32 v4, s2, v4
+; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX11W32-NEXT:    v_mov_b32_e32 v0, 0
+; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W32-NEXT:    v_sub_nc_u32_e32 v1, s2, v1
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W32-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W32-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W32-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = call i32 @llvm.amdgcn.raw.buffer.atomic.sub(i32 %lane, <4 x i32> %inout, i32 0, i32 0, i32 0)
   store i32 %old, ptr addrspace(1) %out
   ret void
 }

llvm/test/CodeGen/AMDGPU/atomic_optimizations_struct_buffer.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -march=amdgcn -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX6 %s
-; RUN: llc -march=amdgcn -mcpu=tonga -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX8 %s
-; RUN: llc -march=amdgcn -mcpu=gfx900 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX9 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX10W64 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX10W32 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX11W64 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-atomic-optimizations=true -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX11W32 %s
+; RUN: llc -march=amdgcn -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX6 %s
+; RUN: llc -march=amdgcn -mcpu=tonga -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX8 %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX9 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX10W64 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX10,GFX10W32 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX11W64 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-atomic-optimizations=true -amdgpu-atomic-optimizer-use-dpp=false -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GFX11,GFX11W32 %s
 
 declare i32 @llvm.amdgcn.workitem.id.x()
 declare i32 @llvm.amdgcn.struct.buffer.atomic.add(i32, <4 x i32>, i32, i32, i32, i32)
 declare i32 @llvm.amdgcn.struct.buffer.atomic.sub(i32, <4 x i32>, i32, i32, i32, i32)
 
 ; Show what the atomic optimization pass will do for struct buffers.
 
 define amdgpu_kernel void @add_i32_constant(ptr addrspace(1) %out, <4 x i32> %inout) {

 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    s_waitcnt vmcnt(0)
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: add_i32_varying_vdata:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    v_mov_b32_e32 v3, 0
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX8-NEXT:    v_mov_b32_e32 v1, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s4, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_add_i32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB2_2
-; GFX8-NEXT:  ; %bb.1:
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB2_4
+; GFX8-NEXT:  ; %bb.3:
 ; GFX8-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s4
+; GFX8-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    buffer_atomic_add v0, v3, s[8:11], 0 idxen glc
-; GFX8-NEXT:  .LBB2_2:
+; GFX8-NEXT:    buffer_atomic_add v0, v2, s[8:11], 0 idxen glc
+; GFX8-NEXT:  .LBB2_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v2
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s2, v0
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, s2, v1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    v_mov_b32_e32 v4, s1
-; GFX8-NEXT:    v_mov_b32_e32 v3, s0
-; GFX8-NEXT:    flat_store_dword v[3:4], v0
+; GFX8-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8-NEXT:    flat_store_dword v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: add_i32_varying_vdata:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s4, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_add_i32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB2_2
-; GFX9-NEXT:  ; %bb.1:
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB2_4
+; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    buffer_atomic_add v0, v3, s[8:11], 0 idxen glc
-; GFX9-NEXT:  .LBB2_2:
+; GFX9-NEXT:    buffer_atomic_add v0, v2, s[8:11], 0 idxen glc
+; GFX9-NEXT:  .LBB2_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v2
-; GFX9-NEXT:    v_add_u32_e32 v0, s2, v0
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    v_add_u32_e32 v0, s2, v1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    global_store_dword v3, v0, s[0:1]
+; GFX9-NEXT:    global_store_dword v2, v0, s[0:1]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX10W64-LABEL: add_i32_varying_vdata:
 ; GFX10W64:       ; %bb.0: ; %entry
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX10W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX10W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX10W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX10W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX10W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX10W64-NEXT:    ; implicit-def: $vgpr4
+; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX10W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX10W64-NEXT:    s_mov_b32 s4, 0
+; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX10W64-NEXT:    ; implicit-def: $vgpr1
+; GFX10W64-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX10W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W64-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX10W64-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX10W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX10W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX10W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX10W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX10W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX10W64-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX10W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX10W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX10W64-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX10W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
+; GFX10W64-NEXT:    ; implicit-def: $vgpr0
 ; GFX10W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX10W64-NEXT:    s_cbranch_execz .LBB2_2
-; GFX10W64-NEXT:  ; %bb.1:
+; GFX10W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX10W64-NEXT:    s_cbranch_execz .LBB2_4
+; GFX10W64-NEXT:  ; %bb.3:
 ; GFX10W64-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, s4
+; GFX10W64-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    buffer_atomic_add v4, v0, s[8:11], 0 idxen glc
-; GFX10W64-NEXT:  .LBB2_2:
+; GFX10W64-NEXT:    buffer_atomic_add v0, v2, s[8:11], 0 idxen glc
+; GFX10W64-NEXT:  .LBB2_4:
 ; GFX10W64-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX10W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX10W64-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W64-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W64-NEXT:    v_add_nc_u32_e32 v1, s2, v1
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W64-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W64-NEXT:    s_endpgm
 ;
 ; GFX10W32-LABEL: add_i32_varying_vdata:
 ; GFX10W32:       ; %bb.0: ; %entry
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX10W32-NEXT:    ; implicit-def: $vgpr4
-; GFX10W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX10W32-NEXT:    s_cbranch_execz .LBB2_2
-; GFX10W32-NEXT:  ; %bb.1:
-; GFX10W32-NEXT:    s_mov_b32 s3, s4
+; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX10W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX10W32-NEXT:    s_mov_b32 s2, 0
+; GFX10W32-NEXT:    ; implicit-def: $vgpr1
+; GFX10W32-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX10W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W32-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX10W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX10W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX10W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX10W32-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX10W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX10W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX10W32-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX10W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
+; GFX10W32-NEXT:    ; implicit-def: $vgpr0
+; GFX10W32-NEXT:    s_and_saveexec_b32 s3, vcc_lo
+; GFX10W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX10W32-NEXT:    s_cbranch_execz .LBB2_4
+; GFX10W32-NEXT:  ; %bb.3:
 ; GFX10W32-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, s3
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, s2
+; GFX10W32-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    buffer_atomic_add v4, v0, s[4:7], 0 idxen glc
-; GFX10W32-NEXT:  .LBB2_2:
+; GFX10W32-NEXT:    buffer_atomic_add v0, v2, s[4:7], 0 idxen glc
+; GFX10W32-NEXT:  .LBB2_4:
 ; GFX10W32-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX10W32-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W32-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W32-NEXT:    v_add_nc_u32_e32 v1, s2, v1
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W32-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W32-NEXT:    s_endpgm
 ;
 ; GFX11W64-LABEL: add_i32_varying_vdata:
 ; GFX11W64:       ; %bb.0: ; %entry
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX11W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX11W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX11W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX11W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX11W64-NEXT:    ; implicit-def: $vgpr4
-; GFX11W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX11W64-NEXT:    s_cbranch_execz .LBB2_2
-; GFX11W64-NEXT:  ; %bb.1:
+; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    s_mov_b32 s4, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX11W64-NEXT:    ; implicit-def: $vgpr1
+; GFX11W64-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX11W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W64-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX11W64-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX11W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX11W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX11W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX11W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX11W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX11W64-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX11W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX11W64-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX11W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    ; implicit-def: $vgpr0
+; GFX11W64-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX11W64-NEXT:    s_cbranch_execz .LBB2_4
+; GFX11W64-NEXT:  ; %bb.3:
 ; GFX11W64-NEXT:    s_load_b128 s[8:11], s[0:1], 0x34
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, s4
+; GFX11W64-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    buffer_atomic_add_u32 v4, v0, s[8:11], 0 idxen glc
-; GFX11W64-NEXT:  .LBB2_2:
+; GFX11W64-NEXT:    buffer_atomic_add_u32 v0, v2, s[8:11], 0 idxen glc
+; GFX11W64-NEXT:  .LBB2_4:
 ; GFX11W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX11W64-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    v_add_nc_u32_e32 v1, s2, v1
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W64-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W64-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W64-NEXT:    s_endpgm
 ;
 ; GFX11W32-LABEL: add_i32_varying_vdata:
 ; GFX11W32:       ; %bb.0: ; %entry
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX11W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX11W32-NEXT:    ; implicit-def: $vgpr4
-; GFX11W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX11W32-NEXT:    s_cbranch_execz .LBB2_2
-; GFX11W32-NEXT:  ; %bb.1:
-; GFX11W32-NEXT:    s_mov_b32 s3, s4
+; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    s_mov_b32 s2, 0
+; GFX11W32-NEXT:    ; implicit-def: $vgpr1
+; GFX11W32-NEXT:  .LBB2_1: ; %ComputeLoop
+; GFX11W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W32-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX11W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX11W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX11W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX11W32-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX11W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX11W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX11W32-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX11W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    ; implicit-def: $vgpr0
+; GFX11W32-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX11W32-NEXT:    s_cbranch_execz .LBB2_4
+; GFX11W32-NEXT:  ; %bb.3:
 ; GFX11W32-NEXT:    s_load_b128 s[4:7], s[0:1], 0x34
-; GFX11W32-NEXT:    v_mov_b32_e32 v4, s3
+; GFX11W32-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11W32-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    buffer_atomic_add_u32 v4, v0, s[4:7], 0 idxen glc
-; GFX11W32-NEXT:  .LBB2_2:
-; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX11W32-NEXT:    buffer_atomic_add_u32 v0, v2, s[4:7], 0 idxen glc
+; GFX11W32-NEXT:  .LBB2_4:
+; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX11W32-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W32-NEXT:    v_mov_b32_e32 v4, v3
+; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_e32 v4, s2, v4
+; GFX11W32-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_add_nc_u32 v1, s2, v1
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W32-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W32-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W32-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = call i32 @llvm.amdgcn.struct.buffer.atomic.add(i32 %lane, <4 x i32> %inout, i32 0, i32 0, i32 0, i32 0)
   store i32 %old, ptr addrspace(1) %out
   ret void
 }

 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    s_waitcnt vmcnt(0)
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: sub_i32_varying_vdata:
 ; GFX8:       ; %bb.0: ; %entry
-; GFX8-NEXT:    v_mov_b32_e32 v3, 0
-; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX8-NEXT:    v_mov_b32_e32 v1, v0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0
-; GFX8-NEXT:    s_not_b64 exec, exec
-; GFX8-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    v_mov_b32_e32 v2, 0
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX8-NEXT:    s_nop 1
-; GFX8-NEXT:    v_add_u32_dpp v1, vcc, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX8-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX8-NEXT:    s_nop 0
-; GFX8-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX8-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX8-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX8-NEXT:    s_mov_b64 s[2:3], exec
+; GFX8-NEXT:    s_mov_b32 s4, 0
+; GFX8-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX8-NEXT:    ; implicit-def: $vgpr1
+; GFX8-NEXT:  .LBB7_1: ; %ComputeLoop
+; GFX8-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX8-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX8-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX8-NEXT:    s_add_i32 s5, s5, 32
+; GFX8-NEXT:    s_min_u32 s5, s6, s5
+; GFX8-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX8-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX8-NEXT:    s_mov_b32 m0, s5
+; GFX8-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX8-NEXT:    s_add_i32 s4, s4, s8
+; GFX8-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX8-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX8-NEXT:    s_cbranch_scc1 .LBB7_1
+; GFX8-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX8-NEXT:    ; implicit-def: $vgpr0
 ; GFX8-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX8-NEXT:    s_cbranch_execz .LBB7_2
-; GFX8-NEXT:  ; %bb.1:
+; GFX8-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX8-NEXT:    s_cbranch_execz .LBB7_4
+; GFX8-NEXT:  ; %bb.3:
 ; GFX8-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s4
+; GFX8-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    buffer_atomic_sub v0, v3, s[8:11], 0 idxen glc
-; GFX8-NEXT:  .LBB7_2:
+; GFX8-NEXT:    buffer_atomic_sub v0, v2, s[8:11], 0 idxen glc
+; GFX8-NEXT:  .LBB7_4:
 ; GFX8-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX8-NEXT:    v_mov_b32_e32 v0, v2
-; GFX8-NEXT:    v_sub_u32_e32 v0, vcc, s2, v0
+; GFX8-NEXT:    v_sub_u32_e32 v2, vcc, s2, v1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    v_mov_b32_e32 v4, s1
-; GFX8-NEXT:    v_mov_b32_e32 v3, s0
-; GFX8-NEXT:    flat_store_dword v[3:4], v0
+; GFX8-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8-NEXT:    flat_store_dword v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: sub_i32_varying_vdata:
 ; GFX9:       ; %bb.0: ; %entry
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX9-NEXT:    v_mov_b32_e32 v1, v0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
-; GFX9-NEXT:    s_not_b64 exec, exec
-; GFX9-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX9-NEXT:    s_nop 1
-; GFX9-NEXT:    v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX9-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX9-NEXT:    s_nop 0
-; GFX9-NEXT:    v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX9-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX9-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX9-NEXT:    s_mov_b64 s[2:3], exec
+; GFX9-NEXT:    s_mov_b32 s4, 0
+; GFX9-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX9-NEXT:    ; implicit-def: $vgpr1
+; GFX9-NEXT:  .LBB7_1: ; %ComputeLoop
+; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX9-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX9-NEXT:    s_add_i32 s5, s5, 32
+; GFX9-NEXT:    s_min_u32 s5, s6, s5
+; GFX9-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX9-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX9-NEXT:    s_mov_b32 m0, s5
+; GFX9-NEXT:    v_writelane_b32 v1, s4, m0
+; GFX9-NEXT:    s_add_i32 s4, s4, s8
+; GFX9-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_cbranch_scc1 .LBB7_1
+; GFX9-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX9-NEXT:    s_cbranch_execz .LBB7_2
-; GFX9-NEXT:  ; %bb.1:
+; GFX9-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX9-NEXT:    s_cbranch_execz .LBB7_4
+; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    buffer_atomic_sub v0, v3, s[8:11], 0 idxen glc
-; GFX9-NEXT:  .LBB7_2:
+; GFX9-NEXT:    buffer_atomic_sub v0, v2, s[8:11], 0 idxen glc
+; GFX9-NEXT:  .LBB7_4:
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX9-NEXT:    v_mov_b32_e32 v0, v2
-; GFX9-NEXT:    v_sub_u32_e32 v0, s2, v0
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    v_sub_u32_e32 v0, s2, v1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    global_store_dword v3, v0, s[0:1]
+; GFX9-NEXT:    global_store_dword v2, v0, s[0:1]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX10W64-LABEL: sub_i32_varying_vdata:
 ; GFX10W64:       ; %bb.0: ; %entry
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W64-NEXT:    s_not_b64 exec, exec
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX10W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX10W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX10W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX10W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX10W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX10W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX10W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX10W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX10W64-NEXT:    ; implicit-def: $vgpr4
+; GFX10W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX10W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX10W64-NEXT:    s_mov_b32 s4, 0
+; GFX10W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX10W64-NEXT:    ; implicit-def: $vgpr1
+; GFX10W64-NEXT:  .LBB7_1: ; %ComputeLoop
+; GFX10W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W64-NEXT:    s_ff1_i32_b32 s5, s3
+; GFX10W64-NEXT:    s_ff1_i32_b32 s6, s2
+; GFX10W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX10W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX10W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX10W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX10W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX10W64-NEXT:    s_andn2_b64 s[2:3], s[2:3], s[6:7]
+; GFX10W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX10W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX10W64-NEXT:    s_cbranch_scc1 .LBB7_1
+; GFX10W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v2
+; GFX10W64-NEXT:    ; implicit-def: $vgpr0
 ; GFX10W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX10W64-NEXT:    s_cbranch_execz .LBB7_2
-; GFX10W64-NEXT:  ; %bb.1:
+; GFX10W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX10W64-NEXT:    s_cbranch_execz .LBB7_4
+; GFX10W64-NEXT:  ; %bb.3:
 ; GFX10W64-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, s4
+; GFX10W64-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    buffer_atomic_sub v4, v0, s[8:11], 0 idxen glc
-; GFX10W64-NEXT:  .LBB7_2:
+; GFX10W64-NEXT:    buffer_atomic_sub v0, v2, s[8:11], 0 idxen glc
+; GFX10W64-NEXT:  .LBB7_4:
 ; GFX10W64-NEXT:    s_waitcnt_depctr 0xffe3
 ; GFX10W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX10W64-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W64-NEXT:    v_sub_nc_u32_e32 v4, s2, v4
+; GFX10W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W64-NEXT:    v_sub_nc_u32_e32 v1, s2, v1
 ; GFX10W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W64-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W64-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W64-NEXT:    s_endpgm
 ;
 ; GFX10W32-LABEL: sub_i32_varying_vdata:
 ; GFX10W32:       ; %bb.0: ; %entry
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX10W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX10W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX10W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX10W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX10W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX10W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX10W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX10W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX10W32-NEXT:    ; implicit-def: $vgpr4
-; GFX10W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX10W32-NEXT:    s_cbranch_execz .LBB7_2
-; GFX10W32-NEXT:  ; %bb.1:
-; GFX10W32-NEXT:    s_mov_b32 s3, s4
+; GFX10W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX10W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX10W32-NEXT:    s_mov_b32 s2, 0
+; GFX10W32-NEXT:    ; implicit-def: $vgpr1
+; GFX10W32-NEXT:  .LBB7_1: ; %ComputeLoop
+; GFX10W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX10W32-NEXT:    s_ff1_i32_b32 s4, s3
+; GFX10W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX10W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX10W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX10W32-NEXT:    s_andn2_b32 s3, s3, s6
+; GFX10W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX10W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX10W32-NEXT:    s_cbranch_scc1 .LBB7_1
+; GFX10W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX10W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v2
+; GFX10W32-NEXT:    ; implicit-def: $vgpr0
+; GFX10W32-NEXT:    s_and_saveexec_b32 s3, vcc_lo
+; GFX10W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX10W32-NEXT:    s_cbranch_execz .LBB7_4
+; GFX10W32-NEXT:  ; %bb.3:
 ; GFX10W32-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, s3
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, s2
+; GFX10W32-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    buffer_atomic_sub v4, v0, s[4:7], 0 idxen glc
-; GFX10W32-NEXT:  .LBB7_2:
+; GFX10W32-NEXT:    buffer_atomic_sub v0, v2, s[4:7], 0 idxen glc
+; GFX10W32-NEXT:  .LBB7_4:
 ; GFX10W32-NEXT:    s_waitcnt_depctr 0xffe3
-; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX10W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX10W32-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX10W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX10W32-NEXT:    v_mov_b32_e32 v4, v3
-; GFX10W32-NEXT:    v_sub_nc_u32_e32 v4, s2, v4
+; GFX10W32-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX10W32-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10W32-NEXT:    v_sub_nc_u32_e32 v1, s2, v1
 ; GFX10W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10W32-NEXT:    global_store_dword v0, v4, s[0:1]
+; GFX10W32-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GFX10W32-NEXT:    s_endpgm
 ;
 ; GFX11W64-LABEL: sub_i32_varying_vdata:
 ; GFX11W64:       ; %bb.0: ; %entry
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W64-NEXT:    s_not_b64 exec, exec
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W64-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W64-NEXT:    v_mov_b32_e32 v2, s4
-; GFX11W64-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
-; GFX11W64-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 15
-; GFX11W64-NEXT:    v_readlane_b32 s5, v1, 31
-; GFX11W64-NEXT:    v_writelane_b32 v3, s4, 16
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_readlane_b32 s4, v1, 63
-; GFX11W64-NEXT:    v_readlane_b32 s6, v1, 47
-; GFX11W64-NEXT:    v_writelane_b32 v3, s5, 32
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W64-NEXT:    s_or_saveexec_b64 s[2:3], -1
-; GFX11W64-NEXT:    v_writelane_b32 v3, s6, 48
-; GFX11W64-NEXT:    s_mov_b64 exec, s[2:3]
-; GFX11W64-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX11W64-NEXT:    ; implicit-def: $vgpr4
-; GFX11W64-NEXT:    s_and_saveexec_b64 s[2:3], vcc
-; GFX11W64-NEXT:    s_cbranch_execz .LBB7_2
-; GFX11W64-NEXT:  ; %bb.1:
+; GFX11W64-NEXT:    v_mbcnt_lo_u32_b32 v1, exec_lo, 0
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    s_mov_b32 s4, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11W64-NEXT:    v_mbcnt_hi_u32_b32 v2, exec_hi, v1
+; GFX11W64-NEXT:    ; implicit-def: $vgpr1
+; GFX11W64-NEXT:  .LBB7_1: ; %ComputeLoop
+; GFX11W64-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W64-NEXT:    s_ctz_i32_b32 s5, s3
+; GFX11W64-NEXT:    s_ctz_i32_b32 s6, s2
+; GFX11W64-NEXT:    s_add_i32 s5, s5, 32
+; GFX11W64-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX11W64-NEXT:    s_min_u32 s5, s6, s5
+; GFX11W64-NEXT:    v_readlane_b32 s8, v0, s5
+; GFX11W64-NEXT:    s_lshl_b64 s[6:7], 1, s5
+; GFX11W64-NEXT:    v_writelane_b32 v1, s4, s5
+; GFX11W64-NEXT:    s_and_not1_b64 s[2:3], s[2:3], s[6:7]
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    s_add_i32 s4, s4, s8
+; GFX11W64-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX11W64-NEXT:    s_cbranch_scc1 .LBB7_1
+; GFX11W64-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W64-NEXT:    s_mov_b64 s[2:3], exec
+; GFX11W64-NEXT:    ; implicit-def: $vgpr0
+; GFX11W64-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W64-NEXT:    s_xor_b64 s[2:3], exec, s[2:3]
+; GFX11W64-NEXT:    s_cbranch_execz .LBB7_4
+; GFX11W64-NEXT:  ; %bb.3:
 ; GFX11W64-NEXT:    s_load_b128 s[8:11], s[0:1], 0x34
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, s4
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, s4
+; GFX11W64-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    buffer_atomic_sub_u32 v4, v0, s[8:11], 0 idxen glc
-; GFX11W64-NEXT:  .LBB7_2:
+; GFX11W64-NEXT:    buffer_atomic_sub_u32 v0, v2, s[8:11], 0 idxen glc
+; GFX11W64-NEXT:  .LBB7_4:
 ; GFX11W64-NEXT:    s_or_b64 exec, exec, s[2:3]
 ; GFX11W64-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W64-NEXT:    v_mov_b32_e32 v4, v3
-; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W64-NEXT:    v_sub_nc_u32_e32 v4, s2, v4
+; GFX11W64-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX11W64-NEXT:    v_mov_b32_e32 v0, 0
+; GFX11W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W64-NEXT:    v_sub_nc_u32_e32 v1, s2, v1
 ; GFX11W64-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W64-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W64-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W64-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W64-NEXT:    s_endpgm
 ;
 ; GFX11W32-LABEL: sub_i32_varying_vdata:
 ; GFX11W32:       ; %bb.0: ; %entry
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, v0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    v_mov_b32_e32 v1, 0
-; GFX11W32-NEXT:    s_not_b32 exec_lo, exec_lo
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_mov_b32_e32 v3, 0
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_mov_b32_e32 v2, v1
-; GFX11W32-NEXT:    v_permlanex16_b32 v2, v2, -1, -1
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11W32-NEXT:    v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s4, v1, 31
-; GFX11W32-NEXT:    v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX11W32-NEXT:    v_readlane_b32 s3, v1, 15
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX11W32-NEXT:    v_mov_b32_e32 v0, 0
-; GFX11W32-NEXT:    s_or_saveexec_b32 s2, -1
-; GFX11W32-NEXT:    v_writelane_b32 v3, s3, 16
-; GFX11W32-NEXT:    s_mov_b32 exec_lo, s2
-; GFX11W32-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX11W32-NEXT:    ; implicit-def: $vgpr4
-; GFX11W32-NEXT:    s_and_saveexec_b32 s2, vcc_lo
-; GFX11W32-NEXT:    s_cbranch_execz .LBB7_2
-; GFX11W32-NEXT:  ; %bb.1:
-; GFX11W32-NEXT:    s_mov_b32 s3, s4
+; GFX11W32-NEXT:    v_mbcnt_lo_u32_b32 v2, exec_lo, 0
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    s_mov_b32 s2, 0
+; GFX11W32-NEXT:    ; implicit-def: $vgpr1
+; GFX11W32-NEXT:  .LBB7_1: ; %ComputeLoop
+; GFX11W32-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX11W32-NEXT:    s_ctz_i32_b32 s4, s3
+; GFX11W32-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
+; GFX11W32-NEXT:    v_readlane_b32 s5, v0, s4
+; GFX11W32-NEXT:    s_lshl_b32 s6, 1, s4
+; GFX11W32-NEXT:    v_writelane_b32 v1, s2, s4
+; GFX11W32-NEXT:    s_and_not1_b32 s3, s3, s6
+; GFX11W32-NEXT:    s_add_i32 s2, s2, s5
+; GFX11W32-NEXT:    s_cmp_lg_u32 s3, 0
+; GFX11W32-NEXT:    s_cbranch_scc1 .LBB7_1
+; GFX11W32-NEXT:  ; %bb.2: ; %ComputeEnd
+; GFX11W32-NEXT:    s_mov_b32 s3, exec_lo
+; GFX11W32-NEXT:    ; implicit-def: $vgpr0
+; GFX11W32-NEXT:    v_cmpx_eq_u32_e32 0, v2
+; GFX11W32-NEXT:    s_xor_b32 s3, exec_lo, s3
+; GFX11W32-NEXT:    s_cbranch_execz .LBB7_4
+; GFX11W32-NEXT:  ; %bb.3:
 ; GFX11W32-NEXT:    s_load_b128 s[4:7], s[0:1], 0x34
-; GFX11W32-NEXT:    v_mov_b32_e32 v4, s3
+; GFX11W32-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11W32-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    buffer_atomic_sub_u32 v4, v0, s[4:7], 0 idxen glc
-; GFX11W32-NEXT:  .LBB7_2:
-; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s2
+; GFX11W32-NEXT:    buffer_atomic_sub_u32 v0, v2, s[4:7], 0 idxen glc
+; GFX11W32-NEXT:  .LBB7_4:
+; GFX11W32-NEXT:    s_or_b32 exec_lo, exec_lo, s3
 ; GFX11W32-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
 ; GFX11W32-NEXT:    s_waitcnt vmcnt(0)
-; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v4
-; GFX11W32-NEXT:    v_mov_b32_e32 v4, v3
-; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11W32-NEXT:    v_sub_nc_u32_e32 v4, s2, v4
+; GFX11W32-NEXT:    v_readfirstlane_b32 s2, v0
+; GFX11W32-NEXT:    v_mov_b32_e32 v0, 0
+; GFX11W32-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11W32-NEXT:    v_sub_nc_u32_e32 v1, s2, v1
 ; GFX11W32-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11W32-NEXT:    global_store_b32 v0, v4, s[0:1]
+; GFX11W32-NEXT:    global_store_b32 v0, v1, s[0:1]
 ; GFX11W32-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; GFX11W32-NEXT:    s_endpgm
 entry:
   %lane = call i32 @llvm.amdgcn.workitem.id.x()
   %old = call i32 @llvm.amdgcn.struct.buffer.atomic.sub(i32 %lane, <4 x i32> %inout, i32 0, i32 0, i32 0, i32 0)
   store i32 %old, ptr addrspace(1) %out
   ret void
 }

llvm/test/CodeGen/AMDGPU/global_atomics_iterative_scan.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN:  opt -S -mtriple=amdgcn-- -amdgpu-atomic-optimizer-use-dpp=false -passes=amdgpu-atomic-optimizer %s | FileCheck -check-prefix=IR %s
+
+define amdgpu_kernel void @uniform_value(ptr addrspace(1) %out, ptr addrspace(1) %val) #0 {
+; IR-LABEL: @uniform_value(
+; IR-NEXT:  entry:
+; IR-NEXT:    [[BAZ_KERNARG_SEGMENT:%.*]] = call nonnull align 16 dereferenceable(52) ptr addrspace(4) @llvm.amdgcn.kernarg.segment.ptr()
+; IR-NEXT:    [[OUT_KERNARG_OFFSET:%.*]] = getelementptr inbounds i8, ptr addrspace(4) [[BAZ_KERNARG_SEGMENT]], i64 36
+; IR-NEXT:    [[TMP0:%.*]] = load <2 x i64>, ptr addrspace(4) [[OUT_KERNARG_OFFSET]], align 4
+; IR-NEXT:    [[OUT_LOAD1:%.*]] = extractelement <2 x i64> [[TMP0]], i32 0
+; IR-NEXT:    [[TMP1:%.*]] = inttoptr i64 [[OUT_LOAD1]] to ptr addrspace(1)
+; IR-NEXT:    [[VAL_LOAD2:%.*]] = extractelement <2 x i64> [[TMP0]], i32 1
+; IR-NEXT:    [[TMP2:%.*]] = inttoptr i64 [[VAL_LOAD2]] to ptr addrspace(1)
+; IR-NEXT:    [[LANE:%.*]] = tail call i32 @llvm.amdgcn.workgroup.id.x()
+; IR-NEXT:    [[IDXPROM:%.*]] = sext i32 [[LANE]] to i64
+; IR-NEXT:    [[ELE:%.*]] = getelementptr i32, ptr addrspace(1) [[TMP2]], i64 [[IDXPROM]]
+; IR-NEXT:    [[VALUE:%.*]] = load i32, ptr addrspace(1) [[ELE]], align 4
+; IR-NEXT:    [[GEP:%.*]] = getelementptr i32, ptr addrspace(1) [[TMP1]], i32 4
+; IR-NEXT:    [[TMP3:%.*]] = call i64 @llvm.amdgcn.ballot.i64(i1 true)
+; IR-NEXT:    [[TMP4:%.*]] = bitcast i64 [[TMP3]] to <2 x i32>
+; IR-NEXT:    [[TMP5:%.*]] = extractelement <2 x i32> [[TMP4]], i32 0
+; IR-NEXT:    [[TMP6:%.*]] = extractelement <2 x i32> [[TMP4]], i32 1
+; IR-NEXT:    [[TMP7:%.*]] = call i32 @llvm.amdgcn.mbcnt.lo(i32 [[TMP5]], i32 0)
+; IR-NEXT:    [[TMP8:%.*]] = call i32 @llvm.amdgcn.mbcnt.hi(i32 [[TMP6]], i32 [[TMP7]])
+; IR-NEXT:    [[TMP9:%.*]] = call i64 @llvm.ctpop.i64(i64 [[TMP3]])
+; IR-NEXT:    [[TMP10:%.*]] = trunc i64 [[TMP9]] to i32
+; IR-NEXT:    [[TMP11:%.*]] = mul i32 [[VALUE]], [[TMP10]]
+; IR-NEXT:    [[TMP12:%.*]] = icmp eq i32 [[TMP8]], 0
+; IR-NEXT:    br i1 [[TMP12]], label [[TMP13:%.*]], label [[TMP15:%.*]]
+; IR:       13:
+; IR-NEXT:    [[TMP14:%.*]] = atomicrmw volatile add ptr addrspace(1) [[GEP]], i32 [[TMP11]] seq_cst, align 4
+; IR-NEXT:    br label [[TMP15]]
+; IR:       15:
+; IR-NEXT:    ret void
+;
+entry:
+  %baz.kernarg.segment = call nonnull align 16 dereferenceable(52) ptr addrspace(4) @llvm.amdgcn.kernarg.segment.ptr()
+  %out.kernarg.offset = getelementptr inbounds i8, ptr addrspace(4) %baz.kernarg.segment, i64 36
+  %0 = load <2 x i64>, ptr addrspace(4) %out.kernarg.offset, align 4
+  %out.load1 = extractelement <2 x i64> %0, i32 0
+  %1 = inttoptr i64 %out.load1 to ptr addrspace(1)
+  %val.load2 = extractelement <2 x i64> %0, i32 1
+  %2 = inttoptr i64 %val.load2 to ptr addrspace(1)
+  %lane = tail call i32 @llvm.amdgcn.workgroup.id.x()
+  %idxprom = sext i32 %lane to i64
+  %ele = getelementptr i32, ptr addrspace(1) %2, i64 %idxprom
+  %value = load i32, ptr addrspace(1) %ele, align 4
+  %gep = getelementptr i32, ptr addrspace(1) %1, i32 4
+  %old = atomicrmw volatile add ptr addrspace(1) %gep, i32 %value seq_cst, align 4
+  ret void
+}
+
+define amdgpu_kernel void @divergent_value(ptr addrspace(1) %out, ptr addrspace(1) %val) #0 {
+; IR-LABEL: @divergent_value(
+; IR-NEXT:  entry:
+; IR-NEXT:    [[BAR_KERNARG_SEGMENT:%.*]] = call nonnull align 16 dereferenceable(52) ptr addrspace(4) @llvm.amdgcn.kernarg.segment.ptr()
+; IR-NEXT:    [[OUT_KERNARG_OFFSET:%.*]] = getelementptr inbounds i8, ptr addrspace(4) [[BAR_KERNARG_SEGMENT]], i64 36
+; IR-NEXT:    [[TMP0:%.*]] = load <2 x i64>, ptr addrspace(4) [[OUT_KERNARG_OFFSET]], align 4
+; IR-NEXT:    [[OUT_LOAD1:%.*]] = extractelement <2 x i64> [[TMP0]], i32 0
+; IR-NEXT:    [[TMP1:%.*]] = inttoptr i64 [[OUT_LOAD1]] to ptr addrspace(1)
+; IR-NEXT:    [[VAL_LOAD2:%.*]] = extractelement <2 x i64> [[TMP0]], i32 1
+; IR-NEXT:    [[TMP2:%.*]] = inttoptr i64 [[VAL_LOAD2]] to ptr addrspace(1)
+; IR-NEXT:    [[LANE:%.*]] = tail call i32 @llvm.amdgcn.workitem.id.x()
+; IR-NEXT:    [[IDXPROM:%.*]] = sext i32 [[LANE]] to i64
+; IR-NEXT:    [[ELE:%.*]] = getelementptr i32, ptr addrspace(1) [[TMP2]], i64 [[IDXPROM]]
+; IR-NEXT:    [[VALUE:%.*]] = load i32, ptr addrspace(1) [[ELE]], align 4
+; IR-NEXT:    [[GEP:%.*]] = getelementptr i32, ptr addrspace(1) [[TMP1]], i32 4
+; IR-NEXT:    [[TMP3:%.*]] = call i64 @llvm.amdgcn.ballot.i64(i1 true)
+; IR-NEXT:    [[TMP4:%.*]] = bitcast i64 [[TMP3]] to <2 x i32>
+; IR-NEXT:    [[TMP5:%.*]] = extractelement <2 x i32> [[TMP4]], i32 0
+; IR-NEXT:    [[TMP6:%.*]] = extractelement <2 x i32> [[TMP4]], i32 1
+; IR-NEXT:    [[TMP7:%.*]] = call i32 @llvm.amdgcn.mbcnt.lo(i32 [[TMP5]], i32 0)
+; IR-NEXT:    [[TMP8:%.*]] = call i32 @llvm.amdgcn.mbcnt.hi(i32 [[TMP6]], i32 [[TMP7]])
+; IR-NEXT:    [[TMP9:%.*]] = call i64 @llvm.amdgcn.ballot.i64(i1 true)
+; IR-NEXT:    br label [[COMPUTELOOP:%.*]]
+; IR:       10:
+; IR-NEXT:    [[TMP11:%.*]] = atomicrmw volatile add ptr addrspace(1) [[GEP]], i32 [[TMP16:%.*]] seq_cst, align 4
+; IR-NEXT:    br label [[TMP12:%.*]]
+; IR:       12:
+; IR-NEXT:    ret void
+; IR:       ComputeLoop:
+; IR-NEXT:    [[ACCUMULATOR:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[TMP16]], [[COMPUTELOOP]] ]
+; IR-NEXT:    [[ACTIVEBITS:%.*]] = phi i64 [ [[TMP9]], [[ENTRY]] ], [ [[TMP19:%.*]], [[COMPUTELOOP]] ]
+; IR-NEXT:    [[TMP13:%.*]] = call i64 @llvm.cttz.i64(i64 [[ACTIVEBITS]], i1 true)
+; IR-NEXT:    [[TMP14:%.*]] = trunc i64 [[TMP13]] to i32
+; IR-NEXT:    [[TMP15:%.*]] = call i32 @llvm.amdgcn.readlane(i32 [[VALUE]], i32 [[TMP14]])
+; IR-NEXT:    [[TMP16]] = add i32 [[ACCUMULATOR]], [[TMP15]]
+; IR-NEXT:    [[TMP17:%.*]] = shl i64 1, [[TMP13]]
+; IR-NEXT:    [[TMP18:%.*]] = xor i64 [[TMP17]], -1
+; IR-NEXT:    [[TMP19]] = and i64 [[ACTIVEBITS]], [[TMP18]]
+; IR-NEXT:    [[TMP20:%.*]] = icmp eq i64 [[TMP19]], 0
+; IR-NEXT:    br i1 [[TMP20]], label [[COMPUTEEND:%.*]], label [[COMPUTELOOP]]
+; IR:       ComputeEnd:
+; IR-NEXT:    [[TMP21:%.*]] = icmp eq i32 [[TMP8]], 0
+; IR-NEXT:    br i1 [[TMP21]], label [[TMP10:%.*]], label [[TMP12]]
+;
+entry:
+  %bar.kernarg.segment = call nonnull align 16 dereferenceable(52) ptr addrspace(4) @llvm.amdgcn.kernarg.segment.ptr()
+  %out.kernarg.offset = getelementptr inbounds i8, ptr addrspace(4) %bar.kernarg.segment, i64 36
+  %0 = load <2 x i64>, ptr addrspace(4) %out.kernarg.offset, align 4
+  %out.load1 = extractelement <2 x i64> %0, i32 0
+  %1 = inttoptr i64 %out.load1 to ptr addrspace(1)
+  %val.load2 = extractelement <2 x i64> %0, i32 1
+  %2 = inttoptr i64 %val.load2 to ptr addrspace(1)
+  %lane = tail call i32 @llvm.amdgcn.workitem.id.x()
+  %idxprom = sext i32 %lane to i64
+  %ele = getelementptr i32, ptr addrspace(1) %2, i64 %idxprom
+  %value = load i32, ptr addrspace(1) %ele, align 4
+  %gep = getelementptr i32, ptr addrspace(1) %1, i32 4
+  %old = atomicrmw volatile add ptr addrspace(1) %gep, i32 %value seq_cst, align 4
+  ret void
+}
+
+declare i32 @llvm.amdgcn.workitem.id.x() #1
+declare i32 @llvm.amdgcn.workgroup.id.x() #1
+
+declare align 4 ptr addrspace(4) @llvm.amdgcn.kernarg.segment.ptr()
+
+attributes #0 = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "target-cpu"="gfx906" "uniform-work-group-size"="false" }
+attributes #1 = { nocallback nofree nosync nounwind speculatable willreturn memory(none)}
+