[AMDGPU] Combine s_or_saveexec, s_xor instructions.

[tsymalla]

Summary

This patch merges a consecutive sequence of

s_or_saveexec s_o, s_i
s_xor exec, exec, s_o

into a single

s_andn2_saveexec s_o, s_i instruction.
This patch also cleans up the SIOptimizeExecMasking pass a bit.

[foad]

This patch also cleans up the SIOptimizeExecMasking pass a bit.

Please put that in a separate NFC patch. It really does make life easier for reviewers.

[tsymalla]

In D129073#3628177, @foad wrote:

This patch also cleans up the SIOptimizeExecMasking pass a bit.

Please put that in a separate NFC patch. It really does make life easier for reviewers.

Will do that.

[tsymalla]

Cleanup submitted as separate NFC change in https://reviews.llvm.org/D129086, will change this patch after the NFC has landed

[tsymalla]

Rebased on top of the latest refactoring in SIOptimizeExecMasking

[tsymalla]

Could you please have a look at this one again?

[nhaehnle]

This is a good start. However, I have some high-level questions:

1. Scanning over entire basic blocks is bad for compile times, and this pass is already doing some scans. For example, optimizeExecSequence already scans for copies to exec. Can this be improved? Notice how the scan in optimizeExecSequence goes backwards in the basic block and limits itself to only a small number of instruction. I could imagine a restructuring of the pass so that every basic block is scanned backwards for an EXEC-writing instruction. Depending on what the instruction is (copy-to-exec, s_and_saveexec, s_xor) one of the optimizations can be applied.
2. Why is this change done in SIOptimizeExecMasking instead of SIOptimizeExecMaskingPreRA? Actually, I don't remember why we have the two passes in the first place. Perhaps @rampitec remembers?

llvm/lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

786

MI == MBB.end() is always false here.

[arsenm]

In D129073#3635410, @nhaehnle wrote:

2. Why is this change done in SIOptimizeExecMasking instead of SIOptimizeExecMaskingPreRA? Actually, I don't remember why we have the two passes in the first place. Perhaps @rampitec remembers?

The primary reason is we can't produce the terminators with output registers before register allocation in case we need to insert spills live out of the block for the save exec. RA needs to insert spills before terminators

[sebastian-ne]

llvm/lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

769–770

I guess this also works if the input register is not the same as the output register?

s_or_saveexec s_o, s_i
s_xor exec, exec, s_o

llvm/test/CodeGen/AMDGPU/s_or_saveexec_xor_combine.mir

99

Can you add a test where S_OR_SAVEEXEC is the last instruction?

[tsymalla]

In D129073#3635410, @nhaehnle wrote:

This is a good start. However, I have some high-level questions:

1. Scanning over entire basic blocks is bad for compile times, and this pass is already doing some scans. For example, optimizeExecSequence already scans for copies to exec. Can this be improved? Notice how the scan in optimizeExecSequence goes backwards in the basic block and limits itself to only a small number of instruction. I could imagine a restructuring of the pass so that every basic block is scanned backwards for an EXEC-writing instruction. Depending on what the instruction is (copy-to-exec, s_and_saveexec, s_xor) one of the optimizations can be applied.

Good idea. I will try to reuse the existing scan to inject the optimizations, if possible.

2. Why is this change done in SIOptimizeExecMasking instead of SIOptimizeExecMaskingPreRA? Actually, I don't remember why we have the two passes in the first place. Perhaps @rampitec remembers?

I don't know what exactly the reasons are to have two passes. However, as in SIOptimizeExecMasking S_*BINOP*_{B32, B64} instructions are swapped with their SAVEEXEC counterpart, it made sense to me to introduce the change here. This is the last time such instructions are inserted by a pass, so it's likely the pattern I am trying to match will only appear after SIOptimizeExecMasking has run. But please correct me if I'm wrong.

[rampitec]

In D129073#3635410, @nhaehnle wrote:

This is a good start. However, I have some high-level questions:

1. Scanning over entire basic blocks is bad for compile times, and this pass is already doing some scans. For example, optimizeExecSequence already scans for copies to exec. Can this be improved? Notice how the scan in optimizeExecSequence goes backwards in the basic block and limits itself to only a small number of instruction. I could imagine a restructuring of the pass so that every basic block is scanned backwards for an EXEC-writing instruction. Depending on what the instruction is (copy-to-exec, s_and_saveexec, s_xor) one of the optimizations can be applied.
2. Why is this change done in SIOptimizeExecMasking instead of SIOptimizeExecMaskingPreRA? Actually, I don't remember why we have the two passes in the first place. Perhaps @rampitec remembers?

We only had post-RA pass initially. AFAIR I have added pre-RA to actually save registers (D35967).

[rampitec]

In D129073#3635410, @nhaehnle wrote:

1. Scanning over entire basic blocks is bad for compile times, and this pass is already doing some scans. For example, optimizeExecSequence already scans for copies to exec. Can this be improved? Notice how the scan in optimizeExecSequence goes backwards in the basic block and limits itself to only a small number of instruction. I could imagine a restructuring of the pass so that every basic block is scanned backwards for an EXEC-writing instruction. Depending on what the instruction is (copy-to-exec, s_and_saveexec, s_xor) one of the optimizations can be applied.

It can probably start from the first terminator, scan backwards, and bail with the first instruction not modifying the exec.

[nhaehnle]

In D129073#3635744, @arsenm wrote:

In D129073#3635410, @nhaehnle wrote:

2. Why is this change done in SIOptimizeExecMasking instead of SIOptimizeExecMaskingPreRA? Actually, I don't remember why we have the two passes in the first place. Perhaps @rampitec remembers?

The primary reason is we can't produce the terminators with output registers before register allocation in case we need to insert spills live out of the block for the save exec. RA needs to insert spills before terminators

That makes sense, thanks.

[tsymalla]

llvm/lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

769–770

No, I don't think so.
If EXEC = 0b111000, s0 = 0b001001, s1 = 0b010010, then after
s_or_saveexec s0, s1
s_xor exec, exec, s0

EXEC = 0b010010

while for s_andn2_saveexec s1, s0 EXEC = s_andn2_saveexec s1, s1 = 0b100100, s_andn2_saveexec s0, s1 EXEC = s_andn2_saveexec s0, s0 = 0b000001.
Same goes if you change the order of operands, so DST of s_or_saveexec must be equal to SRC0 of s_or_saveexec and thus needs to be DST and SRC0 of s_andn2_saveexec, please correct me if I'm wrong

[sebastian-ne]

llvm/lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

769–770

Not quite sure if I’m following, but isn’t it

=> EXEC = 0b111000, s0 = 0b001001, s1 = 0b010010
s_or_saveexec s0, s1
=> EXEC = 0b111010, s0 = 0b111000, s1 = 0b010010
s_xor exec, exec, s0
=> EXEC = 0b000010, s0 = 0b111000, s1 = 0b010010

and

=> EXEC = 0b111000, s0 = 0b001001, s1 = 0b010010
s_andn2_saveexec s0, s1
=> EXEC = 0b000010, s0 = 0b111000, s1 = 0b010010
(EXEC = ~EXEC & s1 = 0b000111 & 0b010010 = 0b000010)

?

[tsymalla]

llvm/lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

769–770

Yes, you are correct. I probably messed up somewhere.

[tsymalla]

Merge two scans into one to reduce compile time.
Combine even if src and dst of s_or_saveexec are not equal.

[sebastian-ne]

Seems fine to me if you update the commit message and the comment.
I’ll leave approval to someone else.

llvm/lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

769–770

Can you update the comment please?

[tsymalla]

Update inline comment.

[arsenm]

llvm/lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

800–801

Why do you need to collect every instance in the function before processing them? Each of these can be handled standalone?

[tsymalla]

llvm/lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

800–801

I find that more convenient and easier to follow, especially if the optimization handles a bunch of cases. Instead of mixing the find pattern-do combine step, I prefer to first exclude all irrelevant matchings and then just transform the findings one by another.

[tsymalla]

Could anyone have a look at this again?

[nhaehnle]

This looks pretty good already, but I do have a bunch of code quality comments inline.

llvm/lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

463

Why the empty line?

626–632

Confusing comment. Just remove it.

739–740

Merge this function and the callee.

742

Inline this function into its caller.

746–749

Move those into optmizeVCmpSaveExecSequence (also maybe rename the method while you're at it? The singular "sequence" already implies that it's a single sequence...)

787–790

This is sort of backwards. When matching code for this kind of combine, the general pattern is to start from the definition of the final value you want to rewrite. In this case, that would be the S_XOR. You can see the approach taken in findPossibleVCMPVCMPXOptimization, using findInstrBackwards.

788–790

This isn't true. The v_cmp may potentially be far away, but the s_and_saveexec must be near the end of the block, so the entire loop should exit after the SearchWindow.

Also, and I know this isn't due to your change, but debug instructions should not be counted towards the search window. The goal is to avoid codegen changes caused by debug.

Also also, shouldn't we be able to exit this loop once we found the first write to EXEC?

801

Remove OrXorPair and use emplace_back here

817–818

Move into the inner scope.

[tsymalla]

llvm/lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

787–790

But wouldn't this cause more work to do? I think it's more likely to first find a S_XOR instruction which the compiler could use as start point for looking for the s_or_saveexec. The issue is, that we basically would require to stop at a lot of s_xor and see if the instruction before is a eligible s_or_saveexec instruction. The amount of checks would be reduced if we could first stop at the s_or_saveexec instruction and _then_ look at the s_xor instruction.

See following (artificial) example:

BB0:
s_or_saveexec_b32 s0, s1
s_xor_b32 exec_lo, exec_lo, s0
s_xor_b32 exec_lo, exec_lo, s1
s_xor_b32 exec_lo, exec_lo, s2

Three checks for three s_xor instructions when checking s_xor first and two of them will fail. In comparison, with the current approach, we'd only check once at the cost of incrementing the iterator.

788–790

I don't get the first part of your comment. If we exit after SearchWindow, we're likely to ignore some v_cmp instructions.

For the debug instructions, you're correct. I missed that.

For the EXEC part, I'll double-check.

[nhaehnle]

llvm/lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

787–790

Can such sequences actually happen, though? Before register allocation, we generally follow the rule that EXEC can only be written by special terminator instructions, and there is really no reason to have more than one of those. So I don't see where this sequence would come from.

See also the note about being able to stop the search after seeing the first instruction (from the end) that writes EXEC.

788–790

My point in the first part was that for v_cmp + s_and_saveexec -> s_mov + v_cmpx, we scan the search window for the s_and_saveexec. And then *from there* we look backwards for the v_cmp. This scan (via findInstrBackwards) isn't limited by SearchWindow.

[tsymalla]

llvm/lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

787–790

Not exactly. My point is, we will see single occurrences of s_xor_b32 exec_lo, exec_lo, s* as terminator instructions, and will stop at any of them even if in a lot of cases the combine won't be applied. But I can change the order of checks anyhow.

[tsymalla]

Addressed @nhaehnle's comments.

[nhaehnle]

LGTM

llvm/lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

 //===-- SIOptimizeExecMasking.cpp -----------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "AMDGPU.h"
 #include "GCNSubtarget.h"
 #include "MCTargetDesc/AMDGPUMCTargetDesc.h"
 #include "SIRegisterInfo.h"
 #include "llvm/CodeGen/LivePhysRegs.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineOperand.h"
+#include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/InitializePasses.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "si-optimize-exec-masking"
 
 namespace {
 
 class SIOptimizeExecMasking : public MachineFunctionPass {
   MachineFunction *MF = nullptr;
   const GCNSubtarget *ST = nullptr;
   const SIRegisterInfo *TRI = nullptr;
   const SIInstrInfo *TII = nullptr;
   const MachineRegisterInfo *MRI = nullptr;
+  MCRegister Exec;
+
+  DenseMap<MachineInstr *, MachineInstr *> SaveExecVCmpMapping;
+  SmallVector<std::pair<MachineInstr *, MachineInstr *>, 1> OrXors;
 
   Register isCopyFromExec(const MachineInstr &MI) const;
   Register isCopyToExec(const MachineInstr &MI) const;
   bool removeTerminatorBit(MachineInstr &MI) const;
   MachineBasicBlock::reverse_iterator
   fixTerminators(MachineBasicBlock &MBB) const;
   MachineBasicBlock::reverse_iterator
   findExecCopy(MachineBasicBlock &MBB, MachineBasicBlock::reverse_iterator I,
                unsigned CopyToExec) const;
 
   bool isRegisterInUseBetween(MachineInstr &Stop, MachineInstr &Start,
                               MCRegister Reg, bool UseLiveOuts = false,
                               bool IgnoreStart = false) const;
   bool isRegisterInUseAfter(MachineInstr &Stop, MCRegister Reg) const;
   MachineInstr *findInstrBackwards(MachineInstr &Origin,
                                    std::function<bool(MachineInstr *)> Pred,
                                    ArrayRef<MCRegister> NonModifiableRegs,
                                    unsigned MaxInstructions = 20) const;
-  MachineInstr *findPossibleVCMPVCMPXOptimization(MachineInstr &SaveExec,
-                                                  MCRegister Exec) const;
-  bool optimizeExecSequence() const;
-  bool optimizeVCmpxAndSaveexecSequence() const;
-  bool optimizeSingleVCMPSaveExecSequence(MachineInstr &SaveExecInstr,
-                                          MachineInstr &VCmp,
-                                          MCRegister Exec) const;
+  bool optimizeExecSequence();
+  void tryRecordVCmpxAndSaveexecSequence(MachineInstr &MI);
+  bool optimizeVCMPSaveExecSequence(MachineInstr &SaveExecInstr,
+                                    MachineInstr &VCmp, MCRegister Exec) const;
+
+  void tryRecordOrSaveexecXorSequence(MachineInstr &MI);
+  bool optimizeOrSaveexecXorSequences();
 
 public:
   static char ID;
 
   SIOptimizeExecMasking() : MachineFunctionPass(ID) {
     initializeSIOptimizeExecMaskingPass(*PassRegistry::getPassRegistry());
   }
 

 Register SIOptimizeExecMasking::isCopyFromExec(const MachineInstr &MI) const {
   switch (MI.getOpcode()) {
   case AMDGPU::COPY:
   case AMDGPU::S_MOV_B64:
   case AMDGPU::S_MOV_B64_term:
   case AMDGPU::S_MOV_B32:
   case AMDGPU::S_MOV_B32_term: {
     const MachineOperand &Src = MI.getOperand(1);
-    if (Src.isReg() && Src.getReg() == TRI->getExec())
+    if (Src.isReg() && Src.getReg() == Exec)
       return MI.getOperand(0).getReg();
   }
   }
 
   return AMDGPU::NoRegister;
 }
 
 /// If \p MI is a copy to exec, return the register copied from.
 Register SIOptimizeExecMasking::isCopyToExec(const MachineInstr &MI) const {
   switch (MI.getOpcode()) {
   case AMDGPU::COPY:
   case AMDGPU::S_MOV_B64:
   case AMDGPU::S_MOV_B32: {
     const MachineOperand &Dst = MI.getOperand(0);
-    if (Dst.isReg() && Dst.getReg() == TRI->getExec() &&
-        MI.getOperand(1).isReg())
+    if (Dst.isReg() && Dst.getReg() == Exec && MI.getOperand(1).isReg())
       return MI.getOperand(1).getReg();
     break;
   }
   case AMDGPU::S_MOV_B64_term:
   case AMDGPU::S_MOV_B32_term:
     llvm_unreachable("should have been replaced");
   }
 

 // inserted for the saved copy of exec.
 //
 //     x = copy exec
 //     z = s_<op>_b64 x, y
 //     exec = copy z
 // =>
 //     x = s_<op>_saveexec_b64 y
 //
-bool SIOptimizeExecMasking::optimizeExecSequence() const {
-  MCRegister Exec = TRI->getExec();
-
+bool SIOptimizeExecMasking::optimizeExecSequence() {
   bool Changed = false;
   for (MachineBasicBlock &MBB : *MF) {
     MachineBasicBlock::reverse_iterator I = fixTerminators(MBB);
     MachineBasicBlock::reverse_iterator E = MBB.rend();
     if (I == E)
       continue;
 
     // It's possible to see other terminator copies after the exec copy. This
for (MachineBasicBlock &MBB : *MF) {
     Register CopyFromExec = CopyFromExecInst->getOperand(0).getReg();
     MachineInstr *SaveExecInst = nullptr;
     SmallVector<MachineInstr *, 4> OtherUseInsts;
 
     for (MachineBasicBlock::iterator
              J = std::next(CopyFromExecInst->getIterator()),
              JE = I->getIterator();
          J != JE; ++J) {
       if (SaveExecInst && J->readsRegister(Exec, TRI)) {
         LLVM_DEBUG(dbgs() << "exec read prevents saveexec: " << *J << '\n');
         // Make sure this is inserted after any VALU ops that may have been
         // scheduled in between.
         SaveExecInst = nullptr;
         break;
       }
 
       bool ReadsCopyFromExec = J->readsRegister(CopyFromExec, TRI);
for (MachineBasicBlock &MBB : *MF) {
     }
 
     Changed = true;
   }
 
   return Changed;
 }
 
-// Tries to find a possibility to optimize a v_cmp ..., s_and_saveexec sequence
-// by looking at an instance of a s_and_saveexec instruction. Returns a pointer
-// to the v_cmp instruction if it is safe to replace the sequence (see the
-// conditions in the function body). This is after register allocation, so some
-// checks on operand dependencies need to be considered.
-MachineInstr *SIOptimizeExecMasking::findPossibleVCMPVCMPXOptimization(
-    MachineInstr &SaveExec, MCRegister Exec) const {
-
-  MachineInstr *VCmp = nullptr;
-
-  Register SaveExecDest = SaveExec.getOperand(0).getReg();
-  if (!TRI->isSGPRReg(*MRI, SaveExecDest))
-    return nullptr;
-
-  MachineOperand *SaveExecSrc0 =
-      TII->getNamedOperand(SaveExec, AMDGPU::OpName::src0);
-  if (!SaveExecSrc0->isReg())
-    return nullptr;
-
-  // Try to find the last v_cmp instruction that defs the saveexec input
-  // operand without any write to Exec or the saveexec input operand inbetween.
-  VCmp = findInstrBackwards(
-      SaveExec,
-      [&](MachineInstr *Check) {
-        return AMDGPU::getVCMPXOpFromVCMP(Check->getOpcode()) != -1 &&
-               Check->modifiesRegister(SaveExecSrc0->getReg(), TRI);
-      },
-      {Exec, SaveExecSrc0->getReg()});
-
-  if (!VCmp)
-    return nullptr;
-
-  MachineOperand *VCmpDest = TII->getNamedOperand(*VCmp, AMDGPU::OpName::sdst);
-  assert(VCmpDest && "Should have an sdst operand!");
-
-  // Check if any of the v_cmp source operands is written by the saveexec.
-  MachineOperand *Src0 = TII->getNamedOperand(*VCmp, AMDGPU::OpName::src0);
-  if (Src0->isReg() && TRI->isSGPRReg(*MRI, Src0->getReg()) &&
-      SaveExec.modifiesRegister(Src0->getReg(), TRI))
-    return nullptr;
-
-  MachineOperand *Src1 = TII->getNamedOperand(*VCmp, AMDGPU::OpName::src1);
-  if (Src1->isReg() && TRI->isSGPRReg(*MRI, Src1->getReg()) &&
-      SaveExec.modifiesRegister(Src1->getReg(), TRI))
-    return nullptr;
-
-  // Don't do the transformation if the destination operand is included in
-  // it's MBB Live-outs, meaning it's used in any of it's successors, leading
-  // to incorrect code if the v_cmp and therefore the def of
-  // the dest operand is removed.
-  if (isLiveOut(*VCmp->getParent(), VCmpDest->getReg()))
-    return nullptr;
-
-  // If the v_cmp target is in use between v_cmp and s_and_saveexec or after the
-  // s_and_saveexec, skip the optimization.
-  if (isRegisterInUseBetween(*VCmp, SaveExec, VCmpDest->getReg(), false,
-                             true) ||
-      isRegisterInUseAfter(SaveExec, VCmpDest->getReg()))
-    return nullptr;
-
-  // Try to determine if there is a write to any of the VCmp
-  // operands between the saveexec and the vcmp.
-  // If yes, additional VGPR spilling might need to be inserted. In this case,
-  // it's not worth replacing the instruction sequence.
-  SmallVector<MCRegister, 2> NonDefRegs;
-  if (Src0->isReg())
-    NonDefRegs.push_back(Src0->getReg());
-
-  if (Src1->isReg())
-    NonDefRegs.push_back(Src1->getReg());
-
-  if (!findInstrBackwards(
-          SaveExec, [&](MachineInstr *Check) { return Check == VCmp; },
-          NonDefRegs))
-    return nullptr;
-
-  return VCmp;
-}
-
 // Inserts the optimized s_mov_b32 / v_cmpx sequence based on the
 // operands extracted from a v_cmp ..., s_and_saveexec pattern.
-bool SIOptimizeExecMasking::optimizeSingleVCMPSaveExecSequence(
+bool SIOptimizeExecMasking::optimizeVCMPSaveExecSequence(
     MachineInstr &SaveExecInstr, MachineInstr &VCmp, MCRegister Exec) const {
   const int NewOpcode = AMDGPU::getVCMPXOpFromVCMP(VCmp.getOpcode());
 
   if (NewOpcode == -1)
     return false;
 
   MachineOperand *Src0 = TII->getNamedOperand(VCmp, AMDGPU::OpName::src0);
   MachineOperand *Src1 = TII->getNamedOperand(VCmp, AMDGPU::OpName::src1);
bool SIOptimizeExecMasking::optimizeVCMPSaveExecSequence(
   TryAddImmediateValueFromNamedOperand(AMDGPU::OpName::clamp);
 
   // The kill flags may no longer be correct.
   if (Src0->isReg())
     MRI->clearKillFlags(Src0->getReg());
   if (Src1->isReg())
     MRI->clearKillFlags(Src1->getReg());
 
+  SaveExecInstr.eraseFromParent();
+  VCmp.eraseFromParent();
+
   return true;
 }
 
-// After all s_op_saveexec instructions are inserted,
-// replace (on GFX10.3 and later)
+// Record (on GFX10.3 and later) occurences of
 // v_cmp_* SGPR, IMM, VGPR
 // s_and_saveexec_b32 EXEC_SGPR_DEST, SGPR
-// with
+// to be replaced with
 // s_mov_b32 EXEC_SGPR_DEST, exec_lo
 // v_cmpx_* IMM, VGPR
 // to reduce pipeline stalls.
-bool SIOptimizeExecMasking::optimizeVCmpxAndSaveexecSequence() const {
+void SIOptimizeExecMasking::tryRecordVCmpxAndSaveexecSequence(
+    MachineInstr &MI) {
   if (!ST->hasGFX10_3Insts())
-    return false;
-
-  bool Changed = false;
+    return;
 
-  DenseMap<MachineInstr *, MachineInstr *> SaveExecVCmpMapping;
-  MCRegister Exec = TRI->getExec();
   const unsigned AndSaveExecOpcode =
       ST->isWave32() ? AMDGPU::S_AND_SAVEEXEC_B32 : AMDGPU::S_AND_SAVEEXEC_B64;
 
-  for (MachineBasicBlock &MBB : *MF) {
-    for (MachineInstr &MI : MBB) {
-      // Record relevant v_cmp / s_and_saveexec instruction pairs for
-      // replacement.
-      if (MI.getOpcode() != AndSaveExecOpcode)
-        continue;
+    return;
+
+  Register SaveExecDest = MI.getOperand(0).getReg();
+  if (!TRI->isSGPRReg(*MRI, SaveExecDest))
+    return;
+
+  MachineOperand *SaveExecSrc0 = TII->getNamedOperand(MI, AMDGPU::OpName::src0);
+  if (!SaveExecSrc0->isReg())
+    return;
+
+  // Tries to find a possibility to optimize a v_cmp ..., s_and_saveexec
+  // sequence by looking at an instance of a s_and_saveexec instruction. Returns
+  // a pointer to the v_cmp instruction if it is safe to replace the sequence
+  // (see the conditions in the function body). This is after register
+  // allocation, so some checks on operand dependencies need to be considered.
+  MachineInstr *VCmp = nullptr;
 
-      if (MachineInstr *VCmp = findPossibleVCMPVCMPXOptimization(MI, Exec))
-        SaveExecVCmpMapping[&MI] = VCmp;
-    }
-  }
+  // Try to find the last v_cmp instruction that defs the saveexec input
+  // operand without any write to Exec or the saveexec input operand inbetween.
+  VCmp = findInstrBackwards(
+      MI,
+      [&](MachineInstr *Check) {
+        return AMDGPU::getVCMPXOpFromVCMP(Check->getOpcode()) != -1 &&
+               Check->modifiesRegister(SaveExecSrc0->getReg(), TRI);
+      },
+      {Exec, SaveExecSrc0->getReg()});
+
+  if (!VCmp)
+    return;
+
+  MachineOperand *VCmpDest = TII->getNamedOperand(*VCmp, AMDGPU::OpName::sdst);
+  assert(VCmpDest && "Should have an sdst operand!");
+
+  // Check if any of the v_cmp source operands is written by the saveexec.
+  MachineOperand *Src0 = TII->getNamedOperand(*VCmp, AMDGPU::OpName::src0);
+  if (Src0->isReg() && TRI->isSGPRReg(*MRI, Src0->getReg()) &&
+      MI.modifiesRegister(Src0->getReg(), TRI))
+    return;
+
+  MachineOperand *Src1 = TII->getNamedOperand(*VCmp, AMDGPU::OpName::src1);
+  if (Src1->isReg() && TRI->isSGPRReg(*MRI, Src1->getReg()) &&
+      MI.modifiesRegister(Src1->getReg(), TRI))
+    return;
+
+  // Don't do the transformation if the destination operand is included in
+  // it's MBB Live-outs, meaning it's used in any of it's successors, leading
+  // to incorrect code if the v_cmp and therefore the def of
+  // the dest operand is removed.
+  if (isLiveOut(*VCmp->getParent(), VCmpDest->getReg()))
+    return;
+
+  // If the v_cmp target is in use between v_cmp and s_and_saveexec or after the
+  // s_and_saveexec, skip the optimization.
+  if (isRegisterInUseBetween(*VCmp, MI, VCmpDest->getReg(), false, true) ||
+      isRegisterInUseAfter(MI, VCmpDest->getReg()))
+    return;
+
+  // Try to determine if there is a write to any of the VCmp
+  // operands between the saveexec and the vcmp.
+  // If yes, additional VGPR spilling might need to be inserted. In this case,
+  // it's not worth replacing the instruction sequence.
+  SmallVector<MCRegister, 2> NonDefRegs;
+  if (Src0->isReg())
+    NonDefRegs.push_back(Src0->getReg());
+
+  if (Src1->isReg())
+    NonDefRegs.push_back(Src1->getReg());
+
+  if (!findInstrBackwards(
+          MI, [&](MachineInstr *Check) { return Check == VCmp; }, NonDefRegs))
+    return;
+
+  if (VCmp)
+
+// Record occurences of
+// s_or_saveexec s_o, s_i
+// s_xor exec, exec, s_o
+// to be replaced with
+// s_andn2_saveexec s_o, s_i.
+void SIOptimizeExecMasking::tryRecordOrSaveexecXorSequence(MachineInstr &MI) {
+  const unsigned XorOpcode =
+      ST->isWave32() ? AMDGPU::S_XOR_B32 : AMDGPU::S_XOR_B64;
+
+  if (MI.getOpcode() == XorOpcode && &MI != &MI.getParent()->front()) {
+    const MachineOperand &XorDst = MI.getOperand(0);
+    const MachineOperand &XorSrc0 = MI.getOperand(1);
+    const MachineOperand &XorSrc1 = MI.getOperand(2);
+
+    if (XorDst.isReg() && XorDst.getReg() == Exec && XorSrc0.isReg() &&
+        XorSrc1.isReg() &&
+        (XorSrc0.getReg() == Exec || XorSrc1.getReg() == Exec)) {
+      const unsigned OrSaveexecOpcode = ST->isWave32()
+                                            ? AMDGPU::S_OR_SAVEEXEC_B32
+                                            : AMDGPU::S_OR_SAVEEXEC_B64;
+
+      // Peek at the previous instruction and check if this is a relevant
+      // s_or_saveexec instruction.
+      MachineInstr &PossibleOrSaveexec = *MI.getPrevNode();
+      if (PossibleOrSaveexec.getOpcode() != OrSaveexecOpcode)
+        return;
+
+      const MachineOperand &OrDst = PossibleOrSaveexec.getOperand(0);
+      const MachineOperand &OrSrc0 = PossibleOrSaveexec.getOperand(1);
+      if (OrDst.isReg() && OrSrc0.isReg()) {
+        if ((XorSrc0.getReg() == Exec && XorSrc1.getReg() == OrDst.getReg()) ||
+            (XorSrc0.getReg() == OrDst.getReg() && XorSrc1.getReg() == Exec)) {
+          OrXors.emplace_back(&PossibleOrSaveexec, &MI);
+        }
+      }
+    }
+  }
 
-  for (const auto &Entry : SaveExecVCmpMapping) {
-    MachineInstr *SaveExecInstr = Entry.getFirst();
-    MachineInstr *VCmpInstr = Entry.getSecond();
+bool SIOptimizeExecMasking::optimizeOrSaveexecXorSequences() {
+  if (OrXors.empty()) {
+    return false;
+  }
 
-    if (optimizeSingleVCMPSaveExecSequence(*SaveExecInstr, *VCmpInstr, Exec)) {
-      SaveExecInstr->eraseFromParent();
-      VCmpInstr->eraseFromParent();
+  bool Changed = false;
+  const unsigned Andn2Opcode = ST->isWave32() ? AMDGPU::S_ANDN2_SAVEEXEC_B32
+                                              : AMDGPU::S_ANDN2_SAVEEXEC_B64;
+
+  for (const auto &Pair : OrXors) {
+    MachineInstr *Or = nullptr;
+    MachineInstr *Xor = nullptr;
+    std::tie(Or, Xor) = Pair;
+    BuildMI(*Or->getParent(), Or->getIterator(), Or->getDebugLoc(),
+            TII->get(Andn2Opcode), Or->getOperand(0).getReg())
+        .addReg(Or->getOperand(1).getReg());
+
+    Or->eraseFromParent();
+    Xor->eraseFromParent();
 
-      Changed = true;
-    }
-  }
 
   return Changed;
 }
 
 bool SIOptimizeExecMasking::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(MF.getFunction()))
     return false;
 
   this->MF = &MF;
   ST = &MF.getSubtarget<GCNSubtarget>();
   TRI = ST->getRegisterInfo();
   TII = ST->getInstrInfo();
   MRI = &MF.getRegInfo();
+  Exec = TRI->getExec();
 
   bool Changed = optimizeExecSequence();
-  Changed |= optimizeVCmpxAndSaveexecSequence();
+
+  OrXors.clear();
+  SaveExecVCmpMapping.clear();
+  static unsigned SearchWindow = 10;
+  for (MachineBasicBlock &MBB : MF) {
+    unsigned SearchCount = 0;
+
+    for (auto &MI : llvm::reverse(MBB)) {
+      if (MI.isDebugInstr())
+        continue;
+
+      if (SearchCount >= SearchWindow) {
+        break;
+      }
+
+      tryRecordOrSaveexecXorSequence(MI);
+      tryRecordVCmpxAndSaveexecSequence(MI);
+
+      if (MI.modifiesRegister(Exec, TRI)) {
+        break;
+      }
+
+      ++SearchCount;
+    }
+  }
+
+  Changed |= optimizeOrSaveexecXorSequences();
+  for (const auto &Entry : SaveExecVCmpMapping) {
+    MachineInstr *SaveExecInstr = Entry.getFirst();
+    MachineInstr *VCmpInstr = Entry.getSecond();
+
+    Changed |= optimizeVCMPSaveExecSequence(*SaveExecInstr, *VCmpInstr, Exec);
+  }
 
   return Changed;
 }

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.memmove.ll

 ; LOOP-NEXT:    v_addc_u32_e32 v7, vcc, v1, v5, vcc
 ; LOOP-NEXT:    v_add_i32_e32 v4, vcc, 1, v4
 ; LOOP-NEXT:    v_addc_u32_e32 v5, vcc, 0, v5, vcc
 ; LOOP-NEXT:    v_cmp_ne_u32_e32 vcc, 4, v4
 ; LOOP-NEXT:    s_waitcnt vmcnt(0)
 ; LOOP-NEXT:    buffer_store_byte v8, v[6:7], s[0:3], 0 addr64
 ; LOOP-NEXT:    s_cbranch_vccnz .LBB0_2
 ; LOOP-NEXT:  .LBB0_3: ; %Flow14
-; LOOP-NEXT:    s_or_saveexec_b64 s[0:1], s[4:5]
-; LOOP-NEXT:    s_xor_b64 exec, exec, s[0:1]
+; LOOP-NEXT:    s_andn2_saveexec_b64 s[0:1], s[4:5]
 ; LOOP-NEXT:    s_cbranch_execz .LBB0_6
 ; LOOP-NEXT:  ; %bb.4: ; %copy_backwards
 ; LOOP-NEXT:    s_mov_b64 s[4:5], 3
 ; LOOP-NEXT:    s_mov_b32 s2, 0
 ; LOOP-NEXT:    s_mov_b32 s3, 0xf000
 ; LOOP-NEXT:    s_mov_b64 s[0:1], 0
 ; LOOP-NEXT:    v_mov_b32_e32 v4, s4
 ; LOOP-NEXT:    v_mov_b32_e32 v5, s5

llvm/test/CodeGen/AMDGPU/GlobalISel/localizer.ll

 ; GFX9-LABEL: localize_internal_globals:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    v_and_b32_e32 v0, 1, v0
 ; GFX9-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
 ; GFX9-NEXT:    s_xor_b64 s[4:5], vcc, -1
 ; GFX9-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
 ; GFX9-NEXT:    s_xor_b64 s[4:5], exec, s[6:7]
-; GFX9-NEXT:    s_cbranch_execz .LBB2_2
-; GFX9-NEXT:  ; %bb.1: ; %bb1
+; GFX9-NEXT:    s_cbranch_execnz .LBB2_3
+; GFX9-NEXT:  ; %bb.1: ; %Flow
+; GFX9-NEXT:    s_andn2_saveexec_b64 s[4:5], s[4:5]
+; GFX9-NEXT:    s_cbranch_execnz .LBB2_4
+; GFX9-NEXT:  .LBB2_2: ; %bb2
+; GFX9-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+; GFX9-NEXT:  .LBB2_3: ; %bb1
 ; GFX9-NEXT:    s_getpc_b64 s[6:7]
 ; GFX9-NEXT:    s_add_u32 s6, s6, static.gv2@rel32@lo+4
 ; GFX9-NEXT:    s_addc_u32 s7, s7, static.gv2@rel32@hi+12
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    global_store_dword v0, v0, s[6:7]
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_getpc_b64 s[6:7]
 ; GFX9-NEXT:    s_add_u32 s6, s6, static.gv3@rel32@lo+4
 ; GFX9-NEXT:    s_addc_u32 s7, s7, static.gv3@rel32@hi+12
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 1
 ; GFX9-NEXT:    global_store_dword v0, v1, s[6:7]
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:  .LBB2_2: ; %Flow
-; GFX9-NEXT:    s_or_saveexec_b64 s[4:5], s[4:5]
-; GFX9-NEXT:    s_xor_b64 exec, exec, s[4:5]
-; GFX9-NEXT:    s_cbranch_execz .LBB2_4
-; GFX9-NEXT:  ; %bb.3: ; %bb0
+; GFX9-NEXT:    s_andn2_saveexec_b64 s[4:5], s[4:5]
+; GFX9-NEXT:    s_cbranch_execz .LBB2_2
+; GFX9-NEXT:  .LBB2_4: ; %bb0
 ; GFX9-NEXT:    s_getpc_b64 s[6:7]
 ; GFX9-NEXT:    s_add_u32 s6, s6, static.gv0@rel32@lo+4
 ; GFX9-NEXT:    s_addc_u32 s7, s7, static.gv0@rel32@hi+12
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    global_store_dword v0, v0, s[6:7]
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_getpc_b64 s[6:7]
 ; GFX9-NEXT:    s_add_u32 s6, s6, static.gv1@rel32@lo+4
 ; GFX9-NEXT:    s_addc_u32 s7, s7, static.gv1@rel32@hi+12
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 1
 ; GFX9-NEXT:    global_store_dword v0, v1, s[6:7]
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:  .LBB2_4: ; %bb2
 ; GFX9-NEXT:    s_or_b64 exec, exec, s[4:5]
-; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 entry:
   br i1 %cond, label %bb0, label %bb1
 
 bb0:
   store volatile i32 0, i32 addrspace(1)* @static.gv0
   store volatile i32 1, i32 addrspace(1)* @static.gv1
   br label %bb2

llvm/test/CodeGen/AMDGPU/GlobalISel/sdiv.i64.ll

 ; CHECK-NEXT:    v_mov_b32_e32 v5, v1
 ; CHECK-NEXT:    v_mov_b32_e32 v4, v0
 ; CHECK-NEXT:    v_or_b32_e32 v1, v5, v3
 ; CHECK-NEXT:    v_mov_b32_e32 v0, 0
 ; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CHECK-NEXT:    s_cbranch_execz .LBB0_2
-; CHECK-NEXT:  ; %bb.1:
+; CHECK-NEXT:    s_cbranch_execnz .LBB0_3
+; CHECK-NEXT:  ; %bb.1: ; %Flow
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CHECK-NEXT:    s_cbranch_execnz .LBB0_4
+; CHECK-NEXT:  .LBB0_2:
+; CHECK-NEXT:    s_or_b64 exec, exec, s[6:7]
+; CHECK-NEXT:    s_setpc_b64 s[30:31]
+; CHECK-NEXT:  .LBB0_3:
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v0, 31, v3
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v2, v0
 ; CHECK-NEXT:    v_addc_u32_e32 v2, vcc, v3, v0, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v6, v1, v0
 ; CHECK-NEXT:    v_xor_b32_e32 v7, v2, v0
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v1, v6
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v2, v7
 ; CHECK-NEXT:    v_sub_i32_e32 v9, vcc, 0, v6

 ; CHECK-NEXT:    v_xor_b32_e32 v3, v9, v0
 ; CHECK-NEXT:    v_cndmask_b32_e32 v2, v8, v2, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, v1, v3
 ; CHECK-NEXT:    v_xor_b32_e32 v1, v2, v3
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v3
 ; CHECK-NEXT:    v_subb_u32_e32 v1, vcc, v1, v3, vcc
 ; CHECK-NEXT:    ; implicit-def: $vgpr2
 ; CHECK-NEXT:    ; implicit-def: $vgpr4
-; CHECK-NEXT:  .LBB0_2: ; %Flow
-; CHECK-NEXT:    s_or_saveexec_b64 s[6:7], s[6:7]
-; CHECK-NEXT:    s_xor_b64 exec, exec, s[6:7]
-; CHECK-NEXT:    s_cbranch_execz .LBB0_4
-; CHECK-NEXT:  ; %bb.3:
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CHECK-NEXT:    s_cbranch_execz .LBB0_2
+; CHECK-NEXT:  .LBB0_4:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v2
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, 0, v2
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v1, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v0, v4, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v0, v2
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, 1, v0
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v4, v1
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v2
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
 ; CHECK-NEXT:    v_sub_i32_e64 v3, s[4:5], v1, v2
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, 1, v0
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v2
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
 ; CHECK-NEXT:    v_mov_b32_e32 v1, 0
-; CHECK-NEXT:  .LBB0_4:
 ; CHECK-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %result = sdiv i64 %num, %den
   ret i64 %result
 }
 
 ; FIXME: This is a workaround for not handling uniform VGPR case.
 declare i32 @llvm.amdgcn.readfirstlane(i32)

 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v12, v2, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v0, v1, v3
 ; CGP-NEXT:    v_xor_b32_e32 v1, v2, v3
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v3
 ; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v3, vcc
 ; CGP-NEXT:    ; implicit-def: $vgpr4
 ; CGP-NEXT:    ; implicit-def: $vgpr10
 ; CGP-NEXT:  .LBB2_2: ; %Flow1
-; CGP-NEXT:    s_or_saveexec_b64 s[6:7], s[6:7]
-; CGP-NEXT:    s_xor_b64 exec, exec, s[6:7]
+; CGP-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
 ; CGP-NEXT:    s_cbranch_execz .LBB2_4
 ; CGP-NEXT:  ; %bb.3:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v0, v4
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, 0, v4
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_lo_u32 v1, v1, v0

 ; CGP-NEXT:  .LBB2_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CGP-NEXT:    v_or_b32_e32 v3, v9, v7
 ; CGP-NEXT:    v_mov_b32_e32 v2, 0
 ; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CGP-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CGP-NEXT:    s_cbranch_execz .LBB2_6
-; CGP-NEXT:  ; %bb.5:
+; CGP-NEXT:    s_cbranch_execnz .LBB2_7
+; CGP-NEXT:  ; %bb.5: ; %Flow
+; CGP-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CGP-NEXT:    s_cbranch_execnz .LBB2_8
+; CGP-NEXT:  .LBB2_6:
+; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
+; CGP-NEXT:    s_setpc_b64 s[30:31]
+; CGP-NEXT:  .LBB2_7:
 ; CGP-NEXT:    v_ashrrev_i32_e32 v2, 31, v7
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, v6, v2
 ; CGP-NEXT:    v_addc_u32_e32 v4, vcc, v7, v2, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v6, v3, v2
 ; CGP-NEXT:    v_xor_b32_e32 v7, v4, v2
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v3, v6
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v4, v7
 ; CGP-NEXT:    v_sub_i32_e32 v11, vcc, 0, v6

 ; CGP-NEXT:    v_xor_b32_e32 v5, v11, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v4, v10, v4, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v2, v3, v5
 ; CGP-NEXT:    v_xor_b32_e32 v3, v4, v5
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v5
 ; CGP-NEXT:    v_subb_u32_e32 v3, vcc, v3, v5, vcc
 ; CGP-NEXT:    ; implicit-def: $vgpr6
 ; CGP-NEXT:    ; implicit-def: $vgpr8
-; CGP-NEXT:  .LBB2_6: ; %Flow
-; CGP-NEXT:    s_or_saveexec_b64 s[6:7], s[6:7]
-; CGP-NEXT:    s_xor_b64 exec, exec, s[6:7]
-; CGP-NEXT:    s_cbranch_execz .LBB2_8
-; CGP-NEXT:  ; %bb.7:
+; CGP-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CGP-NEXT:    s_cbranch_execz .LBB2_6
+; CGP-NEXT:  .LBB2_8:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v6
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v6
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_hi_u32 v2, v8, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v2, v6
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v2
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v8, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v3, v6
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v4, s[4:5], v3, v6
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v3, v4, vcc
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v2
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v3, v6
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v3, 0
-; CGP-NEXT:  .LBB2_8:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = sdiv <2 x i64> %num, %den
   ret <2 x i64> %result
 }
 
 define i64 @v_sdiv_i64_pow2k_denom(i64 %num) {
 ; CHECK-LABEL: v_sdiv_i64_pow2k_denom:

 ; CHECK-NEXT:    v_mov_b32_e32 v4, v1
 ; CHECK-NEXT:    v_mov_b32_e32 v3, v0
 ; CHECK-NEXT:    v_or_b32_e32 v1, v4, v6
 ; CHECK-NEXT:    v_mov_b32_e32 v0, 0
 ; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CHECK-NEXT:    s_cbranch_execz .LBB7_2
-; CHECK-NEXT:  ; %bb.1:
+; CHECK-NEXT:    s_cbranch_execnz .LBB7_3
+; CHECK-NEXT:  ; %bb.1: ; %Flow
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CHECK-NEXT:    s_cbranch_execnz .LBB7_4
+; CHECK-NEXT:  .LBB7_2:
+; CHECK-NEXT:    s_or_b64 exec, exec, s[6:7]
+; CHECK-NEXT:    s_setpc_b64 s[30:31]
+; CHECK-NEXT:  .LBB7_3:
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v0, 31, v6
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v5, v0
 ; CHECK-NEXT:    v_addc_u32_e32 v2, vcc, v6, v0, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v7, v1, v0
 ; CHECK-NEXT:    v_xor_b32_e32 v8, v2, v0
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v1, v7
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v2, v8
 ; CHECK-NEXT:    v_sub_i32_e32 v10, vcc, 0, v7

 ; CHECK-NEXT:    v_xor_b32_e32 v3, v10, v0
 ; CHECK-NEXT:    v_cndmask_b32_e32 v2, v5, v2, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, v1, v3
 ; CHECK-NEXT:    v_xor_b32_e32 v1, v2, v3
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v3
 ; CHECK-NEXT:    v_subb_u32_e32 v1, vcc, v1, v3, vcc
 ; CHECK-NEXT:    ; implicit-def: $vgpr5_vgpr6
 ; CHECK-NEXT:    ; implicit-def: $vgpr3
-; CHECK-NEXT:  .LBB7_2: ; %Flow
-; CHECK-NEXT:    s_or_saveexec_b64 s[6:7], s[6:7]
-; CHECK-NEXT:    s_xor_b64 exec, exec, s[6:7]
-; CHECK-NEXT:    s_cbranch_execz .LBB7_4
-; CHECK-NEXT:  ; %bb.3:
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CHECK-NEXT:    s_cbranch_execz .LBB7_2
+; CHECK-NEXT:  .LBB7_4:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v5
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, 0, v5
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v1, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v0, v3, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v0, v5
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v3, v1
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v5
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CHECK-NEXT:    v_sub_i32_e64 v2, s[4:5], v1, v5
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v2, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v5
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CHECK-NEXT:    v_mov_b32_e32 v1, 0
-; CHECK-NEXT:  .LBB7_4:
 ; CHECK-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %shl.y = shl i64 4096, %y
   %r = sdiv i64 %x, %shl.y
   ret i64 %r
 }
 
 define <2 x i64> @v_sdiv_v2i64_pow2_shl_denom(<2 x i64> %x, <2 x i64> %y) {

 ; CGP-NEXT:  .LBB8_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[8:9]
 ; CGP-NEXT:    v_or_b32_e32 v3, v7, v10
 ; CGP-NEXT:    v_mov_b32_e32 v2, 0
 ; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CGP-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CGP-NEXT:    s_cbranch_execz .LBB8_6
-; CGP-NEXT:  ; %bb.5:
+; CGP-NEXT:    s_cbranch_execnz .LBB8_7
+; CGP-NEXT:  ; %bb.5: ; %Flow
+; CGP-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CGP-NEXT:    s_cbranch_execnz .LBB8_8
+; CGP-NEXT:  .LBB8_6:
+; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
+; CGP-NEXT:    s_setpc_b64 s[30:31]
+; CGP-NEXT:  .LBB8_7:
 ; CGP-NEXT:    v_ashrrev_i32_e32 v2, 31, v10
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, v9, v2
 ; CGP-NEXT:    v_addc_u32_e32 v4, vcc, v10, v2, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v6, v3, v2
 ; CGP-NEXT:    v_xor_b32_e32 v10, v4, v2
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v3, v6
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v4, v10
 ; CGP-NEXT:    v_sub_i32_e32 v12, vcc, 0, v6

 ; CGP-NEXT:    v_xor_b32_e32 v5, v12, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v4, v8, v4, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v2, v3, v5
 ; CGP-NEXT:    v_xor_b32_e32 v3, v4, v5
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v5
 ; CGP-NEXT:    v_subb_u32_e32 v3, vcc, v3, v5, vcc
 ; CGP-NEXT:    ; implicit-def: $vgpr9_vgpr10
 ; CGP-NEXT:    ; implicit-def: $vgpr5
-; CGP-NEXT:  .LBB8_6: ; %Flow
-; CGP-NEXT:    s_or_saveexec_b64 s[6:7], s[6:7]
-; CGP-NEXT:    s_xor_b64 exec, exec, s[6:7]
-; CGP-NEXT:    s_cbranch_execz .LBB8_8
-; CGP-NEXT:  ; %bb.7:
+; CGP-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CGP-NEXT:    s_cbranch_execz .LBB8_6
+; CGP-NEXT:  .LBB8_8:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v9
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v9
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_hi_u32 v2, v5, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v2, v9
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v2
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v5, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v3, v9
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v4, s[4:5], v3, v9
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v3, v4, vcc
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v2
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v3, v9
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v3, 0
-; CGP-NEXT:  .LBB8_8:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %shl.y = shl <2 x i64> <i64 4096, i64 4096>, %y
   %r = sdiv <2 x i64> %x, %shl.y
   ret <2 x i64> %r
 }
 
 define i64 @v_sdiv_i64_24bit(i64 %num, i64 %den) {

llvm/test/CodeGen/AMDGPU/GlobalISel/srem.i64.ll

 ; CHECK-NEXT:    v_mov_b32_e32 v5, v1
 ; CHECK-NEXT:    v_mov_b32_e32 v4, v0
 ; CHECK-NEXT:    v_or_b32_e32 v1, v5, v3
 ; CHECK-NEXT:    v_mov_b32_e32 v0, 0
 ; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CHECK-NEXT:    s_cbranch_execz .LBB0_2
-; CHECK-NEXT:  ; %bb.1:
+; CHECK-NEXT:    s_cbranch_execnz .LBB0_3
+; CHECK-NEXT:  ; %bb.1: ; %Flow
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CHECK-NEXT:    s_cbranch_execnz .LBB0_4
+; CHECK-NEXT:  .LBB0_2:
+; CHECK-NEXT:    s_or_b64 exec, exec, s[4:5]
+; CHECK-NEXT:    s_setpc_b64 s[30:31]
+; CHECK-NEXT:  .LBB0_3:
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v0, 31, v3
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v2, v0
 ; CHECK-NEXT:    v_addc_u32_e32 v2, vcc, v3, v0, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v3, v1, v0
 ; CHECK-NEXT:    v_xor_b32_e32 v6, v2, v0
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v3
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v1, v6
 ; CHECK-NEXT:    v_sub_i32_e32 v8, vcc, 0, v3

 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v2, v1, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, v0, v8
 ; CHECK-NEXT:    v_xor_b32_e32 v1, v1, v8
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v8
 ; CHECK-NEXT:    v_subb_u32_e32 v1, vcc, v1, v8, vcc
 ; CHECK-NEXT:    ; implicit-def: $vgpr2
 ; CHECK-NEXT:    ; implicit-def: $vgpr4
-; CHECK-NEXT:  .LBB0_2: ; %Flow
-; CHECK-NEXT:    s_or_saveexec_b64 s[4:5], s[6:7]
-; CHECK-NEXT:    s_xor_b64 exec, exec, s[4:5]
-; CHECK-NEXT:    s_cbranch_execz .LBB0_4
-; CHECK-NEXT:  ; %bb.3:
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CHECK-NEXT:    s_cbranch_execz .LBB0_2
+; CHECK-NEXT:  .LBB0_4:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v2
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, 0, v2
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v1, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v0, v4, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v0, v0, v2
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v4, v0
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v0, v2
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v0, v2
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CHECK-NEXT:    v_mov_b32_e32 v1, 0
-; CHECK-NEXT:  .LBB0_4:
 ; CHECK-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %result = srem i64 %num, %den
   ret i64 %result
 }
 
 ; FIXME: This is a workaround for not handling uniform VGPR case.
 declare i32 @llvm.amdgcn.readfirstlane(i32)

 ; CGP-NEXT:    v_cndmask_b32_e32 v1, v2, v1, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v0, v0, v12
 ; CGP-NEXT:    v_xor_b32_e32 v1, v1, v12
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v12
 ; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v12, vcc
 ; CGP-NEXT:    ; implicit-def: $vgpr4
 ; CGP-NEXT:    ; implicit-def: $vgpr10
 ; CGP-NEXT:  .LBB2_2: ; %Flow1
-; CGP-NEXT:    s_or_saveexec_b64 s[4:5], s[6:7]
-; CGP-NEXT:    s_xor_b64 exec, exec, s[4:5]
+; CGP-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
 ; CGP-NEXT:    s_cbranch_execz .LBB2_4
 ; CGP-NEXT:  ; %bb.3:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v0, v4
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, 0, v4
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_lo_u32 v1, v1, v0

 ; CGP-NEXT:  .LBB2_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CGP-NEXT:    v_or_b32_e32 v3, v9, v7
 ; CGP-NEXT:    v_mov_b32_e32 v2, 0
 ; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CGP-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CGP-NEXT:    s_cbranch_execz .LBB2_6
-; CGP-NEXT:  ; %bb.5:
+; CGP-NEXT:    s_cbranch_execnz .LBB2_7
+; CGP-NEXT:  ; %bb.5: ; %Flow
+; CGP-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CGP-NEXT:    s_cbranch_execnz .LBB2_8
+; CGP-NEXT:  .LBB2_6:
+; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
+; CGP-NEXT:    s_setpc_b64 s[30:31]
+; CGP-NEXT:  .LBB2_7:
 ; CGP-NEXT:    v_ashrrev_i32_e32 v2, 31, v7
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, v6, v2
 ; CGP-NEXT:    v_addc_u32_e32 v4, vcc, v7, v2, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v5, v3, v2
 ; CGP-NEXT:    v_xor_b32_e32 v6, v4, v2
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v5
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v3, v6
 ; CGP-NEXT:    v_sub_i32_e32 v10, vcc, 0, v5

 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v5, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v4, v3, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v2, v2, v10
 ; CGP-NEXT:    v_xor_b32_e32 v3, v3, v10
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
 ; CGP-NEXT:    v_subb_u32_e32 v3, vcc, v3, v10, vcc
 ; CGP-NEXT:    ; implicit-def: $vgpr6
 ; CGP-NEXT:    ; implicit-def: $vgpr8
-; CGP-NEXT:  .LBB2_6: ; %Flow
-; CGP-NEXT:    s_or_saveexec_b64 s[4:5], s[6:7]
-; CGP-NEXT:    s_xor_b64 exec, exec, s[4:5]
-; CGP-NEXT:    s_cbranch_execz .LBB2_8
-; CGP-NEXT:  ; %bb.7:
+; CGP-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CGP-NEXT:    s_cbranch_execz .LBB2_6
+; CGP-NEXT:  .LBB2_8:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v6
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v6
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_hi_u32 v2, v8, v2
 ; CGP-NEXT:    v_mul_lo_u32 v2, v2, v6
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v8, v2
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v2, v6
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v6
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v3, vcc
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v2, v6
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v6
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v3, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v3, 0
-; CGP-NEXT:  .LBB2_8:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = srem <2 x i64> %num, %den
   ret <2 x i64> %result
 }
 
 define i64 @v_srem_i64_pow2k_denom(i64 %num) {
 ; CHECK-LABEL: v_srem_i64_pow2k_denom:

 ; CHECK-NEXT:    v_mov_b32_e32 v4, v1
 ; CHECK-NEXT:    v_mov_b32_e32 v3, v0
 ; CHECK-NEXT:    v_or_b32_e32 v1, v4, v6
 ; CHECK-NEXT:    v_mov_b32_e32 v0, 0
 ; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CHECK-NEXT:    s_cbranch_execz .LBB7_2
-; CHECK-NEXT:  ; %bb.1:
+; CHECK-NEXT:    s_cbranch_execnz .LBB7_3
+; CHECK-NEXT:  ; %bb.1: ; %Flow
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CHECK-NEXT:    s_cbranch_execnz .LBB7_4
+; CHECK-NEXT:  .LBB7_2:
+; CHECK-NEXT:    s_or_b64 exec, exec, s[4:5]
+; CHECK-NEXT:    s_setpc_b64 s[30:31]
+; CHECK-NEXT:  .LBB7_3:
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v0, 31, v6
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v5, v0
 ; CHECK-NEXT:    v_addc_u32_e32 v2, vcc, v6, v0, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v5, v1, v0
 ; CHECK-NEXT:    v_xor_b32_e32 v6, v2, v0
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v5
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v1, v6
 ; CHECK-NEXT:    v_sub_i32_e32 v8, vcc, 0, v5

 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v2, v1, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, v0, v8
 ; CHECK-NEXT:    v_xor_b32_e32 v1, v1, v8
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v8
 ; CHECK-NEXT:    v_subb_u32_e32 v1, vcc, v1, v8, vcc
 ; CHECK-NEXT:    ; implicit-def: $vgpr5_vgpr6
 ; CHECK-NEXT:    ; implicit-def: $vgpr3
-; CHECK-NEXT:  .LBB7_2: ; %Flow
-; CHECK-NEXT:    s_or_saveexec_b64 s[4:5], s[6:7]
-; CHECK-NEXT:    s_xor_b64 exec, exec, s[4:5]
-; CHECK-NEXT:    s_cbranch_execz .LBB7_4
-; CHECK-NEXT:  ; %bb.3:
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CHECK-NEXT:    s_cbranch_execz .LBB7_2
+; CHECK-NEXT:  .LBB7_4:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v5
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, 0, v5
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v1, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v0, v3, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v0, v0, v5
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v3, v0
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v0, v5
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v5
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v0, v5
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v5
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CHECK-NEXT:    v_mov_b32_e32 v1, 0
-; CHECK-NEXT:  .LBB7_4:
 ; CHECK-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %shl.y = shl i64 4096, %y
   %r = srem i64 %x, %shl.y
   ret i64 %r
 }
 
 define <2 x i64> @v_srem_v2i64_pow2_shl_denom(<2 x i64> %x, <2 x i64> %y) {

 ; CGP-NEXT:  .LBB8_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CGP-NEXT:    v_or_b32_e32 v3, v7, v10
 ; CGP-NEXT:    v_mov_b32_e32 v2, 0
 ; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CGP-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CGP-NEXT:    s_cbranch_execz .LBB8_6
-; CGP-NEXT:  ; %bb.5:
+; CGP-NEXT:    s_cbranch_execnz .LBB8_7
+; CGP-NEXT:  ; %bb.5: ; %Flow
+; CGP-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CGP-NEXT:    s_cbranch_execnz .LBB8_8
+; CGP-NEXT:  .LBB8_6:
+; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
+; CGP-NEXT:    s_setpc_b64 s[30:31]
+; CGP-NEXT:  .LBB8_7:
 ; CGP-NEXT:    v_ashrrev_i32_e32 v2, 31, v10
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, v9, v2
 ; CGP-NEXT:    v_addc_u32_e32 v4, vcc, v10, v2, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v6, v3, v2
 ; CGP-NEXT:    v_xor_b32_e32 v8, v4, v2
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v6
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v3, v8
 ; CGP-NEXT:    v_sub_i32_e32 v10, vcc, 0, v6

 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v6, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v4, v3, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v2, v2, v10
 ; CGP-NEXT:    v_xor_b32_e32 v3, v3, v10
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
 ; CGP-NEXT:    v_subb_u32_e32 v3, vcc, v3, v10, vcc
 ; CGP-NEXT:    ; implicit-def: $vgpr9_vgpr10
 ; CGP-NEXT:    ; implicit-def: $vgpr5
-; CGP-NEXT:  .LBB8_6: ; %Flow
-; CGP-NEXT:    s_or_saveexec_b64 s[4:5], s[6:7]
-; CGP-NEXT:    s_xor_b64 exec, exec, s[4:5]
-; CGP-NEXT:    s_cbranch_execz .LBB8_8
-; CGP-NEXT:  ; %bb.7:
+; CGP-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CGP-NEXT:    s_cbranch_execz .LBB8_6
+; CGP-NEXT:  .LBB8_8:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v9
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v9
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_hi_u32 v2, v5, v2
 ; CGP-NEXT:    v_mul_lo_u32 v2, v2, v9
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v5, v2
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v2, v9
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v9
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v3, vcc
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v2, v9
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v9
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v3, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v3, 0
-; CGP-NEXT:  .LBB8_8:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %shl.y = shl <2 x i64> <i64 4096, i64 4096>, %y
   %r = srem <2 x i64> %x, %shl.y
   ret <2 x i64> %r
 }
 
 define i64 @v_srem_i64_24bit(i64 %num, i64 %den) {

llvm/test/CodeGen/AMDGPU/GlobalISel/udiv.i64.ll

 ; CHECK-NEXT:    v_mov_b32_e32 v4, v0
 ; CHECK-NEXT:    v_mov_b32_e32 v5, v1
 ; CHECK-NEXT:    v_or_b32_e32 v1, v5, v3
 ; CHECK-NEXT:    v_mov_b32_e32 v0, 0
 ; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CHECK-NEXT:    s_cbranch_execz .LBB0_2
-; CHECK-NEXT:  ; %bb.1:
+; CHECK-NEXT:    s_cbranch_execnz .LBB0_3
+; CHECK-NEXT:  ; %bb.1: ; %Flow
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CHECK-NEXT:    s_cbranch_execnz .LBB0_4
+; CHECK-NEXT:  .LBB0_2:
+; CHECK-NEXT:    s_or_b64 exec, exec, s[6:7]
+; CHECK-NEXT:    s_setpc_b64 s[30:31]
+; CHECK-NEXT:  .LBB0_3:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v2
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v1, v3
 ; CHECK-NEXT:    v_sub_i32_e32 v6, vcc, 0, v2
 ; CHECK-NEXT:    v_subb_u32_e32 v7, vcc, 0, v3, vcc
 ; CHECK-NEXT:    v_mac_f32_e32 v0, 0x4f800000, v1
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x5f7ffffc, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v1, 0x2f800000, v0

 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
 ; CHECK-NEXT:    v_cndmask_b32_e32 v2, v10, v8, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v3, v11, v12, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v6
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
 ; CHECK-NEXT:    ; implicit-def: $vgpr2
 ; CHECK-NEXT:    ; implicit-def: $vgpr4
-; CHECK-NEXT:  .LBB0_2: ; %Flow
-; CHECK-NEXT:    s_or_saveexec_b64 s[6:7], s[6:7]
-; CHECK-NEXT:    s_xor_b64 exec, exec, s[6:7]
-; CHECK-NEXT:    s_cbranch_execz .LBB0_4
-; CHECK-NEXT:  ; %bb.3:
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CHECK-NEXT:    s_cbranch_execz .LBB0_2
+; CHECK-NEXT:  .LBB0_4:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v2
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, 0, v2
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v1, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v0, v4, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v0, v2
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, 1, v0
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v4, v1
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v2
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
 ; CHECK-NEXT:    v_sub_i32_e64 v3, s[4:5], v1, v2
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, 1, v0
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v2
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
 ; CHECK-NEXT:    v_mov_b32_e32 v1, 0
-; CHECK-NEXT:  .LBB0_4:
 ; CHECK-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %result = udiv i64 %num, %den
   ret i64 %result
 }
 
 ; FIXME: This is a workaround for not handling uniform VGPR case.
 declare i32 @llvm.amdgcn.readfirstlane(i32)

 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v14, v12, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v15, v16, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v10
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
 ; CGP-NEXT:    ; implicit-def: $vgpr4
 ; CGP-NEXT:    ; implicit-def: $vgpr10
 ; CGP-NEXT:  .LBB2_2: ; %Flow1
-; CGP-NEXT:    s_or_saveexec_b64 s[6:7], s[6:7]
-; CGP-NEXT:    s_xor_b64 exec, exec, s[6:7]
+; CGP-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
 ; CGP-NEXT:    s_cbranch_execz .LBB2_4
 ; CGP-NEXT:  ; %bb.3:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v0, v4
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, 0, v4
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_lo_u32 v1, v1, v0

 ; CGP-NEXT:  .LBB2_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CGP-NEXT:    v_or_b32_e32 v3, v9, v7
 ; CGP-NEXT:    v_mov_b32_e32 v2, 0
 ; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CGP-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CGP-NEXT:    s_cbranch_execz .LBB2_6
-; CGP-NEXT:  ; %bb.5:
+; CGP-NEXT:    s_cbranch_execnz .LBB2_7
+; CGP-NEXT:  ; %bb.5: ; %Flow
+; CGP-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CGP-NEXT:    s_cbranch_execnz .LBB2_8
+; CGP-NEXT:  .LBB2_6:
+; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
+; CGP-NEXT:    s_setpc_b64 s[30:31]
+; CGP-NEXT:  .LBB2_7:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v6
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v3, v7
 ; CGP-NEXT:    v_sub_i32_e32 v4, vcc, 0, v6
 ; CGP-NEXT:    v_subb_u32_e32 v5, vcc, 0, v7, vcc
 ; CGP-NEXT:    v_mac_f32_e32 v2, 0x4f800000, v3
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x5f7ffffc, v2
 ; CGP-NEXT:    v_mul_f32_e32 v3, 0x2f800000, v2

 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
 ; CGP-NEXT:    v_cndmask_b32_e32 v4, v12, v10, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v5, v13, v14, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v8
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc
 ; CGP-NEXT:    ; implicit-def: $vgpr6
 ; CGP-NEXT:    ; implicit-def: $vgpr8
-; CGP-NEXT:  .LBB2_6: ; %Flow
-; CGP-NEXT:    s_or_saveexec_b64 s[6:7], s[6:7]
-; CGP-NEXT:    s_xor_b64 exec, exec, s[6:7]
-; CGP-NEXT:    s_cbranch_execz .LBB2_8
-; CGP-NEXT:  ; %bb.7:
+; CGP-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CGP-NEXT:    s_cbranch_execz .LBB2_6
+; CGP-NEXT:  .LBB2_8:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v6
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v6
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_hi_u32 v2, v8, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v2, v6
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v2
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v8, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v3, v6
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v4, s[4:5], v3, v6
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v3, v4, vcc
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v2
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v3, v6
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v3, 0
-; CGP-NEXT:  .LBB2_8:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = udiv <2 x i64> %num, %den
   ret <2 x i64> %result
 }
 
 define i64 @v_udiv_i64_pow2k_denom(i64 %num) {
 ; CHECK-LABEL: v_udiv_i64_pow2k_denom:

 ; CHECK-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; CHECK-NEXT:    v_lshl_b64 v[5:6], s[4:5], v2
 ; CHECK-NEXT:    v_or_b32_e32 v1, v4, v6
 ; CHECK-NEXT:    v_mov_b32_e32 v0, 0
 ; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CHECK-NEXT:    s_cbranch_execz .LBB7_2
-; CHECK-NEXT:  ; %bb.1:
+; CHECK-NEXT:    s_cbranch_execnz .LBB7_3
+; CHECK-NEXT:  ; %bb.1: ; %Flow
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CHECK-NEXT:    s_cbranch_execnz .LBB7_4
+; CHECK-NEXT:  .LBB7_2:
+; CHECK-NEXT:    s_or_b64 exec, exec, s[6:7]
+; CHECK-NEXT:    s_setpc_b64 s[30:31]
+; CHECK-NEXT:  .LBB7_3:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v5
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v1, v6
 ; CHECK-NEXT:    v_sub_i32_e32 v2, vcc, 0, v5
 ; CHECK-NEXT:    v_subb_u32_e32 v7, vcc, 0, v6, vcc
 ; CHECK-NEXT:    v_mac_f32_e32 v0, 0x4f800000, v1
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x5f7ffffc, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v1, 0x2f800000, v0

 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
 ; CHECK-NEXT:    v_cndmask_b32_e32 v2, v10, v8, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v3, v11, v12, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
 ; CHECK-NEXT:    ; implicit-def: $vgpr5_vgpr6
 ; CHECK-NEXT:    ; implicit-def: $vgpr3
-; CHECK-NEXT:  .LBB7_2: ; %Flow
-; CHECK-NEXT:    s_or_saveexec_b64 s[6:7], s[6:7]
-; CHECK-NEXT:    s_xor_b64 exec, exec, s[6:7]
-; CHECK-NEXT:    s_cbranch_execz .LBB7_4
-; CHECK-NEXT:  ; %bb.3:
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CHECK-NEXT:    s_cbranch_execz .LBB7_2
+; CHECK-NEXT:  .LBB7_4:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v5
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, 0, v5
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v1, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v0, v3, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v0, v5
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v3, v1
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v5
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CHECK-NEXT:    v_sub_i32_e64 v2, s[4:5], v1, v5
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v2, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v5
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CHECK-NEXT:    v_mov_b32_e32 v1, 0
-; CHECK-NEXT:  .LBB7_4:
 ; CHECK-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %shl.y = shl i64 4096, %y
   %r = udiv i64 %x, %shl.y
   ret i64 %r
 }
 
 define <2 x i64> @v_udiv_v2i64_pow2_shl_denom(<2 x i64> %x, <2 x i64> %y) {

 ; CGP-NEXT:  .LBB8_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[8:9]
 ; CGP-NEXT:    v_or_b32_e32 v3, v7, v10
 ; CGP-NEXT:    v_mov_b32_e32 v2, 0
 ; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CGP-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CGP-NEXT:    s_cbranch_execz .LBB8_6
-; CGP-NEXT:  ; %bb.5:
+; CGP-NEXT:    s_cbranch_execnz .LBB8_7
+; CGP-NEXT:  ; %bb.5: ; %Flow
+; CGP-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CGP-NEXT:    s_cbranch_execnz .LBB8_8
+; CGP-NEXT:  .LBB8_6:
+; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
+; CGP-NEXT:    s_setpc_b64 s[30:31]
+; CGP-NEXT:  .LBB8_7:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v9
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v3, v10
 ; CGP-NEXT:    v_sub_i32_e32 v4, vcc, 0, v9
 ; CGP-NEXT:    v_subb_u32_e32 v6, vcc, 0, v10, vcc
 ; CGP-NEXT:    v_mac_f32_e32 v2, 0x4f800000, v3
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x5f7ffffc, v2
 ; CGP-NEXT:    v_mul_f32_e32 v3, 0x2f800000, v2

 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
 ; CGP-NEXT:    v_cndmask_b32_e32 v4, v12, v8, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v5, v13, v14, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v6
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc
 ; CGP-NEXT:    ; implicit-def: $vgpr9_vgpr10
 ; CGP-NEXT:    ; implicit-def: $vgpr5
-; CGP-NEXT:  .LBB8_6: ; %Flow
-; CGP-NEXT:    s_or_saveexec_b64 s[6:7], s[6:7]
-; CGP-NEXT:    s_xor_b64 exec, exec, s[6:7]
-; CGP-NEXT:    s_cbranch_execz .LBB8_8
-; CGP-NEXT:  ; %bb.7:
+; CGP-NEXT:    s_andn2_saveexec_b64 s[6:7], s[6:7]
+; CGP-NEXT:    s_cbranch_execz .LBB8_6
+; CGP-NEXT:  .LBB8_8:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v9
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v9
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_hi_u32 v2, v5, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v2, v9
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v2
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v5, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v3, v9
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v4, s[4:5], v3, v9
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v3, v4, vcc
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v2
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v3, v9
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v3, 0
-; CGP-NEXT:  .LBB8_8:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %shl.y = shl <2 x i64> <i64 4096, i64 4096>, %y
   %r = udiv <2 x i64> %x, %shl.y
   ret <2 x i64> %r
 }
 
 define i64 @v_udiv_i64_24bit(i64 %num, i64 %den) {

llvm/test/CodeGen/AMDGPU/GlobalISel/urem.i64.ll

 ; CHECK-NEXT:    v_mov_b32_e32 v4, v0
 ; CHECK-NEXT:    v_mov_b32_e32 v5, v1
 ; CHECK-NEXT:    v_or_b32_e32 v1, v5, v3
 ; CHECK-NEXT:    v_mov_b32_e32 v0, 0
 ; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CHECK-NEXT:    s_cbranch_execz .LBB0_2
-; CHECK-NEXT:  ; %bb.1:
+; CHECK-NEXT:    s_cbranch_execnz .LBB0_3
+; CHECK-NEXT:  ; %bb.1: ; %Flow
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CHECK-NEXT:    s_cbranch_execnz .LBB0_4
+; CHECK-NEXT:  .LBB0_2:
+; CHECK-NEXT:    s_or_b64 exec, exec, s[4:5]
+; CHECK-NEXT:    s_setpc_b64 s[30:31]
+; CHECK-NEXT:  .LBB0_3:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v2
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v1, v3
 ; CHECK-NEXT:    v_sub_i32_e32 v6, vcc, 0, v2
 ; CHECK-NEXT:    v_subb_u32_e32 v7, vcc, 0, v3, vcc
 ; CHECK-NEXT:    v_mac_f32_e32 v0, 0x4f800000, v1
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x5f7ffffc, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v1, 0x2f800000, v0

 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v3
 ; CHECK-NEXT:    v_cndmask_b32_e32 v2, v6, v2, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v3, v7, v0, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v5
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v1, v2, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v4, v3, vcc
 ; CHECK-NEXT:    ; implicit-def: $vgpr2
 ; CHECK-NEXT:    ; implicit-def: $vgpr4
-; CHECK-NEXT:  .LBB0_2: ; %Flow
-; CHECK-NEXT:    s_or_saveexec_b64 s[4:5], s[6:7]
-; CHECK-NEXT:    s_xor_b64 exec, exec, s[4:5]
-; CHECK-NEXT:    s_cbranch_execz .LBB0_4
-; CHECK-NEXT:  ; %bb.3:
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CHECK-NEXT:    s_cbranch_execz .LBB0_2
+; CHECK-NEXT:  .LBB0_4:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v2
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, 0, v2
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v1, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v0, v4, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v0, v0, v2
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v4, v0
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v0, v2
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v0, v2
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CHECK-NEXT:    v_mov_b32_e32 v1, 0
-; CHECK-NEXT:  .LBB0_4:
 ; CHECK-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %result = urem i64 %num, %den
   ret i64 %result
 }
 
 ; FIXME: This is a workaround for not handling uniform VGPR case.
 declare i32 @llvm.amdgcn.readfirstlane(i32)

 ; CGP-NEXT:    v_cndmask_b32_e32 v4, v10, v4, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v5, v11, v0, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v3
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v1, v4, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v1, v2, v5, vcc
 ; CGP-NEXT:    ; implicit-def: $vgpr4
 ; CGP-NEXT:    ; implicit-def: $vgpr10
 ; CGP-NEXT:  .LBB2_2: ; %Flow1
-; CGP-NEXT:    s_or_saveexec_b64 s[4:5], s[6:7]
-; CGP-NEXT:    s_xor_b64 exec, exec, s[4:5]
+; CGP-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
 ; CGP-NEXT:    s_cbranch_execz .LBB2_4
 ; CGP-NEXT:  ; %bb.3:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v0, v4
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, 0, v4
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_lo_u32 v1, v1, v0

 ; CGP-NEXT:  .LBB2_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CGP-NEXT:    v_or_b32_e32 v3, v9, v7
 ; CGP-NEXT:    v_mov_b32_e32 v2, 0
 ; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CGP-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CGP-NEXT:    s_cbranch_execz .LBB2_6
-; CGP-NEXT:  ; %bb.5:
+; CGP-NEXT:    s_cbranch_execnz .LBB2_7
+; CGP-NEXT:  ; %bb.5: ; %Flow
+; CGP-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CGP-NEXT:    s_cbranch_execnz .LBB2_8
+; CGP-NEXT:  .LBB2_6:
+; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
+; CGP-NEXT:    s_setpc_b64 s[30:31]
+; CGP-NEXT:  .LBB2_7:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v6
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v3, v7
 ; CGP-NEXT:    v_sub_i32_e32 v4, vcc, 0, v6
 ; CGP-NEXT:    v_subb_u32_e32 v5, vcc, 0, v7, vcc
 ; CGP-NEXT:    v_mac_f32_e32 v2, 0x4f800000, v3
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x5f7ffffc, v2
 ; CGP-NEXT:    v_mul_f32_e32 v3, 0x2f800000, v2

 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v7
 ; CGP-NEXT:    v_cndmask_b32_e32 v6, v8, v6, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v7, v9, v2, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v5
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v3, v6, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v4, v7, vcc
 ; CGP-NEXT:    ; implicit-def: $vgpr6
 ; CGP-NEXT:    ; implicit-def: $vgpr8
-; CGP-NEXT:  .LBB2_6: ; %Flow
-; CGP-NEXT:    s_or_saveexec_b64 s[4:5], s[6:7]
-; CGP-NEXT:    s_xor_b64 exec, exec, s[4:5]
-; CGP-NEXT:    s_cbranch_execz .LBB2_8
-; CGP-NEXT:  ; %bb.7:
+; CGP-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CGP-NEXT:    s_cbranch_execz .LBB2_6
+; CGP-NEXT:  .LBB2_8:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v6
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v6
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_hi_u32 v2, v8, v2
 ; CGP-NEXT:    v_mul_lo_u32 v2, v2, v6
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v8, v2
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v2, v6
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v6
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v3, vcc
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v2, v6
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v6
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v3, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v3, 0
-; CGP-NEXT:  .LBB2_8:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = urem <2 x i64> %num, %den
   ret <2 x i64> %result
 }
 
 define i64 @v_urem_i64_pow2k_denom(i64 %num) {
 ; CHECK-LABEL: v_urem_i64_pow2k_denom:

 ; CHECK-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; CHECK-NEXT:    v_lshl_b64 v[5:6], s[4:5], v2
 ; CHECK-NEXT:    v_or_b32_e32 v1, v4, v6
 ; CHECK-NEXT:    v_mov_b32_e32 v0, 0
 ; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CHECK-NEXT:    s_cbranch_execz .LBB7_2
-; CHECK-NEXT:  ; %bb.1:
+; CHECK-NEXT:    s_cbranch_execnz .LBB7_3
+; CHECK-NEXT:  ; %bb.1: ; %Flow
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CHECK-NEXT:    s_cbranch_execnz .LBB7_4
+; CHECK-NEXT:  .LBB7_2:
+; CHECK-NEXT:    s_or_b64 exec, exec, s[4:5]
+; CHECK-NEXT:    s_setpc_b64 s[30:31]
+; CHECK-NEXT:  .LBB7_3:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v5
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v1, v6
 ; CHECK-NEXT:    v_sub_i32_e32 v2, vcc, 0, v5
 ; CHECK-NEXT:    v_subb_u32_e32 v7, vcc, 0, v6, vcc
 ; CHECK-NEXT:    v_mac_f32_e32 v0, 0x4f800000, v1
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x5f7ffffc, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v1, 0x2f800000, v0

 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v6
 ; CHECK-NEXT:    v_cndmask_b32_e32 v4, v4, v5, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v5, v7, v0, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v3
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v1, v4, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v2, v5, vcc
 ; CHECK-NEXT:    ; implicit-def: $vgpr5_vgpr6
 ; CHECK-NEXT:    ; implicit-def: $vgpr3
-; CHECK-NEXT:  .LBB7_2: ; %Flow
-; CHECK-NEXT:    s_or_saveexec_b64 s[4:5], s[6:7]
-; CHECK-NEXT:    s_xor_b64 exec, exec, s[4:5]
-; CHECK-NEXT:    s_cbranch_execz .LBB7_4
-; CHECK-NEXT:  ; %bb.3:
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CHECK-NEXT:    s_cbranch_execz .LBB7_2
+; CHECK-NEXT:  .LBB7_4:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, v5
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, 0, v5
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v1, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v0, v3, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v0, v0, v5
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v3, v0
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v0, v5
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v5
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, v0, v5
 ; CHECK-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v5
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CHECK-NEXT:    v_mov_b32_e32 v1, 0
-; CHECK-NEXT:  .LBB7_4:
 ; CHECK-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %shl.y = shl i64 4096, %y
   %r = urem i64 %x, %shl.y
   ret i64 %r
 }
 
 define <2 x i64> @v_urem_v2i64_pow2_shl_denom(<2 x i64> %x, <2 x i64> %y) {

 ; CGP-NEXT:  .LBB8_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CGP-NEXT:    v_or_b32_e32 v3, v7, v10
 ; CGP-NEXT:    v_mov_b32_e32 v2, 0
 ; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CGP-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
-; CGP-NEXT:    s_cbranch_execz .LBB8_6
-; CGP-NEXT:  ; %bb.5:
+; CGP-NEXT:    s_cbranch_execnz .LBB8_7
+; CGP-NEXT:  ; %bb.5: ; %Flow
+; CGP-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CGP-NEXT:    s_cbranch_execnz .LBB8_8
+; CGP-NEXT:  .LBB8_6:
+; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
+; CGP-NEXT:    s_setpc_b64 s[30:31]
+; CGP-NEXT:  .LBB8_7:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v9
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v3, v10
 ; CGP-NEXT:    v_sub_i32_e32 v4, vcc, 0, v9
 ; CGP-NEXT:    v_subb_u32_e32 v6, vcc, 0, v10, vcc
 ; CGP-NEXT:    v_mac_f32_e32 v2, 0x4f800000, v3
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x5f7ffffc, v2
 ; CGP-NEXT:    v_mul_f32_e32 v3, 0x2f800000, v2

 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v8
 ; CGP-NEXT:    v_cndmask_b32_e32 v6, v6, v9, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v7, v7, v2, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v5
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v3, v6, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v4, v7, vcc
 ; CGP-NEXT:    ; implicit-def: $vgpr9_vgpr10
 ; CGP-NEXT:    ; implicit-def: $vgpr5
-; CGP-NEXT:  .LBB8_6: ; %Flow
-; CGP-NEXT:    s_or_saveexec_b64 s[4:5], s[6:7]
-; CGP-NEXT:    s_xor_b64 exec, exec, s[4:5]
-; CGP-NEXT:    s_cbranch_execz .LBB8_8
-; CGP-NEXT:  ; %bb.7:
+; CGP-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
+; CGP-NEXT:    s_cbranch_execz .LBB8_6
+; CGP-NEXT:  .LBB8_8:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v9
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v9
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_hi_u32 v2, v5, v2
 ; CGP-NEXT:    v_mul_lo_u32 v2, v2, v9
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v5, v2
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v2, v9
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v9
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v3, vcc
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v2, v9
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v9
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v3, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v3, 0
-; CGP-NEXT:  .LBB8_8:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %shl.y = shl <2 x i64> <i64 4096, i64 4096>, %y
   %r = urem <2 x i64> %x, %shl.y
   ret <2 x i64> %r
 }
 
 define i64 @v_urem_i64_24bit(i64 %num, i64 %den) {

llvm/test/CodeGen/AMDGPU/branch-relaxation.ll

 
 ; si_mask_branch
 
 ; GCN-LABEL: {{^}}analyze_mask_branch:
 ; GCN: v_cmp_nlt_f32_e32 vcc
 ; GCN-NEXT: s_and_saveexec_b64 [[TEMP_MASK:s\[[0-9]+:[0-9]+\]]], vcc
 ; GCN-NEXT: s_xor_b64  [[MASK:s\[[0-9]+:[0-9]+\]]], exec, [[TEMP_MASK]]
 
-; GCN: BB{{[0-9]+_[0-9]+}}: ; %Flow
-; GCN-NEXT: s_or_saveexec_b64 [[TEMP_MASK1:s\[[0-9]+:[0-9]+\]]], [[MASK]]
-; GCN-NEXT: s_xor_b64 exec, exec, [[TEMP_MASK1]]
+; GCN: .LBB{{[0-9]+_[0-9]+}}: ; %Flow1
+; GCN-NEXT: s_andn2_saveexec_b64 [[MASK]], [[MASK]]
+; GCN-NEXT: s_cbranch_execnz
 
 ; GCN: .L[[LOOP_BODY:BB[0-9]+_[0-9]+]]: ; %loop{{$}}
 ; GCN: ;;#ASMSTART
 ; GCN: v_nop_e64
 ; GCN: v_nop_e64
 ; GCN: v_nop_e64
 ; GCN: v_nop_e64
 ; GCN: v_nop_e64

llvm/test/CodeGen/AMDGPU/bypass-div.ll

 ; GFX9-NEXT:    v_cndmask_b32_e32 v0, v3, v6, vcc
 ; GFX9-NEXT:    v_xor_b32_e32 v1, v1, v2
 ; GFX9-NEXT:    v_xor_b32_e32 v0, v0, v2
 ; GFX9-NEXT:    v_sub_co_u32_e32 v4, vcc, v1, v2
 ; GFX9-NEXT:    v_subb_co_u32_e32 v5, vcc, v0, v2, vcc
 ; GFX9-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; GFX9-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; GFX9-NEXT:  .LBB0_2: ; %Flow
-; GFX9-NEXT:    s_or_saveexec_b64 s[4:5], s[6:7]
-; GFX9-NEXT:    s_xor_b64 exec, exec, s[4:5]
+; GFX9-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
 ; GFX9-NEXT:    s_cbranch_execz .LBB0_4
 ; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v1, v2
 ; GFX9-NEXT:    v_sub_u32_e32 v3, 0, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v5, 0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v1, v1
 ; GFX9-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v1
 ; GFX9-NEXT:    v_cvt_u32_f32_e32 v1, v1

 ; GFX9-NEXT:    v_cndmask_b32_e64 v4, v11, v9, s[4:5]
 ; GFX9-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
 ; GFX9-NEXT:    v_cndmask_b32_e64 v0, v10, v8, s[4:5]
 ; GFX9-NEXT:    v_cndmask_b32_e32 v5, v7, v4, vcc
 ; GFX9-NEXT:    v_cndmask_b32_e32 v4, v6, v0, vcc
 ; GFX9-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; GFX9-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; GFX9-NEXT:  .LBB1_2: ; %Flow
-; GFX9-NEXT:    s_or_saveexec_b64 s[4:5], s[6:7]
-; GFX9-NEXT:    s_xor_b64 exec, exec, s[4:5]
+; GFX9-NEXT:    s_andn2_saveexec_b64 s[4:5], s[6:7]
 ; GFX9-NEXT:    s_cbranch_execz .LBB1_4
 ; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v1, v2
 ; GFX9-NEXT:    v_sub_u32_e32 v3, 0, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v5, 0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v1, v1
 ; GFX9-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v1
 ; GFX9-NEXT:    v_cvt_u32_f32_e32 v1, v1

 ; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX9-NEXT:    v_xor_b32_e32 v0, v0, v5
 ; GFX9-NEXT:    v_xor_b32_e32 v1, v1, v5
 ; GFX9-NEXT:    v_sub_co_u32_e32 v4, vcc, v0, v5
 ; GFX9-NEXT:    v_subb_co_u32_e32 v5, vcc, v1, v5, vcc
 ; GFX9-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; GFX9-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; GFX9-NEXT:  .LBB2_2: ; %Flow
-; GFX9-NEXT:    s_or_saveexec_b64 s[4:5], s[8:9]
-; GFX9-NEXT:    s_xor_b64 exec, exec, s[4:5]
+; GFX9-NEXT:    s_andn2_saveexec_b64 s[4:5], s[8:9]
 ; GFX9-NEXT:    s_cbranch_execz .LBB2_4
 ; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v1, v2
 ; GFX9-NEXT:    v_sub_u32_e32 v3, 0, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v5, 0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v1, v1
 ; GFX9-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v1
 ; GFX9-NEXT:    v_cvt_u32_f32_e32 v1, v1

 ; GFX9-NEXT:    v_cndmask_b32_e64 v4, v7, v4, s[4:5]
 ; GFX9-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
 ; GFX9-NEXT:    v_cndmask_b32_e32 v5, v1, v4, vcc
 ; GFX9-NEXT:    v_cndmask_b32_e64 v1, v6, v9, s[4:5]
 ; GFX9-NEXT:    v_cndmask_b32_e32 v4, v0, v1, vcc
 ; GFX9-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; GFX9-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; GFX9-NEXT:  .LBB3_2: ; %Flow
-; GFX9-NEXT:    s_or_saveexec_b64 s[4:5], s[8:9]
-; GFX9-NEXT:    s_xor_b64 exec, exec, s[4:5]
+; GFX9-NEXT:    s_andn2_saveexec_b64 s[4:5], s[8:9]
 ; GFX9-NEXT:    s_cbranch_execz .LBB3_4
 ; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v1, v2
 ; GFX9-NEXT:    v_sub_u32_e32 v3, 0, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v5, 0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v1, v1
 ; GFX9-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v1
 ; GFX9-NEXT:    v_cvt_u32_f32_e32 v1, v1

 ; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX9-NEXT:    v_xor_b32_e32 v0, v0, v7
 ; GFX9-NEXT:    v_xor_b32_e32 v1, v1, v7
 ; GFX9-NEXT:    v_sub_co_u32_e32 v6, vcc, v0, v7
 ; GFX9-NEXT:    v_subb_co_u32_e32 v7, vcc, v1, v7, vcc
 ; GFX9-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; GFX9-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; GFX9-NEXT:  .LBB8_2: ; %Flow
-; GFX9-NEXT:    s_or_saveexec_b64 s[4:5], s[10:11]
-; GFX9-NEXT:    s_xor_b64 exec, exec, s[4:5]
+; GFX9-NEXT:    s_andn2_saveexec_b64 s[4:5], s[10:11]
 ; GFX9-NEXT:    s_cbranch_execz .LBB8_4
 ; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v1, v2
 ; GFX9-NEXT:    v_sub_u32_e32 v3, 0, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v5, 0
 ; GFX9-NEXT:    v_mov_b32_e32 v7, v5
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v1, v1
 ; GFX9-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v1

 ; GFX9-NEXT:    v_cndmask_b32_e64 v4, v13, v11, s[6:7]
 ; GFX9-NEXT:    v_cndmask_b32_e32 v7, v1, v3, vcc
 ; GFX9-NEXT:    v_cndmask_b32_e64 v1, v9, v2, s[6:7]
 ; GFX9-NEXT:    v_cndmask_b32_e32 v4, v6, v4, vcc
 ; GFX9-NEXT:    v_cndmask_b32_e32 v6, v0, v1, vcc
 ; GFX9-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; GFX9-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; GFX9-NEXT:  .LBB9_2: ; %Flow
-; GFX9-NEXT:    s_or_saveexec_b64 s[4:5], s[8:9]
-; GFX9-NEXT:    s_xor_b64 exec, exec, s[4:5]
+; GFX9-NEXT:    s_andn2_saveexec_b64 s[4:5], s[8:9]
 ; GFX9-NEXT:    s_cbranch_execz .LBB9_4
 ; GFX9-NEXT:  ; %bb.3:
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v1, v2
 ; GFX9-NEXT:    v_sub_u32_e32 v3, 0, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v5, 0
 ; GFX9-NEXT:    v_mov_b32_e32 v7, v5
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v1, v1
 ; GFX9-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v1

llvm/test/CodeGen/AMDGPU/collapse-endcf.ll

 ; GCN:      s_and_saveexec_b64 [[SAVEEXEC_OUTER:s\[[0-9:]+\]]]
 ; GCN-NEXT: s_cbranch_execz [[ENDIF_OUTER:.LBB[0-9_]+]]
 ; GCN:      s_and_saveexec_b64 [[SAVEEXEC_INNER:s\[[0-9:]+\]]]
 ; GCN-NEXT: s_xor_b64 [[SAVEEXEC_INNER2:s\[[0-9:]+\]]], exec, [[SAVEEXEC_INNER]]
 ; GCN-NEXT: s_cbranch_execz [[THEN_INNER:.LBB[0-9_]+]]
 ; GCN-NEXT: ; %bb.{{[0-9]+}}:
 ; GCN:      store_dword
 ; GCN:      {{^}}[[THEN_INNER]]:
-; GCN-NEXT: s_or_saveexec_b64 [[SAVEEXEC_INNER3:s\[[0-9:]+\]]], [[SAVEEXEC_INNER2]]
-; GCN-NEXT: s_xor_b64 exec, exec, [[SAVEEXEC_INNER3]]
+; GCN-NEXT: s_andn2_saveexec_b64 [[SAVEEXEC_INNER2]], [[SAVEEXEC_INNER2]]
 ; GCN-NEXT: s_cbranch_execz [[ENDIF_OUTER]]
 ; GCN:      store_dword
 ; GCN-NEXT: {{^}}[[ENDIF_OUTER]]:
 ; GCN-NEXT: s_or_b64 exec, exec, [[SAVEEXEC_OUTER]]
 ; GCN: ds_write_b32
 ; GCN: s_endpgm
 ;
 ; GCN-O0-LABEL: {{^}}nested_if_if_else:

 ; GCN:      store_dword
 ; GCN-NEXT: s_and_saveexec_b64 [[SAVEEXEC_INNER_IF_OUTER_ELSE:s\[[0-9:]+\]]]
 ; GCN-NEXT: s_cbranch_execz [[THEN_OUTER_FLOW:.LBB[0-9_]+]]
 ; GCN-NEXT: ; %bb.{{[0-9]+}}:
 ; GCN:      store_dword
 ; GCN-NEXT: {{^}}[[THEN_OUTER_FLOW]]:
 ; GCN-NEXT: s_or_b64 exec, exec, [[SAVEEXEC_INNER_IF_OUTER_ELSE]]
 ; GCN:      {{^}}[[THEN_OUTER]]:
-; GCN-NEXT: s_or_saveexec_b64 [[SAVEEXEC_OUTER3:s\[[0-9:]+\]]], [[SAVEEXEC_OUTER2]]
-; GCN-NEXT: s_xor_b64 exec, exec, [[SAVEEXEC_OUTER3]]
+; GCN-NEXT: s_andn2_saveexec_b64 [[SAVEEXEC_OUTER2]], [[SAVEEXEC_OUTER2]]
 ; GCN-NEXT: s_cbranch_execz [[ENDIF_OUTER:.LBB[0-9_]+]]
 ; GCN-NEXT: ; %bb.{{[0-9]+}}:
 ; GCN:      store_dword
 ; GCN-NEXT: s_and_saveexec_b64 [[SAVEEXEC_ELSE:s\[[0-9:]+\]]],
 ; GCN-NEXT: s_cbranch_execz [[FLOW1:.LBB[0-9_]+]]
 ; GCN-NEXT: ; %bb.{{[0-9]+}}:
 ; GCN:      store_dword
 ; GCN-NEXT: [[FLOW1]]:
 ; GCN-NEXT: s_or_b64 exec, exec, [[SAVEEXEC_ELSE]]
-; GCN:      s_or_b64 exec, exec, [[SAVEEXEC_OUTER3]]
+; GCN:      s_or_b64 exec, exec, [[SAVEEXEC_OUTER2]]
 ; GCN:      ds_write_b32
 ; GCN:      s_endpgm
 ;
 ; GCN-O0-LABEL: {{^}}nested_if_else_if:
 ; GCN-O0:      s_mov_b64 s[{{[0-9:]+}}], exec
 ; GCN-O0-NEXT: s_and_b64 s[{{[0-9:]+}}], s[{{[0-9:]+}}], s[{{[0-9:]+}}]
 ; GCN-O0-NEXT: s_xor_b64 s[{{[0-9:]+}}], s[{{[0-9:]+}}], s[{{[0-9:]+}}]
 ; GCN-O0-DAG:  v_writelane_b32 [[VGPR:v[0-9]+]], s{{[0-9]+}}, [[OUTER_SPILL_LANE_0:[0-9]+]]

llvm/test/CodeGen/AMDGPU/else.ll

 ; RUN: llc -march=amdgcn -mcpu=verde -verify-machineinstrs < %s | FileCheck %s
 ; RUN: llc -march=amdgcn -mcpu=tonga -verify-machineinstrs < %s | FileCheck %s
 
 ; CHECK-LABEL: {{^}}else_no_execfix:
 ; CHECK: ; %Flow
-; CHECK-NEXT: s_or_saveexec_b64 [[DST:s\[[0-9]+:[0-9]+\]]],
-; CHECK-NEXT: s_xor_b64 exec, exec, [[DST]]
+; CHECK-NEXT: s_andn2_saveexec_b64 [[DST:s\[[0-9]+:[0-9]+\]]], [[DST]]
 define amdgpu_ps float @else_no_execfix(i32 %z, float %v) #0 {
 main_body:
   %cc = icmp sgt i32 %z, 5
   br i1 %cc, label %if, label %else
 
 if:
   %v.if = fmul float %v, 2.0
   br label %end

llvm/test/CodeGen/AMDGPU/kill-infinite-loop.ll

 ; CHECK-NEXT:    s_andn2_b64 s[0:1], s[0:1], exec
 ; CHECK-NEXT:    s_cbranch_scc0 .LBB0_6
 ; CHECK-NEXT:  ; %bb.2: ; %loop
 ; CHECK-NEXT:    ; in Loop: Header=BB0_1 Depth=1
 ; CHECK-NEXT:    s_mov_b64 exec, 0
 ; CHECK-NEXT:    s_mov_b64 vcc, 0
 ; CHECK-NEXT:    s_branch .LBB0_1
 ; CHECK-NEXT:  .LBB0_3: ; %Flow1
-; CHECK-NEXT:    s_or_saveexec_b64 s[0:1], s[2:3]
-; CHECK-NEXT:    s_xor_b64 exec, exec, s[0:1]
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[0:1], s[2:3]
 ; CHECK-NEXT:    s_cbranch_execz .LBB0_5
 ; CHECK-NEXT:  ; %bb.4: ; %end
 ; CHECK-NEXT:    v_mov_b32_e32 v0, 1.0
 ; CHECK-NEXT:    v_mov_b32_e32 v1, 0
 ; CHECK-NEXT:    exp mrt0 v1, v1, v1, v0 done vm
 ; CHECK-NEXT:  .LBB0_5: ; %UnifiedReturnBlock
 ; CHECK-NEXT:    s_endpgm
 ; CHECK-NEXT:  .LBB0_6:

 ; CHECK-NEXT:    s_andn2_b64 s[0:1], s[0:1], exec
 ; CHECK-NEXT:    s_cbranch_scc0 .LBB1_6
 ; CHECK-NEXT:  ; %bb.2: ; %loop
 ; CHECK-NEXT:    ; in Loop: Header=BB1_1 Depth=1
 ; CHECK-NEXT:    s_mov_b64 exec, 0
 ; CHECK-NEXT:    s_mov_b64 vcc, 0
 ; CHECK-NEXT:    s_branch .LBB1_1
 ; CHECK-NEXT:  .LBB1_3: ; %Flow1
-; CHECK-NEXT:    s_or_saveexec_b64 s[0:1], s[2:3]
-; CHECK-NEXT:    s_xor_b64 exec, exec, s[0:1]
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[0:1], s[2:3]
 ; CHECK-NEXT:    s_cbranch_execz .LBB1_5
 ; CHECK-NEXT:  ; %bb.4: ; %end
 ; CHECK-NEXT:    v_mov_b32_e32 v0, 0
 ; CHECK-NEXT:    exp mrt0 v0, off, v0, off done compr vm
 ; CHECK-NEXT:  .LBB1_5: ; %UnifiedReturnBlock
 ; CHECK-NEXT:    s_endpgm
 ; CHECK-NEXT:  .LBB1_6:
 ; CHECK-NEXT:    s_mov_b64 exec, 0

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.softwqm.ll

 ; CHECK:       ; %bb.0: ; %main_body
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v1
 ; CHECK-NEXT:    s_and_saveexec_b64 s[0:1], vcc
 ; CHECK-NEXT:    s_xor_b64 s[0:1], exec, s[0:1]
 ; CHECK-NEXT:    s_cbranch_execz .LBB6_2
 ; CHECK-NEXT:  ; %bb.1: ; %ELSE
 ; CHECK-NEXT:    buffer_store_dword v2, v0, s[0:3], 0 idxen
 ; CHECK-NEXT:  .LBB6_2: ; %Flow
-; CHECK-NEXT:    s_or_saveexec_b64 s[0:1], s[0:1]
-; CHECK-NEXT:    s_xor_b64 exec, exec, s[0:1]
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[0:1], s[0:1]
 ; CHECK-NEXT:    s_cbranch_execz .LBB6_4
 ; CHECK-NEXT:  ; %bb.3: ; %IF
 ; CHECK-NEXT:    v_mov_b32_e32 v0, s12
 ; CHECK-NEXT:    v_mov_b32_e32 v1, s13
 ; CHECK-NEXT:    buffer_load_dword v0, v0, s[0:3], 0 idxen
 ; CHECK-NEXT:    buffer_load_dword v1, v1, s[0:3], 0 idxen
 ; CHECK-NEXT:    s_waitcnt vmcnt(0)
 ; CHECK-NEXT:    v_add_f32_e32 v2, v0, v1

 ; CHECK-NEXT:    image_sample v1, v0, s[0:7], s[8:11] dmask:0x1
 ; CHECK-NEXT:    s_and_saveexec_b64 s[18:19], s[14:15]
 ; CHECK-NEXT:    s_waitcnt vmcnt(0)
 ; CHECK-NEXT:    image_sample v1, v1, s[0:7], s[8:11] dmask:0x1
 ; CHECK-NEXT:    s_waitcnt vmcnt(0)
 ; CHECK-NEXT:    buffer_store_dword v1, v0, s[0:3], 0 idxen
 ; CHECK-NEXT:    s_mov_b64 exec, s[18:19]
 ; CHECK-NEXT:  .LBB7_2: ; %Flow
-; CHECK-NEXT:    s_or_saveexec_b64 s[0:1], s[16:17]
-; CHECK-NEXT:    s_xor_b64 exec, exec, s[0:1]
+; CHECK-NEXT:    s_andn2_saveexec_b64 s[0:1], s[16:17]
 ; CHECK-NEXT:    s_cbranch_execz .LBB7_4
 ; CHECK-NEXT:  ; %bb.3: ; %IF
 ; CHECK-NEXT:    v_mov_b32_e32 v0, s12
 ; CHECK-NEXT:    v_mov_b32_e32 v1, s13
 ; CHECK-NEXT:    buffer_load_dword v0, v0, s[0:3], 0 idxen
 ; CHECK-NEXT:    buffer_load_dword v1, v1, s[0:3], 0 idxen
 ; CHECK-NEXT:    s_waitcnt vmcnt(0)
 ; CHECK-NEXT:    v_add_f32_e32 v2, v0, v1

llvm/test/CodeGen/AMDGPU/multi-divergent-exit-region.ll

 ; GCN-DAG:  s_xor_b64
 
 ; GCN: ; %LeafBlock1
 ; GCN-NEXT: s_mov_b64           [[EXIT0]], exec
 ; GCN-NEXT: v_cmp_ne_u32_e32    vcc, 2,
 ; GCN-NEXT: s_and_b64           [[EXIT1]], vcc, exec
 
 ; GCN: ; %Flow
-; GCN-NEXT: s_or_saveexec_b64
-; GCN-NEXT: s_xor_b64
+; GCN-NEXT: s_andn2_saveexec_b64
 
 ; GCN: ; %LeafBlock
 ; GCN-DAG:  v_cmp_eq_u32_e32    vcc, 1,
 ; GCN-DAG:  v_cmp_ne_u32_e64    [[INV:s\[[0-9]+:[0-9]+\]]], 1,
 ; GCN-DAG:  s_andn2_b64         [[EXIT0]], [[EXIT0]], exec
 ; GCN-DAG:  s_andn2_b64         [[EXIT1]], [[EXIT1]], exec
 ; GCN-DAG:  s_and_b64           [[TMP0:s\[[0-9]+:[0-9]+\]]], vcc, exec
 ; GCN-DAG:  s_and_b64           [[TMP1:s\[[0-9]+:[0-9]+\]]], [[INV]], exec

llvm/test/CodeGen/AMDGPU/s_or_saveexec_xor_combine.mir

+# RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32 -run-pass=si-optimize-exec-masking -verify-machineinstrs %s -o - | FileCheck --check-prefixes=GCN,WAVE32 %s
+# RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -run-pass=si-optimize-exec-masking -verify-machineinstrs %s -o - | FileCheck --check-prefixes=GCN,WAVE64 %s
+
+---
+
+# After the Optimize exec masking (post-RA) pass, codegen can end up with the following sequence:
+# s_or_saveexec_b32 s0, s0
+# s_xor_b32 exec_lo, exec_lo, s0
+#
+# This can be combined into one instruction:
+# s_andn2_saveexec_b32 s0, s0
+
+# Ensure the transformation gets applied in the b32 case.
+# GCN-LABEL: name: s_or_saveexec_xor_combine_b32
+# WAVE32: S_ANDN2_SAVEEXEC_B32
+name: s_or_saveexec_xor_combine_b32
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $sgpr0
+    renamable $sgpr0 = S_OR_SAVEEXEC_B32 killed renamable $sgpr0, implicit-def $exec, implicit-def $scc, implicit $exec
+    $exec_lo = S_XOR_B32 $exec_lo, renamable $sgpr0, implicit-def $scc
+...
+
+---
+
+# Ensure the transformation gets applied in the b64 case.
+# GCN-LABEL: name: s_or_saveexec_xor_combine_b64
+# WAVE64: S_ANDN2_SAVEEXEC_B64
+name: s_or_saveexec_xor_combine_b64
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $sgpr0_sgpr1
+    renamable $sgpr0_sgpr1 = S_OR_SAVEEXEC_B64 killed renamable $sgpr0_sgpr1, implicit-def $exec, implicit-def $scc, implicit $exec
+    $exec = S_XOR_B64 $exec, renamable $sgpr0_sgpr1, implicit-def $scc
+...
+
+---
+
+# Ensure the transformation does get applied even if the operands are swapped.
+# GCN-LABEL: name: s_or_saveexec_xor_combine_b32_swap
+# WAVE32: S_ANDN2_SAVEEXEC_B32
+name: s_or_saveexec_xor_combine_b32_swap
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $sgpr0
+    renamable $sgpr0 = S_OR_SAVEEXEC_B32 killed renamable $sgpr0, implicit-def $exec, implicit-def $scc, implicit $exec
+    $exec_lo = S_XOR_B32 renamable $sgpr0, $exec_lo, implicit-def $scc
+...
+
+---
+
+# Ensure the transformation does get applied if source and dest operand for s_or_saveeexec are not equal.
+# GCN-LABEL: name: s_or_saveexec_xor_combine_b32_inequal_operands
+# WAVE32: S_ANDN2_SAVEEXEC
+name: s_or_saveexec_xor_combine_b32_inequal_operands
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $sgpr0, $sgpr1
+    renamable $sgpr0 = S_OR_SAVEEXEC_B32 killed renamable $sgpr1, implicit-def $exec, implicit-def $scc, implicit $exec
+    $exec_lo = S_XOR_B32 $exec_lo, renamable $sgpr0, implicit-def $scc
+...
+
+---
+
+# Ensure the transformation does not get applied if s_xor does not use the dest as input operand.
+# GCN-LABEL: name: s_or_saveexec_xor_combine_b32_wrong_input
+# WAVE32: S_OR_SAVEEXEC
+# WAVE32: S_XOR_B32
+name: s_or_saveexec_xor_combine_b32_wrong_input
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $sgpr0, $sgpr1
+    renamable $sgpr0 = S_OR_SAVEEXEC_B32 killed renamable $sgpr0, implicit-def $exec, implicit-def $scc, implicit $exec
+    $exec_lo = S_XOR_B32 $exec_lo, renamable $sgpr1, implicit-def $scc
+...
+
+---
+
+
+# Ensure the transformation does not get applied if the instructions don't appear sequentially.
+# GCN-LABEL: name: s_or_saveexec_xor_combine_b32_non_sequence
+# WAVE32: S_OR_SAVEEXEC
+# WAVE32: S_MOV_B32
+# WAVE32: S_XOR_B32
+name: s_or_saveexec_xor_combine_b32_non_sequence
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $sgpr0, $sgpr1
+    renamable $sgpr0 = S_OR_SAVEEXEC_B32 killed renamable $sgpr0, implicit-def $exec, implicit-def $scc, implicit $exec
+    renamable $sgpr1 = S_MOV_B32 renamable $sgpr0
+    $exec_lo = S_XOR_B32 $exec_lo, renamable $sgpr1, implicit-def $scc
+...
+
+---
+
+# Don't apply the transformation if the basic block only has a single instruction.
+
+# GCN-LABEL: name: s_or_saveexec_xor_combine_b32_last_inst
+# WAVE32: S_OR_SAVEEXEC
+name: s_or_saveexec_xor_combine_b32_last_inst
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $sgpr0
+    renamable $sgpr0 = S_OR_SAVEEXEC_B32 killed renamable $sgpr0, implicit-def $exec, implicit-def $scc, implicit $exec
+...
+
+---
+
+# Don't apply the transformation if the basic block ends with an S_OR_SAVEEXEC_B32 instruction.
+
+# GCN-LABEL: name: s_or_saveexec_xor_combine_b32_or_saveexec_terminator
+# WAVE32: S_MOV_B32
+# WAVE32: S_OR_SAVEEXEC
+name: s_or_saveexec_xor_combine_b32_or_saveexec_terminator
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $sgpr0, $sgpr1
+    renamable $sgpr1 = S_MOV_B32 renamable $sgpr0
+    renamable $sgpr0 = S_OR_SAVEEXEC_B32 killed renamable $sgpr0, implicit-def $exec, implicit-def $scc, implicit $exec
 No newline at end of file

llvm/test/CodeGen/AMDGPU/set-inactive-wwm-overwrite.ll

 ; GCN-NEXT:    s_and_saveexec_b32 s0, vcc_lo
 ; GCN-NEXT:  ; %bb.1: ; %.bb0
 ; GCN-NEXT:    v_mov_b32_e32 v3, 1
 ; GCN-NEXT:  ; %bb.2: ; %.merge
 ; GCN-NEXT:    s_or_b32 exec_lo, exec_lo, s0
 ; GCN-NEXT:    v_cmp_lt_u32_e32 vcc_lo, 3, v0
 ; GCN-NEXT:    s_and_saveexec_b32 s0, vcc_lo
 ; GCN-NEXT:    s_xor_b32 s0, exec_lo, s0
-; GCN-NEXT:    s_cbranch_execz .LBB1_4
-; GCN-NEXT:  ; %bb.3: ; %.else
+; GCN-NEXT:    s_cbranch_execnz .LBB1_5
+; GCN-NEXT:  ; %bb.3: ; %Flow
+; GCN-NEXT:    s_andn2_saveexec_b32 s0, s0
+; GCN-NEXT:    s_cbranch_execnz .LBB1_6
+; GCN-NEXT:  .LBB1_4: ; %.end
+; GCN-NEXT:    s_endpgm
+; GCN-NEXT:  .LBB1_5: ; %.else
 ; GCN-NEXT:    s_or_saveexec_b32 s1, -1
 ; GCN-NEXT:    v_mov_b32_e32 v1, 0
 ; GCN-NEXT:    s_mov_b32 exec_lo, s1
 ; GCN-NEXT:    v_mov_b32_e32 v2, v3
 ; GCN-NEXT:    s_not_b32 exec_lo, exec_lo
 ; GCN-NEXT:    v_mov_b32_e32 v2, 0
 ; GCN-NEXT:    s_not_b32 exec_lo, exec_lo
 ; GCN-NEXT:    s_or_saveexec_b32 s1, -1
 ; GCN-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
 ; GCN-NEXT:    s_mov_b32 exec_lo, s1
 ; GCN-NEXT:    v_mov_b32_e32 v0, v1
 ; GCN-NEXT:    v_mov_b32_e32 v3, -1
 ; GCN-NEXT:    buffer_store_dword v3, v0, s[4:7], 0 offen
 ; GCN-NEXT:    ; implicit-def: $vgpr3
-; GCN-NEXT:  .LBB1_4: ; %Flow
-; GCN-NEXT:    s_or_saveexec_b32 s0, s0
-; GCN-NEXT:    s_waitcnt_depctr 0xffe3
-; GCN-NEXT:    s_xor_b32 exec_lo, exec_lo, s0
-; GCN-NEXT:    s_cbranch_execz .LBB1_6
-; GCN-NEXT:  ; %bb.5: ; %.then
+; GCN-NEXT:    s_andn2_saveexec_b32 s0, s0
+; GCN-NEXT:    s_cbranch_execz .LBB1_4
+; GCN-NEXT:  .LBB1_6: ; %.then
 ; GCN-NEXT:    v_mov_b32_e32 v0, -1
 ; GCN-NEXT:    buffer_store_dword v0, v3, s[4:7], 0 offen
-; GCN-NEXT:  .LBB1_6: ; %.end
 ; GCN-NEXT:    s_endpgm
 .entry:
   %LocalInvocationId.i0 = extractelement <3 x i32> %LocalInvocationId, i32 0
   %.not10002 = icmp eq i32 %LocalInvocationId.i0, 0
   %i530 = icmp ult i32 %LocalInvocationId.i0, 4
   br i1 %.not10002, label %.merge, label %.bb0
 
 .bb0:

llvm/test/CodeGen/AMDGPU/sgpr-control-flow.ll

 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    buffer_load_dword v0, v[0:1], s[0:3], 0 addr64
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_cmp_gt_i32_e32 vcc, 0, v0
 ; SI-NEXT:    s_and_b64 s[8:9], vcc, exec
 ; SI-NEXT:    ; implicit-def: $vgpr0
 ; SI-NEXT:  .LBB3_2: ; %Flow
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
-; SI-NEXT:    s_or_saveexec_b64 s[0:1], s[10:11]
-; SI-NEXT:    s_xor_b64 exec, exec, s[0:1]
+; SI-NEXT:    s_andn2_saveexec_b64 s[0:1], s[10:11]
 ; SI-NEXT:    s_cbranch_execz .LBB3_4
 ; SI-NEXT:  ; %bb.3: ; %if
 ; SI-NEXT:    s_mov_b32 s15, 0xf000
 ; SI-NEXT:    s_mov_b32 s14, 0
 ; SI-NEXT:    s_mov_b64 s[12:13], s[6:7]
 ; SI-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; SI-NEXT:    v_mov_b32_e32 v1, 0
 ; SI-NEXT:    buffer_load_dword v0, v[0:1], s[12:15], 0 addr64

llvm/test/CodeGen/AMDGPU/transform-block-with-return-to-epilog.ll

 else.if:                                          ; preds = %else.if.cond
   ret float %d
 else:                                             ; preds = %else.if.cond
   store volatile i32 0, i32 addrspace(1)* undef
   unreachable
 }
 
 define amdgpu_ps { <4 x float> } @test_return_to_epilog_with_optimized_kill(float %val) #0 {
-  ; GCN-LABEL: name: test_return_to_epilog_with_optimized_kill
-  ; GCN: bb.0.entry:
-  ; GCN:   successors: %bb.1(0x40000000), %bb.4(0x40000000)
-  ; GCN:   liveins: $vgpr0
-  ; GCN:   renamable $vgpr1 = nofpexcept V_RCP_F32_e32 $vgpr0, implicit $mode, implicit $exec
-  ; GCN:   $sgpr0_sgpr1 = S_MOV_B64 $exec
-  ; GCN:   nofpexcept V_CMP_NGT_F32_e32 0, killed $vgpr1, implicit-def $vcc, implicit $mode, implicit $exec
-  ; GCN:   $sgpr2_sgpr3 = S_AND_SAVEEXEC_B64 killed $vcc, implicit-def $exec, implicit-def $scc, implicit $exec
-  ; GCN:   renamable $sgpr2_sgpr3 = S_XOR_B64 $exec, killed renamable $sgpr2_sgpr3, implicit-def dead $scc
-  ; GCN:   S_CBRANCH_EXECZ %bb.4, implicit $exec
-  ; GCN: bb.1.flow.preheader:
-  ; GCN:   successors: %bb.2(0x80000000)
-  ; GCN:   liveins: $vgpr0, $sgpr0_sgpr1, $sgpr2_sgpr3
-  ; GCN:   nofpexcept V_CMP_NGT_F32_e32 0, killed $vgpr0, implicit-def $vcc, implicit $mode, implicit $exec
-  ; GCN:   renamable $sgpr4_sgpr5 = S_MOV_B64 0
-  ; GCN: bb.2.flow:
-  ; GCN:   successors: %bb.3(0x04000000), %bb.2(0x7c000000)
-  ; GCN:   liveins: $vcc, $sgpr0_sgpr1, $sgpr2_sgpr3, $sgpr4_sgpr5
-  ; GCN:   renamable $sgpr6_sgpr7 = S_AND_B64 $exec, renamable $vcc, implicit-def $scc
-  ; GCN:   renamable $sgpr4_sgpr5 = S_OR_B64 killed renamable $sgpr6_sgpr7, killed renamable $sgpr4_sgpr5, implicit-def $scc
-  ; GCN:   $exec = S_ANDN2_B64 $exec, renamable $sgpr4_sgpr5, implicit-def $scc
-  ; GCN:   S_CBRANCH_EXECNZ %bb.2, implicit $exec
-  ; GCN: bb.3.Flow:
-  ; GCN:   successors: %bb.4(0x80000000)
-  ; GCN:   liveins: $sgpr0_sgpr1, $sgpr2_sgpr3, $sgpr4_sgpr5
-  ; GCN:   $exec = S_OR_B64 $exec, killed renamable $sgpr4_sgpr5, implicit-def $scc
-  ; GCN: bb.4.Flow1:
-  ; GCN:   successors: %bb.5(0x40000000), %bb.7(0x40000000)
-  ; GCN:   liveins: $sgpr0_sgpr1, $sgpr2_sgpr3
-  ; GCN:   renamable $sgpr2_sgpr3 = S_OR_SAVEEXEC_B64 killed renamable $sgpr2_sgpr3, implicit-def $exec, implicit-def $scc, implicit $exec
-  ; GCN:   $exec = S_XOR_B64 $exec, renamable $sgpr2_sgpr3, implicit-def $scc
-  ; GCN:   S_CBRANCH_EXECZ %bb.7, implicit $exec
-  ; GCN: bb.5.kill0:
-  ; GCN:   successors: %bb.6(0x40000000), %bb.8(0x40000000)
-  ; GCN:   liveins: $sgpr0_sgpr1, $sgpr2_sgpr3
-  ; GCN:   dead renamable $sgpr0_sgpr1 = S_ANDN2_B64 killed renamable $sgpr0_sgpr1, $exec, implicit-def $scc
-  ; GCN:   S_CBRANCH_SCC0 %bb.8, implicit $scc
-  ; GCN: bb.6.kill0:
-  ; GCN:   successors: %bb.7(0x80000000)
-  ; GCN:   liveins: $sgpr2_sgpr3, $scc
-  ; GCN:   $exec = S_MOV_B64 0
-  ; GCN: bb.7.end:
-  ; GCN:   successors: %bb.9(0x80000000)
-  ; GCN:   liveins: $sgpr2_sgpr3
-  ; GCN:   $exec = S_OR_B64 $exec, killed renamable $sgpr2_sgpr3, implicit-def $scc
-  ; GCN:   S_BRANCH %bb.9
-  ; GCN: bb.8:
-  ; GCN:   $exec = S_MOV_B64 0
-  ; GCN:   EXP_DONE 9, undef $vgpr0, undef $vgpr0, undef $vgpr0, undef $vgpr0, 1, 0, 0, implicit $exec
-  ; GCN:   S_ENDPGM 0
-  ; GCN: bb.9:
-entry:
+; GCN: bb.0 (%ir-block.0):
+; GCN:   successors: %bb.3(0x40000000), %bb.1(0x40000000)
+; GCN:   S_CBRANCH_EXECNZ %bb.3, implicit $exec
+; GCN: bb.1.Flow1:
+; GCN:   successors: %bb.6(0x40000000), %bb.2(0x40000000)
+; GCN:   liveins: $sgpr0_sgpr1, $sgpr2_sgpr3
+; GCN:   $sgpr2_sgpr3 = S_ANDN2_SAVEEXEC_B64 killed $sgpr2_sgpr3, implicit-def $exec, implicit-def $scc, implicit $exec
+; GCN:   S_CBRANCH_EXECNZ %bb.6, implicit $exec
+; GCN: bb.2.end:
+; GCN:   successors: %bb.9(0x80000000)
+; GCN:   liveins: $sgpr2_sgpr3
+; GCN:   $exec = S_OR_B64 $exec, killed renamable $sgpr2_sgpr3, implicit-def $scc
+; GCN:   S_BRANCH %bb.9
+; GCN: bb.3.flow.preheader:
+; GCN:   successors: %bb.4(0x80000000)
+; GCN: bb.4.flow:
+; GCN:   successors: %bb.5(0x04000000), %bb.4(0x7c000000)
+; GCN:   S_CBRANCH_EXECNZ %bb.4, implicit $exec
+; GCN: bb.5.Flow:
+; GCN:   successors: %bb.6(0x40000000), %bb.2(0x40000000)
+; GCN:   $sgpr2_sgpr3 = S_ANDN2_SAVEEXEC_B64 killed $sgpr2_sgpr3, implicit-def $exec, implicit-def $scc, implicit $exec
+; GCN:   S_CBRANCH_EXECZ %bb.2, implicit $exec
+; GCN: bb.6.kill0:
+; GCN:   successors: %bb.7(0x40000000), %bb.8(0x40000000)
+; GCN: bb.7.kill0:
+; GCN:   successors: %bb.9(0x80000000)
   %.i0 = fdiv reassoc nnan nsz arcp contract afn float 1.000000e+00, %val
   %cmp0 = fcmp olt float %.i0, 0.000000e+00
   br i1 %cmp0, label %kill0, label %flow
 
 kill0:                                            ; preds = %entry
   call void @llvm.amdgcn.kill(i1 false)
   br label %end
 

llvm/test/CodeGen/AMDGPU/valu-i1.ll

 
 ; SI-NEXT: ; %bb.{{[0-9]+}}: ; %LeafBlock3
 ; SI:      s_mov_b64 s[{{[0-9]:[0-9]}}], -1
 ; SI:      s_and_saveexec_b64
 ; SI-NEXT: s_cbranch_execnz
 
 ; v_mov should be after exec modification
 ; SI: [[FLOW_BB]]:
-; SI-NEXT: s_or_saveexec_b64 [[SAVE3:s\[[0-9]+:[0-9]+\]]], [[SAVE2]]
-; SI-NEXT: s_xor_b64 exec, exec, [[SAVE3]]
+; SI-NEXT: s_andn2_saveexec_b64 [[SAVE2]], [[SAVE2]]
 ;
 define amdgpu_kernel void @test_if(i32 %b, i32 addrspace(1)* %src, i32 addrspace(1)* %dst) #1 {
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
   switch i32 %tid, label %default [
     i32 1, label %case1
     i32 2, label %case2
   ]

 
 ; SI-LABEL: {{^}}simple_test_v_if_ret_else_code_ret:
 ; SI: v_cmp_eq_u32_e32 vcc, 0, v{{[0-9]+}}
 ; SI: s_and_saveexec_b64 [[BR_SREG:s\[[0-9]+:[0-9]+\]]], vcc
 ; SI: s_xor_b64 [[BR_SREG]], exec, [[BR_SREG]]
 ; SI: s_cbranch_execnz [[EXIT:.LBB[0-9]+_[0-9]+]]
 
 ; SI-NEXT: {{^.LBB[0-9]+_[0-9]+}}: ; %Flow
-; SI-NEXT: s_or_saveexec_b64
-; SI-NEXT: s_xor_b64 exec, exec
+; SI-NEXT: s_andn2_saveexec_b64 [[BR_SREG]], [[BR_SREG]]
 ; SI-NEXT: s_cbranch_execz [[UNIFIED_RETURN:.LBB[0-9]+_[0-9]+]]
 
 ; SI-NEXT: ; %bb.{{[0-9]+}}: ; %then
 ; SI: s_waitcnt
 ; SI-NEXT: buffer_store_dword
 
 ; SI-NEXT: {{^}}[[UNIFIED_RETURN]]: ; %UnifiedReturnBlock
 ; SI: s_endpgm

 
 ; SI: [[LABEL_LOOP:.LBB[0-9]+_[0-9]+]]:
 ; SI: buffer_load_dword
 ; SI-DAG: buffer_store_dword
 ; SI-DAG: s_cmpk_lg_i32 s{{[0-9]+}}, 0x100
 ; SI: s_cbranch_scc1 [[LABEL_LOOP]]
 ; SI: [[LABEL_EXIT]]:
 ; SI: s_endpgm
-
 define amdgpu_kernel void @simple_test_v_loop(i32 addrspace(1)* %dst, i32 addrspace(1)* %src) #1 {
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
   %is.0 = icmp ne i32 %tid, 0
   %limit = add i32 %tid, 64
   br i1 %is.0, label %loop, label %exit
 
 loop:

 ; SI-NEXT: s_and_b64 [[TMP1:s\[[0-9]+:[0-9]+\]]],
 ; SI-NEXT: s_or_b64 [[COND_STATE]], [[TMP1]], [[COND_STATE]]
 ; SI-NEXT: s_andn2_b64 exec, exec, [[COND_STATE]]
 ; SI-NEXT: s_cbranch_execnz .L[[LABEL_LOOP]]
 
 ; SI: [[LABEL_EXIT]]:
 ; SI-NOT: [[COND_STATE]]
 ; SI: s_endpgm
-
 define amdgpu_kernel void @multi_vcond_loop(i32 addrspace(1)* noalias nocapture %arg, i32 addrspace(1)* noalias nocapture readonly %arg1, i32 addrspace(1)* noalias nocapture readonly %arg2, i32 addrspace(1)* noalias nocapture readonly %arg3) #1 {
 bb:
   %tmp = tail call i32 @llvm.amdgcn.workitem.id.x() #0
   %tmp4 = sext i32 %tmp to i64
   %tmp5 = getelementptr inbounds i32, i32 addrspace(1)* %arg3, i64 %tmp4
   %tmp6 = load i32, i32 addrspace(1)* %tmp5, align 4
   %tmp7 = icmp sgt i32 %tmp6, 0
   %tmp8 = sext i32 %tmp6 to i64

llvm/test/CodeGen/AMDGPU/vcmp-saveexec-to-vcmpx.ll

 l2:
   ret void
 }
 
 ; Omit the transformation if the s_and_saveexec instruction overwrites
 ; any of the v_cmp source operands.
 
 ; GCN-LABEL: check_saveexec_overwrites_vcmp_source:
-; GCN:  ; %bb.1: ; %then
+; GCN:  .LBB7_3: ; %then
 ; GFX1010:          v_cmp_ge_i32_e32 vcc_lo, s[[A:[0-9]+]], v{{.*}}
 ; GFX1010-NEXT:     v_mov_b32_e32 {{.*}}, s[[A]]
 ; GFX1010-NEXT:     s_and_saveexec_b32 s[[A]], vcc_lo
 ; GFX1030:          v_cmp_ge_i32_e32 vcc_lo, s[[A:[0-9]+]], v{{.*}}
 ; GFX1030-NEXT:     v_mov_b32_e32 {{.*}}, s[[A]]
 ; GFX1030-NEXT:     s_and_saveexec_b32 s[[A]], vcc_lo
 define i32 @check_saveexec_overwrites_vcmp_source(i32 inreg %a, i32 inreg %b) {
 entry:

llvm/test/CodeGen/AMDGPU/vgpr-liverange.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -march=amdgcn -mcpu=gfx1010 -amdgpu-opt-vgpr-liverange=true -verify-machineinstrs < %s | FileCheck -check-prefix=SI %s
 
 ; a normal if-else
 define amdgpu_ps float @else1(i32 %z, float %v) #0 {
 ; SI-LABEL: else1:
 ; SI:       ; %bb.0: ; %main_body
 ; SI-NEXT:    v_cmp_gt_i32_e32 vcc_lo, 6, v0
 ; SI-NEXT:    ; implicit-def: $vgpr0
 ; SI-NEXT:    s_and_saveexec_b32 s0, vcc_lo
 ; SI-NEXT:    s_xor_b32 s0, exec_lo, s0
-; SI-NEXT:  ; %bb.1: ; %else
+; SI-NEXT:    s_cbranch_execnz .LBB0_3
+; SI-NEXT:  ; %bb.1: ; %Flow
+; SI-NEXT:    s_andn2_saveexec_b32 s0, s0
+; SI-NEXT:    s_cbranch_execnz .LBB0_4
+; SI-NEXT:  .LBB0_2: ; %end
+; SI-NEXT:    s_or_b32 exec_lo, exec_lo, s0
+; SI-NEXT:    s_branch .LBB0_5
+; SI-NEXT:  .LBB0_3: ; %else
 ; SI-NEXT:    v_mul_f32_e32 v0, 0x40400000, v1
 ; SI-NEXT:    ; implicit-def: $vgpr1
-; SI-NEXT:  ; %bb.2: ; %Flow
-; SI-NEXT:    s_or_saveexec_b32 s0, s0
-; SI-NEXT:    s_xor_b32 exec_lo, exec_lo, s0
-; SI-NEXT:  ; %bb.3: ; %if
+; SI-NEXT:    s_andn2_saveexec_b32 s0, s0
+; SI-NEXT:    s_cbranch_execz .LBB0_2
+; SI-NEXT:  .LBB0_4: ; %if
 ; SI-NEXT:    v_add_f32_e32 v0, v1, v1
-; SI-NEXT:  ; %bb.4: ; %end
 ; SI-NEXT:    s_or_b32 exec_lo, exec_lo, s0
-; SI-NEXT:    ; return to shader part epilog
+; SI-NEXT:    s_branch .LBB0_5
+; SI-NEXT:  .LBB0_5:
 main_body:
   %cc = icmp sgt i32 %z, 5
   br i1 %cc, label %if, label %else
 
 if:
   %v.if = fmul float %v, 2.0
   br label %end
 

 ; SI:       ; %bb.0: ; %main_body
 ; SI-NEXT:    v_cmp_gt_i32_e32 vcc_lo, 6, v0
 ; SI-NEXT:    ; implicit-def: $vgpr0
 ; SI-NEXT:    s_and_saveexec_b32 s0, vcc_lo
 ; SI-NEXT:    s_xor_b32 s0, exec_lo, s0
 ; SI-NEXT:  ; %bb.1: ; %else
 ; SI-NEXT:    v_mul_f32_e32 v0, 0x40400000, v1
 ; SI-NEXT:  ; %bb.2: ; %Flow
-; SI-NEXT:    s_or_saveexec_b32 s0, s0
-; SI-NEXT:    s_xor_b32 exec_lo, exec_lo, s0
+; SI-NEXT:    s_andn2_saveexec_b32 s0, s0
 ; SI-NEXT:  ; %bb.3: ; %if
 ; SI-NEXT:    v_add_f32_e32 v1, v1, v1
 ; SI-NEXT:    v_mov_b32_e32 v0, v1
 ; SI-NEXT:  ; %bb.4: ; %end
 ; SI-NEXT:    s_or_b32 exec_lo, exec_lo, s0
 ; SI-NEXT:    v_add_f32_e32 v0, v1, v0
 ; SI-NEXT:    ; return to shader part epilog
 main_body:

 ; SI-NEXT:    s_xor_b32 s2, exec_lo, s2
 ; SI-NEXT:  ; %bb.3: ; %else
 ; SI-NEXT:    ; in Loop: Header=BB2_2 Depth=1
 ; SI-NEXT:    v_mul_lo_u32 v0, v2, 3
 ; SI-NEXT:    v_mul_f32_e32 v3, v1, v2
 ; SI-NEXT:    ; implicit-def: $vgpr2
 ; SI-NEXT:  ; %bb.4: ; %Flow
 ; SI-NEXT:    ; in Loop: Header=BB2_2 Depth=1
-; SI-NEXT:    s_or_saveexec_b32 s2, s2
-; SI-NEXT:    s_xor_b32 exec_lo, exec_lo, s2
+; SI-NEXT:    s_andn2_saveexec_b32 s2, s2
 ; SI-NEXT:    s_cbranch_execz .LBB2_1
 ; SI-NEXT:  ; %bb.5: ; %if
 ; SI-NEXT:    ; in Loop: Header=BB2_2 Depth=1
 ; SI-NEXT:    v_mul_f32_e32 v3, s1, v1
 ; SI-NEXT:    v_add_nc_u32_e32 v0, 1, v2
 ; SI-NEXT:    s_branch .LBB2_1
 ; SI-NEXT:  .LBB2_6: ; %for.end
 ; SI-NEXT:    v_add_f32_e32 v0, v0, v3

 ; SI-NEXT:    ; implicit-def: $vgpr0
 ; SI-NEXT:    s_xor_b32 exec_lo, exec_lo, s8
 ; SI-NEXT:    s_cbranch_execnz .LBB3_2
 ; SI-NEXT:  ; %bb.3:
 ; SI-NEXT:    s_mov_b32 exec_lo, s7
 ; SI-NEXT:    ; implicit-def: $vgpr0
 ; SI-NEXT:    ; implicit-def: $vgpr2
 ; SI-NEXT:  .LBB3_4: ; %Flow
-; SI-NEXT:    s_or_saveexec_b32 s6, s6
-; SI-NEXT:    s_xor_b32 exec_lo, exec_lo, s6
+; SI-NEXT:    s_andn2_saveexec_b32 s6, s6
 ; SI-NEXT:    s_cbranch_execz .LBB3_8
 ; SI-NEXT:  ; %bb.5: ; %if
 ; SI-NEXT:    s_mov_b32 s7, exec_lo
 ; SI-NEXT:  .LBB3_6: ; =>This Inner Loop Header: Depth=1
 ; SI-NEXT:    v_readfirstlane_b32 s4, v2
 ; SI-NEXT:    v_readfirstlane_b32 s5, v3
 ; SI-NEXT:    v_cmp_eq_u64_e32 vcc_lo, s[4:5], v[2:3]
 ; SI-NEXT:    s_and_saveexec_b32 s8, vcc_lo

 ; SI-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; SI-NEXT:    ; implicit-def: $vgpr4_vgpr5
 ; SI-NEXT:    s_xor_b32 exec_lo, exec_lo, s8
 ; SI-NEXT:    s_cbranch_execnz .LBB4_2
 ; SI-NEXT:  ; %bb.3:
 ; SI-NEXT:    s_mov_b32 exec_lo, s7
 ; SI-NEXT:    ; implicit-def: $vgpr2
 ; SI-NEXT:  .LBB4_4: ; %Flow
-; SI-NEXT:    s_or_saveexec_b32 s6, s6
-; SI-NEXT:    s_xor_b32 exec_lo, exec_lo, s6
+; SI-NEXT:    s_andn2_saveexec_b32 s6, s6
 ; SI-NEXT:    s_cbranch_execz .LBB4_8
 ; SI-NEXT:  ; %bb.5: ; %if
 ; SI-NEXT:    s_mov_b32 s7, exec_lo
 ; SI-NEXT:  .LBB4_6: ; =>This Inner Loop Header: Depth=1
 ; SI-NEXT:    v_readfirstlane_b32 s4, v2
 ; SI-NEXT:    v_readfirstlane_b32 s5, v3
 ; SI-NEXT:    v_cmp_eq_u64_e32 vcc_lo, s[4:5], v[2:3]
 ; SI-NEXT:    s_and_saveexec_b32 s8, vcc_lo

llvm/test/CodeGen/AMDGPU/wqm.ll

 ; GFX9-W64-NEXT:    s_xor_b64 s[14:15], exec, s[14:15]
 ; GFX9-W64-NEXT:    s_cbranch_execz .LBB23_2
 ; GFX9-W64-NEXT:  ; %bb.1: ; %ELSE
 ; GFX9-W64-NEXT:    s_and_saveexec_b64 s[16:17], s[12:13]
 ; GFX9-W64-NEXT:    buffer_store_dword v2, v0, s[0:3], 0 idxen
 ; GFX9-W64-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-W64-NEXT:    s_mov_b64 exec, s[16:17]
 ; GFX9-W64-NEXT:  .LBB23_2: ; %Flow
-; GFX9-W64-NEXT:    s_or_saveexec_b64 s[14:15], s[14:15]
-; GFX9-W64-NEXT:    s_xor_b64 exec, exec, s[14:15]
+; GFX9-W64-NEXT:    s_andn2_saveexec_b64 s[14:15], s[14:15]
 ; GFX9-W64-NEXT:    s_cbranch_execz .LBB23_4
 ; GFX9-W64-NEXT:  ; %bb.3: ; %IF
 ; GFX9-W64-NEXT:    image_sample v0, v0, s[0:7], s[8:11] dmask:0x1
 ; GFX9-W64-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-W64-NEXT:    image_sample v2, v0, s[0:7], s[8:11] dmask:0x1
 ; GFX9-W64-NEXT:  .LBB23_4: ; %END
 ; GFX9-W64-NEXT:    s_or_b64 exec, exec, s[14:15]
 ; GFX9-W64-NEXT:    s_and_b64 exec, exec, s[12:13]

 ; GFX10-W32-NEXT:    s_xor_b32 s13, exec_lo, s13
 ; GFX10-W32-NEXT:    s_cbranch_execz .LBB23_2
 ; GFX10-W32-NEXT:  ; %bb.1: ; %ELSE
 ; GFX10-W32-NEXT:    s_and_saveexec_b32 s14, s12
 ; GFX10-W32-NEXT:    buffer_store_dword v2, v0, s[0:3], 0 idxen
 ; GFX10-W32-NEXT:    ; implicit-def: $vgpr0
 ; GFX10-W32-NEXT:    s_mov_b32 exec_lo, s14
 ; GFX10-W32-NEXT:  .LBB23_2: ; %Flow
-; GFX10-W32-NEXT:    s_or_saveexec_b32 s13, s13
-; GFX10-W32-NEXT:    s_xor_b32 exec_lo, exec_lo, s13
+; GFX10-W32-NEXT:    s_andn2_saveexec_b32 s13, s13
 ; GFX10-W32-NEXT:    s_cbranch_execz .LBB23_4
 ; GFX10-W32-NEXT:  ; %bb.3: ; %IF
 ; GFX10-W32-NEXT:    image_sample v0, v0, s[0:7], s[8:11] dmask:0x1 dim:SQ_RSRC_IMG_1D
 ; GFX10-W32-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-W32-NEXT:    image_sample v2, v0, s[0:7], s[8:11] dmask:0x1 dim:SQ_RSRC_IMG_1D
 ; GFX10-W32-NEXT:  .LBB23_4: ; %END
 ; GFX10-W32-NEXT:    s_or_b32 exec_lo, exec_lo, s13
 ; GFX10-W32-NEXT:    s_and_b32 exec_lo, exec_lo, s12

 ; GFX9-W64-NEXT:    v_cmp_nlt_f32_e32 vcc, 0, v0
 ; GFX9-W64-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-W64-NEXT:    s_and_saveexec_b64 s[14:15], vcc
 ; GFX9-W64-NEXT:    s_xor_b64 s[14:15], exec, s[14:15]
 ; GFX9-W64-NEXT:  ; %bb.1: ; %ELSE
 ; GFX9-W64-NEXT:    v_lshlrev_b32_e32 v0, 2, v5
 ; GFX9-W64-NEXT:    ; implicit-def: $vgpr5
 ; GFX9-W64-NEXT:  ; %bb.2: ; %Flow
-; GFX9-W64-NEXT:    s_or_saveexec_b64 s[14:15], s[14:15]
-; GFX9-W64-NEXT:    s_xor_b64 exec, exec, s[14:15]
+; GFX9-W64-NEXT:    s_andn2_saveexec_b64 s[14:15], s[14:15]
 ; GFX9-W64-NEXT:  ; %bb.3: ; %IF
 ; GFX9-W64-NEXT:    v_mul_lo_u32 v0, v5, 3
 ; GFX9-W64-NEXT:  ; %bb.4: ; %END
 ; GFX9-W64-NEXT:    s_or_b64 exec, exec, s[14:15]
 ; GFX9-W64-NEXT:    s_and_b64 exec, exec, s[12:13]
 ; GFX9-W64-NEXT:    image_sample v[0:3], v0, s[0:7], s[8:11] dmask:0xf
 ; GFX9-W64-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-W64-NEXT:    ; return to shader part epilog

 ; GFX10-W32-NEXT:    ; implicit-def: $vgpr0
 ; GFX10-W32-NEXT:    s_wqm_b32 exec_lo, exec_lo
 ; GFX10-W32-NEXT:    s_and_saveexec_b32 s13, vcc_lo
 ; GFX10-W32-NEXT:    s_xor_b32 s13, exec_lo, s13
 ; GFX10-W32-NEXT:  ; %bb.1: ; %ELSE
 ; GFX10-W32-NEXT:    v_lshlrev_b32_e32 v0, 2, v5
 ; GFX10-W32-NEXT:    ; implicit-def: $vgpr5
 ; GFX10-W32-NEXT:  ; %bb.2: ; %Flow
-; GFX10-W32-NEXT:    s_or_saveexec_b32 s13, s13
-; GFX10-W32-NEXT:    s_xor_b32 exec_lo, exec_lo, s13
+; GFX10-W32-NEXT:    s_andn2_saveexec_b32 s13, s13
 ; GFX10-W32-NEXT:  ; %bb.3: ; %IF
 ; GFX10-W32-NEXT:    v_mul_lo_u32 v0, v5, 3
 ; GFX10-W32-NEXT:  ; %bb.4: ; %END
 ; GFX10-W32-NEXT:    s_or_b32 exec_lo, exec_lo, s13
 ; GFX10-W32-NEXT:    s_and_b32 exec_lo, exec_lo, s12
 ; GFX10-W32-NEXT:    image_sample v[0:3], v0, s[0:7], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
 ; GFX10-W32-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-W32-NEXT:    s_waitcnt_vscnt null, 0x0

 ; GFX9-W64-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-W64-NEXT:    image_sample v1, v1, s[0:7], s[8:11] dmask:0x1
 ; GFX9-W64-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-W64-NEXT:    v_cmp_nlt_f32_e32 vcc, 0, v1
 ; GFX9-W64-NEXT:    buffer_store_dword v1, v0, s[0:3], 0 idxen
 ; GFX9-W64-NEXT:    ; implicit-def: $vgpr0
 ; GFX9-W64-NEXT:    s_and_saveexec_b64 s[0:1], vcc
 ; GFX9-W64-NEXT:    s_xor_b64 s[0:1], exec, s[0:1]
-; GFX9-W64-NEXT:  ; %bb.1: ; %ELSE
+; GFX9-W64-NEXT:    s_cbranch_execnz .LBB26_3
+; GFX9-W64-NEXT:  ; %bb.1: ; %Flow
+; GFX9-W64-NEXT:    s_andn2_saveexec_b64 s[0:1], s[0:1]
+; GFX9-W64-NEXT:    s_cbranch_execnz .LBB26_4
+; GFX9-W64-NEXT:  .LBB26_2: ; %END
+; GFX9-W64-NEXT:    s_or_b64 exec, exec, s[0:1]
+; GFX9-W64-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-W64-NEXT:    s_branch .LBB26_5
+; GFX9-W64-NEXT:  .LBB26_3: ; %ELSE
 ; GFX9-W64-NEXT:    v_mul_f32_e32 v0, 4.0, v1
 ; GFX9-W64-NEXT:    ; implicit-def: $vgpr1
-; GFX9-W64-NEXT:  ; %bb.2: ; %Flow
-; GFX9-W64-NEXT:    s_or_saveexec_b64 s[0:1], s[0:1]
-; GFX9-W64-NEXT:    s_xor_b64 exec, exec, s[0:1]
-; GFX9-W64-NEXT:  ; %bb.3: ; %IF
+; GFX9-W64-NEXT:    s_andn2_saveexec_b64 s[0:1], s[0:1]
+; GFX9-W64-NEXT:    s_cbranch_execz .LBB26_2
+; GFX9-W64-NEXT:  .LBB26_4: ; %IF
 ; GFX9-W64-NEXT:    v_mul_f32_e32 v0, 0x40400000, v1
-; GFX9-W64-NEXT:  ; %bb.4: ; %END
 ; GFX9-W64-NEXT:    s_or_b64 exec, exec, s[0:1]
 ; GFX9-W64-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-W64-NEXT:    ; return to shader part epilog
+; GFX9-W64-NEXT:    s_branch .LBB26_5
+; GFX9-W64-NEXT:  .LBB26_5:
 ;
 ; GFX10-W32-LABEL: test_control_flow_3:
 ; GFX10-W32:       ; %bb.0: ; %main_body
 ; GFX10-W32-NEXT:    s_mov_b32 s12, exec_lo
 ; GFX10-W32-NEXT:    s_wqm_b32 exec_lo, exec_lo
 ; GFX10-W32-NEXT:    image_sample v1, v1, s[0:7], s[8:11] dmask:0x1 dim:SQ_RSRC_IMG_1D
 ; GFX10-W32-NEXT:    s_and_b32 exec_lo, exec_lo, s12
 ; GFX10-W32-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-W32-NEXT:    image_sample v1, v1, s[0:7], s[8:11] dmask:0x1 dim:SQ_RSRC_IMG_1D
 ; GFX10-W32-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-W32-NEXT:    buffer_store_dword v1, v0, s[0:3], 0 idxen
 ; GFX10-W32-NEXT:    s_mov_b32 s0, exec_lo
 ; GFX10-W32-NEXT:    ; implicit-def: $vgpr0
 ; GFX10-W32-NEXT:    v_cmpx_nlt_f32_e32 0, v1
 ; GFX10-W32-NEXT:    s_xor_b32 s0, exec_lo, s0
-; GFX10-W32-NEXT:  ; %bb.1: ; %ELSE
+; GFX10-W32-NEXT:    s_cbranch_execnz .LBB26_3
+; GFX10-W32-NEXT:  ; %bb.1: ; %Flow
+; GFX10-W32-NEXT:    s_andn2_saveexec_b32 s0, s0
+; GFX10-W32-NEXT:    s_cbranch_execnz .LBB26_4
+; GFX10-W32-NEXT:  .LBB26_2: ; %END
+; GFX10-W32-NEXT:    s_or_b32 exec_lo, exec_lo, s0
+; GFX10-W32-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-W32-NEXT:    s_branch .LBB26_5
+; GFX10-W32-NEXT:  .LBB26_3: ; %ELSE
 ; GFX10-W32-NEXT:    v_mul_f32_e32 v0, 4.0, v1
 ; GFX10-W32-NEXT:    ; implicit-def: $vgpr1
-; GFX10-W32-NEXT:  ; %bb.2: ; %Flow
-; GFX10-W32-NEXT:    s_or_saveexec_b32 s0, s0
-; GFX10-W32-NEXT:    s_xor_b32 exec_lo, exec_lo, s0
-; GFX10-W32-NEXT:  ; %bb.3: ; %IF
+; GFX10-W32-NEXT:    s_andn2_saveexec_b32 s0, s0
+; GFX10-W32-NEXT:    s_cbranch_execz .LBB26_2
+; GFX10-W32-NEXT:  .LBB26_4: ; %IF
 ; GFX10-W32-NEXT:    v_mul_f32_e32 v0, 0x40400000, v1
-; GFX10-W32-NEXT:  ; %bb.4: ; %END
 ; GFX10-W32-NEXT:    s_or_b32 exec_lo, exec_lo, s0
 ; GFX10-W32-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-W32-NEXT:    ; return to shader part epilog
+; GFX10-W32-NEXT:    s_branch .LBB26_5
+; GFX10-W32-NEXT:  .LBB26_5:
 main_body:
   %tex = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %coord, <8 x i32> %rsrc, <4 x i32> %sampler, i1 false, i32 0, i32 0) #0
   %tex0 = extractelement <4 x float> %tex, i32 0
   %dtex = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %tex0, <8 x i32> %rsrc, <4 x i32> %sampler, i1 false, i32 0, i32 0) #0
   %dtex.1 = extractelement <4 x float> %dtex, i32 0
   call void @llvm.amdgcn.struct.buffer.store.f32(float %dtex.1, <4 x i32> undef, i32 %idx, i32 0, i32 0, i32 0)
 
   %cc = fcmp ogt float %dtex.1, 0.0