[AMDGPU] Avoid use of V_READLANE into EXEC in SGPR spills

[critson]

Summary

Depends on D81224.
Always prefer to clobber input SGPRs and restore them after the
spill. This applies to both spills to VGPRs and scratch.

[arsenm]

Can you try to constrain the class of the virtual register earlier to avoid this from happening instead? I think that should avoid this coming up (e.g. we do MRI.constrainRegClass(SrcReg, &AMDGPU::SReg_32_XM0RegClass) already to try avoiding m0 spills, plus in foldMemoryOperandImpl to avoid another problem when spills are optimized out)

llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp

871

I'm pretty sure this used to be true, so this should depend on the subtarget?

[critson]

Update spill-special-sgpr.mir for auto updated version.

[critson]

In D81914#2095875, @arsenm wrote:

Can you try to constrain the class of the virtual register earlier to avoid this from happening instead? I think that should avoid this coming up (e.g. we do MRI.constrainRegClass(SrcReg, &AMDGPU::SReg_32_XM0RegClass) already to try avoiding m0 spills, plus in foldMemoryOperandImpl to avoid another problem when spills are optimized out)

I could look at that, but it seems advantageous to be able to spill and restore EXEC directly when we at the point of spilling SGPRs?

llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp

871

If we can establish at what point it became unsupported, but it seems it does not work on anything recent.
I have verified that it does not work on gfx900 or gfx1010.
If I get a chance I will test gfx7.

[critson]

Fix error in test update.

[arsenm]

In D81914#2097845, @critson wrote:

In D81914#2095875, @arsenm wrote:

Can you try to constrain the class of the virtual register earlier to avoid this from happening instead? I think that should avoid this coming up (e.g. we do MRI.constrainRegClass(SrcReg, &AMDGPU::SReg_32_XM0RegClass) already to try avoiding m0 spills, plus in foldMemoryOperandImpl to avoid another problem when spills are optimized out)

I could look at that, but it seems advantageous to be able to spill and restore EXEC directly when we at the point of spilling SGPRs?

As a reserved register, ppilling of exec should be impossible. The only situations where it "spills" are when a copy was coalesced of another register that spilled. We should be able to satisfy everything by register class constraints. Even if that weren't the case, I would prefer to proceed by first constraining the register classes, and then applying the more complex fix if it turns out there are still cases that can't be avoided

[critson]

This is the simplified version which blocks spill/reload of exec from occuring.

[arsenm]

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

7036–7037

This is now way more complicated. You shouldn't need to loop, or consider specific physical registers. You should be apply to just change the register classes that were used here

[arsenm]

I also think this is missing the test to stress foldMemoryOperandImpl; Can you add one following the example of the test added in d9e0a2942ac71327166a3a597e8383192fd19b17?

[critson]

• Add test for foldMemoryOperandImpl
• Rework code in foldMemoryOperandImpl to remove references to specific physical registers

[arsenm]

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

7038–7041

This shouldn't consider the register class of the physical register. Only the register class of the virtual register matters (plus getPhysRegClass is really slow), and you shouldn't need to get the size and check if it's an SGPR.

Something like:
RC = MRI.getRegClass(VirtReg);
if (RC->contains(m0) || RC->contains(exec) ...)

or alternatively
RC->hasSuperClassEq(AMDGPU::SReg_64RegClass)
...
RC->hasSuperClassEq(AMDGPU::SReg_32RegClass)

[critson]

Address comments on foldMemoryOperandImpl.

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

void SIInstrInfo::storeRegToStackSlot(MachineBasicBlock &MBB,
   MachineMemOperand *MMO = MF->getMachineMemOperand(
       PtrInfo, MachineMemOperand::MOStore, FrameInfo.getObjectSize(FrameIndex),
       FrameInfo.getObjectAlign(FrameIndex));
   unsigned SpillSize = TRI->getSpillSize(*RC);
 
   if (RI.isSGPRClass(RC)) {
     MFI->setHasSpilledSGPRs();
     assert(SrcReg != AMDGPU::M0 && "m0 should not be spilled");
+    assert(SrcReg != AMDGPU::EXEC_LO && SrcReg != AMDGPU::EXEC_HI &&
+           SrcReg != AMDGPU::EXEC && "exec should not be spilled");
 
     // We are only allowed to create one new instruction when spilling
     // registers, so we need to use pseudo instruction for spilling SGPRs.
     const MCInstrDesc &OpDesc = get(getSGPRSpillSaveOpcode(SpillSize));
 
     // The SGPR spill/restore instructions only work on number sgprs, so we need
     // to make sure we are using the correct register class.
     if (Register::isVirtualRegister(SrcReg) && SpillSize == 4) {
       MachineRegisterInfo &MRI = MF->getRegInfo();
-      MRI.constrainRegClass(SrcReg, &AMDGPU::SReg_32_XM0RegClass);
+      MRI.constrainRegClass(SrcReg, &AMDGPU::SReg_32_XM0_XEXECRegClass);
     }
 
     BuildMI(MBB, MI, DL, OpDesc)
       .addReg(SrcReg, getKillRegState(isKill)) // data
       .addFrameIndex(FrameIndex)               // addr
       .addMemOperand(MMO)
       .addReg(MFI->getScratchRSrcReg(), RegState::Implicit)
       .addReg(MFI->getStackPtrOffsetReg(), RegState::Implicit);
void SIInstrInfo::loadRegFromStackSlot(MachineBasicBlock &MBB,
 
   MachineMemOperand *MMO = MF->getMachineMemOperand(
       PtrInfo, MachineMemOperand::MOLoad, FrameInfo.getObjectSize(FrameIndex),
       FrameInfo.getObjectAlign(FrameIndex));
 
   if (RI.isSGPRClass(RC)) {
     MFI->setHasSpilledSGPRs();
     assert(DestReg != AMDGPU::M0 && "m0 should not be reloaded into");
+    assert(DestReg != AMDGPU::EXEC_LO && DestReg != AMDGPU::EXEC_HI &&
+           DestReg != AMDGPU::EXEC && "exec should not be spilled");
 
     // FIXME: Maybe this should not include a memoperand because it will be
     // lowered to non-memory instructions.
     const MCInstrDesc &OpDesc = get(getSGPRSpillRestoreOpcode(SpillSize));
     if (DestReg.isVirtual() && SpillSize == 4) {
       MachineRegisterInfo &MRI = MF->getRegInfo();
-      MRI.constrainRegClass(DestReg, &AMDGPU::SReg_32_XM0RegClass);
+      MRI.constrainRegClass(DestReg, &AMDGPU::SReg_32_XM0_XEXECRegClass);
     }
 
     if (RI.spillSGPRToVGPR())
       FrameInfo.setStackID(FrameIndex, TargetStackID::SGPRSpill);
     BuildMI(MBB, MI, DL, OpDesc, DestReg)
       .addFrameIndex(FrameIndex) // addr
       .addMemOperand(MMO)
       .addReg(MFI->getScratchRSrcReg(), RegState::Implicit)
MachineInstr *SIInstrInfo::foldMemoryOperandImpl(
   //
   //   %0:sreg_32 = COPY $m0
   //
   // We explicitly chose SReg_32 for the virtual register so such a copy might
   // be eliminated by RegisterCoalescer. However, that may not be possible, and
   // %0 may even spill. We can't spill $m0 normally (it would require copying to
   // a numbered SGPR anyway), and since it is in the SReg_32 register class,
   // TargetInstrInfo::foldMemoryOperand() is going to try.
+  // A similar issue also exists with spilling and reloading $exec registers.
   //
   // To prevent that, constrain the %0 register class here.
   if (MI.isFullCopy()) {
     Register DstReg = MI.getOperand(0).getReg();
     Register SrcReg = MI.getOperand(1).getReg();
-
-    if (DstReg == AMDGPU::M0 && SrcReg.isVirtual()) {
-      MF.getRegInfo().constrainRegClass(SrcReg, &AMDGPU::SReg_32_XM0RegClass);
+    if ((DstReg.isVirtual() || SrcReg.isVirtual()) &&
+        (DstReg.isVirtual() != SrcReg.isVirtual())) {
+      MachineRegisterInfo &MRI = MF.getRegInfo();
+      Register VirtReg = DstReg.isVirtual() ? DstReg : SrcReg;
+      const TargetRegisterClass *RC = MRI.getRegClass(VirtReg);
+      if (RC->hasSuperClassEq(&AMDGPU::SReg_32RegClass)) {
+        MRI.constrainRegClass(VirtReg, &AMDGPU::SReg_32_XM0_XEXECRegClass);
         return nullptr;
-    }
-
-    if (SrcReg == AMDGPU::M0 && DstReg.isVirtual()) {
-      MF.getRegInfo().constrainRegClass(DstReg, &AMDGPU::SReg_32_XM0RegClass);
+      } else if (RC->hasSuperClassEq(&AMDGPU::SReg_64RegClass)) {
+        MRI.constrainRegClass(VirtReg, &AMDGPU::SReg_64_XEXECRegClass);
         return nullptr;
       }
     }
+  }
 
   return nullptr;
 }
 
 unsigned SIInstrInfo::getInstrLatency(const InstrItineraryData *ItinData,
                                       const MachineInstr &MI,
                                       unsigned *PredCost) const {
   if (MI.isBundle()) {

llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp

void SIRegisterInfo::buildSpillLoadStore(MachineBasicBlock::iterator MI,
   if (ScratchOffsetRegDelta != 0) {
     // Subtract the offset we added to the ScratchOffset register.
     BuildMI(*MBB, MI, DL, TII->get(AMDGPU::S_SUB_U32), SOffset)
         .addReg(SOffset)
         .addImm(ScratchOffsetRegDelta);
   }
 }
 
 // Generate a VMEM access which loads or stores the VGPR containing an SGPR
 // spill such that all the lanes set in VGPRLanes are loaded or stored.
 // This generates exec mask manipulation and will use SGPRs available in MI
 // or VGPR lanes in the VGPR to save and restore the exec mask.
 void SIRegisterInfo::buildSGPRSpillLoadStore(MachineBasicBlock::iterator MI,
                                              int Index, int Offset,
                                              unsigned EltSize, Register VGPR,
                                              int64_t VGPRLanes,
                                              RegScavenger *RS,
                                              bool IsLoad) const {
   MachineBasicBlock *MBB = MI->getParent();
   MachineFunction *MF = MBB->getParent();
   SIMachineFunctionInfo *MFI = MF->getInfo<SIMachineFunctionInfo>();
   const SIInstrInfo *TII = ST.getInstrInfo();
 
   Register SuperReg = MI->getOperand(0).getReg();
   const TargetRegisterClass *RC = getPhysRegClass(SuperReg);
   ArrayRef<int16_t> SplitParts = getRegSplitParts(RC, EltSize);
   unsigned NumSubRegs = SplitParts.empty() ? 1 : SplitParts.size();
-  unsigned FirstPart = isWave32 ? Offset * 16 : Offset * 32;
+  unsigned FirstPart = Offset * 32;
+  unsigned ExecLane = 0;
 
   bool IsKill = MI->getOperand(0).isKill();
   const DebugLoc &DL = MI->getDebugLoc();
 
-  const bool SuperRegIsExec =
-      SuperReg == AMDGPU::EXEC || SuperReg == AMDGPU::EXEC_LO;
-
-  // If exec mask is stored in the VGPR, make sure it is stored after
-  // any lanes used by the spill (16 lanes on Wave32, 32 lanes on Wave64).
-  const unsigned ExecLoLane = SuperRegIsExec ? 0 : (isWave32 ? 16 : 32);
-  const unsigned ExecHiLane = SuperRegIsExec ? 1 : (isWave32 ? 17 : 33);
-
-  // Try to use the src/dst SGPRs to hold a copy of the exec mask.
-  // Use VGPR lanes when this is not possible, i.e. the src value
-  // must be valid after the spill or src is smaller than exec mask.
-  bool StoreExecInVGPR = !IsLoad && (SuperRegIsExec || !IsKill);
+  // Cannot handle load/store to EXEC
+  assert(SuperReg != AMDGPU::EXEC_LO && SuperReg != AMDGPU::EXEC_HI &&
+         SuperReg != AMDGPU::EXEC && "exec should never spill");
 
   // On Wave32 only handle EXEC_LO.
   // On Wave64 only update EXEC_HI if there is sufficent space for a copy.
-  bool OnlyExecLo = isWave32 || NumSubRegs == 1;
+  bool OnlyExecLo = isWave32 || NumSubRegs == 1 || SuperReg == AMDGPU::EXEC_HI;
 
   unsigned ExecMovOpc = OnlyExecLo ? AMDGPU::S_MOV_B32 : AMDGPU::S_MOV_B64;
   Register ExecReg = OnlyExecLo ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
   Register SavedExecReg;
 
   // Backup EXEC
-  if (SuperRegIsExec) {
-    // Do nothing; exec is already stored in VGPR or will be overwritten
-  } else if (StoreExecInVGPR) {
-    BuildMI(*MBB, MI, DL, TII->getMCOpcodeFromPseudo(AMDGPU::V_WRITELANE_B32),
-            VGPR)
-        .addReg(AMDGPU::EXEC_LO)
-        .addImm(ExecLoLane)
-        .addReg(VGPR, getUndefRegState(IsLoad));
-
-    if (!isWave32) {
-      BuildMI(*MBB, MI, DL, TII->getMCOpcodeFromPseudo(AMDGPU::V_WRITELANE_B32),
-              VGPR)
-          .addReg(AMDGPU::EXEC_HI)
-          .addImm(ExecHiLane)
-          .addReg(VGPR);
-    }
-  } else {
-    if (OnlyExecLo) {
-      SavedExecReg = NumSubRegs == 1
-                         ? SuperReg
-                         : getSubReg(SuperReg, SplitParts[FirstPart]);
-    } else {
-      SavedExecReg =
-          getMatchingSuperReg(getSubReg(SuperReg, SplitParts[FirstPart]),
-                              AMDGPU::sub0, &AMDGPU::SReg_64_XEXECRegClass);
-      // If src/dst is an odd size it is possible subreg0 is not aligned.
-      if (!SavedExecReg && NumSubRegs > 2)
-        SavedExecReg =
-            getMatchingSuperReg(getSubReg(SuperReg, SplitParts[FirstPart + 1]),
-                                AMDGPU::sub0, &AMDGPU::SReg_64_XEXECRegClass);
-    }
-
-    assert(SavedExecReg);
-    BuildMI(*MBB, MI, DL, TII->get(ExecMovOpc), SavedExecReg).addReg(ExecReg);
-  }
+                       : getSubReg(SuperReg, SplitParts[FirstPart + ExecLane]);
+    for (; ExecLane < (NumSubRegs - 1); ++ExecLane) {
+      SavedExecReg = getMatchingSuperReg(
+          getSubReg(SuperReg, SplitParts[FirstPart + ExecLane]), AMDGPU::sub0,
+          &AMDGPU::SReg_64_XEXECRegClass);
+      if (SavedExecReg)
+        break;
+    }
 
   // Setup EXEC
   BuildMI(*MBB, MI, DL, TII->get(ExecMovOpc), ExecReg).addImm(VGPRLanes);
 
   // Load/store VGPR
   MachineFrameInfo &FrameInfo = MF->getFrameInfo();
   assert(FrameInfo.getStackID(Index) != TargetStackID::SGPRSpill);
 
void SIRegisterInfo::buildSGPRSpillLoadStore(MachineBasicBlock::iterator MI,
   if (IsLoad) {
     buildSpillLoadStore(MI, AMDGPU::BUFFER_LOAD_DWORD_OFFSET,
           Index,
           VGPR, false,
           MFI->getScratchRSrcReg(), FrameReg,
           Offset * EltSize, MMO,
           RS);
   } else {
-    buildSpillLoadStore(MI, AMDGPU::BUFFER_STORE_DWORD_OFFSET,
-          Index,
-          VGPR, !StoreExecInVGPR,
-          MFI->getScratchRSrcReg(), FrameReg,
-          Offset * EltSize, MMO,
-          RS);
+    buildSpillLoadStore(MI, AMDGPU::BUFFER_STORE_DWORD_OFFSET, Index, VGPR,
+                        IsKill, MFI->getScratchRSrcReg(), FrameReg,
+                        Offset * EltSize, MMO, RS);
     // This only ever adds one VGPR spill
     MFI->addToSpilledVGPRs(1);
   }
 
   // Restore EXEC
-  if (SuperRegIsExec && IsLoad) {
-    // Do nothing; exec will be overwritten
-  } else if (StoreExecInVGPR) {
-    BuildMI(*MBB, MI, DL, TII->getMCOpcodeFromPseudo(AMDGPU::V_READLANE_B32),
-            AMDGPU::EXEC_LO)
-        .addReg(VGPR, getKillRegState(!IsLoad && isWave32))
-        .addImm(ExecLoLane);
-    if (!isWave32) {
+  BuildMI(*MBB, MI, DL, TII->get(ExecMovOpc), ExecReg)
+      .addReg(SavedExecReg, getKillRegState(IsLoad || IsKill));
+
+  // Restore clobbered SGPRs
+  if (IsLoad) {
+    // Nothing to do; register will be overwritten
+  } else if (!IsKill) {
+    // Restore SGPRs from appropriate VGPR lanes
+    if (!OnlyExecLo) {
       BuildMI(*MBB, MI, DL, TII->getMCOpcodeFromPseudo(AMDGPU::V_READLANE_B32),
-              AMDGPU::EXEC_HI)
-          .addReg(VGPR, getKillRegState(!IsLoad))
-          .addImm(ExecHiLane);
+              getSubReg(SuperReg, SplitParts[FirstPart + ExecLane + 1]))
+          .addReg(VGPR)
+          .addImm(ExecLane + 1);
     }
-  } else {
-    assert(SavedExecReg);
-    BuildMI(*MBB, MI, DL, TII->get(ExecMovOpc), ExecReg)
-        .addReg(SavedExecReg, RegState::Kill);
+    BuildMI(*MBB, MI, DL, TII->getMCOpcodeFromPseudo(AMDGPU::V_READLANE_B32),
+            NumSubRegs == 1
+                ? SavedExecReg
+                : getSubReg(SuperReg, SplitParts[FirstPart + ExecLane]))
+        .addReg(VGPR, RegState::Kill)
+        .addImm(ExecLane);
   }
 }
 
 bool SIRegisterInfo::spillSGPR(MachineBasicBlock::iterator MI,
                                int Index,
                                RegScavenger *RS,
                                bool OnlyToVGPR) const {
   MachineBasicBlock *MBB = MI->getParent();
bool SIRegisterInfo::spillSGPR(MachineBasicBlock::iterator MI,
   Register SuperReg = MI->getOperand(0).getReg();
   bool IsKill = MI->getOperand(0).isKill();
   const DebugLoc &DL = MI->getDebugLoc();
 
   assert(SpillToVGPR || (SuperReg != MFI->getStackPtrOffsetReg() &&
                          SuperReg != MFI->getFrameOffsetReg()));
 
   assert(SuperReg != AMDGPU::M0 && "m0 should never spill");
+  assert(SuperReg != AMDGPU::EXEC_LO && SuperReg != AMDGPU::EXEC_HI &&
+         SuperReg != AMDGPU::EXEC && "exec should never spill");
 
   unsigned EltSize = 4;
   const TargetRegisterClass *RC = getPhysRegClass(SuperReg);
 
   ArrayRef<int16_t> SplitParts = getRegSplitParts(RC, EltSize);
   unsigned NumSubRegs = SplitParts.empty() ? 1 : SplitParts.size();
 
   if (SpillToVGPR) {
for (unsigned i = 0, e = NumSubRegs; i < e; ++i) {
       // FIXME: Since this spills to another register instead of an actual
       // frame index, we should delete the frame index when all references to
       // it are fixed.
     }
   } else {
     // Scavenged temporary VGPR to use. It must be scavenged once for any number
     // of spilled subregs.
     Register TmpVGPR = RS->scavengeRegister(&AMDGPU::VGPR_32RegClass, MI, 0);
+    RS->setRegUsed(TmpVGPR);
 
     // SubReg carries the "Kill" flag when SubReg == SuperReg.
     unsigned SubKillState = getKillRegState((NumSubRegs == 1) && IsKill);
 
-    unsigned PerVGPR = isWave32 ? 16 : 32;
+    unsigned PerVGPR = 32;
     unsigned NumVGPRs = (NumSubRegs + (PerVGPR - 1)) / PerVGPR;
     int64_t VGPRLanes = (1LL << std::min(PerVGPR, NumSubRegs)) - 1LL;
 
     for (unsigned Offset = 0; Offset < NumVGPRs; ++Offset) {
       unsigned TmpVGPRFlags = RegState::Undef;
 
       // Write sub registers into the VGPR
       for (unsigned i = Offset * PerVGPR,
if (OnlyToVGPR && !SpillToVGPR)
     return false;
 
   const SIInstrInfo *TII = ST.getInstrInfo();
   const DebugLoc &DL = MI->getDebugLoc();
 
   Register SuperReg = MI->getOperand(0).getReg();
 
   assert(SuperReg != AMDGPU::M0 && "m0 should never spill");
+  assert(SuperReg != AMDGPU::EXEC_LO && SuperReg != AMDGPU::EXEC_HI &&
+         SuperReg != AMDGPU::EXEC && "exec should never spill");
 
   unsigned EltSize = 4;
 
   const TargetRegisterClass *RC = getPhysRegClass(SuperReg);
 
   ArrayRef<int16_t> SplitParts = getRegSplitParts(RC, EltSize);
   unsigned NumSubRegs = SplitParts.empty() ? 1 : SplitParts.size();
 
   if (SpillToVGPR) {
     for (unsigned i = 0, e = NumSubRegs; i < e; ++i) {
       Register SubReg =
           NumSubRegs == 1 ? SuperReg : getSubReg(SuperReg, SplitParts[i]);
 
       SIMachineFunctionInfo::SpilledReg Spill = VGPRSpills[i];
       auto MIB =
         BuildMI(*MBB, MI, DL, TII->getMCOpcodeFromPseudo(AMDGPU::V_READLANE_B32),
                 SubReg)
         .addReg(Spill.VGPR)
         .addImm(Spill.Lane);
-
       if (NumSubRegs > 1 && i == 0)
         MIB.addReg(SuperReg, RegState::ImplicitDefine);
     }
   } else {
     Register TmpVGPR = RS->scavengeRegister(&AMDGPU::VGPR_32RegClass, MI, 0);
+    RS->setRegUsed(TmpVGPR);
 
-    unsigned PerVGPR = isWave32 ? 16 : 32;
+    unsigned PerVGPR = 32;
     unsigned NumVGPRs = (NumSubRegs + (PerVGPR - 1)) / PerVGPR;
     int64_t VGPRLanes = (1LL << std::min(PerVGPR, NumSubRegs)) - 1LL;
 
     for (unsigned Offset = 0; Offset < NumVGPRs; ++Offset) {
       // Load in VGPR data
       buildSGPRSpillLoadStore(MI, Index, Offset, EltSize, TmpVGPR, VGPRLanes,
                               RS, true);
 
       // Unpack lanes
       for (unsigned i = Offset * PerVGPR,
                     e = std::min((Offset + 1) * PerVGPR, NumSubRegs);
            i < e; ++i) {
         Register SubReg =
             NumSubRegs == 1 ? SuperReg : getSubReg(SuperReg, SplitParts[i]);
 
         bool LastSubReg = (i + 1 == e);
         auto MIB =
             BuildMI(*MBB, MI, DL,
                     TII->getMCOpcodeFromPseudo(AMDGPU::V_READLANE_B32), SubReg)
                 .addReg(TmpVGPR, getKillRegState(LastSubReg))
                 .addImm(i);
-
         if (NumSubRegs > 1 && i == 0)
           MIB.addReg(SuperReg, RegState::ImplicitDefine);
       }
     }
   }
 
   MI->eraseFromParent();
   return true;

llvm/test/CodeGen/AMDGPU/fold-reload-into-exec.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -verify-machineinstrs -stress-regalloc=2 -start-before=greedy -stop-after=virtregmap -o - %s | FileCheck %s
+
+# Test that a spill of a copy of exec is not folded to be a spill of exec directly.
+
+---
+
+name:            merge_sgpr_spill_into_copy_from_exec_lo
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    ; CHECK-LABEL: name: merge_sgpr_spill_into_copy_from_exec_lo
+    ; CHECK: liveins: $vgpr0
+    ; CHECK: S_WAITCNT 0
+    ; CHECK: S_NOP 0, implicit-def $exec_lo
+    ; CHECK: $sgpr0 = S_MOV_B32 $exec_lo
+    ; CHECK: $vgpr0 = V_WRITELANE_B32_vi killed $sgpr0, 0, undef $vgpr0
+    ; CHECK: $sgpr0 = V_READLANE_B32_vi $vgpr0, 0
+    ; CHECK: S_NOP 0, implicit-def dead renamable $sgpr1, implicit-def dead renamable $sgpr0, implicit killed renamable $sgpr0
+    ; CHECK: $sgpr0 = V_READLANE_B32_vi killed $vgpr0, 0
+    ; CHECK: $exec_lo = S_MOV_B32 killed $sgpr0
+    ; CHECK: S_SENDMSG 0, implicit $m0, implicit $exec
+    S_NOP 0, implicit-def $exec_lo
+    %0:sreg_32 = COPY $exec_lo
+    S_NOP 0, implicit-def %1:sreg_32, implicit-def %2:sreg_32, implicit %0
+    $exec_lo = COPY %0
+    S_SENDMSG 0, implicit $m0, implicit $exec
+
+...
+---
+
+name:            merge_sgpr_spill_into_copy_from_exec_hi
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    ; CHECK-LABEL: name: merge_sgpr_spill_into_copy_from_exec_hi
+    ; CHECK: liveins: $vgpr0
+    ; CHECK: S_WAITCNT 0
+    ; CHECK: S_NOP 0, implicit-def $exec_hi
+    ; CHECK: $sgpr0 = S_MOV_B32 $exec_hi
+    ; CHECK: $vgpr0 = V_WRITELANE_B32_vi killed $sgpr0, 0, undef $vgpr0
+    ; CHECK: $sgpr0 = V_READLANE_B32_vi $vgpr0, 0
+    ; CHECK: S_NOP 0, implicit-def dead renamable $sgpr1, implicit-def dead renamable $sgpr0, implicit killed renamable $sgpr0
+    ; CHECK: $sgpr0 = V_READLANE_B32_vi killed $vgpr0, 0
+    ; CHECK: $exec_hi = S_MOV_B32 killed $sgpr0
+    ; CHECK: S_SENDMSG 0, implicit $m0, implicit $exec
+    S_NOP 0, implicit-def $exec_hi
+    %0:sreg_32 = COPY $exec_hi
+    S_NOP 0, implicit-def %1:sreg_32, implicit-def %2:sreg_32, implicit %0
+    $exec_hi = COPY %0
+    S_SENDMSG 0, implicit $m0, implicit $exec
+
+...
+---
+
+name:            merge_sgpr_spill_into_copy_from_exec
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    ; CHECK-LABEL: name: merge_sgpr_spill_into_copy_from_exec
+    ; CHECK: liveins: $vgpr0
+    ; CHECK: S_WAITCNT 0
+    ; CHECK: S_NOP 0, implicit-def $exec
+    ; CHECK: $sgpr0_sgpr1 = S_MOV_B64 $exec
+    ; CHECK: $vgpr0 = V_WRITELANE_B32_vi killed $sgpr0, 0, undef $vgpr0
+    ; CHECK: $vgpr0 = V_WRITELANE_B32_vi killed $sgpr1, 1, killed $vgpr0
+    ; CHECK: $sgpr0 = V_READLANE_B32_vi $vgpr0, 0, implicit-def $sgpr0_sgpr1
+    ; CHECK: $sgpr1 = V_READLANE_B32_vi $vgpr0, 1
+    ; CHECK: S_NOP 0, implicit-def dead renamable $sgpr2_sgpr3, implicit-def dead renamable $sgpr0_sgpr1, implicit killed renamable $sgpr0_sgpr1
+    ; CHECK: $sgpr0 = V_READLANE_B32_vi $vgpr0, 0, implicit-def $sgpr0_sgpr1
+    ; CHECK: $sgpr1 = V_READLANE_B32_vi killed $vgpr0, 1
+    ; CHECK: $exec = S_MOV_B64 killed $sgpr0_sgpr1
+    ; CHECK: S_SENDMSG 0, implicit $m0, implicit $exec
+    S_NOP 0, implicit-def $exec
+    %0:sreg_64 = COPY $exec
+    S_NOP 0, implicit-def %1:sreg_64, implicit-def %2:sreg_64, implicit %0
+    $exec = COPY %0
+    S_SENDMSG 0, implicit $m0, implicit $exec
+
+...
+
+# Test that a reload into a copy of exec is not folded to be a reload of exec directly.
+
+---
+
+name:            reload_sgpr_spill_into_copy_to_exec_lo
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    ; CHECK-LABEL: name: reload_sgpr_spill_into_copy_to_exec_lo
+    ; CHECK: liveins: $vgpr0
+    ; CHECK: S_WAITCNT 0
+    ; CHECK: S_NOP 0, implicit-def renamable $sgpr0, implicit-def dead renamable $sgpr1, implicit-def $exec_lo
+    ; CHECK: $vgpr0 = V_WRITELANE_B32_vi killed $sgpr0, 0, undef $vgpr0
+    ; CHECK: $sgpr0 = V_READLANE_B32_vi $vgpr0, 0
+    ; CHECK: S_NOP 0, implicit killed renamable $sgpr0, implicit-def dead renamable $sgpr1, implicit-def dead renamable $sgpr0
+    ; CHECK: $sgpr0 = V_READLANE_B32_vi killed $vgpr0, 0
+    ; CHECK: $exec_lo = S_MOV_B32 killed $sgpr0
+    ; CHECK: S_SENDMSG 0, implicit $m0, implicit $exec
+    S_NOP 0, implicit-def %0:sreg_32, implicit-def %1:sreg_32, implicit-def $exec_lo
+    S_NOP 0, implicit %0, implicit-def %3:sreg_32, implicit-def %4:sreg_32
+    $exec_lo = COPY %0
+    S_SENDMSG 0, implicit $m0, implicit $exec
+
+...
+---
+
+name:            reload_sgpr_spill_into_copy_to_exec_hi
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    ; CHECK-LABEL: name: reload_sgpr_spill_into_copy_to_exec_hi
+    ; CHECK: liveins: $vgpr0
+    ; CHECK: S_WAITCNT 0
+    ; CHECK: S_NOP 0, implicit-def renamable $sgpr0, implicit-def dead renamable $sgpr1, implicit-def $exec_hi
+    ; CHECK: $vgpr0 = V_WRITELANE_B32_vi killed $sgpr0, 0, undef $vgpr0
+    ; CHECK: $sgpr0 = V_READLANE_B32_vi $vgpr0, 0
+    ; CHECK: S_NOP 0, implicit killed renamable $sgpr0, implicit-def dead renamable $sgpr1, implicit-def dead renamable $sgpr0
+    ; CHECK: $sgpr0 = V_READLANE_B32_vi killed $vgpr0, 0
+    ; CHECK: $exec_hi = S_MOV_B32 killed $sgpr0
+    ; CHECK: S_SENDMSG 0, implicit $m0, implicit $exec
+    S_NOP 0, implicit-def %0:sreg_32, implicit-def %1:sreg_32, implicit-def $exec_hi
+    S_NOP 0, implicit %0, implicit-def %3:sreg_32, implicit-def %4:sreg_32
+    $exec_hi = COPY %0
+    S_SENDMSG 0, implicit $m0, implicit $exec
+
+...
+---
+
+name:            reload_sgpr_spill_into_copy_to_exec
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    ; CHECK-LABEL: name: reload_sgpr_spill_into_copy_to_exec
+    ; CHECK: liveins: $vgpr0
+    ; CHECK: S_WAITCNT 0
+    ; CHECK: S_NOP 0, implicit-def renamable $sgpr0_sgpr1, implicit-def dead renamable $sgpr2_sgpr3, implicit-def $exec
+    ; CHECK: $vgpr0 = V_WRITELANE_B32_vi killed $sgpr0, 0, undef $vgpr0
+    ; CHECK: $vgpr0 = V_WRITELANE_B32_vi killed $sgpr1, 1, killed $vgpr0
+    ; CHECK: $sgpr0 = V_READLANE_B32_vi $vgpr0, 0, implicit-def $sgpr0_sgpr1
+    ; CHECK: $sgpr1 = V_READLANE_B32_vi $vgpr0, 1
+    ; CHECK: S_NOP 0, implicit killed renamable $sgpr0_sgpr1, implicit-def dead renamable $sgpr2_sgpr3, implicit-def dead renamable $sgpr0_sgpr1
+    ; CHECK: $sgpr0 = V_READLANE_B32_vi $vgpr0, 0, implicit-def $sgpr0_sgpr1
+    ; CHECK: $sgpr1 = V_READLANE_B32_vi killed $vgpr0, 1
+    ; CHECK: $exec = S_MOV_B64 killed $sgpr0_sgpr1
+    ; CHECK: S_SENDMSG 0, implicit $m0, implicit $exec
+    S_NOP 0, implicit-def %0:sreg_64, implicit-def %1:sreg_64, implicit-def $exec
+    S_NOP 0, implicit %0, implicit-def %3:sreg_64, implicit-def %4:sreg_64
+    $exec = COPY %0
+    S_SENDMSG 0, implicit $m0, implicit $exec
+
+...

llvm/test/CodeGen/AMDGPU/sgpr-spill.mir

 # RUN: llc -mtriple=amdgcn -mcpu=gfx900 -verify-machineinstrs -run-pass=prologepilog %s -o - | FileCheck -check-prefix=CHECK -check-prefix=GCN64 %s
 # RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -verify-machineinstrs -run-pass=prologepilog %s -o - | FileCheck -check-prefix=CHECK -check-prefix=GCN32 %s
 
 
 # CHECK-LABEL: name: check_spill
 
 # S32 with kill
 # CHECK: V_WRITELANE
 # CHECK: $sgpr12 = S_MOV_B32 $exec_lo
 # CHECK: $exec_lo = S_MOV_B32 1
-# CHECK: BUFFER_STORE_DWORD_OFFSET {{(killed )?}}$vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 4
+# CHECK: BUFFER_STORE_DWORD_OFFSET killed $vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 4
 # CHECK: $exec_lo = S_MOV_B32 killed $sgpr12
 
 # S32 without kill
 # CHECK: V_WRITELANE
-# CHECK: V_WRITELANE
+# CHECK: $sgpr12 = S_MOV_B32 $exec_lo
 # CHECK: $exec_lo = S_MOV_B32 1
-# CHECK: BUFFER_STORE_DWORD_OFFSET {{(killed )?}}$vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 4
-# CHECK: $exec_lo = V_READLANE
+# CHECK: BUFFER_STORE_DWORD_OFFSET $vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 4
+# CHECK: $sgpr12 = V_READLANE
 
 # S64 with kill
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # GCN32: $sgpr12 = S_MOV_B32 $exec_lo
 # GCN64: $sgpr12_sgpr13 = S_MOV_B64 $exec
 # GCN32: $exec_lo = S_MOV_B32 3
 # GCN64: $exec = S_MOV_B64 3
-# CHECK: BUFFER_STORE_DWORD_OFFSET {{(killed )?}}$vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 8
+# CHECK: BUFFER_STORE_DWORD_OFFSET killed $vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 8
 # GCN32: $exec_lo = S_MOV_B32 killed $sgpr12
 # GCN64: $exec = S_MOV_B64 killed $sgpr12_sgpr13
 
 # S64 without kill
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
-# CHECK: V_WRITELANE
-# GCN64: V_WRITELANE
+# GCN32: $sgpr12 = S_MOV_B32 $exec_lo
+# GCN64: $sgpr12_sgpr13 = S_MOV_B64 $exec
 # GCN32: $exec_lo = S_MOV_B32 3
 # GCN64: $exec = S_MOV_B64 3
-# CHECK: BUFFER_STORE_DWORD_OFFSET {{(killed )?}}$vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 8
-# CHECK: $exec_lo = V_READLANE
-# GCN64: $exec_hi = V_READLANE
+# CHECK: BUFFER_STORE_DWORD_OFFSET $vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 8
+# GCN32: $exec_lo = S_MOV_B32 $sgpr12
+# GCN64: $exec = S_MOV_B64 $sgpr12_sgpr13
+# GCN64: $sgpr13 = V_READLANE
+# CHECK: $sgpr12 = V_READLANE
 
 # S96
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # GCN32: $sgpr12 = S_MOV_B32 $exec_lo
 # GCN64: $sgpr12_sgpr13 = S_MOV_B64 $exec
 # GCN32: $exec_lo = S_MOV_B32 7
 # GCN64: $exec = S_MOV_B64 7
-# CHECK: BUFFER_STORE_DWORD_OFFSET {{(killed )?}}$vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 16
+# CHECK: BUFFER_STORE_DWORD_OFFSET killed $vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 16
 # GCN32: $exec_lo = S_MOV_B32 killed $sgpr12
 # GCN64: $exec = S_MOV_B64 killed $sgpr12_sgpr13
 
 # S128
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # GCN32: $sgpr12 = S_MOV_B32 $exec_lo
 # GCN64: $sgpr12_sgpr13 = S_MOV_B64 $exec
 # GCN32: $exec_lo = S_MOV_B32 15
 # GCN64: $exec = S_MOV_B64 15
-# CHECK: BUFFER_STORE_DWORD_OFFSET {{(killed )?}}$vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 28
+# CHECK: BUFFER_STORE_DWORD_OFFSET killed $vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 28
 # GCN32: $exec_lo = S_MOV_B32 killed $sgpr12
 # GCN64: $exec = S_MOV_B64 killed $sgpr12_sgpr13
 
 # S160
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE

 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
-# GCN32: $sgpr64 = S_MOV_B32 $exec_lo
-# GCN32: $exec_lo = S_MOV_B32 65535
-# GCN32: BUFFER_STORE_DWORD_OFFSET {{(killed )?}}$vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 160
-# GCN32: $exec_lo = S_MOV_B32 killed $sgpr64
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
 # CHECK: V_WRITELANE
-# GCN32: $sgpr80 = S_MOV_B32 $exec_lo
+# GCN32: $sgpr64 = S_MOV_B32 $exec_lo
 # GCN64: $sgpr64_sgpr65 = S_MOV_B64 $exec
-# GCN32: $exec_lo = S_MOV_B32 65535
+# GCN32: $exec_lo = S_MOV_B32 4294967295
 # GCN64: $exec = S_MOV_B64 4294967295
-# GCN32: BUFFER_STORE_DWORD_OFFSET {{(killed )?}}$vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 164
-# GCN64: BUFFER_STORE_DWORD_OFFSET {{(killed )?}}$vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 160
-# GCN32: $exec_lo = S_MOV_B32 killed $sgpr80
+# CHECK: BUFFER_STORE_DWORD_OFFSET {{(killed )?}}$vgpr{{[0-9]+}}, ${{(sgpr[0-9_]+)*}}, $sgpr33, 160
+# GCN32: $exec_lo = S_MOV_B32 killed $sgpr64
 # GCN64: $exec = S_MOV_B64 killed $sgpr64_sgpr65
 
 --- |
 
   define amdgpu_kernel void @check_spill() #0 {
     ret void
   }
 

 # CHECK: $sgpr24 = V_READLANE
 # CHECK: $sgpr25 = V_READLANE
 # CHECK: $sgpr26 = V_READLANE
 # CHECK: $sgpr27 = V_READLANE
 
 # S1024
 # GCN32: $sgpr64 = S_MOV_B32 $exec_lo
 # GCN64: $sgpr64_sgpr65 = S_MOV_B64 $exec
-# GCN32: $exec_lo = S_MOV_B32 65535
+# GCN32: $exec_lo = S_MOV_B32 4294967295
 # GCN64: $exec = S_MOV_B64 4294967295
 # CHECK: BUFFER_LOAD_DWORD_OFFSET ${{(sgpr[0-9_]+)*}}, $sgpr33, 160
 # GCN32: $exec_lo = S_MOV_B32 killed $sgpr64
 # GCN64: $exec = S_MOV_B64 killed $sgpr64_sgpr65
 # CHECK: $sgpr64 = V_READLANE
 # CHECK: $sgpr65 = V_READLANE
 # CHECK: $sgpr66 = V_READLANE
 # CHECK: $sgpr67 = V_READLANE
 # CHECK: $sgpr68 = V_READLANE
 # CHECK: $sgpr69 = V_READLANE
 # CHECK: $sgpr70 = V_READLANE
 # CHECK: $sgpr71 = V_READLANE
 # CHECK: $sgpr72 = V_READLANE
 # CHECK: $sgpr73 = V_READLANE
 # CHECK: $sgpr74 = V_READLANE
 # CHECK: $sgpr75 = V_READLANE
 # CHECK: $sgpr76 = V_READLANE
 # CHECK: $sgpr77 = V_READLANE
 # CHECK: $sgpr78 = V_READLANE
 # CHECK: $sgpr79 = V_READLANE
-# GCN32: $sgpr80 = S_MOV_B32 $exec_lo
-# GCN32: $exec_lo = S_MOV_B32 65535
-# GCN32: BUFFER_LOAD_DWORD_OFFSET ${{(sgpr[0-9_]+)*}}, $sgpr33, 164
-# GCN32: $exec_lo = S_MOV_B32 killed $sgpr80
 # CHECK: $sgpr80 = V_READLANE
 # CHECK: $sgpr81 = V_READLANE
 # CHECK: $sgpr82 = V_READLANE
 # CHECK: $sgpr83 = V_READLANE
 # CHECK: $sgpr84 = V_READLANE
 # CHECK: $sgpr85 = V_READLANE
 # CHECK: $sgpr86 = V_READLANE
 # CHECK: $sgpr87 = V_READLANE

llvm/test/CodeGen/AMDGPU/si-sgpr-spill.ll

 ; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=GCN -check-prefix=TOVGPR %s
-; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
+; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
 
 ; These tests check that the compiler won't crash when it needs to spill
 ; SGPRs.
 
 ; GCN-LABEL: {{^}}main:
+
+; Make sure there are no direct spills for EXEC registers before WQM
+; GCN-NOT: v_writelane_b32 v{{[0-9]+}}, exec_lo
+; GCN-NOT: v_writelane_b32 v{{[0-9]+}}, exec_hi
+
 ; GCN: s_wqm
 
 ; Make sure not emitting unused scratch resource descriptor setup
 ; GCN-NOT: s_mov_b32
 
 ; GCN: s_mov_b32 m0
 
 ; Make sure scratch space isn't being used for SGPR->VGPR spills
 
 ; Writing to M0 from an SMRD instruction will hang the GPU.
 ; GCN-NOT: s_buffer_load_dword m0
+
+; Make sure there are no direct spills/reloads for EXEC registers
+; GCN-NOT: v_writelane_b32 v{{[0-9]+}}, exec_lo
+; GCN-NOT: v_writelane_b32 v{{[0-9]+}}, exec_hi
+; GCN-NOT: v_readlane_b32 exec_lo
+; GCN-NOT: v_readlane_b32 exec_hi
+
 ; GCN: s_endpgm
 
 ; TOVGPR: ScratchSize: 0{{$}}
 define amdgpu_ps void @main([17 x <4 x i32>] addrspace(4)* inreg %arg, [32 x <4 x i32>] addrspace(4)* inreg %arg1, [16 x <8 x i32>] addrspace(4)* inreg %arg2, float inreg %arg3, i32 inreg %arg4, <2 x i32> %arg5, <2 x i32> %arg6, <2 x i32> %arg7, <3 x i32> %arg8, <2 x i32> %arg9, <2 x i32> %arg10, <2 x i32> %arg11, float %arg12, float %arg13, float %arg14, float %arg15, float %arg16, float %arg17, float %arg18, float %arg19, float %arg20) {
 main_body:
   %lds = inttoptr i32 0 to [64 x i32] addrspace(3)*
   %tmp = getelementptr [17 x <4 x i32>], [17 x <4 x i32>] addrspace(4)* %arg, i64 0, i32 0
   %tmp21 = load <4 x i32>, <4 x i32> addrspace(4)* %tmp, !tbaa !0

llvm/test/CodeGen/AMDGPU/spill-special-sgpr.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -mtriple=amdgcn -mcpu=gfx900 -verify-machineinstrs -run-pass=prologepilog %s -o - | FileCheck -check-prefixes=CHECK,GFX9 %s
 # RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs -run-pass=prologepilog %s -o - | FileCheck -check-prefixes=CHECK,GFX10 %s
 
 --- |
   define amdgpu_kernel void @check_vcc() #0 {
     ret void
   }
 
-  define amdgpu_kernel void @check_exec() #0 {
-    ret void
-  }
-
   attributes #0 = {  "frame-pointer"="all" }
 ...
 ---
 name:            check_vcc
 tracksRegLiveness: true
 liveins:
   - { reg: '$sgpr4_sgpr5' }
   - { reg: '$sgpr6_sgpr7' }
bb.0:
     ; GFX9: $sgpr13 = S_MOV_B32 &SCRATCH_RSRC_DWORD1, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15
     ; GFX9: $sgpr14 = S_MOV_B32 4294967295, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15
     ; GFX9: $sgpr15 = S_MOV_B32 14680064, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15
     ; GFX9: $sgpr12 = S_ADD_U32 $sgpr12, $sgpr9, implicit-def $scc, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15
     ; GFX9: $sgpr13 = S_ADDC_U32 $sgpr13, 0, implicit-def $scc, implicit $scc, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15
     ; GFX9: $vcc = IMPLICIT_DEF
     ; GFX9: $vgpr0 = V_WRITELANE_B32_vi $vcc_lo, 0, undef $vgpr0, implicit $vcc
     ; GFX9: $vgpr0 = V_WRITELANE_B32_vi $vcc_hi, 1, $vgpr0, implicit $vcc
-    ; GFX9: $vgpr0 = V_WRITELANE_B32_vi $exec_lo, 32, $vgpr0
-    ; GFX9: $vgpr0 = V_WRITELANE_B32_vi $exec_hi, 33, $vgpr0
+    ; GFX9: $vcc = S_MOV_B64 $exec
     ; GFX9: $exec = S_MOV_B64 3
     ; GFX9: BUFFER_STORE_DWORD_OFFSET $vgpr0, $sgpr12_sgpr13_sgpr14_sgpr15, $sgpr33, 4, 0, 0, 0, 0, 0, implicit $exec :: (store 4 into %stack.0, addrspace 5)
-    ; GFX9: $exec_lo = V_READLANE_B32_vi $vgpr0, 32
-    ; GFX9: $exec_hi = V_READLANE_B32_vi killed $vgpr0, 33
+    ; GFX9: $exec = S_MOV_B64 $vcc
+    ; GFX9: $vcc_hi = V_READLANE_B32_vi $vgpr0, 1
+    ; GFX9: $vcc_lo = V_READLANE_B32_vi killed $vgpr0, 0
     ; GFX9: $vcc = IMPLICIT_DEF
     ; GFX9: $vgpr0 = V_WRITELANE_B32_vi $vcc_lo, 0, undef $vgpr0, implicit $vcc
     ; GFX9: $vgpr0 = V_WRITELANE_B32_vi $vcc_hi, 1, $vgpr0, implicit killed $vcc
     ; GFX9: $vcc = S_MOV_B64 $exec
     ; GFX9: $exec = S_MOV_B64 3
     ; GFX9: BUFFER_STORE_DWORD_OFFSET killed $vgpr0, $sgpr12_sgpr13_sgpr14_sgpr15, $sgpr33, 4, 0, 0, 0, 0, 0, implicit $exec :: (store 4 into %stack.0, addrspace 5)
     ; GFX9: $exec = S_MOV_B64 killed $vcc
     ; GFX9: $vcc = S_MOV_B64 $exec
     ; GFX9: $exec = S_MOV_B64 3
     ; GFX9: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr12_sgpr13_sgpr14_sgpr15, $sgpr33, 4, 0, 0, 0, 0, 0, implicit $exec :: (load 4 from %stack.0, addrspace 5)
     ; GFX9: $exec = S_MOV_B64 killed $vcc
     ; GFX9: $vcc_lo = V_READLANE_B32_vi $vgpr0, 0, implicit-def $vcc
     ; GFX9: $vcc_hi = V_READLANE_B32_vi killed $vgpr0, 1
 
     ; GFX10: $sgpr33 = S_MOV_B32 0
     ; GFX10: $sgpr96 = S_MOV_B32 &SCRATCH_RSRC_DWORD0, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99
     ; GFX10: $sgpr97 = S_MOV_B32 &SCRATCH_RSRC_DWORD1, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99
     ; GFX10: $sgpr98 = S_MOV_B32 4294967295, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99
     ; GFX10: $sgpr99 = S_MOV_B32 836853760, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99
     ; GFX10: $sgpr96 = S_ADD_U32 $sgpr96, $sgpr9, implicit-def $scc, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99
     ; GFX10: $sgpr97 = S_ADDC_U32 $sgpr97, 0, implicit-def $scc, implicit $scc, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99
     ; GFX10: $vcc = IMPLICIT_DEF
     ; GFX10: $vgpr0 = V_WRITELANE_B32_gfx10 $vcc_lo, 0, undef $vgpr0, implicit $vcc
     ; GFX10: $vgpr0 = V_WRITELANE_B32_gfx10 $vcc_hi, 1, $vgpr0, implicit $vcc
-    ; GFX10: $vgpr0 = V_WRITELANE_B32_gfx10 $exec_lo, 32, $vgpr0
-    ; GFX10: $vgpr0 = V_WRITELANE_B32_gfx10 $exec_hi, 33, $vgpr0
+    ; GFX10: $vcc = S_MOV_B64 $exec
     ; GFX10: $exec = S_MOV_B64 3
     ; GFX10: BUFFER_STORE_DWORD_OFFSET $vgpr0, $sgpr96_sgpr97_sgpr98_sgpr99, $sgpr33, 4, 0, 0, 0, 0, 0, implicit $exec :: (store 4 into %stack.0, addrspace 5)
-    ; GFX10: $exec_lo = V_READLANE_B32_gfx10 $vgpr0, 32
-    ; GFX10: $exec_hi = V_READLANE_B32_gfx10 killed $vgpr0, 33
+    ; GFX10: $exec = S_MOV_B64 $vcc
+    ; GFX10: $vcc_hi = V_READLANE_B32_gfx10 $vgpr0, 1
+    ; GFX10: $vcc_lo = V_READLANE_B32_gfx10 killed $vgpr0, 0
     ; GFX10: $vcc = IMPLICIT_DEF
     ; GFX10: $vgpr0 = V_WRITELANE_B32_gfx10 $vcc_lo, 0, undef $vgpr0, implicit $vcc
     ; GFX10: $vgpr0 = V_WRITELANE_B32_gfx10 $vcc_hi, 1, $vgpr0, implicit killed $vcc
     ; GFX10: $vcc = S_MOV_B64 $exec
     ; GFX10: $exec = S_MOV_B64 3
     ; GFX10: BUFFER_STORE_DWORD_OFFSET killed $vgpr0, $sgpr96_sgpr97_sgpr98_sgpr99, $sgpr33, 4, 0, 0, 0, 0, 0, implicit $exec :: (store 4 into %stack.0, addrspace 5)
     ; GFX10: $exec = S_MOV_B64 killed $vcc
     ; GFX10: $vcc = S_MOV_B64 $exec
     ; GFX10: $exec = S_MOV_B64 3
     ; GFX10: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr96_sgpr97_sgpr98_sgpr99, $sgpr33, 4, 0, 0, 0, 0, 0, implicit $exec :: (load 4 from %stack.0, addrspace 5)
     ; GFX10: $exec = S_MOV_B64 killed $vcc
     ; GFX10: $vcc_lo = V_READLANE_B32_gfx10 $vgpr0, 0, implicit-def $vcc
     ; GFX10: $vcc_hi = V_READLANE_B32_gfx10 killed $vgpr0, 1
     $vcc = IMPLICIT_DEF
     SI_SPILL_S64_SAVE $vcc, %stack.0, implicit $exec, implicit $sgpr96_sgpr97_sgpr98_sgpr99, implicit $sgpr32
 
     $vcc = IMPLICIT_DEF
     SI_SPILL_S64_SAVE killed $vcc, %stack.0, implicit $exec, implicit $sgpr96_sgpr97_sgpr98_sgpr99, implicit $sgpr32
 
     $vcc = SI_SPILL_S64_RESTORE %stack.0, implicit $exec, implicit $sgpr96_sgpr97_sgpr98_sgpr99, implicit $sgpr32
 ...
----
-name:            check_exec
-tracksRegLiveness: true
-liveins:
-  - { reg: '$sgpr4_sgpr5' }
-  - { reg: '$sgpr6_sgpr7' }
-  - { reg: '$sgpr8' }
-frameInfo:
-  maxAlignment:    4
-stack:
-  - { id: 0, type: spill-slot, size: 8, alignment: 4 }
-machineFunctionInfo:
-  isEntryFunction: true
-  waveLimiter:     true
-  scratchRSrcReg:  '$sgpr96_sgpr97_sgpr98_sgpr99'
-  stackPtrOffsetReg: '$sgpr32'
-  frameOffsetReg: '$sgpr33'
-  argumentInfo:
-    privateSegmentBuffer: { reg: '$sgpr0_sgpr1_sgpr2_sgpr3' }
-    dispatchPtr:     { reg: '$sgpr4_sgpr5' }
-    kernargSegmentPtr: { reg: '$sgpr6_sgpr7' }
-    workGroupIDX:    { reg: '$sgpr8' }
-    privateSegmentWaveByteOffset: { reg: '$sgpr9' }
-body:             |
-  bb.0:
-    liveins: $sgpr8, $sgpr4_sgpr5, $sgpr6_sgpr7
-
-    ; CHECK-LABEL: name: check_exec
-    ; CHECK: liveins: $sgpr8, $sgpr4_sgpr5, $sgpr6_sgpr7, $sgpr9
-
-    ; GFX9: $sgpr33 = S_MOV_B32 0
-    ; GFX9: $sgpr12 = S_MOV_B32 &SCRATCH_RSRC_DWORD0, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15
-    ; GFX9: $sgpr13 = S_MOV_B32 &SCRATCH_RSRC_DWORD1, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15
-    ; GFX9: $sgpr14 = S_MOV_B32 4294967295, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15
-    ; GFX9: $sgpr15 = S_MOV_B32 14680064, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15
-    ; GFX9: $sgpr12 = S_ADD_U32 $sgpr12, $sgpr9, implicit-def $scc, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15
-    ; GFX9: $sgpr13 = S_ADDC_U32 $sgpr13, 0, implicit-def $scc, implicit $scc, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15
-    ; GFX9: $vgpr0 = V_WRITELANE_B32_vi $exec_lo, 0, undef $vgpr0, implicit $exec
-    ; GFX9: $vgpr0 = V_WRITELANE_B32_vi $exec_hi, 1, $vgpr0, implicit $exec
-    ; GFX9: $exec = S_MOV_B64 3
-    ; GFX9: BUFFER_STORE_DWORD_OFFSET $vgpr0, $sgpr12_sgpr13_sgpr14_sgpr15, $sgpr33, 4, 0, 0, 0, 0, 0, implicit $exec :: (store 4 into %stack.0, addrspace 5)
-    ; GFX9: $exec_lo = V_READLANE_B32_vi $vgpr0, 0
-    ; GFX9: $exec_hi = V_READLANE_B32_vi killed $vgpr0, 1
-    ; GFX9: $exec = S_MOV_B64 3
-    ; GFX9: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr12_sgpr13_sgpr14_sgpr15, $sgpr33, 4, 0, 0, 0, 0, 0, implicit $exec :: (load 4 from %stack.0, addrspace 5)
-    ; GFX9: $exec_lo = V_READLANE_B32_vi $vgpr0, 0, implicit-def $exec
-    ; GFX9: $exec_hi = V_READLANE_B32_vi killed $vgpr0, 1
-
-    ; GFX10: $sgpr33 = S_MOV_B32 0
-    ; GFX10: $sgpr96 = S_MOV_B32 &SCRATCH_RSRC_DWORD0, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99
-    ; GFX10: $sgpr97 = S_MOV_B32 &SCRATCH_RSRC_DWORD1, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99
-    ; GFX10: $sgpr98 = S_MOV_B32 4294967295, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99
-    ; GFX10: $sgpr99 = S_MOV_B32 836853760, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99
-    ; GFX10: $sgpr96 = S_ADD_U32 $sgpr96, $sgpr9, implicit-def $scc, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99
-    ; GFX10: $sgpr97 = S_ADDC_U32 $sgpr97, 0, implicit-def $scc, implicit $scc, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99
-    ; GFX10: $vgpr0 = V_WRITELANE_B32_gfx10 $exec_lo, 0, undef $vgpr0, implicit $exec
-    ; GFX10: $vgpr0 = V_WRITELANE_B32_gfx10 $exec_hi, 1, $vgpr0, implicit $exec
-    ; GFX10: $exec = S_MOV_B64 3
-    ; GFX10: BUFFER_STORE_DWORD_OFFSET $vgpr0, $sgpr96_sgpr97_sgpr98_sgpr99, $sgpr33, 4, 0, 0, 0, 0, 0, implicit $exec :: (store 4 into %stack.0, addrspace 5)
-    ; GFX10: $exec_lo = V_READLANE_B32_gfx10 $vgpr0, 0
-    ; GFX10: $exec_hi = V_READLANE_B32_gfx10 killed $vgpr0, 1
-    ; GFX10: $exec = S_MOV_B64 3
-    ; GFX10: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr96_sgpr97_sgpr98_sgpr99, $sgpr33, 4, 0, 0, 0, 0, 0, implicit $exec :: (load 4 from %stack.0, addrspace 5)
-    ; GFX10: $exec_lo = V_READLANE_B32_gfx10 $vgpr0, 0, implicit-def $exec
-    ; GFX10: $exec_hi = V_READLANE_B32_gfx10 killed $vgpr0, 1
-    SI_SPILL_S64_SAVE $exec, %stack.0, implicit $exec, implicit $sgpr96_sgpr97_sgpr98_sgpr99, implicit $sgpr32
-
-    $exec = SI_SPILL_S64_RESTORE %stack.0, implicit $exec, implicit $sgpr96_sgpr97_sgpr98_sgpr99, implicit $sgpr32
-...