[AMDGPU] Fix saving fp and bp

[sebastian-ne]

Summary

Spilling the fp or bp to scratch could overwrite VGPRs of inactive
lanes. Fix that by using only the active lanes of the scavenged VGPR.

This builds on the assumptions that

1. a function is never called with exec=0
2. lanes do not die in a function, i.e. exec!=0 in the function epilog
3. no new lanes are active when exiting the function, i.e. exec in the epilog is a subset of exec in the prolog.

[arsenm]

Should comment why the register isn't considered killed

llvm/lib/Target/AMDGPU/SIFrameLowering.cpp

891

Don't need hasValue

903

• instead of getValue

992

Comment the isKill

1007

Typo imlicit

1223

Comment isKill

1238–1239

• instead of getValue

1285

Typo imlicit

[sebastian-ne]

Fix review comments and remove writelane from epilogs.

[sebastian-ne]

In the light of changes in D96980, can we assume that

1. exec != 0 in the function prolog and epilog?
2. exec in the epilog is a subset of exec in the prolog (i.e. no new lanes are active)?

That would simplify the code here.

[arsenm]

In D96869#2596509, @sebastian-ne wrote:

In the light of changes in D96980, can we assume that

1. exec != 0 in the function prolog and epilog?

Yes (at least for the prolog). Allowing functions to kill exec and return is an open question. We have to assume a call to an arbitrary function will have any of the problematic cases that can't run with exec=0.

2. exec in the epilog is a subset of exec in the prolog (i.e. no new lanes are active)?

This should be fine

[sebastian-ne]

In D96869#2600088, @arsenm wrote:

1. exec != 0 in the function prolog and epilog?

Yes (at least for the prolog). Allowing functions to kill exec and return is an open question. We have to assume a call to an arbitrary function will have any of the problematic cases that can't run with exec=0.

Thanks. I’ll leave it as it is then, because we are not sure about the epilog.

[critson]

In D96869#2600444, @sebastian-ne wrote:

In D96869#2600088, @arsenm wrote:

1. exec != 0 in the function prolog and epilog?

Yes (at least for the prolog). Allowing functions to kill exec and return is an open question. We have to assume a call to an arbitrary function will have any of the problematic cases that can't run with exec=0.

Thanks. I’ll leave it as it is then, because we are not sure about the epilog.

If the only way lanes can be killed is llvm.amdgcn.kill (or demote) then epilog will never be reached with EXEC=0 as kill now immediately terminates a shader if all lanes are dead (or helpers). Are there other mechanisms that reduce EXEC? Inline ASM?

[arsenm]

In D96869#2601792, @critson wrote:

In D96869#2600444, @sebastian-ne wrote:

In D96869#2600088, @arsenm wrote:

1. exec != 0 in the function prolog and epilog?

Yes (at least for the prolog). Allowing functions to kill exec and return is an open question. We have to assume a call to an arbitrary function will have any of the problematic cases that can't run with exec=0.

Thanks. I’ll leave it as it is then, because we are not sure about the epilog.

If the only way lanes can be killed is llvm.amdgcn.kill (or demote) then epilog will never be reached with EXEC=0 as kill now immediately terminates a shader if all lanes are dead (or helpers). Are there other mechanisms that reduce EXEC? Inline ASM?

Inline asm theoretically could, but we don't even attempt to handle exec modifications on it. We could also just define the ABI to make that an invalid use

[sebastian-ne]

Rebased

[sebastian-ne]

Rewrite the patch to not rely on D96336 anymore.

Instead of spilling all lanes of the used VGPR, assume that the live lanes are fitting in the prolog and epilog and use only live lanes for spilling.

[sebastian-ne]

I changed the frame-setup-without-sgpr-to-vgpr-spills.ll test to be autogenerated (it is currently not autogenerated on main). I will pre-commit that if it’s ok.

[arsenm]

LGTM. I think "lanes do not die in a function" may not be true in the future, but is good enough for now (not sure whether callers or callees should be responsible for wave termination on kill)

llvm/lib/Target/AMDGPU/SIFrameLowering.cpp

 
 // We need to specially emit stack operations here because a different frame
 // register is used than in the rest of the function, as getFrameRegister would
 // use.
 static void buildPrologSpill(const GCNSubtarget &ST, LivePhysRegs &LiveRegs,
                              MachineBasicBlock &MBB,
                              MachineBasicBlock::iterator I,
                              const SIInstrInfo *TII, Register SpillReg,
-                             Register ScratchRsrcReg, Register SPReg, int FI) {
+                             Register ScratchRsrcReg, Register SPReg, int FI,
+                             bool IsKill = true) {
   MachineFunction *MF = MBB.getParent();
   MachineFrameInfo &MFI = MF->getFrameInfo();
 
   int64_t Offset = MFI.getObjectOffset(FI);
 
   MachineMemOperand *MMO = MF->getMachineMemOperand(
       MachinePointerInfo::getFixedStack(*MF, FI), MachineMemOperand::MOStore, 4,
       MFI.getObjectAlign(FI));
 
   if (ST.enableFlatScratch()) {
     if (TII->isLegalFLATOffset(Offset, AMDGPUAS::PRIVATE_ADDRESS, true)) {
       BuildMI(MBB, I, DebugLoc(), TII->get(AMDGPU::SCRATCH_STORE_DWORD_SADDR))
-        .addReg(SpillReg, RegState::Kill)
+          .addReg(SpillReg, getKillRegState(IsKill))
           .addReg(SPReg)
           .addImm(Offset)
           .addImm(0) // glc
           .addImm(0) // slc
           .addImm(0) // dlc
           .addImm(0) // scc
           .addMemOperand(MMO);
       return;
     }
   } else if (SIInstrInfo::isLegalMUBUFImmOffset(Offset)) {
     BuildMI(MBB, I, DebugLoc(), TII->get(AMDGPU::BUFFER_STORE_DWORD_OFFSET))
-      .addReg(SpillReg, RegState::Kill)
+        .addReg(SpillReg, getKillRegState(IsKill))
         .addReg(ScratchRsrcReg)
         .addReg(SPReg)
         .addImm(Offset)
         .addImm(0) // glc
         .addImm(0) // slc
         .addImm(0) // tfe
         .addImm(0) // dlc
         .addImm(0) // swz
         .addImm(0) // scc
         .addMemOperand(MMO);
     return;
   }
 
   // Don't clobber the TmpVGPR if we also need a scratch reg for the stack
   // offset in the spill.
   LiveRegs.addReg(SpillReg);
 
   if (ST.enableFlatScratch()) {
     MCPhysReg OffsetReg = findScratchNonCalleeSaveRegister(
       MF->getRegInfo(), LiveRegs, AMDGPU::SReg_32_XM0RegClass);
 
     bool HasOffsetReg = OffsetReg;
     if (!HasOffsetReg) {
       // No free register, use stack pointer and restore afterwards.
       OffsetReg = SPReg;
     }
 
     BuildMI(MBB, I, DebugLoc(), TII->get(AMDGPU::S_ADD_U32), OffsetReg)
       .addReg(SPReg)
       .addImm(Offset);
 
     BuildMI(MBB, I, DebugLoc(), TII->get(AMDGPU::SCRATCH_STORE_DWORD_SADDR))
-        .addReg(SpillReg, RegState::Kill)
-        .addReg(OffsetReg, HasOffsetReg ? RegState::Kill : 0)
+        .addReg(SpillReg, getKillRegState(IsKill))
+        .addReg(OffsetReg, getKillRegState(HasOffsetReg))
         .addImm(0) // offset
         .addImm(0) // glc
         .addImm(0) // slc
         .addImm(0) // dlc
         .addImm(0) // scc
         .addMemOperand(MMO);
 
     if (!HasOffsetReg) {
       BuildMI(MBB, I, DebugLoc(), TII->get(AMDGPU::S_SUB_U32), OffsetReg)
           .addReg(SPReg)
           .addImm(Offset);
     }
   } else {
     MCPhysReg OffsetReg = findScratchNonCalleeSaveRegister(
       MF->getRegInfo(), LiveRegs, AMDGPU::VGPR_32RegClass);
 
     if (OffsetReg) {
       BuildMI(MBB, I, DebugLoc(), TII->get(AMDGPU::V_MOV_B32_e32), OffsetReg)
           .addImm(Offset);
 
       BuildMI(MBB, I, DebugLoc(), TII->get(AMDGPU::BUFFER_STORE_DWORD_OFFEN))
-          .addReg(SpillReg, RegState::Kill)
+          .addReg(SpillReg, getKillRegState(IsKill))
           .addReg(OffsetReg, RegState::Kill)
           .addReg(ScratchRsrcReg)
           .addReg(SPReg)
           .addImm(0) // offset
           .addImm(0) // glc
           .addImm(0) // slc
           .addImm(0) // tfe
           .addImm(0) // dlc
           .addImm(0) // swz
           .addImm(0) // scc
           .addMemOperand(MMO);
     } else {
       // No free register, use stack pointer and restore afterwards.
       BuildMI(MBB, I, DebugLoc(), TII->get(AMDGPU::S_ADD_U32), SPReg)
           .addReg(SPReg)
           .addImm(Offset);
 
       BuildMI(MBB, I, DebugLoc(), TII->get(AMDGPU::BUFFER_STORE_DWORD_OFFSET))
-          .addReg(SpillReg, RegState::Kill)
+          .addReg(SpillReg, getKillRegState(IsKill))
           .addReg(ScratchRsrcReg)
           .addReg(SPReg)
           .addImm(0) // offset
           .addImm(0) // glc
           .addImm(0) // slc
           .addImm(0) // tfe
           .addImm(0) // dlc
           .addImm(0) // swz
static Register buildScratchExecCopy(LivePhysRegs &LiveRegs,
 
   const unsigned OrSaveExec =
       ST.isWave32() ? AMDGPU::S_OR_SAVEEXEC_B32 : AMDGPU::S_OR_SAVEEXEC_B64;
   BuildMI(MBB, MBBI, DL, TII->get(OrSaveExec), ScratchExecCopy).addImm(-1);
 
   return ScratchExecCopy;
 }
 
+// Save or restore the frame or base pointer to memory.
+//
+// 1. Save exec into ScratchExecCopy if not already saved
+// 2. Scavenge a TmpVGPR and save it to scratch if not already done
+// 3. When IsProlog: Write SpillSGPR into TmpVGPR and save to FrameIndex
+//    When !IsProlog: Load TmpVGPR from FrameIndex and read into SpillSGPR
+static void
+buildSpillLoadStore(LivePhysRegs &LiveRegs, SIMachineFunctionInfo *FuncInfo,
+                    const GCNSubtarget &ST, const SIInstrInfo *TII,
+                    const MachineFrameInfo &MFI, MachineRegisterInfo &MRI,
+                    MachineFunction &MF, MachineBasicBlock &MBB,
+                    MachineBasicBlock::iterator MBBI, bool IsProlog,
+                    Register StackPtrReg, Register &ScratchExecCopy,
+                    MCRegister &TmpVGPR, Register SpillSGPR, int FrameIndex) {
+  assert(!MFI.isDeadObjectIndex(FrameIndex));
+
+  if (!ScratchExecCopy)
+    ScratchExecCopy = buildScratchExecCopy(LiveRegs, MF, MBB, MBBI, IsProlog);
+
+  if (!TmpVGPR) {
+    TmpVGPR = findScratchNonCalleeSaveRegister(MRI, LiveRegs,
+                                               AMDGPU::VGPR_32RegClass);
+    if (!TmpVGPR)
+      report_fatal_error("failed to find free scratch register");
+    // Get emergency spill slot
+    SIMachineFunctionInfo *SIMFI = MF.getInfo<SIMachineFunctionInfo>();
+    int TmpVGPRIndex =
+        SIMFI->getScavengeFI(MF.getFrameInfo(), *ST.getRegisterInfo());
+
+    BuildMI(MBB, MBBI, DebugLoc(), TII->get(AMDGPU::COPY_INACTIVE_LANES),
+            TmpVGPR);
+
+    // Save TmpVGPR
+    buildPrologSpill(ST, LiveRegs, MBB, MBBI, TII, TmpVGPR,
+                     FuncInfo->getScratchRSrcReg(), StackPtrReg, TmpVGPRIndex,
+                     false);
+  }
+
+  if (IsProlog) {
+    BuildMI(MBB, MBBI, DebugLoc(), TII->get(AMDGPU::V_WRITELANE_B32), TmpVGPR)
+        .addReg(SpillSGPR)
+        .addImm(0)
+        .addReg(TmpVGPR);
+
+    buildPrologSpill(ST, LiveRegs, MBB, MBBI, TII, TmpVGPR,
+                     FuncInfo->getScratchRSrcReg(), StackPtrReg, FrameIndex,
+                     false);
+  } else {
+    buildEpilogReload(ST, LiveRegs, MBB, MBBI, TII, TmpVGPR,
+                      FuncInfo->getScratchRSrcReg(), StackPtrReg, FrameIndex);
+    BuildMI(MBB, MBBI, DebugLoc(), TII->get(AMDGPU::V_READFIRSTLANE_B32),
+            SpillSGPR)
+        .addReg(TmpVGPR);
+  }
+}
+
 void SIFrameLowering::emitPrologue(MachineFunction &MF,
                                    MachineBasicBlock &MBB) const {
   SIMachineFunctionInfo *FuncInfo = MF.getInfo<SIMachineFunctionInfo>();
   if (FuncInfo->isEntryFunction()) {
     emitEntryFunctionPrologue(MF, MBB);
     return;
   }
 
void SIFrameLowering::emitPrologue(MachineFunction &MF,
 
   bool HasFP = false;
   bool HasBP = false;
   uint32_t NumBytes = MFI.getStackSize();
   uint32_t RoundedSize = NumBytes;
   // To avoid clobbering VGPRs in lanes that weren't active on function entry,
   // turn on all lanes before doing the spill to memory.
   Register ScratchExecCopy;
+  MCRegister TmpVGPR;
 
   bool HasFPSaveIndex = FuncInfo->FramePointerSaveIndex.hasValue();
   bool SpillFPToMemory = false;
   // A StackID of SGPRSpill implies that this is a spill from SGPR to VGPR.
   // Otherwise we are spilling the FP to memory.
   if (HasFPSaveIndex) {
     SpillFPToMemory = MFI.getStackID(*FuncInfo->FramePointerSaveIndex) !=
                       TargetStackID::SGPRSpill;
   }
 
   bool HasBPSaveIndex = FuncInfo->BasePointerSaveIndex.hasValue();
   bool SpillBPToMemory = false;
   // A StackID of SGPRSpill implies that this is a spill from SGPR to VGPR.
   // Otherwise we are spilling the BP to memory.
   if (HasBPSaveIndex) {
     SpillBPToMemory = MFI.getStackID(*FuncInfo->BasePointerSaveIndex) !=
                       TargetStackID::SGPRSpill;
   }
 
   for (const SIMachineFunctionInfo::SGPRSpillVGPRCSR &Reg
          : FuncInfo->getSGPRSpillVGPRs()) {
-    if (!Reg.FI.hasValue())
+    if (!Reg.FI)
       continue;
 
     if (!ScratchExecCopy)
       ScratchExecCopy = buildScratchExecCopy(LiveRegs, MF, MBB, MBBI, true);
 
     buildPrologSpill(ST, LiveRegs, MBB, MBBI, TII, Reg.VGPR,
-                     FuncInfo->getScratchRSrcReg(),
-                     StackPtrReg,
-                     Reg.FI.getValue());
+                     FuncInfo->getScratchRSrcReg(), StackPtrReg, *Reg.FI);
   }
 
   if (HasFPSaveIndex && SpillFPToMemory) {
-    assert(!MFI.isDeadObjectIndex(FuncInfo->FramePointerSaveIndex.getValue()));
-
-    if (!ScratchExecCopy)
-      ScratchExecCopy = buildScratchExecCopy(LiveRegs, MF, MBB, MBBI, true);
-
-    MCPhysReg TmpVGPR = findScratchNonCalleeSaveRegister(
-        MRI, LiveRegs, AMDGPU::VGPR_32RegClass);
-    if (!TmpVGPR)
-      report_fatal_error("failed to find free scratch register");
-
-    BuildMI(MBB, MBBI, DL, TII->get(AMDGPU::V_MOV_B32_e32), TmpVGPR)
-        .addReg(FramePtrReg);
-
-    buildPrologSpill(ST, LiveRegs, MBB, MBBI, TII, TmpVGPR,
-                     FuncInfo->getScratchRSrcReg(), StackPtrReg,
-                     FuncInfo->FramePointerSaveIndex.getValue());
+    buildSpillLoadStore(LiveRegs, FuncInfo, ST, TII, MFI, MRI, MF, MBB, MBBI,
+                        /*IsProlog*/ true, StackPtrReg, ScratchExecCopy,
+                        TmpVGPR, FramePtrReg, *FuncInfo->FramePointerSaveIndex);
   }
 
   if (HasBPSaveIndex && SpillBPToMemory) {
-    assert(!MFI.isDeadObjectIndex(*FuncInfo->BasePointerSaveIndex));
-
-    if (!ScratchExecCopy)
-      ScratchExecCopy = buildScratchExecCopy(LiveRegs, MF, MBB, MBBI, true);
-
-    MCPhysReg TmpVGPR = findScratchNonCalleeSaveRegister(
-        MRI, LiveRegs, AMDGPU::VGPR_32RegClass);
-    if (!TmpVGPR)
-      report_fatal_error("failed to find free scratch register");
-
-    BuildMI(MBB, MBBI, DL, TII->get(AMDGPU::V_MOV_B32_e32), TmpVGPR)
-        .addReg(BasePtrReg);
-
-    buildPrologSpill(ST, LiveRegs, MBB, MBBI, TII, TmpVGPR,
-                     FuncInfo->getScratchRSrcReg(), StackPtrReg,
-                     *FuncInfo->BasePointerSaveIndex);
+    buildSpillLoadStore(LiveRegs, FuncInfo, ST, TII, MFI, MRI, MF, MBB, MBBI,
+                        /*IsProlog*/ true, StackPtrReg, ScratchExecCopy,
+                        TmpVGPR, BasePtrReg, *FuncInfo->BasePointerSaveIndex);
+  }
+
+  if (TmpVGPR) {
+    // Restore TmpVGPR
+    // Get emergency spill slot
+    SIMachineFunctionInfo *SIMFI = MF.getInfo<SIMachineFunctionInfo>();
+    int TmpVGPRIndex =
+        SIMFI->getScavengeFI(MF.getFrameInfo(), *ST.getRegisterInfo());
+    buildEpilogReload(ST, LiveRegs, MBB, MBBI, TII, TmpVGPR,
+                      FuncInfo->getScratchRSrcReg(), StackPtrReg, TmpVGPRIndex);
+    // If TmpVGPR is set, ScratchExecCopy must also be set. We add an implicit
+    // use and kill to the exec restore below, so this reload cannot get
+    // eliminated.
   }
 
   if (ScratchExecCopy) {
     // FIXME: Split block and make terminator.
     unsigned ExecMov = ST.isWave32() ? AMDGPU::S_MOV_B32 : AMDGPU::S_MOV_B64;
     MCRegister Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
-    BuildMI(MBB, MBBI, DL, TII->get(ExecMov), Exec)
+    auto I = BuildMI(MBB, MBBI, DL, TII->get(ExecMov), Exec)
                  .addReg(ScratchExecCopy, RegState::Kill);
+    if (TmpVGPR)
+      I.addReg(TmpVGPR, RegState::ImplicitKill);
     LiveRegs.addReg(ScratchExecCopy);
   }
 
   // In this case, spill the FP to a reserved VGPR.
   if (HasFPSaveIndex && !SpillFPToMemory) {
-    const int FI = FuncInfo->FramePointerSaveIndex.getValue();
+    const int FI = *FuncInfo->FramePointerSaveIndex;
     assert(!MFI.isDeadObjectIndex(FI));
 
     assert(MFI.getStackID(FI) == TargetStackID::SGPRSpill);
     ArrayRef<SIMachineFunctionInfo::SpilledReg> Spill =
         FuncInfo->getSGPRToVGPRSpills(FI);
     assert(Spill.size() == 1);
 
     // Save FP before setting it up.
-    // FIXME: This should respect spillSGPRToVGPR;
     BuildMI(MBB, MBBI, DL, TII->get(AMDGPU::V_WRITELANE_B32), Spill[0].VGPR)
         .addReg(FramePtrReg)
         .addImm(Spill[0].Lane)
         .addReg(Spill[0].VGPR, RegState::Undef);
   }
 
   // In this case, spill the BP to a reserved VGPR.
   if (HasBPSaveIndex && !SpillBPToMemory) {
     const int BasePtrFI = *FuncInfo->BasePointerSaveIndex;
     assert(!MFI.isDeadObjectIndex(BasePtrFI));
 
     assert(MFI.getStackID(BasePtrFI) == TargetStackID::SGPRSpill);
     ArrayRef<SIMachineFunctionInfo::SpilledReg> Spill =
         FuncInfo->getSGPRToVGPRSpills(BasePtrFI);
     assert(Spill.size() == 1);
 
     // Save BP before setting it up.
-    // FIXME: This should respect spillSGPRToVGPR;
     BuildMI(MBB, MBBI, DL, TII->get(AMDGPU::V_WRITELANE_B32), Spill[0].VGPR)
         .addReg(BasePtrReg)
         .addImm(Spill[0].Lane)
         .addReg(Spill[0].VGPR, RegState::Undef);
   }
 
   // Emit the copy if we need an FP, and are using a free SGPR to save it.
   if (FuncInfo->SGPRForFPSaveRestoreCopy) {
void SIFrameLowering::emitPrologue(MachineFunction &MF,
 
   assert((HasBP || (!FuncInfo->SGPRForBPSaveRestoreCopy &&
                     !FuncInfo->BasePointerSaveIndex)) &&
          "Saved BP but didn't need it");
 }
 
 void SIFrameLowering::emitEpilogue(MachineFunction &MF,
                                    MachineBasicBlock &MBB) const {
-  const SIMachineFunctionInfo *FuncInfo = MF.getInfo<SIMachineFunctionInfo>();
+  SIMachineFunctionInfo *FuncInfo = MF.getInfo<SIMachineFunctionInfo>();
   if (FuncInfo->isEntryFunction())
     return;
 
   const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
   const SIInstrInfo *TII = ST.getInstrInfo();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   const SIRegisterInfo &TRI = TII->getRegisterInfo();
   MachineBasicBlock::iterator MBBI = MBB.getFirstTerminator();
void SIFrameLowering::emitEpilogue(MachineFunction &MF,
 
   if (FuncInfo->SGPRForBPSaveRestoreCopy) {
     BuildMI(MBB, MBBI, DL, TII->get(AMDGPU::COPY), BasePtrReg)
         .addReg(FuncInfo->SGPRForBPSaveRestoreCopy)
         .setMIFlag(MachineInstr::FrameDestroy);
   }
 
   Register ScratchExecCopy;
+  MCRegister TmpVGPR;
   if (HasFPSaveIndex) {
-    const int FI = FuncInfo->FramePointerSaveIndex.getValue();
+    const int FI = *FuncInfo->FramePointerSaveIndex;
     assert(!MFI.isDeadObjectIndex(FI));
     if (SpillFPToMemory) {
-      if (!ScratchExecCopy)
-        ScratchExecCopy = buildScratchExecCopy(LiveRegs, MF, MBB, MBBI, false);
-
-      MCPhysReg TempVGPR = findScratchNonCalleeSaveRegister(
-          MRI, LiveRegs, AMDGPU::VGPR_32RegClass);
-      if (!TempVGPR)
-        report_fatal_error("failed to find free scratch register");
-      buildEpilogReload(ST, LiveRegs, MBB, MBBI, TII, TempVGPR,
-                        FuncInfo->getScratchRSrcReg(), StackPtrReg, FI);
-      BuildMI(MBB, MBBI, DL, TII->get(AMDGPU::V_READFIRSTLANE_B32), FramePtrReg)
-          .addReg(TempVGPR, RegState::Kill);
+      buildSpillLoadStore(LiveRegs, FuncInfo, ST, TII, MFI, MRI, MF, MBB, MBBI,
+                          /*IsProlog*/ false, StackPtrReg, ScratchExecCopy,
+                          TmpVGPR, FramePtrReg, FI);
     } else {
       // Reload from VGPR spill.
       assert(MFI.getStackID(FI) == TargetStackID::SGPRSpill);
       ArrayRef<SIMachineFunctionInfo::SpilledReg> Spill =
           FuncInfo->getSGPRToVGPRSpills(FI);
       assert(Spill.size() == 1);
       BuildMI(MBB, MBBI, DL, TII->get(AMDGPU::V_READLANE_B32), FramePtrReg)
           .addReg(Spill[0].VGPR)
           .addImm(Spill[0].Lane);
     }
   }
 
   if (HasBPSaveIndex) {
-    const int BasePtrFI = *FuncInfo->BasePointerSaveIndex;
-    assert(!MFI.isDeadObjectIndex(BasePtrFI));
+    const int FI = *FuncInfo->BasePointerSaveIndex;
+    assert(!MFI.isDeadObjectIndex(FI));
     if (SpillBPToMemory) {
-      if (!ScratchExecCopy)
-        ScratchExecCopy = buildScratchExecCopy(LiveRegs, MF, MBB, MBBI, false);
-
-      MCPhysReg TempVGPR = findScratchNonCalleeSaveRegister(
-          MRI, LiveRegs, AMDGPU::VGPR_32RegClass);
-      if (!TempVGPR)
-        report_fatal_error("failed to find free scratch register");
-      buildEpilogReload(ST, LiveRegs, MBB, MBBI, TII, TempVGPR,
-                        FuncInfo->getScratchRSrcReg(), StackPtrReg, BasePtrFI);
-      BuildMI(MBB, MBBI, DL, TII->get(AMDGPU::V_READFIRSTLANE_B32), BasePtrReg)
-          .addReg(TempVGPR, RegState::Kill);
+      buildSpillLoadStore(LiveRegs, FuncInfo, ST, TII, MFI, MRI, MF, MBB, MBBI,
+                          /*IsProlog*/ false, StackPtrReg, ScratchExecCopy,
+                          TmpVGPR, BasePtrReg, FI);
     } else {
       // Reload from VGPR spill.
-      assert(MFI.getStackID(BasePtrFI) == TargetStackID::SGPRSpill);
+      assert(MFI.getStackID(FI) == TargetStackID::SGPRSpill);
       ArrayRef<SIMachineFunctionInfo::SpilledReg> Spill =
-          FuncInfo->getSGPRToVGPRSpills(BasePtrFI);
+          FuncInfo->getSGPRToVGPRSpills(FI);
       assert(Spill.size() == 1);
       BuildMI(MBB, MBBI, DL, TII->get(AMDGPU::V_READLANE_B32), BasePtrReg)
           .addReg(Spill[0].VGPR)
           .addImm(Spill[0].Lane);
     }
   }
 
   for (const SIMachineFunctionInfo::SGPRSpillVGPRCSR &Reg :
        FuncInfo->getSGPRSpillVGPRs()) {
-    if (!Reg.FI.hasValue())
+    if (!Reg.FI)
       continue;
 
     if (!ScratchExecCopy)
       ScratchExecCopy = buildScratchExecCopy(LiveRegs, MF, MBB, MBBI, false);
 
     buildEpilogReload(ST, LiveRegs, MBB, MBBI, TII, Reg.VGPR,
-                      FuncInfo->getScratchRSrcReg(), StackPtrReg,
-                      Reg.FI.getValue());
+                      FuncInfo->getScratchRSrcReg(), StackPtrReg, *Reg.FI);
+  }
+
+  if (TmpVGPR) {
+    // Restore TmpVGPR
+    SIMachineFunctionInfo *SIMFI = MF.getInfo<SIMachineFunctionInfo>();
+    int TmpVGPRIndex =
+        SIMFI->getScavengeFI(MF.getFrameInfo(), *ST.getRegisterInfo());
+    buildEpilogReload(ST, LiveRegs, MBB, MBBI, TII, TmpVGPR,
+                      FuncInfo->getScratchRSrcReg(), StackPtrReg, TmpVGPRIndex);
+    // If TmpVGPR is set, ScratchExecCopy must also be set. We add an implicit
+    // use and kill to the exec restore below, so this reload cannot get
+    // eliminated.
   }
 
   if (ScratchExecCopy) {
     // FIXME: Split block and make terminator.
     unsigned ExecMov = ST.isWave32() ? AMDGPU::S_MOV_B32 : AMDGPU::S_MOV_B64;
     MCRegister Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
-    BuildMI(MBB, MBBI, DL, TII->get(ExecMov), Exec)
+    auto I = BuildMI(MBB, MBBI, DL, TII->get(ExecMov), Exec)
                  .addReg(ScratchExecCopy, RegState::Kill);
+    if (TmpVGPR)
+      I.addReg(TmpVGPR, RegState::ImplicitKill);
   }
 }
 
 #ifndef NDEBUG
 static bool allSGPRSpillsAreDead(const MachineFunction &MF) {
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   const SIMachineFunctionInfo *FuncInfo = MF.getInfo<SIMachineFunctionInfo>();
   for (int I = MFI.getObjectIndexBegin(), E = MFI.getObjectIndexEnd();

llvm/test/CodeGen/AMDGPU/callee-frame-setup.ll

define void @ipra_call_with_stack() #0 {
   store volatile i32 0, i32 addrspace(5)* %alloca
   call void @local_empty_func()
   ret void
 }
 
 ; With no free registers, we must spill the FP to memory.
 ; GCN-LABEL: {{^}}callee_need_to_spill_fp_to_memory:
 ; MUBUF:   s_or_saveexec_b64 [[COPY_EXEC1:s\[[0-9]+:[0-9]+\]]], -1{{$}}
-; MUBUF:   v_mov_b32_e32 [[TMP_VGPR1:v[0-9]+]], s33
-; MUBUF:   buffer_store_dword [[TMP_VGPR1]], off, s[0:3], s32 offset:4
-; MUBUF:   s_mov_b64 exec, [[COPY_EXEC1]]
+; MUBUF-NEXT:    ;;#COPY_INACTIVE_LANES [[TMP_VGPR1:v[0-9]+]]
+; MUBUF-NEXT:    buffer_store_dword [[TMP_VGPR1]], off, s[0:3], s32
+; MUBUF-NEXT:    v_writelane_b32 [[TMP_VGPR1]], s33, 0
+; MUBUF-NEXT:    buffer_store_dword [[TMP_VGPR1]], off, s[0:3], s32 offset:4 ; 4-byte Folded Spill
+; MUBUF-NEXT:    buffer_load_dword [[TMP_VGPR1]], off, s[0:3], s32
+; MUBUF-NEXT:    s_waitcnt vmcnt(0)
+; MUBUF-NEXT:    s_mov_b64 exec, [[COPY_EXEC1]]
 ; FLATSCR: s_mov_b32 s0, s33
 ; GCN:     s_mov_b32 s33, s32
 ; MUBUF:   s_or_saveexec_b64 [[COPY_EXEC2:s\[[0-9]+:[0-9]+\]]], -1{{$}}
-; MUBUF:   buffer_load_dword [[TMP_VGPR2:v[0-9]+]], off, s[0:3], s32 offset:4
+; MUBUF-NEXT:    ;;#COPY_INACTIVE_LANES [[TMP_VGPR2:v[0-9]+]]
+; MUBUF-NEXT:    buffer_store_dword [[TMP_VGPR2]], off, s[0:3], s32
+; MUBUF-NEXT:    buffer_load_dword [[TMP_VGPR2]], off, s[0:3], s32 offset:4 ; 4-byte Folded Reload
+; MUBUF-NEXT:    s_waitcnt vmcnt(0)
+; MUBUF-NEXT:    v_readfirstlane_b32 s33, [[TMP_VGPR2]]
+; MUBUF-NEXT:    buffer_load_dword [[TMP_VGPR2]], off, s[0:3], s32
+; MUBUF-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR: s_mov_b32 s33, s0
-; MUBUF:   s_waitcnt vmcnt(0)
-; MUBUF:   v_readfirstlane_b32 s33, [[TMP_VGPR2]]
 ; MUBUF:   s_mov_b64 exec, [[COPY_EXEC2]]
 ; GCN:     s_setpc_b64
 ; MUBUF:   ScratchSize: 8
 ; FLATSCR: ScratchSize: 0
 define void @callee_need_to_spill_fp_to_memory() #3 {
   call void asm sideeffect "; clobber nonpreserved SGPRs",
     "~{s4},~{s5},~{s6},~{s7},~{s8},~{s9}
     ,~{s10},~{s11},~{s12},~{s13},~{s14},~{s15},~{s16},~{s17},~{s18},~{s19}
     ,~{s20},~{s21},~{s22},~{s23},~{s24},~{s25},~{s26},~{s27},~{s28},~{s29}
     ,~{vcc}"()
 
   call void asm sideeffect "; clobber all VGPRs",
     "~{v0},~{v1},~{v2},~{v3},~{v4},~{v5},~{v6},~{v7},~{v8},~{v9}
     ,~{v10},~{v11},~{v12},~{v13},~{v14},~{v15},~{v16},~{v17},~{v18},~{v19}
     ,~{v20},~{v21},~{v22},~{v23},~{v24},~{v25},~{v26},~{v27},~{v28},~{v29}
     ,~{v30},~{v31},~{v32},~{v33},~{v34},~{v35},~{v36},~{v37},~{v38},~{v39}"()
   ret void
 }
 
 ; If we have a reserved VGPR that can be used for SGPR spills, we may still
 ; need to spill the FP to memory if there are no free lanes in the reserved
 ; VGPR.
 ; GCN-LABEL: {{^}}callee_need_to_spill_fp_to_memory_full_reserved_vgpr:
 ; MUBUF:   s_or_saveexec_b64 [[COPY_EXEC1:s\[[0-9]+:[0-9]+\]]], -1{{$}}
-; MUBUF:   v_mov_b32_e32 [[TMP_VGPR1:v[0-9]+]], s33
+; MUBUF:   v_writelane_b32 [[TMP_VGPR1:v[0-9]+]], s33, 0
 ; MUBUF:   buffer_store_dword [[TMP_VGPR1]], off, s[0:3], s32 offset:[[OFF:[0-9]+]]
 ; MUBUF:   s_mov_b64 exec, [[COPY_EXEC1]]
 ; GCN-NOT: v_writelane_b32 v40, s33
 ; MUBUF:   s_mov_b32 s33, s32
 ; FLATSCR: s_mov_b32 s33, s0
 ; GCN-NOT: v_readlane_b32 s33, v40
 ; MUBUF:   s_or_saveexec_b64 [[COPY_EXEC2:s\[[0-9]+:[0-9]+\]]], -1{{$}}
 ; MUBUF:   buffer_load_dword [[TMP_VGPR2:v[0-9]+]], off, s[0:3], s32 offset:[[OFF]]
call void asm sideeffect "; clobber all VGPRs except CSR v40",
     ,~{v30},~{v31},~{v32},~{v33},~{v34},~{v35},~{v36},~{v37},~{v38},~{v39}"()
   ret void
 }
 
 ; If the size of the offset exceeds the MUBUF offset field we need another
 ; scratch VGPR to hold the offset.
 ; GCN-LABEL: {{^}}spill_fp_to_memory_scratch_reg_needed_mubuf_offset
 ; MUBUF: s_or_saveexec_b64 s[4:5], -1
-; MUBUF: v_mov_b32_e32 v0, 0x1008
-; MUBUF-NEXT:  buffer_store_dword v39, v0, s[0:3], s32 offen ; 4-byte Folded Spill
-; MUBUF: v_mov_b32_e32 v0, s33
+; MUBUF-NEXT:    v_mov_b32_e32 v0, 0x1008
+; MUBUF-NEXT:    buffer_store_dword v39, v0, s[0:3], s32 offen ; 4-byte Folded Spill
+; MUBUF-NEXT:    ;;#COPY_INACTIVE_LANES v0
+; MUBUF-NEXT:    v_mov_b32_e32 v1, 0x1000
+; MUBUF-NEXT:    buffer_store_dword v0, v1, s[0:3], s32 offen
+; MUBUF-NEXT:    v_writelane_b32 v0, s33, 0
 ; GCN-NOT: v_mov_b32_e32 v0, 0x100c
-; MUBUF-NEXT: v_mov_b32_e32 v1, 0x100c
-; MUBUF-NEXT: buffer_store_dword v0, v1, s[0:3], s32 offen ; 4-byte Folded Spill
+; MUBUF-NEXT:    v_mov_b32_e32 v1, 0x100c
+; MUBUF-NEXT:    buffer_store_dword v0, v1, s[0:3], s32 offen ; 4-byte Folded Spill
+; MUBUF-NEXT:    v_mov_b32_e32 v0, 0x1000
+; MUBUF-NEXT:    buffer_load_dword v0, v0, s[0:3], s32 offen
+; MUBUF-NEXT:    s_waitcnt vmcnt(0)
 ; FLATSCR: s_add_u32 [[SOFF:s[0-9]+]], s33, 0x1004
 ; FLATSCR: v_mov_b32_e32 v0, 0
 ; FLATSCR: scratch_store_dword off, v0, [[SOFF]]
+; MUBUF:    s_or_saveexec_b64 s[4:5], -1
+; MUBUF-NEXT:    ;;#COPY_INACTIVE_LANES v0
+; MUBUF-NEXT:    v_mov_b32_e32 v1, 0x1000
+; MUBUF-NEXT:    buffer_store_dword v0, v1, s[0:3], s32 offen
+; MUBUF-NEXT:    v_mov_b32_e32 v0, 0x100c
+; MUBUF-NEXT:    buffer_load_dword v0, v0, s[0:3], s32 offen ; 4-byte Folded Reload
+; MUBUF-NEXT:    s_waitcnt vmcnt(0)
+; MUBUF-NEXT:    v_readfirstlane_b32 s33, v0
+; MUBUF-NEXT:    v_mov_b32_e32 v0, 0x1008
+; MUBUF-NEXT:    buffer_load_dword v39, v0, s[0:3], s32 offen ; 4-byte Folded Reload
+; MUBUF-NEXT:    v_mov_b32_e32 v0, 0x1000
+; MUBUF-NEXT:    buffer_load_dword v0, v0, s[0:3], s32 offen
+; MUBUF-NEXT:    s_waitcnt vmcnt(0)
+; MUBUF-NEXT:    s_mov_b64 exec, s[4:5]
 define void @spill_fp_to_memory_scratch_reg_needed_mubuf_offset([4096 x i8] addrspace(5)* byval([4096 x i8]) align 4 %arg) #3 {
   %alloca = alloca i32, addrspace(5)
   store volatile i32 0, i32 addrspace(5)* %alloca
 
   call void asm sideeffect "; clobber nonpreserved SGPRs and 64 CSRs",
     "~{s4},~{s5},~{s6},~{s7},~{s8},~{s9}
     ,~{s10},~{s11},~{s12},~{s13},~{s14},~{s15},~{s16},~{s17},~{s18},~{s19}
     ,~{s20},~{s21},~{s22},~{s23},~{s24},~{s25},~{s26},~{s27},~{s28},~{s29}

llvm/test/CodeGen/AMDGPU/frame-setup-without-sgpr-to-vgpr-spills.ll

-; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs -amdgpu-spill-sgpr-to-vgpr=true < %s | FileCheck -check-prefixes=GCN,SPILL-TO-VGPR %s
-; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs -amdgpu-spill-sgpr-to-vgpr=false < %s | FileCheck -check-prefixes=GCN,NO-SPILL-TO-VGPR %s
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs -amdgpu-spill-sgpr-to-vgpr=true < %s | FileCheck -check-prefix=SPILL-TO-VGPR %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs -amdgpu-spill-sgpr-to-vgpr=false < %s | FileCheck -check-prefix=NO-SPILL-TO-VGPR %s
 
 ; Check frame setup where SGPR spills to VGPRs are disabled or enabled.
 
 declare hidden void @external_void_func_void() #0
 
-; GCN-LABEL: {{^}}callee_with_stack_and_call:
-; SPILL-TO-VGPR:      buffer_store_dword v40, off, s[0:3], s32 offset:4 ; 4-byte Folded Spill
-; SPILL-TO-VGPR:      v_writelane_b32 v40, s33, 2
-; SPILL-TO-VGPR:      v_writelane_b32 v40, s30, 0
-; SPILL-TO-VGPR:      v_writelane_b32 v40, s31, 1
-; NO-SPILL-TO-VGPR:   v_mov_b32_e32 v0, s33
-; NO-SPILL-TO-VGPR:   buffer_store_dword v0, off, s[0:3], s32 offset:12 ; 4-byte Folded Spill
-; NO-SPILL-TO-VGPR:   v_writelane_b32 v0, s30, 0
-; NO-SPILL-TO-VGPR:   v_writelane_b32 v0, s31, 1
-; NO-SPILL-TO-VGPR:   buffer_store_dword v0, off, s[0:3], s33 offset:4 ; 4-byte Folded Spill
-
-; GCN:                s_swappc_b64 s[30:31], s[4:5]
-
-; SPILL-TO-VGPR:      v_readlane_b32 s4, v40, 0
-; SPILL-TO-VGPR:      v_readlane_b32 s5, v40, 1
-; NO-SPILL-TO-VGPR:   buffer_load_dword v0, off, s[0:3], s33 offset:4 ; 4-byte Folded Reload
-; NO-SPILL-TO-VGPR:   v_readlane_b32 s4, v0, 0
-; NO-SPILL-TO-VGPR:   v_readlane_b32 s5, v0, 1
-
-; SPILL-TO-VGPR:      v_readlane_b32 s33, v40, 2
-; NO-SPILL-TO-VGPR:   buffer_load_dword v0, off, s[0:3], s32 offset:12 ; 4-byte Folded Reload
-; NO-SPILL-TO-VGPR:   v_readfirstlane_b32 s33, v0
 define void @callee_with_stack_and_call() #0 {
+; SPILL-TO-VGPR-LABEL: callee_with_stack_and_call:
+; SPILL-TO-VGPR:       ; %bb.0:
+; SPILL-TO-VGPR-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; SPILL-TO-VGPR-NEXT:    s_or_saveexec_b64 s[4:5], -1
+; SPILL-TO-VGPR-NEXT:    buffer_store_dword v40, off, s[0:3], s32 offset:4 ; 4-byte Folded Spill
+; SPILL-TO-VGPR-NEXT:    s_mov_b64 exec, s[4:5]
+; SPILL-TO-VGPR-NEXT:    v_writelane_b32 v40, s33, 2
+; SPILL-TO-VGPR-NEXT:    v_writelane_b32 v40, s30, 0
+; SPILL-TO-VGPR-NEXT:    s_mov_b32 s33, s32
+; SPILL-TO-VGPR-NEXT:    s_add_u32 s32, s32, 0x400
+; SPILL-TO-VGPR-NEXT:    v_mov_b32_e32 v0, 0
+; SPILL-TO-VGPR-NEXT:    s_getpc_b64 s[4:5]
+; SPILL-TO-VGPR-NEXT:    s_add_u32 s4, s4, external_void_func_void@rel32@lo+4
+; SPILL-TO-VGPR-NEXT:    s_addc_u32 s5, s5, external_void_func_void@rel32@hi+12
+; SPILL-TO-VGPR-NEXT:    v_writelane_b32 v40, s31, 1
+; SPILL-TO-VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], s33
+; SPILL-TO-VGPR-NEXT:    s_waitcnt vmcnt(0)
+; SPILL-TO-VGPR-NEXT:    s_swappc_b64 s[30:31], s[4:5]
+; SPILL-TO-VGPR-NEXT:    v_readlane_b32 s4, v40, 0
+; SPILL-TO-VGPR-NEXT:    v_readlane_b32 s5, v40, 1
+; SPILL-TO-VGPR-NEXT:    s_sub_u32 s32, s32, 0x400
+; SPILL-TO-VGPR-NEXT:    v_readlane_b32 s33, v40, 2
+; SPILL-TO-VGPR-NEXT:    s_or_saveexec_b64 s[6:7], -1
+; SPILL-TO-VGPR-NEXT:    buffer_load_dword v40, off, s[0:3], s32 offset:4 ; 4-byte Folded Reload
+; SPILL-TO-VGPR-NEXT:    s_mov_b64 exec, s[6:7]
+; SPILL-TO-VGPR-NEXT:    s_waitcnt vmcnt(0)
+; SPILL-TO-VGPR-NEXT:    s_setpc_b64 s[4:5]
+;
+; NO-SPILL-TO-VGPR-LABEL: callee_with_stack_and_call:
+; NO-SPILL-TO-VGPR:       ; %bb.0:
+; NO-SPILL-TO-VGPR-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; NO-SPILL-TO-VGPR-NEXT:    s_or_saveexec_b64 s[4:5], -1
+; NO-SPILL-TO-VGPR-NEXT:    ;;#COPY_INACTIVE_LANES v0
+; NO-SPILL-TO-VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], s32 offset:4
+; NO-SPILL-TO-VGPR-NEXT:    v_writelane_b32 v0, s33, 0
+; NO-SPILL-TO-VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], s32 offset:12 ; 4-byte Folded Spill
+; NO-SPILL-TO-VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], s32 offset:4
+; NO-SPILL-TO-VGPR-NEXT:    s_waitcnt vmcnt(0)
+; NO-SPILL-TO-VGPR-NEXT:    s_mov_b64 exec, s[4:5]
+; NO-SPILL-TO-VGPR-NEXT:    s_mov_b32 s33, s32
+; NO-SPILL-TO-VGPR-NEXT:    ;;#COPY_INACTIVE_LANES v0
+; NO-SPILL-TO-VGPR-NEXT:    s_add_u32 s32, s32, 0x800
+; NO-SPILL-TO-VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], s33 offset:4
+; NO-SPILL-TO-VGPR-NEXT:    s_not_b64 exec, exec
+; NO-SPILL-TO-VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], s33 offset:4
+; NO-SPILL-TO-VGPR-NEXT:    s_not_b64 exec, exec
+; NO-SPILL-TO-VGPR-NEXT:    v_writelane_b32 v0, s30, 0
+; NO-SPILL-TO-VGPR-NEXT:    v_writelane_b32 v0, s31, 1
+; NO-SPILL-TO-VGPR-NEXT:    s_mov_b64 s[30:31], exec
+; NO-SPILL-TO-VGPR-NEXT:    s_mov_b64 exec, 3
+; NO-SPILL-TO-VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], s33 offset:4 ; 4-byte Folded Spill
+; NO-SPILL-TO-VGPR-NEXT:    s_mov_b64 exec, s[30:31]
+; NO-SPILL-TO-VGPR-NEXT:    v_readlane_b32 s31, v0, 1
+; NO-SPILL-TO-VGPR-NEXT:    v_readlane_b32 s30, v0, 0
+; NO-SPILL-TO-VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], s33 offset:4
+; NO-SPILL-TO-VGPR-NEXT:    s_not_b64 exec, exec
+; NO-SPILL-TO-VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], s33 offset:4
+; NO-SPILL-TO-VGPR-NEXT:    s_waitcnt vmcnt(0)
+; NO-SPILL-TO-VGPR-NEXT:    s_not_b64 exec, exec
+; NO-SPILL-TO-VGPR-NEXT:    v_mov_b32_e32 v0, 0
+; NO-SPILL-TO-VGPR-NEXT:    s_getpc_b64 s[4:5]
+; NO-SPILL-TO-VGPR-NEXT:    s_add_u32 s4, s4, external_void_func_void@rel32@lo+4
+; NO-SPILL-TO-VGPR-NEXT:    s_addc_u32 s5, s5, external_void_func_void@rel32@hi+12
+; NO-SPILL-TO-VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], s33
+; NO-SPILL-TO-VGPR-NEXT:    s_waitcnt vmcnt(0)
+; NO-SPILL-TO-VGPR-NEXT:    s_swappc_b64 s[30:31], s[4:5]
+; NO-SPILL-TO-VGPR-NEXT:    ;;#COPY_INACTIVE_LANES v0
+; NO-SPILL-TO-VGPR-NEXT:    s_or_saveexec_b64 s[4:5], -1
+; NO-SPILL-TO-VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], s33 offset:4
+; NO-SPILL-TO-VGPR-NEXT:    s_mov_b64 exec, s[4:5]
+; NO-SPILL-TO-VGPR-NEXT:    s_mov_b64 s[4:5], exec
+; NO-SPILL-TO-VGPR-NEXT:    s_mov_b64 exec, 3
+; NO-SPILL-TO-VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], s33 offset:4 ; 4-byte Folded Reload
+; NO-SPILL-TO-VGPR-NEXT:    s_mov_b64 exec, s[4:5]
+; NO-SPILL-TO-VGPR-NEXT:    s_waitcnt vmcnt(0)
+; NO-SPILL-TO-VGPR-NEXT:    v_readlane_b32 s4, v0, 0
+; NO-SPILL-TO-VGPR-NEXT:    v_readlane_b32 s5, v0, 1
+; NO-SPILL-TO-VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], s33 offset:4
+; NO-SPILL-TO-VGPR-NEXT:    s_not_b64 exec, exec
+; NO-SPILL-TO-VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], s33 offset:4
+; NO-SPILL-TO-VGPR-NEXT:    s_waitcnt vmcnt(0)
+; NO-SPILL-TO-VGPR-NEXT:    s_not_b64 exec, exec
+; NO-SPILL-TO-VGPR-NEXT:    s_sub_u32 s32, s32, 0x800
+; NO-SPILL-TO-VGPR-NEXT:    s_or_saveexec_b64 s[6:7], -1
+; NO-SPILL-TO-VGPR-NEXT:    ;;#COPY_INACTIVE_LANES v0
+; NO-SPILL-TO-VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], s32 offset:4
+; NO-SPILL-TO-VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], s32 offset:12 ; 4-byte Folded Reload
+; NO-SPILL-TO-VGPR-NEXT:    s_waitcnt vmcnt(0)
+; NO-SPILL-TO-VGPR-NEXT:    v_readfirstlane_b32 s33, v0
+; NO-SPILL-TO-VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], s32 offset:4
+; NO-SPILL-TO-VGPR-NEXT:    s_waitcnt vmcnt(0)
+; NO-SPILL-TO-VGPR-NEXT:    s_mov_b64 exec, s[6:7]
+; NO-SPILL-TO-VGPR-NEXT:    s_setpc_b64 s[4:5]
   %alloca = alloca i32, addrspace(5)
   store volatile i32 0, i32 addrspace(5)* %alloca
   call void @external_void_func_void()
   ret void
 }
 
 attributes #0 = { nounwind }

llvm/test/CodeGen/AMDGPU/stack-realign.ll

; GCN: s_setpc_b64 s[30:31]
   ret void
 }
 
 define void @no_free_regs_spill_bp_to_memory(<32 x i32> %a, i32 %b) #5 {
 ; If there are no free SGPRs or VGPRs available we must spill the BP to memory.
 
 ; GCN-LABEL: no_free_regs_spill_bp_to_mem
 ; GCN: s_or_saveexec_b64 s[4:5], -1
-; GCN: v_mov_b32_e32 v0, s33
-; GCN: buffer_store_dword v0, off, s[0:3], s32
-; GCN: v_mov_b32_e32 v0, s34
-; GCN-DAG: buffer_store_dword v0, off, s[0:3], s32
+; GCN-NEXT: ;;#COPY_INACTIVE_LANES v0
+; GCN-NEXT: buffer_store_dword v0, off, s[0:3], s32 offset:36
+; GCN-NEXT: v_writelane_b32 v0, s33, 0
+; GCN-NEXT: buffer_store_dword v0, off, s[0:3], s32 offset:136 ; 4-byte Folded Spill
+; GCN-NEXT: v_writelane_b32 v0, s34, 0
+; GCN-NEXT: buffer_store_dword v0, off, s[0:3], s32 offset:140 ; 4-byte Folded Spill
+; GCN-NEXT: buffer_load_dword v0, off, s[0:3], s32 offset:36
   %local_val = alloca i32, align 128, addrspace(5)
   store volatile i32 %b, i32 addrspace(5)* %local_val, align 128
 
   call void asm sideeffect "; clobber nonpreserved SGPRs and 64 CSRs",
     "~{s4},~{s5},~{s6},~{s7},~{s8},~{s9}
     ,~{s10},~{s11},~{s12},~{s13},~{s14},~{s15},~{s16},~{s17},~{s18},~{s19}
     ,~{s20},~{s21},~{s22},~{s23},~{s24},~{s25},~{s26},~{s27},~{s28},~{s29}
     ,~{s40},~{s41},~{s42},~{s43},~{s44},~{s45},~{s46},~{s47},~{s48},~{s49}

 }
 
 define void @spill_bp_to_memory_scratch_reg_needed_mubuf_offset(<32 x i32> %a, i32 %b, [4096 x i8] addrspace(5)* byval([4096 x i8]) align 4 %arg) #5 {
 ; If the size of the offset exceeds the MUBUF offset field we need another
 ; scratch VGPR to hold the offset.
 
 ; GCN-LABEL: spill_bp_to_memory_scratch_reg_needed_mubuf_offset
 ; GCN: s_or_saveexec_b64 s[4:5], -1
-; GCN: v_mov_b32_e32 v0, s33
-; GCN-NOT: v_mov_b32_e32 v0, 0x1088
-; GCN-NEXT: v_mov_b32_e32 v1, 0x1088
+; GCN-NEXT: v_mov_b32_e32 v0, 0x1084
+; GCN-NEXT: buffer_store_dword v39, v0, s[0:3], s32 offen ; 4-byte Folded Spill
+; GCN-NEXT: ;;#COPY_INACTIVE_LANES v0
+; GCN-NEXT: v_mov_b32_e32 v1, 0x1024
 ; GCN-NEXT: buffer_store_dword v0, v1, s[0:3], s32 offen
-; GCN: v_mov_b32_e32 v0, s34
-; GCN-NOT: v_mov_b32_e32 v0, 0x108c
+; GCN-NEXT: v_writelane_b32 v0, s33, 0
+; GCN-NEXT: v_mov_b32_e32 v1, 0x1088
+; GCN-NEXT: buffer_store_dword v0, v1, s[0:3], s32 offen ; 4-byte Folded Spill
+; GCN-NEXT: v_writelane_b32 v0, s34, 0
 ; GCN-NEXT: v_mov_b32_e32 v1, 0x108c
-; GCN-NEXT: buffer_store_dword v0, v1, s[0:3], s32 offen
+; GCN-NEXT: buffer_store_dword v0, v1, s[0:3], s32 offen ; 4-byte Folded Spill
+; GCN-NEXT: v_mov_b32_e32 v0, 0x1024
+; GCN-NEXT: buffer_load_dword v0, v0, s[0:3], s32 offen
+; GCN-NEXT: s_waitcnt vmcnt(0)
   %local_val = alloca i32, align 128, addrspace(5)
   store volatile i32 %b, i32 addrspace(5)* %local_val, align 128
 
   call void asm sideeffect "; clobber nonpreserved SGPRs and 64 CSRs",
     "~{s4},~{s5},~{s6},~{s7},~{s8},~{s9}
     ,~{s10},~{s11},~{s12},~{s13},~{s14},~{s15},~{s16},~{s17},~{s18},~{s19}
     ,~{s20},~{s21},~{s22},~{s23},~{s24},~{s25},~{s26},~{s27},~{s28},~{s29}
     ,~{s40},~{s41},~{s42},~{s43},~{s44},~{s45},~{s46},~{s47},~{s48},~{s49}