[RISCV] Split SP adjustment to reduce the offset of callee saved register spill and restore

[shiva0217]

Summary

We would like to split the SP adjustment to reduce the instructions in prologue and epilogue as the following case. In this way, the offset of the callee saved register could fit in a single store.

add     sp,sp,-2032
sw      ra,2028(sp)
sw      s0,2024(sp)
sw      s1,2020(sp)
sw      s3,2012(sp)
sw      s4,2008(sp)
add     sp,sp,-64

[luismarques]

lib/Target/RISCV/RISCVFrameLowering.h

49

I'm not sure we need to split things into the boolean and the adjustment amount. Why not just have a single function, and if the returned initial adjustment is 0 then we just do the normal SP adjustment, otherwise we do it in two steps?

[luismarques]

lib/Target/RISCV/RISCVFrameLowering.cpp

248

If SecondSPAdjustmentAmount == 0 then haven't we done something wrong? The first SP adjustment becomes the only one, so isSplitSPAdjust can't logically be true, right? If you keep the separate boolean/amount, then I would suggest asserting that SecondSPAdjustmentAmount > 0 when isSplitSPAdjust is true.

[shiva0217]

lib/Target/RISCV/RISCVFrameLowering.cpp

248

When the isSplitSPAdjust return true, the StackSize > 2048, and the FirstSPAdjustAmount = 2048 - StackAlign. So SecondSPAdjustmentAmount = StackSize - FirstSPAdjustAmount should always greater than zero when isSplitSPAdjust return true. We probably could remove the SecondSPAdjustAmount > 0 in the if line. I think it's great idea to add assertion for SecondSPAdjustmentAmount > 0 to catch anything we unexpect even if we merge the functions, thanks.

lib/Target/RISCV/RISCVFrameLowering.h

49

Good idea, I'll merge the functions, thanks.

[shiva0217]

Update patch to address the comments.

Hi @luismarques, thanks for the review :)

[luismarques]

getFirstSPAdjustAmount does a non-trivial amount of work. Instead of calling it repeatedly it would be better to call it once per prologue/epilogue generation, and cache the result in a local variable.
With that fixed it LGTM.

[shiva0217]

Update patch

1. Call getFirstSPAdjustAmount() once in per prologue/epilogue generation as @luismarques suggest
2. Generate CFI directives in large-stack.ll test case to check the CFI generation

[shiva0217]

Update patch

Reducing FP setting offset by setting FP before second SP adjustment.

[luismarques]

In D68011#1683772, @shiva0217 wrote:

Update patch

@shiva0217 Thanks for updating the patch and addressing the previous concerns. This is looking quite good. I have just a couple more concerns before approving this:

• The boundary condition for doing the two-stage sp adjustment probably needs adjustment. For instance, try this:

int main() {
    char xx[2048-16];
    foo(xx);
}
--
        addi    sp, sp, -2032
        sd      ra, 2024(sp)
        addi    sp, sp, -16
...

I think that one would still fit a single-stage sp update. Related to that, where you have the code if (!isInt<12>(StackSize) && (CSI.size() > 0)), be sure that the isInt<12> check properly accounts for the StackSize, CSI size, StackAlign etc. Which brings me to the second point...

• Please include tests showing that the boundary condition is correctly computed. For instance, one test where the stack size is just enough not to have to split the sp adjustment and another where the adjustment is needed.

Thanks!

[shiva0217]

In D68011#1683996, @luismarques wrote:

In D68011#1683772, @shiva0217 wrote:

Update patch

@shiva0217 Thanks for updating the patch and addressing the previous concerns. This is looking quite good. I have just a couple more concerns before approving this:

• The boundary condition for doing the two-stage sp adjustment probably needs adjustment. For instance, try this:

int main() {
    char xx[2048-16];
    foo(xx);
}
--
        addi    sp, sp, -2032
        sd      ra, 2024(sp)
        addi    sp, sp, -16
...

I think that one would still fit a single-stage sp update. Related to that, where you have the code if (!isInt<12>(StackSize) && (CSI.size() > 0)), be sure that the isInt<12> check properly accounts for the StackSize, CSI size, StackAlign etc. Which brings me to the second point...

• Please include tests showing that the boundary condition is correctly computed. For instance, one test where the stack size is just enough not to have to split the sp adjustment and another where the adjustment is needed.

Thanks!

Hi @luismarques,

Thanks for the boundary test case, I'll add the tests that one will split and the other will not.
The case will split in the prologue because in the epilogue +2048 will not fit in signed 12-bit.
The StackSize will contain the callee saved stack area and the StackSize has been aligned by determineFrameLayout();
The condition CSI.size() > 0 is to check that there exists a callee saved register spill.
GCC will split in the prologue and the epilogue in this case to simplify the split logic.

main:
      add     sp,sp,-2032
      sw      ra,2028(sp)
      add     sp,sp,-16
      mv      a0,sp
      call    foo
      add     sp,sp,16
      lw      ra,2028(sp)
      li      a0,0
      add     sp,sp,2032
      jr      ra
      .size   main, .-main
      .ident  "GCC:

Should we sync the GCC behaviour in this case?

[shiva0217]

Add splitting SP adjustment boundary test case as @luismarques suggest.

[luismarques]

LGTM, after fixing the comment.

lib/Target/RISCV/RISCVFrameLowering.cpp

194

This is the prologue so it should say "after saving", not "after restoring".

test/CodeGen/RISCV/split-sp-adjust.ll

20 ↗

(On Diff #222078)

This test would be more clear if the alloca size and the computed stack size better matched, but getStackSize is overestimating the size, as indicated in a comment copied from Lanai. I think this can be committed as is, but we should investigate that stack overallocation and if/when we fix that we should adjust this test.

[shiva0217]

lib/Target/RISCV/RISCVFrameLowering.cpp

194

Thanks for catching this.

test/CodeGen/RISCV/split-sp-adjust.ll

20 ↗

(On Diff #222078)

Ok, we could update the test when the stack estimation changing.

[shiva0217]

Fix comment in the patch.

Hi @luismarques, thanks for the review, really appreciate that.

lib/Target/RISCV/RISCVFrameLowering.h

public:
 
   bool hasFP(const MachineFunction &MF) const override;
 
   bool hasReservedCallFrame(const MachineFunction &MF) const override;
   MachineBasicBlock::iterator
   eliminateCallFramePseudoInstr(MachineFunction &MF, MachineBasicBlock &MBB,
                                 MachineBasicBlock::iterator MI) const override;
 
+  // Get the first stack adjustment amount for SplitSPAdjust.
+  // Return 0 if we don't want to to split the SP adjustment in prologue and
+  // epilogue.
+  uint64_t getFirstSPAdjustAmount(const MachineFunction &MF) const;
+
 protected:
   const RISCVSubtarget &STI;
 
 private:
   void determineFrameLayout(MachineFunction &MF) const;
   void adjustReg(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
                  const DebugLoc &DL, Register DestReg, Register SrcReg,
                  int64_t Val, MachineInstr::MIFlag Flag) const;
 };
 }
 #endif

lib/Target/RISCV/RISCVFrameLowering.cpp

void RISCVFrameLowering::emitPrologue(MachineFunction &MF,
   // FIXME (note copied from Lanai): This appears to be overallocating.  Needs
   // investigation. Get the number of bytes to allocate from the FrameInfo.
   uint64_t StackSize = MFI.getStackSize();
 
   // Early exit if there is no need to allocate on the stack
   if (StackSize == 0 && !MFI.adjustsStack())
     return;
 
+  // Split the SP adjustment to reduce the offsets of callee saved spill.
+  if (getFirstSPAdjustAmount(MF))
+    StackSize = getFirstSPAdjustAmount(MF);
+
   // Allocate space on the stack if necessary.
   adjustReg(MBB, MBBI, DL, SPReg, SPReg, -StackSize, MachineInstr::FrameSetup);
 
   // Emit ".cfi_def_cfa_offset StackSize"
   unsigned CFIIndex = MF.addFrameInst(
       MCCFIInstruction::createDefCfaOffset(nullptr, -StackSize));
   BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
       .addCFIIndex(CFIIndex);
for (const auto &Entry : CSI) {
     int64_t Offset = MFI.getObjectOffset(Entry.getFrameIdx());
     Register Reg = Entry.getReg();
     unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::createOffset(
         nullptr, RI->getDwarfRegNum(Reg, true), Offset));
     BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
         .addCFIIndex(CFIIndex);
   }
 
+  // Emit the second SP adjustment after restoring callee saved registers.
+  if (getFirstSPAdjustAmount(MF)) {
+    uint64_t SecondSPAdjustAmount = MFI.getStackSize() -
+                                    getFirstSPAdjustAmount(MF);
+    assert(SecondSPAdjustAmount > 0 &&
+           "SecondSPAdjustAmount should be greater than zero");
+    adjustReg(MBB, MBBI, DL, SPReg, SPReg, -SecondSPAdjustAmount,
+              MachineInstr::FrameSetup);
+    // Emit ".cfi_def_cfa_offset StackSize"
+    unsigned CFIIndex = MF.addFrameInst(
+        MCCFIInstruction::createDefCfaOffset(nullptr, -MFI.getStackSize()));
+    BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
+        .addCFIIndex(CFIIndex);
+  }
+
   // Generate new FP.
   if (hasFP(MF)) {
     adjustReg(MBB, MBBI, DL, FPReg, SPReg,
-              StackSize - RVFI->getVarArgsSaveSize(), MachineInstr::FrameSetup);
+              MFI.getStackSize() - RVFI->getVarArgsSaveSize(),
+              MachineInstr::FrameSetup);
 
     // Emit ".cfi_def_cfa $fp, 0"
     unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::createDefCfa(
         nullptr, RI->getDwarfRegNum(FPReg, true), 0));
     BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
         .addCFIIndex(CFIIndex);
 
     // Realign Stack
     const RISCVRegisterInfo *RI = STI.getRegisterInfo();
     if (RI->needsStackRealignment(MF)) {
       unsigned MaxAlignment = MFI.getMaxAlignment();
 
       const RISCVInstrInfo *TII = STI.getInstrInfo();
       if (isInt<12>(-(int)MaxAlignment)) {
         BuildMI(MBB, MBBI, DL, TII->get(RISCV::ANDI), SPReg)
             .addReg(SPReg)
void RISCVFrameLowering::emitEpilogue(MachineFunction &MF,
   // necessary if the stack pointer was modified, meaning the stack size is
   // unknown.
   if (RI->needsStackRealignment(MF) || MFI.hasVarSizedObjects()) {
     assert(hasFP(MF) && "frame pointer should not have been eliminated");
     adjustReg(MBB, LastFrameDestroy, DL, SPReg, FPReg, -FPOffset,
               MachineInstr::FrameDestroy);
   }
 
+  if (getFirstSPAdjustAmount(MF)) {
+    uint64_t SecondSPAdjustAmount = MFI.getStackSize() -
+                                    getFirstSPAdjustAmount(MF);
+    assert(SecondSPAdjustAmount > 0 &&
+           "SecondSPAdjustAmount should be greater than zero");
+
+    adjustReg(MBB, LastFrameDestroy, DL, SPReg, SPReg, SecondSPAdjustAmount,
+              MachineInstr::FrameDestroy);
+
+    // Emit ".cfi_def_cfa_offset getFirstSPAdjustAmount(MF)"
+    unsigned CFIIndex =
+        MF.addFrameInst(
+             MCCFIInstruction::createDefCfaOffset(nullptr,
+                                                  -getFirstSPAdjustAmount(MF)));
+    BuildMI(MBB, LastFrameDestroy, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
+        .addCFIIndex(CFIIndex);
+  }
+
   if (hasFP(MF)) {
     // To find the instruction restoring FP from stack.
     for (auto &I = LastFrameDestroy; I != MBBI; ++I) {
       if (I->mayLoad() && I->getOperand(0).isReg()) {
         Register DestReg = I->getOperand(0).getReg();
         if (DestReg == FPReg) {
           // If there is frame pointer, after restoring $fp registers, we
           // need adjust CFA to ($sp - FPOffset).
void RISCVFrameLowering::emitEpilogue(MachineFunction &MF,
   for (const auto &Entry : CSI) {
     Register Reg = Entry.getReg();
     unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::createRestore(
         nullptr, RI->getDwarfRegNum(Reg, true)));
     BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
         .addCFIIndex(CFIIndex);
   }
 
+  if (getFirstSPAdjustAmount(MF))
+    StackSize = getFirstSPAdjustAmount(MF);
+
   // Deallocate stack
   adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackSize, MachineInstr::FrameDestroy);
 
   // After restoring $sp, we need to adjust CFA to $(sp + 0)
   // Emit ".cfi_def_cfa_offset 0"
   unsigned CFIIndex =
       MF.addFrameInst(MCCFIInstruction::createDefCfaOffset(nullptr, 0));
   BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
int RISCVFrameLowering::getFrameIndexReference(const MachineFunction &MF,
 
   if (CSI.size()) {
     MinCSFI = CSI[0].getFrameIdx();
     MaxCSFI = CSI[CSI.size() - 1].getFrameIdx();
   }
 
   if (FI >= MinCSFI && FI <= MaxCSFI) {
     FrameReg = RISCV::X2;
+
+    if (getFirstSPAdjustAmount(MF))
+      Offset += getFirstSPAdjustAmount(MF);
+    else
       Offset += MF.getFrameInfo().getStackSize();
   } else if (RI->needsStackRealignment(MF)) {
     assert(!MFI.hasVarSizedObjects() &&
            "Unexpected combination of stack realignment and varsized objects");
     // If the stack was realigned, the frame pointer is set in order to allow
     // SP to be restored, but we still access stack objects using SP.
     FrameReg = RISCV::X2;
     Offset += MF.getFrameInfo().getStackSize();
   } else {
if (Amount != 0) {
         Amount = -Amount;
 
       adjustReg(MBB, MI, DL, SPReg, SPReg, Amount, MachineInstr::NoFlags);
     }
   }
 
   return MBB.erase(MI);
 }
+
+// We would like to split the SP adjustment to reduce prologue/epilogue
+// as following instructions. In this way, the offset of the callee saved
+// register could fit in a single store.
+//   add     sp,sp,-2032
+//   sw      ra,2028(sp)
+//   sw      s0,2024(sp)
+//   sw      s1,2020(sp)
+//   sw      s3,2012(sp)
+//   sw      s4,2008(sp)
+//   add     sp,sp,-64
+uint64_t
+RISCVFrameLowering::getFirstSPAdjustAmount(const MachineFunction &MF) const {
+  const MachineFrameInfo &MFI = MF.getFrameInfo();
+  const std::vector<CalleeSavedInfo> &CSI = MFI.getCalleeSavedInfo();
+  uint64_t StackSize = MFI.getStackSize();
+  uint64_t StackAlign = getStackAlignment();
+
+  // FIXME: Disable SplitSPAdjust if save-restore libcall enabled when the patch
+  //        landing. The callee saved registers will be pushed by the
+  //        save-restore libcalls, so we don't have to split the SP adjustment
+  //        in this case.
+  if (!isInt<12>(StackSize) && (CSI.size() > 0)) {
+    // We choose 2048 - StackAlign because 2048 will cause sp = sp + 2048 in
+    // epilogue split into multi-instructions. The offset smaller than 2048 can
+    // fit in signle load/store instruction and we have to stick with the stack
+    // alignment.
+    return 2048 - StackAlign;
+  }
+  return 0;
+}

test/CodeGen/RISCV/large-stack.ll

 ; RV32I-FPELIM-NEXT:    sub sp, sp, a0
 ; RV32I-FPELIM-NEXT:    lui a0, 74565
 ; RV32I-FPELIM-NEXT:    addi a0, a0, 1664
 ; RV32I-FPELIM-NEXT:    add sp, sp, a0
 ; RV32I-FPELIM-NEXT:    ret
 ;
 ; RV32I-WITHFP-LABEL: test:
 ; RV32I-WITHFP:       # %bb.0:
+; RV32I-WITHFP-NEXT:    addi sp, sp, -2032
+; RV32I-WITHFP-NEXT:    sw ra, 2028(sp)
+; RV32I-WITHFP-NEXT:    sw s0, 2024(sp)
 ; RV32I-WITHFP-NEXT:    lui a0, 74565
-; RV32I-WITHFP-NEXT:    addi a0, a0, 1680
+; RV32I-WITHFP-NEXT:    addi a0, a0, -352
 ; RV32I-WITHFP-NEXT:    sub sp, sp, a0
 ; RV32I-WITHFP-NEXT:    lui a0, 74565
-; RV32I-WITHFP-NEXT:    addi a0, a0, 1676
-; RV32I-WITHFP-NEXT:    add a0, sp, a0
-; RV32I-WITHFP-NEXT:    sw ra, 0(a0)
-; RV32I-WITHFP-NEXT:    lui a0, 74565
-; RV32I-WITHFP-NEXT:    addi a0, a0, 1672
-; RV32I-WITHFP-NEXT:    add a0, sp, a0
-; RV32I-WITHFP-NEXT:    sw s0, 0(a0)
-; RV32I-WITHFP-NEXT:    lui a0, 74565
 ; RV32I-WITHFP-NEXT:    addi a0, a0, 1680
 ; RV32I-WITHFP-NEXT:    add s0, sp, a0
 ; RV32I-WITHFP-NEXT:    lui a0, 74565
-; RV32I-WITHFP-NEXT:    addi a0, a0, 1672
-; RV32I-WITHFP-NEXT:    add a0, sp, a0
-; RV32I-WITHFP-NEXT:    lw s0, 0(a0)
-; RV32I-WITHFP-NEXT:    lui a0, 74565
-; RV32I-WITHFP-NEXT:    addi a0, a0, 1676
-; RV32I-WITHFP-NEXT:    add a0, sp, a0
-; RV32I-WITHFP-NEXT:    lw ra, 0(a0)
-; RV32I-WITHFP-NEXT:    lui a0, 74565
-; RV32I-WITHFP-NEXT:    addi a0, a0, 1680
+; RV32I-WITHFP-NEXT:    addi a0, a0, -352
 ; RV32I-WITHFP-NEXT:    add sp, sp, a0
+; RV32I-WITHFP-NEXT:    lw s0, 2024(sp)
+; RV32I-WITHFP-NEXT:    lw ra, 2028(sp)
+; RV32I-WITHFP-NEXT:    addi sp, sp, 2032
 ; RV32I-WITHFP-NEXT:    ret
   %tmp = alloca [ 305419896 x i8 ] , align 4
   ret void
 }
 
 ; This test case artificially produces register pressure which should force
 ; use of the emergency spill slot.
 
 define void @test_emergency_spill_slot(i32 %a) nounwind {
 ; RV32I-FPELIM-LABEL: test_emergency_spill_slot:
 ; RV32I-FPELIM:       # %bb.0:
-; RV32I-FPELIM-NEXT:    lui a1, 98
-; RV32I-FPELIM-NEXT:    addi a1, a1, -1392
+; RV32I-FPELIM-NEXT:    addi sp, sp, -2032
+; RV32I-FPELIM-NEXT:    sw s0, 2028(sp)
+; RV32I-FPELIM-NEXT:    sw s1, 2024(sp)
+; RV32I-FPELIM-NEXT:    lui a1, 97
+; RV32I-FPELIM-NEXT:    addi a1, a1, 672
 ; RV32I-FPELIM-NEXT:    sub sp, sp, a1
-; RV32I-FPELIM-NEXT:    lui a1, 98
-; RV32I-FPELIM-NEXT:    addi a1, a1, -1396
-; RV32I-FPELIM-NEXT:    add a1, sp, a1
-; RV32I-FPELIM-NEXT:    sw s0, 0(a1)
-; RV32I-FPELIM-NEXT:    lui a1, 98
-; RV32I-FPELIM-NEXT:    addi a1, a1, -1400
-; RV32I-FPELIM-NEXT:    add a1, sp, a1
-; RV32I-FPELIM-NEXT:    sw s1, 0(a1)
 ; RV32I-FPELIM-NEXT:    lui a1, 78
 ; RV32I-FPELIM-NEXT:    addi a1, a1, 512
 ; RV32I-FPELIM-NEXT:    addi a2, sp, 8
 ; RV32I-FPELIM-NEXT:    add a1, a2, a1
 ; RV32I-FPELIM-NEXT:    #APP
 ; RV32I-FPELIM-NEXT:    nop
 ; RV32I-FPELIM-NEXT:    #NO_APP
 ; RV32I-FPELIM-NEXT:    sw a0, 0(a1)
 ; RV32I-FPELIM-NEXT:    #APP
 ; RV32I-FPELIM-NEXT:    nop
 ; RV32I-FPELIM-NEXT:    #NO_APP
-; RV32I-FPELIM-NEXT:    lui a0, 98
-; RV32I-FPELIM-NEXT:    addi a0, a0, -1400
-; RV32I-FPELIM-NEXT:    add a0, sp, a0
-; RV32I-FPELIM-NEXT:    lw s1, 0(a0)
-; RV32I-FPELIM-NEXT:    lui a0, 98
-; RV32I-FPELIM-NEXT:    addi a0, a0, -1396
-; RV32I-FPELIM-NEXT:    add a0, sp, a0
-; RV32I-FPELIM-NEXT:    lw s0, 0(a0)
-; RV32I-FPELIM-NEXT:    lui a0, 98
-; RV32I-FPELIM-NEXT:    addi a0, a0, -1392
+; RV32I-FPELIM-NEXT:    lui a0, 97
+; RV32I-FPELIM-NEXT:    addi a0, a0, 672
 ; RV32I-FPELIM-NEXT:    add sp, sp, a0
+; RV32I-FPELIM-NEXT:    lw s1, 2024(sp)
+; RV32I-FPELIM-NEXT:    lw s0, 2028(sp)
+; RV32I-FPELIM-NEXT:    addi sp, sp, 2032
 ; RV32I-FPELIM-NEXT:    ret
 ;
 ; RV32I-WITHFP-LABEL: test_emergency_spill_slot:
 ; RV32I-WITHFP:       # %bb.0:
-; RV32I-WITHFP-NEXT:    lui a1, 98
-; RV32I-WITHFP-NEXT:    addi a1, a1, -1376
+; RV32I-WITHFP-NEXT:    addi sp, sp, -2032
+; RV32I-WITHFP-NEXT:    sw ra, 2028(sp)
+; RV32I-WITHFP-NEXT:    sw s0, 2024(sp)
+; RV32I-WITHFP-NEXT:    sw s1, 2020(sp)
+; RV32I-WITHFP-NEXT:    sw s2, 2016(sp)
+; RV32I-WITHFP-NEXT:    lui a1, 97
+; RV32I-WITHFP-NEXT:    addi a1, a1, 688
 ; RV32I-WITHFP-NEXT:    sub sp, sp, a1
 ; RV32I-WITHFP-NEXT:    lui a1, 98
-; RV32I-WITHFP-NEXT:    addi a1, a1, -1380
-; RV32I-WITHFP-NEXT:    add a1, sp, a1
-; RV32I-WITHFP-NEXT:    sw ra, 0(a1)
-; RV32I-WITHFP-NEXT:    lui a1, 98
-; RV32I-WITHFP-NEXT:    addi a1, a1, -1384
-; RV32I-WITHFP-NEXT:    add a1, sp, a1
-; RV32I-WITHFP-NEXT:    sw s0, 0(a1)
-; RV32I-WITHFP-NEXT:    lui a1, 98
-; RV32I-WITHFP-NEXT:    addi a1, a1, -1388
-; RV32I-WITHFP-NEXT:    add a1, sp, a1
-; RV32I-WITHFP-NEXT:    sw s1, 0(a1)
-; RV32I-WITHFP-NEXT:    lui a1, 98
-; RV32I-WITHFP-NEXT:    addi a1, a1, -1392
-; RV32I-WITHFP-NEXT:    add a1, sp, a1
-; RV32I-WITHFP-NEXT:    sw s2, 0(a1)
-; RV32I-WITHFP-NEXT:    lui a1, 98
 ; RV32I-WITHFP-NEXT:    addi a1, a1, -1376
 ; RV32I-WITHFP-NEXT:    add s0, sp, a1
 ; RV32I-WITHFP-NEXT:    lui a1, 78
 ; RV32I-WITHFP-NEXT:    addi a1, a1, 512
 ; RV32I-WITHFP-NEXT:    lui a2, 1048478
 ; RV32I-WITHFP-NEXT:    addi a2, a2, 1388
 ; RV32I-WITHFP-NEXT:    add a2, s0, a2
 ; RV32I-WITHFP-NEXT:    mv a2, a2
 ; RV32I-WITHFP-NEXT:    add a1, a2, a1
 ; RV32I-WITHFP-NEXT:    #APP
 ; RV32I-WITHFP-NEXT:    nop
 ; RV32I-WITHFP-NEXT:    #NO_APP
 ; RV32I-WITHFP-NEXT:    sw a0, 0(a1)
 ; RV32I-WITHFP-NEXT:    #APP
 ; RV32I-WITHFP-NEXT:    nop
 ; RV32I-WITHFP-NEXT:    #NO_APP
-; RV32I-WITHFP-NEXT:    lui a0, 98
-; RV32I-WITHFP-NEXT:    addi a0, a0, -1392
-; RV32I-WITHFP-NEXT:    add a0, sp, a0
-; RV32I-WITHFP-NEXT:    lw s2, 0(a0)
-; RV32I-WITHFP-NEXT:    lui a0, 98
-; RV32I-WITHFP-NEXT:    addi a0, a0, -1388
-; RV32I-WITHFP-NEXT:    add a0, sp, a0
-; RV32I-WITHFP-NEXT:    lw s1, 0(a0)
-; RV32I-WITHFP-NEXT:    lui a0, 98
-; RV32I-WITHFP-NEXT:    addi a0, a0, -1384
-; RV32I-WITHFP-NEXT:    add a0, sp, a0
-; RV32I-WITHFP-NEXT:    lw s0, 0(a0)
-; RV32I-WITHFP-NEXT:    lui a0, 98
-; RV32I-WITHFP-NEXT:    addi a0, a0, -1380
-; RV32I-WITHFP-NEXT:    add a0, sp, a0
-; RV32I-WITHFP-NEXT:    lw ra, 0(a0)
-; RV32I-WITHFP-NEXT:    lui a0, 98
-; RV32I-WITHFP-NEXT:    addi a0, a0, -1376
+; RV32I-WITHFP-NEXT:    lui a0, 97
+; RV32I-WITHFP-NEXT:    addi a0, a0, 688
 ; RV32I-WITHFP-NEXT:    add sp, sp, a0
+; RV32I-WITHFP-NEXT:    lw s2, 2016(sp)
+; RV32I-WITHFP-NEXT:    lw s1, 2020(sp)
+; RV32I-WITHFP-NEXT:    lw s0, 2024(sp)
+; RV32I-WITHFP-NEXT:    lw ra, 2028(sp)
+; RV32I-WITHFP-NEXT:    addi sp, sp, 2032
 ; RV32I-WITHFP-NEXT:    ret
   %data = alloca [ 100000 x i32 ] , align 4
   %ptr = getelementptr inbounds [100000 x i32], [100000 x i32]* %data, i32 0, i32 80000
   %1 = tail call { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } asm sideeffect "nop", "=r,=r,=r,=r,=r,=r,=r,=r,=r,=r,=r,=r,=r,=r,=r"()
   %asmresult0 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %1, 0
   %asmresult1 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %1, 1
   %asmresult2 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %1, 2
   %asmresult3 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %1, 3

test/CodeGen/RISCV/rv64-large-stack.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=riscv64 -verify-machineinstrs < %s | FileCheck %s
 ;
 ; The test case check that RV64 could handle the stack adjustment offset exceed
 ; 32-bit.
 
 define void @foo() nounwind {
 ; CHECK-LABEL: foo:
 ; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    addi sp, sp, -2032
+; CHECK-NEXT:    sd ra, 2024(sp)
 ; CHECK-NEXT:    lui a0, 95
 ; CHECK-NEXT:    addiw a0, a0, 1505
 ; CHECK-NEXT:    slli a0, a0, 13
-; CHECK-NEXT:    addi a0, a0, 32
+; CHECK-NEXT:    addi a0, a0, -2000
 ; CHECK-NEXT:    sub sp, sp, a0
-; CHECK-NEXT:    lui a0, 781250
-; CHECK-NEXT:    addiw a0, a0, 24
-; CHECK-NEXT:    add a0, sp, a0
-; CHECK-NEXT:    sd ra, 0(a0)
 ; CHECK-NEXT:    addi a0, sp, 16
 ; CHECK-NEXT:    call baz
-; CHECK-NEXT:    lui a0, 781250
-; CHECK-NEXT:    addiw a0, a0, 24
-; CHECK-NEXT:    add a0, sp, a0
-; CHECK-NEXT:    ld ra, 0(a0)
 ; CHECK-NEXT:    lui a0, 95
 ; CHECK-NEXT:    addiw a0, a0, 1505
 ; CHECK-NEXT:    slli a0, a0, 13
-; CHECK-NEXT:    addi a0, a0, 32
+; CHECK-NEXT:    addi a0, a0, -2000
 ; CHECK-NEXT:    add sp, sp, a0
+; CHECK-NEXT:    ld ra, 2024(sp)
+; CHECK-NEXT:    addi sp, sp, 2032
 ; CHECK-NEXT:    ret
 entry:
   %w = alloca [100000000 x { fp128, fp128 }], align 16
   %arraydecay = getelementptr inbounds [100000000 x { fp128, fp128 }], [100000000 x { fp128, fp128 }]* %w, i64 0, i64 0
   call void @baz({ fp128, fp128 }* nonnull %arraydecay)
   ret void
 }
 
 declare void @baz({ fp128, fp128 }*)

test/CodeGen/RISCV/stack-realignment.ll

; RV64I-NEXT:    ret
   %1 = alloca i8, align 512
   call void @callee(i8* %1)
   ret void
 }
 
 define void @caller1024() nounwind {
 ; RV32I-LABEL: caller1024:
 ; RV32I:       # %bb.0:
-; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, -1024
-; RV32I-NEXT:    sub sp, sp, a0
-; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, -1028
-; RV32I-NEXT:    add a0, sp, a0
-; RV32I-NEXT:    sw ra, 0(a0)
-; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, -1032
-; RV32I-NEXT:    add a0, sp, a0
-; RV32I-NEXT:    sw s0, 0(a0)
+; RV32I-NEXT:    addi sp, sp, -2032
+; RV32I-NEXT:    sw ra, 2028(sp)
+; RV32I-NEXT:    sw s0, 2024(sp)
+; RV32I-NEXT:    addi sp, sp, -1040
 ; RV32I-NEXT:    lui a0, 1
 ; RV32I-NEXT:    addi a0, a0, -1024
 ; RV32I-NEXT:    add s0, sp, a0
 ; RV32I-NEXT:    andi sp, sp, -1024
 ; RV32I-NEXT:    lui a0, 1
 ; RV32I-NEXT:    addi a0, a0, -2048
 ; RV32I-NEXT:    add a0, sp, a0
 ; RV32I-NEXT:    mv a0, a0
 ; RV32I-NEXT:    call callee
 ; RV32I-NEXT:    lui a0, 1
 ; RV32I-NEXT:    addi a0, a0, -1024
 ; RV32I-NEXT:    sub sp, s0, a0
-; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, -1032
-; RV32I-NEXT:    add a0, sp, a0
-; RV32I-NEXT:    lw s0, 0(a0)
-; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, -1028
-; RV32I-NEXT:    add a0, sp, a0
-; RV32I-NEXT:    lw ra, 0(a0)
-; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, -1024
-; RV32I-NEXT:    add sp, sp, a0
+; RV32I-NEXT:    addi sp, sp, 1040
+; RV32I-NEXT:    lw s0, 2024(sp)
+; RV32I-NEXT:    lw ra, 2028(sp)
+; RV32I-NEXT:    addi sp, sp, 2032
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: caller1024:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, -1024
-; RV64I-NEXT:    sub sp, sp, a0
-; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, -1032
-; RV64I-NEXT:    add a0, sp, a0
-; RV64I-NEXT:    sd ra, 0(a0)
-; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, -1040
-; RV64I-NEXT:    add a0, sp, a0
-; RV64I-NEXT:    sd s0, 0(a0)
+; RV64I-NEXT:    addi sp, sp, -2032
+; RV64I-NEXT:    sd ra, 2024(sp)
+; RV64I-NEXT:    sd s0, 2016(sp)
+; RV64I-NEXT:    addi sp, sp, -1040
 ; RV64I-NEXT:    lui a0, 1
 ; RV64I-NEXT:    addiw a0, a0, -1024
 ; RV64I-NEXT:    add s0, sp, a0
 ; RV64I-NEXT:    andi sp, sp, -1024
 ; RV64I-NEXT:    lui a0, 1
 ; RV64I-NEXT:    addiw a0, a0, -2048
 ; RV64I-NEXT:    add a0, sp, a0
 ; RV64I-NEXT:    mv a0, a0
 ; RV64I-NEXT:    call callee
 ; RV64I-NEXT:    lui a0, 1
 ; RV64I-NEXT:    addiw a0, a0, -1024
 ; RV64I-NEXT:    sub sp, s0, a0
-; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, -1040
-; RV64I-NEXT:    add a0, sp, a0
-; RV64I-NEXT:    ld s0, 0(a0)
-; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, -1032
-; RV64I-NEXT:    add a0, sp, a0
-; RV64I-NEXT:    ld ra, 0(a0)
-; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, -1024
-; RV64I-NEXT:    add sp, sp, a0
+; RV64I-NEXT:    addi sp, sp, 1040
+; RV64I-NEXT:    ld s0, 2016(sp)
+; RV64I-NEXT:    ld ra, 2024(sp)
+; RV64I-NEXT:    addi sp, sp, 2032
 ; RV64I-NEXT:    ret
   %1 = alloca i8, align 1024
   call void @callee(i8* %1)
   ret void
 }
 
 define void @caller_no_realign1024() nounwind "no-realign-stack" {
 ; RV32I-LABEL: caller_no_realign1024:
; RV64I-NEXT:    ret
   %1 = alloca i8, align 1024
   call void @callee(i8* %1)
   ret void
 }
 
 define void @caller2048() nounwind {
 ; RV32I-LABEL: caller2048:
 ; RV32I:       # %bb.0:
-; RV32I-NEXT:    lui a0, 2
-; RV32I-NEXT:    addi a0, a0, -2048
-; RV32I-NEXT:    sub sp, sp, a0
-; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, 2044
-; RV32I-NEXT:    add a0, sp, a0
-; RV32I-NEXT:    sw ra, 0(a0)
+; RV32I-NEXT:    addi sp, sp, -2032
+; RV32I-NEXT:    sw ra, 2028(sp)
+; RV32I-NEXT:    sw s0, 2024(sp)
 ; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, 2040
-; RV32I-NEXT:    add a0, sp, a0
-; RV32I-NEXT:    sw s0, 0(a0)
+; RV32I-NEXT:    addi a0, a0, 16
+; RV32I-NEXT:    sub sp, sp, a0
 ; RV32I-NEXT:    lui a0, 2
 ; RV32I-NEXT:    addi a0, a0, -2048
 ; RV32I-NEXT:    add s0, sp, a0
 ; RV32I-NEXT:    andi sp, sp, -2048
 ; RV32I-NEXT:    lui a0, 1
 ; RV32I-NEXT:    add a0, sp, a0
 ; RV32I-NEXT:    mv a0, a0
 ; RV32I-NEXT:    call callee
 ; RV32I-NEXT:    lui a0, 2
 ; RV32I-NEXT:    addi a0, a0, -2048
 ; RV32I-NEXT:    sub sp, s0, a0
 ; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, 2040
-; RV32I-NEXT:    add a0, sp, a0
-; RV32I-NEXT:    lw s0, 0(a0)
-; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, 2044
-; RV32I-NEXT:    add a0, sp, a0
-; RV32I-NEXT:    lw ra, 0(a0)
-; RV32I-NEXT:    lui a0, 2
-; RV32I-NEXT:    addi a0, a0, -2048
+; RV32I-NEXT:    addi a0, a0, 16
 ; RV32I-NEXT:    add sp, sp, a0
+; RV32I-NEXT:    lw s0, 2024(sp)
+; RV32I-NEXT:    lw ra, 2028(sp)
+; RV32I-NEXT:    addi sp, sp, 2032
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: caller2048:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    lui a0, 2
-; RV64I-NEXT:    addiw a0, a0, -2048
-; RV64I-NEXT:    sub sp, sp, a0
+; RV64I-NEXT:    addi sp, sp, -2032
+; RV64I-NEXT:    sd ra, 2024(sp)
+; RV64I-NEXT:    sd s0, 2016(sp)
 ; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, 2040
-; RV64I-NEXT:    add a0, sp, a0
-; RV64I-NEXT:    sd ra, 0(a0)
-; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, 2032
-; RV64I-NEXT:    add a0, sp, a0
-; RV64I-NEXT:    sd s0, 0(a0)
+; RV64I-NEXT:    addiw a0, a0, 16
+; RV64I-NEXT:    sub sp, sp, a0
 ; RV64I-NEXT:    lui a0, 2
 ; RV64I-NEXT:    addiw a0, a0, -2048
 ; RV64I-NEXT:    add s0, sp, a0
 ; RV64I-NEXT:    andi sp, sp, -2048
 ; RV64I-NEXT:    lui a0, 1
 ; RV64I-NEXT:    add a0, sp, a0
 ; RV64I-NEXT:    mv a0, a0
 ; RV64I-NEXT:    call callee
 ; RV64I-NEXT:    lui a0, 2
 ; RV64I-NEXT:    addiw a0, a0, -2048
 ; RV64I-NEXT:    sub sp, s0, a0
 ; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, 2032
-; RV64I-NEXT:    add a0, sp, a0
-; RV64I-NEXT:    ld s0, 0(a0)
-; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, 2040
-; RV64I-NEXT:    add a0, sp, a0
-; RV64I-NEXT:    ld ra, 0(a0)
-; RV64I-NEXT:    lui a0, 2
-; RV64I-NEXT:    addiw a0, a0, -2048
+; RV64I-NEXT:    addiw a0, a0, 16
 ; RV64I-NEXT:    add sp, sp, a0
+; RV64I-NEXT:    ld s0, 2016(sp)
+; RV64I-NEXT:    ld ra, 2024(sp)
+; RV64I-NEXT:    addi sp, sp, 2032
 ; RV64I-NEXT:    ret
   %1 = alloca i8, align 2048
   call void @callee(i8* %1)
   ret void
 }
 
 define void @caller_no_realign2048() nounwind "no-realign-stack" {
 ; RV32I-LABEL: caller_no_realign2048:
; RV64I-NEXT:    ret
   %1 = alloca i8, align 2048
   call void @callee(i8* %1)
   ret void
 }
 
 define void @caller4096() nounwind {
 ; RV32I-LABEL: caller4096:
 ; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi sp, sp, -2032
+; RV32I-NEXT:    sw ra, 2028(sp)
+; RV32I-NEXT:    sw s0, 2024(sp)
 ; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, -2032
 ; RV32I-NEXT:    sub sp, sp, a0
 ; RV32I-NEXT:    lui a0, 3
-; RV32I-NEXT:    addi a0, a0, -4
-; RV32I-NEXT:    add a0, sp, a0
-; RV32I-NEXT:    sw ra, 0(a0)
-; RV32I-NEXT:    lui a0, 3
-; RV32I-NEXT:    addi a0, a0, -8
-; RV32I-NEXT:    add a0, sp, a0
-; RV32I-NEXT:    sw s0, 0(a0)
-; RV32I-NEXT:    lui a0, 3
 ; RV32I-NEXT:    add s0, sp, a0
 ; RV32I-NEXT:    srli a0, sp, 12
 ; RV32I-NEXT:    slli sp, a0, 12
 ; RV32I-NEXT:    lui a0, 2
 ; RV32I-NEXT:    add a0, sp, a0
 ; RV32I-NEXT:    mv a0, a0
 ; RV32I-NEXT:    call callee
 ; RV32I-NEXT:    lui a0, 3
 ; RV32I-NEXT:    sub sp, s0, a0
 ; RV32I-NEXT:    lui a0, 3
-; RV32I-NEXT:    addi a0, a0, -8
-; RV32I-NEXT:    add a0, sp, a0
-; RV32I-NEXT:    lw s0, 0(a0)
-; RV32I-NEXT:    lui a0, 3
-; RV32I-NEXT:    addi a0, a0, -4
-; RV32I-NEXT:    add a0, sp, a0
-; RV32I-NEXT:    lw ra, 0(a0)
-; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, -2032
 ; RV32I-NEXT:    add sp, sp, a0
+; RV32I-NEXT:    lw s0, 2024(sp)
+; RV32I-NEXT:    lw ra, 2028(sp)
+; RV32I-NEXT:    addi sp, sp, 2032
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: caller4096:
 ; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi sp, sp, -2032
+; RV64I-NEXT:    sd ra, 2024(sp)
+; RV64I-NEXT:    sd s0, 2016(sp)
 ; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, -2032
 ; RV64I-NEXT:    sub sp, sp, a0
 ; RV64I-NEXT:    lui a0, 3
-; RV64I-NEXT:    addiw a0, a0, -8
-; RV64I-NEXT:    add a0, sp, a0
-; RV64I-NEXT:    sd ra, 0(a0)
-; RV64I-NEXT:    lui a0, 3
-; RV64I-NEXT:    addiw a0, a0, -16
-; RV64I-NEXT:    add a0, sp, a0
-; RV64I-NEXT:    sd s0, 0(a0)
-; RV64I-NEXT:    lui a0, 3
 ; RV64I-NEXT:    add s0, sp, a0
 ; RV64I-NEXT:    srli a0, sp, 12
 ; RV64I-NEXT:    slli sp, a0, 12
 ; RV64I-NEXT:    lui a0, 2
 ; RV64I-NEXT:    add a0, sp, a0
 ; RV64I-NEXT:    mv a0, a0
 ; RV64I-NEXT:    call callee
 ; RV64I-NEXT:    lui a0, 3
 ; RV64I-NEXT:    sub sp, s0, a0
 ; RV64I-NEXT:    lui a0, 3
-; RV64I-NEXT:    addiw a0, a0, -16
-; RV64I-NEXT:    add a0, sp, a0
-; RV64I-NEXT:    ld s0, 0(a0)
-; RV64I-NEXT:    lui a0, 3
-; RV64I-NEXT:    addiw a0, a0, -8
-; RV64I-NEXT:    add a0, sp, a0
-; RV64I-NEXT:    ld ra, 0(a0)
-; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, -2032
 ; RV64I-NEXT:    add sp, sp, a0
+; RV64I-NEXT:    ld s0, 2016(sp)
+; RV64I-NEXT:    ld ra, 2024(sp)
+; RV64I-NEXT:    addi sp, sp, 2032
 ; RV64I-NEXT:    ret
   %1 = alloca i8, align 4096
   call void @callee(i8* %1)
   ret void
 }
 
 define void @caller_no_realign4096() nounwind "no-realign-stack" {
 ; RV32I-LABEL: caller_no_realign4096: