[RISCV] Fix offset computation for RVV

[rogfer01]

Summary

In D97111 we changed the RVV frame layout when using sp or bp to address the stack slots so we could address the emergency stack slot. The idea is to put the RVV objects as far as possible (in offset terms) from the frame reference register (sp / fp / bp).

When using fp this happens naturally because the RVV objects are already the top of the stack and due to the constraints of RVV (VLENB being a power of two >= 128) the stack remains aligned. The rest of this summary does not apply to this case.

When using sp / bp we need to skip the non-RVV stack slots. The size of the the non-RVV objects is computed subtracting the callee saved register size (whose computation is added in D97111 itself) to the total size of the stack (which does not account for RVV stack slots). However, when doing so we round to 16 bytes when computing that size and we end emitting a smaller offset that may belong to a scalar stack slot (see D98801). So this change removes that rounding.

Also, because we want the RVV objects be between the non-RVV stack slots and the callee-saved register slots, we need to make sure the RVV objects are properly aligned to 8 bytes. Adding a padding of 8 would render the stack unaligned. So when allocating space for RVV (only when we don't use fp) we need to have extra padding that preserves the stack alignment. This way we can round to 8 bytes the offset that skips the non-RVV objects and we do not misalign the whole stack in the way. In some circumstances this means that the RVV objects may have padding before (=lower offsets from sp/bp) and after (before the CSR stack slots).

Unfortunately we are computing the size of the CSR stack slots (CalleeSavedStackSize) in processFunctionBeforeFrameIndicesReplaced which is called after the prologue has been emitted. So we might be less precise here though my expectation is that this is OK because we handle those now as having XLEN size. If this is a problem, then we can unconditionally add enough padding (getStackAlign()) to retain the alignment while allowing us to adjust the area of RVV-objects as needed.

[rogfer01]

Hum, this is not right. I just realized that we may need some extra padding between the end of the scalar part and the RVV stack slots. That padding hasn't been accounted anywhere.

[rogfer01]

ChangeLog:

• Add extra padding so we can adjust RVV objects within the stack correctly
• Round the offset to the RVV objects to 8 bytes (when not using fp).
• Move computation of CalleeSavedStackSize from processFunctionBeforeFrameIndicesReplaced to processFunctionBeforeFrameFinalized because we need that size before we emit the prologue.

[rogfer01]

I feel this might be a bit complicated to grasp from my attempt of explanation in the summary, so this is how the stack should look like for wrong-stack-slot-rv32.mir assuming VLENB=128.

[frasercrmck]

Looks reasonable to me, though I'm not the best person to judge the ins and outs of the RISC-V frame lowering. I must say, I'm loving the diagram!

Do any of our code-comment diagrams need updating to show off padding?

[StephenFan]

Hi @rogfer01 ! It is reasonable to me. But I think the instruction of BuildMI(MBB, MBBI, DL, TII->get(RISCV::ADDI), SPReg) can be eliminated. Firstly, the value of calleeSavedStackSize can be regarded as a aligned value(For example, aligned to MFI.getStackAlign()). Then we can calculate the padding size by the aligned calleeSavedStackSize minus the original calleeSavedStackSize. When emits prologue, we can minus the value of MFI.getStackSize() - original calleeSavedStackSize + aligned calleeSavedStackSize. When get the offset of the rvv object, we can MFI.getStackSize() - original calleesavedStackSize, because we just want to calculate the non-calleesaved field size. My English is poor and I am a beginner of LLVM-RISCV. So I don't know if it makes sense to you.

[rogfer01]

In D98802#2634791, @StephenFan wrote:

Hi @rogfer01 ! It is reasonable to me. But I think the instruction of BuildMI(MBB, MBBI, DL, TII->get(RISCV::ADDI), SPReg) can be eliminated. Firstly, the value of calleeSavedStackSize can be regarded as a aligned value(For example, aligned to MFI.getStackAlign()). Then we can calculate the padding size by the aligned calleeSavedStackSize minus the original calleeSavedStackSize. When emits prologue, we can minus the value of MFI.getStackSize() - original calleeSavedStackSize + aligned calleeSavedStackSize. When get the offset of the rvv object, we can MFI.getStackSize() - original calleesavedStackSize, because we just want to calculate the non-calleesaved field size.

That sounds like a great idea, I will try your suggestion. Thanks a lot @StephenFan !

[rogfer01]

ChangeLog:

• Fold the padding into the main stack size enlargement: adjust CSR offsets accordingly, as suggested by @StephenFan
• Add {get,set}RVVPadding to RISCVMachineFunctionInfo to materialise the extra padding required by RVV.

[StephenFan]

This comment was removed by StephenFan.

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

882

If we can set the RVVPadding as

RVFI->setRVVPadding(alignTo(Size, getStackAlign().value()) -Size);

or just align to 8, because we just want the RVV stack aligned to 8 bytes.

RVFI->setRVVPadding(alignTo(Size, 8) - Size);

?
In this way, maybe we can save some stack space.

882

If we can set the RVVPadding as

RVFI->setRVVPadding(alignTo(Size, getStackAlign().value()) -Size);

or just align to 8, because we just want the RVV stack aligned to 8 bytes.

RVFI->setRVVPadding(alignTo(Size, 8) - Size);

?
In this way, maybe we can save some stack space.

[rogfer01]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

882

If we can set the RVVPadding as

RVFI->setRVVPadding(alignTo(Size, getStackAlign().value()) -Size);

This is more compact, I'll use that, thanks!.

or just align to 8, because we just want the RVV stack aligned to 8 bytes.

Unless I do something different with the RVVPadding I think I need to use getStackAlign().

Consider RV32 and a single CSR: Size will be 4 so alignTo(Size, 8) - Size is 4. StackSize before adding RVVPadding is already aligned to getStackAlign() (that value is commonly 16, even in RV32). If I add 4 bytes to the StackSize then I can align the RVV to 8 bytes but sp won't be aligned to getStackAlign() anymore. Maybe I'm missing something?

[StephenFan]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

882

You're right. I didn't consider that if the StackSize adds some value that not aligned, it will be not aligned. And then sp will not be aligned.

[rogfer01]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

882

Then I think the first form RVFI->setRVVPadding(alignTo(Size, getStackAlign().value()) -Size); would not be correct either because we might set the padding to a smaller value than getStackAlign().

I'll add a clarifying comment here.

[rogfer01]

ChangeLog:

• Add comment explaining why we add the alignment of the stack as RVV padding when CSR are not aligned to 8 bytes.

[rogfer01]

I have internally validated that the last change (Diff 331637, not Diff 332358 that only adds a comment) is equivalent to the previous change (Diff 331546).

If no objections arise in the next days (and after I have addressed one comment in D98801) I'll land this.

[rogfer01]

ChangeLog:

• Rebase

llvm/lib/Target/RISCV/RISCVFrameLowering.h

public:
   uint64_t getFirstSPAdjustAmount(const MachineFunction &MF) const;
 
   bool canUseAsPrologue(const MachineBasicBlock &MBB) const override;
   bool canUseAsEpilogue(const MachineBasicBlock &MBB) const override;
 
   bool isSupportedStackID(TargetStackID::Value ID) const override;
   TargetStackID::Value getStackIDForScalableVectors() const override;
 
-  void processFunctionBeforeFrameIndicesReplaced(
-      MachineFunction &MF, RegScavenger *RS = nullptr) const override;
-
 protected:
   const RISCVSubtarget &STI;
 
 private:
   void determineFrameLayout(MachineFunction &MF) const;
   void adjustReg(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
                  const DebugLoc &DL, Register DestReg, Register SrcReg,
                  int64_t Val, MachineInstr::MIFlag Flag) const;
   void adjustStackForRVV(MachineFunction &MF, MachineBasicBlock &MBB,
                          MachineBasicBlock::iterator MBBI, const DebugLoc &DL,
                          int64_t Amount) const;
   int64_t assignRVVStackObjectOffsets(MachineFrameInfo &MFI) const;
 };
 }
 #endif

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

if (Amount < 0) {
     Opc = RISCV::SUB;
   }
 
   // 1. Multiply the number of v-slots to the length of registers
   Register FactorRegister = TII->getVLENFactoredAmount(MF, MBB, MBBI, Amount);
   // 2. SP = SP - RVV stack size
   BuildMI(MBB, MBBI, DL, TII->get(Opc), SPReg)
       .addReg(SPReg)
-      .addReg(FactorRegister);
+      .addReg(FactorRegister, RegState::Kill);
 }
 
 void RISCVFrameLowering::emitPrologue(MachineFunction &MF,
                                       MachineBasicBlock &MBB) const {
   MachineFrameInfo &MFI = MF.getFrameInfo();
   auto *RVFI = MF.getInfo<RISCVMachineFunctionInfo>();
   const RISCVRegisterInfo *RI = STI.getRegisterInfo();
   const RISCVInstrInfo *TII = STI.getInstrInfo();
if (int LibCallRegs = getLibCallID(MF, MFI.getCalleeSavedInfo()) + 1) {
     // Calculate the size of the frame managed by the libcall. The libcalls are
     // implemented such that the stack will always be 16 byte aligned.
     unsigned LibCallFrameSize = alignTo((STI.getXLen() / 8) * LibCallRegs, 16);
     RVFI->setLibCallStackSize(LibCallFrameSize);
   }
 
   // FIXME (note copied from Lanai): This appears to be overallocating.  Needs
   // investigation. Get the number of bytes to allocate from the FrameInfo.
-  uint64_t StackSize = MFI.getStackSize();
+  uint64_t StackSize = MFI.getStackSize() + RVFI->getRVVPadding();
   uint64_t RealStackSize = StackSize + RVFI->getLibCallStackSize();
   uint64_t RVVStackSize = RVFI->getRVVStackSize();
 
   // Early exit if there is no need to allocate on the stack
   if (RealStackSize == 0 && !MFI.adjustsStack() && RVVStackSize == 0)
     return;
 
   // If the stack pointer has been marked as reserved, then produce an error if
void RISCVFrameLowering::emitEpilogue(MachineFunction &MF,
 
   // Skip to before the restores of callee-saved registers
   // FIXME: assumes exactly one instruction is used to restore each
   // callee-saved register.
   auto LastFrameDestroy = MBBI;
   if (!CSI.empty())
     LastFrameDestroy = std::prev(MBBI, CSI.size());
 
-  uint64_t StackSize = MFI.getStackSize();
+  uint64_t StackSize = MFI.getStackSize() + RVFI->getRVVPadding();
   uint64_t RealStackSize = StackSize + RVFI->getLibCallStackSize();
   uint64_t FPOffset = RealStackSize - RVFI->getVarArgsSaveSize();
   uint64_t RVVStackSize = RVFI->getRVVStackSize();
 
   // Restore the stack pointer using the value of the frame pointer. Only
   // necessary if the stack pointer was modified, meaning the stack size is
   // unknown.
   if (RI->needsStackRealignment(MF) || MFI.hasVarSizedObjects()) {
RISCVFrameLowering::getFrameIndexReference(const MachineFunction &MF, int FI,
   }
 
   if (FI >= MinCSFI && FI <= MaxCSFI) {
     FrameReg = RISCV::X2;
 
     if (FirstSPAdjustAmount)
       Offset += StackOffset::getFixed(FirstSPAdjustAmount);
     else
-      Offset += StackOffset::getFixed(MFI.getStackSize());
+      Offset +=
+          StackOffset::getFixed(MFI.getStackSize() + RVFI->getRVVPadding());
   } else if (RI->needsStackRealignment(MF) && !MFI.isFixedObjectIndex(FI)) {
     // If the stack was realigned, the frame pointer is set in order to allow
     // SP to be restored, so we need another base register to record the stack
     // after realignment.
     if (hasBP(MF)) {
       FrameReg = RISCVABI::getBPReg();
       // |--------------------------| -- <-- FP
-      // | callee-saved registers   | | <---------.
-      // |--------------------------| --          |
-      // | realignment (the size of | |           |
-      // | this area is not counted | |           |
-      // | in MFI.getStackSize())   | |           |
-      // |--------------------------| --          |-- MFI.getStackSize()
-      // | RVV objects              | |           |
-      // |--------------------------| --          |
-      // | scalar local variables   | | <---------'
+      // | callee-saved registers   | | <----.
+      // |--------------------------| --     |
+      // | realignment (the size of | |      |
+      // | this area is not counted | |      |
+      // | in MFI.getStackSize())   | |      |
+      // |--------------------------| --     |
+      // | Padding after RVV        | |      |
+      // | (not counted in          | |      |
+      // | MFI.getStackSize()       | |      |
+      // |--------------------------| --     |-- MFI.getStackSize()
+      // | RVV objects              | |      |
+      // | (not counted in          | |      |
+      // | MFI.getStackSize()       | |      |
+      // |--------------------------| --     |
+      // | Padding before RVV       | |      |
+      // | (not counted in          | |      |
+      // | MFI.getStackSize()       | |      |
+      // |--------------------------| --     |
+      // | scalar local variables   | | <----'
       // |--------------------------| -- <-- BP
       // | VarSize objects          | |
       // |--------------------------| -- <-- SP
     } else {
       FrameReg = RISCV::X2;
       // |--------------------------| -- <-- FP
-      // | callee-saved registers   | | <---------.
-      // |--------------------------| --          |
-      // | realignment (the size of | |           |
-      // | this area is not counted | |           |
-      // | in MFI.getStackSize())   | |           |
-      // |--------------------------| --          |-- MFI.getStackSize()
-      // | RVV objects              | |           |
-      // |--------------------------| --          |
-      // | scalar local variables   | | <---------'
+      // | callee-saved registers   | | <----.
+      // |--------------------------| --     |
+      // | realignment (the size of | |      |
+      // | this area is not counted | |      |
+      // | in MFI.getStackSize())   | |      |
+      // |--------------------------| --     |
+      // | Padding after RVV        | |      |
+      // | (not counted in          | |      |
+      // | MFI.getStackSize()       | |      |
+      // |--------------------------| --     |-- MFI.getStackSize()
+      // | RVV objects              | |      |
+      // | (not counted in          | |      |
+      // | MFI.getStackSize()       | |      |
+      // |--------------------------| --     |
+      // | Padding before RVV       | |      |
+      // | (not counted in          | |      |
+      // | MFI.getStackSize()       | |      |
+      // |--------------------------| --     |
+      // | scalar local variables   | | <----'
       // |--------------------------| -- <-- SP
     }
+    // The total amount of padding surrounding RVV objects is described by
+    // RVV->getRVVPadding() and it can be zero. It allows us to align the RVV
+    // objects to 8 bytes.
     if (MFI.getStackID(FI) == TargetStackID::Default) {
       Offset += StackOffset::getFixed(MFI.getStackSize());
       if (FI < 0)
         Offset += StackOffset::getFixed(RVFI->getLibCallStackSize());
     } else if (MFI.getStackID(FI) == TargetStackID::ScalableVector) {
-      Offset +=
-          StackOffset::get(MFI.getStackSize() - RVFI->getCalleeSavedStackSize(),
-                           RVFI->getRVVStackSize());
+      Offset += StackOffset::get(
+          alignTo(MFI.getStackSize() - RVFI->getCalleeSavedStackSize(), 8),
     }
   } else {
     FrameReg = RI->getFrameRegister(MF);
     if (hasFP(MF)) {
       Offset += StackOffset::getFixed(RVFI->getVarArgsSaveSize());
       if (FI >= 0)
         Offset -= StackOffset::getFixed(RVFI->getLibCallStackSize());
       // When using FP to access scalable vector objects, we need to minus
if (hasFP(MF)) {
       // | VarSize objects          |
       // |--------------------------| <-- SP
       if (MFI.getStackID(FI) == TargetStackID::ScalableVector)
         Offset -= StackOffset::getFixed(MFI.getStackSize());
     } else {
       // When using SP to access frame objects, we need to add RVV stack size.
       //
       // |--------------------------| -- <-- FP
-      // | callee-saved registers   | |<--------.
-      // |--------------------------| --        |
-      // | RVV objects              | |         |-- MFI.getStackSize()
-      // |--------------------------| --        |
-      // | scalar local variables   | |<--------'
+      // | callee-saved registers   | | <----.
+      // |--------------------------| --     |
+      // | Padding after RVV        | |      |
+      // | (not counted in          | |      |
+      // | MFI.getStackSize()       | |      |
+      // |--------------------------| --     |
+      // | RVV objects              | |      |-- MFI.getStackSize()
+      // | (not counted in          | |      |
+      // | MFI.getStackSize()       | |      |
+      // |--------------------------| --     |
+      // | Padding before RVV       | |      |
+      // | (not counted in          | |      |
+      // | MFI.getStackSize()       | |      |
+      // |--------------------------| --     |
+      // | scalar local variables   | | <----'
       // |--------------------------| -- <-- SP
+      //
+      // The total amount of padding surrounding RVV objects is described by
+      // RVV->getRVVPadding() and it can be zero. It allows us to align the RVV
+      // objects to 8 bytes.
       if (MFI.getStackID(FI) == TargetStackID::Default) {
         Offset += StackOffset::getFixed(MFI.getStackSize());
         if (FI < 0)
           Offset += StackOffset::getFixed(RVFI->getLibCallStackSize());
       } else if (MFI.getStackID(FI) == TargetStackID::ScalableVector) {
-        Offset += StackOffset::get(MFI.getStackSize() -
-                                       RVFI->getCalleeSavedStackSize(),
-                                   RVFI->getRVVStackSize());
+        Offset += StackOffset::get(
+            alignTo(MFI.getStackSize() - RVFI->getCalleeSavedStackSize(), 8),
       }
     }
   }
 
   return Offset;
 }
 
 void RISCVFrameLowering::determineCalleeSaves(MachineFunction &MF,
void RISCVFrameLowering::processFunctionBeforeFrameFinalized(
   // FIXME: It may be possible to craft a function with a small stack that
   // still needs an emergency spill slot for branch relaxation. This case
   // would currently be missed.
   if (!isInt<11>(MFI.estimateStackSize(MF)) || RVVStackSize != 0) {
     int RegScavFI = MFI.CreateStackObject(RegInfo->getSpillSize(*RC),
                                           RegInfo->getSpillAlign(*RC), false);
     RS->addScavengingFrameIndex(RegScavFI);
   }
-}
 
-void RISCVFrameLowering::processFunctionBeforeFrameIndicesReplaced(
-    MachineFunction &MF, RegScavenger *RS) const {
-  auto *RVFI = MF.getInfo<RISCVMachineFunctionInfo>();
-  const MachineFrameInfo &MFI = MF.getFrameInfo();
   if (MFI.getCalleeSavedInfo().empty() || RVFI->useSaveRestoreLibCalls(MF)) {
     RVFI->setCalleeSavedStackSize(0);
     return;
   }
 
-  int64_t MinOffset = std::numeric_limits<int64_t>::max();
-  int64_t MaxOffset = std::numeric_limits<int64_t>::min();
+  unsigned Size = 0;
   for (const auto &Info : MFI.getCalleeSavedInfo()) {
     int FrameIdx = Info.getFrameIdx();
     if (MFI.getStackID(FrameIdx) != TargetStackID::Default)
       continue;
 
-    int64_t Offset = MFI.getObjectOffset(FrameIdx);
-    int64_t ObjSize = MFI.getObjectSize(FrameIdx);
-    MinOffset = std::min<int64_t>(Offset, MinOffset);
-    MaxOffset = std::max<int64_t>(Offset + ObjSize, MaxOffset);
+    Size += MFI.getObjectSize(FrameIdx);
   }
-
-  unsigned Size = alignTo(MaxOffset - MinOffset, 16);
   RVFI->setCalleeSavedStackSize(Size);
+
+  // Padding required to keep the RVV stack aligned to 8 bytes
+  // within the main stack. We only need this when not using FP.
+  if (RVVStackSize && !hasFP(MF) && Size % 8 != 0) {
+    // Because we add the padding to the size of the stack, adding
+    // getStackAlign() will keep it aligned.
+    RVFI->setRVVPadding(getStackAlign().value());
+  }
 }
 
 // Not preserve stack space within prologue for outgoing variables when the
 // function contains variable size objects and let eliminateCallFramePseudoInstr
 // preserve stack space for it.
 bool RISCVFrameLowering::hasReservedCallFrame(const MachineFunction &MF) const {
   return !MF.getFrameInfo().hasVarSizedObjects();
 }

llvm/lib/Target/RISCV/RISCVMachineFunctionInfo.h

private:
   int VarArgsSaveSize = 0;
   /// FrameIndex used for transferring values between 64-bit FPRs and a pair
   /// of 32-bit GPRs via the stack.
   int MoveF64FrameIndex = -1;
   /// Size of any opaque stack adjustment due to save/restore libcalls.
   unsigned LibCallStackSize = 0;
   /// Size of RVV stack.
   uint64_t RVVStackSize = 0;
+  /// Padding required to keep RVV stack aligned within the main stack.
+  uint64_t RVVPadding = 0;
   /// Size of stack frame to save callee saved registers
   unsigned CalleeSavedStackSize = 0;
 
 public:
   RISCVMachineFunctionInfo(const MachineFunction &MF) {}
 
   int getVarArgsFrameIndex() const { return VarArgsFrameIndex; }
   void setVarArgsFrameIndex(int Index) { VarArgsFrameIndex = Index; }
bool useSaveRestoreLibCalls(const MachineFunction &MF) const {
     // function uses a varargs save area.
     return MF.getSubtarget<RISCVSubtarget>().enableSaveRestore() &&
            VarArgsSaveSize == 0 && !MF.getFrameInfo().hasTailCall();
   }
 
   uint64_t getRVVStackSize() const { return RVVStackSize; }
   void setRVVStackSize(uint64_t Size) { RVVStackSize = Size; }
 
+  uint64_t getRVVPadding() const { return RVVPadding; }
+  void setRVVPadding(uint64_t Padding) { RVVPadding = Padding; }
+
   unsigned getCalleeSavedStackSize() const { return CalleeSavedStackSize; }
   void setCalleeSavedStackSize(unsigned Size) { CalleeSavedStackSize = Size; }
 };
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_RISCV_RISCVMACHINEFUNCTIONINFO_H

llvm/test/CodeGen/RISCV/rvv/localvar.ll

 ; RV64IV-NEXT:    addi a0, a0, 15
 ; RV64IV-NEXT:    andi a0, a0, -16
 ; RV64IV-NEXT:    sub a0, sp, a0
 ; RV64IV-NEXT:    mv sp, a0
 ; RV64IV-NEXT:    addi a1, s1, 128
 ; RV64IV-NEXT:    csrr a2, vlenb
 ; RV64IV-NEXT:    slli a2, a2, 1
 ; RV64IV-NEXT:    add a2, s1, a2
-; RV64IV-NEXT:    addi a2, a2, 224
+; RV64IV-NEXT:    addi a2, a2, 232
 ; RV64IV-NEXT:    call notdead2@plt
 ; RV64IV-NEXT:    lw a0, 124(s1)
 ; RV64IV-NEXT:    csrr a0, vlenb
 ; RV64IV-NEXT:    slli a0, a0, 1
 ; RV64IV-NEXT:    add a0, s1, a0
-; RV64IV-NEXT:    addi a0, a0, 224
+; RV64IV-NEXT:    addi a0, a0, 232
 ; RV64IV-NEXT:    vl2r.v v26, (a0)
-; RV64IV-NEXT:    addi a0, s1, 224
+; RV64IV-NEXT:    addi a0, s1, 232
 ; RV64IV-NEXT:    vl2r.v v26, (a0)
 ; RV64IV-NEXT:    lw a0, 120(s1)
 ; RV64IV-NEXT:    addi sp, s0, -256
 ; RV64IV-NEXT:    ld s1, 232(sp) # 8-byte Folded Reload
 ; RV64IV-NEXT:    ld s0, 240(sp) # 8-byte Folded Reload
 ; RV64IV-NEXT:    ld ra, 248(sp) # 8-byte Folded Reload
 ; RV64IV-NEXT:    addi sp, sp, 256
 ; RV64IV-NEXT:    ret

llvm/test/CodeGen/RISCV/rvv/rv32-spill-vector-csr.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-v,+d -O0 < %s \
 ; RUN:    | FileCheck --check-prefix=SPILL-O0 %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-v,+d -O2 < %s \
 ; RUN:    | FileCheck --check-prefix=SPILL-O2 %s
 
 @.str = private unnamed_addr constant [6 x i8] c"hello\00", align 1
 
 define <vscale x 1 x double> @foo(<vscale x 1 x double> %a, <vscale x 1 x double> %b, <vscale x 1 x double> %c, i32 %gvl) nounwind
 ; SPILL-O0-LABEL: foo:
 ; SPILL-O0:       # %bb.0:
-; SPILL-O0-NEXT:    addi sp, sp, -16
-; SPILL-O0-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; SPILL-O0-NEXT:    addi sp, sp, -32
+; SPILL-O0-NEXT:    sw ra, 28(sp) # 4-byte Folded Spill
 ; SPILL-O0-NEXT:    csrr a1, vlenb
 ; SPILL-O0-NEXT:    slli a1, a1, 1
 ; SPILL-O0-NEXT:    sub sp, sp, a1
 ; SPILL-O0-NEXT:    sw a0, 8(sp) # 4-byte Folded Spill
 ; SPILL-O0-NEXT:    csrr a1, vlenb
 ; SPILL-O0-NEXT:    add a1, sp, a1
+; SPILL-O0-NEXT:    addi a1, a1, 16
 ; SPILL-O0-NEXT:    vs1r.v v8, (a1) # Unknown-size Folded Spill
 ; SPILL-O0-NEXT:    vsetvli a0, a0, e64,m1,ta,mu
 ; SPILL-O0-NEXT:    vfadd.vv v25, v8, v9
-; SPILL-O0-NEXT:    vs1r.v v25, (sp) # Unknown-size Folded Spill
+; SPILL-O0-NEXT:    addi a0, sp, 16
+; SPILL-O0-NEXT:    vs1r.v v25, (a0) # Unknown-size Folded Spill
 ; SPILL-O0-NEXT:    lui a0, %hi(.L.str)
 ; SPILL-O0-NEXT:    addi a0, a0, %lo(.L.str)
 ; SPILL-O0-NEXT:    call puts@plt
-; SPILL-O0-NEXT:    vl1r.v v25, (sp) # Unknown-size Folded Reload
+; SPILL-O0-NEXT:    addi a1, sp, 16
+; SPILL-O0-NEXT:    vl1r.v v25, (a1) # Unknown-size Folded Reload
 ; SPILL-O0-NEXT:    csrr a1, vlenb
 ; SPILL-O0-NEXT:    add a1, sp, a1
+; SPILL-O0-NEXT:    addi a1, a1, 16
 ; SPILL-O0-NEXT:    vl1r.v v8, (a1) # Unknown-size Folded Reload
 ; SPILL-O0-NEXT:    # kill: def $x11 killed $x10
-; SPILL-O0-NEXT:    lw a0, 8(sp)  # 4-byte Folded Reload
+; SPILL-O0-NEXT:    lw a0, 8(sp) # 4-byte Folded Reload
 ; SPILL-O0-NEXT:    vsetvli a0, a0, e64,m1,ta,mu
 ; SPILL-O0-NEXT:    vfadd.vv v8, v8, v25
 ; SPILL-O0-NEXT:    csrr a0, vlenb
 ; SPILL-O0-NEXT:    slli a0, a0, 1
 ; SPILL-O0-NEXT:    add sp, sp, a0
-; SPILL-O0-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
-; SPILL-O0-NEXT:    addi sp, sp, 16
+; SPILL-O0-NEXT:    lw ra, 28(sp) # 4-byte Folded Reload
+; SPILL-O0-NEXT:    addi sp, sp, 32
 ; SPILL-O0-NEXT:    ret
 ;
 ; SPILL-O2-LABEL: foo:
 ; SPILL-O2:       # %bb.0:
 ; SPILL-O2-NEXT:    addi sp, sp, -16
 ; SPILL-O2-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
-; SPILL-O2-NEXT:    sw s0, 8(sp)  # 4-byte Folded Spill
+; SPILL-O2-NEXT:    sw s0, 8(sp) # 4-byte Folded Spill
 ; SPILL-O2-NEXT:    csrr a1, vlenb
 ; SPILL-O2-NEXT:    slli a1, a1, 1
 ; SPILL-O2-NEXT:    sub sp, sp, a1
 ; SPILL-O2-NEXT:    mv s0, a0
-; SPILL-O2-NEXT:    vs1r.v v8, (sp) # Unknown-size Folded Spill
+; SPILL-O2-NEXT:    addi a1, sp, 8
+; SPILL-O2-NEXT:    vs1r.v v8, (a1) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    vsetvli a0, a0, e64,m1,ta,mu
 ; SPILL-O2-NEXT:    vfadd.vv v25, v8, v9
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    add a0, sp, a0
+; SPILL-O2-NEXT:    addi a0, a0, 8
 ; SPILL-O2-NEXT:    vs1r.v v25, (a0) # Unknown-size Folded Spill
 ; SPILL-O2-NEXT:    lui a0, %hi(.L.str)
 ; SPILL-O2-NEXT:    addi a0, a0, %lo(.L.str)
 ; SPILL-O2-NEXT:    call puts@plt
 ; SPILL-O2-NEXT:    vsetvli a0, s0, e64,m1,ta,mu
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    add a0, sp, a0
+; SPILL-O2-NEXT:    addi a0, a0, 8
 ; SPILL-O2-NEXT:    vl1r.v v25, (a0) # Unknown-size Folded Reload
-; SPILL-O2-NEXT:    vl1r.v v26, (sp) # Unknown-size Folded Reload
+; SPILL-O2-NEXT:    addi a0, sp, 8
+; SPILL-O2-NEXT:    vl1r.v v26, (a0) # Unknown-size Folded Reload
 ; SPILL-O2-NEXT:    vfadd.vv v8, v26, v25
 ; SPILL-O2-NEXT:    csrr a0, vlenb
 ; SPILL-O2-NEXT:    slli a0, a0, 1
 ; SPILL-O2-NEXT:    add sp, sp, a0
 ; SPILL-O2-NEXT:    lw s0, 8(sp) # 4-byte Folded Reload
 ; SPILL-O2-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; SPILL-O2-NEXT:    addi sp, sp, 16
 ; SPILL-O2-NEXT:    ret
 {
    %x = call <vscale x 1 x double> @llvm.riscv.vfadd.nxv1f64.nxv1f64(<vscale x 1 x double> %a, <vscale x 1 x double> %b, i32 %gvl)
    %call = call signext i32 @puts(i8* getelementptr inbounds ([6 x i8], [6 x i8]* @.str, i64 0, i64 0))
    %z = call <vscale x 1 x double> @llvm.riscv.vfadd.nxv1f64.nxv1f64(<vscale x 1 x double> %a, <vscale x 1 x double> %x, i32 %gvl)
    ret <vscale x 1 x double> %z
 }
 
 declare <vscale x 1 x double> @llvm.riscv.vfadd.nxv1f64.nxv1f64(<vscale x 1 x double> %a, <vscale x 1 x double> %b, i32 %gvl)
-declare i32 @puts(i8*);
+declare i32 @puts(i8*);
 

llvm/test/CodeGen/RISCV/rvv/rv64-spill-vector-csr.ll

 ; SPILL-O0-NEXT:    addi sp, sp, -32
 ; SPILL-O0-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
 ; SPILL-O0-NEXT:    csrr a1, vlenb
 ; SPILL-O0-NEXT:    slli a1, a1, 1
 ; SPILL-O0-NEXT:    sub sp, sp, a1
 ; SPILL-O0-NEXT:    sd a0, 16(sp) # 8-byte Folded Spill
 ; SPILL-O0-NEXT:    csrr a1, vlenb
 ; SPILL-O0-NEXT:    add a1, sp, a1
-; SPILL-O0-NEXT:    addi a1, a1, 16
+; SPILL-O0-NEXT:    addi a1, a1, 24
 ; SPILL-O0-NEXT:    vs1r.v v8, (a1) # Unknown-size Folded Spill
 ; SPILL-O0-NEXT:    vsetvli a0, a0, e64,m1,ta,mu
 ; SPILL-O0-NEXT:    vfadd.vv v25, v8, v9
-; SPILL-O0-NEXT:    addi a0, sp, 16
+; SPILL-O0-NEXT:    addi a0, sp, 24
 ; SPILL-O0-NEXT:    vs1r.v v25, (a0) # Unknown-size Folded Spill
 ; SPILL-O0-NEXT:    lui a0, %hi(.L.str)
 ; SPILL-O0-NEXT:    addi a0, a0, %lo(.L.str)
 ; SPILL-O0-NEXT:    call puts@plt
-; SPILL-O0-NEXT:    addi a1, sp, 16
+; SPILL-O0-NEXT:    addi a1, sp, 24
 ; SPILL-O0-NEXT:    vl1r.v v25, (a1) # Unknown-size Folded Reload
 ; SPILL-O0-NEXT:    csrr a1, vlenb
 ; SPILL-O0-NEXT:    add a1, sp, a1
-; SPILL-O0-NEXT:    addi a1, a1, 16
+; SPILL-O0-NEXT:    addi a1, a1, 24
 ; SPILL-O0-NEXT:    vl1r.v v8, (a1) # Unknown-size Folded Reload
-; SPILL-O0-NEXT:                     # kill: def $x11 killed $x10
+; SPILL-O0-NEXT:    # kill: def $x11 killed $x10
 ; SPILL-O0-NEXT:    ld a0, 16(sp) # 8-byte Folded Reload
 ; SPILL-O0-NEXT:    vsetvli a0, a0, e64,m1,ta,mu
 ; SPILL-O0-NEXT:    vfadd.vv v8, v8, v25
 ; SPILL-O0-NEXT:    csrr a0, vlenb
 ; SPILL-O0-NEXT:    slli a0, a0, 1
 ; SPILL-O0-NEXT:    add sp, sp, a0
 ; SPILL-O0-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
 ; SPILL-O0-NEXT:    addi sp, sp, 32

llvm/test/CodeGen/RISCV/rvv/rvv-framelayout.ll

 ; CHECK-NEXT:    slli a0, a0, 2
 ; CHECK-NEXT:    addi a0, a0, 15
 ; CHECK-NEXT:    andi a0, a0, -16
 ; CHECK-NEXT:    sub a0, sp, a0
 ; CHECK-NEXT:    mv sp, a0
 ; CHECK-NEXT:    csrr a2, vlenb
 ; CHECK-NEXT:    slli a2, a2, 1
 ; CHECK-NEXT:    add a2, s1, a2
-; CHECK-NEXT:    addi a2, a2, 96
+; CHECK-NEXT:    addi a2, a2, 104
 ; CHECK-NEXT:    vl1re64.v v25, (a2)
-; CHECK-NEXT:    addi a2, s1, 96
+; CHECK-NEXT:    addi a2, s1, 104
 ; CHECK-NEXT:    vl2re64.v v26, (a2)
 ; CHECK-NEXT:    lw a2, 64(s1)
 ; CHECK-NEXT:    slli a1, a1, 2
 ; CHECK-NEXT:    add a0, a0, a1
 ; CHECK-NEXT:    lw a0, 0(a0)
 ; CHECK-NEXT:    addi sp, s0, -128
 ; CHECK-NEXT:    ld s1, 104(sp) # 8-byte Folded Reload
 ; CHECK-NEXT:    ld s0, 112(sp) # 8-byte Folded Reload

llvm/test/CodeGen/RISCV/rvv/wrong-stack-slot-rv32.mir

 # NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 # RUN: llc -mtriple=riscv32 -mattr=+m,+experimental-v  -o - %s \
 # RUN:   -start-before=prologepilog | FileCheck %s
 #
 # This test checks that we are assigning the right stack slot to GPRs and to
 # vector registers (VRs). If this test changes, make sure there is no overlap
 # between slots for GPRs and VRs.
 --- |
   define void @foo() #0 {
   ; CHECK-LABEL: foo:
   ; CHECK:       # %bb.0: # %entry
-  ; CHECK-NEXT:    addi sp, sp, -16
-  ; CHECK-NEXT:    sw s9, 12(sp) # 4-byte Folded Spill
+  ; CHECK-NEXT:    addi sp, sp, -32
+  ; CHECK-NEXT:    sw s9, 28(sp) # 4-byte Folded Spill
   ; CHECK-NEXT:    csrr a1, vlenb
   ; CHECK-NEXT:    slli a1, a1, 1
   ; CHECK-NEXT:    sub sp, sp, a1
   ; CHECK-NEXT:    sw a0, 8(sp) # 4-byte Folded Spill
-  ; CHECK-NEXT:    vs2r.v v30, (sp) # Unknown-size Folded Spill
+  ; CHECK-NEXT:    addi a0, sp, 16
+  ; CHECK-NEXT:    vs2r.v v30, (a0) # Unknown-size Folded Spill
   ; CHECK-NEXT:    csrr a0, vlenb
   ; CHECK-NEXT:    slli a0, a0, 1
   ; CHECK-NEXT:    add sp, sp, a0
-  ; CHECK-NEXT:    lw s9, 12(sp) # 4-byte Folded Reload
-  ; CHECK-NEXT:    addi sp, sp, 16
+  ; CHECK-NEXT:    lw s9, 28(sp) # 4-byte Folded Reload
+  ; CHECK-NEXT:    addi sp, sp, 32
   ; CHECK-NEXT:    ret
   entry:
     ret void
   }
 
   attributes #0 = { nounwind }
 ...
 ---

llvm/test/CodeGen/RISCV/rvv/wrong-stack-slot-rv64.mir

--- |
   ; CHECK-LABEL: foo:
   ; CHECK:       # %bb.0: # %entry
   ; CHECK-NEXT:    addi sp, sp, -32
   ; CHECK-NEXT:    sd s9, 24(sp) # 8-byte Folded Spill
   ; CHECK-NEXT:    csrr a1, vlenb
   ; CHECK-NEXT:    slli a1, a1, 1
   ; CHECK-NEXT:    sub sp, sp, a1
   ; CHECK-NEXT:    sd a0, 16(sp) # 8-byte Folded Spill
-  ; CHECK-NEXT:    addi a0, sp, 16
+  ; CHECK-NEXT:    addi a0, sp, 24
   ; CHECK-NEXT:    vs2r.v v30, (a0) # Unknown-size Folded Spill
   ; CHECK-NEXT:    csrr a0, vlenb
   ; CHECK-NEXT:    slli a0, a0, 1
   ; CHECK-NEXT:    add sp, sp, a0
   ; CHECK-NEXT:    ld s9, 24(sp) # 8-byte Folded Reload
   ; CHECK-NEXT:    addi sp, sp, 32
   ; CHECK-NEXT:    ret
   entry: