[RISCV] Frame handling for RISC-V V extension.

[HsiangKai]

Summary

This patch proposes how to deal with RISC-V vector frame objects. The layout of RISC-V vector frame will look like

|---------------------------------|
| RVV incoming arguments          |
|---------------------------------| previous frame
| scalar incoming arguments       |
|---------------------------------| <- start of frame (fp)
| scalar callee-saved registers   |
|---------------------------------|
| scalar local variables          |
|---------------------------------|
| addresses of RVV objects (*)    | current frame
|---------------------------------|
| RVV local variables             |
|---------------------------------|
| RVV outgoing arguments          |
|---------------------------------|
| scalar outgoing arguments       |
|---------------------------------| <- end of frame (sp)

We will reserve stack space for RISC-V vector frame objects and store their base addresses in “addresses of RVV objects” area. The benefits of this way is we have no need to calculate the addresses of RVV objects repeatedly. There is no easy way to get the addresses of the RISC-V vector objects in V instructions. We need to read out VLENB and multiple VLENB with LMUL to know the size of vector objects. Then we need to subtract or add the size from the stack pointer. If there are multiple vector objects in the frame, we need to subtract or add the accumulated size from the stack pointer. If we store the base addresses in the frame first, we only need to load the base address from the stack frame and access the vector object directly.

Spilling for RISC-V V extension will be another patch.

Authored-by: Roger Ferrer Ibanez <[e-mail]>
Co-Authored-by: Hsiangkai Wang <[e-mail]>

[jrtc27]

llvm/include/llvm/CodeGen/MachineFrameInfo.h

226

How come SVE doesn't need this already?

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

280

Hm, looking at the stack-realignment.ll diff, were we double-counting before? If so it'd be worth splitting out that bit of this revision into its own smaller one.

496

:( is there progress being made to specify it?

llvm/lib/Target/RISCV/RISCVInstrInfoVPseudos.td

249

Can you not just do isa<> to check it's the right class and then look at mayLoad/mayStore?

llvm/test/CodeGen/RISCV/rvv/allocate-lmul-2-4-8.ll

5

Only nounwind is needed for these I believe?

[HsiangKai]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

280

I think so. I will create another patch for it.

[StephenFan]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

547

why if the machine function has the scalable vector objects, then the prolog doesn't need to realign the stack address ?

[HsiangKai]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

496

After rethinking it, there is no callee-saved vector registers in the current specification. I think we could remove the if statement.

Reference: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc

[HsiangKai]

llvm/include/llvm/CodeGen/MachineFrameInfo.h

226

I didn't dig into how SVE implements it. I will do some study.

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

547

I need to think about how to deal with it when there are objects needed realignment and RVV objects on the stack at the same time.

llvm/lib/Target/RISCV/RISCVInstrInfoVPseudos.td

249

Good idea! Thanks for your tips.

[StephenFan]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

547

we have an internal implementation that if needs realignment and RVV objects on the stack at the same time. The BP register needs to be used as base address of stack object. So modify the hasBP function in RISCVFrameLowering.cpp may be a good choice. However, maybe there is a better solution.

[StephenFan]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

829

Is This for loop unuseful? because in the FunctionLoweringinfo.cpp has a set function that will setstackid

if (isa<ScalableVectorType>(Ty))
  MF->getFrameInfo().setStackID(FrameIndex,
                                TFI->getStackIDForScalableVectors());

[HsiangKai]

Deal with the case when there are realignment objects and RVV objects on the stack at the same time.

[HsiangKai]

llvm/include/llvm/CodeGen/MachineFrameInfo.h

226

The SVE spill/fill instructions have VL-scaled addressing modes such as:
ldr z8, [fp, #-7 mul vl]

In SVE, there is no need to do address calculation to get the location of SVE frame objects. The number of SVE objects is encoded in the instruction.

In RVV, we have no VL-scaled addressing mode. We need to read out VLENB, do multiplication or shift to know the size of RVV objects, and add/subtract the accumulated size from FP/SP. To avoid the repeated address calculation, we have two areas for RVV objects in the frame. One is for RVV object addresses, and another is for RVV objects themselves. SP will be updated by the size of RVV object region if there are any RVV objects. We still could use SP to access frame objects, but it will need to add multiple of the run-time VLENB value. To ease the complexity, we use FP to access frame objects if there is any RVV object on the stack. That’s why I define hasScalableVectorObjects() to know about it.

[HsiangKai]

Ping.

[jrtc27]

llvm/include/llvm/CodeGen/MachineFrameInfo.h

226

Isn't that basically just a special case of DYNAMIC_STACKALLOC? Can we not reuse some of that logic with BP?

[craig.topper]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

632

unsigned -> Register.

Use RISCV::NoRegister instead of 0

633

BinSizes.data() might be a little better than "&BinSizes[0]"

llvm/lib/Target/RISCV/RISCVMCInstLower.cpp

204

Can we use RISCVSysReg::lookupSysRegByName("VLENB")->Encoding here instead of adding a new enum that repeats the addresses?

llvm/lib/Target/RISCV/RISCVRegisterInfo.cpp

173

Are we just using RVV as a boolean and not a pointer? Should we do something like?

bool IsRVV = RISCVVPseudosTable::getPseudoInfo(MI.getOpcode()) != nullptr;

192

Add 'const' as the pre-merge check suggests.

[jrtc27]

llvm/lib/Target/RISCV/Utils/RISCVBaseInfo.h

438–443

Only VLENB is used, and as Craig says we use lookupSysRegByName for CYCLE[H] already. If we wanted to avoid that we should modify TableGen, not duplicate things here.

[jrtc27]

One thing I don't understand is why RVV incoming/outgoing arguments need their own stack slots. I assume they're just passed indirectly, so why do we need to distinguish between arguments and locals rather than just treat them like anything else that ends up being passed indirectly?

[HsiangKai]

Address @craig.topper and @jrtc27's comments.

[HsiangKai]

llvm/include/llvm/CodeGen/MachineFrameInfo.h

226

It is not a special case of DYNAMIC_STACKALLOC. The RVV objects are statically allocated. The path is different in SelectionDAGBuilder::visitAlloca().

[HsiangKai]

In D93750#2478380, @jrtc27 wrote:

One thing I don't understand is why RVV incoming/outgoing arguments need their own stack slots. I assume they're just passed indirectly, so why do we need to distinguish between arguments and locals rather than just treat them like anything else that ends up being passed indirectly?

It seems the existing logic in https://github.com/llvm/llvm-project/blob/4e0e79dd349a208384449fd8dcdc9bf1644ee0f3/llvm/lib/Target/RISCV/RISCVISelLowering.cpp#L3154.

In current RISC-V implementation, if the argument is passed indirectly, it will create a temporary frame object and pass the address of the temporary frame object.

[HsiangKai]

Let's focus on D94465. If there is anyone oppose it, I could reactivate this commit.

llvm/include/llvm/CodeGen/MIRYamlMapping.h

 };
 
 template <>
 struct ScalarEnumerationTraits<TargetStackID::Value> {
   static void enumeration(yaml::IO &IO, TargetStackID::Value &ID) {
     IO.enumCase(ID, "default", TargetStackID::Default);
     IO.enumCase(ID, "sgpr-spill", TargetStackID::SGPRSpill);
     IO.enumCase(ID, "sve-vec", TargetStackID::SVEVector);
+    IO.enumCase(ID, "riscv-vec", TargetStackID::RISCVVector);
     IO.enumCase(ID, "noalloc", TargetStackID::NoAlloc);
   }
 };
 
 template <> struct MappingTraits<FixedMachineStackObject> {
   static void mapping(yaml::IO &YamlIO, FixedMachineStackObject &Object) {
     YamlIO.mapRequired("id", Object.ID);
     YamlIO.mapOptional(

llvm/include/llvm/CodeGen/MachineFrameInfo.h

 //===-- CodeGen/MachineFrameInfo.h - Abstract Stack Frame Rep. --*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // The file defines the MachineFrameInfo class.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_MACHINEFRAMEINFO_H
 #define LLVM_CODEGEN_MACHINEFRAMEINFO_H
 
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Support/Alignment.h"
 #include "llvm/Support/DataTypes.h"
+#include "llvm/CodeGen/TargetFrameLowering.h"
 #include <cassert>
 #include <vector>
 
 namespace llvm {
 class raw_ostream;
 class MachineFunction;
 class MachineBasicBlock;
 class BitVector;
private:
   /// the stack.  Because fixed objects are stored at a negative index in the
   /// Objects list, this is also the index to the 0th object in the list.
   unsigned NumFixedObjects = 0;
 
   /// This boolean keeps track of whether any variable
   /// sized objects have been allocated yet.
   bool HasVarSizedObjects = false;
 
+  /// This boolean keeps track of whether any scalable
+  /// vector objects have been allocated yet.
+  bool HasScalableVectorObjects = false;
+
   /// This boolean keeps track of whether there is a call
   /// to builtin \@llvm.frameaddress.
   bool FrameAddressTaken = false;
 
   /// This boolean keeps track of whether there is a call
   /// to builtin \@llvm.returnaddress.
   bool ReturnAddressTaken = false;
 
public:
   /// Return true if there are any stack objects in this function.
   bool hasStackObjects() const { return !Objects.empty(); }
 
   /// This method may be called any time after instruction
   /// selection is complete to determine if the stack frame for this function
   /// contains any variable sized objects.
   bool hasVarSizedObjects() const { return HasVarSizedObjects; }
 
+  /// This method may be called any time after instruction
+  /// selection is complete to determine if the stack frame for this function
+  /// contains any scalable vector objects.
+  bool hasScalableVectorObjects() const { return HasScalableVectorObjects; }
+
   /// Return the index for the stack protector object.
   int getStackProtectorIndex() const { return StackProtectorIdx; }
   void setStackProtectorIndex(int I) { StackProtectorIdx = I; }
   bool hasStackProtectorIndex() const { return StackProtectorIdx != -1; }
 
   /// Return the index for the function context object.
   /// This object is used for SjLj exceptions.
   int getFunctionContextIndex() const { return FunctionContextIdx; }
uint8_t getStackID(int ObjectIdx) const {
     return Objects[ObjectIdx+NumFixedObjects].StackID;
   }
 
   /// \see StackID
   void setStackID(int ObjectIdx, uint8_t ID) {
     assert(unsigned(ObjectIdx+NumFixedObjects) < Objects.size() &&
            "Invalid Object Idx!");
     Objects[ObjectIdx+NumFixedObjects].StackID = ID;
+    if (ID == TargetStackID::SVEVector || ID == TargetStackID::RISCVVector)
+      HasScalableVectorObjects = true;
     // If ID > 0, MaxAlignment may now be overly conservative.
     // If ID == 0, MaxAlignment will need to be updated separately.
   }
 
   /// Returns true if the specified index corresponds to a dead object.
   bool isDeadObjectIndex(int ObjectIdx) const {
     assert(unsigned(ObjectIdx+NumFixedObjects) < Objects.size() &&
            "Invalid Object Idx!");

llvm/include/llvm/CodeGen/TargetFrameLowering.h

 
 namespace llvm {
   class BitVector;
   class CalleeSavedInfo;
   class MachineFunction;
   class RegScavenger;
 
 namespace TargetStackID {
   enum Value {
     Default = 0,
     SGPRSpill = 1,
     SVEVector = 2,
+    RISCVVector = 3,
     NoAlloc = 255
   };
 }
 
 /// Information about stack frame layout on the target.  It holds the direction
 /// of stack growth, the known stack alignment on entry to each function, and
 /// the offset to the locals area.
 ///

llvm/lib/Target/AMDGPU/SIFrameLowering.cpp

 
 bool SIFrameLowering::isSupportedStackID(TargetStackID::Value ID) const {
   switch (ID) {
   case TargetStackID::Default:
   case TargetStackID::NoAlloc:
   case TargetStackID::SGPRSpill:
     return true;
   case TargetStackID::SVEVector:
+  case TargetStackID::RISCVVector:
     return false;
   }
   llvm_unreachable("Invalid TargetStackID::Value");
 }
 
 // Activate all lanes, returns saved exec.
 static Register buildScratchExecCopy(LivePhysRegs &LiveRegs,
                                      MachineFunction &MF,

llvm/lib/Target/RISCV/RISCVFrameLowering.h

public:
   // Get the first stack adjustment amount for SplitSPAdjust.
   // Return 0 if we don't want to to split the SP adjustment in prologue and
   // epilogue.
   uint64_t getFirstSPAdjustAmount(const MachineFunction &MF) const;
 
   bool canUseAsPrologue(const MachineBasicBlock &MBB) const override;
   bool canUseAsEpilogue(const MachineBasicBlock &MBB) const override;
 
+  bool isSupportedStackID(TargetStackID::Value ID) const override;
+  TargetStackID::Value getStackIDForScalableVectors() const override;
+
+  /// targetHandlesStackFrameRounding - Returns true if the target is
+  /// responsible for rounding up the stack frame (probably at emitPrologue
+  /// time).
+  bool targetHandlesStackFrameRounding() const override { return true; }
+
 protected:
   const RISCVSubtarget &STI;
 
 private:
   void determineFrameLayout(MachineFunction &MF) const;
   void adjustReg(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
                  const DebugLoc &DL, Register DestReg, Register SrcReg,
                  int64_t Val, MachineInstr::MIFlag Flag) const;
+  void prepareStorageSpilledVR(MachineFunction &MF, MachineBasicBlock &MBB,
+                               MachineBasicBlock::iterator MBBI,
+                               const MachineFrameInfo &MFI,
+                               MachineRegisterInfo &MRI,
+                               const TargetInstrInfo &TII,
+                               const DebugLoc &DL) const;
 };
 }
 #endif

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

 }
 
 bool RISCVFrameLowering::hasFP(const MachineFunction &MF) const {
   const TargetRegisterInfo *RegInfo = MF.getSubtarget().getRegisterInfo();
 
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   return MF.getTarget().Options.DisableFramePointerElim(MF) ||
          RegInfo->needsStackRealignment(MF) || MFI.hasVarSizedObjects() ||
-         MFI.isFrameAddressTaken();
+         MFI.isFrameAddressTaken() || MFI.hasScalableVectorObjects();
 }
 
 bool RISCVFrameLowering::hasBP(const MachineFunction &MF) const {
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   const TargetRegisterInfo *TRI = STI.getRegisterInfo();
 
   return MFI.hasVarSizedObjects() && TRI->needsStackRealignment(MF);
 }
 
 // Determines the size of the frame and maximum call frame size.
 void RISCVFrameLowering::determineFrameLayout(MachineFunction &MF) const {
   MachineFrameInfo &MFI = MF.getFrameInfo();
   const RISCVRegisterInfo *RI = STI.getRegisterInfo();
 
   // Get the number of bytes to allocate from the FrameInfo.
   uint64_t FrameSize = MFI.getStackSize();
 
+  // Account all RVV frame objects taking the size of a pointer.
+  // In the current implementation, we will reserve stack space for RVV
+  // objects and store the base pointers into the frame. We need to reserve
+  // frame space for these pointers.
+  //
+  // |---------| <-- frame pointer
+  // |         |
+  // |         |
+  // |---------|
+  // | new sp  |
+  // |---------| <-- sp
+  // | RVV     | size = VLENB x LMUL
+  // | objects |
+  // |---------| <-- new sp = sp - size
+  for (int FI = MFI.getObjectIndexBegin(), EFI = MFI.getObjectIndexEnd();
+       FI < EFI; FI++) {
+    uint8_t StackID = MFI.getStackID(FI);
+    if (StackID == TargetStackID::Default)
+      continue;
+    if (MFI.isDeadObjectIndex(FI))
+      continue;
+
+    switch (StackID) {
+    case TargetStackID::RISCVVector: {
+      const TargetRegisterInfo *RegInfo = MF.getSubtarget().getRegisterInfo();
+      FrameSize =
+          alignTo(FrameSize, RegInfo->getSpillAlignment(RISCV::GPRRegClass));
+      FrameSize += RegInfo->getSpillSize(RISCV::GPRRegClass);
+      MFI.setObjectOffset(FI, -FrameSize);
+      break;
+    }
+    default:
+      llvm_unreachable("Unexpected StackID");
+    }
+  }
+
+  // We do not want the generic alignment handling in
+  // PEI::calculateFrameObjectOffsets(). We store the pointers of RVV objects
+  // in the frame, instead of storing RVV objects themselves. That is why we
+  // do not want to use the FixedSize of RVV objects to align these objects in
+  // calculateFrameObjectOffsets().
+  // targetHandlesStackFrameRounding() will return true and handle the alignment
+  // here.
+  // MaxCallFrameSize is defined through ADJCALLSTACKDOWN.
+  // Reserve frame space for out going arguments of caller.
+  if (MFI.adjustsStack() && hasReservedCallFrame(MF))
+    FrameSize += MFI.getMaxCallFrameSize();
+
   // Get the alignment.
   Align StackAlign = getStackAlign();
-  if (RI->needsStackRealignment(MF)) {
+  // If there are RVV objects in the stack, MFI.getMaxAlign() is not usable here.
+  // We store the pointer to RVV objects into the frame, instead of storing the
+  // RVV objects. We do not use MFI.getMaxAlign() to align the stack if there
+  // are RVV objects in the frame.
+  if (RI->needsStackRealignment(MF) && !MFI.hasScalableVectorObjects()) {
     Align MaxStackAlign = std::max(StackAlign, MFI.getMaxAlign());
     FrameSize += (MaxStackAlign.value() - StackAlign.value());
     StackAlign = MaxStackAlign;
   }
 
   // Set Max Call Frame Size
   uint64_t MaxCallSize = alignTo(MFI.getMaxCallFrameSize(), StackAlign);
   MFI.setMaxCallFrameSize(MaxCallSize);
getNonLibcallCSI(const std::vector<CalleeSavedInfo> &CSI) {
 
   for (auto &CS : CSI)
     if (CS.getFrameIdx() >= 0)
       NonLibcallCSI.push_back(CS);
 
   return NonLibcallCSI;
 }
 
+bool RISCVFrameLowering::isSupportedStackID(TargetStackID::Value ID) const {
+  switch (ID) {
+  case TargetStackID::Default:
+  case TargetStackID::RISCVVector:
+    return true;
+  case TargetStackID::NoAlloc:
+  case TargetStackID::SGPRSpill:
+  case TargetStackID::SVEVector:
+    return false;
+  }
+  llvm_unreachable("Invalid TargetStackID::Value");
+}
+
 void RISCVFrameLowering::emitPrologue(MachineFunction &MF,
                                       MachineBasicBlock &MBB) const {
   MachineFrameInfo &MFI = MF.getFrameInfo();
   auto *RVFI = MF.getInfo<RISCVMachineFunctionInfo>();
   const RISCVRegisterInfo *RI = STI.getRegisterInfo();
   const RISCVInstrInfo *TII = STI.getInstrInfo();
   MachineBasicBlock::iterator MBBI = MBB.begin();
 
void RISCVFrameLowering::emitPrologue(MachineFunction &MF,
 
   // Allocate space on the stack if necessary.
   adjustReg(MBB, MBBI, DL, SPReg, SPReg, -StackSize, MachineInstr::FrameSetup);
 
   // Emit ".cfi_def_cfa_offset RealStackSize"
   unsigned CFIIndex = MF.addFrameInst(
       MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize));
   BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
-      .addCFIIndex(CFIIndex);
+      .addCFIIndex(CFIIndex)
+      .setMIFlag(MachineInstr::FrameSetup);
 
   const auto &CSI = MFI.getCalleeSavedInfo();
 
   // The frame pointer is callee-saved, and code has been generated for us to
   // save it to the stack. We need to skip over the storing of callee-saved
   // registers as the frame pointer must be modified after it has been saved
   // to the stack, not before.
   // FIXME: assumes exactly one instruction is used to save each callee-saved
   // register.
-  std::advance(MBBI, getNonLibcallCSI(CSI).size());
+  // RVV registers break this assumption but they are to be handled after we
+  // adjust the FP.
+  int InsnToSkip = getNonLibcallCSI(CSI).size();
+  for (auto &CS : CSI) {
+    if (RISCV::VRRegClass.contains(CS.getReg()))
+      InsnToSkip--;
+  }
+  std::advance(MBBI, InsnToSkip);
 
   // Iterate over list of callee-saved registers and emit .cfi_offset
   // directives.
   for (const auto &Entry : CSI) {
     int FrameIdx = Entry.getFrameIdx();
     int64_t Offset;
     // Offsets for objects with fixed locations (IE: those saved by libcall) are
     // simply calculated from the frame index.
     if (FrameIdx < 0)
       Offset = FrameIdx * (int64_t) STI.getXLen() / 8;
     else
       Offset = MFI.getObjectOffset(Entry.getFrameIdx()) -
                RVFI->getLibCallStackSize();
     Register Reg = Entry.getReg();
+    // We don't have sensible DWARF for VRs yet
+    if (RISCV::VRRegClass.contains(Reg))
+      continue;
     unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::createOffset(
         nullptr, RI->getDwarfRegNum(Reg, true), Offset));
     BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
-        .addCFIIndex(CFIIndex);
+        .addCFIIndex(CFIIndex)
+        .setMIFlag(MachineInstr::FrameSetup);
   }
 
   // Generate new FP.
   if (hasFP(MF)) {
     if (STI.isRegisterReservedByUser(FPReg))
       MF.getFunction().getContext().diagnose(DiagnosticInfoUnsupported{
           MF.getFunction(), "Frame pointer required, but has been reserved."});
 
     adjustReg(MBB, MBBI, DL, FPReg, SPReg,
               RealStackSize - RVFI->getVarArgsSaveSize(),
               MachineInstr::FrameSetup);
 
     // Emit ".cfi_def_cfa $fp, RVFI->getVarArgsSaveSize()"
     unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::cfiDefCfa(
         nullptr, RI->getDwarfRegNum(FPReg, true), RVFI->getVarArgsSaveSize()));
     BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
-        .addCFIIndex(CFIIndex);
+        .addCFIIndex(CFIIndex)
+        .setMIFlag(MachineInstr::FrameSetup);
   }
 
   // Emit the second SP adjustment after saving callee saved registers.
   if (FirstSPAdjustAmount) {
     uint64_t SecondSPAdjustAmount = MFI.getStackSize() - FirstSPAdjustAmount;
     assert(SecondSPAdjustAmount > 0 &&
            "SecondSPAdjustAmount should be greater than zero");
     adjustReg(MBB, MBBI, DL, SPReg, SPReg, -SecondSPAdjustAmount,
               MachineInstr::FrameSetup);
 
     // If we are using a frame-pointer, and thus emitted ".cfi_def_cfa fp, 0",
     // don't emit an sp-based .cfi_def_cfa_offset
     if (!hasFP(MF)) {
       // Emit ".cfi_def_cfa_offset StackSize"
       unsigned CFIIndex = MF.addFrameInst(
           MCCFIInstruction::cfiDefCfaOffset(nullptr, MFI.getStackSize()));
       BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
-          .addCFIIndex(CFIIndex);
+          .addCFIIndex(CFIIndex)
+          .setMIFlag(MachineInstr::FrameSetup);
     }
   }
 
   if (hasFP(MF)) {
     // Realign Stack
     const RISCVRegisterInfo *RI = STI.getRegisterInfo();
-    if (RI->needsStackRealignment(MF)) {
+    if (RI->needsStackRealignment(MF) && !MFI.hasScalableVectorObjects()) {
       Align MaxAlignment = MFI.getMaxAlign();
 
       const RISCVInstrInfo *TII = STI.getInstrInfo();
       if (isInt<12>(-(int)MaxAlignment.value())) {
         BuildMI(MBB, MBBI, DL, TII->get(RISCV::ANDI), SPReg)
             .addReg(SPReg)
             .addImm(-(int)MaxAlignment.value());
       } else {
if (RI->needsStackRealignment(MF) && !MFI.hasScalableVectorObjects()) {
       if (hasBP(MF)) {
         // move BP, SP
         BuildMI(MBB, MBBI, DL, TII->get(RISCV::ADDI), BPReg)
             .addReg(SPReg)
             .addImm(0);
       }
     }
   }
+
+  prepareStorageSpilledVR(MF, MBB, MBBI, MFI, MF.getRegInfo(), *TII, DL);
+}
+
+void RISCVFrameLowering::prepareStorageSpilledVR(
+    MachineFunction &MF, MachineBasicBlock &MBB,
+    MachineBasicBlock::iterator MBBI, const MachineFrameInfo &MFI,
+    MachineRegisterInfo &MRI, const TargetInstrInfo &TII,
+    const DebugLoc &DL) const {
+  if (!MFI.hasScalableVectorObjects())
+    return;
+
+  unsigned SizeOfVector = MRI.createVirtualRegister(&RISCV::GPRRegClass);
+  BuildMI(MBB, MBBI, DL, TII.get(RISCV::PseudoReadVLENB), SizeOfVector);
+
+  // Classify the FIs.
+  std::array<llvm::SmallVector<int, 4>, 4> BinSizes;
+  const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
+  // Collect all RVV objects first.
+  for (int FI = MFI.getObjectIndexBegin(), EFI = MFI.getObjectIndexEnd();
+       FI < EFI; FI++) {
+    int8_t StackID = MFI.getStackID(FI);
+    if (StackID == TargetStackID::Default)
+      continue;
+    if (MFI.isDeadObjectIndex(FI))
+      continue;
+    assert(StackID == TargetStackID::RISCVVector && "Unexpected StackID");
+
+    int64_t ObjectSize = MFI.getObjectSize(FI);
+
+    unsigned ShiftAmount;
+    // Mask objects may be logically smaller than the spill size of the VR
+    // class. Reserve the size of VR for mask objects. It is similar for
+    // fractional LMUL objects.
+    if (ObjectSize <= TRI->getSpillSize(RISCV::VRRegClass))
+      ShiftAmount = 0;
+    else if (ObjectSize == TRI->getSpillSize(RISCV::VRM2RegClass))
+      ShiftAmount = 1;
+    else if (ObjectSize == TRI->getSpillSize(RISCV::VRM4RegClass))
+      ShiftAmount = 2;
+    else if (ObjectSize == TRI->getSpillSize(RISCV::VRM8RegClass))
+      ShiftAmount = 3;
+    else
+      llvm_unreachable("Unexpected object size");
+
+    BinSizes[ShiftAmount].push_back(FI);
+  }
+
+  // Do the actual allocation. We will allocate space for LMUL = 1 first,
+  // then LMUL 2, 4, and 8.
+  unsigned SPReg = getSPReg(STI);
+  for (const auto &Bin : BinSizes) {
+    if (Bin.empty())
+      continue;
+
+    unsigned FactorRegister = 0;
+    int ShiftAmount = &Bin - &BinSizes[0];
+    // The maximum LMUL is 8.
+    assert(0 <= ShiftAmount && ShiftAmount <= 3 && "Invalid shift amount!");
+    if (ShiftAmount > 0) {
+      // FactorRegister = VLENB x LMUL.
+      FactorRegister = MRI.createVirtualRegister(&RISCV::GPRRegClass);
+      BuildMI(MBB, MBBI, DL, TII.get(RISCV::SLLI), FactorRegister)
+          .addReg(SizeOfVector)
+          .addImm(ShiftAmount);
+    }
+
+    for (const int &FI : Bin) {
+      if (ShiftAmount > 0) {
+        assert(FactorRegister && "Invalid register!");
+        bool LastUse = &FI == &Bin.back();
+        // SP = SP - VLENB x LMUL.
+        BuildMI(MBB, MBBI, DL, TII.get(RISCV::SUB), SPReg)
+            .addReg(SPReg)
+            .addReg(FactorRegister, LastUse ? RegState::Kill : 0);
+      } else
+        // SP = SP - VLENB.
+        BuildMI(MBB, MBBI, DL, TII.get(RISCV::SUB), SPReg)
+            .addReg(SPReg)
+            .addReg(SizeOfVector);
+
+      TII.storeRegToStackSlot(MBB, MBBI, SPReg, false, FI,
+                              TRI->getMinimalPhysRegClass(SPReg), TRI);
+    }
+  }
 }
 
 void RISCVFrameLowering::emitEpilogue(MachineFunction &MF,
                                       MachineBasicBlock &MBB) const {
   const RISCVRegisterInfo *RI = STI.getRegisterInfo();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   auto *RVFI = MF.getInfo<RISCVMachineFunctionInfo>();
   Register FPReg = getFPReg(STI);
void RISCVFrameLowering::emitEpilogue(MachineFunction &MF,
   }
 
   const auto &CSI = getNonLibcallCSI(MFI.getCalleeSavedInfo());
 
   // Skip to before the restores of callee-saved registers
   // FIXME: assumes exactly one instruction is used to restore each
   // callee-saved register.
   auto LastFrameDestroy = MBBI;
-  if (!CSI.empty())
-    LastFrameDestroy = std::prev(MBBI, CSI.size());
+  if (!CSI.empty()) {
+    // Ignore the VRs as we did in the prologue.
+    int InsnToSkip = CSI.size();
+    for (auto &CS : CSI) {
+      if (RISCV::VRRegClass.contains(CS.getReg()))
+        InsnToSkip--;
+    }
+    LastFrameDestroy = std::prev(MBBI, InsnToSkip);
+  }
 
   uint64_t StackSize = MFI.getStackSize();
   uint64_t RealStackSize = StackSize + RVFI->getLibCallStackSize();
   uint64_t FPOffset = RealStackSize - RVFI->getVarArgsSaveSize();
 
   // Restore the stack pointer using the value of the frame pointer. Only
   // necessary if the stack pointer was modified, meaning the stack size is
   // unknown.
-  if (RI->needsStackRealignment(MF) || MFI.hasVarSizedObjects()) {
+  if (RI->needsStackRealignment(MF) || MFI.hasVarSizedObjects() ||
+      MFI.hasScalableVectorObjects()) {
     assert(hasFP(MF) && "frame pointer should not have been eliminated");
     adjustReg(MBB, LastFrameDestroy, DL, SPReg, FPReg, -FPOffset,
               MachineInstr::FrameDestroy);
   }
 
   uint64_t FirstSPAdjustAmount = getFirstSPAdjustAmount(MF);
   if (FirstSPAdjustAmount) {
     uint64_t SecondSPAdjustAmount = MFI.getStackSize() - FirstSPAdjustAmount;
RISCVFrameLowering::getFrameIndexReference(const MachineFunction &MF, int FI,
 
   uint64_t FirstSPAdjustAmount = getFirstSPAdjustAmount(MF);
 
   if (CSI.size()) {
     MinCSFI = CSI[0].getFrameIdx();
     MaxCSFI = CSI[CSI.size() - 1].getFrameIdx();
   }
 
-  if (FI >= MinCSFI && FI <= MaxCSFI) {
+  if ((FI >= MinCSFI && FI <= MaxCSFI) &&
+      (MFI.getStackID(FI) == TargetStackID::Default)) {
     FrameReg = RISCV::X2;
 
     if (FirstSPAdjustAmount)
       Offset += FirstSPAdjustAmount;
     else
       Offset += MFI.getStackSize();
-  } else if (RI->needsStackRealignment(MF) && !MFI.isFixedObjectIndex(FI)) {
+  } else if (RI->needsStackRealignment(MF) && !MFI.isFixedObjectIndex(FI) &&
+             !MFI.hasScalableVectorObjects()) {
     // If the stack was realigned, the frame pointer is set in order to allow
     // SP to be restored, so we need another base register to record the stack
     // after realignment.
     if (hasBP(MF))
       FrameReg = RISCVABI::getBPReg();
     else
       FrameReg = RISCV::X2;
     Offset += MFI.getStackSize();
RISCVFrameLowering::getFrameIndexReference(const MachineFunction &MF, int FI,
   }
   return StackOffset::getFixed(Offset);
 }
 
 void RISCVFrameLowering::determineCalleeSaves(MachineFunction &MF,
                                               BitVector &SavedRegs,
                                               RegScavenger *RS) const {
   TargetFrameLowering::determineCalleeSaves(MF, SavedRegs, RS);
+
+  MachineFrameInfo &MFI = MF.getFrameInfo();
+
+  // Go through all Stackslots coming from a VR alloca.
+  // We need do this early otherwise we may be answering hasFP false.
+  for (int FI = MFI.getObjectIndexBegin(), EFI = MFI.getObjectIndexEnd();
+       FI < EFI; FI++) {
+    // Get the (LLVM IR) allocation instruction
+    const AllocaInst *Alloca = MFI.getObjectAllocation(FI);
+
+    if (!Alloca)
+      continue;
+
+    if (const ScalableVectorType *VT = dyn_cast<const ScalableVectorType>(
+            Alloca->getType()->getElementType())) {
+      // Set stack ID to RISCVVector will make MFI.hasScalableVectorObjects()
+      // true.
+      MFI.setStackID(FI, TargetStackID::RISCVVector);
+    }
+  }
+
   // Unconditionally spill RA and FP only if the function uses a frame
   // pointer.
   if (hasFP(MF)) {
     SavedRegs.set(RISCV::X1);
     SavedRegs.set(RISCV::X8);
   }
   // Mark BP as used if function has dedicated base pointer.
   if (hasBP(MF))
     SavedRegs.set(RISCVABI::getBPReg());
 
   // If interrupt is enabled and there are calls in the handler,
   // unconditionally save all Caller-saved registers and
   // all FP registers, regardless whether they are used.
-  MachineFrameInfo &MFI = MF.getFrameInfo();
-
   if (MF.getFunction().hasFnAttribute("interrupt") && MFI.hasCalls()) {
 
     static const MCPhysReg CSRegs[] = { RISCV::X1,      /* ra */
       RISCV::X5, RISCV::X6, RISCV::X7,                  /* t0-t2 */
       RISCV::X10, RISCV::X11,                           /* a0-a1, a2-a7 */
       RISCV::X12, RISCV::X13, RISCV::X14, RISCV::X15, RISCV::X16, RISCV::X17,
       RISCV::X28, RISCV::X29, RISCV::X30, RISCV::X31, 0 /* t3-t6 */
     };
if (!isInt<11>(MFI.estimateStackSize(MF))) {
     RS->addScavengingFrameIndex(RegScavFI);
   }
 }
 
 // Not preserve stack space within prologue for outgoing variables when the
 // function contains variable size objects and let eliminateCallFramePseudoInstr
 // preserve stack space for it.
 bool RISCVFrameLowering::hasReservedCallFrame(const MachineFunction &MF) const {
-  return !MF.getFrameInfo().hasVarSizedObjects();
+  const MachineFrameInfo &MFI = MF.getFrameInfo();
+  return !(MFI.hasVarSizedObjects() || MFI.hasScalableVectorObjects());
 }
 
 // Eliminate ADJCALLSTACKDOWN, ADJCALLSTACKUP pseudo instructions.
 MachineBasicBlock::iterator RISCVFrameLowering::eliminateCallFramePseudoInstr(
     MachineFunction &MF, MachineBasicBlock &MBB,
     MachineBasicBlock::iterator MI) const {
   Register SPReg = RISCV::X2;
   DebugLoc DL = MI->getDebugLoc();
bool RISCVFrameLowering::canUseAsEpilogue(const MachineBasicBlock &MBB) const {
   // restore will be eliminated regardless.
   if (!SuccMBB)
     return true;
 
   // The successor can only contain a return, since we would effectively be
   // replacing the successor with our own tail return at the end of our block.
   return SuccMBB->isReturnBlock() && SuccMBB->size() == 1;
 }
+
+TargetStackID::Value RISCVFrameLowering::getStackIDForScalableVectors() const {
+  return TargetStackID::RISCVVector;
+}

llvm/lib/Target/RISCV/RISCVInstrInfoVPseudos.td

 class RISCVVPseudo {
   Pseudo Pseudo = !cast<Pseudo>(NAME); // Used as a key.
   Instruction BaseInstr;
   bits<8> VLIndex = InvalidIndex.V;
   bits<8> SEWIndex = InvalidIndex.V;
   bits<8> MergeOpIndex = InvalidIndex.V;
   bits<3> VLMul;
   bit HasDummyMask = 0;
+  bit IsRVVLoadStore = 0;
 }
 
 // The actual table.
 def RISCVVPseudosTable : GenericTable {
   let FilterClass = "RISCVVPseudo";
   let CppTypeName = "PseudoInfo";
   let Fields = [ "Pseudo", "BaseInstr", "VLIndex", "SEWIndex", "MergeOpIndex",
-                 "VLMul", "HasDummyMask" ];
+                 "VLMul", "HasDummyMask", "IsRVVLoadStore" ];
   let PrimaryKey = [ "Pseudo" ];
   let PrimaryKeyName = "getPseudoInfo";
 }
 
 def RISCVVIntrinsicsTable : GenericTable {
   let FilterClass = "RISCVVIntrinsic";
   let CppTypeName = "RISCVVIntrinsicInfo";
   let Fields = ["IntrinsicID", "ExtendOperand"];
class VPseudoUSLoadNoMask<VReg RetClass>:
   let mayLoad = 1;
   let mayStore = 0;
   let hasSideEffects = 0;
   let usesCustomInserter = 1;
   let Uses = [VL, VTYPE];
   let VLIndex = 2;
   let SEWIndex = 3;
   let HasDummyMask = 1;
+  let IsRVVLoadStore = 1;
   let BaseInstr = !cast<Instruction>(PseudoToVInst<NAME>.VInst);
 }
 
 class VPseudoUSLoadMask<VReg RetClass>:
       Pseudo<(outs GetVRegNoV0<RetClass>.R:$rd),
               (ins GetVRegNoV0<RetClass>.R:$merge,
                    GPR:$rs1,
                    VMaskOp:$vm, GPR:$vl, ixlenimm:$sew),[]>,
       RISCVVPseudo {
   let mayLoad = 1;
   let mayStore = 0;
   let hasSideEffects = 0;
   let usesCustomInserter = 1;
   let Constraints = "$rd = $merge";
   let Uses = [VL, VTYPE];
   let VLIndex = 4;
   let SEWIndex = 5;
   let MergeOpIndex = 1;
+  let IsRVVLoadStore = 1;
   let BaseInstr = !cast<Instruction>(PseudoToVInst<NAME>.VInst);
 }
 
 class VPseudoSLoadNoMask<VReg RetClass>:
       Pseudo<(outs RetClass:$rd),
              (ins GPR:$rs1, GPR:$rs2, GPR:$vl, ixlenimm:$sew),[]>,
       RISCVVPseudo {
   let mayLoad = 1;
   let mayStore = 0;
   let hasSideEffects = 0;
   let usesCustomInserter = 1;
   let Uses = [VL, VTYPE];
   let VLIndex = 3;
   let SEWIndex = 4;
   let HasDummyMask = 1;
+  let IsRVVLoadStore = 1;
   let BaseInstr = !cast<Instruction>(PseudoToVInst<NAME>.VInst);
 }
 
 class VPseudoSLoadMask<VReg RetClass>:
       Pseudo<(outs GetVRegNoV0<RetClass>.R:$rd),
               (ins GetVRegNoV0<RetClass>.R:$merge,
                    GPR:$rs1, GPR:$rs2,
                    VMaskOp:$vm, GPR:$vl, ixlenimm:$sew),[]>,
       RISCVVPseudo {
   let mayLoad = 1;
   let mayStore = 0;
   let hasSideEffects = 0;
   let usesCustomInserter = 1;
   let Constraints = "$rd = $merge";
   let Uses = [VL, VTYPE];
   let VLIndex = 5;
   let SEWIndex = 6;
   let MergeOpIndex = 1;
+  let IsRVVLoadStore = 1;
   let BaseInstr = !cast<Instruction>(PseudoToVInst<NAME>.VInst);
 }
 
 class VPseudoILoadNoMask<VReg RetClass, VReg IdxClass>:
       Pseudo<(outs RetClass:$rd),
              (ins GPR:$rs1, IdxClass:$rs2, GPR:$vl, ixlenimm:$sew),[]>,
       RISCVVPseudo {
   let mayLoad = 1;
   let mayStore = 0;
   let hasSideEffects = 0;
   let usesCustomInserter = 1;
   let Uses = [VL, VTYPE];
   let VLIndex = 3;
   let SEWIndex = 4;
   let HasDummyMask = 1;
+  let IsRVVLoadStore = 1;
   let BaseInstr = !cast<Instruction>(PseudoToVInst<NAME>.VInst);
 }
 
 class VPseudoILoadMask<VReg RetClass, VReg IdxClass>:
       Pseudo<(outs GetVRegNoV0<RetClass>.R:$rd),
               (ins GetVRegNoV0<RetClass>.R:$merge,
                    GPR:$rs1, IdxClass:$rs2,
                    VMaskOp:$vm, GPR:$vl, ixlenimm:$sew),[]>,
       RISCVVPseudo {
   let mayLoad = 1;
   let mayStore = 0;
   let hasSideEffects = 0;
   let usesCustomInserter = 1;
   let Constraints = "$rd = $merge";
   let Uses = [VL, VTYPE];
   let VLIndex = 5;
   let SEWIndex = 6;
   let MergeOpIndex = 1;
+  let IsRVVLoadStore = 1;
   let BaseInstr = !cast<Instruction>(PseudoToVInst<NAME>.VInst);
 }
 
 class VPseudoUSStoreNoMask<VReg StClass>:
       Pseudo<(outs),
               (ins StClass:$rd, GPR:$rs1, GPR:$vl, ixlenimm:$sew),[]>,
       RISCVVPseudo {
   let mayLoad = 0;
   let mayStore = 1;
   let hasSideEffects = 0;
   let usesCustomInserter = 1;
   let Uses = [VL, VTYPE];
   let VLIndex = 2;
   let SEWIndex = 3;
   let HasDummyMask = 1;
+  let IsRVVLoadStore = 1;
   let BaseInstr = !cast<Instruction>(PseudoToVInst<NAME>.VInst);
 }
 
 class VPseudoUSStoreMask<VReg StClass>:
       Pseudo<(outs),
               (ins StClass:$rd, GPR:$rs1, VMaskOp:$vm, GPR:$vl, ixlenimm:$sew),[]>,
       RISCVVPseudo {
   let mayLoad = 0;
   let mayStore = 1;
   let hasSideEffects = 0;
   let usesCustomInserter = 1;
   let Uses = [VL, VTYPE];
   let VLIndex = 3;
   let SEWIndex = 4;
+  let IsRVVLoadStore = 1;
   let BaseInstr = !cast<Instruction>(PseudoToVInst<NAME>.VInst);
 }
 
 class VPseudoSStoreNoMask<VReg StClass>:
       Pseudo<(outs),
               (ins StClass:$rd, GPR:$rs1, GPR:$rs2, GPR:$vl, ixlenimm:$sew),[]>,
       RISCVVPseudo {
   let mayLoad = 0;
   let mayStore = 1;
   let hasSideEffects = 0;
   let usesCustomInserter = 1;
   let Uses = [VL, VTYPE];
   let VLIndex = 3;
   let SEWIndex = 4;
   let HasDummyMask = 1;
+  let IsRVVLoadStore = 1;
   let BaseInstr = !cast<Instruction>(PseudoToVInst<NAME>.VInst);
 }
 
 class VPseudoSStoreMask<VReg StClass>:
       Pseudo<(outs),
               (ins StClass:$rd, GPR:$rs1, GPR:$rs2, VMaskOp:$vm, GPR:$vl, ixlenimm:$sew),[]>,
       RISCVVPseudo {
   let mayLoad = 0;
   let mayStore = 1;
   let hasSideEffects = 0;
   let usesCustomInserter = 1;
   let Uses = [VL, VTYPE];
   let VLIndex = 4;
   let SEWIndex = 5;
+  let IsRVVLoadStore = 1;
   let BaseInstr = !cast<Instruction>(PseudoToVInst<NAME>.VInst);
 }
 
 // Unary instruction that is never masked so HasDummyMask=0.
 class VPseudoUnaryNoDummyMask<VReg RetClass,
                               DAGOperand Op2Class> :
         Pseudo<(outs RetClass:$rd),
                (ins Op2Class:$rs1, GPR:$vl, ixlenimm:$sew), []>,
class VPseudoIStoreNoMask<VReg StClass, VReg IdxClass>:
   let mayLoad = 0;
   let mayStore = 1;
   let hasSideEffects = 0;
   let usesCustomInserter = 1;
   let Uses = [VL, VTYPE];
   let VLIndex = 3;
   let SEWIndex = 4;
   let HasDummyMask = 1;
+  let IsRVVLoadStore = 1;
   let BaseInstr = !cast<Instruction>(PseudoToVInst<NAME>.VInst);
 }
 
 class VPseudoIStoreMask<VReg StClass, VReg IdxClass>:
       Pseudo<(outs),
               (ins StClass:$rd, GPR:$rs1, IdxClass:$rs2, VMaskOp:$vm, GPR:$vl, ixlenimm:$sew),[]>,
       RISCVVPseudo {
   let mayLoad = 0;
   let mayStore = 1;
   let hasSideEffects = 0;
   let usesCustomInserter = 1;
   let Uses = [VL, VTYPE];
   let VLIndex = 4;
   let SEWIndex = 5;
+  let IsRVVLoadStore = 1;
   let BaseInstr = !cast<Instruction>(PseudoToVInst<NAME>.VInst);
 }
 
 class VPseudoBinaryMask<VReg RetClass,
                         VReg Op1Class,
                         DAGOperand Op2Class,
                         string Constraint> :
         Pseudo<(outs GetVRegNoV0<RetClass>.R:$rd),

 //===----------------------------------------------------------------------===//
 let hasSideEffects = 0, mayLoad = 0, mayStore = 0 in {
   def PseudoVMV1R_V : VPseudo<VMV1R_V, V_M1, (outs VR:$vd), (ins VR:$vs2)>;
   def PseudoVMV2R_V : VPseudo<VMV2R_V, V_M2, (outs VRM2:$vd), (ins VRM2:$vs2)>;
   def PseudoVMV4R_V : VPseudo<VMV4R_V, V_M4, (outs VRM4:$vd), (ins VRM4:$vs2)>;
   def PseudoVMV8R_V : VPseudo<VMV8R_V, V_M8, (outs VRM8:$vd), (ins VRM8:$vs2)>;
 }
 
+let hasSideEffects = 0, mayLoad = 0, mayStore = 0, isCodeGenOnly = 1 in {
+  def PseudoReadVLENB : Pseudo<(outs GPR:$rd), (ins), []>;
+}
+
 //===----------------------------------------------------------------------===//
 // 6. Configuration-Setting Instructions
 //===----------------------------------------------------------------------===//
 
 // Pseudos.
 let hasSideEffects = 1, mayLoad = 0, mayStore = 0, Defs = [VL, VTYPE] in {
 def PseudoVSETVLI : Pseudo<(outs GPR:$rd), (ins GPR:$rs1, VTypeIOp:$vtypei), []>;
 

llvm/lib/Target/RISCV/RISCVMCInstLower.cpp

 //===-- RISCVMCInstLower.cpp - Convert RISCV MachineInstr to an MCInst ------=//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file contains code to lower RISCV MachineInstrs to their corresponding
 // MCInst records.
 //
 //===----------------------------------------------------------------------===//
 
 #include "RISCV.h"
 #include "RISCVSubtarget.h"
 #include "MCTargetDesc/RISCVMCExpr.h"
+#include "Utils/RISCVBaseInfo.h"
 #include "llvm/CodeGen/AsmPrinter.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCContext.h"
 #include "llvm/MC/MCExpr.h"
 #include "llvm/MC/MCInst.h"
 #include "llvm/Support/ErrorHandling.h"
void llvm::LowerRISCVMachineInstrToMCInst(const MachineInstr *MI, MCInst &OutMI,
 
   OutMI.setOpcode(MI->getOpcode());
 
   for (const MachineOperand &MO : MI->operands()) {
     MCOperand MCOp;
     if (LowerRISCVMachineOperandToMCOperand(MO, MCOp, AP))
       OutMI.addOperand(MCOp);
   }
+
+  if (OutMI.getOpcode() == RISCV::PseudoReadVLENB) {
+    OutMI.setOpcode(RISCV::CSRRS);
+    OutMI.addOperand(MCOperand::createImm(RISCVCSR::VLENB));
+    OutMI.addOperand(MCOperand::createReg(RISCV::X0));
+    return;
+  }
+
 }

llvm/lib/Target/RISCV/RISCVRegisterInfo.cpp

 
 void RISCVRegisterInfo::eliminateFrameIndex(MachineBasicBlock::iterator II,
                                             int SPAdj, unsigned FIOperandNum,
                                             RegScavenger *RS) const {
   assert(SPAdj == 0 && "Unexpected non-zero SPAdj value");
 
   MachineInstr &MI = *II;
   MachineFunction &MF = *MI.getParent()->getParent();
+  MachineFrameInfo &MFI = MF.getFrameInfo();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   const RISCVInstrInfo *TII = MF.getSubtarget<RISCVSubtarget>().getInstrInfo();
   DebugLoc DL = MI.getDebugLoc();
 
   int FrameIndex = MI.getOperand(FIOperandNum).getIndex();
   Register FrameReg;
   int Offset = getFrameLowering(MF)
                    ->getFrameIndexReference(MF, FrameIndex, FrameReg)
-                   .getFixed() +
-               MI.getOperand(FIOperandNum + 1).getImm();
+                   .getFixed();
+
+  bool NeedsIndirectAddressing = false;
+  const RISCVVPseudosTable::PseudoInfo *RVV =
+      RISCVVPseudosTable::getPseudoInfo(MI.getOpcode());
+  // We store RVV frame object base addresses into the frame. Load the base
+  // address first, then load the RVV object itself.
+  if (RVV && RVV->IsRVVLoadStore)
+    NeedsIndirectAddressing =
+        MFI.getStackID(FrameIndex) == TargetStackID::RISCVVector;
+  else
+    Offset += MI.getOperand(FIOperandNum + 1).getImm();
 
   if (!isInt<32>(Offset)) {
     report_fatal_error(
         "Frame offsets outside of the signed 32-bit range not supported");
   }
 
   MachineBasicBlock &MBB = *MI.getParent();
+  if (NeedsIndirectAddressing) {
+    MachineOperand &SlotAddr = MI.getOperand(FIOperandNum);
+    Register RVVBaseReg = MRI.createVirtualRegister(&RISCV::GPRRegClass);
+    auto &Subtarget = MF.getSubtarget<RISCVSubtarget>();
+    unsigned LoadOpcode = Subtarget.is64Bit() ? RISCV::LD : RISCV::LW;
+    MachineInstr *LoadRVVBase =
+        BuildMI(MBB, II, DL, TII->get(LoadOpcode), RVVBaseReg)
+            .add(SlotAddr)
+            .addImm(0);
+    SlotAddr.ChangeToRegister(RVVBaseReg, false, false, /*isKill=*/true);
+
+    // Resolve FrameIndex in `LD/LW RVVBaseReg, FI, 0`
+    return eliminateFrameIndex(LoadRVVBase, /*SPAdj=*/0, 1, RS);
+  }
+
   bool FrameRegIsKill = false;
 
   if (!isInt<12>(Offset)) {
     assert(isInt<32>(Offset) && "Int32 expected");
     // The offset won't fit in an immediate, so use a scratch register instead
     // Modify Offset and FrameReg appropriately
     Register ScratchReg = MRI.createVirtualRegister(&RISCV::GPRRegClass);
     TII->movImm(MBB, II, DL, ScratchReg, Offset);
     BuildMI(MBB, II, DL, TII->get(RISCV::ADD), ScratchReg)
         .addReg(FrameReg)
         .addReg(ScratchReg, RegState::Kill);
     Offset = 0;
     FrameReg = ScratchReg;
     FrameRegIsKill = true;
   }
 
   MI.getOperand(FIOperandNum)
       .ChangeToRegister(FrameReg, false, false, FrameRegIsKill);
+  if (!(RVV && RVV->IsRVVLoadStore))
     MI.getOperand(FIOperandNum + 1).ChangeToImmediate(Offset);
 }
 
 Register RISCVRegisterInfo::getFrameRegister(const MachineFunction &MF) const {
   const TargetFrameLowering *TFI = getFrameLowering(MF);
   return TFI->hasFP(MF) ? RISCV::X8 : RISCV::X2;
 }
 
 const uint32_t *

llvm/lib/Target/RISCV/Utils/RISCVBaseInfo.h

 struct PseudoInfo {
   unsigned int Pseudo;
   unsigned int BaseInstr;
   uint8_t VLIndex;
   uint8_t SEWIndex;
   uint8_t MergeOpIndex;
   uint8_t VLMul;
   bool HasDummyMask;
+  bool IsRVVLoadStore;
 
   int getVLIndex() const { return static_cast<int8_t>(VLIndex); }
 
   int getSEWIndex() const { return static_cast<int8_t>(SEWIndex); }
 
   int getMergeOpIndex() const { return static_cast<int8_t>(MergeOpIndex); }
 
   bool hasDummyMask() const { return HasDummyMask; }
 };
 
 using namespace RISCV;
 
 #define GET_RISCVVPseudosTable_DECL
 #include "RISCVGenSearchableTables.inc"
 
 } // end namespace RISCVVPseudosTable
 
+namespace RISCVCSR {
+enum {
+  VSTART = 0x008,
+  VXSAT = 0x009,
+  VXRM = 0x00A,
+  VL = 0xC20,
+  VTYPE = 0xC21,
+  VLENB = 0xC22,
+};
+} // end namespace RISCVCSR
+
 } // namespace llvm
 
 #endif

llvm/test/CodeGen/RISCV/rvv/allocate-lmul-2-4-8.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv64 -mattr=+experimental-v -verify-machineinstrs < %s \
+; RUN:    | FileCheck %s
+
+define void @lmul1() nounwind optnone noinline {
+; CHECK-LABEL: lmul1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -32
+; CHECK-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 32
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    addi sp, s0, -32
+; CHECK-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 32
+; CHECK-NEXT:    ret
+  %v = alloca <vscale x 1 x i64>
+  ret void
+}
+
+define void @lmul2() nounwind optnone noinline {
+; CHECK-LABEL: lmul2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -32
+; CHECK-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 32
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    slli a0, a0, 1
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    addi sp, s0, -32
+; CHECK-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 32
+; CHECK-NEXT:    ret
+  %v = alloca <vscale x 2 x i64>
+  ret void
+}
+
+define void @lmul4() nounwind optnone noinline {
+; CHECK-LABEL: lmul4:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -32
+; CHECK-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 32
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    slli a0, a0, 2
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    addi sp, s0, -32
+; CHECK-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 32
+; CHECK-NEXT:    ret
+  %v = alloca <vscale x 4 x i64>
+  ret void
+}
+
+define void @lmul8() nounwind optnone noinline {
+; CHECK-LABEL: lmul8:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -32
+; CHECK-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 32
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    slli a0, a0, 3
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    addi sp, s0, -32
+; CHECK-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 32
+; CHECK-NEXT:    ret
+  %v = alloca <vscale x 8 x i64>
+  ret void
+}
+
+define void @lmul1_and_2() nounwind optnone noinline {
+; CHECK-LABEL: lmul1_and_2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -32
+; CHECK-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 32
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    slli a0, a0, 1
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    addi sp, s0, -32
+; CHECK-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 32
+; CHECK-NEXT:    ret
+  %v1 = alloca <vscale x 1 x i64>
+  %v2 = alloca <vscale x 2 x i64>
+  ret void
+}
+
+define void @lmul2_and_4() nounwind optnone noinline {
+; CHECK-LABEL: lmul2_and_4:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -32
+; CHECK-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 32
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    slli a1, a0, 1
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    slli a0, a0, 2
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    addi sp, s0, -32
+; CHECK-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 32
+; CHECK-NEXT:    ret
+  %v1 = alloca <vscale x 2 x i64>
+  %v2 = alloca <vscale x 4 x i64>
+  ret void
+}
+
+define void @lmul1_and_4() nounwind optnone noinline {
+; CHECK-LABEL: lmul1_and_4:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -32
+; CHECK-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 32
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    slli a0, a0, 2
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    addi sp, s0, -32
+; CHECK-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 32
+; CHECK-NEXT:    ret
+  %v1 = alloca <vscale x 1 x i64>
+  %v2 = alloca <vscale x 4 x i64>
+  ret void
+}
+
+define void @lmul2_and_1() nounwind optnone noinline {
+; CHECK-LABEL: lmul2_and_1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -32
+; CHECK-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 32
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    slli a0, a0, 1
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    addi sp, s0, -32
+; CHECK-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 32
+; CHECK-NEXT:    ret
+  %v1 = alloca <vscale x 2 x i64>
+  %v2 = alloca <vscale x 1 x i64>
+  ret void
+}
+
+define void @lmul4_and_1() nounwind optnone noinline {
+; CHECK-LABEL: lmul4_and_1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -32
+; CHECK-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 32
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    slli a0, a0, 2
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    addi sp, s0, -32
+; CHECK-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 32
+; CHECK-NEXT:    ret
+  %v1 = alloca <vscale x 4 x i64>
+  %v2 = alloca <vscale x 1 x i64>
+  ret void
+}
+
+define void @lmul4_and_2() nounwind optnone noinline {
+; CHECK-LABEL: lmul4_and_2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -32
+; CHECK-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 32
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    slli a1, a0, 1
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    slli a0, a0, 2
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    addi sp, s0, -32
+; CHECK-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 32
+; CHECK-NEXT:    ret
+  %v1 = alloca <vscale x 4 x i64>
+  %v2 = alloca <vscale x 2 x i64>
+  ret void
+}
+
+define void @lmul4_and_2_x2_0() nounwind optnone noinline {
+; CHECK-LABEL: lmul4_and_2_x2_0:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -48
+; CHECK-NEXT:    sd ra, 40(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 32(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 48
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    slli a1, a0, 1
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -48(s0)
+; CHECK-NEXT:    slli a0, a0, 2
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -40(s0)
+; CHECK-NEXT:    addi sp, s0, -48
+; CHECK-NEXT:    ld s0, 32(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 40(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 48
+; CHECK-NEXT:    ret
+  %v1 = alloca <vscale x 4 x i64>
+  %v2 = alloca <vscale x 2 x i64>
+  %v3 = alloca <vscale x 4 x i64>
+  %v4 = alloca <vscale x 2 x i64>
+  ret void
+}
+
+define void @lmul4_and_2_x2_1() nounwind optnone noinline {
+; CHECK-LABEL: lmul4_and_2_x2_1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -48
+; CHECK-NEXT:    sd ra, 40(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 32(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 48
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    slli a1, a0, 1
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -40(s0)
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -48(s0)
+; CHECK-NEXT:    slli a0, a0, 2
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    addi sp, s0, -48
+; CHECK-NEXT:    ld s0, 32(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 40(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 48
+; CHECK-NEXT:    ret
+  %v1 = alloca <vscale x 4 x i64>
+  %v3 = alloca <vscale x 4 x i64>
+  %v2 = alloca <vscale x 2 x i64>
+  %v4 = alloca <vscale x 2 x i64>
+  ret void
+}
+
+
+define void @gpr_and_lmul1_and_2() nounwind optnone noinline {
+; CHECK-LABEL: gpr_and_lmul1_and_2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -48
+; CHECK-NEXT:    sd ra, 40(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 32(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 48
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    slli a0, a0, 1
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -40(s0)
+; CHECK-NEXT:    addi a0, zero, 3
+; CHECK-NEXT:    sd a0, -24(s0)
+; CHECK-NEXT:    addi sp, s0, -48
+; CHECK-NEXT:    ld s0, 32(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 40(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 48
+; CHECK-NEXT:    ret
+  %x1 = alloca i64
+  %v1 = alloca <vscale x 1 x i64>
+  %v2 = alloca <vscale x 2 x i64>
+  store volatile i64 3, i64* %x1
+  ret void
+}
+
+define void @gpr_and_lmul1_and_4() nounwind optnone noinline {
+; CHECK-LABEL: gpr_and_lmul1_and_4:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -48
+; CHECK-NEXT:    sd ra, 40(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 32(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 48
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    slli a0, a0, 2
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -40(s0)
+; CHECK-NEXT:    addi a0, zero, 3
+; CHECK-NEXT:    sd a0, -24(s0)
+; CHECK-NEXT:    addi sp, s0, -48
+; CHECK-NEXT:    ld s0, 32(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 40(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 48
+; CHECK-NEXT:    ret
+  %x1 = alloca i64
+  %v1 = alloca <vscale x 1 x i64>
+  %v2 = alloca <vscale x 4 x i64>
+  store volatile i64 3, i64* %x1
+  ret void
+}
+
+define void @lmul_1_2_4_8() nounwind optnone noinline {
+; CHECK-LABEL: lmul_1_2_4_8:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -48
+; CHECK-NEXT:    sd ra, 40(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 32(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 48
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    slli a1, a0, 1
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    slli a1, a0, 2
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -40(s0)
+; CHECK-NEXT:    slli a0, a0, 3
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -48(s0)
+; CHECK-NEXT:    addi sp, s0, -48
+; CHECK-NEXT:    ld s0, 32(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 40(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 48
+; CHECK-NEXT:    ret
+  %v1 = alloca <vscale x 1 x i64>
+  %v2 = alloca <vscale x 2 x i64>
+  %v4 = alloca <vscale x 4 x i64>
+  %v8 = alloca <vscale x 8 x i64>
+  ret void
+}
+
+define void @lmul_1_2_4_8_x2_0() nounwind optnone noinline {
+; CHECK-LABEL: lmul_1_2_4_8_x2_0:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -80
+; CHECK-NEXT:    sd ra, 72(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 64(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 80
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    slli a1, a0, 1
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -40(s0)
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -48(s0)
+; CHECK-NEXT:    slli a1, a0, 2
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -56(s0)
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -64(s0)
+; CHECK-NEXT:    slli a0, a0, 3
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -72(s0)
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -80(s0)
+; CHECK-NEXT:    addi sp, s0, -80
+; CHECK-NEXT:    ld s0, 64(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 72(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 80
+; CHECK-NEXT:    ret
+  %v1 = alloca <vscale x 1 x i64>
+  %v2 = alloca <vscale x 1 x i64>
+  %v3 = alloca <vscale x 2 x i64>
+  %v4 = alloca <vscale x 2 x i64>
+  %v5 = alloca <vscale x 4 x i64>
+  %v6 = alloca <vscale x 4 x i64>
+  %v7 = alloca <vscale x 8 x i64>
+  %v8 = alloca <vscale x 8 x i64>
+  ret void
+}
+
+define void @lmul_1_2_4_8_x2_1() nounwind optnone noinline {
+; CHECK-LABEL: lmul_1_2_4_8_x2_1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -80
+; CHECK-NEXT:    sd ra, 72(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 64(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 80
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -72(s0)
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -80(s0)
+; CHECK-NEXT:    slli a1, a0, 1
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -56(s0)
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -64(s0)
+; CHECK-NEXT:    slli a1, a0, 2
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -40(s0)
+; CHECK-NEXT:    sub sp, sp, a1
+; CHECK-NEXT:    sd sp, -48(s0)
+; CHECK-NEXT:    slli a0, a0, 3
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    addi sp, s0, -80
+; CHECK-NEXT:    ld s0, 64(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 72(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 80
+; CHECK-NEXT:    ret
+  %v8 = alloca <vscale x 8 x i64>
+  %v7 = alloca <vscale x 8 x i64>
+  %v6 = alloca <vscale x 4 x i64>
+  %v5 = alloca <vscale x 4 x i64>
+  %v4 = alloca <vscale x 2 x i64>
+  %v3 = alloca <vscale x 2 x i64>
+  %v2 = alloca <vscale x 1 x i64>
+  %v1 = alloca <vscale x 1 x i64>
+  ret void
+}
+
+define void @masks() nounwind optnone noinline {
+; CHECK-LABEL: masks:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -48
+; CHECK-NEXT:    sd ra, 40(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 32(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    addi s0, sp, 48
+; CHECK-NEXT:    csrr a0, vlenb
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -24(s0)
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -32(s0)
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -40(s0)
+; CHECK-NEXT:    sub sp, sp, a0
+; CHECK-NEXT:    sd sp, -48(s0)
+; CHECK-NEXT:    addi sp, s0, -48
+; CHECK-NEXT:    ld s0, 32(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld ra, 40(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 48
+; CHECK-NEXT:    ret
+  %v1 = alloca <vscale x 1 x i1>
+  %v2 = alloca <vscale x 2 x i1>
+  %v4 = alloca <vscale x 4 x i1>
+  %v8 = alloca <vscale x 8 x i1>
+  ret void
+}

llvm/test/CodeGen/RISCV/rvv/localvar.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv64 -mattr=+experimental-v < %s \
+; RUN:   | FileCheck %s -check-prefix=RV64IV
+
+define void @local_var_mf8() {
+; RV64IV-LABEL: local_var_mf8:
+; RV64IV:       # %bb.0:
+; RV64IV-NEXT:    addi sp, sp, -32
+; RV64IV-NEXT:    .cfi_def_cfa_offset 32
+; RV64IV-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    .cfi_offset ra, -8
+; RV64IV-NEXT:    .cfi_offset s0, -16
+; RV64IV-NEXT:    addi s0, sp, 32
+; RV64IV-NEXT:    .cfi_def_cfa s0, 0
+; RV64IV-NEXT:    csrr a0, vlenb
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -24(s0)
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -32(s0)
+; RV64IV-NEXT:    vsetvli a0, zero, e8,mf8,ta,mu
+; RV64IV-NEXT:    ld a0, -24(s0)
+; RV64IV-NEXT:    vle8.v v25, (a0)
+; RV64IV-NEXT:    ld a0, -32(s0)
+; RV64IV-NEXT:    vle8.v v25, (a0)
+; RV64IV-NEXT:    addi sp, s0, -32
+; RV64IV-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    addi sp, sp, 32
+; RV64IV-NEXT:    ret
+  %local0 = alloca <vscale x 1 x i8>
+  %local1 = alloca <vscale x 1 x i8>
+  load volatile <vscale x 1 x i8>, <vscale x 1 x i8>* %local0
+  load volatile <vscale x 1 x i8>, <vscale x 1 x i8>* %local1
+  ret void
+}
+
+define void @local_var_m1() {
+; RV64IV-LABEL: local_var_m1:
+; RV64IV:       # %bb.0:
+; RV64IV-NEXT:    addi sp, sp, -32
+; RV64IV-NEXT:    .cfi_def_cfa_offset 32
+; RV64IV-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    .cfi_offset ra, -8
+; RV64IV-NEXT:    .cfi_offset s0, -16
+; RV64IV-NEXT:    addi s0, sp, 32
+; RV64IV-NEXT:    .cfi_def_cfa s0, 0
+; RV64IV-NEXT:    csrr a0, vlenb
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -24(s0)
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -32(s0)
+; RV64IV-NEXT:    vsetvli a0, zero, e8,m1,ta,mu
+; RV64IV-NEXT:    ld a0, -24(s0)
+; RV64IV-NEXT:    vle8.v v25, (a0)
+; RV64IV-NEXT:    ld a0, -32(s0)
+; RV64IV-NEXT:    vle8.v v25, (a0)
+; RV64IV-NEXT:    addi sp, s0, -32
+; RV64IV-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    addi sp, sp, 32
+; RV64IV-NEXT:    ret
+  %local0 = alloca <vscale x 8 x i8>
+  %local1 = alloca <vscale x 8 x i8>
+  load volatile <vscale x 8 x i8>, <vscale x 8 x i8>* %local0
+  load volatile <vscale x 8 x i8>, <vscale x 8 x i8>* %local1
+  ret void
+}
+
+define void @local_var_m2() {
+; RV64IV-LABEL: local_var_m2:
+; RV64IV:       # %bb.0:
+; RV64IV-NEXT:    addi sp, sp, -32
+; RV64IV-NEXT:    .cfi_def_cfa_offset 32
+; RV64IV-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    .cfi_offset ra, -8
+; RV64IV-NEXT:    .cfi_offset s0, -16
+; RV64IV-NEXT:    addi s0, sp, 32
+; RV64IV-NEXT:    .cfi_def_cfa s0, 0
+; RV64IV-NEXT:    csrr a0, vlenb
+; RV64IV-NEXT:    slli a0, a0, 1
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -24(s0)
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -32(s0)
+; RV64IV-NEXT:    vsetvli a0, zero, e8,m2,ta,mu
+; RV64IV-NEXT:    ld a0, -24(s0)
+; RV64IV-NEXT:    vle8.v v26, (a0)
+; RV64IV-NEXT:    ld a0, -32(s0)
+; RV64IV-NEXT:    vle8.v v26, (a0)
+; RV64IV-NEXT:    addi sp, s0, -32
+; RV64IV-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    addi sp, sp, 32
+; RV64IV-NEXT:    ret
+  %local0 = alloca <vscale x 16 x i8>
+  %local1 = alloca <vscale x 16 x i8>
+  load volatile <vscale x 16 x i8>, <vscale x 16 x i8>* %local0
+  load volatile <vscale x 16 x i8>, <vscale x 16 x i8>* %local1
+  ret void
+}
+
+define void @local_var_m4() {
+; RV64IV-LABEL: local_var_m4:
+; RV64IV:       # %bb.0:
+; RV64IV-NEXT:    addi sp, sp, -32
+; RV64IV-NEXT:    .cfi_def_cfa_offset 32
+; RV64IV-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    .cfi_offset ra, -8
+; RV64IV-NEXT:    .cfi_offset s0, -16
+; RV64IV-NEXT:    addi s0, sp, 32
+; RV64IV-NEXT:    .cfi_def_cfa s0, 0
+; RV64IV-NEXT:    csrr a0, vlenb
+; RV64IV-NEXT:    slli a0, a0, 2
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -24(s0)
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -32(s0)
+; RV64IV-NEXT:    vsetvli a0, zero, e8,m4,ta,mu
+; RV64IV-NEXT:    ld a0, -24(s0)
+; RV64IV-NEXT:    vle8.v v28, (a0)
+; RV64IV-NEXT:    ld a0, -32(s0)
+; RV64IV-NEXT:    vle8.v v28, (a0)
+; RV64IV-NEXT:    addi sp, s0, -32
+; RV64IV-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    addi sp, sp, 32
+; RV64IV-NEXT:    ret
+  %local0 = alloca <vscale x 32 x i8>
+  %local1 = alloca <vscale x 32 x i8>
+  load volatile <vscale x 32 x i8>, <vscale x 32 x i8>* %local0
+  load volatile <vscale x 32 x i8>, <vscale x 32 x i8>* %local1
+  ret void
+}
+
+define void @local_var_m8() {
+; RV64IV-LABEL: local_var_m8:
+; RV64IV:       # %bb.0:
+; RV64IV-NEXT:    addi sp, sp, -32
+; RV64IV-NEXT:    .cfi_def_cfa_offset 32
+; RV64IV-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    .cfi_offset ra, -8
+; RV64IV-NEXT:    .cfi_offset s0, -16
+; RV64IV-NEXT:    addi s0, sp, 32
+; RV64IV-NEXT:    .cfi_def_cfa s0, 0
+; RV64IV-NEXT:    csrr a0, vlenb
+; RV64IV-NEXT:    slli a0, a0, 3
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -24(s0)
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -32(s0)
+; RV64IV-NEXT:    vsetvli a0, zero, e8,m8,ta,mu
+; RV64IV-NEXT:    ld a0, -24(s0)
+; RV64IV-NEXT:    vle8.v v8, (a0)
+; RV64IV-NEXT:    ld a0, -32(s0)
+; RV64IV-NEXT:    vle8.v v8, (a0)
+; RV64IV-NEXT:    addi sp, s0, -32
+; RV64IV-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    addi sp, sp, 32
+; RV64IV-NEXT:    ret
+  %local0 = alloca <vscale x 64 x i8>
+  %local1 = alloca <vscale x 64 x i8>
+  load volatile <vscale x 64 x i8>, <vscale x 64 x i8>* %local0
+  load volatile <vscale x 64 x i8>, <vscale x 64 x i8>* %local1
+  ret void
+}
+
+define void @local_var_m2_mix_local_scalar() {
+; RV64IV-LABEL: local_var_m2_mix_local_scalar:
+; RV64IV:       # %bb.0:
+; RV64IV-NEXT:    addi sp, sp, -48
+; RV64IV-NEXT:    .cfi_def_cfa_offset 48
+; RV64IV-NEXT:    sd ra, 40(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    sd s0, 32(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    .cfi_offset ra, -8
+; RV64IV-NEXT:    .cfi_offset s0, -16
+; RV64IV-NEXT:    addi s0, sp, 48
+; RV64IV-NEXT:    .cfi_def_cfa s0, 0
+; RV64IV-NEXT:    csrr a0, vlenb
+; RV64IV-NEXT:    slli a0, a0, 1
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -32(s0)
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -40(s0)
+; RV64IV-NEXT:    lw a0, -20(s0)
+; RV64IV-NEXT:    vsetvli a0, zero, e8,m2,ta,mu
+; RV64IV-NEXT:    ld a0, -32(s0)
+; RV64IV-NEXT:    vle8.v v26, (a0)
+; RV64IV-NEXT:    ld a0, -40(s0)
+; RV64IV-NEXT:    vle8.v v26, (a0)
+; RV64IV-NEXT:    lw a0, -24(s0)
+; RV64IV-NEXT:    addi sp, s0, -48
+; RV64IV-NEXT:    ld s0, 32(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    ld ra, 40(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    addi sp, sp, 48
+; RV64IV-NEXT:    ret
+  %local_scalar0 = alloca i32
+  %local0 = alloca <vscale x 16 x i8>
+  %local1 = alloca <vscale x 16 x i8>
+  %local_scalar1 = alloca i32
+  load volatile i32, i32* %local_scalar0
+  load volatile <vscale x 16 x i8>, <vscale x 16 x i8>* %local0
+  load volatile <vscale x 16 x i8>, <vscale x 16 x i8>* %local1
+  load volatile i32, i32* %local_scalar1
+  ret void
+}

llvm/test/CodeGen/RISCV/rvv/memory-args-with-scalar.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv64 -mattr=+experimental-v -O2 < %s \
+; RUN:   | FileCheck %s -check-prefix=RV64IV
+
+declare <vscale x 64 x i8> @llvm.riscv.vadd.nxv64i8.nxv64i8(
+  <vscale x 64 x i8>,
+  <vscale x 64 x i8>,
+  i64);
+
+define <vscale x 64 x i8> @callee(<vscale x 64 x i8> %arg0, <vscale x 64 x i8> %arg1, <vscale x 64 x i8> %arg2, i64 %vl, i64 %a0, i64 %a1, i64 %a2, i64 %a3, i64 %a4, i64 %a5, i64 %a6, i64 %a7) {
+; RV64IV-LABEL: callee:
+; RV64IV:       # %bb.0:
+; RV64IV-NEXT:    vsetvli t0, zero, e8,m8,ta,mu
+; RV64IV-NEXT:    vle8.v v8, (a1)
+; RV64IV-NEXT:    vle8.v v24, (a0)
+; RV64IV-NEXT:    ld t0, 16(sp)
+; RV64IV-NEXT:    ld a1, 0(sp)
+; RV64IV-NEXT:    ld a0, 8(sp)
+; RV64IV-NEXT:    add a3, a3, a4
+; RV64IV-NEXT:    add a4, a5, a6
+; RV64IV-NEXT:    add a1, a7, a1
+; RV64IV-NEXT:    add a0, a0, t0
+; RV64IV-NEXT:    add a3, a3, a4
+; RV64IV-NEXT:    add a0, a1, a0
+; RV64IV-NEXT:    add a0, a3, a0
+; RV64IV-NEXT:    add a0, a2, a0
+; RV64IV-NEXT:    vsetvli a0, a0, e8,m8,ta,mu
+; RV64IV-NEXT:    vadd.vv v16, v16, v24
+; RV64IV-NEXT:    vadd.vv v16, v16, v8
+; RV64IV-NEXT:    ret
+  %localsum0 = add i64 %a0, %a1
+  %localsum1 = add i64 %a2, %a3
+  %localsum2 = add i64 %a4, %a5
+  %localsum3 = add i64 %a6, %a7
+  %localsum4 = add i64 %localsum0, %localsum1
+  %localsum5 = add i64 %localsum2, %localsum3
+  %localsum6 = add i64 %localsum4, %localsum5
+  %avl = add i64 %vl, %localsum6
+  %tmp = call <vscale x 64 x i8> @llvm.riscv.vadd.nxv64i8.nxv64i8(
+                                  <vscale x 64 x i8> %arg0,
+                                  <vscale x 64 x i8> %arg1, i64 %avl)
+  %ret = call <vscale x 64 x i8> @llvm.riscv.vadd.nxv64i8.nxv64i8(
+                                  <vscale x 64 x i8> %tmp,
+                                  <vscale x 64 x i8> %arg2, i64 %avl)
+  ret <vscale x 64 x i8> %ret
+}
+
+define <vscale x 64 x i8> @caller() {
+; RV64IV-LABEL: caller:
+; RV64IV:       # %bb.0:
+; RV64IV-NEXT:    addi sp, sp, -128
+; RV64IV-NEXT:    .cfi_def_cfa_offset 128
+; RV64IV-NEXT:    sd ra, 120(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    sd s0, 112(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    .cfi_offset ra, -8
+; RV64IV-NEXT:    .cfi_offset s0, -16
+; RV64IV-NEXT:    addi s0, sp, 128
+; RV64IV-NEXT:    .cfi_def_cfa s0, 0
+; RV64IV-NEXT:    csrr a0, vlenb
+; RV64IV-NEXT:    slli a0, a0, 3
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -96(s0)
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -104(s0)
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -112(s0)
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -120(s0)
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -128(s0)
+; RV64IV-NEXT:    vsetvli a0, zero, e8,m8,ta,mu
+; RV64IV-NEXT:    ld a0, -96(s0)
+; RV64IV-NEXT:    vle8.v v16, (a0)
+; RV64IV-NEXT:    ld a0, -104(s0)
+; RV64IV-NEXT:    vle8.v v8, (a0)
+; RV64IV-NEXT:    ld a0, -112(s0)
+; RV64IV-NEXT:    vle8.v v24, (a0)
+; RV64IV-NEXT:    ld a2, -24(s0)
+; RV64IV-NEXT:    ld a3, -32(s0)
+; RV64IV-NEXT:    ld a4, -40(s0)
+; RV64IV-NEXT:    ld a5, -48(s0)
+; RV64IV-NEXT:    ld a6, -56(s0)
+; RV64IV-NEXT:    ld a7, -64(s0)
+; RV64IV-NEXT:    ld t0, -72(s0)
+; RV64IV-NEXT:    ld a1, -80(s0)
+; RV64IV-NEXT:    ld a0, -88(s0)
+; RV64IV-NEXT:    addi sp, sp, -32
+; RV64IV-NEXT:    sd a0, 16(sp)
+; RV64IV-NEXT:    sd a1, 8(sp)
+; RV64IV-NEXT:    sd t0, 0(sp)
+; RV64IV-NEXT:    ld a0, -128(s0)
+; RV64IV-NEXT:    vse8.v v24, (a0)
+; RV64IV-NEXT:    addi a0, s0, -120
+; RV64IV-NEXT:    addi a1, s0, -128
+; RV64IV-NEXT:    ld t0, -120(s0)
+; RV64IV-NEXT:    vse8.v v8, (t0)
+; RV64IV-NEXT:    call callee
+; RV64IV-NEXT:    addi sp, sp, 32
+; RV64IV-NEXT:    addi sp, s0, -128
+; RV64IV-NEXT:    ld s0, 112(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    ld ra, 120(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    addi sp, sp, 128
+; RV64IV-NEXT:    ret
+  %local0 = alloca <vscale x 64 x i8>
+  %local1 = alloca <vscale x 64 x i8>
+  %local2 = alloca <vscale x 64 x i8>
+  %local3 = alloca i64
+  %a0_addr = alloca i64
+  %a1_addr = alloca i64
+  %a2_addr = alloca i64
+  %a3_addr = alloca i64
+  %a4_addr = alloca i64
+  %a5_addr = alloca i64
+  %a6_addr = alloca i64
+  %a7_addr = alloca i64
+  %arg0 = load volatile <vscale x 64 x i8>, <vscale x 64 x i8>* %local0
+  %arg1 = load volatile <vscale x 64 x i8>, <vscale x 64 x i8>* %local1
+  %arg2 = load volatile <vscale x 64 x i8>, <vscale x 64 x i8>* %local2
+  %vl = load volatile i64, i64* %local3
+  %a0 = load volatile i64, i64* %a0_addr
+  %a1 = load volatile i64, i64* %a1_addr
+  %a2 = load volatile i64, i64* %a2_addr
+  %a3 = load volatile i64, i64* %a3_addr
+  %a4 = load volatile i64, i64* %a4_addr
+  %a5 = load volatile i64, i64* %a5_addr
+  %a6 = load volatile i64, i64* %a6_addr
+  %a7 = load volatile i64, i64* %a7_addr
+  %ret = call <vscale x 64 x i8> @callee(<vscale x 64 x i8> %arg0,
+                                         <vscale x 64 x i8> %arg1,
+                                         <vscale x 64 x i8> %arg2,
+                                         i64 %vl,
+                                         i64 %a0, i64 %a1, i64 %a2, i64 %a3,
+                                         i64 %a4, i64 %a5, i64 %a6, i64 %a7)
+  ret <vscale x 64 x i8> %ret
+}

llvm/test/CodeGen/RISCV/rvv/memory-args.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv64 -mattr=+experimental-v -O2 < %s \
+; RUN:   | FileCheck %s -check-prefix=RV64IV
+
+declare <vscale x 64 x i8> @llvm.riscv.vadd.nxv64i8.nxv64i8(
+  <vscale x 64 x i8>,
+  <vscale x 64 x i8>,
+  i64);
+
+define <vscale x 64 x i8> @callee(<vscale x 64 x i8> %arg0, <vscale x 64 x i8> %arg1) {
+; RV64IV-LABEL: callee:
+; RV64IV:       # %bb.0:
+; RV64IV-NEXT:    vsetvli a1, zero, e8,m8,ta,mu
+; RV64IV-NEXT:    vle8.v v8, (a0)
+; RV64IV-NEXT:    addi a0, zero, 1024
+; RV64IV-NEXT:    vsetvli a0, a0, e8,m8,ta,mu
+; RV64IV-NEXT:    vadd.vv v16, v16, v8
+; RV64IV-NEXT:    ret
+  %ret = call <vscale x 64 x i8> @llvm.riscv.vadd.nxv64i8.nxv64i8(
+                                  <vscale x 64 x i8> %arg0,
+                                  <vscale x 64 x i8> %arg1, i64 1024)
+  ret <vscale x 64 x i8> %ret
+}
+
+define <vscale x 64 x i8> @caller() {
+; RV64IV-LABEL: caller:
+; RV64IV:       # %bb.0:
+; RV64IV-NEXT:    addi sp, sp, -48
+; RV64IV-NEXT:    .cfi_def_cfa_offset 48
+; RV64IV-NEXT:    sd ra, 40(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    sd s0, 32(sp) # 8-byte Folded Spill
+; RV64IV-NEXT:    .cfi_offset ra, -8
+; RV64IV-NEXT:    .cfi_offset s0, -16
+; RV64IV-NEXT:    addi s0, sp, 48
+; RV64IV-NEXT:    .cfi_def_cfa s0, 0
+; RV64IV-NEXT:    csrr a0, vlenb
+; RV64IV-NEXT:    slli a0, a0, 3
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -24(s0)
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -32(s0)
+; RV64IV-NEXT:    sub sp, sp, a0
+; RV64IV-NEXT:    sd sp, -40(s0)
+; RV64IV-NEXT:    vsetvli a0, zero, e8,m8,ta,mu
+; RV64IV-NEXT:    ld a0, -24(s0)
+; RV64IV-NEXT:    vle8.v v16, (a0)
+; RV64IV-NEXT:    ld a0, -32(s0)
+; RV64IV-NEXT:    vle8.v v8, (a0)
+; RV64IV-NEXT:    addi a0, s0, -40
+; RV64IV-NEXT:    ld a1, -40(s0)
+; RV64IV-NEXT:    vse8.v v8, (a1)
+; RV64IV-NEXT:    call callee
+; RV64IV-NEXT:    addi sp, s0, -48
+; RV64IV-NEXT:    ld s0, 32(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    ld ra, 40(sp) # 8-byte Folded Reload
+; RV64IV-NEXT:    addi sp, sp, 48
+; RV64IV-NEXT:    ret
+  %local0 = alloca <vscale x 64 x i8>
+  %local1 = alloca <vscale x 64 x i8>
+  %arg0 = load volatile <vscale x 64 x i8>, <vscale x 64 x i8>* %local0
+  %arg1 = load volatile <vscale x 64 x i8>, <vscale x 64 x i8>* %local1
+  %ret = call <vscale x 64 x i8> @callee(<vscale x 64 x i8> %arg0,
+                                         <vscale x 64 x i8> %arg1)
+  ret <vscale x 64 x i8> %ret
+}

llvm/test/CodeGen/RISCV/stack-realignment.ll

; RV64I-NEXT:    ret
   %1 = alloca i8, align 256
   call void @callee(i8* %1)
   ret void
 }
 
 define void @caller512() nounwind {
 ; RV32I-LABEL: caller512:
 ; RV32I:       # %bb.0:
-; RV32I-NEXT:    addi sp, sp, -1536
-; RV32I-NEXT:    sw ra, 1532(sp) # 4-byte Folded Spill
-; RV32I-NEXT:    sw s0, 1528(sp) # 4-byte Folded Spill
-; RV32I-NEXT:    addi s0, sp, 1536
+; RV32I-NEXT:    addi sp, sp, -1024
+; RV32I-NEXT:    sw ra, 1020(sp) # 4-byte Folded Spill
+; RV32I-NEXT:    sw s0, 1016(sp) # 4-byte Folded Spill
+; RV32I-NEXT:    addi s0, sp, 1024
 ; RV32I-NEXT:    andi sp, sp, -512
-; RV32I-NEXT:    addi a0, sp, 1024
+; RV32I-NEXT:    addi a0, sp, 512
 ; RV32I-NEXT:    call callee@plt
-; RV32I-NEXT:    addi sp, s0, -1536
-; RV32I-NEXT:    lw s0, 1528(sp) # 4-byte Folded Reload
-; RV32I-NEXT:    lw ra, 1532(sp) # 4-byte Folded Reload
-; RV32I-NEXT:    addi sp, sp, 1536
+; RV32I-NEXT:    addi sp, s0, -1024
+; RV32I-NEXT:    lw s0, 1016(sp) # 4-byte Folded Reload
+; RV32I-NEXT:    lw ra, 1020(sp) # 4-byte Folded Reload
+; RV32I-NEXT:    addi sp, sp, 1024
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: caller512:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    addi sp, sp, -1536
-; RV64I-NEXT:    sd ra, 1528(sp) # 8-byte Folded Spill
-; RV64I-NEXT:    sd s0, 1520(sp) # 8-byte Folded Spill
-; RV64I-NEXT:    addi s0, sp, 1536
+; RV64I-NEXT:    addi sp, sp, -1024
+; RV64I-NEXT:    sd ra, 1016(sp) # 8-byte Folded Spill
+; RV64I-NEXT:    sd s0, 1008(sp) # 8-byte Folded Spill
+; RV64I-NEXT:    addi s0, sp, 1024
 ; RV64I-NEXT:    andi sp, sp, -512
-; RV64I-NEXT:    addi a0, sp, 1024
+; RV64I-NEXT:    addi a0, sp, 512
 ; RV64I-NEXT:    call callee@plt
-; RV64I-NEXT:    addi sp, s0, -1536
-; RV64I-NEXT:    ld s0, 1520(sp) # 8-byte Folded Reload
-; RV64I-NEXT:    ld ra, 1528(sp) # 8-byte Folded Reload
-; RV64I-NEXT:    addi sp, sp, 1536
+; RV64I-NEXT:    addi sp, s0, -1024
+; RV64I-NEXT:    ld s0, 1008(sp) # 8-byte Folded Reload
+; RV64I-NEXT:    ld ra, 1016(sp) # 8-byte Folded Reload
+; RV64I-NEXT:    addi sp, sp, 1024
 ; RV64I-NEXT:    ret
   %1 = alloca i8, align 512
   call void @callee(i8* %1)
   ret void
 }
 
 define void @caller_no_realign512() nounwind "no-realign-stack" {
 ; RV32I-LABEL: caller_no_realign512:

 
 define void @caller1024() nounwind {
 ; RV32I-LABEL: caller1024:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -2032
 ; RV32I-NEXT:    sw ra, 2028(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    sw s0, 2024(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    addi s0, sp, 2032
-; RV32I-NEXT:    addi sp, sp, -1040
+; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    andi sp, sp, -1024
-; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, -2048
-; RV32I-NEXT:    add a0, sp, a0
-; RV32I-NEXT:    mv a0, a0
+; RV32I-NEXT:    addi a0, sp, 1024
 ; RV32I-NEXT:    call callee@plt
-; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, -1024
-; RV32I-NEXT:    sub sp, s0, a0
-; RV32I-NEXT:    addi sp, sp, 1040
+; RV32I-NEXT:    addi sp, s0, -2048
+; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    lw s0, 2024(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    lw ra, 2028(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 2032
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: caller1024:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -2032
 ; RV64I-NEXT:    sd ra, 2024(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    sd s0, 2016(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    addi s0, sp, 2032
-; RV64I-NEXT:    addi sp, sp, -1040
+; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    andi sp, sp, -1024
-; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, -2048
-; RV64I-NEXT:    add a0, sp, a0
-; RV64I-NEXT:    mv a0, a0
+; RV64I-NEXT:    addi a0, sp, 1024
 ; RV64I-NEXT:    call callee@plt
-; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, -1024
-; RV64I-NEXT:    sub sp, s0, a0
-; RV64I-NEXT:    addi sp, sp, 1040
+; RV64I-NEXT:    addi sp, s0, -2048
+; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ld s0, 2016(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    ld ra, 2024(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 2032
 ; RV64I-NEXT:    ret
   %1 = alloca i8, align 1024
   call void @callee(i8* %1)
   ret void
 }

 define void @caller2048() nounwind {
 ; RV32I-LABEL: caller2048:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -2032
 ; RV32I-NEXT:    sw ra, 2028(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    sw s0, 2024(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    addi s0, sp, 2032
 ; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, 16
+; RV32I-NEXT:    addi a0, a0, -2032
 ; RV32I-NEXT:    sub sp, sp, a0
 ; RV32I-NEXT:    andi sp, sp, -2048
 ; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -2048
 ; RV32I-NEXT:    add a0, sp, a0
 ; RV32I-NEXT:    mv a0, a0
 ; RV32I-NEXT:    call callee@plt
-; RV32I-NEXT:    lui a0, 2
-; RV32I-NEXT:    addi a0, a0, -2048
+; RV32I-NEXT:    lui a0, 1
 ; RV32I-NEXT:    sub sp, s0, a0
 ; RV32I-NEXT:    lui a0, 1
-; RV32I-NEXT:    addi a0, a0, 16
+; RV32I-NEXT:    addi a0, a0, -2032
 ; RV32I-NEXT:    add sp, sp, a0
 ; RV32I-NEXT:    lw s0, 2024(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    lw ra, 2028(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 2032
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: caller2048:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -2032
 ; RV64I-NEXT:    sd ra, 2024(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    sd s0, 2016(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    addi s0, sp, 2032
 ; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, 16
+; RV64I-NEXT:    addiw a0, a0, -2032
 ; RV64I-NEXT:    sub sp, sp, a0
 ; RV64I-NEXT:    andi sp, sp, -2048
 ; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -2048
 ; RV64I-NEXT:    add a0, sp, a0
 ; RV64I-NEXT:    mv a0, a0
 ; RV64I-NEXT:    call callee@plt
-; RV64I-NEXT:    lui a0, 2
-; RV64I-NEXT:    addiw a0, a0, -2048
+; RV64I-NEXT:    lui a0, 1
 ; RV64I-NEXT:    sub sp, s0, a0
 ; RV64I-NEXT:    lui a0, 1
-; RV64I-NEXT:    addiw a0, a0, 16
+; RV64I-NEXT:    addiw a0, a0, -2032
 ; RV64I-NEXT:    add sp, sp, a0
 ; RV64I-NEXT:    ld s0, 2016(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    ld ra, 2024(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 2032
 ; RV64I-NEXT:    ret
   %1 = alloca i8, align 2048
   call void @callee(i8* %1)
   ret void

 
 define void @caller4096() nounwind {
 ; RV32I-LABEL: caller4096:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -2032
 ; RV32I-NEXT:    sw ra, 2028(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    sw s0, 2024(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    addi s0, sp, 2032
-; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    lui a0, 2
 ; RV32I-NEXT:    addi a0, a0, -2032
 ; RV32I-NEXT:    sub sp, sp, a0
 ; RV32I-NEXT:    srli a0, sp, 12
 ; RV32I-NEXT:    slli sp, a0, 12
-; RV32I-NEXT:    lui a0, 2
+; RV32I-NEXT:    lui a0, 1
 ; RV32I-NEXT:    add a0, sp, a0
 ; RV32I-NEXT:    mv a0, a0
 ; RV32I-NEXT:    call callee@plt
-; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    lui a0, 2
 ; RV32I-NEXT:    sub sp, s0, a0
-; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    lui a0, 2
 ; RV32I-NEXT:    addi a0, a0, -2032
 ; RV32I-NEXT:    add sp, sp, a0
 ; RV32I-NEXT:    lw s0, 2024(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    lw ra, 2028(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 2032
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: caller4096:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -2032
 ; RV64I-NEXT:    sd ra, 2024(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    sd s0, 2016(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    addi s0, sp, 2032
-; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    lui a0, 2
 ; RV64I-NEXT:    addiw a0, a0, -2032
 ; RV64I-NEXT:    sub sp, sp, a0
 ; RV64I-NEXT:    srli a0, sp, 12
 ; RV64I-NEXT:    slli sp, a0, 12
-; RV64I-NEXT:    lui a0, 2
+; RV64I-NEXT:    lui a0, 1
 ; RV64I-NEXT:    add a0, sp, a0
 ; RV64I-NEXT:    mv a0, a0
 ; RV64I-NEXT:    call callee@plt
-; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    lui a0, 2
 ; RV64I-NEXT:    sub sp, s0, a0
-; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    lui a0, 2
 ; RV64I-NEXT:    addiw a0, a0, -2032
 ; RV64I-NEXT:    add sp, sp, a0
 ; RV64I-NEXT:    ld s0, 2016(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    ld ra, 2024(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 2032
 ; RV64I-NEXT:    ret
   %1 = alloca i8, align 4096
   call void @callee(i8* %1)