[RISCV][WIP] Move VSPILL/VRELOAD expansion for vector tuples to eliminateFrameIndex.

[craig.topper]

Summary

We need a scratch GPR to increment the base pointer for each subsequent
register. We currently reuse the input GPR for the base pointer without
declaring it as a Def of the pseudo.

We can't add it as a Def of the pseudo at creation time because it doesn't
get register allocated. This was tried in D109405.

Seems the only choice we have is to scavenge the GPR. This patch
moves the expansion to eliminateFrameIndex where we can create
virtual registers that will be scavenged. This also eliminates the
extra operand for passing vlenb from frame lowering to expand pseudos.

I need to do more testing on real world code, but wanted to get this
up for early review.

I hope this will fix the issue reported in D123394, but I haven't
checked yet.

[reames]

llvm/lib/Target/RISCV/RISCVRegisterInfo.cpp

276

Missing {} will cause this to not compile without assertions. Same in the function below.

[reames]

On the surface, this seems not unreasonable. I think I'm missing a bit of context though, so may need to discuss offline before I can offer meaningful review.

[craig.topper]

llvm/lib/Target/RISCV/RISCVRegisterInfo.cpp

276

Why? I moved this from another function that didn't have braces and it has been compiling fine.

[reames]

llvm/lib/Target/RISCV/RISCVRegisterInfo.cpp

276

So, doesn't an assert expand to an empty string when assertions are enabled?

If so, we have a dangling else. The next statement effectively becomes the body of the else.

Looking at the code you copied from, this is really quite interesting as it looks like that code is just blatantly wrong. The key lowering loop would be conditional on the else. Do we have any testing for this? And if we do, does that test pass without asserts?

[craig.topper]

llvm/lib/Target/RISCV/RISCVRegisterInfo.cpp

276

Doesn't the ; that would still be there count as a statement for the else?

[reames]

llvm/lib/Target/RISCV/RISCVRegisterInfo.cpp

276

Yes it does. Apparently I need more coffee...

Sorry for wasting your time.

[reames]

LGTM w/minor comment.

llvm/lib/Target/RISCV/RISCVRegisterInfo.cpp

465–495

Can you add a comment above this? Something along the lines of:

Handle spill/fill of synthetic register classes for segment operations to ensure correctness in the edge case one gets spilled. There are many possible optimizations here, but given the extreme rarity of such spills, we prefer simplicity of implementation for now.

llvm/lib/Target/RISCV/RISCVExpandPseudoInsts.cpp

private:
   bool expandMBB(MachineBasicBlock &MBB);
   bool expandMI(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
                 MachineBasicBlock::iterator &NextMBBI);
   bool expandCCOp(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
                   MachineBasicBlock::iterator &NextMBBI);
   bool expandVSetVL(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI);
   bool expandVMSET_VMCLR(MachineBasicBlock &MBB,
                          MachineBasicBlock::iterator MBBI, unsigned Opcode);
-  bool expandVSPILL(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI);
-  bool expandVRELOAD(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI);
 };
 
 char RISCVExpandPseudo::ID = 0;
 
 bool RISCVExpandPseudo::runOnMachineFunction(MachineFunction &MF) {
   TII = static_cast<const RISCVInstrInfo *>(MF.getSubtarget().getInstrInfo());
   bool Modified = false;
   for (auto &MBB : MF)
bool RISCVExpandPseudo::expandMI(MachineBasicBlock &MBB,
   case RISCV::PseudoVMSET_M_B2:
   case RISCV::PseudoVMSET_M_B4:
   case RISCV::PseudoVMSET_M_B8:
   case RISCV::PseudoVMSET_M_B16:
   case RISCV::PseudoVMSET_M_B32:
   case RISCV::PseudoVMSET_M_B64:
     // vmset.m vd => vmxnor.mm vd, vd, vd
     return expandVMSET_VMCLR(MBB, MBBI, RISCV::VMXNOR_MM);
-  case RISCV::PseudoVSPILL2_M1:
-  case RISCV::PseudoVSPILL2_M2:
-  case RISCV::PseudoVSPILL2_M4:
-  case RISCV::PseudoVSPILL3_M1:
-  case RISCV::PseudoVSPILL3_M2:
-  case RISCV::PseudoVSPILL4_M1:
-  case RISCV::PseudoVSPILL4_M2:
-  case RISCV::PseudoVSPILL5_M1:
-  case RISCV::PseudoVSPILL6_M1:
-  case RISCV::PseudoVSPILL7_M1:
-  case RISCV::PseudoVSPILL8_M1:
-    return expandVSPILL(MBB, MBBI);
-  case RISCV::PseudoVRELOAD2_M1:
-  case RISCV::PseudoVRELOAD2_M2:
-  case RISCV::PseudoVRELOAD2_M4:
-  case RISCV::PseudoVRELOAD3_M1:
-  case RISCV::PseudoVRELOAD3_M2:
-  case RISCV::PseudoVRELOAD4_M1:
-  case RISCV::PseudoVRELOAD4_M2:
-  case RISCV::PseudoVRELOAD5_M1:
-  case RISCV::PseudoVRELOAD6_M1:
-  case RISCV::PseudoVRELOAD7_M1:
-  case RISCV::PseudoVRELOAD8_M1:
-    return expandVRELOAD(MBB, MBBI);
   }
 
   return false;
 }
 
 bool RISCVExpandPseudo::expandCCOp(MachineBasicBlock &MBB,
                                    MachineBasicBlock::iterator MBBI,
                                    MachineBasicBlock::iterator &NextMBBI) {
bool RISCVExpandPseudo::expandVMSET_VMCLR(MachineBasicBlock &MBB,
   const MCInstrDesc &Desc = TII->get(Opcode);
   BuildMI(MBB, MBBI, DL, Desc, DstReg)
       .addReg(DstReg, RegState::Undef)
       .addReg(DstReg, RegState::Undef);
   MBBI->eraseFromParent(); // The pseudo instruction is gone now.
   return true;
 }
 
-bool RISCVExpandPseudo::expandVSPILL(MachineBasicBlock &MBB,
-                                     MachineBasicBlock::iterator MBBI) {
-  const TargetRegisterInfo *TRI =
-      MBB.getParent()->getSubtarget().getRegisterInfo();
-  DebugLoc DL = MBBI->getDebugLoc();
-  Register SrcReg = MBBI->getOperand(0).getReg();
-  Register Base = MBBI->getOperand(1).getReg();
-  Register VL = MBBI->getOperand(2).getReg();
-  auto ZvlssegInfo = RISCV::isRVVSpillForZvlsseg(MBBI->getOpcode());
-  if (!ZvlssegInfo)
-    return false;
-  unsigned NF = ZvlssegInfo->first;
-  unsigned LMUL = ZvlssegInfo->second;
-  assert(NF * LMUL <= 8 && "Invalid NF/LMUL combinations.");
-  unsigned Opcode = RISCV::VS1R_V;
-  unsigned SubRegIdx = RISCV::sub_vrm1_0;
-  static_assert(RISCV::sub_vrm1_7 == RISCV::sub_vrm1_0 + 7,
-                "Unexpected subreg numbering");
-  if (LMUL == 2) {
-    Opcode = RISCV::VS2R_V;
-    SubRegIdx = RISCV::sub_vrm2_0;
-    static_assert(RISCV::sub_vrm2_3 == RISCV::sub_vrm2_0 + 3,
-                  "Unexpected subreg numbering");
-  } else if (LMUL == 4) {
-    Opcode = RISCV::VS4R_V;
-    SubRegIdx = RISCV::sub_vrm4_0;
-    static_assert(RISCV::sub_vrm4_1 == RISCV::sub_vrm4_0 + 1,
-                  "Unexpected subreg numbering");
-  } else
-    assert(LMUL == 1 && "LMUL must be 1, 2, or 4.");
-
-  for (unsigned I = 0; I < NF; ++I) {
-    // Adding implicit-use of super register to describe we are using part of
-    // super register, that prevents machine verifier complaining when part of
-    // subreg is undef, see comment in MachineVerifier::checkLiveness for more
-    // detail.
-    BuildMI(MBB, MBBI, DL, TII->get(Opcode))
-        .addReg(TRI->getSubReg(SrcReg, SubRegIdx + I))
-        .addReg(Base)
-        .addMemOperand(*(MBBI->memoperands_begin()))
-        .addReg(SrcReg, RegState::Implicit);
-    if (I != NF - 1)
-      BuildMI(MBB, MBBI, DL, TII->get(RISCV::ADD), Base)
-          .addReg(Base)
-          .addReg(VL);
-  }
-  MBBI->eraseFromParent();
-  return true;
-}
-
-bool RISCVExpandPseudo::expandVRELOAD(MachineBasicBlock &MBB,
-                                      MachineBasicBlock::iterator MBBI) {
-  const TargetRegisterInfo *TRI =
-      MBB.getParent()->getSubtarget().getRegisterInfo();
-  DebugLoc DL = MBBI->getDebugLoc();
-  Register DestReg = MBBI->getOperand(0).getReg();
-  Register Base = MBBI->getOperand(1).getReg();
-  Register VL = MBBI->getOperand(2).getReg();
-  auto ZvlssegInfo = RISCV::isRVVSpillForZvlsseg(MBBI->getOpcode());
-  if (!ZvlssegInfo)
-    return false;
-  unsigned NF = ZvlssegInfo->first;
-  unsigned LMUL = ZvlssegInfo->second;
-  assert(NF * LMUL <= 8 && "Invalid NF/LMUL combinations.");
-  unsigned Opcode = RISCV::VL1RE8_V;
-  unsigned SubRegIdx = RISCV::sub_vrm1_0;
-  static_assert(RISCV::sub_vrm1_7 == RISCV::sub_vrm1_0 + 7,
-                "Unexpected subreg numbering");
-  if (LMUL == 2) {
-    Opcode = RISCV::VL2RE8_V;
-    SubRegIdx = RISCV::sub_vrm2_0;
-    static_assert(RISCV::sub_vrm2_3 == RISCV::sub_vrm2_0 + 3,
-                  "Unexpected subreg numbering");
-  } else if (LMUL == 4) {
-    Opcode = RISCV::VL4RE8_V;
-    SubRegIdx = RISCV::sub_vrm4_0;
-    static_assert(RISCV::sub_vrm4_1 == RISCV::sub_vrm4_0 + 1,
-                  "Unexpected subreg numbering");
-  } else
-    assert(LMUL == 1 && "LMUL must be 1, 2, or 4.");
-
-  for (unsigned I = 0; I < NF; ++I) {
-    BuildMI(MBB, MBBI, DL, TII->get(Opcode),
-            TRI->getSubReg(DestReg, SubRegIdx + I))
-        .addReg(Base)
-        .addMemOperand(*(MBBI->memoperands_begin()));
-    if (I != NF - 1)
-      BuildMI(MBB, MBBI, DL, TII->get(RISCV::ADD), Base)
-          .addReg(Base)
-          .addReg(VL);
-  }
-  MBBI->eraseFromParent();
-  return true;
-}
-
 class RISCVPreRAExpandPseudo : public MachineFunctionPass {
 public:
   const RISCVInstrInfo *TII;
   static char ID;
 
   RISCVPreRAExpandPseudo() : MachineFunctionPass(ID) {
     initializeRISCVPreRAExpandPseudoPass(*PassRegistry::getPassRegistry());
   }

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

void RISCVInstrInfo::storeRegToStackSlot(MachineBasicBlock &MBB,
   if (I != MBB.end())
     DL = I->getDebugLoc();
 
   MachineFunction *MF = MBB.getParent();
   MachineFrameInfo &MFI = MF->getFrameInfo();
 
   unsigned Opcode;
   bool IsScalableVector = true;
-  bool IsZvlsseg = true;
   if (RISCV::GPRRegClass.hasSubClassEq(RC)) {
     Opcode = TRI->getRegSizeInBits(RISCV::GPRRegClass) == 32 ?
              RISCV::SW : RISCV::SD;
     IsScalableVector = false;
   } else if (RISCV::FPR16RegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::FSH;
     IsScalableVector = false;
   } else if (RISCV::FPR32RegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::FSW;
     IsScalableVector = false;
   } else if (RISCV::FPR64RegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::FSD;
     IsScalableVector = false;
   } else if (RISCV::VRRegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::PseudoVSPILL_M1;
-    IsZvlsseg = false;
   } else if (RISCV::VRM2RegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::PseudoVSPILL_M2;
-    IsZvlsseg = false;
   } else if (RISCV::VRM4RegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::PseudoVSPILL_M4;
-    IsZvlsseg = false;
   } else if (RISCV::VRM8RegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::PseudoVSPILL_M8;
-    IsZvlsseg = false;
   } else if (RISCV::VRN2M1RegClass.hasSubClassEq(RC))
     Opcode = RISCV::PseudoVSPILL2_M1;
   else if (RISCV::VRN2M2RegClass.hasSubClassEq(RC))
     Opcode = RISCV::PseudoVSPILL2_M2;
   else if (RISCV::VRN2M4RegClass.hasSubClassEq(RC))
     Opcode = RISCV::PseudoVSPILL2_M4;
   else if (RISCV::VRN3M1RegClass.hasSubClassEq(RC))
     Opcode = RISCV::PseudoVSPILL3_M1;
else
     llvm_unreachable("Can't store this register to stack slot");
 
   if (IsScalableVector) {
     MachineMemOperand *MMO = MF->getMachineMemOperand(
         MachinePointerInfo::getFixedStack(*MF, FI), MachineMemOperand::MOStore,
         MemoryLocation::UnknownSize, MFI.getObjectAlign(FI));
 
     MFI.setStackID(FI, TargetStackID::ScalableVector);
-    auto MIB = BuildMI(MBB, I, DL, get(Opcode))
-                   .addReg(SrcReg, getKillRegState(IsKill))
-                   .addFrameIndex(FI)
-                   .addMemOperand(MMO);
-    if (IsZvlsseg) {
-      // For spilling/reloading Zvlsseg registers, append the dummy field for
-      // the scaled vector length. The argument will be used when expanding
-      // these pseudo instructions.
-      MIB.addReg(RISCV::X0);
-    }
+    BuildMI(MBB, I, DL, get(Opcode))
   } else {
     MachineMemOperand *MMO = MF->getMachineMemOperand(
         MachinePointerInfo::getFixedStack(*MF, FI), MachineMemOperand::MOStore,
         MFI.getObjectSize(FI), MFI.getObjectAlign(FI));
 
     BuildMI(MBB, I, DL, get(Opcode))
         .addReg(SrcReg, getKillRegState(IsKill))
         .addFrameIndex(FI)
void RISCVInstrInfo::loadRegFromStackSlot(MachineBasicBlock &MBB,
   if (I != MBB.end())
     DL = I->getDebugLoc();
 
   MachineFunction *MF = MBB.getParent();
   MachineFrameInfo &MFI = MF->getFrameInfo();
 
   unsigned Opcode;
   bool IsScalableVector = true;
-  bool IsZvlsseg = true;
   if (RISCV::GPRRegClass.hasSubClassEq(RC)) {
     Opcode = TRI->getRegSizeInBits(RISCV::GPRRegClass) == 32 ?
              RISCV::LW : RISCV::LD;
     IsScalableVector = false;
   } else if (RISCV::FPR16RegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::FLH;
     IsScalableVector = false;
   } else if (RISCV::FPR32RegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::FLW;
     IsScalableVector = false;
   } else if (RISCV::FPR64RegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::FLD;
     IsScalableVector = false;
   } else if (RISCV::VRRegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::PseudoVRELOAD_M1;
-    IsZvlsseg = false;
   } else if (RISCV::VRM2RegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::PseudoVRELOAD_M2;
-    IsZvlsseg = false;
   } else if (RISCV::VRM4RegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::PseudoVRELOAD_M4;
-    IsZvlsseg = false;
   } else if (RISCV::VRM8RegClass.hasSubClassEq(RC)) {
     Opcode = RISCV::PseudoVRELOAD_M8;
-    IsZvlsseg = false;
   } else if (RISCV::VRN2M1RegClass.hasSubClassEq(RC))
     Opcode = RISCV::PseudoVRELOAD2_M1;
   else if (RISCV::VRN2M2RegClass.hasSubClassEq(RC))
     Opcode = RISCV::PseudoVRELOAD2_M2;
   else if (RISCV::VRN2M4RegClass.hasSubClassEq(RC))
     Opcode = RISCV::PseudoVRELOAD2_M4;
   else if (RISCV::VRN3M1RegClass.hasSubClassEq(RC))
     Opcode = RISCV::PseudoVRELOAD3_M1;
else
     llvm_unreachable("Can't load this register from stack slot");
 
   if (IsScalableVector) {
     MachineMemOperand *MMO = MF->getMachineMemOperand(
         MachinePointerInfo::getFixedStack(*MF, FI), MachineMemOperand::MOLoad,
         MemoryLocation::UnknownSize, MFI.getObjectAlign(FI));
 
     MFI.setStackID(FI, TargetStackID::ScalableVector);
-    auto MIB = BuildMI(MBB, I, DL, get(Opcode), DstReg)
-                   .addFrameIndex(FI)
-                   .addMemOperand(MMO);
-    if (IsZvlsseg) {
-      // For spilling/reloading Zvlsseg registers, append the dummy field for
-      // the scaled vector length. The argument will be used when expanding
-      // these pseudo instructions.
-      MIB.addReg(RISCV::X0);
-    }
+    BuildMI(MBB, I, DL, get(Opcode), DstReg)
   } else {
     MachineMemOperand *MMO = MF->getMachineMemOperand(
         MachinePointerInfo::getFixedStack(*MF, FI), MachineMemOperand::MOLoad,
         MFI.getObjectSize(FI), MFI.getObjectAlign(FI));
 
     BuildMI(MBB, I, DL, get(Opcode), DstReg)
         .addFrameIndex(FI)
         .addImm(0)

llvm/lib/Target/RISCV/RISCVInstrInfoVPseudos.td

 }
 
 foreach lmul = MxList in {
   foreach nf = NFSet<lmul>.L in {
     defvar vreg = SegRegClass<lmul, nf>.RC;
     let hasSideEffects = 0, mayLoad = 0, mayStore = 1, isCodeGenOnly = 1,
         Size = !mul(4, !sub(!mul(nf, 2), 1)) in {
       def "PseudoVSPILL" # nf # "_" # lmul.MX :
-        Pseudo<(outs), (ins vreg:$rs1, GPR:$rs2, GPR:$vlenb), []>;
+        Pseudo<(outs), (ins vreg:$rs1, GPR:$rs2), []>;
     }
     let hasSideEffects = 0, mayLoad = 1, mayStore = 0, isCodeGenOnly = 1,
         Size = !mul(4, !sub(!mul(nf, 2), 1)) in {
       def "PseudoVRELOAD" # nf # "_" # lmul.MX :
-        Pseudo<(outs vreg:$rs1), (ins GPR:$rs2, GPR:$vlenb), []>;
+        Pseudo<(outs vreg:$rs1), (ins GPR:$rs2), []>;
     }
   }
 }
 
 //===----------------------------------------------------------------------===//
 // 6. Configuration-Setting Instructions
 //===----------------------------------------------------------------------===//
 

llvm/lib/Target/RISCV/RISCVRegisterInfo.h

void adjustReg(MachineBasicBlock &MBB, MachineBasicBlock::iterator II,
                  const DebugLoc &DL, Register DestReg, Register SrcReg,
                  StackOffset Offset, MachineInstr::MIFlag Flag,
                  MaybeAlign RequiredAlign) const;
 
   bool eliminateFrameIndex(MachineBasicBlock::iterator MI, int SPAdj,
                            unsigned FIOperandNum,
                            RegScavenger *RS = nullptr) const override;
 
+  void lowerVSPILL(MachineBasicBlock::iterator II) const;
+  void lowerVRELOAD(MachineBasicBlock::iterator II) const;
+
   Register getFrameRegister(const MachineFunction &MF) const override;
 
   bool requiresRegisterScavenging(const MachineFunction &MF) const override {
     return true;
   }
 
   bool requiresFrameIndexScavenging(const MachineFunction &MF) const override {
     return true;

llvm/lib/Target/RISCV/RISCVRegisterInfo.cpp

void RISCVRegisterInfo::adjustReg(MachineBasicBlock &MBB,
   Register ScratchReg = MRI.createVirtualRegister(&RISCV::GPRRegClass);
   TII->movImm(MBB, II, DL, ScratchReg, Val, Flag);
   BuildMI(MBB, II, DL, TII->get(Opc), DestReg)
       .addReg(SrcReg, getKillRegState(KillSrcReg))
       .addReg(ScratchReg, RegState::Kill)
       .setMIFlag(Flag);
 }
 
+// Split a VSPILLx_Mx pseudo into multiple whole register stores separated by
+// LMUL*VLENB bytes.
+void RISCVRegisterInfo::lowerVSPILL(MachineBasicBlock::iterator II) const {
+  DebugLoc DL = II->getDebugLoc();
+  MachineBasicBlock &MBB = *II->getParent();
+  MachineFunction &MF = *MBB.getParent();
+  MachineRegisterInfo &MRI = MF.getRegInfo();
+  const TargetInstrInfo *TII = MF.getSubtarget().getInstrInfo();
+  const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
+
+  auto ZvlssegInfo = RISCV::isRVVSpillForZvlsseg(II->getOpcode());
+  unsigned NF = ZvlssegInfo->first;
+  unsigned LMUL = ZvlssegInfo->second;
+  assert(NF * LMUL <= 8 && "Invalid NF/LMUL combinations.");
+  unsigned Opcode, SubRegIdx;
+  switch (LMUL) {
+  default:
+    llvm_unreachable("LMUL must be 1, 2, or 4.");
+  case 1:
+    Opcode = RISCV::VS1R_V;
+    SubRegIdx = RISCV::sub_vrm1_0;
+    break;
+  case 2:
+    Opcode = RISCV::VS2R_V;
+    SubRegIdx = RISCV::sub_vrm2_0;
+    break;
+  case 4:
+    Opcode = RISCV::VS4R_V;
+    SubRegIdx = RISCV::sub_vrm4_0;
+    break;
+  }
+  static_assert(RISCV::sub_vrm1_7 == RISCV::sub_vrm1_0 + 7,
+                "Unexpected subreg numbering");
+  static_assert(RISCV::sub_vrm2_3 == RISCV::sub_vrm2_0 + 3,
+                "Unexpected subreg numbering");
+  static_assert(RISCV::sub_vrm4_1 == RISCV::sub_vrm4_0 + 1,
+                "Unexpected subreg numbering");
+
+  Register VL = MRI.createVirtualRegister(&RISCV::GPRRegClass);
+  BuildMI(MBB, II, DL, TII->get(RISCV::PseudoReadVLENB), VL);
+  uint32_t ShiftAmount = Log2_32(LMUL);
+  if (ShiftAmount != 0)
+    BuildMI(MBB, II, DL, TII->get(RISCV::SLLI), VL)
+        .addReg(VL)
+        .addImm(ShiftAmount);
+
+  Register SrcReg = II->getOperand(0).getReg();
+  Register Base = II->getOperand(1).getReg();
+  bool IsBaseKill = II->getOperand(1).isKill();
+  Register NewBase = MRI.createVirtualRegister(&RISCV::GPRRegClass);
+  for (unsigned I = 0; I < NF; ++I) {
+    // Adding implicit-use of super register to describe we are using part of
+    // super register, that prevents machine verifier complaining when part of
+    // subreg is undef, see comment in MachineVerifier::checkLiveness for more
+    // detail.
+    BuildMI(MBB, II, DL, TII->get(Opcode))
+        .addReg(TRI->getSubReg(SrcReg, SubRegIdx + I))
+        .addReg(Base, getKillRegState(I == NF - 1))
+        .addMemOperand(*(II->memoperands_begin()))
+        .addReg(SrcReg, RegState::Implicit);
+    if (I != NF - 1)
+      BuildMI(MBB, II, DL, TII->get(RISCV::ADD), NewBase)
+          .addReg(Base, getKillRegState(I != 0 || IsBaseKill))
+          .addReg(VL, getKillRegState(I == NF - 2));
+    Base = NewBase;
+  }
+  II->eraseFromParent();
+}
+
+// Split a VSPILLx_Mx pseudo into multiple whole register loads separated by
+// LMUL*VLENB bytes.
+void RISCVRegisterInfo::lowerVRELOAD(MachineBasicBlock::iterator II) const {
+  DebugLoc DL = II->getDebugLoc();
+  MachineBasicBlock &MBB = *II->getParent();
+  MachineFunction &MF = *MBB.getParent();
+  MachineRegisterInfo &MRI = MF.getRegInfo();
+  const TargetInstrInfo *TII = MF.getSubtarget().getInstrInfo();
+  const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
+
+  auto ZvlssegInfo = RISCV::isRVVSpillForZvlsseg(II->getOpcode());
+  unsigned NF = ZvlssegInfo->first;
+  unsigned LMUL = ZvlssegInfo->second;
+  assert(NF * LMUL <= 8 && "Invalid NF/LMUL combinations.");
+  unsigned Opcode, SubRegIdx;
+  switch (LMUL) {
+  default:
+    llvm_unreachable("LMUL must be 1, 2, or 4.");
+  case 1:
+    Opcode = RISCV::VL1RE8_V;
+    SubRegIdx = RISCV::sub_vrm1_0;
+    break;
+  case 2:
+    Opcode = RISCV::VL2RE8_V;
+    SubRegIdx = RISCV::sub_vrm2_0;
+    break;
+  case 4:
+    Opcode = RISCV::VL4RE8_V;
+    SubRegIdx = RISCV::sub_vrm4_0;
+    break;
+  }
+  static_assert(RISCV::sub_vrm1_7 == RISCV::sub_vrm1_0 + 7,
+                "Unexpected subreg numbering");
+  static_assert(RISCV::sub_vrm2_3 == RISCV::sub_vrm2_0 + 3,
+                "Unexpected subreg numbering");
+  static_assert(RISCV::sub_vrm4_1 == RISCV::sub_vrm4_0 + 1,
+                "Unexpected subreg numbering");
+
+  Register VL = MRI.createVirtualRegister(&RISCV::GPRRegClass);
+  BuildMI(MBB, II, DL, TII->get(RISCV::PseudoReadVLENB), VL);
+  uint32_t ShiftAmount = Log2_32(LMUL);
+  if (ShiftAmount != 0)
+    BuildMI(MBB, II, DL, TII->get(RISCV::SLLI), VL)
+        .addReg(VL)
+        .addImm(ShiftAmount);
+
+  Register DestReg = II->getOperand(0).getReg();
+  Register Base = II->getOperand(1).getReg();
+  bool IsBaseKill = II->getOperand(1).isKill();
+  Register NewBase = MRI.createVirtualRegister(&RISCV::GPRRegClass);
+  for (unsigned I = 0; I < NF; ++I) {
+    BuildMI(MBB, II, DL, TII->get(Opcode),
+            TRI->getSubReg(DestReg, SubRegIdx + I))
+        .addReg(Base, getKillRegState(I == NF - 1))
+        .addMemOperand(*(II->memoperands_begin()));
+    if (I != NF - 1)
+      BuildMI(MBB, II, DL, TII->get(RISCV::ADD), NewBase)
+          .addReg(Base, getKillRegState(I != 0 || IsBaseKill))
+          .addReg(VL, getKillRegState(I == NF - 2));
+    Base = NewBase;
+  }
+  II->eraseFromParent();
+}
 
 bool RISCVRegisterInfo::eliminateFrameIndex(MachineBasicBlock::iterator II,
                                             int SPAdj, unsigned FIOperandNum,
                                             RegScavenger *RS) const {
   assert(SPAdj == 0 && "Unexpected non-zero SPAdj value");
 
   MachineInstr &MI = *II;
   MachineFunction &MF = *MI.getParent()->getParent();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   const RISCVSubtarget &ST = MF.getSubtarget<RISCVSubtarget>();
-  const RISCVInstrInfo *TII = ST.getInstrInfo();
   DebugLoc DL = MI.getDebugLoc();
 
   int FrameIndex = MI.getOperand(FIOperandNum).getIndex();
   Register FrameReg;
   StackOffset Offset =
       getFrameLowering(MF)->getFrameIndexReference(MF, FrameIndex, FrameReg);
   bool IsRVVSpill = RISCV::isRVVSpill(MI);
   if (!IsRVVSpill)
bool RISCVRegisterInfo::eliminateFrameIndex(MachineBasicBlock::iterator II,
   // If after materializing the adjustment, we have a pointless ADDI, remove it
   if (MI.getOpcode() == RISCV::ADDI &&
       MI.getOperand(0).getReg() == MI.getOperand(1).getReg() &&
       MI.getOperand(2).getImm() == 0) {
     MI.eraseFromParent();
     return true;
   }
 
-  auto ZvlssegInfo = RISCV::isRVVSpillForZvlsseg(MI.getOpcode());
-  if (ZvlssegInfo) {
-    MachineBasicBlock &MBB = *MI.getParent();
-    Register VL = MRI.createVirtualRegister(&RISCV::GPRRegClass);
-    BuildMI(MBB, II, DL, TII->get(RISCV::PseudoReadVLENB), VL);
-    uint32_t ShiftAmount = Log2_32(ZvlssegInfo->second);
-    if (ShiftAmount != 0)
-      BuildMI(MBB, II, DL, TII->get(RISCV::SLLI), VL)
-          .addReg(VL)
-          .addImm(ShiftAmount);
-    // The last argument of pseudo spilling opcode for zvlsseg is the length of
-    // one element of zvlsseg types. For example, for vint32m2x2_t, it will be
-    // the length of vint32m2_t.
-    MI.getOperand(FIOperandNum + 1).ChangeToRegister(VL, /*isDef=*/false);
+  // Handle spill/fill of synthetic register classes for segment operations to
+  // ensure correctness in the edge case one gets spilled. There are many
+  // possible optimizations here, but given the extreme rarity of such spills,
+  // we prefer simplicity of implementation for now.
+  switch (MI.getOpcode()) {
+  case RISCV::PseudoVSPILL2_M1:
+  case RISCV::PseudoVSPILL2_M2:
+  case RISCV::PseudoVSPILL2_M4:
+  case RISCV::PseudoVSPILL3_M1:
+  case RISCV::PseudoVSPILL3_M2:
+  case RISCV::PseudoVSPILL4_M1:
+  case RISCV::PseudoVSPILL4_M2:
+  case RISCV::PseudoVSPILL5_M1:
+  case RISCV::PseudoVSPILL6_M1:
+  case RISCV::PseudoVSPILL7_M1:
+  case RISCV::PseudoVSPILL8_M1:
+    lowerVSPILL(II);
+    return true;
+  case RISCV::PseudoVRELOAD2_M1:
+  case RISCV::PseudoVRELOAD2_M2:
+  case RISCV::PseudoVRELOAD2_M4:
+  case RISCV::PseudoVRELOAD3_M1:
+  case RISCV::PseudoVRELOAD3_M2:
+  case RISCV::PseudoVRELOAD4_M1:
+  case RISCV::PseudoVRELOAD4_M2:
+  case RISCV::PseudoVRELOAD5_M1:
+  case RISCV::PseudoVRELOAD6_M1:
+  case RISCV::PseudoVRELOAD7_M1:
+  case RISCV::PseudoVRELOAD8_M1:
+    lowerVRELOAD(II);
+    return true;
   }
+
   return false;
 }
 
 Register RISCVRegisterInfo::getFrameRegister(const MachineFunction &MF) const {
   const TargetFrameLowering *TFI = getFrameLowering(MF);
   return TFI->hasFP(MF) ? RISCV::X8 : RISCV::X2;
 }
 

llvm/test/CodeGen/RISCV/rvv/undef-subreg-range.mir

-# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
-# RUN: llc %s -O2 -mtriple riscv64 -riscv-enable-subreg-liveness \
-# RUN:     -verify-machineinstrs -run-pass=riscv-expand-pseudo -o - 2>&1 \
-# RUN:     | FileCheck %s
---- |
-  define void @foo() #0 {
-  entry:
-    ret void
-  }
-...
----
-name:            foo
-alignment:       4
-tracksRegLiveness: true
-fixedStack:      []
-stack:
-  - { id: 0, name: '', type: spill-slot, offset: 0, size: 32, alignment: 8,
-      stack-id: scalable-vector, callee-saved-register: '', callee-saved-restored: true }
-body:             |
-  bb.0.entry:
-    liveins: $v8m2, $x10, $x11
-    ; CHECK-LABEL: name: foo
-    ; CHECK: liveins: $v8m2, $x10, $x11
-    ; CHECK-NEXT: {{  $}}
-    ; CHECK-NEXT: VS2R_V $v8m2, $x10, implicit $v8m2_v10m2 :: (store unknown-size into %stack.0, align 8)
-    ; CHECK-NEXT: $x10 = ADD $x10, $x11
-    ; CHECK-NEXT: VS2R_V $v10m2, $x10, implicit $v8m2_v10m2 :: (store unknown-size into %stack.0, align 8)
-    ; CHECK-NEXT: PseudoRET
-    PseudoVSPILL2_M2 killed $v8m2_v10m2, killed $x10, killed $x11 :: (store unknown-size into %stack.0, align 8)
-    PseudoRET
-
-...

llvm/test/CodeGen/RISCV/rvv/vmv-copy.mir

bb.0:
     ; CHECK-NEXT: $v12m2 = PseudoVMV2R_V $v28m2
     $x15 = PseudoVSETIVLI 4, 73, implicit-def $vl, implicit-def $vtype
     $v26m2 = PseudoVLE16_V_M2 killed $x16, $noreg, 4, implicit $vl, implicit $vtype
     $v8m2 = PseudoVLE16_V_M2 killed $x17, $noreg, 4, implicit $vl, implicit $vtype
     $v28m4 = PseudoVWADD_VV_M2 $v26m2, $v8m2, $noreg, 4, implicit $vl, implicit $vtype
     $v12m2 = COPY $v28m2
 ...
 ---
-name: copy_from_reload
-tracksRegLiveness: true
-body:             |
-  bb.0:
-    liveins: $x14, $x16, $x17
-    ; 73 = e16,m2
-    ; CHECK-LABEL: name: copy_from_reload
-    ; CHECK: liveins: $x14, $x16, $x17
-    ; CHECK-NEXT: {{  $}}
-    ; CHECK-NEXT: $x15 = PseudoVSETVLI $x14, 73 /* e16, m2, ta, mu */, implicit-def $vl, implicit-def $vtype
-    ; CHECK-NEXT: $v2m2 = PseudoVLE16_V_M2 killed $x16, $noreg, 4 /* e16 */, implicit $vl, implicit $vtype
-    ; CHECK-NEXT: $x12 = PseudoReadVLENB
-    ; CHECK-NEXT: $x12 = SLLI $x12, 1
-    ; CHECK-NEXT: $v2m2_v4m2 = PseudoVRELOAD2_M2 killed $x17, killed $x12
-    ; CHECK-NEXT: $v12m2 = PseudoVMV2R_V $v2m2
-    $x15 = PseudoVSETVLI $x14, 73, implicit-def $vl, implicit-def $vtype
-    $v2m2 = PseudoVLE16_V_M2 killed $x16, $noreg, 4, implicit $vl, implicit $vtype
-    $x12 = PseudoReadVLENB
-    $x12 = SLLI $x12, 1
-    $v2m2_v4m2 = PseudoVRELOAD2_M2 killed $x17, killed $x12
-    $v12m2 = COPY $v2m2
-...
----
 name: copy_with_different_vlmax
 tracksRegLiveness: true
 body:             |
   bb.0:
     liveins: $x14, $x16
     ; 82 = e32,m4
     ; 74 = e16,m4
     ; CHECK-LABEL: name: copy_with_different_vlmax

llvm/test/CodeGen/RISCV/rvv/zvlsseg-spill.mir

bb.0:
     ; CHECK-NEXT: $x12 = frame-setup PseudoReadVLENB
     ; CHECK-NEXT: $x12 = frame-setup SLLI killed $x12, 3
     ; CHECK-NEXT: $x2 = frame-setup SUB $x2, killed $x12
     ; CHECK-NEXT: frame-setup CFI_INSTRUCTION escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x08, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22
     ; CHECK-NEXT: dead $x0 = PseudoVSETVLI killed renamable $x11, 216 /* e64, m1, ta, ma */, implicit-def $vl, implicit-def $vtype
     ; CHECK-NEXT: $v0_v1_v2_v3_v4_v5_v6 = PseudoVLSEG7E64_V_M1 renamable $x10, $noreg, 6 /* e64 */, implicit $vl, implicit $vtype
     ; CHECK-NEXT: $x11 = ADDI $x2, 16
     ; CHECK-NEXT: $x12 = PseudoReadVLENB
-    ; CHECK-NEXT: PseudoVSPILL7_M1 killed renamable $v0_v1_v2_v3_v4_v5_v6, killed $x11, killed $x12
+    ; CHECK-NEXT: VS1R_V $v0, $x11, implicit $v0_v1_v2_v3_v4_v5_v6 :: (store unknown-size into %stack.0, align 8)
+    ; CHECK-NEXT: $x11 = ADD killed $x11, $x12
+    ; CHECK-NEXT: VS1R_V $v1, $x11, implicit $v0_v1_v2_v3_v4_v5_v6 :: (store unknown-size into %stack.0, align 8)
+    ; CHECK-NEXT: $x11 = ADD killed $x11, $x12
+    ; CHECK-NEXT: VS1R_V $v2, $x11, implicit $v0_v1_v2_v3_v4_v5_v6 :: (store unknown-size into %stack.0, align 8)
+    ; CHECK-NEXT: $x11 = ADD killed $x11, $x12
+    ; CHECK-NEXT: VS1R_V $v3, $x11, implicit $v0_v1_v2_v3_v4_v5_v6 :: (store unknown-size into %stack.0, align 8)
+    ; CHECK-NEXT: $x11 = ADD killed $x11, $x12
+    ; CHECK-NEXT: VS1R_V $v4, $x11, implicit $v0_v1_v2_v3_v4_v5_v6 :: (store unknown-size into %stack.0, align 8)
+    ; CHECK-NEXT: $x11 = ADD killed $x11, $x12
+    ; CHECK-NEXT: VS1R_V $v5, $x11, implicit $v0_v1_v2_v3_v4_v5_v6 :: (store unknown-size into %stack.0, align 8)
+    ; CHECK-NEXT: $x11 = ADD killed $x11, killed $x12
+    ; CHECK-NEXT: VS1R_V $v6, killed $x11, implicit $v0_v1_v2_v3_v4_v5_v6 :: (store unknown-size into %stack.0, align 8)
     ; CHECK-NEXT: $x11 = ADDI $x2, 16
     ; CHECK-NEXT: $x12 = PseudoReadVLENB
-    ; CHECK-NEXT: dead renamable $v7_v8_v9_v10_v11_v12_v13 = PseudoVRELOAD7_M1 killed $x11, killed $x12, implicit-def $v8
+    ; CHECK-NEXT: $v7 = VL1RE8_V $x11 :: (load unknown-size from %stack.0, align 8)
+    ; CHECK-NEXT: $x11 = ADD killed $x11, $x12
+    ; CHECK-NEXT: $v8 = VL1RE8_V $x11 :: (load unknown-size from %stack.0, align 8)
+    ; CHECK-NEXT: $x11 = ADD killed $x11, $x12
+    ; CHECK-NEXT: $v9 = VL1RE8_V $x11 :: (load unknown-size from %stack.0, align 8)
+    ; CHECK-NEXT: $x11 = ADD killed $x11, $x12
+    ; CHECK-NEXT: $v10 = VL1RE8_V $x11 :: (load unknown-size from %stack.0, align 8)
+    ; CHECK-NEXT: $x11 = ADD killed $x11, $x12
+    ; CHECK-NEXT: $v11 = VL1RE8_V $x11 :: (load unknown-size from %stack.0, align 8)
+    ; CHECK-NEXT: $x11 = ADD killed $x11, $x12
+    ; CHECK-NEXT: $v12 = VL1RE8_V $x11 :: (load unknown-size from %stack.0, align 8)
+    ; CHECK-NEXT: $x11 = ADD killed $x11, killed $x12
+    ; CHECK-NEXT: $v13 = VL1RE8_V killed $x11 :: (load unknown-size from %stack.0, align 8)
     ; CHECK-NEXT: VS1R_V killed $v8, killed renamable $x10
     ; CHECK-NEXT: $x10 = frame-destroy PseudoReadVLENB
     ; CHECK-NEXT: $x10 = frame-destroy SLLI killed $x10, 3
     ; CHECK-NEXT: $x2 = frame-destroy ADD $x2, killed $x10
     ; CHECK-NEXT: $x2 = frame-destroy ADDI $x2, 16
     ; CHECK-NEXT: PseudoRET
     %0:gpr = COPY $x10
     %1:gprnox0 = COPY $x11
     $v0_v1_v2_v3_v4_v5_v6 = PseudoVLSEG7E64_V_M1 %0, %1, 6
-    PseudoVSPILL7_M1 killed renamable $v0_v1_v2_v3_v4_v5_v6, %stack.0, $x0
-    renamable $v7_v8_v9_v10_v11_v12_v13 = PseudoVRELOAD7_M1 %stack.0, $x0
+    PseudoVSPILL7_M1 killed renamable $v0_v1_v2_v3_v4_v5_v6, %stack.0 :: (store unknown-size into %stack.0, align 8)
+    renamable $v7_v8_v9_v10_v11_v12_v13 = PseudoVRELOAD7_M1 %stack.0 :: (load unknown-size from %stack.0, align 8)
     VS1R_V killed $v8, %0:gpr
     PseudoRET
 ...