[AArch64] Stackframe accesses to SVE objects.

[sdesmalen]

Summary

Materialize accesses to SVE frame objects from SP or FP, whichever is
available and beneficial.

This patch still assumes the objects are pre-allocated. The automatic
layout of SVE objects within the stackframe will be added in a separate
patch.

[cameron.mcinally]

lib/Target/AArch64/AArch64InstrInfo.cpp

3366 ↗

(On Diff #220822)

I'm not an LLVM coding standards expert, but does this need an llvm_unreachable()? I think it does...

3453 ↗

(On Diff #220822)

Would you shed some light on what this change is doing?

IsMulVL indicates there are scalable objects on the stack, right? What is the reason for the behavior change of the legacy code when !IsMulVL. I.e. the addition of StackOffset(SOffset.getScalableBytes(), MVT::nxv1i8) in the else block.

[efriedma]

lib/Target/AArch64/AArch64InstrInfo.cpp

3366 ↗

(On Diff #220822)

The reason we add llvm_unreachable() after switches in some cases is related to warnings. Some compilers warn about a missing return after a switch that covers every named enum value, but not every possible enum value. That case doesn't apply here: the switch has a "default".

[sdesmalen]

lib/Target/AArch64/AArch64InstrInfo.cpp

3453 ↗

(On Diff #220822)

isAArch64FrameOffsetLegal tries to determine if the immediate field of the instruction can fit the given StackOffset, and will attempt to fold as much of the offset in the immediate.

Although a StackOffset can contain both a scalable and a non-scalable part, it will depend on the instruction whether the immediate is scalable or non-scalable. For example, in

LDR <Zt>, [<Xn|SP>{, #<imm>, MUL VL}]

the immediate is mul vl, so scalable, which means the instruction can only handle the "scalable" part of the StackOffset. The rest of the offset will need to be handled elsewhere.

The variable int64_t Offset uses either the scalable or non-scalable part of the StackOffset, which happens on line 3411. After that, this function does its magic to determine what part of the offset can be folded into the immediate.

On line 3448, the remaining part of the offset that could *not* be folded into the immediate will need to be reflected in the in/out parameter SOffset, which is a StackOffset.
If IsMulVL is true, then variable Offset is scalable and will at this point contain the part of the scalable offset that could not be folded into the immediate.
SOffset.getBytes() just passes through the fixed-size part of the offset that is not handled by the instruction.
Conversely, if IsMulVL is false, then the variable Offset is non-scalable and will contain the part of the fixed-size offset that could not be folded into the immediate. It then has to pass through SOffset.getScalableBytes() that is not handled by the instruction, so it can be handled elsewhere.

For example:

isAArch64FrameOffsetLegal(AArch64::LDR_ZXI, {16, MVT::i8} + {16, MVT::nxv1i8})

would fold {16, MVT::nxv1i8} into the immediate, and the resulting SOffset would be {16, MVT::i8}. It would return AArch64FrameOffsetCanUpdate.

isAArch64FrameOffsetLegal(AArch64::LDR_ZXI, {16, MVT::i8} + {4096, MVT::nxv1i8})

would fold {4080, MVT::nxv1i8} into the immediate (note that it's immediate goes up to #255 MUL VL), and the resulting SOffset would be {16, MVT::i8} + {16, MVT::nxv1i8}.

[cameron.mcinally]

lib/Target/AArch64/AArch64InstrInfo.cpp

3366 ↗

(On Diff #220822)

Ah, ok. I thought that it was protection in case the the switch is changed in the future. The X86 backend has a few switches with default cases that also have unreachable after them.

[cameron.mcinally]

LGTM with a moderate confidence level. Maybe give other reviewers a day or so to comment before committing.

lib/Target/AArch64/AArch64InstrInfo.cpp

3453 ↗

(On Diff #220822)

Ah, okay. So it's determining the addressing mode...

[sdesmalen]

In D67749#1699760, @cameron.mcinally wrote:

LGTM with a moderate confidence level. Maybe give other reviewers a day or so to comment before committing.

Thanks @cameron.mcinally!

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

const AArch64FrameLowering *TFI =
          MF.getSubtarget<AArch64Subtarget>().getFrameLowering();
 
      // IRG does not allow immediate offset. getTaggedBasePointerOffset should
      // almost always point to SP-after-prologue; if not, emit a longer
      // instruction sequence.
      int BaseOffset = -AFI->getTaggedBasePointerOffset();
      unsigned FrameReg;
      StackOffset FrameRegOffset = TFI->resolveFrameOffsetReference(
-         MF, BaseOffset, false /*isFixed*/, FrameReg,
+         MF, BaseOffset, false /*isFixed*/, false /*isSVE*/, FrameReg,
          /*PreferFP=*/false,
          /*ForSimm=*/true);
      Register SrcReg = FrameReg;
      if (FrameRegOffset) {
        // Use output register as temporary.
        SrcReg = MI.getOperand(0).getReg();
        emitFrameOffset(MBB, &MI, MI.getDebugLoc(), SrcReg, FrameReg,
                        FrameRegOffset, TII);

llvm/lib/Target/AArch64/AArch64FrameLowering.h

public:
 
   int getFrameIndexReference(const MachineFunction &MF, int FI,
                              unsigned &FrameReg) const override;
   StackOffset resolveFrameIndexReference(const MachineFunction &MF, int FI,
                                          unsigned &FrameReg, bool PreferFP,
                                          bool ForSimm) const;
   StackOffset resolveFrameOffsetReference(const MachineFunction &MF,
                                           int ObjectOffset, bool isFixed,
-                                          unsigned &FrameReg, bool PreferFP,
-                                          bool ForSimm) const;
+                                          bool isSVE, unsigned &FrameReg,
+                                          bool PreferFP, bool ForSimm) const;
   bool spillCalleeSavedRegisters(MachineBasicBlock &MBB,
                                  MachineBasicBlock::iterator MI,
                                  const std::vector<CalleeSavedInfo> &CSI,
                                  const TargetRegisterInfo *TRI) const override;
 
   bool restoreCalleeSavedRegisters(MachineBasicBlock &MBB,
                                   MachineBasicBlock::iterator MI,
                                   std::vector<CalleeSavedInfo> &CSI,
bool isSupportedStackID(TargetStackID::Value ID) const override {
     case TargetStackID::NoAlloc:
       return true;
     }
   }
 
 private:
   bool shouldCombineCSRLocalStackBump(MachineFunction &MF,
                                       unsigned StackBumpBytes) const;
+  int64_t determineSVEStackSize(MachineFrameInfo &MF, unsigned &MaxAlign) const;
 };
 
 } // End llvm namespace
 
 #endif

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

 }
 
 StackOffset AArch64FrameLowering::resolveFrameIndexReference(
     const MachineFunction &MF, int FI, unsigned &FrameReg, bool PreferFP,
     bool ForSimm) const {
   const auto &MFI = MF.getFrameInfo();
   int ObjectOffset = MFI.getObjectOffset(FI);
   bool isFixed = MFI.isFixedObjectIndex(FI);
-  return resolveFrameOffsetReference(MF, ObjectOffset, isFixed, FrameReg,
+  bool isSVE = MFI.getStackID(FI) == TargetStackID::SVEVector;
+  return resolveFrameOffsetReference(MF, ObjectOffset, isFixed, isSVE, FrameReg,
                                      PreferFP, ForSimm);
 }
 
 StackOffset AArch64FrameLowering::resolveFrameOffsetReference(
-    const MachineFunction &MF, int ObjectOffset, bool isFixed,
+    const MachineFunction &MF, int ObjectOffset, bool isFixed, bool isSVE,
     unsigned &FrameReg, bool PreferFP, bool ForSimm) const {
   const auto &MFI = MF.getFrameInfo();
   const auto *RegInfo = static_cast<const AArch64RegisterInfo *>(
       MF.getSubtarget().getRegisterInfo());
   const auto *AFI = MF.getInfo<AArch64FunctionInfo>();
   const auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
 
   int FPOffset = getFPOffset(MF, ObjectOffset).getBytes();
   int Offset = getStackOffset(MF, ObjectOffset).getBytes();
   bool isCSR =
       !isFixed && ObjectOffset >= -((int)AFI->getCalleeSavedStackSize());
 
   const StackOffset &SVEStackSize = getSVEStackSize(MF);
-  if (SVEStackSize)
-    llvm_unreachable("Accessing frame indices in presence of SVE "
-                     "not yet supported");
 
   // Use frame pointer to reference fixed objects. Use it for locals if
   // there are VLAs or a dynamically realigned SP (and thus the SP isn't
   // reliable as a base). Make sure useFPForScavengingIndex() does the
   // right thing for the emergency spill slot.
   bool UseFP = false;
-  if (AFI->hasStackFrame()) {
+  if (AFI->hasStackFrame() && !isSVE) {
+    // We shouldn't prefer using the FP when there is an SVE area
+    // in between the FP and the non-SVE locals/spills.
+    PreferFP &= !SVEStackSize;
+
     // Note: Keeping the following as multiple 'if' statements rather than
     // merging to a single expression for readability.
     //
     // Argument access should always use the FP.
     if (isFixed) {
       UseFP = hasFP(MF);
     } else if (isCSR && RegInfo->needsStackRealignment(MF)) {
       // References to the CSR area must use FP if we're re-aligning the stack
if (isFixed) {
 
       if (MFI.hasVarSizedObjects()) {
         // If we have variable sized objects, we can use either FP or BP, as the
         // SP offset is unknown. We can use the base pointer if we have one and
         // FP is not preferred. If not, we're stuck with using FP.
         bool CanUseBP = RegInfo->hasBasePointer(MF);
         if (FPOffsetFits && CanUseBP) // Both are ok. Pick the best.
           UseFP = PreferFP;
-        else if (!CanUseBP) // Can't use BP. Forced to use FP.
+        else if (!CanUseBP) { // Can't use BP. Forced to use FP.
+          assert(!SVEStackSize && "Expected BP to be available");
           UseFP = true;
+        }
         // else we can use BP and FP, but the offset from FP won't fit.
         // That will make us scavenge registers which we can probably avoid by
         // using BP. If it won't fit for BP either, we'll scavenge anyway.
       } else if (FPOffset >= 0) {
         // Use SP or FP, whichever gives us the best chance of the offset
         // being in range for direct access. If the FPOffset is positive,
         // that'll always be best, as the SP will be even further away.
         UseFP = true;
if (isFixed) {
       }
     }
   }
 
   assert(((isFixed || isCSR) || !RegInfo->needsStackRealignment(MF) || !UseFP) &&
          "In the presence of dynamic stack pointer realignment, "
          "non-argument/CSR objects cannot be accessed through the frame pointer");
 
+  if (isSVE) {
+    int64_t OffsetToSVEArea =
+        MFI.getStackSize() - AFI->getCalleeSavedStackSize();
+    StackOffset FPOffset = {ObjectOffset, MVT::nxv1i8};
+    StackOffset SPOffset = SVEStackSize +
+                           StackOffset(ObjectOffset, MVT::nxv1i8) +
+                           StackOffset(OffsetToSVEArea, MVT::i8);
+    // Always use the FP for SVE spills if available and beneficial.
+    if (hasFP(MF) &&
+        (SPOffset.getBytes() ||
+         FPOffset.getScalableBytes() < SPOffset.getScalableBytes() ||
+         RegInfo->needsStackRealignment(MF))) {
+      FrameReg = RegInfo->getFrameRegister(MF);
+      return FPOffset;
+    }
+
+    FrameReg = RegInfo->hasBasePointer(MF) ? RegInfo->getBaseRegister()
+                                           : (unsigned)AArch64::SP;
+    return SPOffset;
+  }
+
+  StackOffset ScalableOffset = {};
+  if (UseFP && !(isFixed || isCSR))
+    ScalableOffset = -SVEStackSize;
+  if (!UseFP && (isFixed || isCSR))
+    ScalableOffset = SVEStackSize;
+
   if (UseFP) {
     FrameReg = RegInfo->getFrameRegister(MF);
-    return StackOffset(FPOffset, MVT::i8);
+    return StackOffset(FPOffset, MVT::i8) + ScalableOffset;
   }
 
   // Use the base pointer if we have one.
   if (RegInfo->hasBasePointer(MF))
     FrameReg = RegInfo->getBaseRegister();
   else {
     assert(!MFI.hasVarSizedObjects() &&
            "Can't use SP when we have var sized objects.");
     FrameReg = AArch64::SP;
     // If we're using the red zone for this function, the SP won't actually
     // be adjusted, so the offsets will be negative. They're also all
     // within range of the signed 9-bit immediate instructions.
     if (canUseRedZone(MF))
       Offset -= AFI->getLocalStackSize();
   }
 
-  return StackOffset(Offset, MVT::i8);
+  return StackOffset(Offset, MVT::i8) + ScalableOffset;
 }
 
 static unsigned getPrologueDeath(MachineFunction &MF, unsigned Reg) {
   // Do not set a kill flag on values that are also marked as live-in. This
   // happens with the @llvm-returnaddress intrinsic and with arguments passed in
   // callee saved registers.
   // Omitting the kill flags is conservatively correct even if the live-in
   // is not used after all.
void AArch64FrameLowering::determineCalleeSaves(MachineFunction &MF,
   }
 
   LLVM_DEBUG(dbgs() << "*** determineCalleeSaves\nSaved CSRs:";
              for (unsigned Reg
                   : SavedRegs.set_bits()) dbgs()
              << ' ' << printReg(Reg, RegInfo);
              dbgs() << "\n";);
 
-  bool HasSVEStackObjects = [&MFI]() {
-    for (int I = MFI.getObjectIndexBegin(); I != 0; ++I)
-      if (MFI.getStackID(I) == TargetStackID::SVEVector &&
-          MFI.getObjectOffset(I) < 0)
-        return true;
-    // Note: We don't take allocatable stack objects into
-    // account yet, because allocation for those is not yet
-    // implemented.
-    return false;
-  }();
-
   // If any callee-saved registers are used, the frame cannot be eliminated.
-  bool CanEliminateFrame = (SavedRegs.count() == 0) && !HasSVEStackObjects;
+  unsigned MaxAlign = getStackAlignment();
+  int64_t SVEStackSize =
+      alignTo(determineSVEStackSize(MFI, MaxAlign), MaxAlign);
+  assert(MaxAlign <= 16 && "Cannot align scalable vectors more than 16 bytes");
+  bool CanEliminateFrame = (SavedRegs.count() == 0) && !SVEStackSize;
 
   // The CSR spill slots have not been allocated yet, so estimateStackSize
   // won't include them.
   unsigned EstimatedStackSizeLimit = estimateRSStackSizeLimit(MF);
-  bool BigStack = (EstimatedStackSize + CSStackSize) > EstimatedStackSizeLimit;
+
+  // Conservatively always assume BigStack when there are SVE spills.
+  bool BigStack = SVEStackSize ||
+                  (EstimatedStackSize + CSStackSize) > EstimatedStackSizeLimit;
   if (BigStack || !CanEliminateFrame || RegInfo->cannotEliminateFrame(MF))
     AFI->setHasStackFrame(true);
 
   // Estimate if we might need to scavenge a register at some point in order
   // to materialize a stack offset. If so, either spill one additional
   // callee-saved register or reserve a special spill slot to facilitate
   // register scavenging. If we already spilled an extra callee-saved register
   // above to keep the number of spills even, we don't need to do anything else

 }
 
 bool AArch64FrameLowering::enableStackSlotScavenging(
     const MachineFunction &MF) const {
   const AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   return AFI->hasCalleeSaveStackFreeSpace();
 }
 
-void AArch64FrameLowering::processFunctionBeforeFrameFinalized(
-    MachineFunction &MF, RegScavenger *RS) const {
-  MachineFrameInfo &MFI = MF.getFrameInfo();
-
-  assert(getStackGrowthDirection() == TargetFrameLowering::StackGrowsDown &&
-         "Upwards growing stack unsupported");
-
-  // Process all fixed stack SVE objects.
+int64_t AArch64FrameLowering::determineSVEStackSize(MachineFrameInfo &MFI,
+                                                    unsigned &MaxAlign) const {
+  // Process all fixed stack objects.
   int64_t Offset = 0;
-  for (int I = MFI.getObjectIndexBegin(); I != 0; ++I) {
-    unsigned StackID = MFI.getStackID(I);
-    if (StackID == TargetStackID::SVEVector) {
+  for (int I = MFI.getObjectIndexBegin(); I != 0; ++I)
+    if (MFI.getStackID(I) == TargetStackID::SVEVector) {
       int64_t FixedOffset = -MFI.getObjectOffset(I);
       if (FixedOffset > Offset)
         Offset = FixedOffset;
     }
-  }
+
+  // Note: We don't take allocatable stack objects into
+  // account yet, because allocation for those is not yet
+  // implemented.
+  return Offset;
 
+void AArch64FrameLowering::processFunctionBeforeFrameFinalized(
+    MachineFunction &MF, RegScavenger *RS) const {
+  MachineFrameInfo &MFI = MF.getFrameInfo();
+
+  assert(getStackGrowthDirection() == TargetFrameLowering::StackGrowsDown &&
+         "Upwards growing stack unsupported");
+
   unsigned MaxAlign = getStackAlignment();
-  uint64_t SVEStackSize = alignTo(Offset, MaxAlign);
+  int64_t SVEStackSize = determineSVEStackSize(MFI, MaxAlign);
 
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
-  AFI->setStackSizeSVE(SVEStackSize);
+  AFI->setStackSizeSVE(alignTo(SVEStackSize, MaxAlign));
   assert(MaxAlign <= 16 && "Cannot align scalable vectors more than 16 bytes");
 
   // If this function isn't doing Win64-style C++ EH, we don't need to do
   // anything.
   if (!MF.hasEHFunclets())
     return;
   const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
   WinEHFuncInfo &EHInfo = *MF.getWinEHFuncInfo();

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

case AArch64::TAGPstack:
     break;
   case AArch64::LDG:
   case AArch64::STGOffset:
   case AArch64::STZGOffset:
     Scale = Width = 16;
     MinOffset = -256;
     MaxOffset = 255;
     break;
+  case AArch64::LDR_PXI:
+  case AArch64::STR_PXI:
+    Scale = Width = 2;
+    MinOffset = -256;
+    MaxOffset = 255;
+    break;
+  case AArch64::LDR_ZXI:
+  case AArch64::STR_ZXI:
+    Scale = Width = 16;
+    MinOffset = -256;
+    MaxOffset = 255;
+    break;
   case AArch64::ST2GOffset:
   case AArch64::STZ2GOffset:
     Scale = 16;
     Width = 32;
     MinOffset = -256;
     MaxOffset = 255;
     break;
   case AArch64::STGPi:
if (IsFill && SrcMO.getSubReg() == 0 && DstMO.isUndef()) {
       }
     }
   }
 
   // Cannot fold.
   return nullptr;
 }
 
+static bool isSVEScaledImmInstruction(unsigned Opcode) {
+  switch (Opcode) {
+  case AArch64::LDR_ZXI:
+  case AArch64::STR_ZXI:
+  case AArch64::LDR_PXI:
+  case AArch64::STR_PXI:
+    return true;
+  default:
+    return false;
+  }
+}
+
 int llvm::isAArch64FrameOffsetLegal(const MachineInstr &MI,
                                     StackOffset &SOffset,
                                     bool *OutUseUnscaledOp,
                                     unsigned *OutUnscaledOp,
                                     int *EmittableOffset) {
   // Set output values in case of early exit.
   if (EmittableOffset)
     *EmittableOffset = 0;
int llvm::isAArch64FrameOffsetLegal(const MachineInstr &MI,
   // Get the min/max offset and the scale.
   unsigned Scale, Width;
   int64_t MinOff, MaxOff;
   if (!AArch64InstrInfo::getMemOpInfo(MI.getOpcode(), Scale, Width, MinOff,
                                       MaxOff))
     llvm_unreachable("unhandled opcode in isAArch64FrameOffsetLegal");
 
   // Construct the complete offset.
+  bool IsMulVL = isSVEScaledImmInstruction(MI.getOpcode());
+  int64_t Offset =
+      IsMulVL ? (SOffset.getScalableBytes()) : (SOffset.getBytes());
+
   const MachineOperand &ImmOpnd =
       MI.getOperand(AArch64InstrInfo::getLoadStoreImmIdx(MI.getOpcode()));
-  int Offset = SOffset.getBytes() + ImmOpnd.getImm() * Scale;
+  Offset += ImmOpnd.getImm() * Scale;
 
   // If the offset doesn't match the scale, we rewrite the instruction to
   // use the unscaled instruction instead. Likewise, if we have a negative
   // offset and there is an unscaled op to use.
   Optional<unsigned> UnscaledOp =
       AArch64InstrInfo::getUnscaledLdSt(MI.getOpcode());
   bool useUnscaledOp = UnscaledOp && (Offset % Scale || Offset < 0);
   if (useUnscaledOp &&
int llvm::isAArch64FrameOffsetLegal(const MachineInstr &MI,
 
   if (EmittableOffset)
     *EmittableOffset = NewOffset;
   if (OutUseUnscaledOp)
     *OutUseUnscaledOp = useUnscaledOp;
   if (OutUnscaledOp && UnscaledOp)
     *OutUnscaledOp = *UnscaledOp;
 
-  SOffset = StackOffset(Offset, MVT::i8);
+  if (IsMulVL)
+    SOffset = StackOffset(Offset, MVT::nxv1i8) +
+              StackOffset(SOffset.getBytes(), MVT::i8);
+  else
+    SOffset = StackOffset(Offset, MVT::i8) +
+              StackOffset(SOffset.getScalableBytes(), MVT::nxv1i8);
   return AArch64FrameOffsetCanUpdate |
-         (Offset == 0 ? AArch64FrameOffsetIsLegal : 0);
+         (SOffset ? 0 : AArch64FrameOffsetIsLegal);
 }
 
 bool llvm::rewriteAArch64FrameIndex(MachineInstr &MI, unsigned FrameRegIdx,
                                     unsigned FrameReg, StackOffset &Offset,
                                     const AArch64InstrInfo *TII) {
   unsigned Opcode = MI.getOpcode();
   unsigned ImmIdx = FrameRegIdx + 1;
 

llvm/test/CodeGen/AArch64/framelayout-sve.mir

 #     |     :       |
 #     +-------------+ <- SP after call and frame-setup
 #
 --- |
 
   define void @test_allocate_sve() nounwind { entry: unreachable }
   define void @test_allocate_sve_gpr_callee_saves() nounwind { entry: unreachable }
   define void @test_allocate_sve_gpr_realigned() nounwind { entry: unreachable }
+  define void @test_address_sve() nounwind { entry: unreachable }
+  define void @test_address_sve_fp() nounwind { entry: unreachable }
+  define void @test_stack_arg_sve() nounwind { entry: unreachable }
+  define void @test_address_sve_out_of_range() nounwind { entry: unreachable }
 
 ...
 # +----------+
+# |scratchreg|  // x29 is used as scratch reg.
+# +----------+
 # | %fixed-  |  // scalable SVE object of n * 18 bytes, aligned to 16 bytes,
 # |  stack.0 |  // to be materialized with 2*ADDVL (<=> 2 * n * 16bytes)
 # +----------+
 # | %stack.0 |  // not scalable
 # +----------+ <- SP
 
 # CHECK-LABEL: name: test_allocate_sve
-# CHECK:       stackSize: 16
+# CHECK:       stackSize: 32
 
 # CHECK:      bb.0.entry:
+# CHECK-NEXT: $sp = frame-setup STRXpre killed $[[SCRATCH:[a-z0-9]+]], $sp, -16
 # CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -2
 # CHECK-NEXT: $sp = frame-setup SUBXri $sp, 16, 0
 
 # CHECK-NEXT: $sp = frame-destroy ADDVL_XXI $sp, 2
 # CHECK-NEXT: $sp = frame-destroy ADDXri $sp, 16, 0
+# CHECK-NEXT: $sp, $[[SCRATCH]] = frame-destroy LDRXpost $sp, 16
 # CHECK-NEXT: RET_ReallyLR
 name:            test_allocate_sve
 fixedStack:
   - { id: 0, stack-id: sve-vec, size: 18, alignment: 2, offset: -18 }
 stack:
   - { id: 0, stack-id: default, size: 16, alignment: 8 }
 body:             |
   bb.0.entry:
     RET_ReallyLR
 ---
 ...
 # +----------+
 # | x20, x21 |  // callee saves
+# |scratchreg|  // x29 is used as scratch reg.
 # +----------+
 # | %fixed-  |  // scalable objects
 # |  stack.0 |
 # +----------+
 # | %stack.0 |  // not scalable
 # +----------+ <- SP
 
 # CHECK-LABEL: name: test_allocate_sve_gpr_callee_saves
-# CHECK:       stackSize: 32
+# CHECK:       stackSize: 48
 
 # CHECK:      bb.0.entry:
-# CHECK-NEXT: $sp = frame-setup STPXpre killed $x21, killed $x20, $sp, -2
+# CHECK-NEXT: $sp = frame-setup STRXpre killed $[[SCRATCH:[a-z0-9]+]], $sp, -32
+# CHECK-NEXT: frame-setup STPXi killed $x21, killed $x20, $sp, 2
 # CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -2
 # CHECK-NEXT: $sp = frame-setup SUBXri $sp, 16, 0
 # CHECK-NEXT: $x20 = IMPLICIT_DEF
 # CHECK-NEXT: $x21 = IMPLICIT_DEF
 # CHECK-NEXT: $sp = frame-destroy ADDVL_XXI $sp, 2
 # CHECK-NEXT: $sp = frame-destroy ADDXri $sp, 16, 0
-# CHECK-NEXT: $sp, $x21, $x20 = frame-destroy LDPXpost $sp, 2
+# CHECK-NEXT: $x21, $x20 = frame-destroy LDPXi $sp, 2
+# CHECK-NEXT: $sp, $[[SCRATCH]] = frame-destroy LDRXpost $sp, 32
 # CHECK-NEXT: RET_ReallyLR
 name:            test_allocate_sve_gpr_callee_saves
 fixedStack:
   - { id: 0, stack-id: sve-vec, size: 18, alignment: 2, offset: -18 }
 stack:
   - { id: 0, stack-id: default, size: 16, alignment: 8 }
 body:             |
   bb.0.entry:

 fixedStack:
   - { id: 0, stack-id: sve-vec, size: 18, alignment: 2, offset: -18 }
 stack:
   - { id: 0, stack-id: default, size: 16, alignment: 32 }
 body:             |
   bb.0.entry:
     RET_ReallyLR
 ---
+...
+# +----------+
+# | x20, x21 |  // callee saves
+# +----------+
+# | %stack.0 |  // scalable @ SP + 16b + 32 scalable bytes
+# | %stack.1 |  // scalable @ SP + 16b + 16 scalable bytes
+# | %stack.2 |  // scalable @ SP + 16b + 14 scalable bytes
+# +----------+
+# | %stack.0 |  // not scalable
+# +----------+ <- SP
+
+# CHECK-LABEL: name: test_address_sve
+# CHECK:       stackSize: 32
+
+# CHECK:      bb.0.entry:
+# CHECK-NEXT: $sp = frame-setup STRXpre killed $[[SCRATCH:[a-z0-9]+]], $sp, -16
+# CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -3
+# CHECK-NEXT: $sp = frame-setup SUBXri $sp, 16, 0
+
+# CHECK-NEXT: $[[TMP:x[0-9]+]] = ADDXri $sp, 16
+# CHECK-NEXT: STR_ZXI $z0, killed $[[TMP]], 2
+# CHECK-NEXT: $[[TMP:x[0-9]+]] = ADDXri $sp, 16
+# CHECK-NEXT: STR_ZXI $z1, killed $[[TMP]], 1
+# CHECK-NEXT: $[[TMP:x[0-9]+]] = ADDXri $sp, 16
+# CHECK-NEXT: STR_PXI $p0, killed $[[TMP]], 7
+
+# CHECK-NEXT: $sp = frame-destroy ADDVL_XXI $sp, 3
+# CHECK-NEXT: $sp = frame-destroy ADDXri $sp, 16, 0
+# CHECK-NEXT: $sp, $[[SCRATCH]] = frame-destroy LDRXpost $sp, 16
+# CHECK-NEXT: RET_ReallyLR
+name:            test_address_sve
+frameInfo:
+  maxAlignment:  16
+fixedStack:
+  - { id: 0, stack-id: sve-vec, size: 16, alignment: 8, offset: -16 }
+  - { id: 1, stack-id: sve-vec, size: 16, alignment: 8, offset: -32 }
+  - { id: 2, stack-id: sve-vec, size:  2, alignment: 2, offset: -34 }
+stack:
+  - { id: 0, stack-id: default, size: 16, alignment: 8 }
+body:             |
+  bb.0.entry:
+    liveins: $z0, $z1, $p0
+
+    STR_ZXI $z0, %fixed-stack.0, 0
+    STR_ZXI $z1, %fixed-stack.1, 0
+    STR_PXI $p0, %fixed-stack.2, 0
+
+    RET_ReallyLR
+---
+...
+# +-----------+
+# | x20, x21  |  // callee saves
+# |  lr, fp   |  // frame record
+# +-----------+ <- FP
+# | %fstack.0 |  // scalable @ FP - 16 scalable bytes
+# | %fstack.1 |  // scalable @ FP - 32 scalable bytes
+# | %fstack.2 |  // scalable @ FP - 34 scalable bytes
+# +-----------+
+# | %stack.0  |  // not scalable
+# +-----------+ <- SP
+
+# CHECK-LABEL: name: test_address_sve_fp
+# CHECK:       stackSize: 32
+
+# CHECK:      bb.0.entry:
+# CHECK-NEXT: $sp = frame-setup STPXpre killed $fp, killed $lr, $sp, -2
+# CHECK-NEXT: $fp = frame-setup ADDXri $sp, 0, 0
+# CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -3
+# CHECK-NEXT: $sp = frame-setup SUBXri $sp, 16, 0
+
+# CHECK-NEXT: STR_ZXI $z0, $fp, -1
+# CHECK-NEXT: STR_ZXI $z1, $fp, -2
+# CHECK-NEXT: STR_PXI $p0, $fp, -17
+
+# CHECK-NEXT: $sp = frame-destroy ADDVL_XXI $sp, 3
+# CHECK:      $sp = frame-destroy ADDXri $sp, 16, 0
+# CHECK-NEXT: $sp, $fp, $lr = frame-destroy LDPXpost $sp, 2
+# CHECK-NEXT: RET_ReallyLR
+name:            test_address_sve_fp
+frameInfo:
+  maxAlignment:  16
+  isFrameAddressTaken: true
+fixedStack:
+  - { id: 0, stack-id: sve-vec, size: 16, alignment: 8, offset: -16 }
+  - { id: 1, stack-id: sve-vec, size: 16, alignment: 8, offset: -32 }
+  - { id: 2, stack-id: sve-vec, size:  2, alignment: 2, offset: -34 }
+stack:
+  - { id: 0, stack-id: default, size: 16, alignment: 8 }
+body:             |
+  bb.0.entry:
+    liveins: $z0, $z1, $p0
+
+    STR_ZXI $z0, %fixed-stack.0, 0
+    STR_ZXI $z1, %fixed-stack.1, 0
+    STR_PXI $p0, %fixed-stack.2, 0
+
+    RET_ReallyLR
+---
+...
+# +-----------+
+# | %fstack.1 |  // stack arg @ SP + 16 scalable bytes + 32 bytes.
+# +-----------+
+# |callee save|  // register saved as scratch reg.
+# +-----------+
+# | %fstack.1 |  // vector of 16 scalable bytes
+# +---------- +
+# | %stack.0  |  // not scalable, 16 bytes
+# +-----------+ <- SP
+# CHECK-LABEL: name: test_stack_arg_sve
+# CHECK:       stackSize: 32
+
+# CHECK:      bb.0.entry:
+# CHECK-NEXT: $sp = frame-setup STRXpre killed $[[SCRATCH:[a-z0-9]+]], $sp, -16
+# CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -1
+# CHECK-NEXT: $sp = frame-setup SUBXri $sp, 16, 0
+
+# CHECK:      $[[TMP:x[0-9]+]] = ADDVL_XXI $sp, 1
+# CHECK-NEXT: $x0 = LDRXui killed $[[TMP]], 4
+
+# CHECK-NEXT: $sp = frame-destroy ADDVL_XXI $sp, 1
+# CHECK-NEXT: $sp = frame-destroy ADDXri $sp, 16, 0
+# CHECK-NEXT: $sp, $[[SCRATCH]] = frame-destroy LDRXpost $sp, 16
+# CHECK-NEXT: RET_ReallyLR
+name:             test_stack_arg_sve
+fixedStack:
+  - { id: 0, stack-id: default, size: 16, alignment: 16, offset: 0 }
+  - { id: 1, stack-id: sve-vec, size: 16, alignment: 16, offset: -16 }
+stack:
+  - { id: 0, stack-id: default, size: 16, alignment: 16 }
+body:             |
+  bb.0.entry:
+    liveins: $x0
+
+    $x0 = LDRXui %fixed-stack.0, 0
+    RET_ReallyLR
+---
+...
+# Test that the address to access an SVE data vector at an offset that
+# does not fit its immediate, is correctly materialized.
+# +----------+
+# |calleesave|  // register saved as scratch reg.
+# +----------+
+# | %stack.0 |  // one SVE data object @ SP + 256 scalable bytes.
+# |::::::::::|
+# |:        :|
+# |:%stack.1:|  // Large object
+# |:        :|
+# |::::::::::|
+# +----------+ <- SP
+# CHECK-LABEL: name: test_address_sve_out_of_range
+# CHECK:       stackSize: 16
+
+# CHECK:      bb.0.entry:
+# CHECK-NEXT: $sp = frame-setup STRXpre killed $[[SCRATCH:[a-z0-9]+]], $sp, -16
+# CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -32
+# CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -32
+# CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -32
+# CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -32
+# CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -32
+# CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -32
+# CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -32
+# CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -32
+# CHECK-NEXT: $sp = frame-setup ADDVL_XXI $sp, -1
+
+# CHECK-NEXT: $[[TMP2:x[0-9]+]] = ADDVL_XXI $sp, 1
+# CHECK-NEXT: STR_ZXI $z0, killed $[[TMP2]], 255
+
+# CHECK-NEXT: $[[TMP2:x[0-9]+]] = ADDPL_XXI $sp, 1
+# CHECK-NEXT: STR_PXI $p0, killed $[[TMP2]], 255
+
+# CHECK:      $sp = frame-destroy ADDVL_XXI $sp, 31
+# CHECK-NEXT: $sp = frame-destroy ADDVL_XXI $sp, 31
+# CHECK-NEXT: $sp = frame-destroy ADDVL_XXI $sp, 31
+# CHECK-NEXT: $sp = frame-destroy ADDVL_XXI $sp, 31
+# CHECK-NEXT: $sp = frame-destroy ADDVL_XXI $sp, 31
+# CHECK-NEXT: $sp = frame-destroy ADDVL_XXI $sp, 31
+# CHECK-NEXT: $sp = frame-destroy ADDVL_XXI $sp, 31
+# CHECK-NEXT: $sp = frame-destroy ADDVL_XXI $sp, 31
+# CHECK-NEXT: $sp = frame-destroy ADDVL_XXI $sp, 9
+# CHECK-NEXT: $sp, $[[SCRATCH]] = frame-destroy LDRXpost $sp, 16
+# CHECK-NEXT: RET_ReallyLR
+name:            test_address_sve_out_of_range
+frameInfo:
+  maxAlignment:  16
+fixedStack:
+  - { id: 0, stack-id: sve-vec, size:   16, alignment: 16, offset: -16 }
+  - { id: 1, stack-id: sve-vec, size: 3584, alignment: 16, offset: -3600 }
+  - { id: 2, stack-id: sve-vec, size:  512, alignment: 16, offset: -4112 }
+
+body:             |
+  bb.0.entry:
+    liveins: $z0, $p0
+
+    STR_ZXI $z0, %fixed-stack.0, 0
+    STR_PXI $p0, %fixed-stack.1, 0
+
+    RET_ReallyLR
+---