[AArch64][SVE] Fix handling of stack protection with SVE

[john.brawn]

Summary

Fix a couple of things that were causing stack protection to not work correctly in functions that have scalable vectors on the stack:

• Fix an assertion failure in the StackProtector pass caused by trying to call getTypeAllocSize on a scalable vector.
• When stack protection is enabled move the stack protector location to the top of the SVE locals, so that any overflow in them (or the other locals which are below that) will be detected.

Fixes PR51795.

[sdesmalen]

Thanks for working on this @john.brawn. I've left some suggestions on your patch.

llvm/lib/CodeGen/StackProtector.cpp

363–364

I'd suggest changing the interface HasAddressTaken to take a TypeSize.

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

110

In your patch you've changed the frame-layout by swapping the areas of fixed-width locals and SVE locals and always requiring/reserving x19 as a base pointer. An alternative idea is to allocate vscale x 16 bytes (i.e. size of one SVE vector) for the stack protector slot, and have that as the first SVE slot after the SVE callee saves:

+-----------------------+
|     Callee Saves      |
+ - - - - - - - - - - - +
|  Frame record (LR/FP) |
+ - - - - - - - - - - - + <- FP
| Neon/SVE callee saves |
+ - - - - - - - - - - - +
|    Stack protector    |
| . . . . . . . . . . . | <- FP - sizeof(Neon/SVE callee saves) - 1*sizeof(SVE vector)
|                       |
| SVE locals/spill-slots|
|                       |
+ - - - - - - - - - - - + 
|                       | 
|        Locals         | 
|                       | 
+------------------------ <- SP/BP

This way, we'd trade a bit of redundant stack space, but we'd avoid having to always reserve x19. We also have the benefit of using the same layout (reducing possibility of bugs being introduced by any new layout which is only enabled through a compile flag). Like with your current patch, we can still access SVE locals directly at <pointer> + <ScalableOffset>, where <pointer> in this case will be FP instead of BP.

I'm not too familiar with the exact requirements for the position of the stack protector slot, but my preliminary understanding is that it must live somewhere below the frame-record and above the locals. The above suggestion would satisfy that requirement.

Can you think of any reason why this might not be feasible?

1121

Could this use hasStackProtectorIndex instead?

[john.brawn]

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

110

I hadn't though about doing it this way, but it looks like it should be feasible and would definitely simplify things. I'll give it a go and see if I can get it working.

[john.brawn]

Moved the stack protector to above the SVE locals instead of moving SVE locals below the stack protector, and also make use of TypeSize.

[john.brawn]

Ping.

[sdesmalen]

Hi @john.brawn, thanks for updating your patch, this approach seems like a good improvement! Just left a few more comments.

llvm/lib/CodeGen/PrologEpilogInserter.cpp

969

nit: unnecessary curly braces

972

Do we need to some way to ensure that the LocalStackSlotAllocationPass doesn't pre-allocate the stack protector value if we know it will be allocated to a different StackID?

llvm/lib/CodeGen/StackProtector.cpp

175–176

This should use TypeSize::isKnownGT(Typesize::getFixed(MemLoc->Size.getValue()), AllocSize).

I doubt it would ever return true when AllocSize is scalable though, becuase from what I can see in the code, MemLoc.hasValue() will return false if I accesses a scalable size, because MemoryLocation will not be precise.

217–222

If the gep is indexing into a scalable vector type, then OffsetSize cannot be fixed size?

347

unnecessary change?

363

same here, perhaps just move this into NFC patch and submit it separately.

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

2958

I think this patch also needs some code to determineSVEStackObjectOffsets to ensure that the stack protector value is ordered before all other objects when allocating their offsets. (this is probably best tested with an MIR test)

llvm/test/CodeGen/AArch64/stack-guard-sve.ll

17

nit: To simplify the test, you could also write:

store <vscale x 4 x float> zeroinitializer, <vscale x 4 x float>* %x, align 16

[john.brawn]

llvm/lib/CodeGen/PrologEpilogInserter.cpp

969

That's done for consistency with the preceding if (which has braces because of the nested if/else).

972

I'll add an assertion to LocalStackSlotAllocationPass to check for that.

llvm/lib/CodeGen/StackProtector.cpp

217–222

Indexing into a scalable vector depends only on the size of the element type, which is known.

347

A relic of the previous version of this patch, I'll remove.

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

2958

Will do.

[john.brawn]

Looking into the interaction with LocalStackSlotAllocation it turned out things were more complicated than I thought as it's run before we had decided to put the stack protector in the SVE stack area. I've gone with moving that decision much earlier (instead of attempting to undo what LocalStackSlotAllocation had done).

The getelementptr interaction in HasAddressTaken also wasn't right. Instead of doing "isKnownLT" we have to do "!isKnownGE" as these functions false in the "don't know" case. We also have to be more careful about the subtraction, as you can't subtract a fixed value from a scalable value.

[john.brawn]

Ping.

[sdesmalen]

Thanks for the update, the patch looks structurally good to me. Just left a few nits and a request for an extra test.

llvm/lib/CodeGen/StackProtector.cpp

175

nit: TypeSize::isKnownLT ?

220–221

nit: This can just be:

TypeSize NewAllocSize = TypeSize::Fixed(AllocSize.getKnownMinValue()) - OffsetSize;

Can you also add a test for this case? I guess that would require something like:

%ptr = alloca <vscale x 4 x i32>
%ptr.bc = bitcast <vscale x 4 x i32>* to i32*
%gep = getelementptr i32, i32* %ptr.bc, i64 2 ; no stack protector needed
                                              ; (2*sizeof(i32)) < (vscale x 4 x sizeof(i32))
store i32 %something, i32* %gep
%gep2 = getelementptr i32, i32* %gep, i64 2   ; now a stack protector is needed because
                                              ; it exceeds `sizeof(4 x i32) - 2 x sizeof(i32)`
store i32 %something, i32* %gep2

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

18431

Is it worth adding a FIXME saying that it would be nice to have this in a new interface, something like getStackIDForStackProtectorIndex, rather than doing it in finalizeLowering?

llvm/test/CodeGen/AArch64/stack-guard-reassign-sve.mir

20

nit: It would be better to reorder these objects so that the StackGuardSlot is not the first object, so that you can test it still gets allocated at the top of the frame. (because the algorithm processes the objects in order of id)

[john.brawn]

llvm/lib/CodeGen/StackProtector.cpp

175

It needs to be a negated comparison because the TypeSize::isKnownXY functions return false for "I don't know" and in those cases we should assume the access is larger than AllocSize.

[john.brawn]

Adjusted based on review comments.

[sdesmalen]

Thanks, LGTM!

llvm/include/llvm/CodeGen/StackProtector.h

private:
   /// for it.
   /// \param [out] IsLarge is set to true if a protectable array is found and
   /// it is "large" ( >= ssp-buffer-size).  In the case of a structure with
   /// multiple arrays, this gets set if any of them is large.
   bool ContainsProtectableArray(Type *Ty, bool &IsLarge, bool Strong = false,
                                 bool InStruct = false) const;
 
   /// Check whether a stack allocation has its address taken.
-  bool HasAddressTaken(const Instruction *AI, uint64_t AllocSize);
+  bool HasAddressTaken(const Instruction *AI, TypeSize AllocSize);
 
   /// RequiresStackProtector - Check whether or not this function needs a
   /// stack protector based upon the stack protector level.
   bool RequiresStackProtector();
 
 public:
   static char ID; // Pass identification, replacement for typeid.
 

llvm/lib/CodeGen/LocalStackSlotAllocation.cpp

if (MFI.hasStackProtectorIndex()) {
     // that is **not** covering the protected objects.
     assert(!MFI.isObjectPreAllocated(StackProtectorFI) &&
            "Stack protector pre-allocated in LocalStackSlotAllocation");
 
     StackObjSet LargeArrayObjs;
     StackObjSet SmallArrayObjs;
     StackObjSet AddrOfObjs;
 
-    AdjustStackOffset(MFI, StackProtectorFI, Offset, StackGrowsDown, MaxAlign);
+    // Only place the stack protector in the local stack area if the target
+    // allows it.
+    if (TFI.isStackIdSafeForLocalArea(MFI.getStackID(StackProtectorFI)))
+      AdjustStackOffset(MFI, StackProtectorFI, Offset, StackGrowsDown,
+                        MaxAlign);
 
     // Assign large stack objects first.
     for (unsigned i = 0, e = MFI.getObjectIndexEnd(); i != e; ++i) {
       if (MFI.isDeadObjectIndex(i))
         continue;
       if (StackProtectorFI == (int)i)
         continue;
       if (!TFI.isStackIdSafeForLocalArea(MFI.getStackID(i)))

llvm/lib/CodeGen/PrologEpilogInserter.cpp

if (MFI.hasStackProtectorIndex()) {
     StackObjSet LargeArrayObjs;
     StackObjSet SmallArrayObjs;
     StackObjSet AddrOfObjs;
 
     // If we need a stack protector, we need to make sure that
     // LocalStackSlotPass didn't already allocate a slot for it.
     // If we are told to use the LocalStackAllocationBlock, the stack protector
     // is expected to be already pre-allocated.
-    if (!MFI.getUseLocalStackAllocationBlock())
-      AdjustStackOffset(MFI, StackProtectorFI, StackGrowsDown, Offset, MaxAlign,
-                        Skew);
-    else if (!MFI.isObjectPreAllocated(MFI.getStackProtectorIndex()))
+    if (MFI.getStackID(StackProtectorFI) != TargetStackID::Default) {
+      // If the stack protector isn't on the default stack then it's up to the
+      // target to set the stack offset.
+      assert(MFI.getObjectOffset(StackProtectorFI) != 0 &&
+             "Offset of stack protector on non-default stack expected to be "
+             "already set.");
+      assert(!MFI.isObjectPreAllocated(MFI.getStackProtectorIndex()) &&
+             "Stack protector on non-default stack expected to not be "
+             "pre-allocated by LocalStackSlotPass.");
+    } else if (!MFI.getUseLocalStackAllocationBlock()) {
+      AdjustStackOffset(MFI, StackProtectorFI, StackGrowsDown, Offset,
+                        MaxAlign, Skew);
+    } else if (!MFI.isObjectPreAllocated(MFI.getStackProtectorIndex())) {
       llvm_unreachable(
           "Stack protector not pre-allocated by LocalStackSlotPass.");
+    }
 
     // Assign large stack objects first.
     for (unsigned i = 0, e = MFI.getObjectIndexEnd(); i != e; ++i) {
       if (MFI.isObjectPreAllocated(i) && MFI.getUseLocalStackAllocationBlock())
         continue;
       if (i >= MinCSFrameIndex && i <= MaxCSFrameIndex)
         continue;
       if (RS && RS->isScavengingFrameIndex((int)i))

llvm/lib/CodeGen/StackProtector.cpp

if (ContainsProtectableArray(*I, IsLarge, Strong, true)) {
         return true;
       NeedsProtector = true;
     }
 
   return NeedsProtector;
 }
 
 bool StackProtector::HasAddressTaken(const Instruction *AI,
-                                     uint64_t AllocSize) {
+                                     TypeSize AllocSize) {
   const DataLayout &DL = M->getDataLayout();
   for (const User *U : AI->users()) {
     const auto *I = cast<Instruction>(U);
     // If this instruction accesses memory make sure it doesn't access beyond
     // the bounds of the allocated object.
     Optional<MemoryLocation> MemLoc = MemoryLocation::getOrNone(I);
     if (MemLoc.hasValue() && MemLoc->Size.hasValue() &&
-        MemLoc->Size.getValue() > AllocSize)
+        !TypeSize::isKnownGE(AllocSize,
+                             TypeSize::getFixed(MemLoc->Size.getValue())))
       return true;
     switch (I->getOpcode()) {
     case Instruction::Store:
       if (AI == cast<StoreInst>(I)->getValueOperand())
         return true;
       break;
     case Instruction::AtomicCmpXchg:
       // cmpxchg conceptually includes both a load and store from the same
for (const User *U : AI->users()) {
     case Instruction::Invoke:
       return true;
     case Instruction::GetElementPtr: {
       // If the GEP offset is out-of-bounds, or is non-constant and so has to be
       // assumed to be potentially out-of-bounds, then any memory access that
       // would use it could also be out-of-bounds meaning stack protection is
       // required.
       const GetElementPtrInst *GEP = cast<GetElementPtrInst>(I);
-      unsigned TypeSize = DL.getIndexTypeSizeInBits(I->getType());
-      APInt Offset(TypeSize, 0);
-      APInt MaxOffset(TypeSize, AllocSize);
-      if (!GEP->accumulateConstantOffset(DL, Offset) || Offset.ugt(MaxOffset))
+      unsigned IndexSize = DL.getIndexTypeSizeInBits(I->getType());
+      APInt Offset(IndexSize, 0);
+      if (!GEP->accumulateConstantOffset(DL, Offset))
+        return true;
+      TypeSize OffsetSize = TypeSize::Fixed(Offset.getLimitedValue());
+      if (!TypeSize::isKnownGT(AllocSize, OffsetSize))
         return true;
       // Adjust AllocSize to be the space remaining after this offset.
-      if (HasAddressTaken(I, AllocSize - Offset.getLimitedValue()))
+      // We can't subtract a fixed size from a scalable one, so in that case
+      // assume the scalable value is of minimum size.
+      TypeSize MinAllocSize = TypeSize::Fixed(AllocSize.getKnownMinValue());
+      TypeSize NewAllocSize =
+          (AllocSize.isScalable() ? MinAllocSize : AllocSize) - OffsetSize;
+      if (HasAddressTaken(I, NewAllocSize))
         return true;
       break;
     }
     case Instruction::BitCast:
     case Instruction::Select:
     case Instruction::AddrSpaceCast:
       if (HasAddressTaken(I, AllocSize))
         return true;
for (const Instruction &I : BB) {
                                          MachineFrameInfo::SSPLK_LargeArray));
             ORE.emit(RemarkBuilder);
             NeedsProtector = true;
           }
           continue;
         }
 
         bool IsLarge = false;
         if (ContainsProtectableArray(AI->getAllocatedType(), IsLarge, Strong)) {
           Layout.insert(std::make_pair(AI, IsLarge
                                        ? MachineFrameInfo::SSPLK_LargeArray
                                        : MachineFrameInfo::SSPLK_SmallArray));
           ORE.emit([&]() {
             return OptimizationRemark(DEBUG_TYPE, "StackProtectorBuffer", &I)
                    << "Stack protection applied to function "
                    << ore::NV("Function", F)
                    << " due to a stack allocated buffer or struct containing a "
                       "buffer";
           });
           NeedsProtector = true;
           continue;
         }
 
         if (Strong && HasAddressTaken(AI, M->getDataLayout().getTypeAllocSize(
                                               AI->getAllocatedType()))) {
           ++NumAddrTaken;
           Layout.insert(std::make_pair(AI, MachineFrameInfo::SSPLK_AddrOf));
           ORE.emit([&]() {
             return OptimizationRemark(DEBUG_TYPE, "StackProtectorAddressTaken",
                                       &I)
                    << "Stack protection applied to function "
                    << ore::NV("Function", F)
                    << " due to the address of a local variable being taken";
           });

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

 // '#-7 mul vl' is an offset that can only be evaluated at runtime. With this
 // layout, we don't need to add an unscaled offset to the framepointer before
 // accessing the SVE object in the frame.
 //
 // In some cases when a base pointer is not strictly needed, it is generated
 // anyway when offsets from the frame pointer to access local variables become
 // so large that the offset can't be encoded in the immediate fields of loads
 // or stores.
 //
 // Outgoing function arguments must be at the bottom of the stack frame when
 // calling another function. If we do not have variable-sized stack objects, we
 // can allocate a "reserved call frame" area at the bottom of the local
 // variable area, large enough for all outgoing calls. If we do have VLAs, then
 // the stack pointer must be decremented and incremented around each call to
 // make space for the arguments below the VLAs.
 //
 // FIXME: also explain the redzone concept.
bool needsFrameMoves =
       MF.needsFrameMoves() && !MF.getTarget().getMCAsmInfo()->usesWindowsCFI();
   bool HasFP = hasFP(MF);
   bool NeedsWinCFI = needsWinCFI(MF);
   bool HasWinCFI = false;
   auto Cleanup = make_scope_exit([&]() { MF.setHasWinCFI(HasWinCFI); });
 
   bool IsFunclet = MBB.isEHFuncletEntry();
 
   // At this point, we're going to decide whether or not the function uses a
   // redzone. In most cases, the function doesn't have a redzone so let's
   // assume that's false and set it to true in the case that there's a redzone.
   AFI->setHasRedZone(false);
 
   // Debug location must be unknown since the first debug location is used
   // to determine the end of the prologue.
   DebugLoc DL;
 
bool AArch64FrameLowering::assignCalleeSavedSpillSlots(
   auto *AFI = MF.getInfo<AArch64FunctionInfo>();
   for (auto &CS : CSI) {
     Register Reg = CS.getReg();
     const TargetRegisterClass *RC = RegInfo->getMinimalPhysRegClass(Reg);
 
     unsigned Size = RegInfo->getSpillSize(*RC);
     Align Alignment(RegInfo->getSpillAlign(*RC));
     int FrameIdx = MFI.CreateStackObject(Size, Alignment, true);
     CS.setFrameIdx(FrameIdx);
 
     if ((unsigned)FrameIdx < MinCSFrameIndex) MinCSFrameIndex = FrameIdx;
     if ((unsigned)FrameIdx > MaxCSFrameIndex) MaxCSFrameIndex = FrameIdx;
 
     // Grab 8 bytes below FP for the extended asynchronous frame info.
     if (hasFP(MF) && AFI->hasSwiftAsyncContext() && Reg == AArch64::FP) {
       FrameIdx = MFI.CreateStackObject(8, Alignment, true);
       AFI->setSwiftAsyncContextFrameIdx(FrameIdx);
if (getSVECalleeSaveSlotRange(MFI, MinCSFrameIndex, MaxCSFrameIndex)) {
     }
   }
 
   // Ensure that the Callee-save area is aligned to 16bytes.
   Offset = alignTo(Offset, Align(16U));
 
   // Create a buffer of SVE objects to allocate and sort it.
   SmallVector<int, 8> ObjectsToAllocate;
+  // If we have a stack protector, and we've previously decided that we have SVE
+  // objects on the stack and thus need it to go in the SVE stack area, then it
+  // needs to go first.
+  int StackProtectorFI = -1;
+  if (MFI.hasStackProtectorIndex()) {
+    StackProtectorFI = MFI.getStackProtectorIndex();
+    if (MFI.getStackID(StackProtectorFI) == TargetStackID::ScalableVector)
+      ObjectsToAllocate.push_back(StackProtectorFI);
+  }
   for (int I = 0, E = MFI.getObjectIndexEnd(); I != E; ++I) {
     unsigned StackID = MFI.getStackID(I);
     if (StackID != TargetStackID::ScalableVector)
       continue;
+    if (I == StackProtectorFI)
+      continue;
     if (MaxCSFrameIndex >= I && I >= MinCSFrameIndex)
       continue;
     if (MFI.isDeadObjectIndex(I))
       continue;
 
     ObjectsToAllocate.push_back(I);
   }
 

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

 AArch64TargetLowering::getVaListSizeInBits(const DataLayout &DL) const {
   if (Subtarget->isTargetDarwin() || Subtarget->isTargetWindows())
     return getPointerTy(DL).getSizeInBits();
 
   return 3 * getPointerTy(DL).getSizeInBits() + 2 * 32;
 }
 
 void AArch64TargetLowering::finalizeLowering(MachineFunction &MF) const {
-  MF.getFrameInfo().computeMaxCallFrameSize(MF);
+  MachineFrameInfo &MFI = MF.getFrameInfo();
+  // If we have any vulnerable SVE stack objects then the stack protector
+  // needs to be placed at the top of the SVE stack area, as the SVE locals
+  // are placed above the other locals, so we allocate it as if it were a
+  // scalable vector.
+  if (MFI.hasStackProtectorIndex()) {
+    for (unsigned int i = 0, e = MFI.getObjectIndexEnd(); i != e; ++i) {
+      if (MFI.getStackID(i) == TargetStackID::ScalableVector &&
+          MFI.getObjectSSPLayout(i) != MachineFrameInfo::SSPLK_None) {
+        MFI.setStackID(MFI.getStackProtectorIndex(),
+                       TargetStackID::ScalableVector);
+        MFI.setObjectAlignment(MFI.getStackProtectorIndex(), Align(16));
+        break;
+      }
+    }
+  }
+  MFI.computeMaxCallFrameSize(MF);
   TargetLoweringBase::finalizeLowering(MF);
 }
 
 // Unlike X86, we let frame lowering assign offsets to all catch objects.
 bool AArch64TargetLowering::needsFixedCatchObjects() const {
   return false;
 }
 

llvm/test/CodeGen/AArch64/stack-guard-reassign-sve.mir

+# RUN: llc -mtriple=aarch64--linux-gnu -mattr=+sve -start-before=localstackalloc -stop-after=prologepilog -o - %s | FileCheck %s
+
+--- |
+  @__stack_chk_guard = external global i8*
+  define i32 @main(i32, i8**) {
+    %StackGuardSlot = alloca i8*
+    unreachable
+  }
+...
+---
+name:            main
+tracksRegLiveness: true
+frameInfo:
+# CHECK: stackSize: 544
+# CHECK: localFrameSize: 516
+  stackProtector:  '%stack.0.StackGuardSlot'
+stack:
+# Stack objects 1 and 2 should end up in the local stack area, objects 0 and 3
+# should end up in the SVE stack area with 0 (the stack guard) on top.
+  - { id: 0, name: StackGuardSlot, size: 8, alignment: 16, stack-id: scalable-vector }
+# CHECK:       - { id: 0, name: StackGuardSlot, type: default, offset: -16, size: 8,
+# CHECK-NEXT:      alignment: 16, stack-id: scalable-vector, callee-saved-register: '',
+# CHECK-NEXT:      callee-saved-restored: true, debug-info-variable: '', debug-info-expression: '',
+# CHECK-NEXT:      debug-info-location: '' }
+  - { id: 1, size: 512, alignment: 1, stack-id: default }
+# CHECK:       - { id: 1, name: '', type: default, offset: -528, size: 512, alignment: 1,
+# CHECK-NEXT:      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+# CHECK-NEXT:      local-offset: -512, debug-info-variable: '', debug-info-expression: '',
+# CHECK-NEXT:      debug-info-location: '' }
+  - { id: 2, size: 4, alignment: 4, stack-id: default }
+# CHECK:       - { id: 2, name: '', type: default, offset: -532, size: 4, alignment: 4,
+# CHECK-NEXT:      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+# CHECK-NEXT:      local-offset: -516, debug-info-variable: '', debug-info-expression: '',
+# CHECK-NEXT:      debug-info-location: '' }
+  - { id: 3, size: 16, alignment: 16, stack-id: scalable-vector }
+# CHECK:       - { id: 3, name: '', type: default, offset: -32, size: 16, alignment: 16,
+# CHECK-NEXT:      stack-id: scalable-vector, callee-saved-register: '', callee-saved-restored: true,
+# CHECK-NEXT:      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+body:             |
+  bb.0:
+    %25:gpr64common = LOAD_STACK_GUARD :: (dereferenceable invariant load (s64) from @__stack_chk_guard)
+    STRXui killed %25, %stack.0.StackGuardSlot, 0 :: (volatile store (s64) into %stack.0.StackGuardSlot)
+    %28:gpr64 = LDRXui %stack.0.StackGuardSlot, 0 :: (volatile load (s64) from %stack.0.StackGuardSlot)
+    %29:gpr64common = LOAD_STACK_GUARD :: (dereferenceable invariant load (s64) from @__stack_chk_guard)
+    RET_ReallyLR implicit undef $w0, implicit killed %28, implicit killed %29
+
+...

llvm/test/CodeGen/AArch64/stack-guard-sve.ll

+; RUN: llc -mtriple=aarch64--linux-gnu -mattr=+sve < %s | FileCheck %s
+
+declare dso_local void @val_fn(<vscale x 4 x float>)
+declare dso_local void @ptr_fn(<vscale x 4 x float>*)
+
+; An alloca of a scalable vector shouldn't trigger stack protection.
+
+; CHECK-LABEL: call_value:
+; CHECK-NOT: mov x19, sp
+; CHECK: addvl sp, sp, #-1
+; CHECK-NOT: __stack_chk_guard
+; CHECK: st1w { {{z[0-9]+.s}} }, {{p[0-9]+}}, [x29, #-1, mul vl]
+define void @call_value() #0 {
+entry:
+  %x = alloca <vscale x 4 x float>, align 16
+  store <vscale x 4 x float> zeroinitializer, <vscale x 4 x float>* %x, align 16
+  %0 = load <vscale x 4 x float>, <vscale x 4 x float>* %x, align 16
+  call void @val_fn(<vscale x 4 x float> %0)
+  ret void
+}
+
+; CHECK-LABEL: call_value_strong:
+; CHECK-NOT: mov x19, sp
+; CHECK: addvl sp, sp, #-1
+; CHECK-NOT: __stack_chk_guard
+; CHECK: st1w { {{z[0-9]+.s}} }, {{p[0-9]+}}, [x29, #-1, mul vl]
+define void @call_value_strong() #1 {
+entry:
+  %x = alloca <vscale x 4 x float>, align 16
+  store <vscale x 4 x float> zeroinitializer, <vscale x 4 x float>* %x, align 16
+  %0 = load <vscale x 4 x float>, <vscale x 4 x float>* %x, align 16
+  call void @val_fn(<vscale x 4 x float> %0)
+  ret void
+}
+
+; Address-taking of a scalable vector should trigger stack protection only with
+; sspstrong, and the scalable vector should be be placed below the stack guard.
+
+; CHECK-LABEL: call_ptr:
+; CHECK-NOT: mov x19, sp
+; CHECK: addvl sp, sp, #-1
+; CHECK-NOT: __stack_chk_guard
+; CHECK: addvl x0, x29, #-1
+; CHECK: bl ptr_fn
+define void @call_ptr() #0 {
+entry:
+  %x = alloca <vscale x 4 x float>, align 16
+  call void @ptr_fn(<vscale x 4 x float>* %x)
+  ret void
+}
+
+; CHECK-LABEL: call_ptr_strong:
+; CHECK: mov x29, sp
+; CHECK: addvl sp, sp, #-2
+; CHECK-DAG: addvl [[ADDR:x[0-9]+]], x29, #-1
+; CHECK-DAG: ldr [[VAL:x[0-9]+]], [{{x[0-9]+}}, :lo12:__stack_chk_guard]
+; CHECK-DAG: str [[VAL]], {{\[}}[[ADDR]]]
+; CHECK-DAG: addvl x0, x29, #-2
+; CHECK: bl ptr_fn
+define void @call_ptr_strong() #1 {
+entry:
+  %x = alloca <vscale x 4 x float>, align 16
+  call void @ptr_fn(<vscale x 4 x float>* %x)
+  ret void
+}
+
+; Check that both variables are addressed in the same way
+
+; CHECK-LABEL: call_both:
+; CHECK: mov x29, sp
+; CHECK: addvl sp, sp, #-2
+; CHECK-NOT: __stack_chk_guard
+; CHECK: st1w { {{z[0-9]+.s}} }, {{p[0-9]+}}, [x29, #-1, mul vl]
+; CHECK: bl val_fn
+; CHECK: addvl x0, x29, #-2
+; CHECK: bl ptr_fn
+define void @call_both() #0 {
+entry:
+  %x = alloca <vscale x 4 x float>, align 16
+  %y = alloca <vscale x 4 x float>, align 16
+  store <vscale x 4 x float> zeroinitializer, <vscale x 4 x float>* %x, align 16
+  %0 = load <vscale x 4 x float>, <vscale x 4 x float>* %x, align 16
+  call void @val_fn(<vscale x 4 x float> %0)
+  call void @ptr_fn(<vscale x 4 x float>* %y)
+  ret void
+}
+
+; CHECK-LABEL: call_both_strong:
+; CHECK: mov x29, sp
+; CHECK: addvl sp, sp, #-3
+; CHECK-DAG: addvl [[ADDR:x[0-9]+]], x29, #-1
+; CHECK-DAG: ldr [[VAL:x[0-9]+]], [{{x[0-9]+}}, :lo12:__stack_chk_guard]
+; CHECK-DAG: str [[VAL]], {{\[}}[[ADDR]]]
+; CHECK-DAG: st1w { {{z[0-9]+.s}} }, {{p[0-9]+}}, [x29, #-2, mul vl]
+; CHECK: bl val_fn
+; CHECK: addvl x0, x29, #-3
+; CHECK: bl ptr_fn
+define void @call_both_strong() #1 {
+entry:
+  %x = alloca <vscale x 4 x float>, align 16
+  %y = alloca <vscale x 4 x float>, align 16
+  store <vscale x 4 x float> zeroinitializer, <vscale x 4 x float>* %x, align 16
+  %0 = load <vscale x 4 x float>, <vscale x 4 x float>* %x, align 16
+  call void @val_fn(<vscale x 4 x float> %0)
+  call void @ptr_fn(<vscale x 4 x float>* %y)
+  ret void
+}
+
+; Pushed callee-saved regs should be above the stack guard
+
+; CHECK-LABEL: callee_save:
+; CHECK: mov x29, sp
+; CHECK: addvl sp, sp, #-18
+; CHECK: str {{z[0-9]+}}, [sp, #{{[0-9]+}}, mul vl]
+; CHECK-NOT: mov x29, sp
+; CHECK: addvl sp, sp, #-1
+; CHECK-NOT: __stack_chk_guard
+; CHECK: addvl [[REG:x[0-9]+]], x29, #-11
+; CHECK: st1w { {{z[0-9]+.s}} }, {{p[0-9]+}}, {{\[}}[[REG]], #-8, mul vl]
+define void @callee_save(<vscale x 4 x float> %x) #0 {
+entry:
+  %x.addr = alloca <vscale x 4 x float>, align 16
+  store <vscale x 4 x float> %x, <vscale x 4 x float>* %x.addr, align 16
+  call void @ptr_fn(<vscale x 4 x float>* %x.addr)
+  ret void
+}
+
+; CHECK-LABEL: callee_save_strong:
+; CHECK: mov x29, sp
+; CHECK: addvl sp, sp, #-18
+; CHECK: str {{z[0-9]+}}, [sp, #{{[0-9]+}}, mul vl]
+; CHECK: addvl sp, sp, #-2
+; CHECK-DAG: addvl [[ADDR:x[0-9]+]], x29, #-19
+; CHECK-DAG: ldr [[VAL:x[0-9]+]], [{{x[0-9]+}}, :lo12:__stack_chk_guard]
+; CHECK-DAG: str [[VAL]], {{\[}}[[ADDR]]]
+; CHECK-DAG: addvl [[ADDR2:x[0-9]+]], x29, #-12
+; CHECK-DAG: st1w { z0.s }, p0, {{\[}}[[ADDR2]], #-8, mul vl]
+define void @callee_save_strong(<vscale x 4 x float> %x) #1 {
+entry:
+  %x.addr = alloca <vscale x 4 x float>, align 16
+  store <vscale x 4 x float> %x, <vscale x 4 x float>* %x.addr, align 16
+  call void @ptr_fn(<vscale x 4 x float>* %x.addr)
+  ret void
+}
+
+; Check that local stack allocation works correctly both when we have a stack
+; guard but no vulnerable SVE objects, and when we do have such objects.
+
+; CHECK-LABEL: local_stack_alloc:
+; CHECK: mov x29, sp
+; CHECK: addvl sp, sp, #-2
+; CHECK: sub sp, sp, #16, lsl #12
+; CHECK: sub sp, sp, #16
+
+; Stack guard is placed below the SVE stack area
+; CHECK-DAG: ldr [[STACK_GUARD:x[0-9]+]], [{{x[0-9]+}}, :lo12:__stack_chk_guard]
+; CHECK-DAG: addvl [[STACK_GUARD_POS:x[0-9]+]], x29, #-2
+; CHECK-DAG: stur [[STACK_GUARD]], {{\[}}[[STACK_GUARD_POS]], #-8]
+
+; char_arr is below the stack guard
+; CHECK-DAG: sub [[CHAR_ARR_1:x[0-9]+]], x29, #16
+; CHECK-DAG: addvl [[CHAR_ARR_2:x[0-9]+]], [[CHAR_ARR_1]], #-2
+; CHECK-DAG: strb wzr, {{\[}}[[CHAR_ARR_2]]]
+
+; large1 is accessed via a virtual base register
+; CHECK-DAG: add [[LARGE1:x[0-9]+]], sp, #8, lsl #12
+; CHECK-DAG: stp x0, x0, {{\[}}[[LARGE1]]]
+
+; large2 is at the bottom of the stack
+; CHECK-DAG: stp x0, x0, [sp]
+
+; vec1 and vec2 are in the SVE stack immediately below fp
+; CHECK-DAG: addvl x0, x29, #-1
+; CHECK-DAG: bl ptr_fn
+; CHECK-DAG: addvl x0, x29, #-2
+; CHECK-DAG: bl ptr_fn
+define void @local_stack_alloc(i64 %val) #0 {
+entry:
+  %char_arr = alloca [8 x i8], align 4
+  %gep0 = getelementptr [8 x i8], [8 x i8]* %char_arr, i64 0, i64 0
+  store i8 0, i8* %gep0, align 8
+  %large1 = alloca [4096 x i64], align 8
+  %large2 = alloca [4096 x i64], align 8
+  %vec_1 = alloca <vscale x 4 x float>, align 16
+  %vec_2 = alloca <vscale x 4 x float>, align 16
+  %gep1 = getelementptr [4096 x i64], [4096 x i64]* %large1, i64 0, i64 0
+  %gep2 = getelementptr [4096 x i64], [4096 x i64]* %large1, i64 0, i64 1
+  store i64 %val, i64* %gep1, align 8
+  store i64 %val, i64* %gep2, align 8
+  %gep3 = getelementptr [4096 x i64], [4096 x i64]* %large2, i64 0, i64 0
+  %gep4 = getelementptr [4096 x i64], [4096 x i64]* %large2, i64 0, i64 1
+  store i64 %val, i64* %gep3, align 8
+  store i64 %val, i64* %gep4, align 8
+  call void @ptr_fn(<vscale x 4 x float>* %vec_1)
+  call void @ptr_fn(<vscale x 4 x float>* %vec_2)
+  ret void
+}
+
+; CHECK-LABEL: local_stack_alloc_strong:
+; CHECK: mov x29, sp
+; CHECK: addvl sp, sp, #-3
+; CHECK: sub sp, sp, #16, lsl #12
+; CHECK: sub sp, sp, #16
+
+; Stack guard is placed at the top of the SVE stack area
+; CHECK-DAG: ldr [[STACK_GUARD:x[0-9]+]], [{{x[0-9]+}}, :lo12:__stack_chk_guard]
+; CHECK-DAG: addvl [[STACK_GUARD_POS:x[0-9]+]], x29, #-1
+; CHECK-DAG: str [[STACK_GUARD]], {{\[}}[[STACK_GUARD_POS]]]
+
+; char_arr is below the SVE stack area
+; CHECK-DAG: addvl [[CHAR_ARR:x[0-9]+]], x29, #-3
+; CHECK-DAG: sturb wzr, {{\[}}[[CHAR_ARR]], #-8]
+
+; large1 is accessed via a virtual base register
+; CHECK-DAG: add [[LARGE1:x[0-9]+]], sp, #8, lsl #12
+; CHECK-DAG: stp x0, x0, {{\[}}[[LARGE1]], #8]
+
+; large2 is at the bottom of the stack
+; CHECK-DAG: stp x0, x0, [sp, #8]
+
+; vec1 and vec2 are in the SVE stack area below the stack guard
+; CHECK-DAG: addvl x0, x29, #-2
+; CHECK-DAG: bl ptr_fn
+; CHECK-DAG: addvl x0, x29, #-3
+; CHECK-DAG: bl ptr_fn
+define void @local_stack_alloc_strong(i64 %val) #1 {
+entry:
+  %char_arr = alloca [8 x i8], align 4
+  %gep0 = getelementptr [8 x i8], [8 x i8]* %char_arr, i64 0, i64 0
+  store i8 0, i8* %gep0, align 8
+  %large1 = alloca [4096 x i64], align 8
+  %large2 = alloca [4096 x i64], align 8
+  %vec_1 = alloca <vscale x 4 x float>, align 16
+  %vec_2 = alloca <vscale x 4 x float>, align 16
+  %gep1 = getelementptr [4096 x i64], [4096 x i64]* %large1, i64 0, i64 0
+  %gep2 = getelementptr [4096 x i64], [4096 x i64]* %large1, i64 0, i64 1
+  store i64 %val, i64* %gep1, align 8
+  store i64 %val, i64* %gep2, align 8
+  %gep3 = getelementptr [4096 x i64], [4096 x i64]* %large2, i64 0, i64 0
+  %gep4 = getelementptr [4096 x i64], [4096 x i64]* %large2, i64 0, i64 1
+  store i64 %val, i64* %gep3, align 8
+  store i64 %val, i64* %gep4, align 8
+  call void @ptr_fn(<vscale x 4 x float>* %vec_1)
+  call void @ptr_fn(<vscale x 4 x float>* %vec_2)
+  ret void
+}
+
+; A GEP addressing into a vector of <vscale x 4 x float> is in-bounds for
+; offsets up to 3, but out-of-bounds (and so triggers stack protection with
+; sspstrong) after that.
+
+; CHECK-LABEL: vector_gep_3:
+; CHECK-NOT: __stack_chk_guard
+define void @vector_gep_3() #0 {
+entry:
+  %vec = alloca <vscale x 4 x float>, align 16
+  %gep = getelementptr <vscale x 4 x float>, <vscale x 4 x float>* %vec, i64 0, i64 3
+  store float 0.0, float* %gep, align 4
+  ret void
+}
+
+; CHECK-LABEL: vector_gep_4:
+; CHECK-NOT: __stack_chk_guard
+define void @vector_gep_4() #0 {
+entry:
+  %vec = alloca <vscale x 4 x float>, align 16
+  %gep = getelementptr <vscale x 4 x float>, <vscale x 4 x float>* %vec, i64 0, i64 4
+  store float 0.0, float* %gep, align 4
+  ret void
+}
+
+; CHECK-LABEL: vector_gep_n:
+; CHECK-NOT: __stack_chk_guard
+define void @vector_gep_n(i64 %n) #0 {
+entry:
+  %vec = alloca <vscale x 4 x float>, align 16
+  %gep = getelementptr <vscale x 4 x float>, <vscale x 4 x float>* %vec, i64 0, i64 %n
+  store float 0.0, float* %gep, align 4
+  ret void
+}
+
+; CHECK-LABEL: vector_gep_3_strong:
+; CHECK-NOT: __stack_chk_guard
+define void @vector_gep_3_strong() #1 {
+entry:
+  %vec = alloca <vscale x 4 x float>, align 16
+  %gep = getelementptr <vscale x 4 x float>, <vscale x 4 x float>* %vec, i64 0, i64 3
+  store float 0.0, float* %gep, align 4
+  ret void
+}
+
+; CHECK-LABEL: vector_gep_4_strong:
+; CHECK: __stack_chk_guard
+define void @vector_gep_4_strong(i64 %val) #1 {
+entry:
+  %vec = alloca <vscale x 4 x float>, align 16
+  %gep = getelementptr <vscale x 4 x float>, <vscale x 4 x float>* %vec, i64 0, i64 4
+  store float 0.0, float* %gep, align 4
+  ret void
+}
+
+; CHECK-LABEL: vector_gep_n_strong:
+; CHECK: __stack_chk_guard
+define void @vector_gep_n_strong(i64 %n) #1 {
+entry:
+  %vec = alloca <vscale x 4 x float>, align 16
+  %gep = getelementptr <vscale x 4 x float>, <vscale x 4 x float>* %vec, i64 0, i64 %n
+  store float 0.0, float* %gep, align 4
+  ret void
+}
+
+attributes #0 = { ssp "frame-pointer"="non-leaf" }
+attributes #1 = { sspstrong "frame-pointer"="non-leaf" }