[SVE][AArch64] Refine hasSVEArgsOrReturn

[MattDevereau]

Summary

As described in aapcs64 (https://github.com/ARM-software/abi-aa/blob/2022Q1/aapcs64/aapcs64.rst#scalable-vector-registers) AAVPCS is used only when registers z0-z7 take an SVE argument. This fixes the case where floats occupy the lower bits of registers z0-z7 but SVE arguments in registers greater than z7 cause a function to use AAVPCS where it should use AAPCS.

Moving SVE function deduction from AArch64RegisterInfo::hasSVEArgsOrReturn to AArch64TargetLowering::LowerFormalArguments where physical register lowering is more accurate fixes this.

[efriedma]

llvm/lib/Target/AArch64/AArch64RegisterInfo.cpp

81

Using isFloatTy() like this isn't going to be accurate. If you want to correctly compute the registers used for a call, you really should just use the result of analyzeCallOperands() from isel. Maybe compute it in isel, then save the result in AArch64FunctionInfo.

[efriedma]

llvm/lib/Target/AArch64/AArch64RegisterInfo.cpp

81

Ignore my reference to analyzeCallOperands(). The actual computation of the registers used happens in AArch64TargetLowering::LowerFormalArguments.

[MattDevereau]

Moved SVE function deduction from AArch64RegisterInfo::hasSVEArgsOrReturn to AArch64TargetLowering::LowerFormalArguments
Added IsSVE bool to AArch64FunctionInfo which describes if an AArch64Function has SVE args or return type in physical registers

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64RegisterInfo.cpp

69–85

Does AArch64FunctionInfo contain all the information we need? It looks like it contains a MachineFunction *. I ask because I'm wondering if we can just get rid of AArch64RegisterInfo::hasSVEArgsOrReturn entirely.

[MattDevereau]

llvm/lib/Target/AArch64/AArch64RegisterInfo.cpp

69–85

AArch64TargetLowering::isEligibleForTailCallOptimization and AArch64AsmPrinter::emitFunctionEntryLabel() also use AArch64RegisterInfo::hasSVEArgsOrReturn. As AArch64FunctionInfo has a reference to a machine function, it should be possible to move isa<ScalableVectorType>(MF->getFunction().getReturnType()) into the new AArch64FunctionInfo::IsSVE method.

So yes, I do think AArch64FunctionInfo has all the information we need

[MattDevereau]

Removed hasSVEArgsOrReturn

[efriedma]

LGTM

[efriedma]

llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h

199 ↗

(On Diff #435880)

Hmm... actually, looking at this again, I'm a little concerned about checking the IR return type directly, instead of using AnalyzeReturn. I'm not sure if this actually comes up in practice, but calling convention lowering does various transforms on the return type. Maybe if you write something silly like "<vscale x 1000 x double>".

Maybe we can add a bit of code to LowerFormalArguments to also check the return type?

[MattDevereau]

llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h

199 ↗

(On Diff #435880)

Modifying test @sve_caller_sve_callee() in sve-tailcall.ll to use return type <vscale x 1000 x double> caused the output "LLVM ERROR: Invalid size request on a scalable vector.".

In similar fashion to the changes in LowerFormalArguments, I checked the types of the return values in AArch64TargetLowering::LowerReturn instead of checking the IR return type directly. This ends up passing on the rest of the tests except sve_caller_sve_callee and sve_caller_sve_callee_fastcc, where the return type is not eligible for tail optimization and the frame-pointer is incorrectly unpreserved. Is directly checking the IR return type in AArch64TargetLowering::LowerReturn after AnalyzeReturn still not sufficient?

[efriedma]

llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h

199 ↗

(On Diff #435880)

I don't think we call AArch64TargetLowering::LowerReturn for all functions, in general, only functions that return. We need to do the computation somewhere that's called for every function.

Besides my <vscale x 1000 x double> example, I don't have any specific cases where just checking isa<ScalableVectorType> doesn't work, but I'd like to avoid falling out of sync if we do ever add support for, for example, returning fixed-width vectors in SVE registers.

Modifying test @sve_caller_sve_callee() in sve-tailcall.ll to use return type <vscale x 1000 x double> caused the output "LLVM ERROR: Invalid size request on a scalable vector.".

That sounds like a bug we should fix. Although maybe not a high priority one.

[sdesmalen]

llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h

199 ↗

(On Diff #435880)

I don't think we call AArch64TargetLowering::LowerReturn for all functions, in general, only functions that return. We need to do the computation somewhere that's called for every function.

I'm not sure I understand this point.

For arguments this check is done in LowerFormalArguments, so that case is all covered. For the return value, I'd expect that LowerReturn is sufficient, because if the function doesn't return (and thus doesn't call LowerReturn), it also can't return a value in an SVE register.

201 ↗

(On Diff #435880)

nit: Should this be something like UsesSVERegsForArgsOrReturnValue ?

[efriedma]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

6196 ↗

(On Diff #437524)

I don't understand what you're doing here; FuncInfo describes the caller, not the callee.

llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h

199 ↗

(On Diff #435880)

A function that doesn't return can still throw an exception.

[MattDevereau]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

6196 ↗

(On Diff #437524)

Not doing this causes sve_caller_sve_callee and sve_caller_sve_callee_fastcc in sve-tailcall.ll to fail.
From ISD::InputArg:

/// InputArg - This struct carries flags and type information about a
/// single incoming (formal) argument or incoming (from the perspective
/// of the caller) return value virtual register.
///

[efriedma]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

6196 ↗

(On Diff #437524)

In @sve_caller_sve_callee, the return type is an SVE type, so we should be calling setIsSVECC(true) elsewhere.

I think this maybe another case of the issue that LowerReturn doesn't run/runs too late?

[sdesmalen]

llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h

199 ↗

(On Diff #435880)

If a function never returns because the only 'return' is an exceptional return, then it doesn't have to preserve the entire contents of z8-z23, which I think is what this patch tries to ascertain. Perhaps I was just reading too much into your statement, which confused me. I would have expected the function that analyses/lowers the return to be the right place to handle this, if it is called at the right time of course.

[efriedma]

llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h

199 ↗

(On Diff #435880)

If we throw an exception, the unwinder has to restore z8-z23, I think? Or is there some carveout for unwinding that lets us get away without preserving those registers somehow? Is that documented somewhere?

There's also the issue that tail calls don't go through LowerReturn.

In any case, given that we're calling isSVECC() inside isel, we should ensure that all the setIsSVECC() happen before we start calling isSVECC(); anything else is confusing at best...

[sdesmalen]

llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h

199 ↗

(On Diff #435880)

If we throw an exception, the unwinder has to restore z8-z23, I think? Or is there some carveout for unwinding that lets us get away without preserving those registers somehow? Is that documented somewhere?

I'm not entirely sure where this is officially documented (or whether this is just common practice of existing unwinders), but D84737 states that unwinders may only preserve the lower 64bits.

There's also the issue that tail calls don't go through LowerReturn.

In any case, given that we're calling isSVECC() inside isel, we should ensure that all the setIsSVECC() happen before we start calling isSVECC(); anything else is confusing at best...

Yes, I agree.

[MattDevereau]

llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h

199 ↗

(On Diff #435880)

Is there any particular case where isSVECC() is likely to be called before setIsSVECC()? I've not observed this so far while working on this

[efriedma]

llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h

199 ↗

(On Diff #435880)

isEligibleForTailCallOptimization() calls isSVECC(); I expect that runs before LowerReturn().

[peterwaller-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

6186–6192 ↗

(On Diff #437524)

Is this logic not the logic that this patch is trying to fix? This needs to be changed to make use of the isSVECC flag.

I see here that the calling convention of the Callee is based on this, but doesn't it need to account for the rules as described in the summary of the patch?

Thinking aloud, my interpretation of what's going on is that the underlying CallLoweringInfo::CallConv is C/Fast, because that's what the 'user' specified, but this is 'magically' implied to be a AArch64_SVE_VectorCall only when it matters during the lowering of call/return.

Architecturally this is a little confusing since it means we have these bits of code which 'patch up' the calling convention at the last moment. Is it possible we can fix the CallingConv earlier? I take it not easily, because the details of which registers are used is decided by e.g. AnalyzeReturn, and this only runs during the lowering.

[MattDevereau]

Added logic to LowerFormalArguments to check the return type for scalable vector types. This is more reliable than checking the return types in LowerReturn as LowerReturn does not always run when checking calls, i.e. in the tail call case

[efriedma]

LGTM

[peterwaller-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

6202 ↗

(On Diff #441023)

Note: "Ins" contains the return values of the call, and "Outs" contains the parameters to the call. I think these are confusingly named, at best.

This update of CallConv is a reference to CLI.CallConv, so it's setting the CallConv in the CLI, which gets passed around for various decision making.

I note that the logic for setting CalleeOutSVE is incorrect, because it does not account for the position of the arguments. If I put a scalable vector in any parameter it will update the calling convention, even though we know that's not the correct logic.

I didn't yet find if/where this matters. If it does not matter I think it may be best removed to save future confusion, since it's incorrect.

[peterwaller-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

6202 ↗

(On Diff #441023)

With reflection, I'm changing my position on this for this patch, so please go ahead.

llvm/lib/Target/AArch64/AArch64RegisterInfo.cpp

if (AArch64::ZPRRegClass.contains(Reg)) {
     }
     return false;
   }
 
   RegToUseForCFI = Reg;
   return true;
 }
 
 bool AArch64RegisterInfo::hasSVEArgsOrReturn(const MachineFunction *MF) {
   const Function &F = MF->getFunction();
+  unsigned ZRegsUsed = 0;
   return isa<ScalableVectorType>(F.getReturnType()) ||
-         any_of(F.args(), [](const Argument &Arg) {
-           return isa<ScalableVectorType>(Arg.getType());
+         any_of(F.args(), [&ZRegsUsed](const Argument &Arg) {
+           auto Ty = Arg.getType();
+           if (isa<ScalableVectorType>(Ty)) {
+             if (cast<ScalableVectorType>(Ty)->getElementType() ==
+                 Type::getInt1Ty(Arg.getContext()))
+               return true;
+             return ZRegsUsed++ < 8;
+           }
+           if (Ty->isFloatTy())
+             ZRegsUsed++;
+           return false;
          });
 }
 
 const MCPhysReg *
 AArch64RegisterInfo::getCalleeSavedRegs(const MachineFunction *MF) const {
   assert(MF && "Invalid MachineFunction pointer.");
 
   if (MF->getFunction().getCallingConv() == CallingConv::GHC)
     // GHC set of callee saved regs is empty as all those regs are
     // used for passing STG regs around

llvm/test/CodeGen/AArch64/sve-calling-convention-mixed.ll

 entry:
   %ptr1.bc = bitcast double * %ptr1 to <vscale x 8 x double> *
   store volatile <vscale x 8 x double> %x2, <vscale x 8 x double>* %ptr1.bc
   %ptr2.bc = bitcast double * %ptr2 to <vscale x 6 x double> *
   store volatile <vscale x 6 x double> %x3, <vscale x 6 x double>* %ptr2.bc
   ret double %x0
 }
 
+; Use AAVPCS, SVE register in z0-z7 used
+
+define void @aavpcs1(i32 %s0, i32 %s1, i32 %s2, i32 %s3, i32 %s4, i32 %s5, i32 %s6, <vscale x 4 x i32> %s7, <vscale x 4 x i32> %s8, <vscale x 4 x i32> %s9, <vscale x 4 x i32> %s10, <vscale x 4 x i32> %s11, <vscale x 4 x i32> %s12, <vscale x 4 x i32> %s13, <vscale x 4 x i32> %s14, <vscale x 4 x i32> %s15, <vscale x 4 x i32> %s16, i32 * %ptr) nounwind {
+; CHECK-LABEL: aavpcs1:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ldp x8, x9, [sp]
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1w { z3.s }, p0/z, [x8]
+; CHECK-NEXT:    ld1w { z24.s }, p0/z, [x7]
+; CHECK-NEXT:    st1w { z0.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z1.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z2.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z4.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z5.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z6.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z7.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z24.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z3.s }, p0, [x9]
+; CHECK-NEXT:    ret
+entry:
+  %ptr1.bc = bitcast i32 * %ptr to <vscale x 4 x i32> *
+  store volatile <vscale x 4 x i32> %s7, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s8, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s9, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s11, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s12, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s13, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s14, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s15, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s16, <vscale x 4 x i32>* %ptr1.bc
+  ret void
+}
+
+; Use AAVPCS, SVE register in z0-z7 used
+
+define void @aavpcs2(float %s0, float %s1, float %s2, float %s3, float %s4, float %s5, float %s6, <vscale x 4 x float> %s7, <vscale x 4 x float> %s8, <vscale x 4 x float> %s9, <vscale x 4 x float> %s10, <vscale x 4 x float> %s11, <vscale x 4 x float> %s12,<vscale x 4 x float> %s13,<vscale x 4 x float> %s14,<vscale x 4 x float> %s15,<vscale x 4 x float> %s16,float * %ptr) nounwind {
+; CHECK-LABEL: aavpcs2:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ldp x8, x9, [sp]
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x8]
+; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x7]
+; CHECK-NEXT:    ld1w { z2.s }, p0/z, [x6]
+; CHECK-NEXT:    ld1w { z3.s }, p0/z, [x5]
+; CHECK-NEXT:    ld1w { z4.s }, p0/z, [x4]
+; CHECK-NEXT:    ld1w { z5.s }, p0/z, [x3]
+; CHECK-NEXT:    ld1w { z6.s }, p0/z, [x1]
+; CHECK-NEXT:    ld1w { z24.s }, p0/z, [x0]
+; CHECK-NEXT:    st1w { z7.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z24.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z6.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z5.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z4.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z3.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z2.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z1.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z0.s }, p0, [x9]
+; CHECK-NEXT:    ret
+entry:
+  %ptr1.bc = bitcast float * %ptr to <vscale x 4 x float> *
+  store volatile <vscale x 4 x float> %s7, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s8, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s9, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s11, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s12, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s13, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s14, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s15, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s16, <vscale x 4 x float>* %ptr1.bc
+  ret void
+}
+
+; Use AAVPCS, no SVE register in z0-z7 used (floats occupy z0-z7) but predicate arg is used
+
+define void @aavpcs3(float %s0, float %s1, float %s2, float %s3, float %s4, float %s5, float %s6, float %s7, <vscale x 4 x float> %s8, <vscale x 4 x float> %s9, <vscale x 4 x float> %s10, <vscale x 4 x float> %s11, <vscale x 4 x float> %s12, <vscale x 4 x float> %s13, <vscale x 4 x float> %s14, <vscale x 4 x float> %s15, <vscale x 4 x float> %s16, <vscale x 4 x float> %s17, <vscale x 16 x i1> %p0, float * %ptr) nounwind {
+; CHECK-LABEL: aavpcs3:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ldr x8, [sp]
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x8]
+; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x7]
+; CHECK-NEXT:    ld1w { z2.s }, p0/z, [x6]
+; CHECK-NEXT:    ld1w { z3.s }, p0/z, [x5]
+; CHECK-NEXT:    ld1w { z4.s }, p0/z, [x4]
+; CHECK-NEXT:    ld1w { z5.s }, p0/z, [x3]
+; CHECK-NEXT:    ld1w { z6.s }, p0/z, [x2]
+; CHECK-NEXT:    ld1w { z7.s }, p0/z, [x1]
+; CHECK-NEXT:    ld1w { z24.s }, p0/z, [x0]
+; CHECK-NEXT:    ldr x8, [sp, #16]
+; CHECK-NEXT:    st1w { z24.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z7.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z6.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z5.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z4.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z3.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z2.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z1.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z0.s }, p0, [x8]
+; CHECK-NEXT:    ret
+entry:
+  %ptr1.bc = bitcast float * %ptr to <vscale x 4 x float> *
+  store volatile <vscale x 4 x float> %s8, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s9, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s10, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s11, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s12, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s13, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s14, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s15, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s16, <vscale x 4 x float>* %ptr1.bc
+  ret void
+}
+
+; use AAVPCS, SVE register in z0-z7 used (i32s dont occupy z0-z7)
+
+define void @aavpcs4(i32 %s0, i32 %s1, i32 %s2, i32 %s3, i32 %s4, i32 %s5, i32 %s6, i32 %s7, <vscale x 4 x i32> %s8, <vscale x 4 x i32> %s9, <vscale x 4 x i32> %s10, <vscale x 4 x i32> %s11, <vscale x 4 x i32> %s12, <vscale x 4 x i32> %s13, <vscale x 4 x i32> %s14, <vscale x 4 x i32> %s15, <vscale x 4 x i32> %s16, <vscale x 4 x i32> %s17, i32 * %ptr) nounwind {
+; CHECK-LABEL: aavpcs4:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ldr x8, [sp]
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ldr x9, [sp, #16]
+; CHECK-NEXT:    ld1w { z24.s }, p0/z, [x8]
+; CHECK-NEXT:    st1w { z0.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z1.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z2.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z3.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z4.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z5.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z6.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z7.s }, p0, [x9]
+; CHECK-NEXT:    st1w { z24.s }, p0, [x9]
+; CHECK-NEXT:    ret
+entry:
+  %ptr1.bc = bitcast i32 * %ptr to <vscale x 4 x i32> *
+  store volatile <vscale x 4 x i32> %s8, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s9, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s10, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s11, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s12, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s13, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s14, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s15, <vscale x 4 x i32>* %ptr1.bc
+  store volatile <vscale x 4 x i32> %s16, <vscale x 4 x i32>* %ptr1.bc
+  ret void
+}
+
+; Use AAPCS, no SVE register in z0-7 used (floats occupy z0-z7)
+
+define void @aapcs1(float %s0, float %s1, float %s2, float %s3, float %s4, float %s5, float %s6, float %s7, <vscale x 4 x float> %s8, <vscale x 4 x float> %s9, <vscale x 4 x float> %s10, <vscale x 4 x float> %s11, <vscale x 4 x float> %s12, <vscale x 4 x float> %s13, <vscale x 4 x float> %s14, <vscale x 4 x float> %s15, <vscale x 4 x float> %s16, <vscale x 4 x float> %s17, float * %ptr) nounwind {
+; CHECK-LABEL: aapcs1:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ldr x8, [sp]
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x8]
+; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x7]
+; CHECK-NEXT:    ld1w { z2.s }, p0/z, [x6]
+; CHECK-NEXT:    ld1w { z3.s }, p0/z, [x5]
+; CHECK-NEXT:    ld1w { z4.s }, p0/z, [x4]
+; CHECK-NEXT:    ld1w { z5.s }, p0/z, [x3]
+; CHECK-NEXT:    ld1w { z6.s }, p0/z, [x2]
+; CHECK-NEXT:    ld1w { z7.s }, p0/z, [x1]
+; CHECK-NEXT:    ld1w { z16.s }, p0/z, [x0]
+; CHECK-NEXT:    ldr x8, [sp, #16]
+; CHECK-NEXT:    st1w { z16.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z7.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z6.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z5.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z4.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z3.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z2.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z1.s }, p0, [x8]
+; CHECK-NEXT:    st1w { z0.s }, p0, [x8]
+; CHECK-NEXT:    ret
+entry:
+  %ptr1.bc = bitcast float * %ptr to <vscale x 4 x float> *
+  store volatile <vscale x 4 x float> %s8, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s9, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s10, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s11, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s12, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s13, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s14, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s15, <vscale x 4 x float>* %ptr1.bc
+  store volatile <vscale x 4 x float> %s16, <vscale x 4 x float>* %ptr1.bc
+  ret void
+}
+
 declare float @callee1(float, <vscale x 8 x double>, <vscale x 8 x double>, <vscale x 2 x double>)
 declare float @callee2(i32, i32, i32, i32, i32, i32, i32, i32, float, <vscale x 8 x double>, <vscale x 8 x double>)
 declare float @callee3(float, float, <vscale x 8 x double>, <vscale x 6 x double>, <vscale x 2 x double>)
 
 declare <vscale x 16 x i1> @llvm.aarch64.sve.ptrue.nxv16i1(i32 immarg)
 declare <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1>)
 declare <vscale x 8 x double> @llvm.aarch64.sve.ld4.nxv8f64.nxv2i1(<vscale x 2 x i1>, double*)
 declare <vscale x 6 x double> @llvm.aarch64.sve.ld3.nxv6f64.nxv2i1(<vscale x 2 x i1>, double*)
 declare <vscale x 2 x double> @llvm.aarch64.sve.ld1.nxv2f64(<vscale x 2 x i1>, double*)
 declare double @llvm.aarch64.sve.faddv.nxv2f64(<vscale x 2 x i1>, <vscale x 2 x double>)
 declare <vscale x 2 x double> @llvm.aarch64.sve.tuple.get.nxv2f64.nxv8f64(<vscale x 8 x double>, i32 immarg)
 declare <vscale x 2 x double> @llvm.aarch64.sve.tuple.get.nxv2f64.nxv6f64(<vscale x 6 x double>, i32 immarg)