[AArch64][SVE] Preserve full vector regs over EH edge.

[sdesmalen]

Summary

Unwinders may only preserve the lower 64bits of Neon and SVE registers,
as only the registers in the base ABI are guaranteed to be preserved
over the exception edge. The caller will need to preserve additional
registers for when the call throws an exception and the unwinder has
tried to recover state.

For e.g.

svint32_t bar(svint32_t);
svint32_t foo(svint32_t x, bool *err) {
  try { bar(x); } catch (...) { *err = true; }
  return x;
}

z0 needs to be spilled before the call to bar(x) and reloaded before
returning from foo, as the exception handler may have clobbered z0.

[sdesmalen]

Fixed patch to look at call instruction and callee.

[rsandifo-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

4710 ↗

(On Diff #281237)

Like we talked about off-list:

• it might be better to drop the doesNotThrow check. We should only have an InvokedInst if the call can “return” to a known landing pad.
• It might be better to do this by adding clobbers at the start of the landing pad instead, a bit like the existing getBeginClobberMask. That would also handle the corresponding problem for aarch64_vector_pcs functions and would mean that normal returns weren't penalised by the exceptional case.

[sdesmalen]

• Set clobber mask for EH pad block, instead of changing the CC of the call.
• Rebased patch

[rsandifo-arm]

Thanks the update, the idea looks good in general. Two questions though:

• Is it worth having two functions? I thought it might be simpler to have just getCustomEHPadPreservedMask(), with nullptr meaning “no clobbers”.
• It looks from the tests like the function isn't saving or restoring z8-z23 and p4-p15, is that right? That's fine for landing pads that can't return directly to the callee (e.g. because they rethrow). But in these tests the landing pad does return to the caller. Since the extra callee-saved registers are clobbered on the EH edge, we need to save and restore those registers ourselves on control paths that include an EH edge and a return.

[sdesmalen]

• Made getCustomEHPadPreservedMask() return nullptr by default (meaning 'no clobbers')
• Fixed missing callee saves/restores for clobbered registers.
• Updated tests (and also added a RUN line for regalloc=fast)

[sdesmalen]

In D84737#2216322, @rsandifo-arm wrote:

Thanks the update, the idea looks good in general.

Thanks for the feedback!

Two questions though:

• Is it worth having two functions? I thought it might be simpler to have just getCustomEHPadPreservedMask(), with nullptr meaning “no clobbers”.

The reason I had two functions at first was because TargetRegisterInfo::getCallPreservedMask() returns nullptr as well, but in that case this means "clobbers everything", where for this function it means the opposite. I have changed the default now, and clarified the default in the comment of the method declaration.

• It looks from the tests like the function isn't saving or restoring z8-z23 and p4-p15, is that right? That's fine for landing pads that can't return directly to the callee (e.g. because they rethrow). But in these tests the landing pad does return to the caller. Since the extra callee-saved registers are clobbered on the EH edge, we need to save and restore those registers ourselves on control paths that include an EH edge and a return.

Yes you're right, thanks for pointing that out! I have fixed that in the latest revision by calling addPhysRegsUsedFromRegMask for the mask returned by getCustomEHPadPreservedMask. I tried doing this for the mask returned by MachineBasicBlock::getBeginClobberMask (which seemed to me like the right thing to do considering it's name), but that broke mosts tests for Windows EH (on both AArch64 as well as X86).

[rsandifo-arm]

LGTM, but I don't think I should be the one to accept.

[efriedma]

llvm/lib/CodeGen/RegAllocFast.cpp

1215

This seems like a really weird location for this call; it's modifying function-global state in the middle of register allocation.

I'd prefer to do one of the following:

1. Attach the register mask to the landingpad instruction itself.
2. Update the function-global state when we construct the landing pad instruction/node.

[sdesmalen]

• Updated patch with option 2: Update the function-global state when we construct the landing pad instruction/node.

[efriedma]

Does MIRParser also need to be modified?

[sdesmalen]

• MIRParser now correctly marks registers as clobbered and added corresponding .mir test file.
• Updated RUN lines of unwind-preserved.ll to test with/without global isel.

[sdesmalen]

In D84737#2242900, @efriedma wrote:

Does MIRParser also need to be modified?

You're right, that was missing. Thanks for pointing out!

llvm/lib/CodeGen/RegAllocFast.cpp

1215

This seems like a really weird location for this call; it's modifying function-global state in the middle of register allocation.

The same seems to happen in allocateInstruction(MI) (see line 989) for an instruction when one of the operands is a register mask.

[sdesmalen]

llvm/lib/CodeGen/RegAllocFast.cpp

1215

Sorry, please ignore this comment (I wrote it before I updated this revision the other day, but forgot to remove it).

[sdesmalen]

Hi @efriedma, what do you think of the latest changes to this patch?

(I'm really sorry for the quick prod, but I'm (perhaps falsely) hoping this fix may still have a small chance to make it into the release after it lands. Of course not before you're happy with it first)

[efriedma]

LGTM

I'm not completely confident this is the best approach; it might come out cleaner if the clobber is attached to a MachineInstr. As it is, it's depending on sort of fragile invariants to ensure the value isn't live across the unwind edge: every register allocator has to be aware of getCustomEHPadPreservedMask, and any post-regalloc optimizations that the unwind edge is special. But I can't come up with any concrete example that would actually cause trouble.

[sdesmalen]

Thanks for reviewing the patch!

In D84737#2250735, @efriedma wrote:

I'm not completely confident this is the best approach; it might come out cleaner if the clobber is attached to a MachineInstr.

What MachineInstr would it need to be attached to? When adding the clobbers to the invoke instruction that penalises the normal return. When adding the clobbers to e.g. the EH_LABEL instruction in the unwind-block, the register spills are added at the start of the unwinding block rather than before the call instruction.

As it is, it's depending on sort of fragile invariants to ensure the value isn't live across the unwind edge: every register allocator has to be aware of getCustomEHPadPreservedMask, and any post-regalloc optimizations that the unwind edge is special.

The clobber mask is applied when calculating the live intervals which is used as input to the register allocator, so I would think that register allocators themselves don't need to be aware of this as well. I.e. Once this information is reflected in the live ranges, there isn't anything that the register allocator should do to change that.

[efriedma]

What MachineInstr would it need to be attached to?

We'd need some instruction that's guaranteed to be at the beginning of the block. Probably EH_LABEL should fall into that category. Granted, I guess the register allocators aren't aware of this restriction at the moment.

The clobber mask is applied when calculating the live intervals which is used as input to the register allocator, so I would think that register allocators themselves don't need to be aware of this as well. I.e. Once this information is reflected in the live ranges, there isn't anything that the register allocator should do to change that.

We have at least one register allocator that doesn't use LiveIntervals. Fast regalloc doesn't keep any values live across blocks, though

llvm/include/llvm/CodeGen/TargetRegisterInfo.h

public:
   /// registers.
   ///
   virtual const uint32_t *getCallPreservedMask(const MachineFunction &MF,
                                                CallingConv::ID) const {
     // The default mask clobbers everything.  All targets should override.
     return nullptr;
   }
 
+  /// Return a register mask for the registers preserved by the unwinder,
+  /// or nullptr if no custom mask is needed.
+  virtual const uint32_t *
+  getCustomEHPadPreservedMask(const MachineFunction &MF) const {
+    return nullptr;
+  }
+
   /// Return a register mask that clobbers everything.
   virtual const uint32_t *getNoPreservedMask() const {
     llvm_unreachable("target does not provide no preserved mask");
   }
 
   /// Return a list of all of the registers which are clobbered "inside" a call
   /// to the given function. For example, these might be needed for PLT
   /// sequences of long-branch veneers.

llvm/lib/CodeGen/LiveIntervals.cpp

for (const MachineBasicBlock &MBB : *MF) {
     RMB.first = RegMaskSlots.size();
 
     // Some block starts, such as EH funclets, create masks.
     if (const uint32_t *Mask = MBB.getBeginClobberMask(TRI)) {
       RegMaskSlots.push_back(Indexes->getMBBStartIdx(&MBB));
       RegMaskBits.push_back(Mask);
     }
 
+    // Unwinders may clobber additional registers.
+    // FIXME: This functionality can possibly be merged into
+    // MachineBasicBlock::getBeginClobberMask().
+    if (MBB.isEHPad())
+      if (auto *Mask = TRI->getCustomEHPadPreservedMask(*MBB.getParent())) {
+        RegMaskSlots.push_back(Indexes->getMBBStartIdx(&MBB));
+        RegMaskBits.push_back(Mask);
+      }
+
     for (const MachineInstr &MI : MBB) {
       for (const MachineOperand &MO : MI.operands()) {
         if (!MO.isRegMask())
           continue;
         RegMaskSlots.push_back(Indexes->getInstructionIndex(MI).getRegSlot());
         RegMaskBits.push_back(MO.getRegMask());
       }
     }

llvm/lib/CodeGen/MIRParser/MIRParser.cpp

bool MIRParserImpl::setupRegisterInfo(const PerFunctionMIParsingState &PFS,
 
   for (auto P : PFS.VRegInfos) {
     const VRegInfo &Info = *P.second;
     populateVRegInfo(Info, Twine(P.first));
   }
 
   // Compute MachineRegisterInfo::UsedPhysRegMask
   for (const MachineBasicBlock &MBB : MF) {
+    // Make sure MRI knows about registers clobbered by unwinder.
+    const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
+    if (MBB.isEHPad())
+      if (auto *RegMask = TRI->getCustomEHPadPreservedMask(MF))
+        MRI.addPhysRegsUsedFromRegMask(RegMask);
+
     for (const MachineInstr &MI : MBB) {
       for (const MachineOperand &MO : MI.operands()) {
         if (!MO.isRegMask())
           continue;
         MRI.addPhysRegsUsedFromRegMask(MO.getRegMask());
       }
     }
   }

llvm/lib/CodeGen/RegAllocFast.cpp

for (const MachineBasicBlock::RegisterMaskPair &LI : MBB.liveins())
     if (MRI->isAllocatable(LI.PhysReg))
       definePhysReg(MII, LI.PhysReg, regReserved);
 
   VirtDead.clear();
   Coalesced.clear();
 
   // Otherwise, sequentially allocate each instruction in the MBB.
   for (MachineInstr &MI : MBB) {
+    // Make sure MRI knows about registers clobbered by unwinder.
+    if (MBB.isEHPad())
+      if (auto *RegMask = TRI->getCustomEHPadPreservedMask(*MBB.getParent()))
+        MRI->addPhysRegsUsedFromRegMask(RegMask);
+
     LLVM_DEBUG(
       dbgs() << "\n>> " << MI << "Regs:";
       dumpState()
     );
 
     // Special handling for debug values. Note that they are not allowed to
     // affect codegen of the other instructions in any way.
     if (MI.isDebugValue()) {

llvm/lib/CodeGen/VirtRegMap.cpp

void VirtRegRewriter::rewrite() {
   bool NoSubRegLiveness = !MRI->subRegLivenessEnabled();
   SmallVector<Register, 8> SuperDeads;
   SmallVector<Register, 8> SuperDefs;
   SmallVector<Register, 8> SuperKills;
 
   for (MachineFunction::iterator MBBI = MF->begin(), MBBE = MF->end();
        MBBI != MBBE; ++MBBI) {
     LLVM_DEBUG(MBBI->print(dbgs(), Indexes));
+
+    // Make sure MRI knows about registers clobbered by unwinder.
+    if (MBBI->isEHPad())
+      if (auto *RegMask = TRI->getCustomEHPadPreservedMask(*MBBI->getParent()))
+        MRI->addPhysRegsUsedFromRegMask(RegMask);
+
     for (MachineBasicBlock::instr_iterator
            MII = MBBI->instr_begin(), MIE = MBBI->instr_end(); MII != MIE;) {
       MachineInstr *MI = &*MII;
       ++MII;
 
       for (MachineInstr::mop_iterator MOI = MI->operands_begin(),
            MOE = MI->operands_end(); MOI != MOE; ++MOI) {
         MachineOperand &MO = *MOI;

llvm/lib/Target/AArch64/AArch64RegisterInfo.h

public:
 
   // Calls involved in thread-local variable lookup save more registers than
   // normal calls, so they need a different mask to represent this.
   const uint32_t *getTLSCallPreservedMask() const;
 
   // Funclets on ARM64 Windows don't preserve any registers.
   const uint32_t *getNoPreservedMask() const override;
 
+  // Unwinders may not preserve all Neon and SVE registers.
+  const uint32_t *
+  getCustomEHPadPreservedMask(const MachineFunction &MF) const override;
+
   /// getThisReturnPreservedMask - Returns a call preserved mask specific to the
   /// case that 'returned' is on an i64 first argument if the calling convention
   /// is one that can (partially) model this attribute with a preserved mask
   /// (i.e. it is a calling convention that uses the same register for the first
   /// i64 argument and an i64 return value)
   ///
   /// Should return NULL in the case that the calling convention does not have
   /// this property

llvm/lib/Target/AArch64/AArch64RegisterInfo.cpp

return SCS ? CSR_AArch64_AAPCS_SwiftError_SCS_RegMask
                : CSR_AArch64_AAPCS_SwiftError_RegMask;
   if (CC == CallingConv::PreserveMost)
     return SCS ? CSR_AArch64_RT_MostRegs_SCS_RegMask
                : CSR_AArch64_RT_MostRegs_RegMask;
   else
     return SCS ? CSR_AArch64_AAPCS_SCS_RegMask : CSR_AArch64_AAPCS_RegMask;
 }
 
+const uint32_t *AArch64RegisterInfo::getCustomEHPadPreservedMask(
+    const MachineFunction &MF) const {
+  if (MF.getSubtarget<AArch64Subtarget>().isTargetLinux())
+    return CSR_AArch64_AAPCS_RegMask;
+
+  return nullptr;
+}
+
 const uint32_t *AArch64RegisterInfo::getTLSCallPreservedMask() const {
   if (TT.isOSDarwin())
     return CSR_Darwin_AArch64_TLS_RegMask;
 
   assert(TT.isOSBinFormatELF() && "Invalid target");
   return CSR_AArch64_TLS_ELF_RegMask;
 }
 

llvm/test/CodeGen/AArch64/unwind-preserved.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --extra_scrub
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve -O0 -regalloc=fast < %s | FileCheck %s --check-prefix=FAST
+
+; Test that z0 is saved/restored, as the unwinder may only retain the low 64bits (d0).
+define <vscale x 4 x i32> @invoke_callee_may_throw_sve(<vscale x 4 x i32> %v) personality i8 0 {
+; CHECK-LABEL: invoke_callee_may_throw_sve:
+; CHECK:       .Lfunc_begin0:
+; CHECK-NEXT:    .cfi_startproc
+; CHECK-NEXT:  // %bb.0:
+; CHECK-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT:    addvl sp, sp, #-18
+; CHECK-NEXT:    str p15, [sp, #4, mul vl] // 2-byte Folded Spill
+; CHECK-NEXT:    str p14, [sp, #5, mul vl] // 2-byte Folded Spill
+; CHECK-NEXT:    str p13, [sp, #6, mul vl] // 2-byte Folded Spill
+; CHECK-NEXT:    str p12, [sp, #7, mul vl] // 2-byte Folded Spill
+; CHECK-NEXT:    str p11, [sp, #8, mul vl] // 2-byte Folded Spill
+; CHECK-NEXT:    str p10, [sp, #9, mul vl] // 2-byte Folded Spill
+; CHECK-NEXT:    str p9, [sp, #10, mul vl] // 2-byte Folded Spill
+; CHECK-NEXT:    str p8, [sp, #11, mul vl] // 2-byte Folded Spill
+; CHECK-NEXT:    str p7, [sp, #12, mul vl] // 2-byte Folded Spill
+; CHECK-NEXT:    str p6, [sp, #13, mul vl] // 2-byte Folded Spill
+; CHECK-NEXT:    str p5, [sp, #14, mul vl] // 2-byte Folded Spill
+; CHECK-NEXT:    str p4, [sp, #15, mul vl] // 2-byte Folded Spill
+; CHECK-NEXT:    str z23, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z22, [sp, #3, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z21, [sp, #4, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z20, [sp, #5, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z19, [sp, #6, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z18, [sp, #7, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z17, [sp, #8, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z16, [sp, #9, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z15, [sp, #10, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z14, [sp, #11, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z13, [sp, #12, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z12, [sp, #13, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z11, [sp, #14, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z10, [sp, #15, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z9, [sp, #16, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    str z8, [sp, #17, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT:    addvl sp, sp, #-1
+; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x8f, 0x00, 0x11, 0x10, 0x22, 0x11, 0x98, 0x01, 0x92, 0x2e, 0x00, 0x1e, 0x22 // sp + 16 + 152 * VG
+; CHECK-NEXT:    .cfi_escape 0x10, 0x48, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x78, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d8 @ cfa - 16 - 8 * VG
+; CHECK-NEXT:    .cfi_escape 0x10, 0x49, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x70, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d9 @ cfa - 16 - 16 * VG
+; CHECK-NEXT:    .cfi_escape 0x10, 0x4a, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x68, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d10 @ cfa - 16 - 24 * VG
+; CHECK-NEXT:    .cfi_escape 0x10, 0x4b, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x60, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d11 @ cfa - 16 - 32 * VG
+; CHECK-NEXT:    .cfi_escape 0x10, 0x4c, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x58, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d12 @ cfa - 16 - 40 * VG
+; CHECK-NEXT:    .cfi_escape 0x10, 0x4d, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x50, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d13 @ cfa - 16 - 48 * VG
+; CHECK-NEXT:    .cfi_escape 0x10, 0x4e, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x48, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d14 @ cfa - 16 - 56 * VG
+; CHECK-NEXT:    .cfi_escape 0x10, 0x4f, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x40, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d15 @ cfa - 16 - 64 * VG
+; CHECK-NEXT:    .cfi_offset w30, -8
+; CHECK-NEXT:    .cfi_offset w29, -16
+; CHECK-NEXT:  .Ltmp0:
+; CHECK-NEXT:    str z0, [sp] // 16-byte Folded Spill
+; CHECK-NEXT:    bl may_throw_sve
+; CHECK-NEXT:  .Ltmp1:
+; CHECK-NEXT:  .LBB0_1: // %.Lcontinue
+; CHECK-NEXT:    addvl sp, sp, #1
+; CHECK-NEXT:    ldr p15, [sp, #4, mul vl] // 2-byte Folded Reload
+; CHECK-NEXT:    ldr p14, [sp, #5, mul vl] // 2-byte Folded Reload
+; CHECK-NEXT:    ldr p13, [sp, #6, mul vl] // 2-byte Folded Reload
+; CHECK-NEXT:    ldr p12, [sp, #7, mul vl] // 2-byte Folded Reload
+; CHECK-NEXT:    ldr p11, [sp, #8, mul vl] // 2-byte Folded Reload
+; CHECK-NEXT:    ldr p10, [sp, #9, mul vl] // 2-byte Folded Reload
+; CHECK-NEXT:    ldr p9, [sp, #10, mul vl] // 2-byte Folded Reload
+; CHECK-NEXT:    ldr p8, [sp, #11, mul vl] // 2-byte Folded Reload
+; CHECK-NEXT:    ldr p7, [sp, #12, mul vl] // 2-byte Folded Reload
+; CHECK-NEXT:    ldr p6, [sp, #13, mul vl] // 2-byte Folded Reload
+; CHECK-NEXT:    ldr p5, [sp, #14, mul vl] // 2-byte Folded Reload
+; CHECK-NEXT:    ldr p4, [sp, #15, mul vl] // 2-byte Folded Reload
+; CHECK-NEXT:    ldr z23, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z22, [sp, #3, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z21, [sp, #4, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z20, [sp, #5, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z19, [sp, #6, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z18, [sp, #7, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z17, [sp, #8, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z16, [sp, #9, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z15, [sp, #10, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z14, [sp, #11, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z13, [sp, #12, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z12, [sp, #13, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z11, [sp, #14, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z10, [sp, #15, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z9, [sp, #16, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr z8, [sp, #17, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT:    addvl sp, sp, #18
+; CHECK-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT:    ret
+; CHECK-NEXT:  .LBB0_2: // %.Lunwind
+; CHECK-NEXT:  .Ltmp2:
+; CHECK-NEXT:    ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT:    b .LBB0_1
+;
+; FAST-LABEL: invoke_callee_may_throw_sve:
+; FAST:       .Lfunc_begin0:
+; FAST-NEXT:    .cfi_startproc
+; FAST-NEXT:  // %bb.0:
+; FAST-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; FAST-NEXT:    addvl sp, sp, #-18
+; FAST-NEXT:    str p15, [sp, #4, mul vl] // 2-byte Folded Spill
+; FAST-NEXT:    str p14, [sp, #5, mul vl] // 2-byte Folded Spill
+; FAST-NEXT:    str p13, [sp, #6, mul vl] // 2-byte Folded Spill
+; FAST-NEXT:    str p12, [sp, #7, mul vl] // 2-byte Folded Spill
+; FAST-NEXT:    str p11, [sp, #8, mul vl] // 2-byte Folded Spill
+; FAST-NEXT:    str p10, [sp, #9, mul vl] // 2-byte Folded Spill
+; FAST-NEXT:    str p9, [sp, #10, mul vl] // 2-byte Folded Spill
+; FAST-NEXT:    str p8, [sp, #11, mul vl] // 2-byte Folded Spill
+; FAST-NEXT:    str p7, [sp, #12, mul vl] // 2-byte Folded Spill
+; FAST-NEXT:    str p6, [sp, #13, mul vl] // 2-byte Folded Spill
+; FAST-NEXT:    str p5, [sp, #14, mul vl] // 2-byte Folded Spill
+; FAST-NEXT:    str p4, [sp, #15, mul vl] // 2-byte Folded Spill
+; FAST-NEXT:    str z23, [sp, #2, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z22, [sp, #3, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z21, [sp, #4, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z20, [sp, #5, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z19, [sp, #6, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z18, [sp, #7, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z17, [sp, #8, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z16, [sp, #9, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z15, [sp, #10, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z14, [sp, #11, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z13, [sp, #12, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z12, [sp, #13, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z11, [sp, #14, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z10, [sp, #15, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z9, [sp, #16, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    str z8, [sp, #17, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    addvl sp, sp, #-2
+; FAST-NEXT:    .cfi_escape 0x0f, 0x0d, 0x8f, 0x00, 0x11, 0x10, 0x22, 0x11, 0xa0, 0x01, 0x92, 0x2e, 0x00, 0x1e, 0x22 // sp + 16 + 160 * VG
+; FAST-NEXT:    .cfi_escape 0x10, 0x48, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x78, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d8 @ cfa - 16 - 8 * VG
+; FAST-NEXT:    .cfi_escape 0x10, 0x49, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x70, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d9 @ cfa - 16 - 16 * VG
+; FAST-NEXT:    .cfi_escape 0x10, 0x4a, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x68, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d10 @ cfa - 16 - 24 * VG
+; FAST-NEXT:    .cfi_escape 0x10, 0x4b, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x60, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d11 @ cfa - 16 - 32 * VG
+; FAST-NEXT:    .cfi_escape 0x10, 0x4c, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x58, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d12 @ cfa - 16 - 40 * VG
+; FAST-NEXT:    .cfi_escape 0x10, 0x4d, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x50, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d13 @ cfa - 16 - 48 * VG
+; FAST-NEXT:    .cfi_escape 0x10, 0x4e, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x48, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d14 @ cfa - 16 - 56 * VG
+; FAST-NEXT:    .cfi_escape 0x10, 0x4f, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x40, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d15 @ cfa - 16 - 64 * VG
+; FAST-NEXT:    .cfi_offset w30, -8
+; FAST-NEXT:    .cfi_offset w29, -16
+; FAST-NEXT:  .Ltmp0:
+; FAST-NEXT:    str z0, [sp, #1, mul vl] // 16-byte Folded Spill
+; FAST-NEXT:    bl may_throw_sve
+; FAST-NEXT:  .Ltmp1:
+; FAST-NEXT:    str z0, [sp] // 16-byte Folded Spill
+; FAST-NEXT:    b .LBB0_1
+; FAST-NEXT:  .LBB0_1: // %.Lcontinue
+; FAST-NEXT:    ldr z0, [sp] // 16-byte Folded Reload
+; FAST-NEXT:    addvl sp, sp, #2
+; FAST-NEXT:    ldr p15, [sp, #4, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p14, [sp, #5, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p13, [sp, #6, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p12, [sp, #7, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p11, [sp, #8, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p10, [sp, #9, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p9, [sp, #10, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p8, [sp, #11, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p7, [sp, #12, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p6, [sp, #13, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p5, [sp, #14, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p4, [sp, #15, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr z23, [sp, #2, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z22, [sp, #3, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z21, [sp, #4, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z20, [sp, #5, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z19, [sp, #6, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z18, [sp, #7, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z17, [sp, #8, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z16, [sp, #9, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z15, [sp, #10, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z14, [sp, #11, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z13, [sp, #12, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z12, [sp, #13, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z11, [sp, #14, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z10, [sp, #15, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z9, [sp, #16, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z8, [sp, #17, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    addvl sp, sp, #18
+; FAST-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; FAST-NEXT:    ret
+; FAST-NEXT:  .LBB0_2: // %.Lunwind
+; FAST-NEXT:  .Ltmp2:
+; FAST-NEXT:    ldr z0, [sp, #1, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    addvl sp, sp, #2
+; FAST-NEXT:    ldr p15, [sp, #4, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p14, [sp, #5, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p13, [sp, #6, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p12, [sp, #7, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p11, [sp, #8, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p10, [sp, #9, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p9, [sp, #10, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p8, [sp, #11, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p7, [sp, #12, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p6, [sp, #13, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p5, [sp, #14, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr p4, [sp, #15, mul vl] // 2-byte Folded Reload
+; FAST-NEXT:    ldr z23, [sp, #2, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z22, [sp, #3, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z21, [sp, #4, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z20, [sp, #5, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z19, [sp, #6, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z18, [sp, #7, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z17, [sp, #8, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z16, [sp, #9, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z15, [sp, #10, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z14, [sp, #11, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z13, [sp, #12, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z12, [sp, #13, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z11, [sp, #14, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z10, [sp, #15, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z9, [sp, #16, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    ldr z8, [sp, #17, mul vl] // 16-byte Folded Reload
+; FAST-NEXT:    addvl sp, sp, #18
+; FAST-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; FAST-NEXT:    ret
+  %result = invoke <vscale x 4 x i32> @may_throw_sve(<vscale x 4 x i32> %v) to label %.Lcontinue unwind label %.Lunwind
+.Lcontinue:
+  ret <vscale x 4 x i32> %result
+.Lunwind:
+  %lp = landingpad { i8*, i32 } cleanup
+  ret <vscale x 4 x i32> %v;
+}
+
+declare <vscale x 4 x i32> @may_throw_sve(<vscale x 4 x i32> %v);
+
+
+; Test that q0 is saved/restored, as the unwinder may only retain the low 64bits (d0).
+define aarch64_vector_pcs <4 x i32> @invoke_callee_may_throw_neon(<4 x i32> %v) personality i8 0 {
+; CHECK-LABEL: invoke_callee_may_throw_neon:
+; CHECK:       .Lfunc_begin1:
+; CHECK-NEXT:    .cfi_startproc
+; CHECK-NEXT:  // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #288 // =288
+; CHECK-NEXT:    stp q23, q22, [sp, #16] // 32-byte Folded Spill
+; CHECK-NEXT:    stp q21, q20, [sp, #48] // 32-byte Folded Spill
+; CHECK-NEXT:    stp q19, q18, [sp, #80] // 32-byte Folded Spill
+; CHECK-NEXT:    stp q17, q16, [sp, #112] // 32-byte Folded Spill
+; CHECK-NEXT:    stp q15, q14, [sp, #144] // 32-byte Folded Spill
+; CHECK-NEXT:    stp q13, q12, [sp, #176] // 32-byte Folded Spill
+; CHECK-NEXT:    stp q11, q10, [sp, #208] // 32-byte Folded Spill
+; CHECK-NEXT:    stp q9, q8, [sp, #240] // 32-byte Folded Spill
+; CHECK-NEXT:    stp x29, x30, [sp, #272] // 16-byte Folded Spill
+; CHECK-NEXT:    .cfi_def_cfa_offset 288
+; CHECK-NEXT:    .cfi_offset w30, -8
+; CHECK-NEXT:    .cfi_offset w29, -16
+; CHECK-NEXT:    .cfi_offset b8, -32
+; CHECK-NEXT:    .cfi_offset b9, -48
+; CHECK-NEXT:    .cfi_offset b10, -64
+; CHECK-NEXT:    .cfi_offset b11, -80
+; CHECK-NEXT:    .cfi_offset b12, -96
+; CHECK-NEXT:    .cfi_offset b13, -112
+; CHECK-NEXT:    .cfi_offset b14, -128
+; CHECK-NEXT:    .cfi_offset b15, -144
+; CHECK-NEXT:    .cfi_offset b16, -160
+; CHECK-NEXT:    .cfi_offset b17, -176
+; CHECK-NEXT:    .cfi_offset b18, -192
+; CHECK-NEXT:    .cfi_offset b19, -208
+; CHECK-NEXT:    .cfi_offset b20, -224
+; CHECK-NEXT:    .cfi_offset b21, -240
+; CHECK-NEXT:    .cfi_offset b22, -256
+; CHECK-NEXT:    .cfi_offset b23, -272
+; CHECK-NEXT:  .Ltmp3:
+; CHECK-NEXT:    str q0, [sp] // 16-byte Folded Spill
+; CHECK-NEXT:    bl may_throw_neon
+; CHECK-NEXT:  .Ltmp4:
+; CHECK-NEXT:  .LBB1_1: // %.Lcontinue
+; CHECK-NEXT:    ldp x29, x30, [sp, #272] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp q9, q8, [sp, #240] // 32-byte Folded Reload
+; CHECK-NEXT:    ldp q11, q10, [sp, #208] // 32-byte Folded Reload
+; CHECK-NEXT:    ldp q13, q12, [sp, #176] // 32-byte Folded Reload
+; CHECK-NEXT:    ldp q15, q14, [sp, #144] // 32-byte Folded Reload
+; CHECK-NEXT:    ldp q17, q16, [sp, #112] // 32-byte Folded Reload
+; CHECK-NEXT:    ldp q19, q18, [sp, #80] // 32-byte Folded Reload
+; CHECK-NEXT:    ldp q21, q20, [sp, #48] // 32-byte Folded Reload
+; CHECK-NEXT:    ldp q23, q22, [sp, #16] // 32-byte Folded Reload
+; CHECK-NEXT:    add sp, sp, #288 // =288
+; CHECK-NEXT:    ret
+; CHECK-NEXT:  .LBB1_2: // %.Lunwind
+; CHECK-NEXT:  .Ltmp5:
+; CHECK-NEXT:    ldr q0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT:    b .LBB1_1
+;
+; FAST-LABEL: invoke_callee_may_throw_neon:
+; FAST:       .Lfunc_begin1:
+; FAST-NEXT:    .cfi_startproc
+; FAST-NEXT:  // %bb.0:
+; FAST-NEXT:    sub sp, sp, #304 // =304
+; FAST-NEXT:    stp q23, q22, [sp, #32] // 32-byte Folded Spill
+; FAST-NEXT:    stp q21, q20, [sp, #64] // 32-byte Folded Spill
+; FAST-NEXT:    stp q19, q18, [sp, #96] // 32-byte Folded Spill
+; FAST-NEXT:    stp q17, q16, [sp, #128] // 32-byte Folded Spill
+; FAST-NEXT:    stp q15, q14, [sp, #160] // 32-byte Folded Spill
+; FAST-NEXT:    stp q13, q12, [sp, #192] // 32-byte Folded Spill
+; FAST-NEXT:    stp q11, q10, [sp, #224] // 32-byte Folded Spill
+; FAST-NEXT:    stp q9, q8, [sp, #256] // 32-byte Folded Spill
+; FAST-NEXT:    stp x29, x30, [sp, #288] // 16-byte Folded Spill
+; FAST-NEXT:    .cfi_def_cfa_offset 304
+; FAST-NEXT:    .cfi_offset w30, -8
+; FAST-NEXT:    .cfi_offset w29, -16
+; FAST-NEXT:    .cfi_offset b8, -32
+; FAST-NEXT:    .cfi_offset b9, -48
+; FAST-NEXT:    .cfi_offset b10, -64
+; FAST-NEXT:    .cfi_offset b11, -80
+; FAST-NEXT:    .cfi_offset b12, -96
+; FAST-NEXT:    .cfi_offset b13, -112
+; FAST-NEXT:    .cfi_offset b14, -128
+; FAST-NEXT:    .cfi_offset b15, -144
+; FAST-NEXT:    .cfi_offset b16, -160
+; FAST-NEXT:    .cfi_offset b17, -176
+; FAST-NEXT:    .cfi_offset b18, -192
+; FAST-NEXT:    .cfi_offset b19, -208
+; FAST-NEXT:    .cfi_offset b20, -224
+; FAST-NEXT:    .cfi_offset b21, -240
+; FAST-NEXT:    .cfi_offset b22, -256
+; FAST-NEXT:    .cfi_offset b23, -272
+; FAST-NEXT:  .Ltmp3:
+; FAST-NEXT:    str q0, [sp, #16] // 16-byte Folded Spill
+; FAST-NEXT:    bl may_throw_neon
+; FAST-NEXT:  .Ltmp4:
+; FAST-NEXT:    str q0, [sp] // 16-byte Folded Spill
+; FAST-NEXT:    b .LBB1_1
+; FAST-NEXT:  .LBB1_1: // %.Lcontinue
+; FAST-NEXT:    ldr q0, [sp] // 16-byte Folded Reload
+; FAST-NEXT:    ldp x29, x30, [sp, #288] // 16-byte Folded Reload
+; FAST-NEXT:    ldp q9, q8, [sp, #256] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q11, q10, [sp, #224] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q13, q12, [sp, #192] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q15, q14, [sp, #160] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q17, q16, [sp, #128] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q19, q18, [sp, #96] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q21, q20, [sp, #64] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q23, q22, [sp, #32] // 32-byte Folded Reload
+; FAST-NEXT:    add sp, sp, #304 // =304
+; FAST-NEXT:    ret
+; FAST-NEXT:  .LBB1_2: // %.Lunwind
+; FAST-NEXT:  .Ltmp5:
+; FAST-NEXT:    ldr q0, [sp, #16] // 16-byte Folded Reload
+; FAST-NEXT:    ldp x29, x30, [sp, #288] // 16-byte Folded Reload
+; FAST-NEXT:    ldp q9, q8, [sp, #256] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q11, q10, [sp, #224] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q13, q12, [sp, #192] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q15, q14, [sp, #160] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q17, q16, [sp, #128] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q19, q18, [sp, #96] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q21, q20, [sp, #64] // 32-byte Folded Reload
+; FAST-NEXT:    ldp q23, q22, [sp, #32] // 32-byte Folded Reload
+; FAST-NEXT:    add sp, sp, #304 // =304
+; FAST-NEXT:    ret
+  %result = invoke aarch64_vector_pcs <4 x i32> @may_throw_neon(<4 x i32> %v) to label %.Lcontinue unwind label %.Lunwind
+.Lcontinue:
+  ret <4 x i32> %result
+.Lunwind:
+  %lp = landingpad { i8*, i32 } cleanup
+  ret <4 x i32> %v;
+}
+
+declare aarch64_vector_pcs <4 x i32> @may_throw_neon(<4 x i32> %v);