[AArch64] Homogeneous Prolog and Epilog for Size Optimization

[kyulee]

Summary

Prolog and epilog to handle callee-save registers tend to be irregular with different immediate offsets, which are not often being outlined (by machine outliner) when optimizing for size. From D18619, combining stack operations stretched irregularity further.
This patch tries to emit homogeneous stores and loads with the same offset for prolog and epilog respectively. We have observed that this homogeneous prolog and epilog significantly increased the chance of outlining, resulting in a code size reduction. However, it was still far from the minimum size code due to requiring the special handling of the return register, x30.
This patch custom-outlines prolog and epilog in place:

• Injects HOM_Prolog and HOM_Epilog psuedo instructions in Prolog and Epilog Injection Pass
• Lower and optimize them in AArchLowerHomogneousPrologEpilog Pass
• Outlined helpers are created on demand. Identical helpers are merged by the linker.
• An opt-in flag is introduced to enable this feature. Another threshold flag is also introduced to control the aggressiveness of outlining for application's need.

This reduced an average of 4% of code size on LLVM-TestSuite/CTMark targeting arm64/-Oz.

[dmgreen]

Hello. I like the idea. It's something we thought about internally but no-one has ever worked on enough to see how much of an improvement it gives in general.

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

230

Can you explain what CompactUnwindFrame is, and why it's needed for this to work? I'm not really an expert on Frame Lowering, but is there a way to get this to work without that restriction?

[kyulee]

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

230

This guarantees a pair register use and the ordering of CSR save locations are fixed, which simplifies the current logic for correctness. An immediate remedy to this is to support the non-pair register case, but I'm not either an expert for all other platforms and calling conventions, and I'm not sure how I test and validate such a relaxation. I think probably it's better leaving this extension to folks who know the details for their platforms.

[kyulee]

Updating for Linux support.

[plotfi]

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

1616

Is the bit that was removed a non-functional change here? If so, can this be a separate NFC commit?

[plotfi]

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

242

This is so small that I feel it would be more descriptive at the call site of SavedRegs.set/test to have:

/// true if CSRs should be paired
const bool producePairRegisters = produceCompactUnwindFrame(MF) || homogeneousPrologEpilog(MF);

With some additional comments on the register paring in the context of homogenous-prolog-epilog.

[kyulee]

Refactor getArgumentPopSize() to D83456
Add comment on producePairRegisters()

[kyulee]

Fix for merge

[kyulee]

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

242

Updated the comment.

1616

It's refactored to D83456.

[plotfi]

@kyulee Update please (since the NFC has landed), so that Harbormaster can run again without conflict.

[kyulee]

Rebase after D83456

[kyulee]

rebase

[plotfi]

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

251

I think this one could potentially also be a separate NFC as well.

[plotfi]

@t.p.northover Hi Tim, would you have some bandwidth to take a look at this patch for homogeneous prolog/epilog callee-save registers opti?

[kyulee]

Fix for Outlined Epilog for a Tail-Call

Added a case x16 is live across the epilog helper, which is bailed-out.

[plotfi]

@t.p.northover gentle ping

[plotfi]

@sdesmalen @efriedma @dmgreen Hi Sander, Eli, Dave. Would any of you have some time to help review this Prolog Epilog Size optimization patch? Much appreciated if you do. Me and @kyulee would be available to chat on IRC or discord about it for more info.

[efriedma]

Have you looked at the RISCV prologue/epilogue lowering support? How does this compare?

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

245

A long list of random conditions like this is asking for trouble: someone is inevitably going to miss some case where it needs to be updated in the future. Is this really the only way?

llvm/test/CodeGen/AArch64/arm64-homogeneous-prolog-epilog-frame-tail.ll

72

If we're going to save fp/lr in the caller, can't we just save them to the correct position, instead of copying them to a new location in the outlined function?

Also, in general, pre-increment instructions are more expensive then non-pre-increment instructions. It would be better to rearrange the operations so you can emit exactly one pre-increment instruction, instead of making every instruction pre-increment.

[kyulee]

@efriedma Unlike other work that uses a helper call to runtime library, this work synthesizes outlined functions with a custom-calling convention. This is handy and flexible to extend this logic for other patterns or purposes like instrumentation, etc inside the helper within the compiler.

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

245

This patch is meant to find a regular/homogeneous stack frame as possible that can be easily outlined for the size. In theory, all these random conditions/restrictions can be relaxed with more thorough implementation, but I think this may be good to start with. Let me know how I can improve this.

llvm/test/CodeGen/AArch64/arm64-homogeneous-prolog-epilog-frame-tail.ll

72

I agree this is not ideal for the Linux case which has the different (opposite) order of CSR than Darwin/iOS which does not require this pattern (see the above without SAVELR case). The initial version disallowed this while supporting compact unwind case (Darwin) only but from @dmgreen suggestion, I added this Linux support, but tries to keep as the same pattern as possible instead of introducing different shape at the call-site. This is an important aspect when we want to extend this homogeneous function entry for other purpose (like instrumentation, etc) which actually I'm working on in another context.

I understand pre/post-inc/dec uses are more expensive, and this code can be optimized with additional complexity.
The goal is to minimize the code by exposing the homogeneous patterns and outlining them at the cost of performance -- in fact, outlining itself may hurt perf significantly on a CPU bound case although page-fault or working set is improved for the large binary with this aggressive size optimization.
@efriedma Can I follow up the specialized/optimized outlined function after this patch? Or I can work on the improved version in place now.

[efriedma]

Should this be controlled by the clang -moutline/-mno-outline?

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

245

I'm not sure how hasVarSizedObjects() or needsStackRealignment() are relevant here.

For the other bits, I guess they're directly related to what you're doing. Maybe leave comments in the relevant parts of prologue and epilogue lowering noting that they need to stay in sync with homogeneousPrologEpilog.

llvm/test/CodeGen/AArch64/arm64-homogeneous-prolog-epilog-frame-tail.ll

72

If fixing this is going to substantially change the way the implementation looks, I'd rather not review it twice.

[kyulee]

This rewrites the helper functions:

• One SP Adjustment to reduce micro-opts for the better perf.
• Align Linux/Darwin implementation in the same way . X29/X30 is stored at the call-site and that stack space is skipped in the prolog helper.
• Add some comments and expand the enabling check.

I'm not sure how we want to enable this via the clang flag -moutline .
I think the initial approach is a bit conservative so that this change is kicked in only when -moutline AND minsize opts are applied. Let me know about this.

[kyulee]

Rebase

[kyulee]

@efriedma Could you have a chance to take a look this again?

[kyulee]

Rebase.

I would appreciate if anyone can review this.

[kyulee]

Reabse

[t.p.northover]

llvm/lib/Target/AArch64/AArch64LowerHomogeneousPrologEpilog.cpp

216–218

What does adding two identical mem-operands achieve? I assumed it would be treated identically to just one.

304

I think it would be clearer if the LRIter became LRIdx, folding the std::distance thing into the same line as the find (or by another means). We don't really use it as an iterator and spreading the calculation out confuses me.

306

This calculation is quite dense and the dynamic sequence of stores we're aiming for isn't really explicit in any one location (comment at the top of the file?). I had to copy/paste it into a main.cpp and run some test cases before I really got what it was trying to do.

Also, and I think this is part of the same problem, it's not obvious that we even save all registers we need to. The loop looks like it's going to skip the last two elements unless the if is executed, and I had to really think to convince myself that matched the case where the last two are LR/FP and so already saved.

Sitcking to the separate emitStore calls it might be better as

// If the register stored to the lowest address is not LR, we must subtract more from SP here.
if (LRIdx != Size - 2)
  emitStore(...);

though there's an argument for merging them

// If LR is not stored at the lowest address (i.e. the final pair in the list) then
// the caller will only have subtracted part of the callee save area from SP. 
// Our first store must do the rest.
int RemainingSPAdjustment = Size - 2 - LRIdx;

for (int I = Size - 2; I >= 0; I -= 2) {
  if (Regs[I] == AArch64::LR)
    continue;
  emitStore(..., Regs[I], Regs[I+1], Size - 2 - I - RemainingSPAdjustment, RemainingSPAdjustment != 0);
  RemainingSPAdjustment = 0;
}

354

I know we're not hugely concerned about performance (the indexing is minor, for example), but this seems to misalign the BL/RET chain that CPUs often hard-code into their branch predictors. I seem to remember reading that's an absolutely horrible thing to do to a CPU.

Fortunately, the native AArch64::RET instruction does actually take a register operand (it's only difference from BR is precisely that it's a hint to the CPU that you're popping from any microarchitectural return-address stack it may be keeping). So probably best to use that.

368

Is this really a dynamic property? The frame layout is pretty solidly dictated by the ABI so the offset is known from the list of registers we're saving.

Even if it is dynamic, perhaps it's best to encode into HOM_prolog as an operand rather than re-deriving it by grubbing around the basic block.

407

Let the compiler convert a divide to a shift, it's not the 80s.

[smeenai]

llvm/lib/Target/AArch64/AArch64LowerHomogeneousPrologEpilog.cpp

407

If you're compiling without assertions, the compiler wouldn't be able to just generate a shift directly (because it has to account for negative numbers), right? You'd need something like a __builtin_assume to get just the shift to be generated. (See https://godbolt.org/z/vzx9hx)

[JuunChen]

Hi, I may have found a problem.

It says in the official document ---- the setter and getter of properties will become direct if objc_direct_members is placed on the @implementation block：

When objc_direct_members is placed on an @implementation block, methods defined in the block are considered to be declared as direct unless they have been previously declared as non-direct in any interface of the class. This includes the implicit method definitions introduced by synthesized properties, including auto-synthesized properties.

But I found that this way didn't make property's setter or getter became direct method.
I wrote 3 demos, the code is as follows:

//normal_excutable
@interface ViewController ()
@property (nonatomic, copy) NSString *name;
@end

@implementation ViewController
@end

//use_objc_direct_members
@interface ViewController ()
@property (nonatomic, copy) NSString *name;
@end

__attribute__((objc_direct_members))
@implementation ViewController
@end

//use_direct_property
@interface ViewController ()
@property (nonatomic, copy, direct) NSString *name;
@end

@implementation ViewController
@end

Compile them separately and compare their Mach-O products:

From the picture:

• In normal executable, the name and setName: are in the ViewController's method list, there are not direct methods.
• When placed __attribute__((objc_direct_members)) above @implementation, nothing is happened, the name and setName: are also not direct methods. This does not seem to be consistent with what is described in the documentation.
• When used direct in property, it's works well, the name and setName: are not in the method list, they become the direct methods.

So, I don't know if I misunderstood the sentence:

"This includes the implicit method definitions introduced by synthesized properties, including auto-synthesized properties."

Or the document writes error ? Or the objc_direct_members has a bug?

I would be very, very grateful if you could answer my questions.

[kyulee]

Update for @t.p.northover feedbacks

• Pass FPOffset as an argument to expand it directly
• Use ret instead of br in the epilog return
• Clean up iterator use for register indexing
• Use / instead of >> for the readability with unsigned type.

[kyulee]

llvm/lib/Target/AArch64/AArch64LowerHomogeneousPrologEpilog.cpp

216–218

In fact, these mem-operands are not needed so I just delete them.

306

I took the former approach for the readability even though the latter merges all these code.

354

Nice to know this! I used RET instead of BR.

368

I just passed FPOffset as additional argument to HOM_Prolog. Indeed your suggestion cleans up the code quite a bit. Thanks!

407

Thanks for the point. As shown in https://godbolt.org/z/Wo5r4W, I use a division with unsigned type for readability, but it becomes a shift.

[kyulee]

• Fix clang-tidy/format issues

[kyulee]

• Fix pre-merge checks by using variable_ops in pseudo.

[kyulee]

• Rebase it to the latest
• Expand the epilog corner case where x16 lives across blocks
• Fix a comment

[kyulee]

• Fix clang-tidy from pre-merge check

[kyulee]

@ t.p.northover Could you take a look again when you're available? Thanks!

[t.p.northover]

Thanks for applying the suggested fixes. I think I'm happy with it now.

llvm/lib/Target/AArch64/AArch64.h

 FunctionPass *createAArch64ISelDag(AArch64TargetMachine &TM,
                                  CodeGenOpt::Level OptLevel);
 FunctionPass *createAArch64StorePairSuppressPass();
 FunctionPass *createAArch64ExpandPseudoPass();
 FunctionPass *createAArch64SLSHardeningPass();
 FunctionPass *createAArch64IndirectThunks();
 FunctionPass *createAArch64SpeculationHardeningPass();
 FunctionPass *createAArch64LoadStoreOptimizationPass();
+ModulePass *createAArch64LowerHomogeneousPrologEpilogPass();
 FunctionPass *createAArch64SIMDInstrOptPass();
 ModulePass *createAArch64PromoteConstantPass();
 FunctionPass *createAArch64ConditionOptimizerPass();
 FunctionPass *createAArch64A57FPLoadBalancing();
 FunctionPass *createAArch64A53Fix835769();
 FunctionPass *createFalkorHWPFFixPass();
 FunctionPass *createFalkorMarkStridedAccessesPass();
 FunctionPass *createAArch64BranchTargetsPass();

 void initializeAArch64CompressJumpTablesPass(PassRegistry&);
 void initializeAArch64ConditionalComparesPass(PassRegistry&);
 void initializeAArch64ConditionOptimizerPass(PassRegistry&);
 void initializeAArch64DeadRegisterDefinitionsPass(PassRegistry&);
 void initializeAArch64ExpandPseudoPass(PassRegistry&);
 void initializeAArch64SLSHardeningPass(PassRegistry&);
 void initializeAArch64SpeculationHardeningPass(PassRegistry&);
 void initializeAArch64LoadStoreOptPass(PassRegistry&);
+void initializeAArch64LowerHomogeneousPrologEpilogPass(PassRegistry &);
 void initializeAArch64SIMDInstrOptPass(PassRegistry&);
 void initializeAArch64PreLegalizerCombinerPass(PassRegistry&);
 void initializeAArch64PostLegalizerCombinerPass(PassRegistry &);
 void initializeAArch64PostLegalizerLoweringPass(PassRegistry &);
 void initializeAArch64PostSelectOptimizePass(PassRegistry &);
 void initializeAArch64PromoteConstantPass(PassRegistry&);
 void initializeAArch64RedundantCopyEliminationPass(PassRegistry&);
 void initializeAArch64StorePairSuppressPass(PassRegistry&);

llvm/lib/Target/AArch64/AArch64FrameLowering.h

bool isStackIdSafeForLocalArea(unsigned StackId) const override {
     return StackId != TargetStackID::SVEVector;
   }
 
   void
   orderFrameObjects(const MachineFunction &MF,
                     SmallVectorImpl<int> &ObjectsToAllocate) const override;
 
 private:
+  /// Returns true if a homogeneous prolog or epilog code can be emitted
+  /// for the size optimization. If so, HOM_Prolog/HOM_Epilog pseudo
+  /// instructions are emitted in place. When Exit block is given, this check is
+  /// for epilog.
+  bool homogeneousPrologEpilog(MachineFunction &MF,
+                               MachineBasicBlock *Exit = nullptr) const;
+
+  /// Returns true if CSRs should be paired.
+  bool producePairRegisters(MachineFunction &MF) const;
+
   bool shouldCombineCSRLocalStackBump(MachineFunction &MF,
                                       uint64_t StackBumpBytes) const;
 
   int64_t estimateSVEStackObjectOffsets(MachineFrameInfo &MF) const;
   int64_t assignSVEStackObjectOffsets(MachineFrameInfo &MF,
                                       int &MinCSFrameIndex,
                                       int &MaxCSFrameIndex) const;
   MCCFIInstruction

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

static cl::opt<bool> StackTaggingMergeSetTag(
     "stack-tagging-merge-settag",
     cl::desc("merge settag instruction in function epilog"), cl::init(true),
     cl::Hidden);
 
 static cl::opt<bool> OrderFrameObjects("aarch64-order-frame-objects",
                                        cl::desc("sort stack allocations"),
                                        cl::init(true), cl::Hidden);
 
+cl::opt<bool> EnableHomogeneousPrologEpilog(
+    "homogeneous-prolog-epilog", cl::init(false), cl::ZeroOrMore, cl::Hidden,
+    cl::desc("Emit homogeneous prologue and epilogue for the size "
+             "optimization (default = off)"));
+
 STATISTIC(NumRedZoneFunctions, "Number of functions using red zone");
 
 /// Returns the argument pop size.
 static uint64_t getArgumentPopSize(MachineFunction &MF,
                                    MachineBasicBlock &MBB) {
   MachineBasicBlock::iterator MBBI = MBB.getLastNonDebugInstr();
   bool IsTailCallReturn = false;
   if (MBB.end() != MBBI) {
if (IsTailCallReturn) {
     // LowerFormalArguments. This will, of course, be zero for the C calling
     // convention.
     ArgumentPopSize = AFI->getArgumentStackToRestore();
   }
 
   return ArgumentPopSize;
 }
 
+static bool produceCompactUnwindFrame(MachineFunction &MF);
+static bool needsWinCFI(const MachineFunction &MF);
+static StackOffset getSVEStackSize(const MachineFunction &MF);
+
+/// Returns true if a homogeneous prolog or epilog code can be emitted
+/// for the size optimization. If possible, a frame helper call is injected.
+/// When Exit block is given, this check is for epilog.
+bool AArch64FrameLowering::homogeneousPrologEpilog(
+    MachineFunction &MF, MachineBasicBlock *Exit) const {
+  if (!MF.getFunction().hasMinSize())
+    return false;
+  if (!EnableHomogeneousPrologEpilog)
+    return false;
+  if (ReverseCSRRestoreSeq)
+    return false;
+  if (EnableRedZone)
+    return false;
+
+  // TODO: Window is supported yet.
+  if (needsWinCFI(MF))
+    return false;
+  // TODO: SVE is not supported yet.
+  if (getSVEStackSize(MF))
+    return false;
+
+  // Bail on stack adjustment needed on return for simplicity.
+  const MachineFrameInfo &MFI = MF.getFrameInfo();
+  const TargetRegisterInfo *RegInfo = MF.getSubtarget().getRegisterInfo();
+  if (MFI.hasVarSizedObjects() || RegInfo->needsStackRealignment(MF))
+    return false;
+  if (Exit && getArgumentPopSize(MF, *Exit))
+    return false;
+
+  return true;
+}
+
+/// Returns true if CSRs should be paired.
+bool AArch64FrameLowering::producePairRegisters(MachineFunction &MF) const {
+  return produceCompactUnwindFrame(MF) || homogeneousPrologEpilog(MF);
+}
+
 /// This is the biggest offset to the stack pointer we can encode in aarch64
 /// instructions (without using a separate calculation and a temp register).
 /// Note that the exception here are vector stores/loads which cannot encode any
 /// displacements (see estimateRSStackSizeLimit(), isAArch64FrameOffsetLegal()).
 static const unsigned DefaultSafeSPDisplacement = 255;
 
 /// Look at each instruction that references stack frames and return the stack
 /// size limit beyond which some of these instructions will require a scratch

 }
 
 bool AArch64FrameLowering::shouldCombineCSRLocalStackBump(
     MachineFunction &MF, uint64_t StackBumpBytes) const {
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   const AArch64RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
+  if (homogeneousPrologEpilog(MF))
+    return false;
 
   if (AFI->getLocalStackSize() == 0)
     return false;
 
   // For WinCFI, if optimizing for size, prefer to not combine the stack bump
   // (to force a stp with predecrement) to match the packed unwind format,
   // provided that there actually are any callee saved registers to merge the
   // decrement with.
void AArch64FrameLowering::emitPrologue(MachineFunction &MF,
   AFI->setLocalStackSize(NumBytes - PrologueSaveSize);
   bool CombineSPBump = shouldCombineCSRLocalStackBump(MF, NumBytes);
   if (CombineSPBump) {
     assert(!SVEStackSize && "Cannot combine SP bump with SVE");
     emitFrameOffset(MBB, MBBI, DL, AArch64::SP, AArch64::SP,
                     {-NumBytes, MVT::i8}, TII, MachineInstr::FrameSetup, false,
                     NeedsWinCFI, &HasWinCFI);
     NumBytes = 0;
+  } else if (homogeneousPrologEpilog(MF)) {
+    // Stack has been already adjusted.
+    NumBytes -= PrologueSaveSize;
   } else if (PrologueSaveSize != 0) {
     MBBI = convertCalleeSaveRestoreToSPPrePostIncDec(
         MBB, MBBI, DL, TII, -PrologueSaveSize, NeedsWinCFI, &HasWinCFI);
     NumBytes -= PrologueSaveSize;
   }
   assert(NumBytes >= 0 && "Negative stack allocation size!?");
 
   // Move past the saves of the callee-saved registers, fixing up the offsets
void AArch64FrameLowering::emitEpilogue(MachineFunction &MF,
 
   // All calls are tail calls in GHC calling conv, and functions have no
   // prologue/epilogue.
   if (MF.getFunction().getCallingConv() == CallingConv::GHC)
     return;
 
   // Initial and residual are named for consistency with the prologue. Note that
   // in the epilogue, the residual adjustment is executed first.
   uint64_t ArgumentPopSize = getArgumentPopSize(MF, MBB);
 
   // The stack frame should be like below,
   //
   //      ----------------------                     ---
   //      |                    |                      |
   //      | BytesInStackArgArea|              CalleeArgStackSize
   //      | (NumReusableBytes) |                (of tail call)
   //      |                    |                     ---
void AArch64FrameLowering::emitEpilogue(MachineFunction &MF,
   uint64_t AfterCSRPopSize = ArgumentPopSize;
   auto PrologueSaveSize = AFI->getCalleeSavedStackSize() + FixedObject;
   // We cannot rely on the local stack size set in emitPrologue if the function
   // has funclets, as funclets have different local stack size requirements, and
   // the current value set in emitPrologue may be that of the containing
   // function.
   if (MF.hasEHFunclets())
     AFI->setLocalStackSize(NumBytes - PrologueSaveSize);
+  if (homogeneousPrologEpilog(MF, &MBB)) {
+    assert(!NeedsWinCFI);
+    auto LastPopI = MBB.getFirstTerminator();
+    if (LastPopI != MBB.begin()) {
+      auto HomogeneousEpilog = std::prev(LastPopI);
+      if (HomogeneousEpilog->getOpcode() == AArch64::HOM_Epilog)
+        LastPopI = HomogeneousEpilog;
+    }
+
+    // Adjust local stack
+    uint64_t LocalStackSize = AFI->getLocalStackSize();
+    emitFrameOffset(MBB, LastPopI, DL, AArch64::SP, AArch64::SP,
+                    {(int64_t)LocalStackSize, MVT::i8}, TII,
+                    MachineInstr::FrameDestroy, false, NeedsWinCFI);
+
+    // SP has been already adjusted while restoring callee save regs.
+    // We've bailed-out the case with adjusting SP for arguments.
+    assert(AfterCSRPopSize == 0);
+    return;
+  }
   bool CombineSPBump = shouldCombineCSRLocalStackBumpInEpilogue(MBB, NumBytes);
   // Assume we can't combine the last pop with the sp restore.
 
   if (!CombineSPBump && PrologueSaveSize != 0) {
     MachineBasicBlock::iterator Pop = std::prev(MBB.getFirstTerminator());
     while (AArch64InstrInfo::isSEHInstruction(*Pop))
       Pop = std::prev(Pop);
     // Converting the last ldp to a post-index ldp is valid only if the last
if (!MF.getFunction().hasFnAttribute(Attribute::NoUnwind)) {
           .addCFIIndex(CFIIndex)
           .setMIFlag(MachineInstr::FrameSetup);
     }
 
     // This instruction also makes x18 live-in to the entry block.
     MBB.addLiveIn(AArch64::X18);
   }
 
+  if (homogeneousPrologEpilog(MF)) {
+    auto MIB = BuildMI(MBB, MI, DL, TII.get(AArch64::HOM_Prolog))
+                   .setMIFlag(MachineInstr::FrameSetup);
+
+    for (auto &RPI : RegPairs) {
+      MIB.addReg(RPI.Reg1, RegState::Implicit);
+      MIB.addReg(RPI.Reg2, RegState::Implicit);
+
+      // Update register live in.
+      if (!MRI.isReserved(RPI.Reg1))
+        MBB.addLiveIn(RPI.Reg1);
+      if (!MRI.isReserved(RPI.Reg2))
+        MBB.addLiveIn(RPI.Reg2);
+    }
+    return true;
+  }
   for (auto RPII = RegPairs.rbegin(), RPIE = RegPairs.rend(); RPII != RPIE;
        ++RPII) {
     RegPairInfo RPI = *RPII;
     unsigned Reg1 = RPI.Reg1;
     unsigned Reg2 = RPI.Reg2;
     unsigned StrOpc;
 
     // Issue sequence of spills for cs regs.  The first spill may be converted
bool AArch64FrameLowering::restoreCalleeSavedRegisters(
   for (const RegPairInfo &RPI : reverse(RegPairs))
     if (RPI.isScalable())
       EmitMI(RPI);
 
   if (ReverseCSRRestoreSeq) {
     for (const RegPairInfo &RPI : reverse(RegPairs))
       if (!RPI.isScalable())
         EmitMI(RPI);
+  } else if (homogeneousPrologEpilog(MF, &MBB)) {
+    auto MIB = BuildMI(MBB, MI, DL, TII.get(AArch64::HOM_Epilog))
+                   .setMIFlag(MachineInstr::FrameDestroy);
+    for (auto &RPI : RegPairs) {
+      MIB.addReg(RPI.Reg1, RegState::Implicit | RegState::Define);
+      MIB.addReg(RPI.Reg2, RegState::Implicit | RegState::Define);
+    }
+    return true;
   } else
     for (const RegPairInfo &RPI : RegPairs)
       if (!RPI.isScalable())
         EmitMI(RPI);
 
   if (NeedShadowCallStackProlog) {
     // Shadow call stack epilog: ldr x30, [x18, #-8]!
     BuildMI(MBB, MI, DL, TII.get(AArch64::LDRXpre))
if (!RegUsed) {
         UnspilledCSGPRPaired = PairedReg;
       }
       continue;
     }
 
     // MachO's compact unwind format relies on all registers being stored in
     // pairs.
     // FIXME: the usual format is actually better if unwinding isn't needed.
-    if (produceCompactUnwindFrame(MF) && PairedReg != AArch64::NoRegister &&
+    if (producePairRegisters(MF) && PairedReg != AArch64::NoRegister &&
         !SavedRegs.test(PairedReg)) {
       SavedRegs.set(PairedReg);
       if (AArch64::GPR64RegClass.contains(PairedReg) &&
           !RegInfo->isReservedReg(MF, PairedReg))
         ExtraCSSpill = PairedReg;
     }
   }
 
void AArch64FrameLowering::determineCalleeSaves(MachineFunction &MF,
   if (BigStack) {
     if (!ExtraCSSpill && UnspilledCSGPR != AArch64::NoRegister) {
       LLVM_DEBUG(dbgs() << "Spilling " << printReg(UnspilledCSGPR, RegInfo)
                         << " to get a scratch register.\n");
       SavedRegs.set(UnspilledCSGPR);
       // MachO's compact unwind format relies on all registers being stored in
       // pairs, so if we need to spill one extra for BigStack, then we need to
       // store the pair.
-      if (produceCompactUnwindFrame(MF))
+      if (producePairRegisters(MF))
         SavedRegs.set(UnspilledCSGPRPaired);
       ExtraCSSpill = UnspilledCSGPR;
     }
 
     // If we didn't find an extra callee-saved register to spill, create
     // an emergency spill slot.
     if (!ExtraCSSpill || MF.getRegInfo().isPhysRegUsed(ExtraCSSpill)) {
       const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();

llvm/lib/Target/AArch64/AArch64InstrInfo.td

 let isTerminator = 1, hasSideEffects = 1, isBarrier = 1, hasCtrlDep = 1,
     isCodeGenOnly = 1, isReturn = 1, isEHScopeReturn = 1, isPseudo = 1 in {
    def CLEANUPRET : Pseudo<(outs), (ins), [(cleanupret)]>, Sched<[]>;
    let usesCustomInserter = 1 in
      def CATCHRET : Pseudo<(outs), (ins am_brcond:$dst, am_brcond:$src), [(catchret bb:$dst, bb:$src)]>,
                     Sched<[]>;
 }
 
+// Pseudo instructions for homogeneous prolog/epilog
+let isPseudo = 1 in {
+  def HOM_Prolog : Pseudo<(outs), (ins), []>, Sched<[]>;
+  def HOM_Epilog : Pseudo<(outs), (ins), []>, Sched<[]>;
+}
 //===----------------------------------------------------------------------===//
 // Floating point immediate move.
 //===----------------------------------------------------------------------===//
 
 let isReMaterializable = 1, isAsCheapAsAMove = 1 in {
 defm FMOV : FPMoveImmediate<"fmov">;
 }
 

llvm/lib/Target/AArch64/AArch64LowerHomogeneousPrologEpilog.cpp

+//===- AArch64LowerHomogeneousPrologEpilog.cpp ----------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains a pass that lowers homogeneous prolog/epilog instructions.
+//
+//===----------------------------------------------------------------------===//
+
+#include "AArch64InstrInfo.h"
+#include "AArch64Subtarget.h"
+#include "MCTargetDesc/AArch64InstPrinter.h"
+#include "Utils/AArch64BaseInfo.h"
+#include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/CodeGen/MachineFunction.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineInstr.h"
+#include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/CodeGen/MachineModuleInfo.h"
+#include "llvm/CodeGen/MachineOperand.h"
+#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/IR/DebugLoc.h"
+#include "llvm/Pass.h"
+#include "llvm/Support/raw_ostream.h"
+#include <sstream>
+
+using namespace llvm;
+
+#define AARCH64_LOWER_HOMOGENEOUS_PROLOG_EPILOG_NAME                           \
+  "AArch64 homogeneous prolog/epilog lowering pass"
+
+cl::opt<int> FrameHelperSizeThreshold(
+    "frame-helper-size-threshold", cl::init(2), cl::Hidden,
+    cl::desc("The minimum number of instructions that are outlined in a frame "
+             "helper (default = 2)"));
+
+namespace {
+
+class AArch64LowerHomogeneousPE {
+public:
+  const AArch64InstrInfo *TII;
+
+  AArch64LowerHomogeneousPE(Module *M, MachineModuleInfo *MMI)
+      : M(M), MMI(MMI) {}
+
+  bool run();
+  bool runOnMachineFunction(MachineFunction &Fn);
+
+private:
+  Module *M;
+  MachineModuleInfo *MMI;
+
+  bool runOnMBB(MachineBasicBlock &MBB);
+  bool runOnMI(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
+               MachineBasicBlock::iterator &NextMBBI);
+
+  /// Lower a HOM_Prolog pseudo instruction into a helper call
+  /// or a sequence of homogeneous stores.
+  /// When a a fp setup follows, it can be optimized.
+  bool lowerHOM_Prolog(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
+                       MachineBasicBlock::iterator &NextMBBI);
+  /// Lower a HOM_Epilog pseudo instruction into a helper call
+  /// or a sequence of homogeneous loads.
+  /// When a return follow, it can be optimized.
+  bool lowerHOM_Epilog(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
+                       MachineBasicBlock::iterator &NextMBBI);
+};
+
+class AArch64LowerHomogeneousPrologEpilog : public ModulePass {
+public:
+  static char ID;
+
+  AArch64LowerHomogeneousPrologEpilog() : ModulePass(ID) {
+    initializeAArch64LowerHomogeneousPrologEpilogPass(
+        *PassRegistry::getPassRegistry());
+  }
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.addRequired<MachineModuleInfoWrapperPass>();
+    AU.addPreserved<MachineModuleInfoWrapperPass>();
+    AU.setPreservesAll();
+    ModulePass::getAnalysisUsage(AU);
+  }
+  bool runOnModule(Module &M) override;
+
+  StringRef getPassName() const override {
+    return AARCH64_LOWER_HOMOGENEOUS_PROLOG_EPILOG_NAME;
+  }
+};
+
+} // end anonymous namespace
+
+char AArch64LowerHomogeneousPrologEpilog::ID = 0;
+
+INITIALIZE_PASS(AArch64LowerHomogeneousPrologEpilog,
+                "aarch64-lower-homogeneous-prolog-epilog",
+                AARCH64_LOWER_HOMOGENEOUS_PROLOG_EPILOG_NAME, false, false)
+
+bool AArch64LowerHomogeneousPrologEpilog::runOnModule(Module &M) {
+  if (skipModule(M))
+    return false;
+
+  MachineModuleInfo *MMI =
+      &getAnalysis<MachineModuleInfoWrapperPass>().getMMI();
+  return AArch64LowerHomogeneousPE(&M, MMI).run();
+}
+
+bool AArch64LowerHomogeneousPE::run() {
+  bool Changed = false;
+  for (auto &F : *M) {
+    if (F.empty())
+      continue;
+
+    MachineFunction *MF = MMI->getMachineFunction(F);
+    if (!MF)
+      continue;
+    Changed |= runOnMachineFunction(*MF);
+  }
+
+  return Changed;
+}
+enum FrameHelperType { Prolog, PrologFrame, Epilog, EpilogTail };
+
+/// Return a frame helper name with the given CSRs and the helper type.
+/// For instance, a prolog helper that saves x19 and x20 is named as
+/// OUTLINED_FUNCTION_PROLOG_x19x20.
+static std::string getFrameHelperName(SmallVectorImpl<unsigned> &Regs,
+                                      FrameHelperType Type, unsigned FpOffset) {
+  std::ostringstream RegStream;
+  switch (Type) {
+  case FrameHelperType::Prolog:
+    RegStream << "OUTLINED_FUNCTION_PROLOG_";
+    break;
+  case FrameHelperType::PrologFrame:
+    RegStream << "OUTLINED_FUNCTION_PROLOG_FRAME" << FpOffset << "_";
+    break;
+  case FrameHelperType::Epilog:
+    RegStream << "OUTLINED_FUNCTION_EPILOG_";
+    break;
+  case FrameHelperType::EpilogTail:
+    RegStream << "OUTLINED_FUNCTION_EPILOG_TAIL_";
+    break;
+  }
+
+  for (auto Reg : Regs)
+    RegStream << AArch64InstPrinter::getRegisterName(Reg);
+
+  return RegStream.str();
+}
+
+/// Create a Function for the unique frame helper with the given name.
+/// Return a newly created MachineFunction with an empty MachineBasicBlock.
+static MachineFunction &createFrameHelperMachineFunction(Module *M,
+                                                         MachineModuleInfo *MMI,
+                                                         StringRef Name) {
+  LLVMContext &C = M->getContext();
+  Function *F = M->getFunction(Name);
+  assert(F == nullptr && "Function has been created before");
+  F = Function::Create(FunctionType::get(Type::getVoidTy(C), false),
+                       Function::ExternalLinkage, Name, M);
+  assert(F && "Function was null!");
+
+  // Use ODR linkage to avoid duplication.
+  F->setLinkage(GlobalValue::LinkOnceODRLinkage);
+  F->setUnnamedAddr(GlobalValue::UnnamedAddr::Global);
+
+  // Set no-opt/minsize, so we don't insert padding between outlined
+  // functions.
+  F->addFnAttr(Attribute::OptimizeNone);
+  F->addFnAttr(Attribute::NoInline);
+  F->addFnAttr(Attribute::MinSize);
+  F->addFnAttr(Attribute::Naked);
+
+  MachineFunction &MF = MMI->getOrCreateMachineFunction(*F);
+  // Remove unnecessary register liveness and set NoVRegs.
+  MF.getProperties().reset(MachineFunctionProperties::Property::TracksLiveness);
+  MF.getProperties().reset(MachineFunctionProperties::Property::IsSSA);
+  MF.getProperties().set(MachineFunctionProperties::Property::NoVRegs);
+  MF.getRegInfo().freezeReservedRegs(MF);
+
+  // Create entry block.
+  BasicBlock *EntryBB = BasicBlock::Create(C, "entry", F);
+  IRBuilder<> Builder(EntryBB);
+  Builder.CreateRetVoid();
+
+  // Insert the new block into the function.
+  MachineBasicBlock *MBB = MF.CreateMachineBasicBlock();
+  MF.insert(MF.begin(), MBB);
+
+  return MF;
+}
+
+/// Emit a store-pair instruction for frame-setup.
+static void emitStore(MachineFunction &MF, MachineBasicBlock &MBB,
+                      MachineBasicBlock::iterator Pos,
+                      const TargetInstrInfo &TII, unsigned Reg1, unsigned Reg2,
+                      int Offset, bool IsPreDec) {
+  bool IsFloat = AArch64::FPR64RegClass.contains(Reg1);
+  assert(!(IsFloat ^ AArch64::FPR64RegClass.contains(Reg2)));
+  unsigned Opc;
+  if (IsPreDec)
+    Opc = IsFloat ? AArch64::STPDpre : AArch64::STPXpre;
+  else
+    Opc = IsFloat ? AArch64::STPDi : AArch64::STPXi;
+
+  MachineInstrBuilder MIB = BuildMI(MBB, Pos, DebugLoc(), TII.get(Opc));
+  if (IsPreDec)
+    MIB.addDef(AArch64::SP);
+  MIB.addReg(Reg2)
+      .addReg(Reg1)
+      .addReg(AArch64::SP)
+      .addImm(Offset)
+      .addMemOperand(
+          MF.getMachineMemOperand(MachinePointerInfo::getUnknownStack(MF),
+                                  MachineMemOperand::MOStore, 8, Align(8)))
+      .addMemOperand(
+          MF.getMachineMemOperand(MachinePointerInfo::getUnknownStack(MF),
+                                  MachineMemOperand::MOStore, 8, Align(8)))
+      .setMIFlag(MachineInstr::FrameSetup);
+}
+
+/// Emit a load-pair instruction for frame-destroy.
+static void emitLoad(MachineFunction &MF, MachineBasicBlock &MBB,
+                     MachineBasicBlock::iterator Pos,
+                     const TargetInstrInfo &TII, unsigned Reg1, unsigned Reg2,
+                     int Offset, bool IsPostDec) {
+  bool IsFloat = AArch64::FPR64RegClass.contains(Reg1);
+  assert(!(IsFloat ^ AArch64::FPR64RegClass.contains(Reg2)));
+  unsigned Opc;
+  if (IsPostDec)
+    Opc = IsFloat ? AArch64::LDPDpost : AArch64::LDPXpost;
+  else
+    Opc = IsFloat ? AArch64::LDPDi : AArch64::LDPXi;
+
+  MachineInstrBuilder MIB = BuildMI(MBB, Pos, DebugLoc(), TII.get(Opc));
+  if (IsPostDec)
+    MIB.addDef(AArch64::SP);
+  MIB.addReg(Reg2)
+      .addReg(Reg1)
+      .addReg(AArch64::SP)
+      .addImm(Offset)
+      .addMemOperand(
+          MF.getMachineMemOperand(MachinePointerInfo::getUnknownStack(MF),
+                                  MachineMemOperand::MOLoad, 8, Align(8)))
+      .addMemOperand(
+          MF.getMachineMemOperand(MachinePointerInfo::getUnknownStack(MF),
+                                  MachineMemOperand::MOLoad, 8, Align(8)))
+      .setMIFlag(MachineInstr::FrameDestroy);
+}
+
+/// Return a unique function if a helper can be formed with the given Regs
+/// and frame type.
+/// 1) _OUTLINED_FUNCTION_PROLOG_x30x29x19x20x21x22:
+///    stp x22, x21, [sp, #-32]!    ; x29/x30 has been stored at the caller
+///    stp x20, x19, [sp, #16]
+///    ret
+///
+/// 2) _OUTLINED_FUNCTION_PROLOG_FRAME32_x30x29x19x20x21x22:
+///    stp x22, x21, [sp, #-32]!    ; x29/x30 has been stored at the caller
+///    stp x20, x19, [sp, #16]
+///    add fp, sp, #32
+///    ret
+///
+/// 3) _OUTLINED_FUNCTION_EPILOG_x30x29x19x20x21x22:
+///    mov x16, x30
+///    ldp x29, x30, [sp, #32]
+///    ldp x20, x19, [sp, #16]
+///    ldp x22, x21, [sp], #48
+///    br x16
+///
+/// 4) _OUTLINED_FUNCTION_EPILOG_TAIL_x30x29x19x20x21x22:
+///    ldp x29, x30, [sp, #32]
+///    ldp x20, x19, [sp, #-16]
+///    ldp x22, x21, [sp], #48
+///    ret
+/// @param M module
+/// @param MMI machine module info
+/// @param Regs callee save regs that the helper will handle
+/// @param Type frame helper type
+/// @return a helper function
+static Function *getOrCreateFrameHelper(Module *M, MachineModuleInfo *MMI,
+                                        SmallVectorImpl<unsigned> &Regs,
+                                        FrameHelperType Type,
+                                        unsigned FpOffset = 0) {
+  assert(Regs.size() >= 2);
+  auto Name = getFrameHelperName(Regs, Type, FpOffset);
+  auto F = M->getFunction(Name);
+  if (F)
+    return F;
+
+  auto &MF = createFrameHelperMachineFunction(M, MMI, Name);
+  MachineBasicBlock &MBB = *MF.begin();
+  const TargetSubtargetInfo &STI = MF.getSubtarget();
+  const TargetInstrInfo &TII = *STI.getInstrInfo();
+
+  int Size = (int)Regs.size();
+  switch (Type) {
+  case FrameHelperType::Prolog:
+  case FrameHelperType::PrologFrame: {
+    // Compute the remaining SP adjust beyond FP/LR.
+    auto LRIter = std::find(Regs.begin(), Regs.end(), AArch64::LR);
+    assert(LRIter != Regs.end());
+    int SPAdjust = Size - std::distance(Regs.begin(), LRIter) - 2;
+    // Save the the first CSR with pre-increment of SP.
+    if (SPAdjust) {
+      // FP/LR is not the bottom-most save.
+      assert(Regs[Size - 2] != AArch64::LR);
+      emitStore(MF, MBB, MBB.end(), TII, Regs[Size - 2], Regs[Size - 1],
+                -SPAdjust, true);
+    }
+
+    // Store CSRs in the reverse order.
+    for (int I = Size - 3; I >= 0; I -= 2) {
+      // FP/LR has been stored at call-site.
+      if (Regs[I - 1] == AArch64::LR)
+        continue;
+      emitStore(MF, MBB, MBB.end(), TII, Regs[I - 1], Regs[I], Size - I - 1,
+                false);
+    }
+    if (Type == FrameHelperType::PrologFrame)
+      BuildMI(MBB, MBB.end(), DebugLoc(), TII.get(AArch64::ADDXri))
+          .addDef(AArch64::FP)
+          .addUse(AArch64::SP)
+          .addImm(FpOffset)
+          .addImm(0)
+          .setMIFlag(MachineInstr::FrameSetup);
+
+    BuildMI(MBB, MBB.end(), DebugLoc(), TII.get(AArch64::RET))
+        .addReg(AArch64::LR, RegState::Undef);
+    break;
+  }
+  case FrameHelperType::Epilog:
+  case FrameHelperType::EpilogTail:
+    if (Type == FrameHelperType::Epilog)
+      // Stash LR to X16
+      BuildMI(MBB, MBB.end(), DebugLoc(), TII.get(AArch64::ORRXrs))
+          .addDef(AArch64::X16)
+          .addReg(AArch64::XZR)
+          .addUse(AArch64::LR)
+          .addImm(0);
+
+    for (int I = 0; I < Size - 2; I += 2)
+      emitLoad(MF, MBB, MBB.end(), TII, Regs[I], Regs[I + 1], Size - I - 2,
+               false);
+    // Restore the last CSR with post-increment of SP.
+    emitLoad(MF, MBB, MBB.end(), TII, Regs[Size - 2], Regs[Size - 1], Size,
+             true);
+
+    if (Type == FrameHelperType::Epilog)
+      // Branch on X16 not to trash LR.
+      BuildMI(MBB, MBB.end(), DebugLoc(), TII.get(AArch64::BR))
+          .addUse(AArch64::X16);
+    else
+      BuildMI(MBB, MBB.end(), DebugLoc(), TII.get(AArch64::RET))
+          .addReg(AArch64::LR, RegState::Undef);
+    break;
+  }
+
+  return M->getFunction(Name);
+}
+
+/// Get a valid non-negative adjustment to set fp from sp.
+/// @param MBBI instruciton setting fp from sp.
+/// @return a valid non-negative adjustment. Or -1 for any other case.
+int getFpAdjustmentFromSp(MachineBasicBlock::iterator &MBBI) {
+  MachineInstr &MI = *MBBI;
+  if (!MI.getFlag(MachineInstr::FrameSetup))
+    return -1;
+  unsigned Opcode = MI.getOpcode();
+  if (Opcode != AArch64::ADDXri && Opcode != AArch64::SUBXri)
+    return -1;
+  if (!MI.getOperand(0).isReg())
+    return -1;
+  if (MI.getOperand(0).getReg() != AArch64::FP)
+    return -1;
+  if (!MI.getOperand(1).isReg())
+    return -1;
+  if (MI.getOperand(1).getReg() != AArch64::SP)
+    return -1;
+
+  int Imm = MI.getOperand(2).getImm();
+  if (Opcode == AArch64::ADDXri && Imm >= 0)
+    return Imm;
+  else if (Opcode == AArch64::SUBXri && Imm <= 0)
+    return -Imm;
+
+  return -1;
+}
+
+/// This function checks if a frame helper should be used for
+/// HOM_Prolog/HOM_Epilog pseudo instruction expansion.
+/// @param MBB machine basic block
+/// @param NextMBBI  next instruction following HOM_Prolog/HOM_Epilog
+/// @param Regs callee save registers that are saved or restored.
+/// @param Type frame helper type
+/// @return True if a use of helper is qualified.
+static bool shouldUseFrameHelper(MachineBasicBlock &MBB,
+                                 MachineBasicBlock::iterator &NextMBBI,
+                                 SmallVectorImpl<unsigned> &Regs,
+                                 FrameHelperType Type) {
+  int RegCount = (int)Regs.size();
+  assert(RegCount > 0 && (RegCount % 2 == 0));
+  // # of instructions that will be outlined.
+  int InstCount = RegCount >> 1;
+
+  // Do not use a helper call when not saving LR.
+  if (std::find(Regs.begin(), Regs.end(), AArch64::LR) == Regs.end())
+    return false;
+
+  switch (Type) {
+  case FrameHelperType::Prolog:
+    // Prolog helper cannot save FP/LR.
+    InstCount--;
+    break;
+  case FrameHelperType::PrologFrame: {
+    // Prolog helper cannot save FP/LR.
+    // Check if the following instruction is beneficial to be included.
+    if (NextMBBI == MBB.end())
+      return false;
+    int FpAdjustment = getFpAdjustmentFromSp(NextMBBI);
+    if (FpAdjustment == -1)
+      return false;
+    // Effecitvely no change in InstCount since FpAdjusment is included.
+    break;
+  }
+  case FrameHelperType::Epilog:
+    // Bail-out if X16 is live across the epilog helper because it is used in
+    // the helper to handle X30.
+    if (NextMBBI != MBB.end() && NextMBBI->readsRegister(AArch64::X16))
+      return false;
+    // No change in InstCount for the regular epilog case.
+    break;
+  case FrameHelperType::EpilogTail: {
+    // EpilogTail helper includes the caller's return.
+    if (NextMBBI == MBB.end())
+      return false;
+    if (NextMBBI->getOpcode() != AArch64::RET_ReallyLR)
+      return false;
+    InstCount++;
+    break;
+  }
+  }
+
+  return InstCount >= FrameHelperSizeThreshold;
+}
+
+/// Lower a HOM_Epilog pseudo instruction into a helper call while
+/// creating the helper on demand. Or emit a sequence of loads in place when not
+/// using a helper call.
+///
+/// 1. With a helper including ret
+///    HOM_Epilog x30, x29, x19, x20, x21, x22              ; MBBI
+///    ret                                                  ; NextMBBI
+///    =>
+///    b _OUTLINED_FUNCTION_EPILOG_TAIL_x30x29x19x20x21x22
+///    ...                                                  ; NextMBBI
+///
+/// 2. With a helper
+///    HOM_Epilog x30, x29, x19, x20, x21, x22
+///    =>
+///    bl _OUTLINED_FUNCTION_EPILOG_x30x29x19x20x21x22
+///
+/// 3. Without a helper
+///    HOM_Epilog x30, x29, x19, x20, x21, x22
+///    =>
+///    ldp x29, x30, [sp, #32]
+///    ldp x20, x19, [sp, #16]
+///    ldp x22, x21, [sp], #48
+bool AArch64LowerHomogeneousPE::lowerHOM_Epilog(
+    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
+    MachineBasicBlock::iterator &NextMBBI) {
+  auto &MF = *MBB.getParent();
+  MachineInstr &MI = *MBBI;
+
+  DebugLoc DL = MI.getDebugLoc();
+  SmallVector<unsigned, 8> Regs;
+  for (auto &MO : MI.implicit_operands())
+    if (MO.isReg())
+      Regs.push_back(MO.getReg());
+  int Size = (int)Regs.size();
+  if (Size == 0)
+    return false;
+  // Registers are in pair.
+  assert(Size % 2 == 0);
+  assert(MI.getOpcode() == AArch64::HOM_Epilog);
+
+  auto Return = NextMBBI;
+  if (shouldUseFrameHelper(MBB, NextMBBI, Regs, FrameHelperType::EpilogTail)) {
+    // When MBB ends with a return, emit a tail-call to the epilog helper
+    auto EpilogTailHelper =
+        getOrCreateFrameHelper(M, MMI, Regs, FrameHelperType::EpilogTail);
+    BuildMI(MBB, MBBI, DL, TII->get(AArch64::TCRETURNdi))
+        .addGlobalAddress(EpilogTailHelper)
+        .addImm(0)
+        .setMIFlag(MachineInstr::FrameDestroy)
+        .copyImplicitOps(MI)
+        .copyImplicitOps(*Return);
+    NextMBBI = std::next(Return);
+    Return->removeFromParent();
+  } else if (shouldUseFrameHelper(MBB, NextMBBI, Regs,
+                                  FrameHelperType::Epilog)) {
+    // The default epilog helper case.
+    auto EpilogHelper =
+        getOrCreateFrameHelper(M, MMI, Regs, FrameHelperType::Epilog);
+    BuildMI(MBB, MBBI, DL, TII->get(AArch64::BL))
+        .addGlobalAddress(EpilogHelper)
+        .setMIFlag(MachineInstr::FrameDestroy)
+        .copyImplicitOps(MI);
+  } else {
+    // Fall back to no-helper.
+    for (int I = 0; I < Size - 2; I += 2)
+      emitLoad(MF, MBB, MBBI, *TII, Regs[I], Regs[I + 1], Size - I - 2, false);
+    // Restore the last CSR with post-increment of SP.
+    emitLoad(MF, MBB, MBBI, *TII, Regs[Size - 2], Regs[Size - 1], Size, true);
+  }
+
+  MBBI->removeFromParent();
+  return true;
+}
+
+/// Lower a HOM_Prolog pseudo instruction into a helper call while
+/// creating the helper on demand. Or emit a sequence of stores in place when
+/// not using a helper call.
+///
+/// 1. With a helper including frame-setup
+///    HOM_Prolog x30, x29, x19, x20, x21, x22      ; MBBI
+///    add x29, x30, #32                            ; NextMBBI
+///    =>
+///    stp x29, x30, [sp, #-16]!
+///    bl _OUTLINED_FUNCTION_PROLOG_FRAME32_x30x29x19x20x21x22
+///    ...                                          ; NextMBBI
+///
+/// 2. With a helper
+///    HOM_Prolog x30, x29, x19, x20, x21, x22
+///    =>
+///    stp x29, x30, [sp, #-16]!
+///    bl _OUTLINED_FUNCTION_PROLOG_x30x29x19x20x21x22
+///
+/// 3. Without a helper
+///    HOM_Prolog x30, x29, x19, x20, x21, x22
+///    =>
+///    stp  x22, x21, [sp, #-48]!
+///    stp  x20, x19, [sp, #16]
+///    stp  x29, x30, [sp, #32]
+bool AArch64LowerHomogeneousPE::lowerHOM_Prolog(
+    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
+    MachineBasicBlock::iterator &NextMBBI) {
+  auto &MF = *MBB.getParent();
+  MachineInstr &MI = *MBBI;
+
+  DebugLoc DL = MI.getDebugLoc();
+  SmallVector<unsigned, 8> Regs;
+  int LRIdx = 0;
+  for (auto &MO : MI.implicit_operands()) {
+    if (MO.isReg()) {
+      if (MO.getReg() == AArch64::LR)
+        LRIdx = Regs.size();
+      Regs.push_back(MO.getReg());
+    }
+  }
+  int Size = (int)Regs.size();
+  if (Size == 0)
+    return false;
+  // Allow compact unwind case only for oww.
+  assert(Size % 2 == 0);
+  assert(MI.getOpcode() == AArch64::HOM_Prolog);
+
+  auto FpAdjustment = NextMBBI;
+  if (shouldUseFrameHelper(MBB, NextMBBI, Regs, FrameHelperType::PrologFrame)) {
+    // FP/LR is stored at the top of stack before the prolog helper call.
+    emitStore(MF, MBB, MBBI, *TII, AArch64::LR, AArch64::FP, -LRIdx - 2, true);
+    auto FpOffset = getFpAdjustmentFromSp(NextMBBI);
+    auto PrologFrameHelper = getOrCreateFrameHelper(
+        M, MMI, Regs, FrameHelperType::PrologFrame, FpOffset);
+    BuildMI(MBB, MBBI, DL, TII->get(AArch64::BL))
+        .addGlobalAddress(PrologFrameHelper)
+        .setMIFlag(MachineInstr::FrameSetup)
+        .copyImplicitOps(MI)
+        .copyImplicitOps(*FpAdjustment)
+        .addReg(AArch64::FP, RegState::Implicit | RegState::Define)
+        .addReg(AArch64::SP, RegState::Implicit);
+    NextMBBI = std::next(FpAdjustment);
+    FpAdjustment->removeFromParent();
+  } else if (shouldUseFrameHelper(MBB, NextMBBI, Regs,
+                                  FrameHelperType::Prolog)) {
+    // FP/LR is stored at the top of stack before the prolog helper call.
+    emitStore(MF, MBB, MBBI, *TII, AArch64::LR, AArch64::FP, -LRIdx - 2, true);
+    auto PrologHelper =
+        getOrCreateFrameHelper(M, MMI, Regs, FrameHelperType::Prolog);
+    BuildMI(MBB, MBBI, DL, TII->get(AArch64::BL))
+        .addGlobalAddress(PrologHelper)
+        .setMIFlag(MachineInstr::FrameSetup)
+        .copyImplicitOps(MI);
+  } else {
+    // Fall back to no-helper.
+    emitStore(MF, MBB, MBBI, *TII, Regs[Size - 2], Regs[Size - 1], -Size, true);
+    for (int I = Size - 3; I >= 0; I -= 2)
+      emitStore(MF, MBB, MBBI, *TII, Regs[I - 1], Regs[I], Size - I - 1, false);
+  }
+
+  MBBI->removeFromParent();
+  return true;
+}
+
+/// Process each machine instruction
+/// @param MBB machine basic block
+/// @param MBBI current instruction iterator
+/// @param NextMBBIT next instruction iterator which can be updated
+/// @return True when IR is changed.
+bool AArch64LowerHomogeneousPE::runOnMI(MachineBasicBlock &MBB,
+                                        MachineBasicBlock::iterator MBBI,
+                                        MachineBasicBlock::iterator &NextMBBI) {
+  MachineInstr &MI = *MBBI;
+  unsigned Opcode = MI.getOpcode();
+  switch (Opcode) {
+  default:
+    break;
+  case AArch64::HOM_Prolog:
+    return lowerHOM_Prolog(MBB, MBBI, NextMBBI);
+  case AArch64::HOM_Epilog:
+    return lowerHOM_Epilog(MBB, MBBI, NextMBBI);
+  }
+  return false;
+}
+
+bool AArch64LowerHomogeneousPE::runOnMBB(MachineBasicBlock &MBB) {
+  bool Modified = false;
+
+  MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
+  while (MBBI != E) {
+    MachineBasicBlock::iterator NMBBI = std::next(MBBI);
+    Modified |= runOnMI(MBB, MBBI, NMBBI);
+    MBBI = NMBBI;
+  }
+
+  return Modified;
+}
+
+bool AArch64LowerHomogeneousPE::runOnMachineFunction(MachineFunction &MF) {
+  TII = static_cast<const AArch64InstrInfo *>(MF.getSubtarget().getInstrInfo());
+
+  bool Modified = false;
+  for (auto &MBB : MF)
+    Modified |= runOnMBB(MBB);
+  return Modified;
+}
+
+ModulePass *llvm::createAArch64LowerHomogeneousPrologEpilogPass() {
+  return new AArch64LowerHomogeneousPrologEpilog();
+}

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

 static cl::opt<bool> EnableFalkorHWPFFix("aarch64-enable-falkor-hwpf-fix",
                                          cl::init(true), cl::Hidden);
 
 static cl::opt<bool>
     EnableBranchTargets("aarch64-enable-branch-targets", cl::Hidden,
                         cl::desc("Enable the AAcrh64 branch target pass"),
                         cl::init(true));
 
+extern cl::opt<bool> EnableHomogeneousPrologEpilog;
+
 extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAArch64Target() {
   // Register the target.
   RegisterTargetMachine<AArch64leTargetMachine> X(getTheAArch64leTarget());
   RegisterTargetMachine<AArch64beTargetMachine> Y(getTheAArch64beTarget());
   RegisterTargetMachine<AArch64leTargetMachine> Z(getTheARM64Target());
   RegisterTargetMachine<AArch64leTargetMachine> W(getTheARM64_32Target());
   RegisterTargetMachine<AArch64leTargetMachine> V(getTheAArch64_32Target());
   auto PR = PassRegistry::getPassRegistry();
extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAArch64Target() {
   initializeFalkorHWPFFixPass(*PR);
   initializeFalkorMarkStridedAccessesLegacyPass(*PR);
   initializeLDTLSCleanupPass(*PR);
   initializeSVEIntrinsicOptsPass(*PR);
   initializeAArch64SpeculationHardeningPass(*PR);
   initializeAArch64SLSHardeningPass(*PR);
   initializeAArch64StackTaggingPass(*PR);
   initializeAArch64StackTaggingPreRAPass(*PR);
+  initializeAArch64LowerHomogeneousPrologEpilogPass(*PR);
 }
 
 //===----------------------------------------------------------------------===//
 // AArch64 Lowering public interface.
 //===----------------------------------------------------------------------===//
 static std::unique_ptr<TargetLoweringObjectFile> createTLOF(const Triple &TT) {
   if (TT.isOSBinFormatMachO())
     return std::make_unique<AArch64_MachoTargetObjectFile>();
if (TM->getOptLevel() != CodeGenOpt::None && EnableRedundantCopyElimination)
     addPass(createAArch64RedundantCopyEliminationPass());
 
   if (TM->getOptLevel() != CodeGenOpt::None && usingDefaultRegAlloc())
     // Improve performance for some FP/SIMD code for A57.
     addPass(createAArch64A57FPLoadBalancing());
 }
 
 void AArch64PassConfig::addPreSched2() {
+  // Lower homogeneous frame instructions
+  if (EnableHomogeneousPrologEpilog)
+    addPass(createAArch64LowerHomogeneousPrologEpilogPass());
   // Expand some pseudo instructions to allow proper scheduling.
   addPass(createAArch64ExpandPseudoPass());
   // Use load/store pair instructions when possible.
   if (TM->getOptLevel() != CodeGenOpt::None) {
     if (EnableLoadStoreOpt)
       addPass(createAArch64LoadStoreOptimizationPass());
   }
 

llvm/lib/Target/AArch64/CMakeLists.txt

add_llvm_target(AArch64CodeGen
   AArch64FrameLowering.cpp
   AArch64CompressJumpTables.cpp
   AArch64ConditionOptimizer.cpp
   AArch64RedundantCopyElimination.cpp
   AArch64ISelDAGToDAG.cpp
   AArch64ISelLowering.cpp
   AArch64InstrInfo.cpp
   AArch64LoadStoreOptimizer.cpp
+  AArch64LowerHomogeneousPrologEpilog.cpp
   AArch64MachineFunctionInfo.cpp
   AArch64MacroFusion.cpp
   AArch64MCInstLower.cpp
   AArch64PromoteConstant.cpp
   AArch64PBQPRegAlloc.cpp
   AArch64RegisterInfo.cpp
   AArch64SLSHardening.cpp
   AArch64SelectionDAGInfo.cpp

llvm/test/CodeGen/AArch64/arm64-homogeneous-prolog-epilog-bad-outline.mir

+# RUN: llc -mtriple=arm64-applie-ios7.0 -homogeneous-prolog-epilog %s -o - | FileCheck %s
+#
+# This test ensure no outlined epilog is formed when X16 is live across the helper.
+--- |
+  @FuncPtr = local_unnamed_addr global i32 (i32)* null, align 8
+
+  define i32 @_Z3fooi(i32) minsize "frame-pointer"="all" {
+    ret i32 0
+  }
+
+  declare i32 @_Z3gooii(i32, i32)
+...
+---
+name:            _Z3fooi
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $w0
+
+    $w1 = COPY $w0
+    $w19 = nsw ADDWri $w0, 1, 0
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    $w0 = COPY renamable $w19
+    BL @_Z3gooii, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit $w0, implicit $w1, implicit-def $sp, implicit-def $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+    $x8 = ADRP target-flags(aarch64-page) @FuncPtr
+    $x16 = LDRXui killed renamable $x8, target-flags(aarch64-pageoff, aarch64-nc) @FuncPtr
+    $w0 = nsw ADDWrr killed renamable $w0, killed renamable $w19
+    TCRETURNri killed renamable $x16, 0, csr_aarch64_aapcs, implicit $sp, implicit $w0
+
+# CHECK: _OUTLINED_FUNCTION_PROLOG_FRAME16_x30x29x19x20:
+# CHECK-NOT: _OUTLINED_FUNCTION_EPILOG_x30x29x19x20:

llvm/test/CodeGen/AArch64/arm64-homogeneous-prolog-epilog-frame-tail.ll

+; RUN: llc < %s -mtriple=arm64-apple-ios7.0  -homogeneous-prolog-epilog | FileCheck %s
+; RUN: llc < %s -mtriple=aarch64-unknown-linux-gnu  -homogeneous-prolog-epilog | FileCheck %s --check-prefixes=CHECK-LINUX
+
+; CHECK-LABEL: __Z3foofffi:
+; CHECK:      stp     x29, x30, [sp, #-16]!
+; CHECK-NEXT: bl      _OUTLINED_FUNCTION_PROLOG_FRAME48_x30x29x19x20d8d9d10d11
+; CHECK:      bl      __Z3goof
+; CHECK:      bl      __Z3goof
+; CHECK:      b       _OUTLINED_FUNCTION_EPILOG_TAIL_x30x29x19x20d8d9d10d11
+
+; CHECK-LINUX-LABEL: _Z3foofffi:
+; CHECK-LINUX:      stp     x29, x30, [sp, #-32]!
+; CHECK-LINUX-NEXT: bl      OUTLINED_FUNCTION_PROLOG_FRAME32_x19x20x30x29d8d9d10d11
+; CHECK-LINUX:      bl      _Z3goof
+; CHECK-LINUX:      bl      _Z3goof
+; CHECK-LINUX:      b       OUTLINED_FUNCTION_EPILOG_TAIL_x19x20x30x29d8d9d10d11
+
+define float @_Z3foofffi(float %b, float %x, float %y, i32 %z) ssp minsize "frame-pointer"="non-leaf" {
+entry:
+  %inc = fadd float %b, 1.000000e+00
+  %add = fadd float %inc, %x
+  %add1 = fadd float %add, %y
+  %conv = sitofp i32 %z to float
+  %sub = fsub float %add1, %conv
+  %dec = add nsw i32 %z, -1
+  %call = tail call float @_Z3goof(float %inc) #2
+  %call2 = tail call float @_Z3goof(float %sub) #2
+  %add3 = fadd float %call, %call2
+  %mul = fmul float %inc, %add3
+  %add4 = fadd float %sub, %mul
+  %conv5 = sitofp i32 %dec to float
+  %sub6 = fsub float %add4, %conv5
+  ret float %sub6
+}
+
+; CHECK-LABEL: _Z3zoov:
+; CHECK:      stp     x29, x30, [sp, #-16]!
+; CHECK:      bl      __Z3hoo
+; CHECK:      b       _OUTLINED_FUNCTION_EPILOG_TAIL_x30x29
+
+define i32 @_Z3zoov() nounwind ssp minsize {
+  %1 = tail call i32 @_Z3hoov() #2
+  %2 = add nsw i32 %1, 1
+  ret i32 %2
+}
+
+
+declare float @_Z3goof(float) nounwind ssp minsize
+declare i32 @_Z3hoov() nounwind ssp optsize
+
+; CHECK-LABEL:  _OUTLINED_FUNCTION_PROLOG_FRAME48_x30x29x19x20d8d9d10d11:
+; CHECK:      stp     d11, d10, [sp, #-48]!
+; CHECK-NEXT: stp     d9, d8, [sp, #16]
+; CHECK-NEXT: stp     x20, x19, [sp, #32]
+; CHECK-NEXT: add     x29, sp, #48
+; CHECK-NEXT: ret
+
+; CHECK-LABEL: _OUTLINED_FUNCTION_EPILOG_TAIL_x30x29x19x20d8d9d10d11:
+; CHECK:      ldp     x29, x30, [sp, #48]
+; CHECK-NEXT: ldp     x20, x19, [sp, #32]
+; CHECK-NEXT: ldp     d9, d8, [sp, #16]
+; CHECK-NEXT: ldp     d11, d10, [sp], #64
+; CHECK-NEXT: ret
+
+; CHECK-LABEL: _OUTLINED_FUNCTION_EPILOG_TAIL_x30x29:
+; CHECK:      ldp     x29, x30, [sp], #16
+; CHECK-NEXT: ret
+
+; CHECK-LINUX-LABEL:  OUTLINED_FUNCTION_PROLOG_FRAME32_x19x20x30x29d8d9d10d11:
+; CHECK-LINUX:      stp     d11, d10, [sp, #-32]!
+; CHECK-LINUX-NEXT: stp     d9, d8, [sp, #16]
+; CHECK-LINUX-NEXT: stp     x20, x19, [sp, #48]
+; CHECK-LINUX-NEXT: add     x29, sp, #32
+; CHECK-LINUX-NEXT: ret
+
+; CHECK-LINUX-LABEL: OUTLINED_FUNCTION_EPILOG_TAIL_x19x20x30x29d8d9d10d11:
+; CHECK-LINUX:      ldp     x20, x19, [sp, #48]
+; CHECK-LINUX-NEXT: ldp     x29, x30, [sp, #32]
+; CHECK-LINUX-NEXT: ldp     d9, d8, [sp, #16]
+; CHECK-LINUX-NEXT: ldp     d11, d10, [sp], #64
+; CHECK-LINUX-NEXT: ret
+
+; CHECK-LINUX-LABEL: OUTLINED_FUNCTION_EPILOG_TAIL_x30x29:
+; CHECK-LINUX:      ldp     x29, x30, [sp], #16
+; CHECK-LINUX-NEXT: ret

llvm/test/CodeGen/AArch64/arm64-homogeneous-prolog-epilog-no-helper.ll

+; RUN: llc < %s -mtriple=arm64-apple-ios7.0  -homogeneous-prolog-epilog -frame-helper-size-threshold=6 | FileCheck %s
+; RUN: llc < %s -mtriple=aarch64-unknown-linux-gnu  -homogeneous-prolog-epilog -frame-helper-size-threshold=6 | FileCheck %s --check-prefixes=CHECK-LINUX
+
+; CHECK-LABEL: __Z3foofffi:
+; CHECK:      stp     d11, d10, [sp, #-64]!
+; CHECK-NEXT: stp     d9, d8, [sp, #16]
+; CHECK-NEXT: stp     x20, x19, [sp, #32]
+; CHECK-NEXT: stp     x29, x30, [sp, #48]
+; CHECK-NEXT: add     x29, sp, #48
+; CHECK:      bl      __Z3goof
+; CHECK:      bl      __Z3goof
+; CHECK:      ldp     x29, x30, [sp, #48]
+; CHECK:      ldp     x20, x19, [sp, #32]
+; CHECK:      ldp     d9, d8, [sp, #16]
+; CHECK:      ldp     d11, d10, [sp], #64
+; CHECK:      ret
+
+; CHECK-LINUX-LABEL: _Z3foofffi:
+; CHECK-LINUX:      stp     d11, d10, [sp, #-64]!
+; CHECK-LINUX-NEXT: stp     d9, d8, [sp, #16]
+; CHECK-LINUX-NEXT: stp     x29, x30, [sp, #32]
+; CHECK-LINUX-NEXT: stp     x20, x19, [sp, #48]
+; CHECK-LINUX-NEXT: add     x29, sp, #32
+; CHECK-LINUX:      bl      _Z3goof
+; CHECK-LINUX:      bl      _Z3goof
+; CHECK-LINUX:      ldp     x20, x19, [sp, #48]
+; CHECK-LINUX:      ldp     x29, x30, [sp, #32]
+; CHECK-LINUX:      ldp     d9, d8, [sp, #16]
+; CHECK-LINUX:      ldp     d11, d10, [sp], #64
+; CHECK-LINUX:      ret
+
+define float @_Z3foofffi(float %b, float %x, float %y, i32 %z) uwtable ssp minsize "frame-pointer"="non-leaf" {
+entry:
+  %inc = fadd float %b, 1.000000e+00
+  %add = fadd float %inc, %x
+  %add1 = fadd float %add, %y
+  %conv = sitofp i32 %z to float
+  %sub = fsub float %add1, %conv
+  %dec = add nsw i32 %z, -1
+  %call = tail call float @_Z3goof(float %inc) #2
+  %call2 = tail call float @_Z3goof(float %sub) #2
+  %add3 = fadd float %call, %call2
+  %mul = fmul float %inc, %add3
+  %add4 = fadd float %sub, %mul
+  %conv5 = sitofp i32 %dec to float
+  %sub6 = fsub float %add4, %conv5
+  ret float %sub6
+}
+
+; CHECK-LABEL: __Z3zoov:
+; CHECK:      stp     x29, x30, [sp, #-16]!
+; CHECK:      bl      __Z3hoo
+; CHECK:      ldp     x29, x30, [sp], #16
+; CHECK-NEXT: ret
+
+; CHECK-LINUX-LABEL: _Z3zoov:
+; CHECK-LINUX:      stp     x29, x30, [sp, #-16]!
+; CHECK-LINUX:      bl      _Z3hoo
+; CHECK-LINUX:      ldp     x29, x30, [sp], #16
+; CHECK-LINUX-NEXT: ret
+
+define i32 @_Z3zoov() nounwind ssp minsize {
+  %1 = tail call i32 @_Z3hoov() #2
+  %2 = add nsw i32 %1, 1
+  ret i32 %2
+}
+
+
+declare float @_Z3goof(float) nounwind ssp minsize
+declare i32 @_Z3hoov() nounwind ssp minsize

llvm/test/CodeGen/AArch64/arm64-homogeneous-prolog-epilog.ll

+; RUN: llc < %s -mtriple=arm64-apple-ios7.0 -homogeneous-prolog-epilog| FileCheck %s
+; RUN: llc < %s -mtriple=aarch64-unknown-linux-gnu  -homogeneous-prolog-epilog | FileCheck %s --check-prefixes=CHECK-LINUX
+
+; CHECK-LABEL: __Z3hooii:
+; CHECK:      stp     x29, x30, [sp, #-16]!
+; CHECK-NEXT: bl      _OUTLINED_FUNCTION_PROLOG_x30x29x19x20x21x22
+; CHECK:      bl      __Z3gooi
+; CHECK:      bl      __Z3gooi
+; CHECK:      bl      _OUTLINED_FUNCTION_EPILOG_x30x29x19x20x21x22
+; CHECK-NEXT: b __Z3gooi
+
+; CHECK-LINUX-LABEL: _Z3hooii:
+; CHECK-LINUX:      stp     x29, x30, [sp, #-48]!
+; CHECK-LINUX-NEXT: bl      OUTLINED_FUNCTION_PROLOG_x19x20x21x22x30x29
+; CHECK-LINUX:      bl      _Z3gooi
+; CHECK-LINUX:      bl      _Z3gooi
+; CHECK-LINUX:      bl      OUTLINED_FUNCTION_EPILOG_x19x20x21x22x30x29
+; CHECK-LINUX-NEXT: b _Z3gooi
+
+define i32 @_Z3hooii(i32 %b, i32 %a) nounwind ssp minsize {
+  %1 = tail call i32 @_Z3gooi(i32 %b)
+  %2 = tail call i32 @_Z3gooi(i32 %a)
+  %3 = add i32 %a, %b
+  %4 = add i32 %3, %1
+  %5 = add i32 %4, %2
+  %6 = tail call i32 @_Z3gooi(i32 %5)
+  ret i32 %6
+}
+
+declare i32 @_Z3gooi(i32);
+
+
+; CHECK-LABEL: _OUTLINED_FUNCTION_PROLOG_x30x29x19x20x21x22:
+; CHECK:      stp     x22, x21, [sp, #-32]!
+; CHECK-NEXT: stp     x20, x19, [sp, #16]
+; CHECK-NEXT: ret
+
+; CHECK-LABEL: _OUTLINED_FUNCTION_EPILOG_x30x29x19x20x21x22:
+; CHECK:      mov     x16, x30
+; CHECK-NEXT: ldp     x29, x30, [sp, #32]
+; CHECK-NEXT: ldp     x20, x19, [sp, #16]
+; CHECK-NEXT: ldp     x22, x21, [sp], #48
+; CHECK-NEXT: br      x16
+
+; CHECK-LINUX-LABEL: OUTLINED_FUNCTION_PROLOG_x19x20x21x22x30x29:
+; CHECK-LINUX:      stp     x22, x21, [sp, #16]
+; CHECK-LINUX-NEXT: stp     x20, x19, [sp, #32]
+; CHECK-LINUX-NEXT: ret
+
+; CHECK-LINUX-LABEL: OUTLINED_FUNCTION_EPILOG_x19x20x21x22x30x29:
+; CHECK-LINUX:      mov     x16, x30
+; CHECK-LINUX-NEXT: ldp     x20, x19, [sp, #32]
+; CHECK-LINUX-NEXT: ldp     x22, x21, [sp, #16]
+; CHECK-LINUX-NEXT: ldp     x29, x30, [sp], #48
+; CHECK-LINUX-NEXT: br      x16