IR/AArch64/X86: add "swifttailcc" calling convention.

[t.p.northover]

Summary

Swift's new concurrency features are going to require guaranteed tail calls so that they don't consume excessive amounts of stack space. This would normally mean tailcc, but there are also Swift-specific ABI desires that don't naturally go along with "tailcc" so this adds another calling convention that's the combination of swiftcc and tailcc.

Since the changes to support tailcc in AArch64 were virtually identical to supporting swifttailcc I decided it would be silly not to do that at the same time. I can split them up again for commit if people would prefer.

[varungandhi-apple]

Could you add a short entry on swifttailcc to LangRef.rst?

[t.p.northover]

Of course, should have spotted that one.

[varungandhi-apple]

llvm/docs/CodeGenerator.rst

2066–2077

I think this section needs to be updated to also mention swifttailcc.

[t.p.northover]

• Updated documentation as requested.
• Changed CallSeq_start AArch64 operand so we don't reserve the stack space twice.

[t.p.northover]

Spotted a couple of issues over the last week:

• We sometimes restored SP to the very beginning and then subtracted it back again to "reallocate" argument storage on AArch64. If there's no red-zone this leaves arguments vulnerable to an interrupt.
• Added new keyword to Emacs & Vim highlighters

[t.p.northover]

In this update:

• Switched swiftasync and swiftself parameters to caller-save because otherwise they'd be clobbered by the epilogue in a tail call.

[t.p.northover]

Ping.

[t.p.northover]

Ping.

[t.p.northover]

Ping.

[t.p.northover]

Added support for i386 (for the watch simulator). Mostly came for free but some slight issues around swiftself and CSRs.

[t.p.northover]

Ping.

[t.p.northover]

Ping.

[jroelofs]

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

196–203

measured in bytes?

321

@compnerd might know?

1754

What if the pop size is larger than the immediate field in the ldp is able to encode?

llvm/test/CodeGen/ARM/swifttailcc.ll

3 ↗

(On Diff #335488)

Is the swifttail attribute implicit here?

Seems like there should be tighter CHECKs for what is expected around the vadd.f32.

[paquette]

llvm/docs/CodeGenerator.rst

2071–2072

Is tailcc supported on AArch64 yet? If not, this is somewhat misleading.

[paquette]

llvm/docs/CodeGenerator.rst

2071–2072

oh, duh, this patch adds that

[paquette]

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

196–203

Should be a doxygen comment?

llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h

57

This should be a Doxygen comment?

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

1018

Why did this change?

[t.p.northover]

Thanks for the comments, I'll upload a new patch.

llvm/docs/CodeGenerator.rst

2071–2072

Yes. I debated whether it was closely related enough to include, but it just seemed silly to be updating exactly the places that would need to know about tailcc for it to work and not do it at the same time.

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

196–203

Done because it's harmless either way, but does doxygen care about static functions?

321

Possibly, but that's a whole other topic. Just making sure we fail noisily.

1754

The function is probably better named tryConvert..., if it can't fold it'll insert a normal sub sp, sp, #imm.

But you're right I think there is a risk of overflow with the arg area possibly being large, and there's not actually an overflow check in that function. I'll add one.

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

1018

Minor bugfix. The immediate tells stack allocation how much space it should reserve for this call's stack-based arguments, but because it's a tail call its stack space is actually allocated independently at the other end of the frame (before even the callee-saved registers get pushed).

The practical effect of this is that with NumBytes the function was allocating space in both places, using more stack than is necessary.

llvm/test/CodeGen/ARM/swifttailcc.ll

3 ↗

(On Diff #335488)

I have no idea what this file is here for. It should probably be part of the ARM implementation if anything, but even there it seems useless. I'll remove it from this review.

[paquette]

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

196–203

Yep: https://llvm.org/doxygen/AArch64InstructionSelector_8cpp.html

321

I feel like this should actually be something heavier than an assert? If someone builds without asserts and tries to use this on Windows, it'd be unfortunate if they ended up with something totally broken.

2387

The assert in the original code checks that ByteOffset % 16 == 0, why did that change here?

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

1018

Ah, I see. Makes sense.

[t.p.northover]

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

2387

If you're asking why the direction of the test changed, it's because the assert was after ByteOffset had been incremented by +/-8 in this block.

The reason the check's needed at all now is that there are multiple potential sources of misalignment (odd # of CSRs, and the new context store), and they might or might not combine to realign the stack again.

[paquette]

LGTM

[t.p.northover]

Thanks. Committed as 82a0e808bb2c.

llvm/docs/BitCodeFormat.rst

* *callingconv*: The calling convention number:
   * ``coldcc``: code 9
   * ``webkit_jscc``: code 12
   * ``anyregcc``: code 13
   * ``preserve_mostcc``: code 14
   * ``preserve_allcc``: code 15
   * ``swiftcc`` : code 16
   * ``cxx_fast_tlscc``: code 17
   * ``tailcc`` : code 18
+  * ``cfguard_checkcc`` : code 19
+  * ``swifttailcc`` : code 20
   * ``x86_stdcallcc``: code 64
   * ``x86_fastcallcc``: code 65
   * ``arm_apcscc``: code 66
   * ``arm_aapcscc``: code 67
   * ``arm_aapcs_vfpcc``: code 68
 
 * isproto*: Non-zero if this entry represents a declaration rather than a
   definition

llvm/docs/CodeGenerator.rst

 object files are not marked the way the gold linker expects, but simple Go
 programs can be built by dragonegg.
 
 .. _tail call section:
 
 Tail call optimization
 ----------------------
 
 Tail call optimization, callee reusing the stack of the caller, is currently
-supported on x86/x86-64, PowerPC, and WebAssembly. It is performed on x86/x86-64
-and PowerPC if:
+supported on x86/x86-64, PowerPC, AArch64, and WebAssembly. It is performed on
+x86/x86-64, PowerPC, and AArch64 if:
 
 * Caller and callee have the calling convention ``fastcc``, ``cc 10`` (GHC
-  calling convention), ``cc 11`` (HiPE calling convention), or ``tailcc``.
+  calling convention), ``cc 11`` (HiPE calling convention), ``tailcc``, or
+  ``swifttailcc``.
 
 * The call is a tail call - in tail position (ret immediately follows call and
   ret uses value of call or is void).
 
 * Option ``-tailcallopt`` is enabled or the calling convention is ``tailcc``.
 
 * Platform-specific constraints are met.
 
 x86/x86-64 constraints:
 
 * No variable argument lists are used.
 
 * On x86-64 when generating GOT/PIC code only module-local calls (visibility =

 
 * No variable argument lists are used
 
 * The 'tail-call' target attribute is enabled.
 
 * The caller and callee's return types must match. The caller cannot
   be void unless the callee is, too.
 
+AArch64 constraints:
+
+* No variable argument lists are used.
+
 Example:
 
 Call as ``llc -tailcallopt test.ll``.
 
 .. code-block:: llvm
 
   declare fastcc i32 @tailcallee(i32 inreg %a1, i32 inreg %a2, i32 %a3, i32 %a4)
 

llvm/docs/LangRef.rst

"``cxx_fast_tlscc``" - The `CXX_FAST_TLS` calling convention for access functions
     how arguments and return values are passed, but it uses a different set of
     caller/callee-saved registers.
 
     Given that each platform has its own lowering sequence, hence its own set
     of preserved registers, we can't use the existing `PreserveMost`.
 
     - On X86-64 the callee preserves all general purpose registers, except for
       RDI and RAX.
-"``swiftcc``" - This calling convention is used for Swift language.
-    - On X86-64 RCX and R8 are available for additional integer returns, and
-      XMM2 and XMM3 are available for additional FP/vector returns.
-    - On iOS platforms, we use AAPCS-VFP calling convention.
 "``tailcc``" - Tail callable calling convention
     This calling convention ensures that calls in tail position will always be
     tail call optimized. This calling convention is equivalent to fastcc,
     except for an additional guarantee that tail calls will be produced
     whenever possible. `Tail calls can only be optimized when this, the fastcc,
     the GHC or the HiPE convention is used. <CodeGenerator.html#id80>`_ This
     calling convention does not support varargs and requires the prototype of
     all callees to exactly match the prototype of the function definition.
+"``swiftcc``" - This calling convention is used for Swift language.
+    - On X86-64 RCX and R8 are available for additional integer returns, and
+      XMM2 and XMM3 are available for additional FP/vector returns.
+    - On iOS platforms, we use AAPCS-VFP calling convention.
+"``swifttailcc``"
+    This calling convention is like ``swiftcc`` in most respects, but also the
+    callee pops the argument area of the stack so that mandatory tail calls are 
+    possible as in ``tailcc``.
 "``cfguard_checkcc``" - Windows Control Flow Guard (Check mechanism)
     This calling convention is used for the Control Flow Guard check function,
     calls to which can be inserted before indirect calls to check that the call
     target is a valid function address. The check function has no return value,
     but it will trigger an OS-level error if the address is not a valid target.
     The set of registers preserved by the check function, and the register
     containing the target address are architecture-specific.
 

 
 The '``llvm.swift.async.context.addr``' intrinsic returns a pointer to
 the part of the extended frame record containing the asynchronous
 context of a Swift execution.
 
 Semantics:
 """"""""""
 
-If the function has a ``swiftasync`` parameter, that argument will initially
+If the caller has a ``swiftasync`` parameter, that argument will initially
 be stored at the returned address. If not, it will be initialized to null.
 
 '``llvm.localescape``' and '``llvm.localrecover``' Intrinsics
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 

llvm/include/llvm/AsmParser/LLToken.h

enum Kind {
   kw_ptx_device,
   kw_spir_kernel,
   kw_spir_func,
   kw_x86_64_sysvcc,
   kw_win64cc,
   kw_webkit_jscc,
   kw_anyregcc,
   kw_swiftcc,
+  kw_swifttailcc,
   kw_preserve_mostcc,
   kw_preserve_allcc,
   kw_ghccc,
   kw_x86_intrcc,
   kw_hhvmcc,
   kw_hhvm_ccc,
   kw_cxx_fast_tlscc,
   kw_amdgpu_vs,

llvm/include/llvm/IR/CallingConv.h

enum {
     Tail = 18,
 
     /// Special calling convention on Windows for calling the Control
     /// Guard Check ICall funtion. The function takes exactly one argument
     /// (address of the target function) passed in the first argument register,
     /// and has no return value. All register values are preserved.
     CFGuard_Check = 19,
 
+    /// SwiftTail - This follows the Swift calling convention in how arguments
+    /// are passed but guarantees tail calls will be made by making the callee
+    /// clean up their stack.
+    SwiftTail = 20,
+
     // Target - This is the start of the target-specific calling conventions,
     // e.g. fastcall and thiscall on X86.
     FirstTargetCC = 64,
 
     /// X86_StdCall - stdcall is the calling conventions mostly used by the
     /// Win32 API. It is basically the same as the C convention with the
     /// difference in that the callee is responsible for popping the arguments
     /// from the stack.

llvm/lib/AsmParser/LLLexer.cpp

#define KEYWORD(STR)                                                           \
   KEYWORD(spir_kernel);
   KEYWORD(spir_func);
   KEYWORD(intel_ocl_bicc);
   KEYWORD(x86_64_sysvcc);
   KEYWORD(win64cc);
   KEYWORD(x86_regcallcc);
   KEYWORD(webkit_jscc);
   KEYWORD(swiftcc);
+  KEYWORD(swifttailcc);
   KEYWORD(anyregcc);
   KEYWORD(preserve_mostcc);
   KEYWORD(preserve_allcc);
   KEYWORD(ghccc);
   KEYWORD(x86_intrcc);
   KEYWORD(hhvmcc);
   KEYWORD(hhvm_ccc);
   KEYWORD(cxx_fast_tlscc);

llvm/lib/AsmParser/LLParser.cpp

 ///   ::= 'x86_64_sysvcc'
 ///   ::= 'win64cc'
 ///   ::= 'webkit_jscc'
 ///   ::= 'anyregcc'
 ///   ::= 'preserve_mostcc'
 ///   ::= 'preserve_allcc'
 ///   ::= 'ghccc'
 ///   ::= 'swiftcc'
+///   ::= 'swifttailcc'
 ///   ::= 'x86_intrcc'
 ///   ::= 'hhvmcc'
 ///   ::= 'hhvm_ccc'
 ///   ::= 'cxx_fast_tlscc'
 ///   ::= 'amdgpu_vs'
 ///   ::= 'amdgpu_ls'
 ///   ::= 'amdgpu_hs'
 ///   ::= 'amdgpu_es'
bool LLParser::parseOptionalCallingConv(unsigned &CC) {
   case lltok::kw_x86_64_sysvcc:  CC = CallingConv::X86_64_SysV; break;
   case lltok::kw_win64cc:        CC = CallingConv::Win64; break;
   case lltok::kw_webkit_jscc:    CC = CallingConv::WebKit_JS; break;
   case lltok::kw_anyregcc:       CC = CallingConv::AnyReg; break;
   case lltok::kw_preserve_mostcc:CC = CallingConv::PreserveMost; break;
   case lltok::kw_preserve_allcc: CC = CallingConv::PreserveAll; break;
   case lltok::kw_ghccc:          CC = CallingConv::GHC; break;
   case lltok::kw_swiftcc:        CC = CallingConv::Swift; break;
+  case lltok::kw_swifttailcc:    CC = CallingConv::SwiftTail; break;
   case lltok::kw_x86_intrcc:     CC = CallingConv::X86_INTR; break;
   case lltok::kw_hhvmcc:         CC = CallingConv::HHVM; break;
   case lltok::kw_hhvm_ccc:       CC = CallingConv::HHVM_C; break;
   case lltok::kw_cxx_fast_tlscc: CC = CallingConv::CXX_FAST_TLS; break;
   case lltok::kw_amdgpu_vs:      CC = CallingConv::AMDGPU_VS; break;
   case lltok::kw_amdgpu_gfx:     CC = CallingConv::AMDGPU_Gfx; break;
   case lltok::kw_amdgpu_ls:      CC = CallingConv::AMDGPU_LS; break;
   case lltok::kw_amdgpu_hs:      CC = CallingConv::AMDGPU_HS; break;

llvm/lib/CodeGen/Analysis.cpp

bool llvm::isInTailCallPosition(const CallBase &Call, const TargetMachine &TM) {
   // The block must end in a return statement or unreachable.
   //
   // FIXME: Decline tailcall if it's not guaranteed and if the block ends in
   // an unreachable, for now. The way tailcall optimization is currently
   // implemented means it will add an epilogue followed by a jump. That is
   // not profitable. Also, if the callee is a special function (e.g.
   // longjmp on x86), it can end up causing miscompilation that has not
   // been fully understood.
-  if (!Ret &&
-      ((!TM.Options.GuaranteedTailCallOpt &&
-        Call.getCallingConv() != CallingConv::Tail) || !isa<UnreachableInst>(Term)))
+  if (!Ret && ((!TM.Options.GuaranteedTailCallOpt &&
+                Call.getCallingConv() != CallingConv::Tail &&
+                Call.getCallingConv() != CallingConv::SwiftTail) ||
+               !isa<UnreachableInst>(Term)))
     return false;
 
   // If I will have a chain, make sure no other instruction that will have a
   // chain interposes between I and the return.
   // Check for all calls including speculatable functions.
   for (BasicBlock::const_iterator BBI = std::prev(ExitBB->end(), 2);; --BBI) {
     if (&*BBI == &Call)
       break;

llvm/lib/IR/AsmWriter.cpp

static void PrintCallingConv(unsigned cc, raw_ostream &Out) {
   case CallingConv::AVR_SIGNAL:    Out << "avr_signalcc "; break;
   case CallingConv::PTX_Kernel:    Out << "ptx_kernel"; break;
   case CallingConv::PTX_Device:    Out << "ptx_device"; break;
   case CallingConv::X86_64_SysV:   Out << "x86_64_sysvcc"; break;
   case CallingConv::Win64:         Out << "win64cc"; break;
   case CallingConv::SPIR_FUNC:     Out << "spir_func"; break;
   case CallingConv::SPIR_KERNEL:   Out << "spir_kernel"; break;
   case CallingConv::Swift:         Out << "swiftcc"; break;
+  case CallingConv::SwiftTail:     Out << "swifttailcc"; break;
   case CallingConv::X86_INTR:      Out << "x86_intrcc"; break;
   case CallingConv::HHVM:          Out << "hhvmcc"; break;
   case CallingConv::HHVM_C:        Out << "hhvm_ccc"; break;
   case CallingConv::AMDGPU_VS:     Out << "amdgpu_vs"; break;
   case CallingConv::AMDGPU_LS:     Out << "amdgpu_ls"; break;
   case CallingConv::AMDGPU_HS:     Out << "amdgpu_hs"; break;
   case CallingConv::AMDGPU_ES:     Out << "amdgpu_es"; break;
   case CallingConv::AMDGPU_GS:     Out << "amdgpu_gs"; break;

llvm/lib/Target/AArch64/AArch64CallingConvention.td

 
 // Functions taking SVE arguments or returning an SVE type
 // must (additionally) preserve full Z8-Z23 and predicate registers P4-P15
 def CSR_AArch64_SVE_AAPCS : CalleeSavedRegs<(add (sequence "Z%u", 8, 23),
                                                  (sequence "P%u", 4, 15),
                                                  X19, X20, X21, X22, X23, X24,
                                                  X25, X26, X27, X28, LR, FP)>;
 
+def CSR_AArch64_AAPCS_SwiftTail
+    : CalleeSavedRegs<(sub CSR_AArch64_AAPCS, X20, X22)>;
+
 // Constructors and destructors return 'this' in the iOS 64-bit C++ ABI; since
 // 'this' and the pointer return value are both passed in X0 in these cases,
 // this can be partially modelled by treating X0 as a callee-saved register;
 // only the resulting RegMask is used; the SaveList is ignored
 //
 // (For generic ARM 64-bit ABI code, clang will not generate constructors or
 // destructors with 'this' returns, so this RegMask will not be used in that
 // case)
def CSR_Darwin_AArch64_AAVPCS : CalleeSavedRegs<(add LR, FP, X19, X20, X21,
                                                  X22, X23, X24, X25, X26, X27,
                                                  X28, (sequence "Q%u", 8, 23))>;
 def CSR_Darwin_AArch64_AAPCS_ThisReturn
     : CalleeSavedRegs<(add CSR_Darwin_AArch64_AAPCS, X0)>;
 
 def CSR_Darwin_AArch64_AAPCS_SwiftError
     : CalleeSavedRegs<(sub CSR_Darwin_AArch64_AAPCS, X21)>;
 
+def CSR_Darwin_AArch64_AAPCS_SwiftTail
+    : CalleeSavedRegs<(sub CSR_Darwin_AArch64_AAPCS, X20, X22)>;
+
 // The function used by Darwin to obtain the address of a thread-local variable
 // guarantees more than a normal AAPCS function. x16 and x17 are used on the
 // fast path for calculation, but other registers except X0 (argument/return)
 // and LR (it is a call, after all) are preserved.
 def CSR_Darwin_AArch64_TLS
     : CalleeSavedRegs<(add (sub (sequence "X%u", 1, 28), X16, X17),
                            FP,
                            (sequence "Q%u", 0, 31))>;

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

 
 cl::opt<bool> EnableHomogeneousPrologEpilog(
     "homogeneous-prolog-epilog", cl::init(false), cl::ZeroOrMore, cl::Hidden,
     cl::desc("Emit homogeneous prologue and epilogue for the size "
              "optimization (default = off)"));
 
 STATISTIC(NumRedZoneFunctions, "Number of functions using red zone");
 
-/// Returns the argument pop size.
-static uint64_t getArgumentPopSize(MachineFunction &MF,
+/// Returns how much of the incoming argument stack area (in bytes) we should
+/// clean up in an epilogue. For the C calling convention this will be 0, for
+/// guaranteed tail call conventions it can be positive (a normal return or a
+/// tail call to a function that uses less stack space for arguments) or
+/// negative (for a tail call to a function that needs more stack space than us
+/// for arguments).
+static int64_t getArgumentStackToRestore(MachineFunction &MF,
                                          MachineBasicBlock &MBB) {
   MachineBasicBlock::iterator MBBI = MBB.getLastNonDebugInstr();
   bool IsTailCallReturn = false;
   if (MBB.end() != MBBI) {
     unsigned RetOpcode = MBBI->getOpcode();
     IsTailCallReturn = RetOpcode == AArch64::TCRETURNdi ||
                        RetOpcode == AArch64::TCRETURNri ||
                        RetOpcode == AArch64::TCRETURNriBTI;
   }
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
 
-  uint64_t ArgumentPopSize = 0;
+  int64_t ArgumentPopSize = 0;
   if (IsTailCallReturn) {
     MachineOperand &StackAdjust = MBBI->getOperand(1);
 
     // For a tail-call in a callee-pops-arguments environment, some or all of
     // the stack may actually be in use for the call's arguments, this is
     // calculated during LowerCall and consumed here...
     ArgumentPopSize = StackAdjust.getImm();
   } else {
bool AArch64FrameLowering::homogeneousPrologEpilog(
   if (getSVEStackSize(MF))
     return false;
 
   // Bail on stack adjustment needed on return for simplicity.
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   const TargetRegisterInfo *RegInfo = MF.getSubtarget().getRegisterInfo();
   if (MFI.hasVarSizedObjects() || RegInfo->hasStackRealignment(MF))
     return false;
-  if (Exit && getArgumentPopSize(MF, *Exit))
+  if (Exit && getArgumentStackToRestore(MF, *Exit))
     return false;
 
   return true;
 }
 
 /// Returns true if CSRs should be paired.
 bool AArch64FrameLowering::producePairRegisters(MachineFunction &MF) const {
   return produceCompactUnwindFrame(MF) || homogeneousPrologEpilog(MF);

 }
 
 /// Returns the size of the fixed object area (allocated next to sp on entry)
 /// On Win64 this may include a var args area and an UnwindHelp object for EH.
 static unsigned getFixedObjectSize(const MachineFunction &MF,
                                    const AArch64FunctionInfo *AFI, bool IsWin64,
                                    bool IsFunclet) {
   if (!IsWin64 || IsFunclet) {
-    // Only Win64 uses fixed objects, and then only for the function (not
-    // funclets)
-    return 0;
+    return AFI->getTailCallReservedStack();
   } else {
+    if (AFI->getTailCallReservedStack() != 0)
+      report_fatal_error("cannot generate ABI-changing tail call for Win64");
     // Var args are stored here in the primary function.
     const unsigned VarArgsArea = AFI->getVarArgsGPRSize();
     // To support EH funclets we allocate an UnwindHelp object
     const unsigned UnwindHelpObject = (MF.hasEHFunclets() ? 8 : 0);
     return alignTo(VarArgsArea + UnwindHelpObject, 16);
   }
 }
 
static MachineBasicBlock::iterator convertCalleeSaveRestoreToSPPrePostIncDec(
   while (MBBI->getOpcode() == AArch64::STRXpost ||
          MBBI->getOpcode() == AArch64::LDRXpre ||
          MBBI->getOpcode() == AArch64::CFI_INSTRUCTION) {
     if (MBBI->getOpcode() != AArch64::CFI_INSTRUCTION)
       assert(MBBI->getOperand(0).getReg() != AArch64::SP);
     ++MBBI;
   }
   unsigned NewOpc;
-  int Scale = 1;
   switch (MBBI->getOpcode()) {
   default:
     llvm_unreachable("Unexpected callee-save save/restore opcode!");
   case AArch64::STPXi:
     NewOpc = AArch64::STPXpre;
-    Scale = 8;
     break;
   case AArch64::STPDi:
     NewOpc = AArch64::STPDpre;
-    Scale = 8;
     break;
   case AArch64::STPQi:
     NewOpc = AArch64::STPQpre;
-    Scale = 16;
     break;
   case AArch64::STRXui:
     NewOpc = AArch64::STRXpre;
     break;
   case AArch64::STRDui:
     NewOpc = AArch64::STRDpre;
     break;
   case AArch64::STRQui:
     NewOpc = AArch64::STRQpre;
     break;
   case AArch64::LDPXi:
     NewOpc = AArch64::LDPXpost;
-    Scale = 8;
     break;
   case AArch64::LDPDi:
     NewOpc = AArch64::LDPDpost;
-    Scale = 8;
     break;
   case AArch64::LDPQi:
     NewOpc = AArch64::LDPQpost;
-    Scale = 16;
     break;
   case AArch64::LDRXui:
     NewOpc = AArch64::LDRXpost;
     break;
   case AArch64::LDRDui:
     NewOpc = AArch64::LDRDpost;
     break;
   case AArch64::LDRQui:
     NewOpc = AArch64::LDRQpost;
     break;
   }
   // Get rid of the SEH code associated with the old instruction.
   if (NeedsWinCFI) {
     auto SEH = std::next(MBBI);
     if (AArch64InstrInfo::isSEHInstruction(*SEH))
       SEH->eraseFromParent();
   }
 
+  TypeSize Scale = TypeSize::Fixed(1);
+  unsigned Width;
+  int64_t MinOffset, MaxOffset;
+  bool Success = static_cast<const AArch64InstrInfo *>(TII)->getMemOpInfo(
+      NewOpc, Scale, Width, MinOffset, MaxOffset);
+  (void)Success;
+  assert(Success && "unknown load/store opcode");
+
   // If the first store isn't right where we want SP then we can't fold the
   // update in so create a normal arithmetic instruction instead.
-  if (MBBI->getOperand(MBBI->getNumOperands() - 1).getImm() != 0) {
+  if (MBBI->getOperand(MBBI->getNumOperands() - 1).getImm() != 0 ||
+      CSStackSizeInc < MinOffset || CSStackSizeInc > MaxOffset) {
     emitFrameOffset(MBB, MBBI, DL, AArch64::SP, AArch64::SP,
                     StackOffset::getFixed(CSStackSizeInc), TII,
                     InProlog ? MachineInstr::FrameSetup
                              : MachineInstr::FrameDestroy);
     return std::prev(MBBI);
   }
 
   MachineInstrBuilder MIB = BuildMI(MBB, MBBI, DL, TII->get(NewOpc));
   MIB.addReg(AArch64::SP, RegState::Define);
 
   // Copy all operands other than the immediate offset.
   unsigned OpndIdx = 0;
   for (unsigned OpndEnd = MBBI->getNumOperands() - 1; OpndIdx < OpndEnd;
        ++OpndIdx)
     MIB.add(MBBI->getOperand(OpndIdx));
 
   assert(MBBI->getOperand(OpndIdx).getImm() == 0 &&
          "Unexpected immediate offset in first/last callee-save save/restore "
          "instruction!");
   assert(MBBI->getOperand(OpndIdx - 1).getReg() == AArch64::SP &&
          "Unexpected base register in callee-save save/restore instruction!");
   assert(CSStackSizeInc % Scale == 0);
-  MIB.addImm(CSStackSizeInc / Scale);
+  MIB.addImm(CSStackSizeInc / (int)Scale);
 
   MIB.setMIFlags(MBBI->getFlags());
   MIB.setMemRefs(MBBI->memoperands());
 
   // Generate a new SEH code that corresponds to the new instruction.
   if (NeedsWinCFI) {
     *HasWinCFI = true;
     InsertSEH(*MIB, *TII,
int64_t NumBytes = IsFunclet ? getWinEHFuncletFrameSize(MF)
                                : MFI.getStackSize();
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
 
   // All calls are tail calls in GHC calling conv, and functions have no
   // prologue/epilogue.
   if (MF.getFunction().getCallingConv() == CallingConv::GHC)
     return;
 
-  // Initial and residual are named for consistency with the prologue. Note that
-  // in the epilogue, the residual adjustment is executed first.
-  uint64_t ArgumentPopSize = getArgumentPopSize(MF, MBB);
+  // How much of the stack used by incoming arguments this function is expected
+  // to restore in this particular epilogue.
+  int64_t ArgumentStackToRestore = getArgumentStackToRestore(MF, MBB);
 
   // The stack frame should be like below,
   //
   //      ----------------------                     ---
   //      |                    |                      |
   //      | BytesInStackArgArea|              CalleeArgStackSize
   //      | (NumReusableBytes) |                (of tail call)
   //      |                    |                     ---
void AArch64FrameLowering::emitEpilogue(MachineFunction &MF,
   // it as the 2nd argument of AArch64ISD::TC_RETURN.
 
   auto Cleanup = make_scope_exit([&] { InsertReturnAddressAuth(MF, MBB); });
 
   bool IsWin64 =
       Subtarget.isCallingConvWin64(MF.getFunction().getCallingConv());
   unsigned FixedObject = getFixedObjectSize(MF, AFI, IsWin64, IsFunclet);
 
-  uint64_t AfterCSRPopSize = ArgumentPopSize;
+  int64_t AfterCSRPopSize = ArgumentStackToRestore;
   auto PrologueSaveSize = AFI->getCalleeSavedStackSize() + FixedObject;
   // We cannot rely on the local stack size set in emitPrologue if the function
   // has funclets, as funclets have different local stack size requirements, and
   // the current value set in emitPrologue may be that of the containing
   // function.
   if (MF.hasEHFunclets())
     AFI->setLocalStackSize(NumBytes - PrologueSaveSize);
   if (homogeneousPrologEpilog(MF, &MBB)) {
void AArch64FrameLowering::emitEpilogue(MachineFunction &MF,
 
   if (!CombineSPBump && PrologueSaveSize != 0) {
     MachineBasicBlock::iterator Pop = std::prev(MBB.getFirstTerminator());
     while (AArch64InstrInfo::isSEHInstruction(*Pop))
       Pop = std::prev(Pop);
     // Converting the last ldp to a post-index ldp is valid only if the last
     // ldp's offset is 0.
     const MachineOperand &OffsetOp = Pop->getOperand(Pop->getNumOperands() - 1);
-    // If the offset is 0, convert it to a post-index ldp.
-    if (OffsetOp.getImm() == 0)
+    // If the offset is 0 and the AfterCSR pop is not actually trying to
+    // allocate more stack for arguments (in space that an untimely interrupt
+    // may clobber), convert it to a post-index ldp.
+    if (OffsetOp.getImm() == 0 && AfterCSRPopSize >= 0)
       convertCalleeSaveRestoreToSPPrePostIncDec(
           MBB, Pop, DL, TII, PrologueSaveSize, NeedsWinCFI, &HasWinCFI, false);
     else {
       // If not, make sure to emit an add after the last ldp.
       // We're doing this by transfering the size to be restored from the
       // adjustment *before* the CSR pops to the adjustment *after* the CSR
       // pops.
       AfterCSRPopSize += PrologueSaveSize;
if (!IsFunclet && (MFI.hasVarSizedObjects() || AFI->isStackRealigned())) {
     emitFrameOffset(MBB, LastPopI, DL, AArch64::SP, AArch64::SP,
                     StackOffset::getFixed(NumBytes), TII,
                     MachineInstr::FrameDestroy, false, NeedsWinCFI);
 
   // This must be placed after the callee-save restore code because that code
   // assumes the SP is at the same location as it was after the callee-save save
   // code in the prologue.
   if (AfterCSRPopSize) {
+    assert(AfterCSRPopSize > 0 && "attempting to reallocate arg stack that an "
+                                  "interrupt may have clobbered");
     // Find an insertion point for the first ldp so that it goes before the
     // shadow call stack epilog instruction. This ensures that the restore of
     // lr from x18 is placed after the restore from sp.
     auto FirstSPPopI = MBB.getFirstTerminator();
     while (FirstSPPopI != Begin) {
       auto Prev = std::prev(FirstSPPopI);
       if (Prev->getOpcode() != AArch64::LDRXpre ||
           Prev->getOperand(0).getReg() == AArch64::SP)
         break;
       FirstSPPopI = Prev;
     }
 
     adaptForLdStOpt(MBB, FirstSPPopI, LastPopI);
 
     emitFrameOffset(MBB, FirstSPPopI, DL, AArch64::SP, AArch64::SP,
-                    StackOffset::getFixed((int64_t)AfterCSRPopSize), TII,
+                    StackOffset::getFixed(AfterCSRPopSize), TII,
                     MachineInstr::FrameDestroy, false, NeedsWinCFI, &HasWinCFI);
   }
   if (HasWinCFI)
     BuildMI(MBB, MBB.getFirstTerminator(), DL, TII->get(AArch64::SEH_EpilogEnd))
         .setMIFlag(MachineInstr::FrameDestroy);
 }
 
 /// getFrameIndexReference - Provide a base+offset reference to an FI slot for
static unsigned getPrologueDeath(MachineFunction &MF, unsigned Reg) {
   return getKillRegState(!IsLiveIn);
 }
 
 static bool produceCompactUnwindFrame(MachineFunction &MF) {
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   AttributeList Attrs = MF.getFunction().getAttributes();
   return Subtarget.isTargetMachO() &&
          !(Subtarget.getTargetLowering()->supportSwiftError() &&
-           Attrs.hasAttrSomewhere(Attribute::SwiftError));
+           Attrs.hasAttrSomewhere(Attribute::SwiftError)) &&
+         MF.getFunction().getCallingConv() != CallingConv::SwiftTail;
 }
 
 static bool invalidateWindowsRegisterPairing(unsigned Reg1, unsigned Reg2,
                                              bool NeedsWinCFI, bool IsFirst) {
   // If we are generating register pairs for a Windows function that requires
   // EH support, then pair consecutive registers only.  There are no unwind
   // opcodes for saves/restores of non-consectuve register pairs.
   // The unwind opcodes are save_regp, save_regp_x, save_fregp, save_frepg_x,
if (NeedsWinCFI) {
     ByteOffset = 0;
     StackFillDir = 1;
     // As the CSI array is reversed to match PrologEpilogInserter, iterate
     // backwards, to pair up registers starting from lower numbered registers.
     RegInc = -1;
     FirstReg = Count - 1;
   }
   int ScalableByteOffset = AFI->getSVECalleeSavedStackSize();
+  bool NeedGapToAlignStack = AFI->hasCalleeSaveStackFreeSpace();
 
   // When iterating backwards, the loop condition relies on unsigned wraparound.
   for (unsigned i = FirstReg; i < Count; i += RegInc) {
     RegPairInfo RPI;
     RPI.Reg1 = CSI[i].getReg();
 
     if (AArch64::GPR64RegClass.contains(RPI.Reg1))
       RPI.Type = RegPairInfo::GPR;
if (NeedsFrameRecord && AFI->hasSwiftAsyncContext() &&
         RPI.Reg2 == AArch64::FP)
       ByteOffset += StackFillDir * 8;
 
     assert(!(RPI.isScalable() && RPI.isPaired()) &&
            "Paired spill/fill instructions don't exist for SVE vectors");
 
     // Round up size of non-pair to pair size if we need to pad the
     // callee-save area to ensure 16-byte alignment.
-    if (AFI->hasCalleeSaveStackFreeSpace() && !NeedsWinCFI &&
+    if (NeedGapToAlignStack && !NeedsWinCFI &&
         !RPI.isScalable() && RPI.Type != RegPairInfo::FPR128 &&
-        !RPI.isPaired()) {
+        !RPI.isPaired() && ByteOffset % 16 != 0) {
       ByteOffset += 8 * StackFillDir;
-      assert(ByteOffset % 16 == 0);
       assert(MFI.getObjectAlign(RPI.FrameIdx) <= Align(16));
       // A stack frame with a gap looks like this, bottom up:
       // d9, d8. x21, gap, x20, x19.
-      // Set extra alignment on the x21 object (the only unpaired register)
-      // to create the gap above it.
+      // Set extra alignment on the x21 object to create the gap above it.
       MFI.setObjectAlignment(RPI.FrameIdx, Align(16));
+      NeedGapToAlignStack = false;
     }
 
     int OffsetPost = RPI.isScalable() ? ScalableByteOffset : ByteOffset;
     assert(OffsetPost % Scale == 0);
     // If filling top down (default), we want the offset after incrementing it.
     // If fillibg bootom up (WinCFI) we need the original offset.
     int Offset = NeedsWinCFI ? OffsetPre : OffsetPost;
 

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

case CallingConv::WebKit_JS:
     return CC_AArch64_WebKit_JS;
   case CallingConv::GHC:
     return CC_AArch64_GHC;
   case CallingConv::C:
   case CallingConv::Fast:
   case CallingConv::PreserveMost:
   case CallingConv::CXX_FAST_TLS:
   case CallingConv::Swift:
+  case CallingConv::SwiftTail:
+  case CallingConv::Tail:
     if (Subtarget->isTargetWindows() && IsVarArg)
       return CC_AArch64_Win64_VarArg;
     if (!Subtarget->isTargetDarwin())
       return CC_AArch64_AAPCS;
     if (!IsVarArg)
       return CC_AArch64_DarwinPCS;
     return Subtarget->isTargetILP32() ? CC_AArch64_DarwinPCS_ILP32_VarArg
                                       : CC_AArch64_DarwinPCS_VarArg;
for (unsigned i = 0; i != RVLocs.size(); ++i) {
 
     InVals.push_back(Val);
   }
 
   return Chain;
 }
 
 /// Return true if the calling convention is one that we can guarantee TCO for.
-static bool canGuaranteeTCO(CallingConv::ID CC) {
-  return CC == CallingConv::Fast;
+static bool canGuaranteeTCO(CallingConv::ID CC, bool GuaranteeTailCalls) {
+  return (CC == CallingConv::Fast && GuaranteeTailCalls) ||
+         CC == CallingConv::Tail || CC == CallingConv::SwiftTail;
 }
 
 /// Return true if we might ever do TCO for calls with this calling convention.
 static bool mayTailCallThisCC(CallingConv::ID CC) {
   switch (CC) {
   case CallingConv::C:
   case CallingConv::AArch64_SVE_VectorCall:
   case CallingConv::PreserveMost:
   case CallingConv::Swift:
+  case CallingConv::SwiftTail:
+  case CallingConv::Tail:
+  case CallingConv::Fast:
     return true;
   default:
-    return canGuaranteeTCO(CC);
+    return false;
   }
 }
 
 bool AArch64TargetLowering::isEligibleForTailCallOptimization(
     SDValue Callee, CallingConv::ID CalleeCC, bool isVarArg,
     const SmallVectorImpl<ISD::OutputArg> &Outs,
     const SmallVectorImpl<SDValue> &OutVals,
     const SmallVectorImpl<ISD::InputArg> &Ins, SelectionDAG &DAG) const {
for (Function::const_arg_iterator i = CallerF.arg_begin(),
     // call opt interferes with this. So we disable tail call opt when the
     // caller has an argument with "inreg" attribute.
 
     // FIXME: Check whether the callee also has an "inreg" argument.
     if (i->hasInRegAttr())
       return false;
   }
 
-  if (getTargetMachine().Options.GuaranteedTailCallOpt)
-    return canGuaranteeTCO(CalleeCC) && CCMatch;
+  if (canGuaranteeTCO(CalleeCC, getTargetMachine().Options.GuaranteedTailCallOpt))
+    return CCMatch;
 
   // Externally-defined functions with weak linkage should not be
   // tail-called on AArch64 when the OS does not support dynamic
   // pre-emption of symbols, as the AAELF spec requires normal calls
   // to undefined weak functions to be replaced with a NOP or jump to the
   // next instruction. The behaviour of branch instructions in this
   // situation (as used for tail calls) is implementation-defined, so we
   // cannot rely on the linker replacing the tail call with a return.
if (LoadSDNode *L = dyn_cast<LoadSDNode>(*U))
         }
 
   // Build a tokenfactor for all the chains.
   return DAG.getNode(ISD::TokenFactor, SDLoc(Chain), MVT::Other, ArgChains);
 }
 
 bool AArch64TargetLowering::DoesCalleeRestoreStack(CallingConv::ID CallCC,
                                                    bool TailCallOpt) const {
-  return CallCC == CallingConv::Fast && TailCallOpt;
+  return (CallCC == CallingConv::Fast && TailCallOpt) ||
+         CallCC == CallingConv::Tail || CallCC == CallingConv::SwiftTail;
 }
 
 /// LowerCall - Lower a call to a callseq_start + CALL + callseq_end chain,
 /// and add input and output parameter nodes.
 SDValue
 AArch64TargetLowering::LowerCall(CallLoweringInfo &CLI,
                                  SmallVectorImpl<SDValue> &InVals) const {
   SelectionDAG &DAG = CLI.DAG;
if (IsTailCall) {
     IsTailCall = isEligibleForTailCallOptimization(
         Callee, CallConv, IsVarArg, Outs, OutVals, Ins, DAG);
     if (!IsTailCall && CLI.CB && CLI.CB->isMustTailCall())
       report_fatal_error("failed to perform tail call elimination on a call "
                          "site marked musttail");
 
     // A sibling call is one where we're under the usual C ABI and not planning
     // to change that but can still do a tail call:
-    if (!TailCallOpt && IsTailCall)
+    if (!TailCallOpt && IsTailCall && CallConv != CallingConv::Tail &&
+        CallConv != CallingConv::SwiftTail)
       IsSibCall = true;
 
     if (IsTailCall)
       ++NumTailCalls;
   }
 
   // Analyze operands of the call, assigning locations to each operand.
   SmallVector<CCValAssign, 16> ArgLocs;
if (IsTailCall && !IsSibCall) {
     // popped size 16-byte aligned.
     NumBytes = alignTo(NumBytes, 16);
 
     // FPDiff will be negative if this tail call requires more space than we
     // would automatically have in our incoming argument space. Positive if we
     // can actually shrink the stack.
     FPDiff = NumReusableBytes - NumBytes;
 
+    // Update the required reserved area if this is the tail call requiring the
+    // most argument stack space.
+    if (FPDiff < 0 && FuncInfo->getTailCallReservedStack() < (unsigned)-FPDiff)
+      FuncInfo->setTailCallReservedStack(-FPDiff);
+
     // The stack pointer must be 16-byte aligned at all times it's used for a
     // memory operation, which in practice means at *all* times and in
     // particular across call boundaries. Therefore our own arguments started at
     // a 16-byte aligned SP and the delta applied for the tail call should
     // satisfy the same constraint.
     assert(FPDiff % 16 == 0 && "unaligned stack on tail call");
   }
 
   // Adjust the stack pointer for the new arguments...
   // These operations are automatically eliminated by the prolog/epilog pass
   if (!IsSibCall)
-    Chain = DAG.getCALLSEQ_START(Chain, NumBytes, 0, DL);
+    Chain = DAG.getCALLSEQ_START(Chain, IsTailCall ? 0 : NumBytes, 0, DL);
 
   SDValue StackPtr = DAG.getCopyFromReg(Chain, DL, AArch64::SP,
                                         getPointerTy(DAG.getDataLayout()));
 
   SmallVector<std::pair<unsigned, SDValue>, 8> RegsToPass;
   SmallSet<unsigned, 8> RegsUsed;
   SmallVector<SDValue, 8> MemOpChains;
   auto PtrVT = getPointerTy(DAG.getDataLayout());
if (auto *G = dyn_cast<GlobalAddressSDNode>(Callee)) {
     }
   }
 
   // We don't usually want to end the call-sequence here because we would tidy
   // the frame up *after* the call, however in the ABI-changing tail-call case
   // we've carefully laid out the parameters so that when sp is reset they'll be
   // in the correct location.
   if (IsTailCall && !IsSibCall) {
-    Chain = DAG.getCALLSEQ_END(Chain, DAG.getIntPtrConstant(NumBytes, DL, true),
+    Chain = DAG.getCALLSEQ_END(Chain, DAG.getIntPtrConstant(0, DL, true),
                                DAG.getIntPtrConstant(0, DL, true), InFlag, DL);
     InFlag = Chain.getValue(1);
   }
 
   std::vector<SDValue> Ops;
   Ops.push_back(Chain);
   Ops.push_back(Callee);
 

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

bool AArch64InstrInfo::getMemOpInfo(unsigned Opcode, TypeSize &Scale,
   case AArch64::LDRSBXui:
   case AArch64::STRBui:
   case AArch64::STRBBui:
     Scale = TypeSize::Fixed(1);
     Width = 1;
     MinOffset = 0;
     MaxOffset = 4095;
     break;
+  case AArch64::STPXpre:
+  case AArch64::LDPXpost:
+  case AArch64::STPDpre:
+  case AArch64::LDPDpost:
+    Scale = TypeSize::Fixed(8);
+    Width = 8;
+    MinOffset = -512;
+    MaxOffset = 504;
+    break;
+  case AArch64::STPQpre:
+  case AArch64::LDPQpost:
+    Scale = TypeSize::Fixed(16);
+    Width = 16;
+    MinOffset = -1024;
+    MaxOffset = 1008;
+    break;
+  case AArch64::STRXpre:
+  case AArch64::STRDpre:
+  case AArch64::LDRXpost:
+  case AArch64::LDRDpost:
+    Scale = TypeSize::Fixed(1);
+    Width = 8;
+    MinOffset = -256;
+    MaxOffset = 255;
+    break;
+  case AArch64::STRQpre:
+  case AArch64::LDRQpost:
+    Scale = TypeSize::Fixed(1);
+    Width = 16;
+    MinOffset = -256;
+    MaxOffset = 255;
+    break;
   case AArch64::ADDG:
     Scale = TypeSize::Fixed(16);
     Width = 0;
     MinOffset = 0;
     MaxOffset = 63;
     break;
   case AArch64::TAGPstack:
     Scale = TypeSize::Fixed(16);

llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h

class AArch64FunctionInfo final : public MachineFunctionInfo {
   /// callee.
   unsigned BytesInStackArgArea = 0;
 
   /// The number of bytes to restore to deallocate space for incoming
   /// arguments. Canonically 0 in the C calling convention, but non-zero when
   /// callee is expected to pop the args.
   unsigned ArgumentStackToRestore = 0;
 
+  /// Space just below incoming stack pointer reserved for arguments being
+  /// passed on the stack during a tail call. This will be the difference
+  /// between the largest tail call argument space needed in this function and
+  /// what's already available by reusing space of incoming arguments.
+  unsigned TailCallReservedStack = 0;
+
   /// HasStackFrame - True if this function has a stack frame. Set by
   /// determineCalleeSaves().
   bool HasStackFrame = false;
 
   /// Amount of stack frame size, not including callee-saved registers.
   uint64_t LocalStackSize = 0;
 
   /// The start and end frame indices for the SVE callee saves.
public:
   unsigned getBytesInStackArgArea() const { return BytesInStackArgArea; }
   void setBytesInStackArgArea(unsigned bytes) { BytesInStackArgArea = bytes; }
 
   unsigned getArgumentStackToRestore() const { return ArgumentStackToRestore; }
   void setArgumentStackToRestore(unsigned bytes) {
     ArgumentStackToRestore = bytes;
   }
 
+  unsigned getTailCallReservedStack() const { return TailCallReservedStack; }
+  void setTailCallReservedStack(unsigned bytes) {
+    TailCallReservedStack = bytes;
+  }
+
   bool hasCalculatedStackSizeSVE() const { return HasCalculatedStackSizeSVE; }
 
   void setStackSizeSVE(uint64_t S) {
     HasCalculatedStackSizeSVE = true;
     StackSizeSVE = S;
   }
 
   uint64_t getStackSizeSVE() const { return StackSizeSVE; }

llvm/lib/Target/AArch64/AArch64RegisterInfo.cpp

if (MF->getFunction().getCallingConv() == CallingConv::AArch64_VectorCall)
     return CSR_AArch64_AAVPCS_SaveList;
   if (MF->getFunction().getCallingConv() == CallingConv::AArch64_SVE_VectorCall)
     return CSR_AArch64_SVE_AAPCS_SaveList;
   if (MF->getSubtarget<AArch64Subtarget>().getTargetLowering()
           ->supportSwiftError() &&
       MF->getFunction().getAttributes().hasAttrSomewhere(
           Attribute::SwiftError))
     return CSR_AArch64_AAPCS_SwiftError_SaveList;
+  if (MF->getFunction().getCallingConv() == CallingConv::SwiftTail)
+    return CSR_AArch64_AAPCS_SwiftTail_SaveList;
   if (MF->getFunction().getCallingConv() == CallingConv::PreserveMost)
     return CSR_AArch64_RT_MostRegs_SaveList;
   if (MF->getFunction().getCallingConv() == CallingConv::Win64)
     // This is for OSes other than Windows; Windows is a separate case further
     // above.
     return CSR_AArch64_AAPCS_X18_SaveList;
   if (hasSVEArgsOrReturn(MF))
     return CSR_AArch64_SVE_AAPCS_SaveList;
if (MF->getFunction().getCallingConv() == CallingConv::CXX_FAST_TLS)
     return MF->getInfo<AArch64FunctionInfo>()->isSplitCSR()
                ? CSR_Darwin_AArch64_CXX_TLS_PE_SaveList
                : CSR_Darwin_AArch64_CXX_TLS_SaveList;
   if (MF->getSubtarget<AArch64Subtarget>().getTargetLowering()
           ->supportSwiftError() &&
       MF->getFunction().getAttributes().hasAttrSomewhere(
           Attribute::SwiftError))
     return CSR_Darwin_AArch64_AAPCS_SwiftError_SaveList;
+  if (MF->getFunction().getCallingConv() == CallingConv::SwiftTail)
+    return CSR_Darwin_AArch64_AAPCS_SwiftTail_SaveList;
   if (MF->getFunction().getCallingConv() == CallingConv::PreserveMost)
     return CSR_Darwin_AArch64_RT_MostRegs_SaveList;
   return CSR_Darwin_AArch64_AAPCS_SaveList;
 }
 
 const MCPhysReg *AArch64RegisterInfo::getCalleeSavedRegsViaCopy(
     const MachineFunction *MF) const {
   assert(MF && "Invalid MachineFunction pointer.");
AArch64RegisterInfo::getDarwinCallPreservedMask(const MachineFunction &MF,
   if (CC == CallingConv::CFGuard_Check)
     report_fatal_error(
         "Calling convention CFGuard_Check is unsupported on Darwin.");
   if (MF.getSubtarget<AArch64Subtarget>()
           .getTargetLowering()
           ->supportSwiftError() &&
       MF.getFunction().getAttributes().hasAttrSomewhere(Attribute::SwiftError))
     return CSR_Darwin_AArch64_AAPCS_SwiftError_RegMask;
+  if (CC == CallingConv::SwiftTail)
+    return CSR_Darwin_AArch64_AAPCS_SwiftTail_RegMask;
   if (CC == CallingConv::PreserveMost)
     return CSR_Darwin_AArch64_RT_MostRegs_RegMask;
   return CSR_Darwin_AArch64_AAPCS_RegMask;
 }
 
 const uint32_t *
 AArch64RegisterInfo::getCallPreservedMask(const MachineFunction &MF,
                                           CallingConv::ID CC) const {
return SCS ? CSR_AArch64_SVE_AAPCS_SCS_RegMask
                : CSR_AArch64_SVE_AAPCS_RegMask;
   if (CC == CallingConv::CFGuard_Check)
     return CSR_Win_AArch64_CFGuard_Check_RegMask;
   if (MF.getSubtarget<AArch64Subtarget>().getTargetLowering()
           ->supportSwiftError() &&
       MF.getFunction().getAttributes().hasAttrSomewhere(Attribute::SwiftError))
     return SCS ? CSR_AArch64_AAPCS_SwiftError_SCS_RegMask
                : CSR_AArch64_AAPCS_SwiftError_RegMask;
+  if (CC == CallingConv::SwiftTail) {
+    if (SCS)
+      report_fatal_error("ShadowCallStack attribute not supported with swifttail");
+    return CSR_AArch64_AAPCS_SwiftTail_RegMask;
+  }
   if (CC == CallingConv::PreserveMost)
     return SCS ? CSR_AArch64_RT_MostRegs_SCS_RegMask
                : CSR_AArch64_RT_MostRegs_RegMask;
   else
     return SCS ? CSR_AArch64_AAPCS_SCS_RegMask : CSR_AArch64_AAPCS_RegMask;
 }
 
 const uint32_t *AArch64RegisterInfo::getCustomEHPadPreservedMask(

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

struct OutgoingArgHandler : public CallLowering::OutgoingValueHandler {
   // Cache the SP register vreg if we need it more than once in this call site.
   Register SPReg;
 
   const AArch64Subtarget &Subtarget;
 };
 } // namespace
 
 static bool doesCalleeRestoreStack(CallingConv::ID CallConv, bool TailCallOpt) {
-  return CallConv == CallingConv::Fast && TailCallOpt;
+  return (CallConv == CallingConv::Fast && TailCallOpt) ||
+         CallConv == CallingConv::Tail || CallConv == CallingConv::SwiftTail;
 }
 
 bool AArch64CallLowering::lowerReturn(MachineIRBuilder &MIRBuilder,
                                       const Value *Val,
                                       ArrayRef<Register> VRegs,
                                       FunctionLoweringInfo &FLI,
                                       Register SwiftErrorVReg) const {
   auto MIB = MIRBuilder.buildInstrNoInsert(AArch64::RET_ReallyLR);
bool AArch64CallLowering::lowerFormalArguments(
 
   // Move back to the end of the basic block.
   MIRBuilder.setMBB(MBB);
 
   return true;
 }
 
 /// Return true if the calling convention is one that we can guarantee TCO for.
-static bool canGuaranteeTCO(CallingConv::ID CC) {
-  return CC == CallingConv::Fast;
+static bool canGuaranteeTCO(CallingConv::ID CC, bool GuaranteeTailCalls) {
+  return (CC == CallingConv::Fast && GuaranteeTailCalls) ||
+         CC == CallingConv::Tail || CC == CallingConv::SwiftTail;
 }
 
 /// Return true if we might ever do TCO for calls with this calling convention.
 static bool mayTailCallThisCC(CallingConv::ID CC) {
   switch (CC) {
   case CallingConv::C:
   case CallingConv::PreserveMost:
   case CallingConv::Swift:
+  case CallingConv::SwiftTail:
+  case CallingConv::Tail:
+  case CallingConv::Fast:
     return true;
   default:
-    return canGuaranteeTCO(CC);
+    return false;
   }
 }
 
 /// Returns a pair containing the fixed CCAssignFn and the vararg CCAssignFn for
 /// CC.
 static std::pair<CCAssignFn *, CCAssignFn *>
 getAssignFnsForCC(CallingConv::ID CC, const AArch64TargetLowering &TLI) {
   return {TLI.CCAssignFnForCall(CC, false), TLI.CCAssignFnForCall(CC, true)};
if (GV->hasExternalWeakLinkage() &&
          TT.isOSBinFormatMachO())) {
       LLVM_DEBUG(dbgs() << "... Cannot tail call externally-defined function "
                            "with weak linkage for this OS.\n");
       return false;
     }
   }
 
   // If we have -tailcallopt, then we're done.
-  if (MF.getTarget().Options.GuaranteedTailCallOpt)
-    return canGuaranteeTCO(CalleeCC) && CalleeCC == CallerF.getCallingConv();
+  if (canGuaranteeTCO(CalleeCC, MF.getTarget().Options.GuaranteedTailCallOpt))
+    return CalleeCC == CallerF.getCallingConv();
 
   // We don't have -tailcallopt, so we're allowed to change the ABI (sibcall).
   // Try to find cases where we can do that.
 
   // I want anyone implementing a new calling convention to think long and hard
   // about this assert.
   assert((!Info.IsVarArg || CalleeCC == CallingConv::C) &&
          "Unexpected variadic calling convention");
bool AArch64CallLowering::lowerTailCall(
     SmallVectorImpl<ArgInfo> &OutArgs) const {
   MachineFunction &MF = MIRBuilder.getMF();
   const Function &F = MF.getFunction();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   const AArch64TargetLowering &TLI = *getTLI<AArch64TargetLowering>();
   AArch64FunctionInfo *FuncInfo = MF.getInfo<AArch64FunctionInfo>();
 
   // True when we're tail calling, but without -tailcallopt.
-  bool IsSibCall = !MF.getTarget().Options.GuaranteedTailCallOpt;
+  bool IsSibCall = !MF.getTarget().Options.GuaranteedTailCallOpt &&
+                   Info.CallConv != CallingConv::Tail &&
+                   Info.CallConv != CallingConv::SwiftTail;
 
   // TODO: Right now, regbankselect doesn't know how to handle the rtcGPR64
   // register class. Until we can do that, we should fall back here.
   if (MF.getInfo<AArch64FunctionInfo>()->branchTargetEnforcement()) {
     LLVM_DEBUG(
         dbgs() << "Cannot lower indirect tail calls with BTI enabled yet.\n");
     return false;
   }
if (!IsSibCall) {
     // keep it 16-byte aligned.
     NumBytes = alignTo(OutInfo.getNextStackOffset(), 16);
 
     // FPDiff will be negative if this tail call requires more space than we
     // would automatically have in our incoming argument space. Positive if we
     // actually shrink the stack.
     FPDiff = NumReusableBytes - NumBytes;
 
+    // Update the required reserved area if this is the tail call requiring the
+    // most argument stack space.
+    if (FPDiff < 0 && FuncInfo->getTailCallReservedStack() < (unsigned)-FPDiff)
+      FuncInfo->setTailCallReservedStack(-FPDiff);
+
     // The stack pointer must be 16-byte aligned at all times it's used for a
     // memory operation, which in practice means at *all* times and in
     // particular across call boundaries. Therefore our own arguments started at
     // a 16-byte aligned SP and the delta applied for the tail call should
     // satisfy the same constraint.
     assert(FPDiff % 16 == 0 && "unaligned stack on tail call");
   }
 
for (const auto &F : Forwards) {
       MIB.addReg(ForwardedReg, RegState::Implicit);
     }
   }
 
   // If we have -tailcallopt, we need to adjust the stack. We'll do the call
   // sequence start and end here.
   if (!IsSibCall) {
     MIB->getOperand(1).setImm(FPDiff);
-    CallSeqStart.addImm(NumBytes).addImm(0);
+    CallSeqStart.addImm(0).addImm(0);
     // End the call sequence *before* emitting the call. Normally, we would
     // tidy the frame up after the call. However, here, we've laid out the
     // parameters so that when SP is reset, they will be in the correct
     // location.
-    MIRBuilder.buildInstr(AArch64::ADJCALLSTACKUP).addImm(NumBytes).addImm(0);
+    MIRBuilder.buildInstr(AArch64::ADJCALLSTACKUP).addImm(0).addImm(0);
   }
 
   // Now we can add the actual call instruction to the correct basic block.
   MIRBuilder.insertInstr(MIB);
 
   // If Callee is a reg, since it is used by a target specific instruction,
   // it must have a register class matching the constraint of that instruction.
   if (Info.Callee.isReg())

llvm/lib/Target/ARM/ARMISelLowering.cpp

ARMTargetLowering::getEffectiveCallingConv(CallingConv::ID CC,
   case CallingConv::ARM_APCS:
   case CallingConv::GHC:
   case CallingConv::CFGuard_Check:
     return CC;
   case CallingConv::PreserveMost:
     return CallingConv::PreserveMost;
   case CallingConv::ARM_AAPCS_VFP:
   case CallingConv::Swift:
+  case CallingConv::SwiftTail:
     return isVarArg ? CallingConv::ARM_AAPCS : CallingConv::ARM_AAPCS_VFP;
   case CallingConv::C:
     if (!Subtarget->isAAPCS_ABI())
       return CallingConv::ARM_APCS;
     else if (Subtarget->hasVFP2Base() && !Subtarget->isThumb1Only() &&
              getTargetMachine().Options.FloatABIType == FloatABI::Hard &&
              !isVarArg)
       return CallingConv::ARM_AAPCS_VFP;

llvm/lib/Target/X86/X86CallingConv.td

def RetCC_X86_64 : CallingConv<[
   CCIfCC<"CallingConv::HiPE", CCDelegateTo<RetCC_X86_64_HiPE>>,
 
   // Handle JavaScript calls.
   CCIfCC<"CallingConv::WebKit_JS", CCDelegateTo<RetCC_X86_64_WebKit_JS>>,
   CCIfCC<"CallingConv::AnyReg", CCDelegateTo<RetCC_X86_64_AnyReg>>,
 
   // Handle Swift calls.
   CCIfCC<"CallingConv::Swift", CCDelegateTo<RetCC_X86_64_Swift>>,
+  CCIfCC<"CallingConv::SwiftTail", CCDelegateTo<RetCC_X86_64_Swift>>,
 
   // Handle explicit CC selection
   CCIfCC<"CallingConv::Win64", CCDelegateTo<RetCC_X86_Win64_C>>,
   CCIfCC<"CallingConv::X86_64_SysV", CCDelegateTo<RetCC_X86_64_C>>,
 
   // Handle Vectorcall CC
   CCIfCC<"CallingConv::X86_VectorCall", CCDelegateTo<RetCC_X86_64_Vectorcall>>,
 
def CC_X86_64_C : CallingConv<[
 
   // A SwiftError is passed in R12.
   CCIfSwiftError<CCIfType<[i64], CCAssignToReg<[R12]>>>,
 
   // Pass SwiftAsync in an otherwise callee saved register so that calls to
   // normal functions don't need to save it somewhere.
   CCIfSwiftAsync<CCIfType<[i64], CCAssignToReg<[R14]>>>,
 
-  // For Swift Calling Convention, pass sret in %rax.
+  // For Swift Calling Conventions, pass sret in %rax.
   CCIfCC<"CallingConv::Swift",
     CCIfSRet<CCIfType<[i64], CCAssignToReg<[RAX]>>>>,
+  CCIfCC<"CallingConv::SwiftTail",
+    CCIfSRet<CCIfType<[i64], CCAssignToReg<[RAX]>>>>,
 
   // Pointers are always passed in full 64-bit registers.
   CCIfPtr<CCCustom<"CC_X86_64_Pointer">>,
 
   // The first 6 integer arguments are passed in integer registers.
   CCIfType<[i32], CCAssignToReg<[EDI, ESI, EDX, ECX, R8D, R9D]>>,
   CCIfType<[i64], CCAssignToReg<[RDI, RSI, RDX, RCX, R8 , R9 ]>>,
 

 
 def CC_X86_32_C : CallingConv<[
   // Promote i1/i8/i16/v1i1 arguments to i32.
   CCIfType<[i1, i8, i16, v1i1], CCPromoteToType<i32>>,
 
   // The 'nest' parameter, if any, is passed in ECX.
   CCIfNest<CCAssignToReg<[ECX]>>,
 
+  // On swifttailcc pass swiftself in ECX.
+  CCIfCC<"CallingConv::SwiftTail",
+         CCIfSwiftSelf<CCIfType<[i32], CCAssignToReg<[ECX]>>>>,
+
   // The first 3 integer arguments, if marked 'inreg' and if the call is not
   // a vararg call, are passed in integer registers.
   CCIfNotVarArg<CCIfInReg<CCIfType<[i32], CCAssignToReg<[EAX, EDX, ECX]>>>>,
 
   // Otherwise, same as everything else.
   CCDelegateTo<CC_X86_32_Common>
 ]>;
 

 //===----------------------------------------------------------------------===//
 
 def CSR_NoRegs : CalleeSavedRegs<(add)>;
 
 def CSR_32 : CalleeSavedRegs<(add ESI, EDI, EBX, EBP)>;
 def CSR_64 : CalleeSavedRegs<(add RBX, R12, R13, R14, R15, RBP)>;
 
 def CSR_64_SwiftError : CalleeSavedRegs<(sub CSR_64, R12)>;
+def CSR_64_SwiftTail : CalleeSavedRegs<(sub CSR_64, R13, R14)>;
 
 def CSR_32EHRet : CalleeSavedRegs<(add EAX, EDX, CSR_32)>;
 def CSR_64EHRet : CalleeSavedRegs<(add RAX, RDX, CSR_64)>;
 
 def CSR_Win64_NoSSE : CalleeSavedRegs<(add RBX, RBP, RDI, RSI, R12, R13, R14, R15)>;
 
 def CSR_Win64 : CalleeSavedRegs<(add CSR_Win64_NoSSE,
                                      (sequence "XMM%u", 6, 15))>;
 
 def CSR_Win64_SwiftError : CalleeSavedRegs<(sub CSR_Win64, R12)>;
+def CSR_Win64_SwiftTail : CalleeSavedRegs<(sub CSR_Win64, R13, R14)>;
 
 // The function used by Darwin to obtain the address of a thread-local variable
 // uses rdi to pass a single parameter and rax for the return value. All other
 // GPRs are preserved.
 def CSR_64_TLS_Darwin : CalleeSavedRegs<(add CSR_64, RCX, RDX, RSI,
                                              R8, R9, R10, R11)>;
 
 // CSRs that are handled by prologue, epilogue.

llvm/lib/Target/X86/X86FastISel.cpp

bool X86FastISel::X86SelectRet(const Instruction *I) {
 
   if (TLI.supportSplitCSR(FuncInfo.MF))
     return false;
 
   CallingConv::ID CC = F.getCallingConv();
   if (CC != CallingConv::C &&
       CC != CallingConv::Fast &&
       CC != CallingConv::Tail &&
+      CC != CallingConv::SwiftTail &&
       CC != CallingConv::X86_FastCall &&
       CC != CallingConv::X86_StdCall &&
       CC != CallingConv::X86_ThisCall &&
       CC != CallingConv::X86_64_SysV &&
       CC != CallingConv::Win64)
     return false;
 
   // Don't handle popping bytes if they don't fit the ret's immediate.
   if (!isUInt<16>(X86MFInfo->getBytesToPopOnReturn()))
     return false;
 
   // fastcc with -tailcallopt is intended to provide a guaranteed
   // tail call optimization. Fastisel doesn't know how to do that.
   if ((CC == CallingConv::Fast && TM.Options.GuaranteedTailCallOpt) ||
-      CC == CallingConv::Tail)
+      CC == CallingConv::Tail || CC == CallingConv::SwiftTail)
     return false;
 
   // Let SDISel handle vararg functions.
   if (F.isVarArg())
     return false;
 
   // Build a list of return value registers.
   SmallVector<unsigned, 4> RetRegs;
bool X86FastISel::X86SelectRet(const Instruction *I) {
 
   // Swift calling convention does not require we copy the sret argument
   // into %rax/%eax for the return, and SRetReturnReg is not set for Swift.
 
   // All x86 ABIs require that for returning structs by value we copy
   // the sret argument into %rax/%eax (depending on ABI) for the return.
   // We saved the argument into a virtual register in the entry block,
   // so now we copy the value out and into %rax/%eax.
-  if (F.hasStructRetAttr() && CC != CallingConv::Swift) {
+  if (F.hasStructRetAttr() && CC != CallingConv::Swift &&
+      CC != CallingConv::SwiftTail) {
     Register Reg = X86MFInfo->getSRetReturnReg();
     assert(Reg &&
            "SRetReturnReg should have been set in LowerFormalArguments()!");
     unsigned RetReg = Subtarget->isTarget64BitLP64() ? X86::RAX : X86::EAX;
     BuildMI(*FuncInfo.MBB, FuncInfo.InsertPt, DbgLoc,
             TII.get(TargetOpcode::COPY), RetReg).addReg(Reg);
     RetRegs.push_back(RetReg);
   }

 static unsigned computeBytesPoppedByCalleeForSRet(const X86Subtarget *Subtarget,
                                                   CallingConv::ID CC,
                                                   const CallBase *CB) {
   if (Subtarget->is64Bit())
     return 0;
   if (Subtarget->getTargetTriple().isOSMSVCRT())
     return 0;
   if (CC == CallingConv::Fast || CC == CallingConv::GHC ||
-      CC == CallingConv::HiPE || CC == CallingConv::Tail)
+      CC == CallingConv::HiPE || CC == CallingConv::Tail ||
+      CC == CallingConv::SwiftTail)
     return 0;
 
   if (CB)
     if (CB->arg_empty() || !CB->paramHasAttr(0, Attribute::StructRet) ||
         CB->paramHasAttr(0, Attribute::InReg) || Subtarget->isTargetMCU())
       return 0;
 
   return 4;
bool X86FastISel::fastLowerCall(CallLoweringInfo &CLI) {
   // Handle only C, fastcc, and webkit_js calling conventions for now.
   switch (CC) {
   default: return false;
   case CallingConv::C:
   case CallingConv::Fast:
   case CallingConv::Tail:
   case CallingConv::WebKit_JS:
   case CallingConv::Swift:
+  case CallingConv::SwiftTail:
   case CallingConv::X86_FastCall:
   case CallingConv::X86_StdCall:
   case CallingConv::X86_ThisCall:
   case CallingConv::Win64:
   case CallingConv::X86_64_SysV:
   case CallingConv::CFGuard_Check:
     break;
   }
 
   // Allow SelectionDAG isel to handle tail calls.
   if (IsTailCall)
     return false;
 
   // fastcc with -tailcallopt is intended to provide a guaranteed
   // tail call optimization. Fastisel doesn't know how to do that.
   if ((CC == CallingConv::Fast && TM.Options.GuaranteedTailCallOpt) ||
-      CC == CallingConv::Tail)
+      CC == CallingConv::Tail || CC == CallingConv::SwiftTail)
     return false;
 
   // Don't know how to handle Win64 varargs yet.  Nothing special needed for
   // x86-32. Special handling for x86-64 is implemented.
   if (IsVarArg && IsWin64)
     return false;
 
   // Don't know about inalloca yet.

llvm/lib/Target/X86/X86ISelLowering.cpp

return DAG.getMemcpy(
       /*isVolatile*/ false, /*AlwaysInline=*/true,
       /*isTailCall*/ false, MachinePointerInfo(), MachinePointerInfo());
 }
 
 /// Return true if the calling convention is one that we can guarantee TCO for.
 static bool canGuaranteeTCO(CallingConv::ID CC) {
   return (CC == CallingConv::Fast || CC == CallingConv::GHC ||
           CC == CallingConv::X86_RegCall || CC == CallingConv::HiPE ||
-          CC == CallingConv::HHVM || CC == CallingConv::Tail);
+          CC == CallingConv::HHVM || CC == CallingConv::Tail ||
+          CC == CallingConv::SwiftTail);
 }
 
 /// Return true if we might ever do TCO for calls with this calling convention.
 static bool mayTailCallThisCC(CallingConv::ID CC) {
   switch (CC) {
   // C calling conventions:
   case CallingConv::C:
   case CallingConv::Win64:
static bool mayTailCallThisCC(CallingConv::ID CC) {
   default:
     return canGuaranteeTCO(CC);
   }
 }
 
 /// Return true if the function is being made into a tailcall target by
 /// changing its ABI.
 static bool shouldGuaranteeTCO(CallingConv::ID CC, bool GuaranteedTailCallOpt) {
-  return (GuaranteedTailCallOpt && canGuaranteeTCO(CC)) || CC == CallingConv::Tail;
+  return (GuaranteedTailCallOpt && canGuaranteeTCO(CC)) ||
+         CC == CallingConv::Tail || CC == CallingConv::SwiftTail;
 }
 
 bool X86TargetLowering::mayBeEmittedAsTailCall(const CallInst *CI) const {
   if (!CI->isTailCall())
     return false;
 
   CallingConv::ID CalleeCC = CI->getCallingConv();
   if (!mayTailCallThisCC(CalleeCC))
if (Ins[I].Flags.isSwiftAsync()) {
                                   DAG.getFrameIndex(FI, MVT::i32),
                                   MachinePointerInfo::getFixedStack(MF, FI));
         Chain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, St, Chain);
       }
     }
 
     // Swift calling convention does not require we copy the sret argument
     // into %rax/%eax for the return. We don't set SRetReturnReg for Swift.
-    if (CallConv == CallingConv::Swift)
+    if (CallConv == CallingConv::Swift || CallConv == CallingConv::SwiftTail)
       continue;
 
     // All x86 ABIs require that for returning structs by value we copy the
     // sret argument into %rax/%eax (depending on ABI) for the return. Save
     // the argument into a virtual register so that we can access it from the
     // return points.
     if (Ins[I].Flags.isSRet()) {
       Register Reg = FuncInfo->getSRetReturnReg();
X86TargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI,
   const auto *CB                        = CLI.CB;
 
   MachineFunction &MF = DAG.getMachineFunction();
   bool Is64Bit        = Subtarget.is64Bit();
   bool IsWin64        = Subtarget.isCallingConvWin64(CallConv);
   StructReturnType SR = callIsStructReturn(Outs, Subtarget.isTargetMCU());
   bool IsSibcall      = false;
   bool IsGuaranteeTCO = MF.getTarget().Options.GuaranteedTailCallOpt ||
-      CallConv == CallingConv::Tail;
+      CallConv == CallingConv::Tail || CallConv == CallingConv::SwiftTail;
   X86MachineFunctionInfo *X86Info = MF.getInfo<X86MachineFunctionInfo>();
   bool HasNCSR = (CB && isa<CallInst>(CB) &&
                   CB->hasFnAttr("no_caller_saved_registers"));
   bool HasNoCfCheck = (CB && CB->doesNoCfCheck());
   bool IsIndirectCall = (CB && isa<CallInst>(CB) && CB->isIndirectCall());
   const Module *M = MF.getMMI().getModule();
   Metadata *IsCFProtectionSupported = M->getModuleFlag("cf-protection-branch");
 
bool X86TargetLowering::IsEligibleForTailCallOptimization(
   if (CallerF.getReturnType()->isX86_FP80Ty() && !RetTy->isX86_FP80Ty())
     return false;
 
   CallingConv::ID CallerCC = CallerF.getCallingConv();
   bool CCMatch = CallerCC == CalleeCC;
   bool IsCalleeWin64 = Subtarget.isCallingConvWin64(CalleeCC);
   bool IsCallerWin64 = Subtarget.isCallingConvWin64(CallerCC);
   bool IsGuaranteeTCO = DAG.getTarget().Options.GuaranteedTailCallOpt ||
-      CalleeCC == CallingConv::Tail;
+      CalleeCC == CallingConv::Tail || CalleeCC == CallingConv::SwiftTail;
 
   // Win64 functions have extra shadow space for argument homing. Don't do the
   // sibcall if the caller and callee have mismatched expectations for this
   // space.
   if (IsCalleeWin64 != IsCallerWin64)
     return false;
 
   if (IsGuaranteeTCO) {
case CallingConv::X86_StdCall: {
         }
       }
       break;
     }
     case CallingConv::X86_FastCall:
     case CallingConv::X86_ThisCall:
     case CallingConv::Fast:
     case CallingConv::Tail:
+    case CallingConv::SwiftTail:
       // Pass 'nest' parameter in EAX.
       // Must be kept in sync with X86CallingConv.td
       NestReg = X86::EAX;
       break;
     }
 
     SDValue OutChains[4];
     SDValue Addr, Disp;

llvm/lib/Target/X86/X86RegisterInfo.cpp

X86RegisterInfo::getCalleeSavedRegs(const MachineFunction *MF) const {
   case CallingConv::Cold:
     if (Is64Bit)
       return CSR_64_MostRegs_SaveList;
     break;
   case CallingConv::Win64:
     if (!HasSSE)
       return CSR_Win64_NoSSE_SaveList;
     return CSR_Win64_SaveList;
+  case CallingConv::SwiftTail:
+    if (!Is64Bit)
+      return CSR_32_SaveList;
+    return IsWin64 ? CSR_Win64_SwiftTail_SaveList : CSR_64_SwiftTail_SaveList;
   case CallingConv::X86_64_SysV:
     if (CallsEHReturn)
       return CSR_64EHRet_SaveList;
     return CSR_64_SaveList;
   case CallingConv::X86_INTR:
     if (Is64Bit) {
       if (HasAVX512)
         return CSR_64_AllRegs_AVX512_SaveList;
case CallingConv::CFGuard_Check:
     return (HasSSE ? CSR_Win32_CFGuard_Check_RegMask
                    : CSR_Win32_CFGuard_Check_NoSSE_RegMask);
   case CallingConv::Cold:
     if (Is64Bit)
       return CSR_64_MostRegs_RegMask;
     break;
   case CallingConv::Win64:
     return CSR_Win64_RegMask;
+  case CallingConv::SwiftTail:
+    if (!Is64Bit)
+      return CSR_32_RegMask;
+    return IsWin64 ? CSR_Win64_SwiftTail_RegMask : CSR_64_SwiftTail_RegMask;
   case CallingConv::X86_64_SysV:
     return CSR_64_RegMask;
   case CallingConv::X86_INTR:
     if (Is64Bit) {
       if (HasAVX512)
         return CSR_64_AllRegs_AVX512_RegMask;
       if (HasAVX)
         return CSR_64_AllRegs_AVX_RegMask;
X86RegisterInfo::getCallPreservedMask(const MachineFunction &MF,
   // Unlike getCalleeSavedRegs(), we don't have MMI so we can't check
   // callsEHReturn().
   if (Is64Bit) {
     const Function &F = MF.getFunction();
     bool IsSwiftCC = Subtarget.getTargetLowering()->supportSwiftError() &&
                      F.getAttributes().hasAttrSomewhere(Attribute::SwiftError);
     if (IsSwiftCC)
       return IsWin64 ? CSR_Win64_SwiftError_RegMask : CSR_64_SwiftError_RegMask;
+
     return IsWin64 ? CSR_Win64_RegMask : CSR_64_RegMask;
   }
 
   return CSR_32_RegMask;
 }
 
 const uint32_t*
 X86RegisterInfo::getNoPreservedMask() const {

llvm/lib/Target/X86/X86Subtarget.h

public:
 
   bool isCallingConvWin64(CallingConv::ID CC) const {
     switch (CC) {
     // On Win64, all these conventions just use the default convention.
     case CallingConv::C:
     case CallingConv::Fast:
     case CallingConv::Tail:
     case CallingConv::Swift:
+    case CallingConv::SwiftTail:
     case CallingConv::X86_FastCall:
     case CallingConv::X86_StdCall:
     case CallingConv::X86_ThisCall:
     case CallingConv::X86_VectorCall:
     case CallingConv::Intel_OCL_BI:
       return isTargetWin64();
     // This convention allows using the Win64 convention on other targets.
     case CallingConv::Win64:

llvm/test/Bitcode/compatibility.ll

 declare webkit_jscc void @f.webkit_jscc()
 ; CHECK: declare webkit_jscc void @f.webkit_jscc()
 declare anyregcc void @f.anyregcc()
 ; CHECK: declare anyregcc void @f.anyregcc()
 declare preserve_mostcc void @f.preserve_mostcc()
 ; CHECK: declare preserve_mostcc void @f.preserve_mostcc()
 declare preserve_allcc void @f.preserve_allcc()
 ; CHECK: declare preserve_allcc void @f.preserve_allcc()
+declare swifttailcc void @f.swifttailcc()
+; CHECK: declare swifttailcc void @f.swifttailcc()
 declare cc64 void @f.cc64()
 ; CHECK: declare x86_stdcallcc void @f.cc64()
 declare x86_stdcallcc void @f.x86_stdcallcc()
 ; CHECK: declare x86_stdcallcc void @f.x86_stdcallcc()
 declare cc65 void @f.cc65()
 ; CHECK: declare x86_fastcallcc void @f.cc65()
 declare x86_fastcallcc void @f.x86_fastcallcc()
 ; CHECK: declare x86_fastcallcc void @f.x86_fastcallcc()

llvm/test/CodeGen/AArch64/swift-async.ll

 ; RUN: llc -mtriple=arm64-apple-ios %s -o - | FileCheck %s --check-prefixes=CHECK-NOAUTH,CHECK
 ; RUN: llc -mtriple=arm64-apple-ios -mcpu=apple-a13 %s -o - | FileCheck %s --check-prefixes=CHECK-NOAUTH,CHECK
 ; RUN: llc -mtriple=arm64e-apple-ios %s -o - | FileCheck %s --check-prefixes=CHECK-AUTH,CHECK
 
 ; Important details in prologue:
 ;   * x22 is stored just below x29
 ;   * Enough stack space is allocated for everything
-define void @simple(i8* swiftasync %ctx) "frame-pointer"="all" {
+define swifttailcc void @simple(i8* swiftasync %ctx) "frame-pointer"="all" {
 ; CHECK-LABEL: simple:
 ; CHECK: orr x29, x29, #0x100000000000000
 ; CHECK: sub sp, sp, #32
 ; CHECK: stp x29, x30, [sp, #16]
 
 ; CHECK-NOAUTH: str x22, [sp, #8]
 ; CHECK-AUTH: add x16, sp, #8
 ; CHECK-AUTH: movk x16, #49946, lsl #48

 
 ; CHECK: ldp x29, x30, [sp, #16]
 ; CHECK: and x29, x29, #0xefffffffffffffff
 ; CHECK: add sp, sp, #32
 
   ret void
 }
 
-define void @more_csrs(i8* swiftasync %ctx) "frame-pointer"="all" {
+define swifttailcc void @more_csrs(i8* swiftasync %ctx) "frame-pointer"="all" {
 ; CHECK-LABEL: more_csrs:
 ; CHECK: orr x29, x29, #0x100000000000000
-; CHECK: sub sp, sp, #48
-; CHECK: stp x24, x23, [sp, #8]
-; CHECK: stp x29, x30, [sp, #32]
+; CHECK: str x23, [sp, #-32]!
+; CHECK: stp x29, x30, [sp, #16]
 
-; CHECK-NOAUTH: str x22, [sp, #24]
-; CHECK-AUTH: add x16, sp, #24
+; CHECK-NOAUTH: str x22, [sp, #8]
+; CHECK-AUTH: add x16, sp, #8
 ; CHECK-AUTH: movk x16, #49946, lsl #48
 ; CHECK-AUTH: mov x17, x22
 ; CHECK-AUTH: pacdb x17, x16
-; CHECK-AUTH: str x17, [sp, #24]
+; CHECK-AUTH: str x17, [sp, #8]
 
-; CHECK: add x29, sp, #32
+; CHECK: add x29, sp, #16
 ; CHECK: .cfi_def_cfa w29, 16
 ; CHECK: .cfi_offset w30, -8
 ; CHECK: .cfi_offset w29, -16
 ; CHECK: .cfi_offset w23, -32
 
 ; [...]
 
-; CHECK: ldp x29, x30, [sp, #32]
-; CHECK: ldp x24, x23, [sp, #8]
+; CHECK: ldp x29, x30, [sp, #16]
+; CHECK: ldr x23, [sp], #32
 ; CHECK: and x29, x29, #0xefffffffffffffff
-; CHECK: add sp, sp, #48
   call void asm sideeffect "", "~{x23}"()
   ret void
 }
 
-define void @locals(i8* swiftasync %ctx) "frame-pointer"="all" {
+define swifttailcc void @locals(i8* swiftasync %ctx) "frame-pointer"="all" {
 ; CHECK-LABEL: locals:
 ; CHECK: orr x29, x29, #0x100000000000000
 ; CHECK: sub sp, sp, #64
 ; CHECK: stp x29, x30, [sp, #48]
 
 ; CHECK-NOAUTH: str x22, [sp, #40]
 ; CHECK-AUTH: add x16, sp, #40
 ; CHECK-AUTH: movk x16, #49946, lsl #48

 ; CHECK: ldp x29, x30, [sp, #48]
 ; CHECK: and x29, x29, #0xefffffffffffffff
 ; CHECK: add sp, sp, #64
   %var = alloca i32, i32 10
   call void @bar(i32* %var)
   ret void
 }
 
-define void @use_input_context(i8* swiftasync %ctx, i8** %ptr) "frame-pointer"="all" {
+define swifttailcc void @use_input_context(i8* swiftasync %ctx, i8** %ptr) "frame-pointer"="all" {
 ; CHECK-LABEL: use_input_context:
 
 ; CHECK-NOAUTH: str x22, [sp
 ; CHECK-AUTH: mov x17, x22
 
 ; CHECK-NOT: x22
 ; CHECK: str x22, [x0]
 
   store i8* %ctx, i8** %ptr
   ret void
 }
 
-define i8** @context_in_func() "frame-pointer"="non-leaf" {
+define swifttailcc i8** @context_in_func() "frame-pointer"="non-leaf" {
 ; CHECK-LABEL: context_in_func:
 
 ; CHECK-NOAUTH: str xzr, [sp, #8]
 ; CHECK-AUTH: add x16, sp, #8
 ; CHECK-AUTH: movk x16, #49946, lsl #48
 ; CHECK-AUTH: mov x17, xzr
 ; CHECK-AUTH: pacdb x17, x16
 ; CHECK-AUTH: str x17, [sp, #8]
 
   %ptr = call i8** @llvm.swift.async.context.addr()
   ret i8** %ptr
 }
 
-define void @write_frame_context(i8* swiftasync %ctx, i8* %newctx) "frame-pointer"="non-leaf" {
+define swifttailcc void @write_frame_context(i8* swiftasync %ctx, i8* %newctx) "frame-pointer"="non-leaf" {
 ; CHECK-LABEL: write_frame_context:
 ; CHECK: sub x[[ADDR:[0-9]+]], x29, #8
 ; CHECK: str x0, [x[[ADDR]]]
   %ptr = call i8** @llvm.swift.async.context.addr()
   store i8* %newctx, i8** %ptr
   ret void
 }
 
-define void @simple_fp_elim(i8* swiftasync %ctx) "frame-pointer"="non-leaf" {
+define swifttailcc void @simple_fp_elim(i8* swiftasync %ctx) "frame-pointer"="non-leaf" {
 ; CHECK-LABEL: simple_fp_elim:
 ; CHECK-NOT: orr x29, x29, #0x100000000000000
 
   ret void
 }
 
-define void @large_frame(i8* swiftasync %ctx) "frame-pointer"="all" {
+define swifttailcc void @large_frame(i8* swiftasync %ctx) "frame-pointer"="all" {
 ; CHECK-LABEL: large_frame:
-; CHECK: sub sp, sp, #48
-; CHECK: stp x28, x27, [sp, #8]
-; CHECK: stp x29, x30, [sp, #32]
-; CHECK-NOAUTH: str x22, [sp, #24]
-; CHECK: add x29, sp, #32
+; CHECK: str x28, [sp, #-32]!
+; CHECK: stp x29, x30, [sp, #16]
+; CHECK-NOAUTH: str x22, [sp, #8]
+; CHECK: add x29, sp, #16
 ; CHECK: sub sp, sp, #1024
 ; [...]
 ; CHECK: add sp, sp, #1024
-; CHECK: ldp x29, x30, [sp, #32]
-; CHECK: ldp x28, x27, [sp, #8]
+; CHECK: ldp x29, x30, [sp, #16]
+; CHECK: ldr x28, [sp], #32
 ; CHECK: ret
   %var = alloca i8, i32 1024
   ret void
 }
 
-declare void @bar(i32*)
+; Important point is that there is just one 8-byte gap in the CSR region (right
+; now just above d8) to realign the stack.
+define swifttailcc void @two_unpaired_csrs(i8* swiftasync) "frame-pointer"="all" {
+; CHECK-LABEL: two_unpaired_csrs:
+; CHECK: str d8, [sp, #-48]!
+; CHECK: str x19, [sp, #16]
+; CHECK: stp x29, x30, [sp, #32]
+; CHECK-NOAUTH: str x22, [sp, #24]
+; CHECK: add x29, sp, #32
+
+; CHECK: .cfi_def_cfa w29, 16
+; CHECK: .cfi_offset w30, -8
+; CHECK: .cfi_offset w29, -16
+; CHECK: .cfi_offset w19, -32
+; CHECK: .cfi_offset b8, -48
+
+  call void asm "","~{x19},~{d8}"()
+  call swifttailcc void @bar(i32* undef)
+  ret void
+}
+declare swifttailcc void @bar(i32*)
 declare i8** @llvm.swift.async.context.addr()

llvm/test/CodeGen/AArch64/swifttail-async.ll

+; RUN: llc -mtriple=arm64-apple-ios %s -o - | FileCheck %s
+
+
+declare swifttailcc void @swifttail_callee()
+define swifttailcc void @swifttail() {
+; CHECK-LABEL: swifttail:
+; CHECK-NOT: ld{{.*}}x22
+  call void asm "","~{x22}"()
+  tail call swifttailcc void @swifttail_callee()
+  ret void
+}
+
+define swifttailcc void @no_preserve_swiftself() {
+; CHECK-LABEL: no_preserve_swiftself:
+; CHECK-NOT: ld{{.*}}x20
+  call void asm "","~{x20}"()
+  ret void
+}

llvm/test/CodeGen/AArch64/swifttail-call.ll

+; RUN: llc -verify-machineinstrs < %s -mtriple=aarch64-none-linux-gnu | FileCheck %s --check-prefixes=SDAG,COMMON
+; RUN: llc -global-isel -global-isel-abort=1 -verify-machineinstrs < %s -mtriple=aarch64-none-linux-gnu  | FileCheck %s --check-prefixes=GISEL,COMMON
+
+declare swifttailcc void @callee_stack0()
+declare swifttailcc void @callee_stack8([8 x i64], i64)
+declare swifttailcc void @callee_stack16([8 x i64], i64, i64)
+declare extern_weak swifttailcc void @callee_weak()
+
+define swifttailcc void @caller_to0_from0() nounwind {
+; COMMON-LABEL: caller_to0_from0:
+; COMMON-NEXT: // %bb.
+
+  tail call swifttailcc void @callee_stack0()
+  ret void
+
+; COMMON-NEXT: b callee_stack0
+}
+
+define swifttailcc void @caller_to0_from8([8 x i64], i64) {
+; COMMON-LABEL: caller_to0_from8:
+
+  tail call swifttailcc void @callee_stack0()
+  ret void
+
+; COMMON: add sp, sp, #16
+; COMMON-NEXT: b callee_stack0
+}
+
+define swifttailcc void @caller_to8_from0() {
+; COMMON-LABEL: caller_to8_from0:
+
+; Key point is that the "42" should go #16 below incoming stack
+; pointer (we didn't have arg space to reuse).
+  tail call swifttailcc void @callee_stack8([8 x i64] undef, i64 42)
+  ret void
+
+; COMMON: str {{x[0-9]+}}, [sp, #-16]!
+; COMMON-NEXT: b callee_stack8
+}
+
+define swifttailcc void @caller_to8_from8([8 x i64], i64 %a) {
+; COMMON-LABEL: caller_to8_from8:
+; COMMON-NOT: sub sp,
+
+; Key point is that the "%a" should go where at SP on entry.
+  tail call swifttailcc void @callee_stack8([8 x i64] undef, i64 42)
+  ret void
+
+; COMMON: str {{x[0-9]+}}, [sp]
+; COMMON-NEXT: b callee_stack8
+}
+
+define swifttailcc void @caller_to16_from8([8 x i64], i64 %a) {
+; COMMON-LABEL: caller_to16_from8:
+; COMMON-NOT: sub sp,
+
+; Important point is that the call reuses the "dead" argument space
+; above %a on the stack. If it tries to go below incoming-SP then the
+; callee will not deallocate the space, even in swifttailcc.
+  tail call swifttailcc void @callee_stack16([8 x i64] undef, i64 42, i64 2)
+
+; COMMON: stp {{x[0-9]+}}, {{x[0-9]+}}, [sp]
+; COMMON-NEXT: b callee_stack16
+  ret void
+}
+
+
+define swifttailcc void @caller_to8_from24([8 x i64], i64 %a, i64 %b, i64 %c) {
+; COMMON-LABEL: caller_to8_from24:
+; COMMON-NOT: sub sp,
+
+; Key point is that the "%a" should go where at #16 above SP on entry.
+  tail call swifttailcc void @callee_stack8([8 x i64] undef, i64 42)
+  ret void
+
+; COMMON: str {{x[0-9]+}}, [sp, #16]!
+; COMMON-NEXT: b callee_stack8
+}
+
+
+define swifttailcc void @caller_to16_from16([8 x i64], i64 %a, i64 %b) {
+; COMMON-LABEL: caller_to16_from16:
+; COMMON-NOT: sub sp,
+
+; Here we want to make sure that both loads happen before the stores:
+; otherwise either %a or %b will be wrongly clobbered.
+  tail call swifttailcc void @callee_stack16([8 x i64] undef, i64 %b, i64 %a)
+  ret void
+
+; COMMON: ldp {{x[0-9]+}}, {{x[0-9]+}}, [sp]
+; COMMON: stp {{x[0-9]+}}, {{x[0-9]+}}, [sp]
+; COMMON-NEXT: b callee_stack16
+}
+
+define swifttailcc void @disable_tail_calls() nounwind "disable-tail-calls"="true" {
+; COMMON-LABEL: disable_tail_calls:
+; COMMON-NEXT: // %bb.
+
+  tail call swifttailcc void @callee_stack0()
+  ret void
+
+; COMMON: bl callee_stack0
+; COMMON: ret
+}
+
+; Weakly-referenced extern functions cannot be tail-called, as AAELF does
+; not define the behaviour of branch instructions to undefined weak symbols.
+define swifttailcc void @caller_weak() {
+; COMMON-LABEL: caller_weak:
+; COMMON: bl callee_weak
+  tail call void @callee_weak()
+  ret void
+}
+
+declare { [2 x float] } @get_vec2()
+
+define { [3 x float] } @test_add_elem() {
+; SDAG-LABEL: test_add_elem:
+; SDAG: bl get_vec2
+; SDAG: fmov s2, #1.0
+; SDAG: ret
+; GISEL-LABEL: test_add_elem:
+; GISEL: str  x30, [sp, #-16]!
+; GISEL: bl get_vec2
+; GISEL: fmov s2, #1.0
+; GISEL: ldr  x30, [sp], #16
+; GISEL: ret
+
+  %call = tail call { [2 x float] } @get_vec2()
+  %arr = extractvalue { [2 x float] } %call, 0
+  %arr.0 = extractvalue [2 x float] %arr, 0
+  %arr.1 = extractvalue [2 x float] %arr, 1
+
+  %res.0 = insertvalue { [3 x float] } undef, float %arr.0, 0, 0
+  %res.01 = insertvalue { [3 x float] } %res.0, float %arr.1, 0, 1
+  %res.012 = insertvalue { [3 x float] } %res.01, float 1.000000e+00, 0, 2
+  ret { [3 x float] } %res.012
+}
+
+declare double @get_double()
+define { double, [2 x double] } @test_mismatched_insert() {
+; COMMON-LABEL: test_mismatched_insert:
+; COMMON: bl get_double
+; COMMON: bl get_double
+; COMMON: bl get_double
+; COMMON: ret
+
+  %val0 = call double @get_double()
+  %val1 = call double @get_double()
+  %val2 = tail call double @get_double()
+
+  %res.0 = insertvalue { double, [2 x double] } undef, double %val0, 0
+  %res.01 = insertvalue { double, [2 x double] } %res.0, double %val1, 1, 0
+  %res.012 = insertvalue { double, [2 x double] } %res.01, double %val2, 1, 1
+
+  ret { double, [2 x double] } %res.012
+}
+
+define void @fromC_totail() {
+; COMMON-LABEL: fromC_totail:
+; COMMON: sub sp, sp, #48
+
+; COMMON-NOT: sub sp,
+; COMMON: mov w[[TMP:[0-9]+]], #42
+; COMMON: str x[[TMP]], [sp]
+; COMMON: bl callee_stack8
+  ; We must reset the stack to where it was before the call by undoing its extra stack pop.
+; COMMON: str x[[TMP]], [sp, #-16]!
+; COMMON: bl callee_stack8
+; COMMON: sub sp, sp, #16
+
+  call swifttailcc void @callee_stack8([8 x i64] undef, i64 42)
+  call swifttailcc void @callee_stack8([8 x i64] undef, i64 42)
+  ret void
+}
+
+define void @fromC_totail_noreservedframe(i32 %len) {
+; COMMON-LABEL: fromC_totail_noreservedframe:
+; COMMON: stp x29, x30, [sp, #-48]!
+
+; COMMON: mov w[[TMP:[0-9]+]], #42
+  ; Note stack is subtracted here to allocate space for arg
+; COMMON: str x[[TMP]], [sp, #-16]!
+; COMMON: bl callee_stack8
+  ; And here.
+; COMMON: str x[[TMP]], [sp, #-16]!
+; COMMON: bl callee_stack8
+  ; But not restored here because callee_stack8 did that for us.
+; COMMON-NOT: sub sp,
+
+  ; Variable sized allocation prevents reserving frame at start of function so each call must allocate any stack space it needs.
+  %var = alloca i32, i32 %len
+
+  call swifttailcc void @callee_stack8([8 x i64] undef, i64 42)
+  call swifttailcc void @callee_stack8([8 x i64] undef, i64 42)
+  ret void
+}
+
+declare void @Ccallee_stack8([8 x i64], i64)
+
+define swifttailcc void @fromtail_toC() {
+; COMMON-LABEL: fromtail_toC:
+; COMMON: sub sp, sp, #32
+
+; COMMON-NOT: sub sp,
+; COMMON: mov w[[TMP:[0-9]+]], #42
+; COMMON: str x[[TMP]], [sp]
+; COMMON: bl Ccallee_stack8
+  ; C callees will return with the stack exactly where we left it, so we mustn't try to fix anything.
+; COMMON-NOT: add sp,
+; COMMON-NOT: sub sp,
+; COMMON: str x[[TMP]], [sp]{{$}}
+; COMMON: bl Ccallee_stack8
+; COMMON-NOT: sub sp,
+
+
+  call void @Ccallee_stack8([8 x i64] undef, i64 42)
+  call void @Ccallee_stack8([8 x i64] undef, i64 42)
+  ret void
+}
+
+declare swifttailcc i8* @SwiftSelf(i8 * swiftasync %context, i8* swiftself %closure)
+define swiftcc i8* @CallSwiftSelf(i8* swiftself %closure, i8* %context) {
+; CHECK-LABEL: CallSwiftSelf:
+; CHECK: stp x20
+  ;call void asm "","~{r13}"() ; We get a push r13 but why not with the call
+  ; below?
+  %res = call swifttailcc i8* @SwiftSelf(i8 * swiftasync %context, i8* swiftself %closure)
+  ret i8* %res
+}

llvm/test/CodeGen/AArch64/tail-call.ll

; COMMON-LABEL: caller_to0_from8:
   ret void
 
 ; COMMON: add sp, sp, #16
 ; COMMON-NEXT: b callee_stack0
 }
 
 define fastcc void @caller_to8_from0() {
 ; COMMON-LABEL: caller_to8_from0:
-; COMMON: sub sp, sp, #32
 
 ; Key point is that the "42" should go #16 below incoming stack
 ; pointer (we didn't have arg space to reuse).
   tail call fastcc void @callee_stack8([8 x i64] undef, i64 42)
   ret void
 
-; COMMON: str {{x[0-9]+}}, [sp, #16]!
+; COMMON: str {{x[0-9]+}}, [sp, #-16]!
 ; COMMON-NEXT: b callee_stack8
 }
 
 define fastcc void @caller_to8_from8([8 x i64], i64 %a) {
 ; COMMON-LABEL: caller_to8_from8:
-; COMMON: sub sp, sp, #16
+; COMMON-NOT: sub sp,
 
 ; Key point is that the "%a" should go where at SP on entry.
   tail call fastcc void @callee_stack8([8 x i64] undef, i64 42)
   ret void
 
-; COMMON: str {{x[0-9]+}}, [sp, #16]!
+; COMMON: str {{x[0-9]+}}, [sp]
 ; COMMON-NEXT: b callee_stack8
 }
 
 define fastcc void @caller_to16_from8([8 x i64], i64 %a) {
 ; COMMON-LABEL: caller_to16_from8:
-; COMMON: sub sp, sp, #16
+; COMMON-NOT: sub sp,
 
 ; Important point is that the call reuses the "dead" argument space
 ; above %a on the stack. If it tries to go below incoming-SP then the
 ; callee will not deallocate the space, even in fastcc.
   tail call fastcc void @callee_stack16([8 x i64] undef, i64 42, i64 2)
 
-; COMMON: stp {{x[0-9]+}}, {{x[0-9]+}}, [sp, #16]!
+; COMMON: stp {{x[0-9]+}}, {{x[0-9]+}}, [sp]
 ; COMMON-NEXT: b callee_stack16
   ret void
 }
 
 
 define fastcc void @caller_to8_from24([8 x i64], i64 %a, i64 %b, i64 %c) {
 ; COMMON-LABEL: caller_to8_from24:
-; COMMON: sub sp, sp, #16
+; COMMON-NOT: sub sp,
 
 ; Key point is that the "%a" should go where at #16 above SP on entry.
   tail call fastcc void @callee_stack8([8 x i64] undef, i64 42)
   ret void
 
-; COMMON: str {{x[0-9]+}}, [sp, #32]!
+; COMMON: str {{x[0-9]+}}, [sp, #16]!
 ; COMMON-NEXT: b callee_stack8
 }
 
 
 define fastcc void @caller_to16_from16([8 x i64], i64 %a, i64 %b) {
 ; COMMON-LABEL: caller_to16_from16:
-; COMMON: sub sp, sp, #16
+; COMMON-NOT: sub sp,
 
 ; Here we want to make sure that both loads happen before the stores:
 ; otherwise either %a or %b will be wrongly clobbered.
   tail call fastcc void @callee_stack16([8 x i64] undef, i64 %b, i64 %a)
   ret void
 
-; COMMON: ldp {{x[0-9]+}}, {{x[0-9]+}}, [sp, #16]
-; COMMON: stp {{x[0-9]+}}, {{x[0-9]+}}, [sp, #16]!
+; COMMON: ldp {{x[0-9]+}}, {{x[0-9]+}}, [sp]
+; COMMON: stp {{x[0-9]+}}, {{x[0-9]+}}, [sp]
 ; COMMON-NEXT: b callee_stack16
 }
 
 define fastcc void @disable_tail_calls() nounwind "disable-tail-calls"="true" {
 ; COMMON-LABEL: disable_tail_calls:
 ; COMMON-NEXT: // %bb.
 
   tail call fastcc void @callee_stack0()

llvm/test/CodeGen/AArch64/tailcc-tail-call.ll

+; RUN: llc -verify-machineinstrs < %s -mtriple=aarch64-none-linux-gnu | FileCheck %s --check-prefixes=SDAG,COMMON
+; RUN: llc -global-isel -global-isel-abort=1 -verify-machineinstrs < %s -mtriple=aarch64-none-linux-gnu  | FileCheck %s --check-prefixes=GISEL,COMMON
+
+declare tailcc void @callee_stack0()
+declare tailcc void @callee_stack8([8 x i64], i64)
+declare tailcc void @callee_stack16([8 x i64], i64, i64)
+declare extern_weak tailcc void @callee_weak()
+
+define tailcc void @caller_to0_from0() nounwind {
+; COMMON-LABEL: caller_to0_from0:
+; COMMON-NEXT: // %bb.
+
+  tail call tailcc void @callee_stack0()
+  ret void
+
+; COMMON-NEXT: b callee_stack0
+}
+
+define tailcc void @caller_to0_from8([8 x i64], i64) {
+; COMMON-LABEL: caller_to0_from8:
+
+  tail call tailcc void @callee_stack0()
+  ret void
+
+; COMMON: add sp, sp, #16
+; COMMON-NEXT: b callee_stack0
+}
+
+define tailcc void @caller_to8_from0() "frame-pointer"="all"{
+; COMMON-LABEL: caller_to8_from0:
+
+; Key point is that the "42" should go #16 below incoming stack
+; pointer (we didn't have arg space to reuse).
+  tail call tailcc void @callee_stack8([8 x i64] undef, i64 42)
+  ret void
+
+; COMMON: str {{x[0-9]+}}, [x29, #16]
+; COMMON: ldp x29, x30, [sp], #16
+  ; If there is a sub here then the 42 will be briefly exposed to corruption
+  ; from an interrupt if the kernel does not honour a red-zone, and a larger
+  ; call could well overflow the red zone even if it is present.
+; COMMON-NOT: sub sp,
+; COMMON-NEXT: b callee_stack8
+}
+
+define tailcc void @caller_to8_from8([8 x i64], i64 %a) {
+; COMMON-LABEL: caller_to8_from8:
+; COMMON-NOT: sub sp,
+
+; Key point is that the "%a" should go where at SP on entry.
+  tail call tailcc void @callee_stack8([8 x i64] undef, i64 42)
+  ret void
+
+; COMMON: str {{x[0-9]+}}, [sp]
+; COMMON-NEXT: b callee_stack8
+}
+
+define tailcc void @caller_to16_from8([8 x i64], i64 %a) {
+; COMMON-LABEL: caller_to16_from8:
+; COMMON-NOT: sub sp,
+
+; Important point is that the call reuses the "dead" argument space
+; above %a on the stack. If it tries to go below incoming-SP then the
+; callee will not deallocate the space, even in tailcc.
+  tail call tailcc void @callee_stack16([8 x i64] undef, i64 42, i64 2)
+
+; COMMON: stp {{x[0-9]+}}, {{x[0-9]+}}, [sp]
+; COMMON-NEXT: b callee_stack16
+  ret void
+}
+
+
+define tailcc void @caller_to8_from24([8 x i64], i64 %a, i64 %b, i64 %c) {
+; COMMON-LABEL: caller_to8_from24:
+; COMMON-NOT: sub sp,
+
+; Key point is that the "%a" should go where at #16 above SP on entry.
+  tail call tailcc void @callee_stack8([8 x i64] undef, i64 42)
+  ret void
+
+; COMMON: str {{x[0-9]+}}, [sp, #16]!
+; COMMON-NEXT: b callee_stack8
+}
+
+
+define tailcc void @caller_to16_from16([8 x i64], i64 %a, i64 %b) {
+; COMMON-LABEL: caller_to16_from16:
+; COMMON-NOT: sub sp,
+
+; Here we want to make sure that both loads happen before the stores:
+; otherwise either %a or %b will be wrongly clobbered.
+  tail call tailcc void @callee_stack16([8 x i64] undef, i64 %b, i64 %a)
+  ret void
+
+; COMMON: ldp {{x[0-9]+}}, {{x[0-9]+}}, [sp]
+; COMMON: stp {{x[0-9]+}}, {{x[0-9]+}}, [sp]
+; COMMON-NEXT: b callee_stack16
+}
+
+define tailcc void @disable_tail_calls() nounwind "disable-tail-calls"="true" {
+; COMMON-LABEL: disable_tail_calls:
+; COMMON-NEXT: // %bb.
+
+  tail call tailcc void @callee_stack0()
+  ret void
+
+; COMMON: bl callee_stack0
+; COMMON: ret
+}
+
+; Weakly-referenced extern functions cannot be tail-called, as AAELF does
+; not define the behaviour of branch instructions to undefined weak symbols.
+define tailcc void @caller_weak() {
+; COMMON-LABEL: caller_weak:
+; COMMON: bl callee_weak
+  tail call void @callee_weak()
+  ret void
+}
+
+declare { [2 x float] } @get_vec2()
+
+define { [3 x float] } @test_add_elem() {
+; SDAG-LABEL: test_add_elem:
+; SDAG: bl get_vec2
+; SDAG: fmov s2, #1.0
+; SDAG: ret
+; GISEL-LABEL: test_add_elem:
+; GISEL: str  x30, [sp, #-16]!
+; GISEL: bl get_vec2
+; GISEL: fmov s2, #1.0
+; GISEL: ldr  x30, [sp], #16
+; GISEL: ret
+
+  %call = tail call { [2 x float] } @get_vec2()
+  %arr = extractvalue { [2 x float] } %call, 0
+  %arr.0 = extractvalue [2 x float] %arr, 0
+  %arr.1 = extractvalue [2 x float] %arr, 1
+
+  %res.0 = insertvalue { [3 x float] } undef, float %arr.0, 0, 0
+  %res.01 = insertvalue { [3 x float] } %res.0, float %arr.1, 0, 1
+  %res.012 = insertvalue { [3 x float] } %res.01, float 1.000000e+00, 0, 2
+  ret { [3 x float] } %res.012
+}
+
+declare double @get_double()
+define { double, [2 x double] } @test_mismatched_insert() {
+; COMMON-LABEL: test_mismatched_insert:
+; COMMON: bl get_double
+; COMMON: bl get_double
+; COMMON: bl get_double
+; COMMON: ret
+
+  %val0 = call double @get_double()
+  %val1 = call double @get_double()
+  %val2 = tail call double @get_double()
+
+  %res.0 = insertvalue { double, [2 x double] } undef, double %val0, 0
+  %res.01 = insertvalue { double, [2 x double] } %res.0, double %val1, 1, 0
+  %res.012 = insertvalue { double, [2 x double] } %res.01, double %val2, 1, 1
+
+  ret { double, [2 x double] } %res.012
+}
+
+define void @fromC_totail() {
+; COMMON-LABEL: fromC_totail:
+; COMMON: sub sp, sp, #32
+
+; COMMON-NOT: sub sp,
+; COMMON: mov w[[TMP:[0-9]+]], #42
+; COMMON: str x[[TMP]], [sp]
+; COMMON: bl callee_stack8
+  ; We must reset the stack to where it was before the call by undoing its extra stack pop.
+; COMMON: str x[[TMP]], [sp, #-16]!
+; COMMON: bl callee_stack8
+; COMMON: sub sp, sp, #16
+
+  call tailcc void @callee_stack8([8 x i64] undef, i64 42)
+  call tailcc void @callee_stack8([8 x i64] undef, i64 42)
+  ret void
+}
+
+define void @fromC_totail_noreservedframe(i32 %len) {
+; COMMON-LABEL: fromC_totail_noreservedframe:
+; COMMON: stp x29, x30, [sp, #-32]!
+
+; COMMON: mov w[[TMP:[0-9]+]], #42
+  ; Note stack is subtracted here to allocate space for arg
+; COMMON: str x[[TMP]], [sp, #-16]!
+; COMMON: bl callee_stack8
+  ; And here.
+; COMMON: str x[[TMP]], [sp, #-16]!
+; COMMON: bl callee_stack8
+  ; But not restored here because callee_stack8 did that for us.
+; COMMON-NOT: sub sp,
+
+  ; Variable sized allocation prevents reserving frame at start of function so each call must allocate any stack space it needs.
+  %var = alloca i32, i32 %len
+
+  call tailcc void @callee_stack8([8 x i64] undef, i64 42)
+  call tailcc void @callee_stack8([8 x i64] undef, i64 42)
+  ret void
+}
+
+declare void @Ccallee_stack8([8 x i64], i64)
+
+define tailcc void @fromtail_toC() {
+; COMMON-LABEL: fromtail_toC:
+; COMMON: sub sp, sp, #32
+
+; COMMON-NOT: sub sp,
+; COMMON: mov w[[TMP:[0-9]+]], #42
+; COMMON: str x[[TMP]], [sp]
+; COMMON: bl Ccallee_stack8
+  ; C callees will return with the stack exactly where we left it, so we mustn't try to fix anything.
+; COMMON-NOT: add sp,
+; COMMON-NOT: sub sp,
+; COMMON: str x[[TMP]], [sp]{{$}}
+; COMMON: bl Ccallee_stack8
+; COMMON-NOT: sub sp,
+
+
+  call void @Ccallee_stack8([8 x i64] undef, i64 42)
+  call void @Ccallee_stack8([8 x i64] undef, i64 42)
+  ret void
+}

llvm/test/CodeGen/X86/swifttail-async-i386.ll

+; RUN: llc -mtriple=i386-apple-darwin %s -o - | FileCheck %s
+
+declare void @clobber()
+
+declare swifttailcc void @swifttail_callee()
+define swifttailcc void @swifttail() {
+; CHECK-LABEL: swifttail:
+; CHECK-NOT: %rbx
+  call void @clobber()
+  tail call swifttailcc void @swifttail_callee()
+  ret void
+}
+
+declare swifttailcc void @swiftself(i8* swiftself)
+
+define swifttailcc void @swifttail2(i8* %arg) {
+; CHECK-LABEL: swifttail2:
+; CHECK: movl {{.*}}, %ecx
+; CHECK: jmp _swiftself
+  tail call swifttailcc void @swiftself(i8* swiftself %arg)
+  ret void
+}

llvm/test/CodeGen/X86/swifttail-async.ll

+; RUN: llc -mtriple=x86_64-apple-darwin %s -o - | FileCheck %s
+
+
+declare swifttailcc void @swifttail_callee()
+define swifttailcc void @swifttail() {
+; CHECK-LABEL: swifttail:
+; CHECK-NOT: popq %r14
+  call void asm "","~{r14}"()
+  tail call swifttailcc void @swifttail_callee()
+  ret void
+}
+
+define swifttailcc void @no_preserve_swiftself() {
+; CHECK-LABEL: no_preserve_swiftself:
+; CHECK-NOT: popq %r13
+  call void asm "","~{r13}"()
+  ret void
+}
+
+declare swifttailcc i8* @SwiftSelf(i8 * swiftasync %context, i8* swiftself %closure)
+define swiftcc i8* @CallSwiftSelf(i8* swiftself %closure, i8* %context) {
+; CHECK-LABEL: CallSwiftSelf:
+; CHECK: pushq %r13
+  ;call void asm "","~{r13}"() ; We get a push r13 but why not with the call
+  ; below?
+  %res = call swifttailcc i8* @SwiftSelf(i8 * swiftasync %context, i8* swiftself %closure)
+  ret i8* %res
+}

llvm/test/CodeGen/X86/swifttail-return.ll

+; RUN: llc -verify-machineinstrs < %s -mtriple=x86_64-unknown-unknown | FileCheck %s
+; RUN: llc -verify-machineinstrs < %s -mtriple=x86_64-unknown-unknown -O0 | FileCheck %s
+
+define swifttailcc [4 x i64] @return_int() {
+; CHECK-LABEL: return_int:
+; CHECK-DAG: movl $1, %eax
+; CHECK-DAG: movl $2, %edx
+; CHECK-DAG: movl $3, %ecx
+; CHECK-DAG: movl $4, %r8d
+
+  ret [4 x i64] [i64 1, i64 2, i64 3, i64 4]
+}
+
+
+; CHECK: [[ONE:.LCPI.*]]:
+; CHECK-NEXT: # double 1
+; CHECK: [[TWO:.LCPI.*]]:
+; CHECK-NEXT: # double 2
+; CHECK: [[THREE:.LCPI.*]]:
+; CHECK-NEXT: # double 3
+
+define swifttailcc [4 x double] @return_float() {
+; CHECK-LABEL: return_float:
+; CHECK-DAG: movsd [[ONE]](%rip), %xmm1
+; CHECK-DAG: movsd [[TWO]](%rip), %xmm2
+; CHECK-DAG: movsd [[THREE]](%rip), %xmm3
+; CHECK-DAG: xorps %xmm0, %xmm0
+  ret [4 x double] [double 0.0, double 1.0, double 2.0, double 3.0]
+}

llvm/test/CodeGen/X86/tailcall-swifttailcc.ll

+; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s
+
+declare dso_local swifttailcc i32 @tailcallee(i32 %a1, i32 %a2, i32 %a3, i32 %a4)
+
+define dso_local swifttailcc i32 @tailcaller(i32 %in1, i32 %in2) nounwind {
+; CHECK-LABEL: tailcaller:
+; CHECK-NOT: subq
+; CHECK-NOT: addq
+; CHECK: jmp tailcallee
+entry:
+  %tmp11 = tail call swifttailcc i32 @tailcallee(i32 %in1, i32 %in2, i32 %in1, i32 %in2)
+  ret i32 %tmp11
+}
+
+declare dso_local swifttailcc i8* @alias_callee()
+
+define swifttailcc noalias i8* @noalias_caller() nounwind {
+; CHECK-LABEL: noalias_caller:
+; CHECK:    jmp alias_callee
+  %p = musttail call swifttailcc i8* @alias_callee()
+  ret i8* %p
+}
+
+declare dso_local swifttailcc noalias i8* @noalias_callee()
+
+define dso_local swifttailcc i8* @alias_caller() nounwind {
+; CHECK-LABEL: alias_caller:
+; CHECK:    jmp noalias_callee # TAILCALL
+  %p = tail call swifttailcc noalias i8* @noalias_callee()
+  ret i8* %p
+}
+
+declare dso_local swifttailcc i32 @i32_callee()
+
+define dso_local swifttailcc i32 @ret_undef() nounwind {
+; CHECK-LABEL: ret_undef:
+; CHECK:    jmp i32_callee # TAILCALL
+  %p = tail call swifttailcc i32 @i32_callee()
+  ret i32 undef
+}
+
+declare dso_local swifttailcc void @does_not_return()
+
+define dso_local swifttailcc i32 @noret() nounwind {
+; CHECK-LABEL: noret:
+; CHECK:    jmp does_not_return
+  tail call swifttailcc void @does_not_return()
+  unreachable
+}
+
+define dso_local swifttailcc void @void_test(i32, i32, i32, i32) {
+; CHECK-LABEL: void_test:
+; CHECK:    jmp void_test
+  entry:
+   tail call swifttailcc void @void_test( i32 %0, i32 %1, i32 %2, i32 %3)
+   ret void
+}
+
+define dso_local swifttailcc i1 @i1test(i32, i32, i32, i32) {
+; CHECK-LABEL: i1test:
+; CHECK:    jmp i1test
+  entry:
+  %4 = tail call swifttailcc i1 @i1test( i32 %0, i32 %1, i32 %2, i32 %3)
+  ret i1 %4
+}

llvm/utils/emacs/llvm-mode.el

`(,(regexp-opt
          ;; Linkage types
          "private" "internal" "weak" "weak_odr" "linkonce" "linkonce_odr" "available_externally" "appending" "common" "extern_weak" "external"
          "uninitialized" "implementation" "..."
          ;; Values
          "true" "false" "null" "undef" "zeroinitializer" "none" "c" "asm" "blockaddress" "poison"
 
          ;; Calling conventions
          "ccc" "fastcc" "coldcc" "webkit_jscc" "anyregcc" "preserve_mostcc" "preserve_allcc"
-         "cxx_fast_tlscc" "swiftcc" "tailcc" "cfguard_checkcc"
+         "cxx_fast_tlscc" "swiftcc" "tailcc" "swifttailcc" "cfguard_checkcc"
          ;; Visibility styles
          "default" "hidden" "protected"
          ;; DLL storages
          "dllimport" "dllexport"
          ;; Thread local
          "thread_local" "localdynamic" "initialexec" "localexec"
          ;; Runtime preemption specifiers
          "dso_preemptable" "dso_local" "dso_local_equivalent"

llvm/utils/vim/syntax/llvm.vim

syn keyword llvmKeyword
       \ spir_kernel
       \ sret
       \ ssp
       \ sspreq
       \ sspstrong
       \ strictfp
       \ swiftcc
       \ swifterror
+      \ swifttailcc
       \ swiftself
       \ syncscope
       \ tail
       \ tailcc
       \ target
       \ thread_local
       \ to
       \ triple