IR+AArch64: add `swiftasync` function parameter attribute

[t.p.northover]

Summary

Swift is getting some new asynchronous execution abilities which means functions will want to pass special execution contexts along in a way that is preserved by unaware functions much like swiftself. So this adds a parallel attribute to support that, which will map to a callee saved-register. That accounts for the fairly straightforward IR changes here.

Also, when this kind of argument is present, much of the control flow will be via guaranteed tail-calls which since they destroy the stack frame don't preserve backtrace information well. So when one of these contexts is present it will extend the usual (on AArch64) [FP, LR] frame record by putting the context just before it: [Ctx, FP, LR] (before rather than afterwards so that the same layout can be used on x86 where callq pushes the return address).

Finally, so that system tools and other things capturing stack traces can detect this kind of frame, the in-memory FP gets bit 60 set to 1.

On arm64e, this context will be signed in an address-discriminated way.

[kristof.beyls]

Just a few quick, knee-jerk-style, questions. My apologies if these are too simplistic or vague.

• Do you happen to have a pointer to some more documentation about this new Swift feature?
• I wonder if any features from other languages could also map to this IR/backend implementation? E.g. I haven't looked at how C++ co-routines are implemented, but would they be able to reuse this feature (assuming they would need a somewhat similar feature in mid- and back-end).
• As this seems to be introducing new ABI - do you have an idea of what the most appropriate ABI specification is this would be best documented in? Do I guess correctly that this is not a Darwin-only feature? If this is really swift-specific (i.e. it does not seem to be a good trade-off to have this mid/back-end feature support similar language features for multiple front-ends), I guess this is Swift-specific ABI. Do I guess correctly that there isn't really a specification for Swift-specific ABI - it's purely defined by its primary implementation?
• I just had one very detailed nit-pick comment on the documentation of frame layout - see specific comment.

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

49–54

Would it be useful to update the documentation here on the frame record for AArch64 to cover "So when one of these contexts is present it will extend the usual (on AArch64) [FP, LR] frame record by putting the context just before it: [Ctx, FP, LR]"?

[t.p.northover]

Thanks for the questions. I'm quite fuzzy on the Swift side of this I'm afraid, so take my replies with a pinch of salt.

Do you happen to have a pointer to some more documentation about this new Swift feature?

Most of the ABI-specific things are unfortunately on internal documents. The Swift proposals exist and I'll put links here, but kind of understandably they focus on the implications for Swift itself. But with that in mind, there's:

https://github.com/DougGregor/swift-evolution/blob/async-await/proposals/nnnn-async-await.md
https://github.com/DougGregor/swift-evolution/blob/structured-concurrency/proposals/nnnn-structured-concurrency.md
https://github.com/DougGregor/swift-evolution/blob/actors/proposals/nnnn-actors.md

I think that third one about Actors is probably most significant here, as far as I understand the idea is that the context will store the current Actor, the current Task, and an asynchronous backtrace structure. I'm afraid I don't yet know what form that will take, even from our own pages.

I wonder if any features from other languages could also map to this IR/backend implementation? E.g. I haven't looked at how C++ co-routines are implemented, but would they be able to reuse this feature (assuming they would need a somewhat similar feature in mid- and back-end).

I think it's generic enough that other languages could use it. The proposed context prefix appears to be:

struct SwiftAsyncContext {
  struct SwiftAsyncContext *callerFrame;
  void *callerReturnAddress;
};

File off the name Swift and I think any front-end could construct such a structure. Unfortunately, I think it would be an ABI break for C++ at this stage; an old caller would pass rubbish into a function trying to store this context and that's not good.

As this seems to be introducing new ABI - do you have an idea of what the most appropriate ABI specification is this would be best documented in?

I think we should document the reusable bits, so probably up to that struct above and state that what follows those two pointers is up to the language.

Do I guess correctly that this is not a Darwin-only feature?

That's right. I think it's mostly a quality-of-life feature (so it's not critical for me to get this done on AArch32 right now, for example), but Swift runs on Linux & Windows too, and I'd hope they can use it (even if the default system tools
don't catch up for a while, or ever). I haven't done extensive testing on them yet though.

If this is really swift-specific (i.e. it does not seem to be a good trade-off to have this mid/back-end feature support similar language features for multiple front-ends), I guess this is Swift-specific ABI. Do I guess correctly that there isn't really a specification for Swift-specific ABI - it's purely defined by its primary implementation?

I think that's right.

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

49–54

Yep, I'll do that.

[t.p.northover]

1. Applied Kristof's FrameLowering suggestion.
2. Added IR-level tests for the attribute I'd forgotten about.

[varungandhi-apple]

llvm/include/llvm/Bitcode/LLVMBitCodes.h

661

This is going to hit a merge conflict or a miscompile; on main, 73 is already taken by ATTR_KIND_NO_PROFILE.

[t.p.northover]

llvm/include/llvm/Bitcode/LLVMBitCodes.h

661

Yep, for this kind of thing the last step is always going to be rebasing and refreshing magic constants. If I've done my job right, the regression tests should be enough to flag it even if I forget.

[t.p.northover]

Ping.

[t.p.northover]

Ping.

[t.p.northover]

Just a rebase really (minor comment & doc changes). Also ping.

[t.p.northover]

Ping.

[t.p.northover]

Ping.

[t.p.northover]

Ping.

[t.p.northover]

Ping.

[t.p.northover]

Ping.

[jroelofs]

llvm/docs/LangRef.rst

12360

I think the ^s need to be the same length as the text above it, otherwise we'll get sphinx warnings.

12379

by "the function" do you mean the caller?

llvm/include/llvm/CodeGen/TargetFrameLowering.h

157

Are callers expected to know whether the implementation of this API sets {Min,Max}CSFrameIndex? If there's a way to derive them, or set them to some default value, I think that would make this easier to hold.

Maybe it's better to implement the other one in terms of this one, to force them to be correctly populated even if they're not used?

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

732

The immediate probably needs an explanation. IIUC this is the discriminator?

737

If it's supposed to be zeroing X17, wouldn't MOVZ be better?

The comment above suggests it should be doing a mov x17, x22 though, and neither behavior matches what the code is doing here.

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

1147

do you mean #0x1000_1000_0000_0000?

1793

I'm having trouble making sense of this immediate. Comment above says it clears the MSB, but this looks like it does more than that.

2910

sink the declaration of FrameIdx to its definition.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

139

This is for that pacdb sequence right? Might be worth a comment here showing what the 5 instructions are.

[thegameg]

Flags/attributes/calling conventions look very straightforward.

FrameLowering is always getting harder and harder to review but I think your approach with the StoreSwiftAsyncContext makes it much easier! Thanks!

This LGTM (with the rest of the comments addressed).

llvm/lib/Target/AArch64/MCTargetDesc/AArch64AsmBackend.cpp

636

I'm having trouble seeing what this is checking for. I understand we want to fallback to DWARF like you explained in llvm/test/CodeGen/AArch64/swift-async-unwind.ll, but I'm not sure I get the way CurOffset is used to check for that.

[t.p.northover]

Thanks for the comments. I've implemented a lot of them and explained the reasoning behind what I did for some. I'll upload a new patch.

llvm/include/llvm/CodeGen/TargetFrameLowering.h

157

There's just one caller of this function, in PrologEpilogInserter.cpp, and it calls this new version. Targets that want to set the new variables will override this one, targets that don't will override the other one.

If the target doesn't set them, PrologEpilogInserter supplies defaults (essentially saying no registers count as callee-saved) and I believe that changes some details like spill order.

I'm not sure if there's a point to that, just explaining some of the background really.

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

737

On AArch64, mov xD, xS is actually just an alias for the instruction fully described as orr xD, xzr, xS, lsl #0 which is this what we're doing here (mov x17, CtxReg). I think that matches the comment.

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

1147

No, it's just that one bit. The encoding for AArch64 bitwise immediates is weird (it allows weird repeating patterns etc).

1793

Same as above, weird encoding for immediates. Hence the comment, really.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

139

This switch is notorious for getting out of date when expansion changes. But the biggest problem isn't people not knowing what a pseudo might expand to; it's forgetting the switch exists at all.

A comment here is just one more thing to go stale rather than a useful thing you might act on (IMO).

llvm/lib/Target/AArch64/MCTargetDesc/AArch64AsmBackend.cpp

636

Until now, this generator assumed that the .cfi directives it saw always started at an offset just after where FP was stored and counted up by 8 each time.

With the gap for the new context (and possibly padding) this is no longer true so we have to check that assumption and bail to DWARF if it's violated. This seemed like the least fragile way to do that, even though the specific hole we're looking for could be checked more easily.

[t.p.northover]

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

2910

Promise I have done this, I just forgot to save the file before committing when I uploaded the new diff.

[jroelofs]

LGTM

llvm/include/llvm/CodeGen/TargetFrameLowering.h

157

ok, sounds good.

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

737

ack

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

1147

ack

1793

ack

2910

ack

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

139

ack

[t.p.northover]

Thanks Jon. Committed as ea0eec69f16e

llvm/docs/LangRef.rst

``dereferenceable_or_null(<n>)``
     ``dereferenceable(<n>)``). This attribute may only be applied to
     pointer typed parameters.
 
 ``swiftself``
     This indicates that the parameter is the self/context parameter. This is not
     a valid attribute for return values and can only be applied to one
     parameter.
 
+``swiftasync``
+    This indicates that the parameter is the asynchronous context parameter and
+    triggers the creation of a target-specific extended frame record to store
+    this pointer. This is not a valid attribute for return values and can only
+    be applied to one parameter.
+
 ``swifterror``
     This attribute is motivated to model and optimize Swift error handling. It
     can be applied to a parameter with pointer to pointer type or a
     pointer-sized alloca. At the call site, the actual argument that corresponds
     to a ``swifterror`` parameter has to come from a ``swifterror`` alloca or
     the ``swifterror`` parameter of the caller. A ``swifterror`` value (either
     the parameter or the alloca) can only be loaded and stored from, or used as
     a ``swifterror`` argument. This is not a valid attribute for return values

 cannot be identified. The value returned by this intrinsic is likely to
 be incorrect or 0 for arguments other than zero, so it should only be
 used for debugging purposes.
 
 Note that calling this intrinsic does not prevent function inlining or
 other aggressive transformations, so the value returned may not be that
 of the obvious source-language caller.
 
+'``llvm.swift.async.context.addr``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+
+::
+
+      declare i8** @llvm.swift.async.context.addr()
+
+Overview:
+"""""""""
+
+The '``llvm.swift.async.context.addr``' intrinsic returns a pointer to
+the part of the extended frame record containing the asynchronous
+context of a Swift execution.
+
+Semantics:
+""""""""""
+
+If the function has a ``swiftasync`` parameter, that argument will initially
+be stored at the returned address. If not, it will be initialized to null.
+
 '``llvm.localescape``' and '``llvm.localrecover``' Intrinsics
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 ::
 

llvm/include/llvm/AsmParser/LLToken.h

enum Kind {
   kw_shadowcallstack,
   kw_sret,
   kw_sanitize_thread,
   kw_sanitize_memory,
   kw_speculative_load_hardening,
   kw_strictfp,
   kw_swifterror,
   kw_swiftself,
+  kw_swiftasync,
   kw_uwtable,
   kw_vscale_range,
   kw_willreturn,
   kw_writeonly,
   kw_zeroext,
   kw_immarg,
   kw_byref,
   kw_mustprogress,

llvm/include/llvm/Bitcode/LLVMBitCodes.h

enum AttributeKindCodes {
   ATTR_KIND_PREALLOCATED = 65,
   ATTR_KIND_NO_MERGE = 66,
   ATTR_KIND_NULL_POINTER_IS_VALID = 67,
   ATTR_KIND_NOUNDEF = 68,
   ATTR_KIND_BYREF = 69,
   ATTR_KIND_MUSTPROGRESS = 70,
   ATTR_KIND_NO_CALLBACK = 71,
   ATTR_KIND_HOT = 72,
   ATTR_KIND_NO_PROFILE = 73,
   ATTR_KIND_VSCALE_RANGE = 74,
+  ATTR_KIND_SWIFT_ASYNC = 75,
 };
 
 enum ComdatSelectionKindCodes {
   COMDAT_SELECTION_KIND_ANY = 1,
   COMDAT_SELECTION_KIND_EXACT_MATCH = 2,
   COMDAT_SELECTION_KIND_LARGEST = 3,
   COMDAT_SELECTION_KIND_NO_DUPLICATES = 4,
   COMDAT_SELECTION_KIND_SAME_SIZE = 5,

llvm/include/llvm/CodeGen/TargetCallingConv.h

private:
     unsigned IsByRef : 1;    ///< Passed in memory
     unsigned IsNest : 1;     ///< Nested fn static chain
     unsigned IsReturned : 1; ///< Always returned
     unsigned IsSplit : 1;
     unsigned IsInAlloca : 1;   ///< Passed with inalloca
     unsigned IsPreallocated : 1; ///< ByVal without the copy
     unsigned IsSplitEnd : 1;   ///< Last part of a split
     unsigned IsSwiftSelf : 1;  ///< Swift self parameter
+    unsigned IsSwiftAsync : 1;  ///< Swift async context parameter
     unsigned IsSwiftError : 1; ///< Swift error parameter
     unsigned IsCFGuardTarget : 1; ///< Control Flow Guard target
     unsigned IsHva : 1;        ///< HVA field for
     unsigned IsHvaStart : 1;   ///< HVA structure start
     unsigned IsSecArgPass : 1; ///< Second argument
     unsigned MemAlign : 4;     ///< Log 2 of alignment when arg is passed in memory
                                ///< (including byval/byref)
     unsigned OrigAlign : 5;    ///< Log 2 of original alignment
     unsigned IsInConsecutiveRegsLast : 1;
     unsigned IsInConsecutiveRegs : 1;
     unsigned IsCopyElisionCandidate : 1; ///< Argument copy elision candidate
     unsigned IsPointer : 1;
 
     unsigned ByValOrByRefSize; ///< Byval or byref struct size
 
     unsigned PointerAddrSpace; ///< Address space of pointer argument
 
   public:
     ArgFlagsTy()
         : IsZExt(0), IsSExt(0), IsInReg(0), IsSRet(0), IsByVal(0), IsByRef(0),
-          IsNest(0), IsReturned(0), IsSplit(0), IsInAlloca(0), IsPreallocated(0),
-          IsSplitEnd(0), IsSwiftSelf(0), IsSwiftError(0), IsCFGuardTarget(0),
-          IsHva(0), IsHvaStart(0), IsSecArgPass(0), MemAlign(0),
-          OrigAlign(0), IsInConsecutiveRegsLast(0), IsInConsecutiveRegs(0),
+          IsNest(0), IsReturned(0), IsSplit(0), IsInAlloca(0),
+          IsPreallocated(0), IsSplitEnd(0), IsSwiftSelf(0), IsSwiftAsync(0),
+          IsSwiftError(0), IsCFGuardTarget(0), IsHva(0), IsHvaStart(0),
+          IsSecArgPass(0), MemAlign(0), OrigAlign(0),
+          IsInConsecutiveRegsLast(0), IsInConsecutiveRegs(0),
           IsCopyElisionCandidate(0), IsPointer(0), ByValOrByRefSize(0),
           PointerAddrSpace(0) {
       static_assert(sizeof(*this) == 3 * sizeof(unsigned), "flags are too big");
     }
 
     bool isZExt() const { return IsZExt; }
     void setZExt() { IsZExt = 1; }
 
public:
     void setInAlloca() { IsInAlloca = 1; }
 
     bool isPreallocated() const { return IsPreallocated; }
     void setPreallocated() { IsPreallocated = 1; }
 
     bool isSwiftSelf() const { return IsSwiftSelf; }
     void setSwiftSelf() { IsSwiftSelf = 1; }
 
+    bool isSwiftAsync() const { return IsSwiftAsync; }
+    void setSwiftAsync() { IsSwiftAsync = 1; }
+
     bool isSwiftError() const { return IsSwiftError; }
     void setSwiftError() { IsSwiftError = 1; }
 
     bool isCFGuardTarget() const { return IsCFGuardTarget; }
     void setCFGuardTarget() { IsCFGuardTarget = 1; }
 
     bool isHva() const { return IsHva; }
     void setHva() { IsHva = 1; }

llvm/include/llvm/CodeGen/TargetFrameLowering.h

public:
   virtual bool isFPCloseToIncomingSP() const { return true; }
 
   /// assignCalleeSavedSpillSlots - Allows target to override spill slot
   /// assignment logic.  If implemented, assignCalleeSavedSpillSlots() should
   /// assign frame slots to all CSI entries and return true.  If this method
   /// returns false, spill slots will be assigned using generic implementation.
   /// assignCalleeSavedSpillSlots() may add, delete or rearrange elements of
   /// CSI.
+  virtual bool assignCalleeSavedSpillSlots(MachineFunction &MF,
+                                           const TargetRegisterInfo *TRI,
+                                           std::vector<CalleeSavedInfo> &CSI,
+                                           unsigned &MinCSFrameIndex,
+                                           unsigned &MaxCSFrameIndex) const {
+    return assignCalleeSavedSpillSlots(MF, TRI, CSI);
+  }
+
   virtual bool
   assignCalleeSavedSpillSlots(MachineFunction &MF,
                               const TargetRegisterInfo *TRI,
                               std::vector<CalleeSavedInfo> &CSI) const {
     return false;
   }
 
   /// getCalleeSavedSpillSlots - This method returns a pointer to an array of

llvm/include/llvm/CodeGen/TargetLowering.h

public:
     bool IsSRet : 1;
     bool IsNest : 1;
     bool IsByVal : 1;
     bool IsByRef : 1;
     bool IsInAlloca : 1;
     bool IsPreallocated : 1;
     bool IsReturned : 1;
     bool IsSwiftSelf : 1;
+    bool IsSwiftAsync : 1;
     bool IsSwiftError : 1;
     bool IsCFGuardTarget : 1;
     MaybeAlign Alignment = None;
     // Type for byval, inalloca, or preallocated.
     Type *IndirectType = nullptr;
 
     ArgListEntry()
         : IsSExt(false), IsZExt(false), IsInReg(false), IsSRet(false),
           IsNest(false), IsByVal(false), IsByRef(false), IsInAlloca(false),
           IsPreallocated(false), IsReturned(false), IsSwiftSelf(false),
-          IsSwiftError(false), IsCFGuardTarget(false) {}
+          IsSwiftAsync(false), IsSwiftError(false), IsCFGuardTarget(false) {}
 
     void setAttributes(const CallBase *Call, unsigned ArgIdx);
   };
   using ArgListTy = std::vector<ArgListEntry>;
 
   virtual void markLibCallAttributes(MachineFunction *MF, unsigned CC,
                                      ArgListTy &Args) const {};
 

llvm/include/llvm/IR/Attributes.td

 def SpeculativeLoadHardening : EnumAttr<"speculative_load_hardening">;
 
 /// Argument is swift error.
 def SwiftError : EnumAttr<"swifterror">;
 
 /// Argument is swift self/context.
 def SwiftSelf : EnumAttr<"swiftself">;
 
+/// Argument is swift async context.
+def SwiftAsync : EnumAttr<"swiftasync">;
+
 /// Function must be in a unwind table.
 def UWTable : EnumAttr<"uwtable">;
 
 /// Minimum/Maximum vscale value for function.
 def VScaleRange : IntAttr<"vscale_range">;
 
 /// Function always comes back to callsite.
 def WillReturn : EnumAttr<"willreturn">;

llvm/include/llvm/IR/Intrinsics.td

def int_objc_arc_annotation_topdown_bbend   : Intrinsic<[],
                                                         [llvm_ptrptr_ty,
                                                          llvm_ptrptr_ty]>;
 def int_objc_arc_annotation_bottomup_bbstart  : Intrinsic<[],
                                                           [llvm_ptrptr_ty,
                                                            llvm_ptrptr_ty]>;
 def int_objc_arc_annotation_bottomup_bbend  : Intrinsic<[],
                                                         [llvm_ptrptr_ty,
                                                          llvm_ptrptr_ty]>;
+//===--------------- Swift asynchronous context intrinsics ----------------===//
 
+// Returns the location of the Swift asynchronous context (usually stored just
+// before the frame pointer), and triggers the creation of a null context if it
+// would otherwise be unneeded.
+def int_swift_async_context_addr : Intrinsic<[llvm_ptrptr_ty], [], [IntrNoMem]>;
 
 //===--------------------- Code Generator Intrinsics ----------------------===//
 //
 def int_returnaddress : DefaultAttrsIntrinsic<[llvm_ptr_ty], [llvm_i32_ty],
                                   [IntrNoMem, ImmArg<ArgIndex<0>>]>;
 def int_addressofreturnaddress : DefaultAttrsIntrinsic<[llvm_anyptr_ty], [], [IntrNoMem]>;
 def int_frameaddress : DefaultAttrsIntrinsic<[llvm_anyptr_ty], [llvm_i32_ty],
                                  [IntrNoMem, ImmArg<ArgIndex<0>>]>;

llvm/include/llvm/Target/TargetCallingConv.td

 class CCIfPreallocated<CCAction A> : CCIf<"ArgFlags.isPreallocated()", A> {
 }
 
 /// CCIfSwiftSelf - If the current argument has swiftself parameter attribute,
 /// apply Action A.
 class CCIfSwiftSelf<CCAction A> : CCIf<"ArgFlags.isSwiftSelf()", A> {
 }
 
+/// CCIfSwiftAsync - If the current argument has swiftasync parameter attribute,
+/// apply Action A.
+class CCIfSwiftAsync<CCAction A> : CCIf<"ArgFlags.isSwiftAsync()", A> {
+}
+
 /// CCIfSwiftError - If the current argument has swifterror parameter attribute,
 /// apply Action A.
 class CCIfSwiftError<CCAction A> : CCIf<"ArgFlags.isSwiftError()", A> {
 }
 
 /// CCIfCFGuardTarget - If the current argument has cfguardtarget parameter
 /// attribute, apply Action A.
 class CCIfCFGuardTarget<CCAction A> : CCIf<"ArgFlags.isCFGuardTarget()", A> {

llvm/lib/AsmParser/LLLexer.cpp

#define KEYWORD(STR)                                                           \
   KEYWORD(sanitize_address);
   KEYWORD(sanitize_hwaddress);
   KEYWORD(sanitize_memtag);
   KEYWORD(sanitize_thread);
   KEYWORD(sanitize_memory);
   KEYWORD(speculative_load_hardening);
   KEYWORD(swifterror);
   KEYWORD(swiftself);
+  KEYWORD(swiftasync);
   KEYWORD(uwtable);
   KEYWORD(vscale_range);
   KEYWORD(willreturn);
   KEYWORD(writeonly);
   KEYWORD(zeroext);
   KEYWORD(immarg);
   KEYWORD(byref);
   KEYWORD(mustprogress);

llvm/lib/AsmParser/LLParser.cpp

while (true) {
     case lltok::kw_noalias:
     case lltok::kw_noundef:
     case lltok::kw_nocapture:
     case lltok::kw_nonnull:
     case lltok::kw_returned:
     case lltok::kw_sret:
     case lltok::kw_swifterror:
     case lltok::kw_swiftself:
+    case lltok::kw_swiftasync:
     case lltok::kw_immarg:
     case lltok::kw_byref:
       HaveError |=
           error(Lex.getLoc(),
                 "invalid use of parameter-only attribute on a function");
       break;
     }
 
while (true) {
     case lltok::kw_nofree:          B.addAttribute(Attribute::NoFree); break;
     case lltok::kw_nonnull:         B.addAttribute(Attribute::NonNull); break;
     case lltok::kw_readnone:        B.addAttribute(Attribute::ReadNone); break;
     case lltok::kw_readonly:        B.addAttribute(Attribute::ReadOnly); break;
     case lltok::kw_returned:        B.addAttribute(Attribute::Returned); break;
     case lltok::kw_signext:         B.addAttribute(Attribute::SExt); break;
     case lltok::kw_swifterror:      B.addAttribute(Attribute::SwiftError); break;
     case lltok::kw_swiftself:       B.addAttribute(Attribute::SwiftSelf); break;
+    case lltok::kw_swiftasync:      B.addAttribute(Attribute::SwiftAsync); break;
     case lltok::kw_writeonly:       B.addAttribute(Attribute::WriteOnly); break;
     case lltok::kw_zeroext:         B.addAttribute(Attribute::ZExt); break;
     case lltok::kw_immarg:          B.addAttribute(Attribute::ImmArg); break;
 
     case lltok::kw_alwaysinline:
     case lltok::kw_argmemonly:
     case lltok::kw_builtin:
     case lltok::kw_inlinehint:
while (true) {
     case lltok::kw_byval:
     case lltok::kw_inalloca:
     case lltok::kw_nest:
     case lltok::kw_nocapture:
     case lltok::kw_returned:
     case lltok::kw_sret:
     case lltok::kw_swifterror:
     case lltok::kw_swiftself:
+    case lltok::kw_swiftasync:
     case lltok::kw_immarg:
     case lltok::kw_byref:
       HaveError |=
           error(Lex.getLoc(), "invalid use of parameter-only attribute");
       break;
 
     case lltok::kw_alignstack:
     case lltok::kw_alwaysinline:

llvm/lib/Bitcode/Reader/BitcodeReader.cpp

static Attribute::AttrKind getAttrFromCode(uint64_t Code) {
   case bitc::ATTR_KIND_SANITIZE_MEMORY:
     return Attribute::SanitizeMemory;
   case bitc::ATTR_KIND_SPECULATIVE_LOAD_HARDENING:
     return Attribute::SpeculativeLoadHardening;
   case bitc::ATTR_KIND_SWIFT_ERROR:
     return Attribute::SwiftError;
   case bitc::ATTR_KIND_SWIFT_SELF:
     return Attribute::SwiftSelf;
+  case bitc::ATTR_KIND_SWIFT_ASYNC:
+    return Attribute::SwiftAsync;
   case bitc::ATTR_KIND_UW_TABLE:
     return Attribute::UWTable;
   case bitc::ATTR_KIND_VSCALE_RANGE:
     return Attribute::VScaleRange;
   case bitc::ATTR_KIND_WILLRETURN:
     return Attribute::WillReturn;
   case bitc::ATTR_KIND_WRITEONLY:
     return Attribute::WriteOnly;

llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

static uint64_t getAttrKindEncoding(Attribute::AttrKind Kind) {
   case Attribute::SanitizeMemory:
     return bitc::ATTR_KIND_SANITIZE_MEMORY;
   case Attribute::SpeculativeLoadHardening:
     return bitc::ATTR_KIND_SPECULATIVE_LOAD_HARDENING;
   case Attribute::SwiftError:
     return bitc::ATTR_KIND_SWIFT_ERROR;
   case Attribute::SwiftSelf:
     return bitc::ATTR_KIND_SWIFT_SELF;
+  case Attribute::SwiftAsync:
+    return bitc::ATTR_KIND_SWIFT_ASYNC;
   case Attribute::UWTable:
     return bitc::ATTR_KIND_UW_TABLE;
   case Attribute::VScaleRange:
     return bitc::ATTR_KIND_VSCALE_RANGE;
   case Attribute::WillReturn:
     return bitc::ATTR_KIND_WILLRETURN;
   case Attribute::WriteOnly:
     return bitc::ATTR_KIND_WRITEONLY;

llvm/lib/CodeGen/GlobalISel/CallLowering.cpp

addFlagsUsingAttrFn(ISD::ArgFlagsTy &Flags,
   if (AttrFn(Attribute::Preallocated))
     Flags.setPreallocated();
   if (AttrFn(Attribute::InAlloca))
     Flags.setInAlloca();
   if (AttrFn(Attribute::Returned))
     Flags.setReturned();
   if (AttrFn(Attribute::SwiftSelf))
     Flags.setSwiftSelf();
+  if (AttrFn(Attribute::SwiftAsync))
+    Flags.setSwiftAsync();
   if (AttrFn(Attribute::SwiftError))
     Flags.setSwiftError();
 }
 
 ISD::ArgFlagsTy CallLowering::getAttributesForArgIdx(const CallBase &Call,
                                                      unsigned ArgIdx) const {
   ISD::ArgFlagsTy Flags;
   addFlagsUsingAttrFn(Flags, [&Call, &ArgIdx](Attribute::AttrKind Attr) {

llvm/lib/CodeGen/PrologEpilogInserter.cpp

static void assignCalleeSavedSpillSlots(MachineFunction &F,
   for (unsigned i = 0; CSRegs[i]; ++i) {
     unsigned Reg = CSRegs[i];
     if (SavedRegs.test(Reg))
       CSI.push_back(CalleeSavedInfo(Reg));
   }
 
   const TargetFrameLowering *TFI = F.getSubtarget().getFrameLowering();
   MachineFrameInfo &MFI = F.getFrameInfo();
-  if (!TFI->assignCalleeSavedSpillSlots(F, RegInfo, CSI)) {
+  if (!TFI->assignCalleeSavedSpillSlots(F, RegInfo, CSI, MinCSFrameIndex,
+                                        MaxCSFrameIndex)) {
     // If target doesn't implement this, use generic code.
 
     if (CSI.empty())
       return; // Early exit if no callee saved registers are modified!
 
     unsigned NumFixedSpillSlots;
     const TargetFrameLowering::SpillSlot *FixedSpillSlots =
         TFI->getCalleeSavedSpillSlots(NumFixedSpillSlots);
computeFreeStackSlots(MachineFrameInfo &MFI, bool StackGrowsDown,
   StackBytesFree.resize(FixedCSEnd, true);
 
   SmallVector<int, 16> AllocatedFrameSlots;
   // Add fixed objects.
   for (int i = MFI.getObjectIndexBegin(); i != 0; ++i)
     // StackSlot scavenging is only implemented for the default stack.
     if (MFI.getStackID(i) == TargetStackID::Default)
       AllocatedFrameSlots.push_back(i);
-  // Add callee-save objects.
+  // Add callee-save objects if there are any.
+  if (MinCSFrameIndex <= MaxCSFrameIndex) {
     for (int i = MinCSFrameIndex; i <= (int)MaxCSFrameIndex; ++i)
       if (MFI.getStackID(i) == TargetStackID::Default)
         AllocatedFrameSlots.push_back(i);
+  }
 
   for (int i : AllocatedFrameSlots) {
     // These are converted from int64_t, but they should always fit in int
     // because of the FixedCSEnd check above.
     int ObjOffset = MFI.getObjectOffset(i);
     int ObjSize = MFI.getObjectSize(i);
     int ObjStart, ObjEnd;
     if (StackGrowsDown) {
if (StackGrowsDown) {
       // address of the object.
       FixedOff = MFI.getObjectOffset(i) + MFI.getObjectSize(i);
     }
     if (FixedOff > Offset) Offset = FixedOff;
   }
 
   // First assign frame offsets to stack objects that are used to spill
   // callee saved registers.
-  if (StackGrowsDown) {
+  if (StackGrowsDown && MaxCSFrameIndex >= MinCSFrameIndex) {
     for (unsigned i = MinCSFrameIndex; i <= MaxCSFrameIndex; ++i) {
       if (MFI.getStackID(i) !=
           TargetStackID::Default) // Only allocate objects on the default stack.
         continue;
 
       // If the stack grows down, we need to add the size to find the lowest
       // address of the object.
       Offset += MFI.getObjectSize(i);

llvm/lib/CodeGen/SelectionDAG/FastISel.cpp

for (auto &Arg : CLI.getArgs()) {
     if (Arg.IsSExt)
       Flags.setSExt();
     if (Arg.IsInReg)
       Flags.setInReg();
     if (Arg.IsSRet)
       Flags.setSRet();
     if (Arg.IsSwiftSelf)
       Flags.setSwiftSelf();
+    if (Arg.IsSwiftAsync)
+      Flags.setSwiftAsync();
     if (Arg.IsSwiftError)
       Flags.setSwiftError();
     if (Arg.IsCFGuardTarget)
       Flags.setCFGuardTarget();
     if (Arg.IsByVal)
       Flags.setByVal();
     if (Arg.IsInAlloca) {
       Flags.setInAlloca();

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

if (!CanLowerReturn) {
     Entry.IsZExt = false;
     Entry.IsInReg = false;
     Entry.IsSRet = true;
     Entry.IsNest = false;
     Entry.IsByVal = false;
     Entry.IsByRef = false;
     Entry.IsReturned = false;
     Entry.IsSwiftSelf = false;
+    Entry.IsSwiftAsync = false;
     Entry.IsSwiftError = false;
     Entry.IsCFGuardTarget = false;
     Entry.Alignment = Alignment;
     CLI.getArgs().insert(CLI.getArgs().begin(), Entry);
     CLI.NumFixedArgs += 1;
     CLI.RetTy = Type::getVoidTy(CLI.RetTy->getContext());
 
     // sret demotion isn't compatible with tail-calls, since the sret argument
for (unsigned Value = 0, NumValues = ValueVTs.size(); Value != NumValues;
         }
         // Set InReg Flag
         Flags.setInReg();
       }
       if (Args[i].IsSRet)
         Flags.setSRet();
       if (Args[i].IsSwiftSelf)
         Flags.setSwiftSelf();
+      if (Args[i].IsSwiftAsync)
+        Flags.setSwiftAsync();
       if (Args[i].IsSwiftError)
         Flags.setSwiftError();
       if (Args[i].IsCFGuardTarget)
         Flags.setCFGuardTarget();
       if (Args[i].IsByVal)
         Flags.setByVal();
       if (Args[i].IsByRef)
         Flags.setByRef();
for (unsigned Value = 0, NumValues = ValueVTs.size();
         }
         // Set InReg Flag
         Flags.setInReg();
       }
       if (Arg.hasAttribute(Attribute::StructRet))
         Flags.setSRet();
       if (Arg.hasAttribute(Attribute::SwiftSelf))
         Flags.setSwiftSelf();
+      if (Arg.hasAttribute(Attribute::SwiftAsync))
+        Flags.setSwiftAsync();
       if (Arg.hasAttribute(Attribute::SwiftError))
         Flags.setSwiftError();
       if (Arg.hasAttribute(Attribute::ByVal))
         Flags.setByVal();
       if (Arg.hasAttribute(Attribute::ByRef))
         Flags.setByRef();
       if (Arg.hasAttribute(Attribute::InAlloca)) {
         Flags.setInAlloca();

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

void TargetLoweringBase::ArgListEntry::setAttributes(const CallBase *Call,
 
   IsSExt = Attrs.hasParamAttribute(ArgIdx, Attribute::SExt);
   IsZExt = Attrs.hasParamAttribute(ArgIdx, Attribute::ZExt);
   IsInReg = Attrs.hasParamAttribute(ArgIdx, Attribute::InReg);
   IsSRet = Attrs.hasParamAttribute(ArgIdx, Attribute::StructRet);
   IsNest = Attrs.hasParamAttribute(ArgIdx, Attribute::Nest);
   IsReturned = Attrs.hasParamAttribute(ArgIdx, Attribute::Returned);
   IsSwiftSelf = Attrs.hasParamAttribute(ArgIdx, Attribute::SwiftSelf);
+  IsSwiftAsync = Attrs.hasParamAttr(ArgIdx, Attribute::SwiftAsync);
   IsSwiftError = Attrs.hasParamAttribute(ArgIdx, Attribute::SwiftError);
   Alignment = Attrs.getParamStackAlignment(ArgIdx);
 
   IsByVal = Attrs.hasParamAttribute(ArgIdx, Attribute::ByVal);
   IsInAlloca = Attrs.hasParamAttribute(ArgIdx, Attribute::InAlloca);
   IsPreallocated = Attrs.hasParamAttribute(ArgIdx, Attribute::Preallocated);
 
   assert(IsByVal + IsInAlloca + IsPreallocated <= 1 &&

llvm/lib/IR/Attributes.cpp

std::string Attribute::getAsString(bool InAttrGrp) const {
   if (hasAttribute(Attribute::Builtin))
     return "builtin";
   if (hasAttribute(Attribute::Convergent))
     return "convergent";
   if (hasAttribute(Attribute::SwiftError))
     return "swifterror";
   if (hasAttribute(Attribute::SwiftSelf))
     return "swiftself";
+  if (hasAttribute(Attribute::SwiftAsync))
+    return "swiftasync";
   if (hasAttribute(Attribute::InaccessibleMemOnly))
     return "inaccessiblememonly";
   if (hasAttribute(Attribute::InaccessibleMemOrArgMemOnly))
     return "inaccessiblemem_or_argmemonly";
   if (hasAttribute(Attribute::InlineHint))
     return "inlinehint";
   if (hasAttribute(Attribute::InReg))
     return "inreg";

llvm/lib/IR/Verifier.cpp

for (const auto &AttrSet : Attrs) {
       }
     }
   }
 
   bool SawNest = false;
   bool SawReturned = false;
   bool SawSRet = false;
   bool SawSwiftSelf = false;
+  bool SawSwiftAsync = false;
   bool SawSwiftError = false;
 
   // Verify return value attributes.
   AttributeSet RetAttrs = Attrs.getRetAttributes();
   Assert((!RetAttrs.hasAttribute(Attribute::ByVal) &&
           !RetAttrs.hasAttribute(Attribute::Nest) &&
           !RetAttrs.hasAttribute(Attribute::StructRet) &&
           !RetAttrs.hasAttribute(Attribute::NoCapture) &&
           !RetAttrs.hasAttribute(Attribute::NoFree) &&
           !RetAttrs.hasAttribute(Attribute::Returned) &&
           !RetAttrs.hasAttribute(Attribute::InAlloca) &&
           !RetAttrs.hasAttribute(Attribute::Preallocated) &&
           !RetAttrs.hasAttribute(Attribute::ByRef) &&
           !RetAttrs.hasAttribute(Attribute::SwiftSelf) &&
+          !RetAttrs.hasAttribute(Attribute::SwiftAsync) &&
           !RetAttrs.hasAttribute(Attribute::SwiftError)),
          "Attributes 'byval', 'inalloca', 'preallocated', 'byref', "
          "'nest', 'sret', 'nocapture', 'nofree', "
-         "'returned', 'swiftself', and 'swifterror' do not apply to return "
-         "values!",
+         "'returned', 'swiftself', 'swiftasync', and 'swifterror'"
+         " do not apply to return values!",
          V);
   Assert((!RetAttrs.hasAttribute(Attribute::ReadOnly) &&
           !RetAttrs.hasAttribute(Attribute::WriteOnly) &&
           !RetAttrs.hasAttribute(Attribute::ReadNone)),
          "Attribute '" + RetAttrs.getAsString() +
              "' does not apply to function returns",
          V);
   verifyParameterAttrs(RetAttrs, FT->getReturnType(), V);
if (ArgAttrs.hasAttribute(Attribute::StructRet)) {
       SawSRet = true;
     }
 
     if (ArgAttrs.hasAttribute(Attribute::SwiftSelf)) {
       Assert(!SawSwiftSelf, "Cannot have multiple 'swiftself' parameters!", V);
       SawSwiftSelf = true;
     }
 
+    if (ArgAttrs.hasAttribute(Attribute::SwiftAsync)) {
+      Assert(!SawSwiftAsync, "Cannot have multiple 'swiftasync' parameters!", V);
+      SawSwiftAsync = true;
+    }
+
     if (ArgAttrs.hasAttribute(Attribute::SwiftError)) {
       Assert(!SawSwiftError, "Cannot have multiple 'swifterror' parameters!",
              V);
       SawSwiftError = true;
     }
 
     if (ArgAttrs.hasAttribute(Attribute::InAlloca)) {
       Assert(i == FT->getNumParams() - 1,
static bool isTypeCongruent(Type *L, Type *R) {
   PointerType *PR = dyn_cast<PointerType>(R);
   if (!PL || !PR)
     return false;
   return PL->getAddressSpace() == PR->getAddressSpace();
 }
 
 static AttrBuilder getParameterABIAttributes(int I, AttributeList Attrs) {
   static const Attribute::AttrKind ABIAttrs[] = {
-      Attribute::StructRet,    Attribute::ByVal,     Attribute::InAlloca,
-      Attribute::InReg,        Attribute::SwiftSelf, Attribute::SwiftError,
-      Attribute::Preallocated, Attribute::ByRef,     Attribute::StackAlignment};
+      Attribute::StructRet,  Attribute::ByVal,      Attribute::InAlloca,
+      Attribute::InReg,      Attribute::SwiftSelf,  Attribute::SwiftAsync,
+      Attribute::SwiftError, Attribute::SwiftAsync, Attribute::Preallocated,
+      Attribute::ByRef};
   AttrBuilder Copy;
   for (auto AK : ABIAttrs) {
     if (Attrs.hasParamAttribute(I, AK))
       Copy.addAttribute(AK);
   }
 
   // `align` is ABI-affecting only in combination with `byval` or `byref`.
   if (Attrs.hasParamAttribute(I, Attribute::Alignment) &&

llvm/lib/Target/AArch64/AArch64CallingConvention.td

def CC_AArch64_AAPCS : CallingConv<[
   CCIfNest<CCAssignToReg<[X18]>>,
 
   // Pass SwiftSelf in a callee saved register.
   CCIfSwiftSelf<CCIfType<[i64], CCAssignToRegWithShadow<[X20], [W20]>>>,
 
   // A SwiftError is passed in X21.
   CCIfSwiftError<CCIfType<[i64], CCAssignToRegWithShadow<[X21], [W21]>>>,
 
+  // Pass SwiftAsync in an otherwise callee saved register so that it will be
+  // preserved for normal function calls.
+  CCIfSwiftAsync<CCIfType<[i64], CCAssignToRegWithShadow<[X22], [W22]>>>,
+
   CCIfConsecutiveRegs<CCCustom<"CC_AArch64_Custom_Block">>,
 
   CCIfType<[nxv16i8, nxv8i16, nxv4i32, nxv2i64, nxv2f16, nxv4f16, nxv8f16,
             nxv2bf16, nxv4bf16, nxv8bf16, nxv2f32, nxv4f32, nxv2f64],
            CCAssignToReg<[Z0, Z1, Z2, Z3, Z4, Z5, Z6, Z7]>>,
   CCIfType<[nxv16i8, nxv8i16, nxv4i32, nxv2i64, nxv2f16, nxv4f16, nxv8f16,
             nxv2bf16, nxv4bf16, nxv8bf16, nxv2f32, nxv4f32, nxv2f64],
            CCPassIndirect<i64>>,
def CC_AArch64_DarwinPCS : CallingConv<[
   CCIfByVal<CCPassByVal<8, 8>>,
 
   // Pass SwiftSelf in a callee saved register.
   CCIfSwiftSelf<CCIfType<[i64], CCAssignToRegWithShadow<[X20], [W20]>>>,
 
   // A SwiftError is passed in X21.
   CCIfSwiftError<CCIfType<[i64], CCAssignToRegWithShadow<[X21], [W21]>>>,
 
+  // Pass SwiftAsync in an otherwise callee saved register so that it will be
+  // preserved for normal function calls.
+  CCIfSwiftAsync<CCIfType<[i64], CCAssignToRegWithShadow<[X22], [W22]>>>,
+
   CCIfConsecutiveRegs<CCCustom<"CC_AArch64_Custom_Block">>,
 
   // Handle i1, i8, i16, i32, i64, f32, f64 and v2f64 by passing in registers,
   // up to eight each of GPR and FPR.
   CCIfType<[i1, i8, i16], CCPromoteToType<i32>>,
   CCIfType<[i32], CCAssignToRegWithShadow<[W0, W1, W2, W3, W4, W5, W6, W7],
                                           [X0, X1, X2, X3, X4, X5, X6, X7]>>,
   // i128 is split to two i64s, we can't fit half to register X7.

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

private:
   bool expandSetTagLoop(MachineBasicBlock &MBB,
                         MachineBasicBlock::iterator MBBI,
                         MachineBasicBlock::iterator &NextMBBI);
   bool expandSVESpillFill(MachineBasicBlock &MBB,
                           MachineBasicBlock::iterator MBBI, unsigned Opc,
                           unsigned N);
   bool expandCALL_RVMARKER(MachineBasicBlock &MBB,
                            MachineBasicBlock::iterator MBBI);
+  bool expandStoreSwiftAsyncContext(MachineBasicBlock &MBB,
+                                    MachineBasicBlock::iterator MBBI);
 };
 
 } // end anonymous namespace
 
 char AArch64ExpandPseudo::ID = 0;
 
 INITIALIZE_PASS(AArch64ExpandPseudo, "aarch64-expand-pseudo",
                 AARCH64_EXPAND_PSEUDO_NAME, false, false)
bool AArch64ExpandPseudo::expandCALL_RVMARKER(
   if (MI.shouldUpdateCallSiteInfo())
     MBB.getParent()->moveCallSiteInfo(&MI, Marker);
   MI.eraseFromParent();
   finalizeBundle(MBB, OriginalCall->getIterator(),
                  std::next(Marker->getIterator()));
   return true;
 }
 
+bool AArch64ExpandPseudo::expandStoreSwiftAsyncContext(
+    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI) {
+  Register CtxReg = MBBI->getOperand(0).getReg();
+  Register BaseReg = MBBI->getOperand(1).getReg();
+  int Offset = MBBI->getOperand(2).getImm();
+  DebugLoc DL(MBBI->getDebugLoc());
+  auto &STI = MBB.getParent()->getSubtarget<AArch64Subtarget>();
+
+  if (STI.getTargetTriple().getArchName() != "arm64e") {
+    BuildMI(MBB, MBBI, DL, TII->get(AArch64::STRXui))
+        .addUse(CtxReg)
+        .addUse(BaseReg)
+        .addImm(Offset / 8)
+        .setMIFlag(MachineInstr::FrameSetup);
+    MBBI->eraseFromParent();
+    return true;
+  }
+
+  // We need to sign the context in an address-discriminated way. 0xc31a is a
+  // fixed random value, chosen as part of the ABI.
+  //     add x16, xBase, #Offset
+  //     movk x16, #0xc31a, lsl #48
+  //     mov x17, x22/xzr
+  //     pacdb x17, x16
+  //     str x17, [xBase, #Offset]
+  unsigned Opc = Offset >= 0 ? AArch64::ADDXri : AArch64::SUBXri;
+  BuildMI(MBB, MBBI, DL, TII->get(Opc), AArch64::X16)
+      .addUse(BaseReg)
+      .addImm(abs(Offset))
+      .addImm(0)
+      .setMIFlag(MachineInstr::FrameSetup);
+  BuildMI(MBB, MBBI, DL, TII->get(AArch64::MOVKXi), AArch64::X16)
+      .addUse(AArch64::X16)
+      .addImm(0xc31a)
+      .addImm(48)
+      .setMIFlag(MachineInstr::FrameSetup);
+  // We're not allowed to clobber X22 (and couldn't clobber XZR if we tried), so
+  // move it somewhere before signing.
+  BuildMI(MBB, MBBI, DL, TII->get(AArch64::ORRXrs), AArch64::X17)
+      .addUse(AArch64::XZR)
+      .addUse(CtxReg)
+      .addImm(0)
+      .setMIFlag(MachineInstr::FrameSetup);
+  BuildMI(MBB, MBBI, DL, TII->get(AArch64::PACDB), AArch64::X17)
+      .addUse(AArch64::X17)
+      .addUse(AArch64::X16)
+      .setMIFlag(MachineInstr::FrameSetup);
+  BuildMI(MBB, MBBI, DL, TII->get(AArch64::STRXui))
+      .addUse(AArch64::X17)
+      .addUse(BaseReg)
+      .addImm(Offset / 8)
+      .setMIFlag(MachineInstr::FrameSetup);
+
+  MBBI->eraseFromParent();
+  return true;
+}
+
 /// If MBBI references a pseudo instruction that should be expanded here,
 /// do the expansion and return true.  Otherwise return false.
 bool AArch64ExpandPseudo::expandMI(MachineBasicBlock &MBB,
                                    MachineBasicBlock::iterator MBBI,
                                    MachineBasicBlock::iterator &NextMBBI) {
   MachineInstr &MI = *MBBI;
   unsigned Opcode = MI.getOpcode();
 
bool AArch64ExpandPseudo::expandMI(MachineBasicBlock &MBB,
    case AArch64::LDR_ZZZZXI:
      return expandSVESpillFill(MBB, MBBI, AArch64::LDR_ZXI, 4);
    case AArch64::LDR_ZZZXI:
      return expandSVESpillFill(MBB, MBBI, AArch64::LDR_ZXI, 3);
    case AArch64::LDR_ZZXI:
      return expandSVESpillFill(MBB, MBBI, AArch64::LDR_ZXI, 2);
    case AArch64::BLR_RVMARKER:
      return expandCALL_RVMARKER(MBB, MBBI);
+   case AArch64::StoreSwiftAsyncContext:
+     return expandStoreSwiftAsyncContext(MBB, MBBI);
   }
   return false;
 }
 
 /// Iterate over the instructions in basic block MBB and expand any
 /// pseudo instructions.  Return true if anything was modified.
 bool AArch64ExpandPseudo::expandMBB(MachineBasicBlock &MBB) {
   bool Modified = false;

llvm/lib/Target/AArch64/AArch64FastISel.cpp

bool AArch64FastISel::fastLowerArguments() {
   // Only handle simple cases of up to 8 GPR and FPR each.
   unsigned GPRCnt = 0;
   unsigned FPRCnt = 0;
   for (auto const &Arg : F->args()) {
     if (Arg.hasAttribute(Attribute::ByVal) ||
         Arg.hasAttribute(Attribute::InReg) ||
         Arg.hasAttribute(Attribute::StructRet) ||
         Arg.hasAttribute(Attribute::SwiftSelf) ||
+        Arg.hasAttribute(Attribute::SwiftAsync) ||
         Arg.hasAttribute(Attribute::SwiftError) ||
         Arg.hasAttribute(Attribute::Nest))
       return false;
 
     Type *ArgTy = Arg.getType();
     if (ArgTy->isStructTy() || ArgTy->isArrayTy())
       return false;
 
bool AArch64FastISel::fastLowerCall(CallLoweringInfo &CLI) {
   MVT RetVT;
   if (CLI.RetTy->isVoidTy())
     RetVT = MVT::isVoid;
   else if (!isTypeLegal(CLI.RetTy, RetVT))
     return false;
 
   for (auto Flag : CLI.OutFlags)
     if (Flag.isInReg() || Flag.isSRet() || Flag.isNest() || Flag.isByVal() ||
-        Flag.isSwiftSelf() || Flag.isSwiftError())
+        Flag.isSwiftSelf() || Flag.isSwiftAsync() || Flag.isSwiftError())
       return false;
 
   // Set up the argument vectors.
   SmallVector<MVT, 16> OutVTs;
   OutVTs.reserve(CLI.OutVals.size());
 
   for (auto *Val : CLI.OutVals) {
     MVT VT;

llvm/lib/Target/AArch64/AArch64FrameLowering.h

restoreCalleeSavedRegisters(MachineBasicBlock &MBB,
                               const TargetRegisterInfo *TRI) const override;
 
   /// Can this function use the red zone for local allocations.
   bool canUseRedZone(const MachineFunction &MF) const;
 
   bool hasFP(const MachineFunction &MF) const override;
   bool hasReservedCallFrame(const MachineFunction &MF) const override;
 
-  bool
-  assignCalleeSavedSpillSlots(MachineFunction &MF,
+  bool hasSwiftExtendedFrame(const MachineFunction &MF) const;
+
+  bool assignCalleeSavedSpillSlots(MachineFunction &MF,
                                    const TargetRegisterInfo *TRI,
-                              std::vector<CalleeSavedInfo> &CSI) const override;
+                                   std::vector<CalleeSavedInfo> &CSI,
+                                   unsigned &MinCSFrameIndex,
+                                   unsigned &MaxCSFrameIndex) const override;
 
   void determineCalleeSaves(MachineFunction &MF, BitVector &SavedRegs,
                             RegScavenger *RS) const override;
 
   /// Returns true if the target will correctly handle shrink wrapping.
   bool enableShrinkWrapping(const MachineFunction &MF) const override {
     return true;
   }

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

 // |-----------------------------------|
 // |                                   |
 // | (Win64 only) varargs from reg     |
 // |                                   |
 // |-----------------------------------|
 // |                                   |
 // | callee-saved gpr registers        | <--.
 // |                                   |    | On Darwin platforms these
 // |- - - - - - - - - - - - - - - - - -|    | callee saves are swapped,
-// |                                   |    | (frame record first)
-// | prev_fp, prev_lr                  | <--'
+// | prev_lr                           |    | (frame record first)
+// | prev_fp                           | <--'
+// | async context if needed           |
 // | (a.k.a. "frame record")           |
 // |-----------------------------------| <- fp(=x29)
 // |                                   |
 // | callee-saved fp/simd/SVE regs     |
 // |                                   |
 // |-----------------------------------|
 // |                                   |
 // |        SVE stack objects          |
 // |                                   |
 // |-----------------------------------|
static MachineBasicBlock::iterator convertCalleeSaveRestoreToSPPrePostIncDec(
   }
   // Get rid of the SEH code associated with the old instruction.
   if (NeedsWinCFI) {
     auto SEH = std::next(MBBI);
     if (AArch64InstrInfo::isSEHInstruction(*SEH))
       SEH->eraseFromParent();
   }
 
+  // If the first store isn't right where we want SP then we can't fold the
+  // update in so create a normal arithmetic instruction instead.
+  if (MBBI->getOperand(MBBI->getNumOperands() - 1).getImm() != 0) {
+    emitFrameOffset(MBB, MBBI, DL, AArch64::SP, AArch64::SP,
+                    StackOffset::getFixed(CSStackSizeInc), TII,
+                    InProlog ? MachineInstr::FrameSetup
+                             : MachineInstr::FrameDestroy);
+    return std::prev(MBBI);
+  }
+
   MachineInstrBuilder MIB = BuildMI(MBB, MBBI, DL, TII->get(NewOpc));
   MIB.addReg(AArch64::SP, RegState::Define);
 
   // Copy all operands other than the immediate offset.
   unsigned OpndIdx = 0;
   for (unsigned OpndEnd = MBBI->getNumOperands() - 1; OpndIdx < OpndEnd;
        ++OpndIdx)
     MIB.add(MBBI->getOperand(OpndIdx));
if (MFnI.shouldSignReturnAddress()) {
 
     unsigned CFIIndex =
         MF.addFrameInst(MCCFIInstruction::createNegateRAState(nullptr));
     BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
         .addCFIIndex(CFIIndex)
         .setMIFlags(MachineInstr::FrameSetup);
   }
 
+  // We signal the presence of a Swift extended frame to external tools by
+  // storing FP with 0b0001 in bits 63:60. In normal userland operation a simple
+  // ORR is sufficient, it is assumed a Swift kernel would initialize the TBI
+  // bits so that is still true.
+  if (HasFP && AFI->hasSwiftAsyncContext()) {
+    // ORR x29, x29, #0x1000_0000_0000_0000
+    BuildMI(MBB, MBBI, DL, TII->get(AArch64::ORRXri), AArch64::FP)
+        .addUse(AArch64::FP)
+        .addImm(0x1100)
+        .setMIFlag(MachineInstr::FrameSetup);
+  }
+
   // All calls are tail calls in GHC calling conv, and functions have no
   // prologue/epilogue.
   if (MF.getFunction().getCallingConv() == CallingConv::GHC)
     return;
 
   // Set tagged base pointer to the requested stack slot.
   // Ideally it should match SP value after prologue.
   Optional<int> TBPI = AFI->getTaggedBasePointerIndex();
void AArch64FrameLowering::emitPrologue(MachineFunction &MF,
   // For funclets the FP belongs to the containing function.
   if (!IsFunclet && HasFP) {
     // Only set up FP if we actually need to.
     int64_t FPOffset = AFI->getCalleeSaveBaseToFrameRecordOffset();
 
     if (CombineSPBump)
       FPOffset += AFI->getLocalStackSize();
 
+    if (AFI->hasSwiftAsyncContext()) {
+      // Before we update the live FP we have to ensure there's a valid (or
+      // null) asynchronous context in its slot just before FP in the frame
+      // record, so store it now.
+      const auto &Attrs = MF.getFunction().getAttributes();
+      bool HaveInitialContext = Attrs.hasAttrSomewhere(Attribute::SwiftAsync);
+
+      BuildMI(MBB, MBBI, DL, TII->get(AArch64::StoreSwiftAsyncContext))
+          .addUse(HaveInitialContext ? AArch64::X22 : AArch64::XZR)
+          .addUse(AArch64::SP)
+          .addImm(FPOffset - 8)
+          .setMIFlags(MachineInstr::FrameSetup);
+    }
+
     if (HomPrologEpilog) {
       auto Prolog = MBBI;
       --Prolog;
       assert(Prolog->getOpcode() == AArch64::HOM_Prolog);
       Prolog->addOperand(MachineOperand::CreateImm(FPOffset));
     } else {
       // Issue    sub fp, sp, FPOffset or
       //          mov fp,sp          when FPOffset is zero.
if (MF.hasWinCFI()) {
     // (And as we didn't generate any prologue at all, it would be asymmetrical
     // to the epilogue.) By the end of the function, we assert that
     // HasWinCFI is equal to MF.hasWinCFI(), to verify this assumption.
     HasWinCFI = true;
     BuildMI(MBB, LastPopI, DL, TII->get(AArch64::SEH_EpilogStart))
         .setMIFlag(MachineInstr::FrameDestroy);
   }
 
+  if (hasFP(MF) && AFI->hasSwiftAsyncContext()) {
+    // We need to reset FP to its untagged state on return. Bit 60 is currently
+    // used to show the presence of an extended frame.
+
+    // BIC x29, x29, #0x1000_0000_0000_0000
+    BuildMI(MBB, MBB.getFirstTerminator(), DL, TII->get(AArch64::ANDXri),
+            AArch64::FP)
+        .addUse(AArch64::FP)
+        .addImm(0x10fe)
+        .setMIFlag(MachineInstr::FrameDestroy);
+  }
+
   const StackOffset &SVEStackSize = getSVEStackSize(MF);
 
   // If there is a single SP update, insert it before the ret and we're done.
   if (CombineSPBump) {
     assert(!SVEStackSize && "Cannot combine SP bump with SVE");
     emitFrameOffset(MBB, MBB.getFirstTerminator(), DL, AArch64::SP, AArch64::SP,
                     StackOffset::getFixed(NumBytes + (int64_t)AfterCSRPopSize),
                     TII, MachineInstr::FrameDestroy, false, NeedsWinCFI,
for (unsigned i = FirstReg; i < Count; i += RegInc) {
     int OffsetPre = RPI.isScalable() ? ScalableByteOffset : ByteOffset;
     assert(OffsetPre % Scale == 0);
 
     if (RPI.isScalable())
       ScalableByteOffset += StackFillDir * Scale;
     else
       ByteOffset += StackFillDir * (RPI.isPaired() ? 2 * Scale : Scale);
 
+    // Swift's async context is directly before FP, so allocate an extra
+    // 8 bytes for it.
+    if (NeedsFrameRecord && AFI->hasSwiftAsyncContext() &&
+        RPI.Reg2 == AArch64::FP)
+      ByteOffset += StackFillDir * 8;
+
     assert(!(RPI.isScalable() && RPI.isPaired()) &&
            "Paired spill/fill instructions don't exist for SVE vectors");
 
     // Round up size of non-pair to pair size if we need to pad the
     // callee-save area to ensure 16-byte alignment.
     if (AFI->hasCalleeSaveStackFreeSpace() && !NeedsWinCFI &&
         !RPI.isScalable() && RPI.Type != RegPairInfo::FPR128 &&
         !RPI.isPaired()) {
       ByteOffset += 8 * StackFillDir;
       assert(ByteOffset % 16 == 0);
       assert(MFI.getObjectAlign(RPI.FrameIdx) <= Align(16));
       // A stack frame with a gap looks like this, bottom up:
       // d9, d8. x21, gap, x20, x19.
       // Set extra alignment on the x21 object (the only unpaired register)
       // to create the gap above it.
       MFI.setObjectAlignment(RPI.FrameIdx, Align(16));
     }
 
     int OffsetPost = RPI.isScalable() ? ScalableByteOffset : ByteOffset;
     assert(OffsetPost % Scale == 0);
     // If filling top down (default), we want the offset after incrementing it.
     // If fillibg bootom up (WinCFI) we need the original offset.
     int Offset = NeedsWinCFI ? OffsetPre : OffsetPost;
+
+    // The FP, LR pair goes 8 bytes into our expanded 24-byte slot so that the
+    // Swift context can directly precede FP.
+    if (NeedsFrameRecord && AFI->hasSwiftAsyncContext() &&
+        RPI.Reg2 == AArch64::FP)
+      Offset += 8;
     RPI.Offset = Offset / Scale;
 
     assert(((!RPI.isScalable() && RPI.Offset >= -64 && RPI.Offset <= 63) ||
             (RPI.isScalable() && RPI.Offset >= -256 && RPI.Offset <= 255)) &&
            "Offset out of bounds for LDP/STP immediate");
 
     // Save the offset to frame record so that the FP register can point to the
     // innermost frame record (spilled FP and LR registers).
if (!ExtraCSSpill || MF.getRegInfo().isPhysRegUsed(ExtraCSSpill)) {
       RS->addScavengingFrameIndex(FI);
       LLVM_DEBUG(dbgs() << "No available CS registers, allocated fi#" << FI
                         << " as the emergency spill slot.\n");
     }
   }
 
   // Adding the size of additional 64bit GPR saves.
   CSStackSize += 8 * (SavedRegs.count() - NumSavedRegs);
+
+  // A Swift asynchronous context extends the frame record with a pointer
+  // directly before FP.
+  if (hasFP(MF) && AFI->hasSwiftAsyncContext())
+    CSStackSize += 8;
+
   uint64_t AlignedCSStackSize = alignTo(CSStackSize, 16);
   LLVM_DEBUG(dbgs() << "Estimated stack frame size: "
                << EstimatedStackSize + AlignedCSStackSize
                << " bytes.\n");
 
   assert((!MFI.isCalleeSavedInfoValid() ||
           AFI->getCalleeSavedStackSize() == AlignedCSStackSize) &&
          "Should not invalidate callee saved info");
 
   // Round up to register pair alignment to avoid additional SP adjustment
   // instructions.
   AFI->setCalleeSavedStackSize(AlignedCSStackSize);
   AFI->setCalleeSaveStackHasFreeSpace(AlignedCSStackSize != CSStackSize);
   AFI->setSVECalleeSavedStackSize(alignTo(SVECSStackSize, 16));
 }
 
 bool AArch64FrameLowering::assignCalleeSavedSpillSlots(
-    MachineFunction &MF, const TargetRegisterInfo *TRI,
-    std::vector<CalleeSavedInfo> &CSI) const {
+    MachineFunction &MF, const TargetRegisterInfo *RegInfo,
+    std::vector<CalleeSavedInfo> &CSI, unsigned &MinCSFrameIndex,
+    unsigned &MaxCSFrameIndex) const {
   bool NeedsWinCFI = needsWinCFI(MF);
   // To match the canonical windows frame layout, reverse the list of
   // callee saved registers to get them laid out by PrologEpilogInserter
   // in the right order. (PrologEpilogInserter allocates stack objects top
   // down. Windows canonical prologs store higher numbered registers at
   // the top, thus have the CSI array start from the highest registers.)
   if (NeedsWinCFI)
     std::reverse(CSI.begin(), CSI.end());
-  // Let the generic code do the rest of the setup.
-  return false;
+
+  if (CSI.empty())
+    return true; // Early exit if no callee saved registers are modified!
+
+  // Now that we know which registers need to be saved and restored, allocate
+  // stack slots for them.
+  MachineFrameInfo &MFI = MF.getFrameInfo();
+  auto *AFI = MF.getInfo<AArch64FunctionInfo>();
+  for (auto &CS : CSI) {
+    Register Reg = CS.getReg();
+    const TargetRegisterClass *RC = RegInfo->getMinimalPhysRegClass(Reg);
+
+    int FrameIdx;
+    unsigned Size = RegInfo->getSpillSize(*RC);
+
+    Align Alignment(RegInfo->getSpillAlign(*RC));
+    FrameIdx = MFI.CreateStackObject(Size, Alignment, true);
+    CS.setFrameIdx(FrameIdx);
+
+    if ((unsigned)FrameIdx < MinCSFrameIndex) MinCSFrameIndex = FrameIdx;
+    if ((unsigned)FrameIdx > MaxCSFrameIndex) MaxCSFrameIndex = FrameIdx;
+
+    // Grab 8 bytes below FP for the extended asynchronous frame info.
+    if (hasFP(MF) && AFI->hasSwiftAsyncContext() && Reg == AArch64::FP) {
+      FrameIdx = MFI.CreateStackObject(8, Alignment, true);
+      AFI->setSwiftAsyncContextFrameIdx(FrameIdx);
+      if ((unsigned)FrameIdx < MinCSFrameIndex) MinCSFrameIndex = FrameIdx;
+      if ((unsigned)FrameIdx > MaxCSFrameIndex) MaxCSFrameIndex = FrameIdx;
+    }
+  }
+  return true;
 }
 
 bool AArch64FrameLowering::enableStackSlotScavenging(
     const MachineFunction &MF) const {
   const AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   return AFI->hasCalleeSaveStackFreeSpace();
 }
 

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

case Intrinsic::aarch64_neon_tbx4:
                                            : AArch64::TBXv16i8Four,
                   true);
       return;
     case Intrinsic::aarch64_neon_smull:
     case Intrinsic::aarch64_neon_umull:
       if (tryMULLV64LaneV128(IntNo, Node))
         return;
       break;
+    case Intrinsic::swift_async_context_addr: {
+      SDLoc DL(Node);
+      CurDAG->SelectNodeTo(Node, AArch64::SUBXri, MVT::i64,
+                           CurDAG->getCopyFromReg(CurDAG->getEntryNode(), DL,
+                                                  AArch64::FP, MVT::i64),
+                           CurDAG->getTargetConstant(8, DL, MVT::i32),
+                           CurDAG->getTargetConstant(0, DL, MVT::i32));
+      auto &MF = CurDAG->getMachineFunction();
+      MF.getFrameInfo().setFrameAddressIsTaken(true);
+      MF.getInfo<AArch64FunctionInfo>()->setHasSwiftAsyncContext(true);
+      return;
+    }
     }
     break;
   }
   case ISD::INTRINSIC_VOID: {
     unsigned IntNo = cast<ConstantSDNode>(Node->getOperand(1))->getZExtValue();
     if (Node->getNumOperands() >= 3)
       VT = Node->getOperand(2)->getValueType(0);
     switch (IntNo) {

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (Ins[i].Flags.isByVal()) {
       unsigned FrameIdx =
         MFI.CreateFixedObject(8 * NumRegs, VA.getLocMemOffset(), false);
       SDValue FrameIdxN = DAG.getFrameIndex(FrameIdx, PtrVT);
       InVals.push_back(FrameIdxN);
 
       continue;
     }
 
+    if (Ins[i].Flags.isSwiftAsync())
+      MF.getInfo<AArch64FunctionInfo>()->setHasSwiftAsyncContext(true);
+
     SDValue ArgValue;
     if (VA.isRegLoc()) {
       // Arguments stored in registers.
       EVT RegVT = VA.getLocVT();
       const TargetRegisterClass *RC;
 
       if (RegVT == MVT::i32)
         RC = &AArch64::GPR32RegClass;

llvm/lib/Target/AArch64/AArch64InstrFormats.td

 }
 
 class OneXRegData<bits<3> opc, string asm, SDPatternOperator node>
     : BaseOneOperandData<opc, GPR64, asm, node> {
   let Inst{31} = 1;
 }
 
 class SignAuthOneData<bits<3> opcode_prefix, bits<2> opcode, string asm>
-  : I<(outs GPR64:$Rd), (ins GPR64sp:$Rn), asm, "\t$Rd, $Rn", "",
+  : I<(outs GPR64:$Rd), (ins GPR64:$src, GPR64sp:$Rn), asm, "\t$Rd, $Rn",
+      "$Rd = $src",
       []>,
     Sched<[WriteI, ReadI]> {
   bits<5> Rd;
   bits<5> Rn;
   let Inst{31-15} = 0b11011010110000010;
   let Inst{14-12} = opcode_prefix;
   let Inst{11-10} = opcode;
   let Inst{9-5} = Rn;
   let Inst{4-0} = Rd;
 }
 
 class SignAuthZero<bits<3> opcode_prefix, bits<2> opcode, string asm>
-  : I<(outs GPR64:$Rd), (ins), asm, "\t$Rd", "", []>, Sched<[]> {
+  : I<(outs GPR64:$Rd), (ins GPR64:$src), asm, "\t$Rd", "$Rd = $src",
+      []>, Sched<[]> {
   bits<5> Rd;
   let Inst{31-15} = 0b11011010110000010;
   let Inst{14-12} = opcode_prefix;
   let Inst{11-10} = opcode;
   let Inst{9-5} = 0b11111;
   let Inst{4-0} = Rd;
 }
 

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

unsigned AArch64InstrInfo::getInstSizeInBytes(const MachineInstr &MI) const {
   case AArch64::JumpTableDest32:
   case AArch64::JumpTableDest16:
   case AArch64::JumpTableDest8:
     NumBytes = 12;
     break;
   case AArch64::SPACE:
     NumBytes = MI.getOperand(1).getImm();
     break;
+  case AArch64::StoreSwiftAsyncContext:
+    NumBytes = 20;
+    break;
   case TargetOpcode::BUNDLE:
     NumBytes = getInstBundleLength(MI);
     break;
   }
 
   return NumBytes;
 }
 
bool AArch64InstrInfo::getMemOpInfo(unsigned Opcode, TypeSize &Scale,
   case AArch64::LDRDui:
   case AArch64::STRXui:
   case AArch64::STRDui:
     Scale = TypeSize::Fixed(8);
     Width = 8;
     MinOffset = 0;
     MaxOffset = 4095;
     break;
+  case AArch64::StoreSwiftAsyncContext:
+    // Store is an STRXui, but there might be an ADDXri in the expansion too.
+    Scale = TypeSize::Fixed(1);
+    Width = 8;
+    MinOffset = 0;
+    MaxOffset = 4095;
+    break;
   case AArch64::LDPWi:
   case AArch64::LDPSi:
   case AArch64::LDNPWi:
   case AArch64::LDNPSi:
   case AArch64::STPWi:
   case AArch64::STPSi:
   case AArch64::STNPWi:
   case AArch64::STNPSi:

llvm/lib/Target/AArch64/AArch64InstrInfo.td

class ST64BPattern<Intrinsic intrinsic, Instruction instruction>
     : Pat<(intrinsic GPR64sp:$addr, GPR64:$x0, GPR64:$x1, GPR64:$x2, GPR64:$x3, GPR64:$x4, GPR64:$x5, GPR64:$x6, GPR64:$x7),
           (instruction (REG_SEQUENCE GPR64x8Class, $x0, x8sub_0, $x1, x8sub_1, $x2, x8sub_2, $x3, x8sub_3, $x4, x8sub_4, $x5, x8sub_5, $x6, x8sub_6, $x7, x8sub_7), $addr)>;
 
   def : ST64BPattern<int_aarch64_st64b, ST64B>;
   def : ST64BPattern<int_aarch64_st64bv, ST64BV>;
   def : ST64BPattern<int_aarch64_st64bv0, ST64BV0>;
 }
 
+let Defs = [X16, X17], mayStore = 1, isCodeGenOnly = 1 in
+def StoreSwiftAsyncContext
+      : Pseudo<(outs), (ins GPR64:$ctx, GPR64sp:$base, simm9:$offset),
+               []>, Sched<[]>;
+
 include "AArch64InstrAtomics.td"
 include "AArch64SVEInstrInfo.td"
 
 include "AArch64InstrGISel.td"

llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h

class AArch64FunctionInfo final : public MachineFunctionInfo {
 
   /// SignWithBKey modifies the default PAC-RET mode to signing with the B key.
   bool SignWithBKey = false;
 
   /// BranchTargetEnforcement enables placing BTI instructions at potential
   /// indirect branch destinations.
   bool BranchTargetEnforcement = false;
 
+  /// Whether this function has an extended frame record [Ctx, FP, LR]. If so,
+  /// bit 60 of the in-memory FP will be 1 to enable other tools to detect the
+  /// extended record.
+  bool HasSwiftAsyncContext = false;
+
+  /// The stack slot where the Swift asynchronous context is stored.
+  int SwiftAsyncContextFrameIdx = std::numeric_limits<int>::max();
+
 public:
   explicit AArch64FunctionInfo(MachineFunction &MF);
 
   void initializeBaseYamlFields(const yaml::AArch64FunctionInfo &YamlMFI);
 
   unsigned getBytesInStackArgArea() const { return BytesInStackArgArea; }
   void setBytesInStackArgArea(unsigned bytes) { BytesInStackArgArea = bytes; }
 
if (!HasCalleeSavedStackSize || ValidateCalleeSavedStackSize) {
         if (MFI.getStackID(FrameIdx) != TargetStackID::Default)
           continue;
         int64_t Offset = MFI.getObjectOffset(FrameIdx);
         int64_t ObjSize = MFI.getObjectSize(FrameIdx);
         MinOffset = std::min<int64_t>(Offset, MinOffset);
         MaxOffset = std::max<int64_t>(Offset + ObjSize, MaxOffset);
       }
 
+      if (SwiftAsyncContextFrameIdx != std::numeric_limits<int>::max()) {
+        int64_t Offset = MFI.getObjectOffset(getSwiftAsyncContextFrameIdx());
+        int64_t ObjSize = MFI.getObjectSize(getSwiftAsyncContextFrameIdx());
+        MinOffset = std::min<int64_t>(Offset, MinOffset);
+        MaxOffset = std::max<int64_t>(Offset + ObjSize, MaxOffset);
+      }
+
       unsigned Size = alignTo(MaxOffset - MinOffset, 16);
       assert((!HasCalleeSavedStackSize || getCalleeSavedStackSize() == Size) &&
              "Invalid size calculated for callee saves");
       return Size;
     }
 
     return getCalleeSavedStackSize();
   }
#endif
 
   bool shouldSignReturnAddress() const;
   bool shouldSignReturnAddress(bool SpillsLR) const;
 
   bool shouldSignWithBKey() const { return SignWithBKey; }
 
   bool branchTargetEnforcement() const { return BranchTargetEnforcement; }
 
+  void setHasSwiftAsyncContext(bool HasContext) {
+    HasSwiftAsyncContext = HasContext;
+  }
+  bool hasSwiftAsyncContext() const { return HasSwiftAsyncContext; }
+
+  void setSwiftAsyncContextFrameIdx(int FI) {
+    SwiftAsyncContextFrameIdx = FI;
+  }
+  int getSwiftAsyncContextFrameIdx() const { return SwiftAsyncContextFrameIdx; }
+
 private:
   // Hold the lists of LOHs.
   MILOHContainer LOHContainerSet;
   SetOfInstructions LOHRelated;
 
   SmallVector<std::pair<unsigned, MCSymbol *>, 2> JumpTableEntryInfo;
 };
 

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

bool AArch64CallLowering::lowerFormalArguments(
   unsigned i = 0;
   for (auto &Arg : F.args()) {
     if (DL.getTypeStoreSize(Arg.getType()).isZero())
       continue;
 
     ArgInfo OrigArg{VRegs[i], Arg};
     setArgFlags(OrigArg, i + AttributeList::FirstArgIndex, DL, F);
 
+    if (Arg.hasAttribute(Attribute::SwiftAsync))
+      MF.getInfo<AArch64FunctionInfo>()->setHasSwiftAsyncContext(true);
+
     splitToValueTypes(OrigArg, SplitArgs, DL, F.getCallingConv());
     ++i;
   }
 
   if (!MBB.empty())
     MIRBuilder.setInstr(*MBB.begin());
 
   const AArch64TargetLowering &TLI = *getTLI<AArch64TargetLowering>();

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

else {
         MIB.buildInstr(AArch64::XPACLRI);
         MIB.buildCopy({DstReg}, {Register(AArch64::LR)});
       }
     }
 
     I.eraseFromParent();
     return true;
   }
+  case Intrinsic::swift_async_context_addr:
+    auto Sub = MIB.buildInstr(AArch64::SUBXri, {I.getOperand(0).getReg()},
+                              {Register(AArch64::FP)})
+                   .addImm(8)
+                   .addImm(0);
+    constrainSelectedInstRegOperands(*Sub, TII, TRI, RBI);
+
+    MF->getFrameInfo().setFrameAddressIsTaken(true);
+    MF->getInfo<AArch64FunctionInfo>()->setHasSwiftAsyncContext(true);
+    I.eraseFromParent();
+    return true;
   }
   return false;
 }
 
 InstructionSelector::ComplexRendererFns
 AArch64InstructionSelector::selectShiftA_32(const MachineOperand &Root) const {
   auto MaybeImmed = getImmedFromMO(Root);
   if (MaybeImmed == None || *MaybeImmed > 31)

llvm/lib/Target/AArch64/MCTargetDesc/AArch64AsmBackend.cpp

uint32_t generateCompactUnwindEncoding(
                              ArrayRef<MCCFIInstruction> Instrs) const override {
     if (Instrs.empty())
       return CU::UNWIND_ARM64_MODE_FRAMELESS;
 
     bool HasFP = false;
     unsigned StackSize = 0;
 
     uint32_t CompactUnwindEncoding = 0;
+    int CurOffset = 0;
     for (size_t i = 0, e = Instrs.size(); i != e; ++i) {
       const MCCFIInstruction &Inst = Instrs[i];
 
       switch (Inst.getOperation()) {
       default:
         // Cannot handle this directive:  bail out.
         return CU::UNWIND_ARM64_MODE_DWARF;
       case MCCFIInstruction::OpDefCfa: {
for (size_t i = 0, e = Instrs.size(); i != e; ++i) {
 
         const MCCFIInstruction &LRPush = Instrs[++i];
         assert(LRPush.getOperation() == MCCFIInstruction::OpOffset &&
                "Link register not pushed!");
         const MCCFIInstruction &FPPush = Instrs[++i];
         assert(FPPush.getOperation() == MCCFIInstruction::OpOffset &&
                "Frame pointer not pushed!");
 
+        assert(FPPush.getOffset() + 8 == LRPush.getOffset());
+        CurOffset = FPPush.getOffset();
+
         unsigned LRReg = *MRI.getLLVMRegNum(LRPush.getRegister(), true);
         unsigned FPReg = *MRI.getLLVMRegNum(FPPush.getRegister(), true);
 
         LRReg = getXRegFromWReg(LRReg);
         FPReg = getXRegFromWReg(FPReg);
 
         assert(LRReg == AArch64::LR && FPReg == AArch64::FP &&
                "Pushing invalid registers for frame!");
for (size_t i = 0, e = Instrs.size(); i != e; ++i) {
       }
       case MCCFIInstruction::OpOffset: {
         // Registers are saved in pairs. We expect there to be two consecutive
         // `.cfi_offset' instructions with the appropriate registers specified.
         unsigned Reg1 = *MRI.getLLVMRegNum(Inst.getRegister(), true);
         if (i + 1 == e)
           return CU::UNWIND_ARM64_MODE_DWARF;
 
+        if (CurOffset != 0 && Inst.getOffset() != CurOffset - 8)
+          return CU::UNWIND_ARM64_MODE_DWARF;
+        CurOffset = Inst.getOffset();
+
         const MCCFIInstruction &Inst2 = Instrs[++i];
         if (Inst2.getOperation() != MCCFIInstruction::OpOffset)
           return CU::UNWIND_ARM64_MODE_DWARF;
         unsigned Reg2 = *MRI.getLLVMRegNum(Inst2.getRegister(), true);
 
+        if (Inst2.getOffset() != CurOffset - 8)
+          return CU::UNWIND_ARM64_MODE_DWARF;
+        CurOffset = Inst2.getOffset();
+
         // N.B. The encodings must be in register number order, and the X
         // registers before the D registers.
 
         // X19/X20 pair = 0x00000001,
         // X21/X22 pair = 0x00000002,
         // X23/X24 pair = 0x00000004,
         // X25/X26 pair = 0x00000008,
         // X27/X28 pair = 0x00000010

llvm/lib/Transforms/Utils/CodeExtractor.cpp

if (Attr.isStringAttribute()) {
       case Attribute::Returned:
       case Attribute::ReturnsTwice:
       case Attribute::SExt:
       case Attribute::Speculatable:
       case Attribute::StackAlignment:
       case Attribute::StructRet:
       case Attribute::SwiftError:
       case Attribute::SwiftSelf:
+      case Attribute::SwiftAsync:
       case Attribute::WillReturn:
       case Attribute::WriteOnly:
       case Attribute::ZExt:
       case Attribute::ImmArg:
       case Attribute::ByRef:
       case Attribute::EndAttrKinds:
       case Attribute::EmptyKey:
       case Attribute::TombstoneKey:

llvm/test/Bitcode/attributes.ll

 
 ; CHECK: define void @f75()
 ; CHECK-NOT: define void @f75() #
 define void @f75() vscale_range(0,0)
 {
   ret void
 }
 
+; CHECK: define void @f76(i8* swiftasync %0)
+define void @f76(i8* swiftasync %0)
+{
+  ret void;
+}
+
 ; CHECK: attributes #0 = { noreturn }
 ; CHECK: attributes #1 = { nounwind }
 ; CHECK: attributes #2 = { readnone }
 ; CHECK: attributes #3 = { readonly }
 ; CHECK: attributes #4 = { noinline }
 ; CHECK: attributes #5 = { alwaysinline }
 ; CHECK: attributes #6 = { optsize }
 ; CHECK: attributes #7 = { ssp }

llvm/test/Bitcode/compatibility.ll

 declare void @f.param.nonnull(i8* nonnull)
 ; CHECK: declare void @f.param.nonnull(i8* nonnull)
 declare void @f.param.dereferenceable(i8* dereferenceable(4))
 ; CHECK: declare void @f.param.dereferenceable(i8* dereferenceable(4))
 declare void @f.param.dereferenceable_or_null(i8* dereferenceable_or_null(4))
 ; CHECK: declare void @f.param.dereferenceable_or_null(i8* dereferenceable_or_null(4))
 declare void @f.param.stack_align([2 x double] alignstack(16))
 ; CHECK: declare void @f.param.stack_align([2 x double] alignstack(16))
+declare void @f.param.swiftself(i8* swiftself)
+; CHECK: declare void @f.param.swiftself(i8* swiftself)
+declare void @f.param.swiftasync(i8* swiftasync)
+; CHECK: declare void @f.param.swiftasync(i8* swiftasync)
+declare void @f.param.swifterror(i8** swifterror)
+; CHECK: declare void @f.param.swifterror(i8** swifterror)
 
 ; Functions -- unnamed_addr and local_unnamed_addr
 declare void @f.unnamed_addr() unnamed_addr
 ; CHECK: declare void @f.unnamed_addr() unnamed_addr
 declare void @f.local_unnamed_addr() local_unnamed_addr
 ; CHECK: declare void @f.local_unnamed_addr() local_unnamed_addr
 
 ; Functions -- fn Attrs (Function attributes)

llvm/test/CodeGen/AArch64/swift-async-reg.ll

+; RUN: llc -mtriple=arm64-apple-ios %s -o - | FileCheck %s
+; RUN: llc -mtriple=arm64-apple-ios %s -o - -global-isel | FileCheck %s
+; RUN: llc -mtriple=arm64-apple-ios %s -o - -fast-isel | FileCheck %s
+
+define i8* @argument(i8* swiftasync %in) {
+; CHECK-LABEL: argument:
+; CHECK: mov x0, x22
+
+  ret i8* %in
+}
+
+define void @call(i8* %in) {
+; CHECK-LABEL: call:
+; CHECK: mov x22, x0
+
+  call i8* @argument(i8* swiftasync %in)
+  ret void
+}

llvm/test/CodeGen/AArch64/swift-async-unwind.ll

+; RUN: llc -mtriple=arm64-apple-ios %s -filetype=obj -o - | llvm-objdump --unwind-info - | FileCheck %s
+
+; Swift asynchronous context is incompatible with the compact unwind encoding
+; that currently exists and assumes callee-saved registers are right next to FP
+; in a particular order. This isn't a problem now because C++ exceptions aren't
+; allowed to unwind through Swift code, but at least make sure the compact info
+; says to use DWARF correctly.
+
+; CHECK: compact encoding: 0x03000000
+define void @foo(i8* swiftasync %in) "frame-pointer"="all" {
+  call void asm sideeffect "", "~{x23}"()
+  ret void
+}

llvm/test/CodeGen/AArch64/swift-async.ll

+; RUN: llc -mtriple=arm64-apple-ios %s -o - | FileCheck %s --check-prefixes=CHECK-NOAUTH,CHECK
+; RUN: llc -mtriple=arm64-apple-ios -mcpu=apple-a13 %s -o - | FileCheck %s --check-prefixes=CHECK-NOAUTH,CHECK
+; RUN: llc -mtriple=arm64e-apple-ios %s -o - | FileCheck %s --check-prefixes=CHECK-AUTH,CHECK
+
+; Important details in prologue:
+;   * x22 is stored just below x29
+;   * Enough stack space is allocated for everything
+define void @simple(i8* swiftasync %ctx) "frame-pointer"="all" {
+; CHECK-LABEL: simple:
+; CHECK: orr x29, x29, #0x100000000000000
+; CHECK: sub sp, sp, #32
+; CHECK: stp x29, x30, [sp, #16]
+
+; CHECK-NOAUTH: str x22, [sp, #8]
+; CHECK-AUTH: add x16, sp, #8
+; CHECK-AUTH: movk x16, #49946, lsl #48
+; CHECK-AUTH: mov x17, x22
+; CHECK-AUTH: pacdb x17, x16
+; CHECK-AUTH: str x17, [sp, #8]
+
+; CHECK: add x29, sp, #16
+; CHECK: .cfi_def_cfa w29, 16
+; CHECK: .cfi_offset w30, -8
+; CHECK: .cfi_offset w29, -16
+
+;[...]
+
+; CHECK: ldp x29, x30, [sp, #16]
+; CHECK: and x29, x29, #0xefffffffffffffff
+; CHECK: add sp, sp, #32
+
+  ret void
+}
+
+define void @more_csrs(i8* swiftasync %ctx) "frame-pointer"="all" {
+; CHECK-LABEL: more_csrs:
+; CHECK: orr x29, x29, #0x100000000000000
+; CHECK: sub sp, sp, #48
+; CHECK: stp x24, x23, [sp, #8]
+; CHECK: stp x29, x30, [sp, #32]
+
+; CHECK-NOAUTH: str x22, [sp, #24]
+; CHECK-AUTH: add x16, sp, #24
+; CHECK-AUTH: movk x16, #49946, lsl #48
+; CHECK-AUTH: mov x17, x22
+; CHECK-AUTH: pacdb x17, x16
+; CHECK-AUTH: str x17, [sp, #24]
+
+; CHECK: add x29, sp, #32
+; CHECK: .cfi_def_cfa w29, 16
+; CHECK: .cfi_offset w30, -8
+; CHECK: .cfi_offset w29, -16
+; CHECK: .cfi_offset w23, -32
+
+; [...]
+
+; CHECK: ldp x29, x30, [sp, #32]
+; CHECK: ldp x24, x23, [sp, #8]
+; CHECK: and x29, x29, #0xefffffffffffffff
+; CHECK: add sp, sp, #48
+  call void asm sideeffect "", "~{x23}"()
+  ret void
+}
+
+define void @locals(i8* swiftasync %ctx) "frame-pointer"="all" {
+; CHECK-LABEL: locals:
+; CHECK: orr x29, x29, #0x100000000000000
+; CHECK: sub sp, sp, #64
+; CHECK: stp x29, x30, [sp, #48]
+
+; CHECK-NOAUTH: str x22, [sp, #40]
+; CHECK-AUTH: add x16, sp, #40
+; CHECK-AUTH: movk x16, #49946, lsl #48
+; CHECK-AUTH: mov x17, x22
+; CHECK-AUTH: pacdb x17, x16
+; CHECK-AUTH: str x17, [sp, #40]
+
+; CHECK: add x29, sp, #48
+; CHECK: .cfi_def_cfa w29, 16
+; CHECK: .cfi_offset w30, -8
+; CHECK: .cfi_offset w29, -16
+
+; CHECK: mov x0, sp
+; CHECK: bl _bar
+
+; [...]
+
+; CHECK: ldp x29, x30, [sp, #48]
+; CHECK: and x29, x29, #0xefffffffffffffff
+; CHECK: add sp, sp, #64
+  %var = alloca i32, i32 10
+  call void @bar(i32* %var)
+  ret void
+}
+
+define void @use_input_context(i8* swiftasync %ctx, i8** %ptr) "frame-pointer"="all" {
+; CHECK-LABEL: use_input_context:
+
+; CHECK-NOAUTH: str x22, [sp
+; CHECK-AUTH: mov x17, x22
+
+; CHECK-NOT: x22
+; CHECK: str x22, [x0]
+
+  store i8* %ctx, i8** %ptr
+  ret void
+}
+
+define i8** @context_in_func() "frame-pointer"="non-leaf" {
+; CHECK-LABEL: context_in_func:
+
+; CHECK-NOAUTH: str xzr, [sp, #8]
+; CHECK-AUTH: add x16, sp, #8
+; CHECK-AUTH: movk x16, #49946, lsl #48
+; CHECK-AUTH: mov x17, xzr
+; CHECK-AUTH: pacdb x17, x16
+; CHECK-AUTH: str x17, [sp, #8]
+
+  %ptr = call i8** @llvm.swift.async.context.addr()
+  ret i8** %ptr
+}
+
+define void @write_frame_context(i8* swiftasync %ctx, i8* %newctx) "frame-pointer"="non-leaf" {
+; CHECK-LABEL: write_frame_context:
+; CHECK: sub x[[ADDR:[0-9]+]], x29, #8
+; CHECK: str x0, [x[[ADDR]]]
+  %ptr = call i8** @llvm.swift.async.context.addr()
+  store i8* %newctx, i8** %ptr
+  ret void
+}
+
+define void @simple_fp_elim(i8* swiftasync %ctx) "frame-pointer"="non-leaf" {
+; CHECK-LABEL: simple_fp_elim:
+; CHECK-NOT: orr x29, x29, #0x100000000000000
+
+  ret void
+}
+
+define void @large_frame(i8* swiftasync %ctx) "frame-pointer"="all" {
+; CHECK-LABEL: large_frame:
+; CHECK: sub sp, sp, #48
+; CHECK: stp x28, x27, [sp, #8]
+; CHECK: stp x29, x30, [sp, #32]
+; CHECK-NOAUTH: str x22, [sp, #24]
+; CHECK: add x29, sp, #32
+; CHECK: sub sp, sp, #1024
+; [...]
+; CHECK: add sp, sp, #1024
+; CHECK: ldp x29, x30, [sp, #32]
+; CHECK: ldp x28, x27, [sp, #8]
+; CHECK: ret
+  %var = alloca i8, i32 1024
+  ret void
+}
+
+declare void @bar(i32*)
+declare i8** @llvm.swift.async.context.addr()

llvm/test/Verifier/swiftasync.ll

+; RUN: not llvm-as %s -o /dev/null 2>&1 | FileCheck %s
+
+declare void @a(i32* swiftasync %a, i32* swiftasync %b)
+; CHECK: Cannot have multiple 'swiftasync' parameters!