Swift Calling Convention: target-specific changes

[manmanren]

Summary

This patch applies on top of (reviews.llvm.org/D18108). This is the last patch for swifterror, the first patch was committed as r265189.

[manmanren]

Per review comment in reviews.llvm.org/D18108), I will add a testing case: multiple swifterror values exist in the caller and the actual argument is not the first entry in SwiftErrorVals.

[manmanren]

Adding a testing case with multiple swifterror values, this testing case will fail if we always use the first entry in SwiftErrorVals in LowerCallTo.

[qcolombet]

Hi Manman,

The functionality seems mostly good to me. I have highlighted on the first target the places that need the same kind of fixes you did for the target independent patch. You need to apply this to the three targets.

Regarding the tests, please add a few comments on what each function tests and also run opt -instnamer.

Thanks,
-Quentin

lib/Target/AArch64/AArch64CallingConvention.td

284 ↗

(On Diff #52816)

sub CSR_AArch64_AAPCS, X19?

lib/Target/AArch64/AArch64FastISel.cpp

1912 ↗

(On Diff #52816)

Same comment as the generic code of the previous review: Guard the whole thing with supportSwiftError and add a comment why we need to check these two.

2087 ↗

(On Diff #52816)

Ditto.

3671 ↗

(On Diff #52816)

swap the two conditions. hasAttrSomewhere is probably more expensive.

lib/Target/AArch64/AArch64FrameLowering.cpp

713 ↗

(On Diff #52816)

Shouldn’t we check for swift error support as well?

lib/Target/AArch64/AArch64RegisterInfo.cpp

56 ↗

(On Diff #52816)

Shouldn’t this be guard on supportSwiftError or something?

83 ↗

(On Diff #52816)

Ditto.

lib/Target/ARM/ARMBaseRegisterInfo.cpp

92 ↗

(On Diff #52816)

Ditto.

120 ↗

(On Diff #52816)

Ditto.

lib/Target/X86/X86CallingConv.td

854 ↗

(On Diff #52816)

sub CSR_64, R12

lib/Target/X86/X86ISelLowering.cpp

2359 ↗

(On Diff #52816)

Unrelated change?

lib/Target/X86/X86RegisterInfo.cpp

304 ↗

(On Diff #52816)

Shouldn’t this be guarded by supportSwiftError()?

test/CodeGen/ARM/swifterror.ll

8 ↗

(On Diff #52816)

Add a comment on what particular configuration this function tests.

27 ↗

(On Diff #52816)

Run opt -instnamer to get rid of the %[0-9]+ variables.

32 ↗

(On Diff #52816)

Ditto for the comment.

[manmanren]

Hi Quentin,

Thanks for reviewing the patch!

Manman

lib/Target/AArch64/AArch64CallingConvention.td

284 ↗

(On Diff #52816)

Yes, will do

lib/Target/AArch64/AArch64FastISel.cpp

1912 ↗

(On Diff #52816)

Will do

2087 ↗

(On Diff #52816)

Same here

3671 ↗

(On Diff #52816)

Here too

lib/Target/AArch64/AArch64FrameLowering.cpp

713 ↗

(On Diff #52816)

I don't see an access to TLI from frame lowering. Should we put it in AArch64FunctionInfo? Any other simpler way?

lib/Target/AArch64/AArch64RegisterInfo.cpp

56 ↗

(On Diff #52816)

I don't see an access to TLI from frame lowering. Should we put it in AArch64FunctionInfo? Any other simpler way?

lib/Target/ARM/ARMFastISel.cpp

1075 ↗

(On Diff #52816)

Will change this too.

1201 ↗

(On Diff #52816)

Same here

2112 ↗

(On Diff #52816)

Same here

lib/Target/X86/X86CallingConv.td

854 ↗

(On Diff #52816)

Yes

lib/Target/X86/X86FastISel.cpp

984 ↗

(On Diff #52816)

Will change here too.

lib/Target/X86/X86ISelLowering.cpp

2359 ↗

(On Diff #52816)

This is actually needed for the case of swifterror with sret. The testing case will fail without this change.

More info: We need to change the input chain for "CopyFromReg" of sret. It should have no effect when swifterror is not used, since the input chain should be the same with and without this change.

When sret is used with swifterror, we create a few SDNodes: "CopyToReg" of swifterror, "CopyFromReg" of sret, "CopyToReg" of sret. Before this change, the input chain for "CopyFromReg" of sret comes from "CopyToReg" of swifterror, and we will have a cycle in Unit graph. "CopyToReg" of swifterror and sret will be glued together into a single unit (Unit A), "CopyFromReg" of sret will be in a different unit (Unit B). We will have cyclic dependency between Unit A and Unit B:
Data dependency from Unit B to Unit A due to "CopyToReg" of sret using data from "CopyFromReg" of sret
Chain dependency from Unit A to Unit B due to "CopyFromReg" of sret using chain from "CopyToReg" of swifterror

This change uses the chain prior to handling the result values, as input chain for "CopyFromReg" of sret, to remove the cyclic dependency.

I will update the comment.

test/CodeGen/ARM/swifterror.ll

8 ↗

(On Diff #52816)

will do

27 ↗

(On Diff #52816)

Yes

32 ↗

(On Diff #52816)

Same here

[manmanren]

Addressing review comments.

[manmanren]

The part that is not addressed is using the flag "supportSwiftError" from RegisterInfo or FrameLowering. I am not sure the best way to do this.
Currently supportSwiftError is in TLI, and FrameLowering and RegisterInfo do not have access to TLI, I think.

MachineFunctionInfo doesn't look like the right place to add this, since this is not specific to each function.

Thanks for reviewing, Quentin!

Manman

[qcolombet]

Hi Manman,

I haven’t checked all the places, but I think you should have access to TLI through the MachineFunction then the sub target everywhere.

Cheers,
-Quentin

lib/Target/AArch64/AArch64FrameLowering.cpp

713 ↗

(On Diff #53273)

You should have access to it through the sub target you get from the function.

lib/Target/AArch64/AArch64RegisterInfo.cpp

56 ↗

(On Diff #53273)

MF->getSubtarget()->getTargetLowering().

lib/Target/X86/X86ISelLowering.cpp

2304 ↗

(On Diff #53273)

The comment helps :).

[manmanren]

Addressing comments

[manmanren]

Cheers,
Manman

lib/Target/AArch64/AArch64FrameLowering.cpp

713 ↗

(On Diff #53302)

Yes, you are right.

[qcolombet]

test/CodeGen/AArch64/swifterror.ll

8 ↗

(On Diff #53302)

tests -> test

36 ↗

(On Diff #53302)

Could you add a description for the test?

249 ↗

(On Diff #53302)

Missing description.

339 ↗

(On Diff #53302)

Description.

[MatzeB]

llvm/trunk/lib/Target/AArch64/AArch64RegisterInfo.cpp

84–87

In case of a call the MachineFunction passed to getCallPreservedMask() appears to be the callers machine function (because in general we do not have a machine function for the callee), but I believe this check should be about the callee...

llvm/trunk/lib/Target/AArch64/AArch64CallingConvention.td

def CC_AArch64_AAPCS : CallingConv<[
   CCIfType<[f128, v2i64, v4i32, v8i16, v16i8, v4f32, v2f64, v8f16],
            CCAssignToStack<16, 16>>
 ]>;
 
 def RetCC_AArch64_AAPCS : CallingConv<[
   CCIfType<[v2f32], CCBitConvertToType<v2i32>>,
   CCIfType<[v2f64, v4f32], CCBitConvertToType<v2i64>>,
 
+  CCIfSwiftError<CCIfType<[i64], CCAssignToRegWithShadow<[X19], [W19]>>>,
+
   // Big endian vectors must be passed as if they were 1-element vectors so that
   // their lanes are in a consistent order.
   CCIfBigEndian<CCIfType<[v2i32, v2f32, v4i16, v4f16, v8i8],
                          CCBitConvertToType<f64>>>,
   CCIfBigEndian<CCIfType<[v2i64, v2f64, v4i32, v4f32, v8i16, v8f16, v16i8],
                          CCBitConvertToType<f128>>>,
 
   CCIfType<[i32], CCAssignToRegWithShadow<[W0, W1, W2, W3, W4, W5, W6, W7],
def CC_AArch64_DarwinPCS : CallingConv<[
 
   // Put ByVal arguments directly on the stack. Minimum size and alignment of a
   // slot is 64-bit.
   CCIfByVal<CCPassByVal<8, 8>>,
 
   // A SwiftSelf is passed in X9.
   CCIfSwiftSelf<CCIfType<[i64], CCAssignToRegWithShadow<[X9], [W9]>>>,
 
+  // A SwiftError is passed in X19.
+  CCIfSwiftError<CCIfType<[i64], CCAssignToRegWithShadow<[X19], [W19]>>>,
+
   CCIfConsecutiveRegs<CCCustom<"CC_AArch64_Custom_Block">>,
 
   // Handle i1, i8, i16, i32, i64, f32, f64 and v2f64 by passing in registers,
   // up to eight each of GPR and FPR.
   CCIfType<[i1, i8, i16], CCPromoteToType<i32>>,
   CCIfType<[i32], CCAssignToRegWithShadow<[W0, W1, W2, W3, W4, W5, W6, W7],
                                           [X0, X1, X2, X3, X4, X5, X6, X7]>>,
   // i128 is split to two i64s, we can't fit half to register X7.

 // this can be partially modelled by treating X0 as a callee-saved register;
 // only the resulting RegMask is used; the SaveList is ignored
 //
 // (For generic ARM 64-bit ABI code, clang will not generate constructors or
 // destructors with 'this' returns, so this RegMask will not be used in that
 // case)
 def CSR_AArch64_AAPCS_ThisReturn : CalleeSavedRegs<(add CSR_AArch64_AAPCS, X0)>;
 
+def CSR_AArch64_AAPCS_SwiftError
+    : CalleeSavedRegs<(sub CSR_AArch64_AAPCS, X19)>;
+
 // The function used by Darwin to obtain the address of a thread-local variable
 // guarantees more than a normal AAPCS function. x16 and x17 are used on the
 // fast path for calculation, but other registers except X0 (argument/return)
 // and LR (it is a call, after all) are preserved.
 def CSR_AArch64_TLS_Darwin
     : CalleeSavedRegs<(add (sub (sequence "X%u", 1, 28), X16, X17),
                            FP,
                            (sequence "Q%u", 0, 31))>;

llvm/trunk/lib/Target/AArch64/AArch64FastISel.cpp

bool AArch64FastISel::selectLoad(const Instruction *I) {
   MVT VT;
   // Verify we have a legal type before going any further.  Currently, we handle
   // simple types that will directly fit in a register (i32/f32/i64/f64) or
   // those that can be sign or zero-extended to a basic operation (i1/i8/i16).
   if (!isTypeSupported(I->getType(), VT, /*IsVectorAllowed=*/true) ||
       cast<LoadInst>(I)->isAtomic())
     return false;
 
+  const Value *SV = I->getOperand(0);
+  if (TLI.supportSwiftError()) {
+    // Swifterror values can come from either a function parameter with
+    // swifterror attribute or an alloca with swifterror attribute.
+    if (const Argument *Arg = dyn_cast<Argument>(SV)) {
+      if (Arg->hasSwiftErrorAttr())
+        return false;
+    }
+
+    if (const AllocaInst *Alloca = dyn_cast<AllocaInst>(SV)) {
+      if (Alloca->isSwiftError())
+        return false;
+    }
+  }
+
   // See if we can handle this address.
   Address Addr;
   if (!computeAddress(I->getOperand(0), Addr, I->getType()))
     return false;
 
   // Fold the following sign-/zero-extend into the load instruction.
   bool WantZExt = true;
   MVT RetVT = VT;
bool AArch64FastISel::selectStore(const Instruction *I) {
   const Value *Op0 = I->getOperand(0);
   // Verify we have a legal type before going any further.  Currently, we handle
   // simple types that will directly fit in a register (i32/f32/i64/f64) or
   // those that can be sign or zero-extended to a basic operation (i1/i8/i16).
   if (!isTypeSupported(Op0->getType(), VT, /*IsVectorAllowed=*/true) ||
       cast<StoreInst>(I)->isAtomic())
     return false;
 
+  const Value *PtrV = I->getOperand(1);
+  if (TLI.supportSwiftError()) {
+    // Swifterror values can come from either a function parameter with
+    // swifterror attribute or an alloca with swifterror attribute.
+    if (const Argument *Arg = dyn_cast<Argument>(PtrV)) {
+      if (Arg->hasSwiftErrorAttr())
+        return false;
+    }
+
+    if (const AllocaInst *Alloca = dyn_cast<AllocaInst>(PtrV)) {
+      if (Alloca->isSwiftError())
+        return false;
+    }
+  }
+
   // Get the value to be stored into a register. Use the zero register directly
   // when possible to avoid an unnecessary copy and a wasted register.
   unsigned SrcReg = 0;
   if (const auto *CI = dyn_cast<ConstantInt>(Op0)) {
     if (CI->isZero())
       SrcReg = (VT == MVT::i64) ? AArch64::XZR : AArch64::WZR;
   } else if (const auto *CF = dyn_cast<ConstantFP>(Op0)) {
     if (CF->isZero() && !CF->isNegative()) {
bool AArch64FastISel::fastLowerArguments() {
   unsigned Idx = 0;
   for (auto const &Arg : F->args()) {
     // The first argument is at index 1.
     ++Idx;
     if (F->getAttributes().hasAttribute(Idx, Attribute::ByVal) ||
         F->getAttributes().hasAttribute(Idx, Attribute::InReg) ||
         F->getAttributes().hasAttribute(Idx, Attribute::StructRet) ||
         F->getAttributes().hasAttribute(Idx, Attribute::SwiftSelf) ||
+        F->getAttributes().hasAttribute(Idx, Attribute::SwiftError) ||
         F->getAttributes().hasAttribute(Idx, Attribute::Nest))
       return false;
 
     Type *ArgTy = Arg.getType();
     if (ArgTy->isStructTy() || ArgTy->isArrayTy())
       return false;
 
     EVT ArgVT = TLI.getValueType(DL, ArgTy);
bool AArch64FastISel::fastLowerCall(CallLoweringInfo &CLI) {
   MVT RetVT;
   if (CLI.RetTy->isVoidTy())
     RetVT = MVT::isVoid;
   else if (!isTypeLegal(CLI.RetTy, RetVT))
     return false;
 
   for (auto Flag : CLI.OutFlags)
     if (Flag.isInReg() || Flag.isSRet() || Flag.isNest() || Flag.isByVal() ||
-        Flag.isSwiftSelf())
+        Flag.isSwiftSelf() || Flag.isSwiftError())
       return false;
 
   // Set up the argument vectors.
   SmallVector<MVT, 16> OutVTs;
   OutVTs.reserve(CLI.OutVals.size());
 
   for (auto *Val : CLI.OutVals) {
     MVT VT;
bool AArch64FastISel::selectRet(const Instruction *I) {
   const Function &F = *I->getParent()->getParent();
 
   if (!FuncInfo.CanLowerReturn)
     return false;
 
   if (F.isVarArg())
     return false;
 
+  if (TLI.supportSwiftError() &&
+      F.getAttributes().hasAttrSomewhere(Attribute::SwiftError))
+    return false;
+
   if (TLI.supportSplitCSR(FuncInfo.MF))
     return false;
 
   // Build a list of return value registers.
   SmallVector<unsigned, 4> RetRegs;
 
   if (Ret->getNumOperands() > 0) {
     CallingConv::ID CC = F.getCallingConv();

llvm/trunk/lib/Target/AArch64/AArch64FrameLowering.cpp

if (Reg != AArch64::LR)
     return getKillRegState(true);
 
   // LR maybe referred to later by an @llvm.returnaddress intrinsic.
   bool LRLiveIn = MF.getRegInfo().isLiveIn(AArch64::LR);
   bool LRKill = !(LRLiveIn && MF.getFrameInfo()->isReturnAddressTaken());
   return getKillRegState(LRKill);
 }
 
+static bool produceCompactUnwindFrame(MachineFunction &MF) {
+  const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
+  AttributeSet Attrs = MF.getFunction()->getAttributes();
+  return Subtarget.isTargetMachO() &&
+         !(Subtarget.getTargetLowering()->supportSwiftError() &&
+           Attrs.hasAttrSomewhere(Attribute::SwiftError));
+}
+
+
 struct RegPairInfo {
   RegPairInfo() : Reg1(AArch64::NoRegister), Reg2(AArch64::NoRegister) {}
   unsigned Reg1;
   unsigned Reg2;
   int FrameIdx;
   int Offset;
   bool IsGPR;
   bool isPaired() const { return Reg2 != AArch64::NoRegister; }
 };
 
 static void computeCalleeSaveRegisterPairs(
     MachineFunction &MF, const std::vector<CalleeSavedInfo> &CSI,
     const TargetRegisterInfo *TRI, SmallVectorImpl<RegPairInfo> &RegPairs) {
 
   if (CSI.empty())
     return;
 
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   MachineFrameInfo *MFI = MF.getFrameInfo();
   CallingConv::ID CC = MF.getFunction()->getCallingConv();
   unsigned Count = CSI.size();
   (void)CC;
   // MachO's compact unwind format relies on all registers being stored in
   // pairs.
-  assert((!MF.getSubtarget<AArch64Subtarget>().isTargetMachO() ||
+  assert((!produceCompactUnwindFrame(MF) ||
           CC == CallingConv::PreserveMost ||
           (Count & 1) == 0) &&
          "Odd number of callee-saved regs to spill!");
   unsigned Offset = AFI->getCalleeSavedStackSize();
 
   for (unsigned i = 0; i < Count; ++i) {
     RegPairInfo RPI;
     RPI.Reg1 = CSI[i].getReg();
for (unsigned i = 0; i < Count; ++i) {
     // The order of the registers in the list is controlled by
     // getCalleeSavedRegs(), so they will always be in-order, as well.
     assert((!RPI.isPaired() ||
             (CSI[i].getFrameIdx() + 1 == CSI[i + 1].getFrameIdx())) &&
            "Out of order callee saved regs!");
 
     // MachO's compact unwind format relies on all registers being stored in
     // adjacent register pairs.
-    assert((!MF.getSubtarget<AArch64Subtarget>().isTargetMachO() ||
+    assert((!produceCompactUnwindFrame(MF) ||
             CC == CallingConv::PreserveMost ||
             (RPI.isPaired() &&
              ((RPI.Reg1 == AArch64::LR && RPI.Reg2 == AArch64::FP) ||
               RPI.Reg1 + 1 == RPI.Reg2))) &&
            "Callee-save registers not saved as adjacent register pair!");
 
     RPI.FrameIdx = CSI[i].getFrameIdx();
 
void AArch64FrameLowering::determineCalleeSaves(MachineFunction &MF,
   // prologue/epilogue.
   if (MF.getFunction()->getCallingConv() == CallingConv::GHC)
     return;
 
   TargetFrameLowering::determineCalleeSaves(MF, SavedRegs, RS);
   const AArch64RegisterInfo *RegInfo = static_cast<const AArch64RegisterInfo *>(
       MF.getSubtarget().getRegisterInfo());
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
-  const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   unsigned UnspilledCSGPR = AArch64::NoRegister;
   unsigned UnspilledCSGPRPaired = AArch64::NoRegister;
 
   // The frame record needs to be created by saving the appropriate registers
   if (hasFP(MF)) {
     SavedRegs.set(AArch64::FP);
     SavedRegs.set(AArch64::LR);
   }
if (!RegUsed) {
         UnspilledCSGPRPaired = PairedReg;
       }
       continue;
     }
 
     // MachO's compact unwind format relies on all registers being stored in
     // pairs.
     // FIXME: the usual format is actually better if unwinding isn't needed.
-    if (Subtarget.isTargetMachO() && !SavedRegs.test(PairedReg)) {
+    if (produceCompactUnwindFrame(MF) && !SavedRegs.test(PairedReg)) {
       SavedRegs.set(PairedReg);
       ExtraCSSpill = true;
     }
   }
 
   DEBUG(dbgs() << "*** determineCalleeSaves\nUsed CSRs:";
         for (int Reg = SavedRegs.find_first(); Reg != -1;
              Reg = SavedRegs.find_next(Reg))
void AArch64FrameLowering::determineCalleeSaves(MachineFunction &MF,
   if (BigStack && !ExtraCSSpill) {
     if (UnspilledCSGPR != AArch64::NoRegister) {
       DEBUG(dbgs() << "Spilling " << PrintReg(UnspilledCSGPR, RegInfo)
             << " to get a scratch register.\n");
       SavedRegs.set(UnspilledCSGPR);
       // MachO's compact unwind format relies on all registers being stored in
       // pairs, so if we need to spill one extra for BigStack, then we need to
       // store the pair.
-      if (Subtarget.isTargetMachO())
+      if (produceCompactUnwindFrame(MF))
         SavedRegs.set(UnspilledCSGPRPaired);
       ExtraCSSpill = true;
       NumRegsSpilled = SavedRegs.count();
     }
 
     // If we didn't find an extra callee-saved register to spill, create
     // an emergency spill slot.
     if (!ExtraCSSpill) {

llvm/trunk/lib/Target/AArch64/AArch64ISelLowering.h

bool supportSplitCSR(MachineFunction *MF) const override {
     return MF->getFunction()->getCallingConv() == CallingConv::CXX_FAST_TLS &&
            MF->getFunction()->hasFnAttribute(Attribute::NoUnwind);
   }
   void initializeSplitCSR(MachineBasicBlock *Entry) const override;
   void insertCopiesSplitCSR(
       MachineBasicBlock *Entry,
       const SmallVectorImpl<MachineBasicBlock *> &Exits) const override;
 
+  bool supportSwiftError() const override {
+    return true;
+  }
+
 private:
   bool isExtFreeImpl(const Instruction *Ext) const override;
 
   /// Keep a pointer to the AArch64Subtarget around so that we can
   /// make the right decision when generating code for different targets.
   const AArch64Subtarget *Subtarget;
 
   void addTypeForNEON(EVT VT, EVT PromotedBitwiseVT);

llvm/trunk/lib/Target/AArch64/AArch64RegisterInfo.cpp

if (MF->getFunction()->getCallingConv() == CallingConv::GHC)
     // used for passing STG regs around
     return CSR_AArch64_NoRegs_SaveList;
   if (MF->getFunction()->getCallingConv() == CallingConv::AnyReg)
     return CSR_AArch64_AllRegs_SaveList;
   if (MF->getFunction()->getCallingConv() == CallingConv::CXX_FAST_TLS)
     return MF->getInfo<AArch64FunctionInfo>()->isSplitCSR() ?
            CSR_AArch64_CXX_TLS_Darwin_PE_SaveList :
            CSR_AArch64_CXX_TLS_Darwin_SaveList;
+  if (MF->getSubtarget<AArch64Subtarget>().getTargetLowering()
+          ->supportSwiftError() &&
+      MF->getFunction()->getAttributes().hasAttrSomewhere(
+          Attribute::SwiftError))
+    return CSR_AArch64_AAPCS_SwiftError_SaveList;
   if (MF->getFunction()->getCallingConv() == CallingConv::PreserveMost)
     return CSR_AArch64_RT_MostRegs_SaveList;
   else
     return CSR_AArch64_AAPCS_SaveList;
 }
 
 const MCPhysReg *AArch64RegisterInfo::getCalleeSavedRegsViaCopy(
     const MachineFunction *MF) const {
AArch64RegisterInfo::getCallPreservedMask(const MachineFunction &MF,
                                           CallingConv::ID CC) const {
   if (CC == CallingConv::GHC)
     // This is academic becase all GHC calls are (supposed to be) tail calls
     return CSR_AArch64_NoRegs_RegMask;
   if (CC == CallingConv::AnyReg)
     return CSR_AArch64_AllRegs_RegMask;
   if (CC == CallingConv::CXX_FAST_TLS)
     return CSR_AArch64_CXX_TLS_Darwin_RegMask;
+  if (MF.getSubtarget<AArch64Subtarget>().getTargetLowering()
+          ->supportSwiftError() &&
+      MF.getFunction()->getAttributes().hasAttrSomewhere(Attribute::SwiftError))
+    return CSR_AArch64_AAPCS_SwiftError_RegMask;
   if (CC == CallingConv::PreserveMost)
     return CSR_AArch64_RT_MostRegs_RegMask;
   else
     return CSR_AArch64_AAPCS_RegMask;
 }
 
 const uint32_t *AArch64RegisterInfo::getTLSCallPreservedMask() const {
   if (TT.isOSDarwin())

llvm/trunk/lib/Target/ARM/ARMBaseRegisterInfo.cpp

if (STI.isMClass()) {
       return CSR_FIQ_SaveList;
     } else {
       // Generally only R13-R14 (i.e. SP, LR) are automatically preserved by
       // exception handling.
       return CSR_GenericInt_SaveList;
     }
   }
 
+  if (STI.isTargetDarwin() && STI.getTargetLowering()->supportSwiftError() &&
+      F->getAttributes().hasAttrSomewhere(Attribute::SwiftError))
+    return CSR_iOS_SwiftError_SaveList;
+
   if (STI.isTargetDarwin() && F->getCallingConv() == CallingConv::CXX_FAST_TLS)
     return MF->getInfo<ARMFunctionInfo>()->isSplitCSR()
                ? CSR_iOS_CXX_TLS_PE_SaveList
                : CSR_iOS_CXX_TLS_SaveList;
   return RegList;
 }
 
 const MCPhysReg *ARMBaseRegisterInfo::getCalleeSavedRegsViaCopy(
     const MachineFunction *MF) const {
   assert(MF && "Invalid MachineFunction pointer.");
   if (MF->getFunction()->getCallingConv() == CallingConv::CXX_FAST_TLS &&
       MF->getInfo<ARMFunctionInfo>()->isSplitCSR())
     return CSR_iOS_CXX_TLS_ViaCopy_SaveList;
   return nullptr;
 }
 
 const uint32_t *
 ARMBaseRegisterInfo::getCallPreservedMask(const MachineFunction &MF,
                                           CallingConv::ID CC) const {
   const ARMSubtarget &STI = MF.getSubtarget<ARMSubtarget>();
   if (CC == CallingConv::GHC)
     // This is academic becase all GHC calls are (supposed to be) tail calls
     return CSR_NoRegs_RegMask;
+
+  if (STI.isTargetDarwin() && STI.getTargetLowering()->supportSwiftError() &&
+      MF.getFunction()->getAttributes().hasAttrSomewhere(Attribute::SwiftError))
+    return CSR_iOS_SwiftError_RegMask;
+
   if (STI.isTargetDarwin() && CC == CallingConv::CXX_FAST_TLS)
     return CSR_iOS_CXX_TLS_RegMask;
   return STI.isTargetDarwin() ? CSR_iOS_RegMask : CSR_AAPCS_RegMask;
 }
 
 const uint32_t*
 ARMBaseRegisterInfo::getNoPreservedMask() const {
   return CSR_NoRegs_RegMask;

llvm/trunk/lib/Target/ARM/ARMCallingConv.td

def CC_ARM_APCS : CallingConv<[
   // Handles byval parameters.
   CCIfByVal<CCPassByVal<4, 4>>,
     
   CCIfType<[i1, i8, i16], CCPromoteToType<i32>>,
 
   // A SwiftSelf is passed in R9.
   CCIfSwiftSelf<CCIfType<[i32], CCAssignToReg<[R9]>>>,
 
+  // A SwiftError is passed in R6.
+  CCIfSwiftError<CCIfType<[i32], CCAssignToReg<[R6]>>>,
+
   // Handle all vector types as either f64 or v2f64.
   CCIfType<[v1i64, v2i32, v4i16, v8i8, v2f32], CCBitConvertToType<f64>>,
   CCIfType<[v2i64, v4i32, v8i16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
 
   // f64 and v2f64 are passed in adjacent GPRs, possibly split onto the stack
   CCIfType<[f64, v2f64], CCCustom<"CC_ARM_APCS_Custom_f64">>,
 
   CCIfType<[f32], CCBitConvertToType<i32>>,
   CCIfType<[i32], CCAssignToReg<[R0, R1, R2, R3]>>,
 
   CCIfType<[i32], CCAssignToStack<4, 4>>,
   CCIfType<[f64], CCAssignToStack<8, 4>>,
   CCIfType<[v2f64], CCAssignToStack<16, 4>>
 ]>;
 
 def RetCC_ARM_APCS : CallingConv<[
   CCIfType<[i1, i8, i16], CCPromoteToType<i32>>,
   CCIfType<[f32], CCBitConvertToType<i32>>,
 
+  // A SwiftError is returned in R6.
+  CCIfSwiftError<CCIfType<[i32], CCAssignToReg<[R6]>>>,
+
   // Handle all vector types as either f64 or v2f64.
   CCIfType<[v1i64, v2i32, v4i16, v8i8, v2f32], CCBitConvertToType<f64>>,
   CCIfType<[v2i64, v4i32, v8i16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
 
   CCIfType<[f64, v2f64], CCCustom<"RetCC_ARM_APCS_Custom_f64">>,
 
   CCIfType<[i32], CCAssignToReg<[R0, R1, R2, R3]>>,
   CCIfType<[i64], CCAssignToRegWithShadow<[R0, R2], [R1, R3]>>
def CC_ARM_AAPCS : CallingConv<[
 
   // Handle all vector types as either f64 or v2f64.
   CCIfType<[v1i64, v2i32, v4i16, v8i8, v2f32], CCBitConvertToType<f64>>,
   CCIfType<[v2i64, v4i32, v8i16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
 
   // A SwiftSelf is passed in R9.
   CCIfSwiftSelf<CCIfType<[i32], CCAssignToReg<[R9]>>>,
 
+  // A SwiftError is passed in R6.
+  CCIfSwiftError<CCIfType<[i32], CCAssignToReg<[R6]>>>,
+
   CCIfType<[f64, v2f64], CCCustom<"CC_ARM_AAPCS_Custom_f64">>,
   CCIfType<[f32], CCBitConvertToType<i32>>,
   CCDelegateTo<CC_ARM_AAPCS_Common>
 ]>;
 
 def RetCC_ARM_AAPCS : CallingConv<[
   // Handle all vector types as either f64 or v2f64.
   CCIfType<[v1i64, v2i32, v4i16, v8i8, v2f32], CCBitConvertToType<f64>>,
   CCIfType<[v2i64, v4i32, v8i16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
 
+  // A SwiftError is returned in R6.
+  CCIfSwiftError<CCIfType<[i32], CCAssignToReg<[R6]>>>,
+
   CCIfType<[f64, v2f64], CCCustom<"RetCC_ARM_AAPCS_Custom_f64">>,
   CCIfType<[f32], CCBitConvertToType<i32>>,
   CCDelegateTo<RetCC_ARM_AAPCS_Common>
 ]>;
 
 //===----------------------------------------------------------------------===//
 // ARM AAPCS-VFP (EABI) Calling Convention
 // Also used for FastCC (when VFP2 or later is available)
 //===----------------------------------------------------------------------===//
 
 def CC_ARM_AAPCS_VFP : CallingConv<[
   // Handles byval parameters.
   CCIfByVal<CCPassByVal<4, 4>>,
 
   // Handle all vector types as either f64 or v2f64.
   CCIfType<[v1i64, v2i32, v4i16, v8i8, v2f32], CCBitConvertToType<f64>>,
   CCIfType<[v2i64, v4i32, v8i16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
 
   // A SwiftSelf is passed in R9.
   CCIfSwiftSelf<CCIfType<[i32], CCAssignToReg<[R9]>>>,
 
+  // A SwiftError is passed in R6.
+  CCIfSwiftError<CCIfType<[i32], CCAssignToReg<[R6]>>>,
+
   // HFAs are passed in a contiguous block of registers, or on the stack
   CCIfConsecutiveRegs<CCCustom<"CC_ARM_AAPCS_Custom_Aggregate">>,
 
   CCIfType<[v2f64], CCAssignToReg<[Q0, Q1, Q2, Q3]>>,
   CCIfType<[f64], CCAssignToReg<[D0, D1, D2, D3, D4, D5, D6, D7]>>,
   CCIfType<[f32], CCAssignToReg<[S0, S1, S2, S3, S4, S5, S6, S7, S8,
                                  S9, S10, S11, S12, S13, S14, S15]>>,
   CCDelegateTo<CC_ARM_AAPCS_Common>
 ]>;
 
 def RetCC_ARM_AAPCS_VFP : CallingConv<[
   // Handle all vector types as either f64 or v2f64.
   CCIfType<[v1i64, v2i32, v4i16, v8i8, v2f32], CCBitConvertToType<f64>>,
   CCIfType<[v2i64, v4i32, v8i16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
 
+  // A SwiftError is returned in R6.
+  CCIfSwiftError<CCIfType<[i32], CCAssignToReg<[R6]>>>,
+
   CCIfType<[v2f64], CCAssignToReg<[Q0, Q1, Q2, Q3]>>,
   CCIfType<[f64], CCAssignToReg<[D0, D1, D2, D3, D4, D5, D6, D7]>>,
   CCIfType<[f32], CCAssignToReg<[S0, S1, S2, S3, S4, S5, S6, S7, S8,
                                  S9, S10, S11, S12, S13, S14, S15]>>,
   CCDelegateTo<RetCC_ARM_AAPCS_Common>
 ]>;
 
 //===----------------------------------------------------------------------===//

 def CSR_AAPCS_ThisReturn : CalleeSavedRegs<(add LR, R11, R10, R9, R8, R7, R6,
                                             R5, R4, (sequence "D%u", 15, 8),
                                             R0)>;
 
 // iOS ABI deviates from ARM standard ABI. R9 is not a callee-saved register.
 // Also save R7-R4 first to match the stack frame fixed spill areas.
 def CSR_iOS : CalleeSavedRegs<(add LR, R7, R6, R5, R4, (sub CSR_AAPCS, R9))>;
 
+// R6 is used to pass swifterror, remove it from CSR.
+def CSR_iOS_SwiftError : CalleeSavedRegs<(sub CSR_iOS, R6)>;
+
 def CSR_iOS_ThisReturn : CalleeSavedRegs<(add LR, R7, R6, R5, R4,
                                          (sub CSR_AAPCS_ThisReturn, R9))>;
 
 def CSR_iOS_TLSCall : CalleeSavedRegs<(add LR, SP,
                                            (sequence "R%u", 12, 1),
                                            (sequence "D%u", 31, 0))>;
 
 // C++ TLS access function saves all registers except SP. Try to match

llvm/trunk/lib/Target/ARM/ARMFastISel.cpp

bool ARMFastISel::ARMEmitLoad(MVT VT, unsigned &ResultReg, Address &Addr,
   return true;
 }
 
 bool ARMFastISel::SelectLoad(const Instruction *I) {
   // Atomic loads need special handling.
   if (cast<LoadInst>(I)->isAtomic())
     return false;
 
+  const Value *SV = I->getOperand(0);
+  if (TLI.supportSwiftError()) {
+    // Swifterror values can come from either a function parameter with
+    // swifterror attribute or an alloca with swifterror attribute.
+    if (const Argument *Arg = dyn_cast<Argument>(SV)) {
+      if (Arg->hasSwiftErrorAttr())
+        return false;
+    }
+
+    if (const AllocaInst *Alloca = dyn_cast<AllocaInst>(SV)) {
+      if (Alloca->isSwiftError())
+        return false;
+    }
+  }
+
   // Verify we have a legal type before going any further.
   MVT VT;
   if (!isLoadTypeLegal(I->getType(), VT))
     return false;
 
   // See if we can handle this address.
   Address Addr;
   if (!ARMComputeAddress(I->getOperand(0), Addr)) return false;

 bool ARMFastISel::SelectStore(const Instruction *I) {
   Value *Op0 = I->getOperand(0);
   unsigned SrcReg = 0;
 
   // Atomic stores need special handling.
   if (cast<StoreInst>(I)->isAtomic())
     return false;
 
+  const Value *PtrV = I->getOperand(1);
+  if (TLI.supportSwiftError()) {
+    // Swifterror values can come from either a function parameter with
+    // swifterror attribute or an alloca with swifterror attribute.
+    if (const Argument *Arg = dyn_cast<Argument>(PtrV)) {
+      if (Arg->hasSwiftErrorAttr())
+        return false;
+    }
+
+    if (const AllocaInst *Alloca = dyn_cast<AllocaInst>(PtrV)) {
+      if (Alloca->isSwiftError())
+        return false;
+    }
+  }
+
   // Verify we have a legal type before going any further.
   MVT VT;
   if (!isLoadTypeLegal(I->getOperand(0)->getType(), VT))
     return false;
 
   // Get the value to be stored into a register.
   SrcReg = getRegForValue(Op0);
   if (SrcReg == 0) return false;

 
 bool ARMFastISel::SelectRet(const Instruction *I) {
   const ReturnInst *Ret = cast<ReturnInst>(I);
   const Function &F = *I->getParent()->getParent();
 
   if (!FuncInfo.CanLowerReturn)
     return false;
 
+  if (TLI.supportSwiftError() &&
+      F.getAttributes().hasAttrSomewhere(Attribute::SwiftError))
+    return false;
+
   if (TLI.supportSplitCSR(FuncInfo.MF))
     return false;
 
   // Build a list of return value registers.
   SmallVector<unsigned, 4> RetRegs;
 
   CallingConv::ID CC = F.getCallingConv();
   if (Ret->getNumOperands() > 0) {
if (CS.paramHasAttr(AttrInd, Attribute::SExt))
       Flags.setSExt();
     if (CS.paramHasAttr(AttrInd, Attribute::ZExt))
       Flags.setZExt();
 
     // FIXME: Only handle *easy* calls for now.
     if (CS.paramHasAttr(AttrInd, Attribute::InReg) ||
         CS.paramHasAttr(AttrInd, Attribute::StructRet) ||
         CS.paramHasAttr(AttrInd, Attribute::SwiftSelf) ||
+        CS.paramHasAttr(AttrInd, Attribute::SwiftError) ||
         CS.paramHasAttr(AttrInd, Attribute::Nest) ||
         CS.paramHasAttr(AttrInd, Attribute::ByVal))
       return false;
 
     Type *ArgTy = (*i)->getType();
     MVT ArgVT;
     if (!isTypeLegal(ArgTy, ArgVT) && ArgVT != MVT::i16 && ArgVT != MVT::i8 &&
         ArgVT != MVT::i1)
bool ARMFastISel::fastLowerArguments() {
   for (Function::const_arg_iterator I = F->arg_begin(), E = F->arg_end();
        I != E; ++I, ++Idx) {
     if (Idx > 4)
       return false;
 
     if (F->getAttributes().hasAttribute(Idx, Attribute::InReg) ||
         F->getAttributes().hasAttribute(Idx, Attribute::StructRet) ||
         F->getAttributes().hasAttribute(Idx, Attribute::SwiftSelf) ||
+        F->getAttributes().hasAttribute(Idx, Attribute::SwiftError) ||
         F->getAttributes().hasAttribute(Idx, Attribute::ByVal))
       return false;
 
     Type *ArgTy = I->getType();
     if (ArgTy->isStructTy() || ArgTy->isArrayTy() || ArgTy->isVectorTy())
       return false;
 
     EVT ArgVT = TLI.getValueType(DL, ArgTy);

llvm/trunk/lib/Target/ARM/ARMISelLowering.h

public:
     bool useLoadStackGuardNode() const override;
 
     bool canCombineStoreAndExtract(Type *VectorTy, Value *Idx,
                                    unsigned &Cost) const override;
 
     bool isCheapToSpeculateCttz() const override;
     bool isCheapToSpeculateCtlz() const override;
 
+    bool supportSwiftError() const override {
+      return true;
+    }
+
   protected:
     std::pair<const TargetRegisterClass *, uint8_t>
     findRepresentativeClass(const TargetRegisterInfo *TRI,
                             MVT VT) const override;
 
   private:
     /// Subtarget - Keep a pointer to the ARMSubtarget around so that we can
     /// make the right decision when generating code for different targets.

llvm/trunk/lib/Target/X86/X86CallingConv.td

 def RetCC_X86_64_C : CallingConv<[
   // The X86-64 calling convention always returns FP values in XMM0.
   CCIfType<[f32], CCAssignToReg<[XMM0, XMM1]>>,
   CCIfType<[f64], CCAssignToReg<[XMM0, XMM1]>>,
   CCIfType<[f128], CCAssignToReg<[XMM0, XMM1]>>,
 
   // MMX vector types are always returned in XMM0.
   CCIfType<[x86mmx], CCAssignToReg<[XMM0, XMM1]>>,
+
+  CCIfSwiftError<CCIfType<[i64], CCAssignToReg<[R12]>>>,
+
   CCDelegateTo<RetCC_X86Common>
 ]>;
 
 // X86-Win64 C return-value convention.
 def RetCC_X86_Win64_C : CallingConv<[
   // The X86-Win64 calling convention always returns __m64 values in RAX.
   CCIfType<[x86mmx], CCBitConvertToType<i64>>,
 
def CC_X86_64_C : CallingConv<[
 
   // The 'nest' parameter, if any, is passed in R10.
   CCIfNest<CCIfSubtarget<"isTarget64BitILP32()", CCAssignToReg<[R10D]>>>,
   CCIfNest<CCAssignToReg<[R10]>>,
 
   // A SwiftSelf is passed in R10.
   CCIfSwiftSelf<CCIfType<[i64], CCAssignToReg<[R10]>>>,
 
+  // A SwiftError is passed in R12.
+  CCIfSwiftError<CCIfType<[i64], CCAssignToReg<[R12]>>>,
+
   // The first 6 integer arguments are passed in integer registers.
   CCIfType<[i32], CCAssignToReg<[EDI, ESI, EDX, ECX, R8D, R9D]>>,
   CCIfType<[i64], CCAssignToReg<[RDI, RSI, RDX, RCX, R8 , R9 ]>>,
 
   // The first 8 MMX vector arguments are passed in XMM registers on Darwin.
   CCIfType<[x86mmx],
             CCIfSubtarget<"isTargetDarwin()",
             CCIfSubtarget<"hasSSE2()",

 // Callee-saved Registers.
 //===----------------------------------------------------------------------===//
 
 def CSR_NoRegs : CalleeSavedRegs<(add)>;
 
 def CSR_32 : CalleeSavedRegs<(add ESI, EDI, EBX, EBP)>;
 def CSR_64 : CalleeSavedRegs<(add RBX, R12, R13, R14, R15, RBP)>;
 
+def CSR_64_SwiftError : CalleeSavedRegs<(sub CSR_64, R12)>;
+
 def CSR_32EHRet : CalleeSavedRegs<(add EAX, EDX, CSR_32)>;
 def CSR_64EHRet : CalleeSavedRegs<(add RAX, RDX, CSR_64)>;
 
 def CSR_Win64 : CalleeSavedRegs<(add RBX, RBP, RDI, RSI, R12, R13, R14, R15,
                                      (sequence "XMM%u", 6, 15))>;
 
 // The function used by Darwin to obtain the address of a thread-local variable
 // uses rdi to pass a single parameter and rax for the return value. All other

llvm/trunk/lib/Target/X86/X86FastISel.cpp

 /// X86SelectStore - Select and emit code to implement store instructions.
 bool X86FastISel::X86SelectStore(const Instruction *I) {
   // Atomic stores need special handling.
   const StoreInst *S = cast<StoreInst>(I);
 
   if (S->isAtomic())
     return false;
 
+  const Value *PtrV = I->getOperand(1);
+  if (TLI.supportSwiftError()) {
+    // Swifterror values can come from either a function parameter with
+    // swifterror attribute or an alloca with swifterror attribute.
+    if (const Argument *Arg = dyn_cast<Argument>(PtrV)) {
+      if (Arg->hasSwiftErrorAttr())
+        return false;
+    }
+
+    if (const AllocaInst *Alloca = dyn_cast<AllocaInst>(PtrV)) {
+      if (Alloca->isSwiftError())
+        return false;
+    }
+  }
+
   const Value *Val = S->getValueOperand();
   const Value *Ptr = S->getPointerOperand();
 
   MVT VT;
   if (!isTypeLegal(Val->getType(), VT, /*AllowI1=*/true))
     return false;
 
   unsigned Alignment = S->getAlignment();
bool X86FastISel::X86SelectRet(const Instruction *I) {
   const ReturnInst *Ret = cast<ReturnInst>(I);
   const Function &F = *I->getParent()->getParent();
   const X86MachineFunctionInfo *X86MFInfo =
       FuncInfo.MF->getInfo<X86MachineFunctionInfo>();
 
   if (!FuncInfo.CanLowerReturn)
     return false;
 
+  if (TLI.supportSwiftError() &&
+      F.getAttributes().hasAttrSomewhere(Attribute::SwiftError))
+    return false;
+
   if (TLI.supportSplitCSR(FuncInfo.MF))
     return false;
 
   CallingConv::ID CC = F.getCallingConv();
   if (CC != CallingConv::C &&
       CC != CallingConv::Fast &&
       CC != CallingConv::X86_FastCall &&
       CC != CallingConv::X86_64_SysV)

 ///
 bool X86FastISel::X86SelectLoad(const Instruction *I) {
   const LoadInst *LI = cast<LoadInst>(I);
 
   // Atomic loads need special handling.
   if (LI->isAtomic())
     return false;
 
+  const Value *SV = I->getOperand(0);
+  if (TLI.supportSwiftError()) {
+    // Swifterror values can come from either a function parameter with
+    // swifterror attribute or an alloca with swifterror attribute.
+    if (const Argument *Arg = dyn_cast<Argument>(SV)) {
+      if (Arg->hasSwiftErrorAttr())
+        return false;
+    }
+
+    if (const AllocaInst *Alloca = dyn_cast<AllocaInst>(SV)) {
+      if (Alloca->isSwiftError())
+        return false;
+    }
+  }
+
   MVT VT;
   if (!isTypeLegal(LI->getType(), VT, /*AllowI1=*/true))
     return false;
 
   const Value *Ptr = LI->getPointerOperand();
 
   X86AddressMode AM;
   if (!X86SelectAddress(Ptr, AM))
bool X86FastISel::fastLowerArguments() {
   unsigned Idx = 0;
   for (auto const &Arg : F->args()) {
     // The first argument is at index 1.
     ++Idx;
     if (F->getAttributes().hasAttribute(Idx, Attribute::ByVal) ||
         F->getAttributes().hasAttribute(Idx, Attribute::InReg) ||
         F->getAttributes().hasAttribute(Idx, Attribute::StructRet) ||
         F->getAttributes().hasAttribute(Idx, Attribute::SwiftSelf) ||
+        F->getAttributes().hasAttribute(Idx, Attribute::SwiftError) ||
         F->getAttributes().hasAttribute(Idx, Attribute::Nest))
       return false;
 
     Type *ArgTy = Arg.getType();
     if (ArgTy->isStructTy() || ArgTy->isArrayTy() || ArgTy->isVectorTy())
       return false;
 
     EVT ArgVT = TLI.getValueType(DL, ArgTy);
bool X86FastISel::fastLowerCall(CallLoweringInfo &CLI) {
   // x86-32. Special handling for x86-64 is implemented.
   if (IsVarArg && IsWin64)
     return false;
 
   // Don't know about inalloca yet.
   if (CLI.CS && CLI.CS->hasInAllocaArgument())
     return false;
 
+  for (auto Flag : CLI.OutFlags)
+    if (Flag.isSwiftError())
+      return false;
+
   // Fast-isel doesn't know about callee-pop yet.
   if (X86::isCalleePop(CC, Subtarget->is64Bit(), IsVarArg,
                        TM.Options.GuaranteedTailCallOpt))
     return false;
 
   SmallVector<MVT, 16> OutVTs;
   SmallVector<unsigned, 16> ArgRegs;
 

llvm/trunk/lib/Target/X86/X86ISelLowering.h

public:
     bool isNoopAddrSpaceCast(unsigned SrcAS, unsigned DestAS) const override;
 
     bool useLoadStackGuardNode() const override;
     /// \brief Customize the preferred legalization strategy for certain types.
     LegalizeTypeAction getPreferredVectorAction(EVT VT) const override;
 
     bool isIntDivCheap(EVT VT, AttributeSet Attr) const override;
 
+    bool supportSwiftError() const override {
+      return true;
+    }
+
   protected:
     std::pair<const TargetRegisterClass *, uint8_t>
     findRepresentativeClass(const TargetRegisterInfo *TRI,
                             MVT VT) const override;
 
   private:
     /// Keep a reference to the X86Subtarget around so that we can
     /// make the right decision when generating code for different targets.

llvm/trunk/lib/Target/X86/X86ISelLowering.cpp

X86TargetLowering::LowerReturn(SDValue Chain,
   // We saved the argument into a virtual register in the entry block,
   // so now we copy the value out and into %rax/%eax.
   //
   // Checking Function.hasStructRetAttr() here is insufficient because the IR
   // may not have an explicit sret argument. If FuncInfo.CanLowerReturn is
   // false, then an sret argument may be implicitly inserted in the SelDAG. In
   // either case FuncInfo->setSRetReturnReg() will have been called.
   if (unsigned SRetReg = FuncInfo->getSRetReturnReg()) {
-    SDValue Val = DAG.getCopyFromReg(Chain, dl, SRetReg,
+    // When we have both sret and another return value, we should use the
+    // original Chain stored in RetOps[0], instead of the current Chain updated
+    // in the above loop. If we only have sret, RetOps[0] equals to Chain.
+
+    // For the case of sret and another return value, we have
+    //   Chain_0 at the function entry
+    //   Chain_1 = getCopyToReg(Chain_0) in the above loop
+    // If we use Chain_1 in getCopyFromReg, we will have
+    //   Val = getCopyFromReg(Chain_1)
+    //   Chain_2 = getCopyToReg(Chain_1, Val) from below
+
+    // getCopyToReg(Chain_0) will be glued together with
+    // getCopyToReg(Chain_1, Val) into Unit A, getCopyFromReg(Chain_1) will be
+    // in Unit B, and we will have cyclic dependency between Unit A and Unit B:
+    //   Data dependency from Unit B to Unit A due to usage of Val in
+    //     getCopyToReg(Chain_1, Val)
+    //   Chain dependency from Unit A to Unit B
+
+    // So here, we use RetOps[0] (i.e Chain_0) for getCopyFromReg.
+    SDValue Val = DAG.getCopyFromReg(RetOps[0], dl, SRetReg,
                                      getPointerTy(MF.getDataLayout()));
 
     unsigned RetValReg
         = (Subtarget.is64Bit() && !Subtarget.isTarget64BitILP32()) ?
           X86::RAX : X86::EAX;
     Chain = DAG.getCopyToReg(Chain, dl, RetValReg, Val, Flag);
     Flag = Chain.getValue(1);
 

llvm/trunk/lib/Target/X86/X86RegisterInfo.cpp

default:
     break;
   }
 
   if (Is64Bit) {
     if (IsWin64)
       return CSR_Win64_SaveList;
     if (CallsEHReturn)
       return CSR_64EHRet_SaveList;
+    if (Subtarget.getTargetLowering()->supportSwiftError() &&
+        MF->getFunction()->getAttributes().hasAttrSomewhere(
+            Attribute::SwiftError))
+      return CSR_64_SwiftError_SaveList;
     return CSR_64_SaveList;
   }
   if (CallsEHReturn)
     return CSR_32EHRet_SaveList;
   return CSR_32_SaveList;
 }
 
 const MCPhysReg *X86RegisterInfo::getCalleeSavedRegsViaCopy(
default:
       break;
   }
 
   // Unlike getCalleeSavedRegs(), we don't have MMI so we can't check
   // callsEHReturn().
   if (Is64Bit) {
     if (IsWin64)
       return CSR_Win64_RegMask;
+    if (Subtarget.getTargetLowering()->supportSwiftError() &&
+        MF.getFunction()->getAttributes().hasAttrSomewhere(
+            Attribute::SwiftError))
+      return CSR_64_SwiftError_RegMask;
     return CSR_64_RegMask;
   }
   return CSR_32_RegMask;
 }
 
 const uint32_t*
 X86RegisterInfo::getNoPreservedMask() const {
   return CSR_NoRegs_RegMask;

llvm/trunk/test/CodeGen/AArch64/swifterror.ll

+; RUN: llc -verify-machineinstrs < %s -mtriple=aarch64-apple-ios -disable-post-ra | FileCheck --check-prefix=CHECK-APPLE %s
+; RUN: llc -verify-machineinstrs -O0 < %s -mtriple=aarch64-apple-ios -disable-post-ra | FileCheck --check-prefix=CHECK-O0 %s
+
+declare i8* @malloc(i64)
+declare void @free(i8*)
+%swift_error = type {i64, i8}
+
+; This tests the basic usage of a swifterror parameter. "foo" is the function
+; that takes a swifterror parameter and "caller" is the caller of "foo".
+define float @foo(%swift_error** swifterror %error_ptr_ref) {
+; CHECK-APPLE-LABEL: foo:
+; CHECK-APPLE: orr w0, wzr, #0x10
+; CHECK-APPLE: malloc
+; CHECK-APPLE: orr [[ID:w[0-9]+]], wzr, #0x1
+; CHECK-APPLE: strb [[ID]], [x0, #8]
+; CHECK-APPLE: mov x19, x0
+; CHECK-APPLE-NOT: x19
+
+; CHECK-O0-LABEL: foo:
+; CHECK-O0: orr w{{.*}}, wzr, #0x10
+; CHECK-O0: malloc
+; CHECK-O0: mov [[ID2:x[0-9]+]], x0
+; CHECK-O0: orr [[ID:w[0-9]+]], wzr, #0x1
+; CHECK-O0: strb [[ID]], [x0, #8]
+; CHECK-O0: mov x19, [[ID2]]
+; CHECK-O0-NOT: x19
+entry:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+  ret float 1.0
+}
+
+; "caller" calls "foo" that takes a swifterror parameter.
+define float @caller(i8* %error_ref) {
+; CHECK-APPLE-LABEL: caller:
+; CHECK-APPLE: mov [[ID:x[0-9]+]], x0
+; CHECK-APPLE: mov x19, xzr
+; CHECK-APPLE: bl {{.*}}foo
+; CHECK-APPLE: cbnz x19
+; Access part of the error object and save it to error_ref
+; CHECK-APPLE: ldrb [[CODE:w[0-9]+]], [x19, #8]
+; CHECK-APPLE: strb [[CODE]], [{{.*}}[[ID]]]
+; CHECK-APPLE: mov x0, x19
+; CHECK_APPLE: bl {{.*}}free
+
+; CHECK-O0-LABEL: caller:
+; CHECK-O0: mov x19
+; CHECK-O0: bl {{.*}}foo
+; CHECK-O0: mov [[ID:x[0-9]+]], x19
+; CHECK-O0: cbnz [[ID]]
+entry:
+  %error_ptr_ref = alloca swifterror %swift_error*
+  store %swift_error* null, %swift_error** %error_ptr_ref
+  %call = call float @foo(%swift_error** swifterror %error_ptr_ref)
+  %error_from_foo = load %swift_error*, %swift_error** %error_ptr_ref
+  %had_error_from_foo = icmp ne %swift_error* %error_from_foo, null
+  %tmp = bitcast %swift_error* %error_from_foo to i8*
+  br i1 %had_error_from_foo, label %handler, label %cont
+cont:
+  %v1 = getelementptr inbounds %swift_error, %swift_error* %error_from_foo, i64 0, i32 1
+  %t = load i8, i8* %v1
+  store i8 %t, i8* %error_ref
+  br label %handler
+handler:
+  call void @free(i8* %tmp)
+  ret float 1.0
+}
+
+; "caller2" is the caller of "foo", it calls "foo" inside a loop.
+define float @caller2(i8* %error_ref) {
+; CHECK-APPLE-LABEL: caller2:
+; CHECK-APPLE: mov [[ID:x[0-9]+]], x0
+; CHECK-APPLE: fmov [[CMP:s[0-9]+]], #1.0
+; CHECK-APPLE: mov x19, xzr
+; CHECK-APPLE: bl {{.*}}foo
+; CHECK-APPLE: cbnz x19
+; CHECK-APPLE: fcmp s0, [[CMP]]
+; CHECK-APPLE: b.le
+; Access part of the error object and save it to error_ref
+; CHECK-APPLE: ldrb [[CODE:w[0-9]+]], [x19, #8]
+; CHECK-APPLE: strb [[CODE]], [{{.*}}[[ID]]]
+; CHECK-APPLE: mov x0, x19
+; CHECK_APPLE: bl {{.*}}free
+
+; CHECK-O0-LABEL: caller2:
+; CHECK-O0: mov x19
+; CHECK-O0: bl {{.*}}foo
+; CHECK-O0: mov [[ID:x[0-9]+]], x19
+; CHECK-O0: cbnz [[ID]]
+entry:
+  %error_ptr_ref = alloca swifterror %swift_error*
+  br label %bb_loop
+bb_loop:
+  store %swift_error* null, %swift_error** %error_ptr_ref
+  %call = call float @foo(%swift_error** swifterror %error_ptr_ref)
+  %error_from_foo = load %swift_error*, %swift_error** %error_ptr_ref
+  %had_error_from_foo = icmp ne %swift_error* %error_from_foo, null
+  %tmp = bitcast %swift_error* %error_from_foo to i8*
+  br i1 %had_error_from_foo, label %handler, label %cont
+cont:
+  %cmp = fcmp ogt float %call, 1.000000e+00
+  br i1 %cmp, label %bb_end, label %bb_loop
+bb_end:
+  %v1 = getelementptr inbounds %swift_error, %swift_error* %error_from_foo, i64 0, i32 1
+  %t = load i8, i8* %v1
+  store i8 %t, i8* %error_ref
+  br label %handler
+handler:
+  call void @free(i8* %tmp)
+  ret float 1.0
+}
+
+; "foo_if" is a function that takes a swifterror parameter, it sets swifterror
+; under a certain condition.
+define float @foo_if(%swift_error** swifterror %error_ptr_ref, i32 %cc) {
+; CHECK-APPLE-LABEL: foo_if:
+; CHECK-APPLE: cbz w0
+; CHECK-APPLE: orr w0, wzr, #0x10
+; CHECK-APPLE: malloc
+; CHECK-APPLE: orr [[ID:w[0-9]+]], wzr, #0x1
+; CHECK-APPLE: strb [[ID]], [x0, #8]
+; CHECK-APPLE: mov x19, x0
+; CHECK-APPLE-NOT: x19
+; CHECK-APPLE: ret
+
+; CHECK-O0-LABEL: foo_if:
+; spill x19
+; CHECK-O0: str x19
+; CHECK-O0: cbz w0
+; CHECK-O0: orr w{{.*}}, wzr, #0x10
+; CHECK-O0: malloc
+; CHECK-O0: mov [[ID:x[0-9]+]], x0
+; CHECK-O0: orr [[ID2:w[0-9]+]], wzr, #0x1
+; CHECK-O0: strb [[ID2]], [x0, #8]
+; CHECK-O0: mov x19, [[ID]]
+; CHECK-O0: ret
+; reload from stack
+; CHECK-O0: ldr x19
+; CHECK-O0: ret
+entry:
+  %cond = icmp ne i32 %cc, 0
+  br i1 %cond, label %gen_error, label %normal
+
+gen_error:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+  ret float 1.0
+
+normal:
+  ret float 0.0
+}
+
+; "foo_loop" is a function that takes a swifterror parameter, it sets swifterror
+; under a certain condition inside a loop.
+define float @foo_loop(%swift_error** swifterror %error_ptr_ref, i32 %cc, float %cc2) {
+; CHECK-APPLE-LABEL: foo_loop:
+; CHECK-APPLE: mov x0, x19
+; CHECK-APPLE: cbz
+; CHECK-APPLE: orr w0, wzr, #0x10
+; CHECK-APPLE: malloc
+; CHECK-APPLE: strb w{{.*}}, [x0, #8]
+; CHECK-APPLE: fcmp
+; CHECK-APPLE: b.le
+; CHECK-APPLE: mov x19, x0
+; CHECK-APPLE: ret
+
+; CHECK-O0-LABEL: foo_loop:
+; spill x19
+; CHECK-O0: str x19
+; CHECk-O0: cbz
+; CHECK-O0: orr w{{.*}}, wzr, #0x10
+; CHECK-O0: malloc
+; CHECK-O0: mov [[ID:x[0-9]+]], x0
+; CHECK-O0: strb w{{.*}}, [{{.*}}[[ID]], #8]
+; spill x0
+; CHECK-O0: str x0
+; CHECK-O0: fcmp
+; CHECK-O0: b.le
+; reload from stack
+; CHECK-O0: ldr x19
+; CHECK-O0: ret
+entry:
+  br label %bb_loop
+
+bb_loop:
+  %cond = icmp ne i32 %cc, 0
+  br i1 %cond, label %gen_error, label %bb_cont
+
+gen_error:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+  br label %bb_cont
+
+bb_cont:
+  %cmp = fcmp ogt float %cc2, 1.000000e+00
+  br i1 %cmp, label %bb_end, label %bb_loop
+bb_end:
+  ret float 0.0
+}
+
+%struct.S = type { i32, i32, i32, i32, i32, i32 }
+
+; "foo_sret" is a function that takes a swifterror parameter, it also has a sret
+; parameter.
+define void @foo_sret(%struct.S* sret %agg.result, i32 %val1, %swift_error** swifterror %error_ptr_ref) {
+; CHECK-APPLE-LABEL: foo_sret:
+; CHECK-APPLE: mov [[SRET:x[0-9]+]], x8
+; CHECK-APPLE: orr w0, wzr, #0x10
+; CHECK-APPLE: malloc
+; CHECK-APPLE: orr [[ID:w[0-9]+]], wzr, #0x1
+; CHECK-APPLE: strb [[ID]], [x0, #8]
+; CHECK-APPLE: str w{{.*}}, [{{.*}}[[SRET]], #4]
+; CHECK-APPLE: mov x19, x0
+; CHECK-APPLE-NOT: x19
+
+; CHECK-O0-LABEL: foo_sret:
+; CHECK-O0: orr w{{.*}}, wzr, #0x10
+; spill x8
+; CHECK-O0-DAG: str x8
+; spill x19
+; CHECK-O0-DAG: str x19
+; CHECK-O0: malloc
+; CHECK-O0: orr [[ID:w[0-9]+]], wzr, #0x1
+; CHECK-O0: strb [[ID]], [x0, #8]
+; reload from stack
+; CHECK-O0: ldr [[SRET:x[0-9]+]]
+; CHECK-O0: str w{{.*}}, [{{.*}}[[SRET]], #4]
+; CHECK-O0: mov x19
+; CHECK-O0-NOT: x19
+entry:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+  %v2 = getelementptr inbounds %struct.S, %struct.S* %agg.result, i32 0, i32 1
+  store i32 %val1, i32* %v2
+  ret void
+}
+
+; "caller3" calls "foo_sret" that takes a swifterror parameter.
+define float @caller3(i8* %error_ref) {
+; CHECK-APPLE-LABEL: caller3:
+; CHECK-APPLE: mov [[ID:x[0-9]+]], x0
+; CHECK-APPLE: mov x19, xzr
+; CHECK-APPLE: bl {{.*}}foo_sret
+; CHECK-APPLE: cbnz x19
+; Access part of the error object and save it to error_ref
+; CHECK-APPLE: ldrb [[CODE:w[0-9]+]], [x19, #8]
+; CHECK-APPLE: strb [[CODE]], [{{.*}}[[ID]]]
+; CHECK-APPLE: mov x0, x19
+; CHECK_APPLE: bl {{.*}}free
+
+; CHECK-O0-LABEL: caller3:
+; spill x0
+; CHECK-O0: str x0
+; CHECK-O0: mov x19
+; CHECK-O0: bl {{.*}}foo_sret
+; CHECK-O0: mov [[ID2:x[0-9]+]], x19
+; CHECK-O0: cbnz [[ID2]]
+; Access part of the error object and save it to error_ref
+; reload from stack
+; CHECK-O0: ldrb [[CODE:w[0-9]+]]
+; CHECK-O0: ldr [[ID:x[0-9]+]]
+; CHECK-O0: strb [[CODE]], [{{.*}}[[ID]]]
+; CHECK_O0: bl {{.*}}free
+entry:
+  %s = alloca %struct.S, align 8
+  %error_ptr_ref = alloca swifterror %swift_error*
+  store %swift_error* null, %swift_error** %error_ptr_ref
+  call void @foo_sret(%struct.S* sret %s, i32 1, %swift_error** swifterror %error_ptr_ref)
+  %error_from_foo = load %swift_error*, %swift_error** %error_ptr_ref
+  %had_error_from_foo = icmp ne %swift_error* %error_from_foo, null
+  %tmp = bitcast %swift_error* %error_from_foo to i8*
+  br i1 %had_error_from_foo, label %handler, label %cont
+cont:
+  %v1 = getelementptr inbounds %swift_error, %swift_error* %error_from_foo, i64 0, i32 1
+  %t = load i8, i8* %v1
+  store i8 %t, i8* %error_ref
+  br label %handler
+handler:
+  call void @free(i8* %tmp)
+  ret float 1.0
+}
+
+; "foo_vararg" is a function that takes a swifterror parameter, it also has
+; variable number of arguments.
+declare void @llvm.va_start(i8*) nounwind
+define float @foo_vararg(%swift_error** swifterror %error_ptr_ref, ...) {
+; CHECK-APPLE-LABEL: foo_vararg:
+; CHECK-APPLE: orr w0, wzr, #0x10
+; CHECK-APPLE: malloc
+; CHECK-APPLE: orr [[ID:w[0-9]+]], wzr, #0x1
+; CHECK-FIXMEAPPLE: add [[ARGS:x[0-9]+]], [[TMP:x[0-9]+]], #16
+; CHECK-APPLE: strb [[ID]], [x0, #8]
+
+; First vararg
+; CHECK-FIXMEAPPLE-DAG: orr {{x[0-9]+}}, [[ARGS]], #0x8
+; CHECK-FIXMEAPPLE-DAG: ldr {{w[0-9]+}}, [{{.*}}[[TMP]], #16]
+; CHECK-APPLE: add {{x[0-9]+}}, {{x[0-9]+}}, #8
+; Second vararg
+; CHECK-APPLE: ldr {{w[0-9]+}}, [{{x[0-9]+}}]
+; CHECK-APPLE: add {{x[0-9]+}}, {{x[0-9]+}}, #8
+; Third vararg
+; CHECK-APPLE: ldr {{w[0-9]+}}, [{{x[0-9]+}}]
+
+; CHECK-APPLE: mov x19, x0
+; CHECK-APPLE-NOT: x19
+entry:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+
+  %args = alloca i8*, align 8
+  %a10 = alloca i32, align 4
+  %a11 = alloca i32, align 4
+  %a12 = alloca i32, align 4
+  %v10 = bitcast i8** %args to i8*
+  call void @llvm.va_start(i8* %v10)
+  %v11 = va_arg i8** %args, i32
+  store i32 %v11, i32* %a10, align 4
+  %v12 = va_arg i8** %args, i32
+  store i32 %v12, i32* %a11, align 4
+  %v13 = va_arg i8** %args, i32
+  store i32 %v13, i32* %a12, align 4
+
+  ret float 1.0
+}
+
+; "caller4" calls "foo_vararg" that takes a swifterror parameter.
+define float @caller4(i8* %error_ref) {
+; CHECK-APPLE-LABEL: caller4:
+
+; CHECK-APPLE: mov [[ID:x[0-9]+]], x0
+; CHECK-APPLE: stp {{x[0-9]+}}, {{x[0-9]+}}, [sp, #8]
+; CHECK-APPLE: str {{x[0-9]+}}, [sp]
+
+; CHECK-APPLE: mov x19, xzr
+; CHECK-APPLE: bl {{.*}}foo_vararg
+; CHECK-APPLE: cbnz x19
+; Access part of the error object and save it to error_ref
+; CHECK-APPLE: ldrb [[CODE:w[0-9]+]], [x19, #8]
+; CHECK-APPLE: strb [[CODE]], [{{.*}}[[ID]]]
+; CHECK-APPLE: mov x0, x19
+; CHECK_APPLE: bl {{.*}}free
+entry:
+  %error_ptr_ref = alloca swifterror %swift_error*
+  store %swift_error* null, %swift_error** %error_ptr_ref
+
+  %a10 = alloca i32, align 4
+  %a11 = alloca i32, align 4
+  %a12 = alloca i32, align 4
+  store i32 10, i32* %a10, align 4
+  store i32 11, i32* %a11, align 4
+  store i32 12, i32* %a12, align 4
+  %v10 = load i32, i32* %a10, align 4
+  %v11 = load i32, i32* %a11, align 4
+  %v12 = load i32, i32* %a12, align 4
+
+  %call = call float (%swift_error**, ...) @foo_vararg(%swift_error** swifterror %error_ptr_ref, i32 %v10, i32 %v11, i32 %v12)
+  %error_from_foo = load %swift_error*, %swift_error** %error_ptr_ref
+  %had_error_from_foo = icmp ne %swift_error* %error_from_foo, null
+  %tmp = bitcast %swift_error* %error_from_foo to i8*
+  br i1 %had_error_from_foo, label %handler, label %cont
+
+cont:
+  %v1 = getelementptr inbounds %swift_error, %swift_error* %error_from_foo, i64 0, i32 1
+  %t = load i8, i8* %v1
+  store i8 %t, i8* %error_ref
+  br label %handler
+handler:
+  call void @free(i8* %tmp)
+  ret float 1.0
+}

llvm/trunk/test/CodeGen/ARM/swifterror.ll

+; RUN: llc -verify-machineinstrs < %s -mtriple=armv7-apple-ios | FileCheck --check-prefix=CHECK-APPLE %s
+; RUN: llc -verify-machineinstrs -O0 < %s -mtriple=armv7-apple-ios | FileCheck --check-prefix=CHECK-O0 %s
+
+declare i8* @malloc(i64)
+declare void @free(i8*)
+%swift_error = type { i64, i8 }
+%struct.S = type { i32, i32, i32, i32, i32, i32 }
+
+; This tests the basic usage of a swifterror parameter. "foo" is the function
+; that takes a swifterror parameter and "caller" is the caller of "foo".
+define float @foo(%swift_error** swifterror %error_ptr_ref) {
+; CHECK-APPLE-LABEL: foo:
+; CHECK-APPLE: mov r0, #16
+; CHECK-APPLE: malloc
+; CHECK-APPLE-DAG: mov [[ID:r[0-9]+]], #1
+; CHECK-APPLE-DAG: mov r6, r{{.*}}
+; CHECK-APPLE-DAG: strb [[ID]], [r{{.*}}, #8]
+
+; CHECK-O0-LABEL: foo:
+; CHECK-O0: mov r{{.*}}, #16
+; CHECK-O0: malloc
+; CHECK-O0: mov [[ID2:r[0-9]+]], r0
+; CHECK-O0: mov [[ID:r[0-9]+]], #1
+; CHECK-O0: strb [[ID]], [r0, #8]
+; CHECK-O0: mov r6, [[ID2]]
+entry:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+  ret float 1.0
+}
+
+; "caller" calls "foo" that takes a swifterror parameter.
+define float @caller(i8* %error_ref) {
+; CHECK-APPLE-LABEL: caller:
+; CHECK-APPLE-DAG: mov [[ID:r[0-9]+]], r0
+; CHECK-APPLE-DAG: mov r6, #0
+; CHECK-APPLE: bl {{.*}}foo
+; CHECK-APPLE: cmp r6, #0
+; Access part of the error object and save it to error_ref
+; CHECK-APPLE: ldrbeq [[CODE:r[0-9]+]], [r6, #8]
+; CHECK-APPLE: strbeq [[CODE]], [{{.*}}[[ID]]]
+; CHECK-APPLE: mov r0, r6
+; CHECK_APPLE: bl {{.*}}free
+
+; CHECK-O0-LABEL: caller:
+; spill r0
+; CHECK-O0-DAG: str r0,
+; CHECK-O0-DAG: mov r6, #0
+; CHECK-O0: bl {{.*}}foo
+; CHECK-O0: mov r{{.*}}, r6
+; CHECK-O0: bne
+; CHECK-O0: ldrb [[CODE:r[0-9]+]], [r0, #8]
+; reload r0
+; CHECK-O0: ldr [[ID:r[0-9]+]],
+; CHECK-O0: strb [[CODE]], [{{.*}}[[ID]]]
+; CHECK-O0: mov r0,
+; CHECK-O0: free
+entry:
+  %error_ptr_ref = alloca swifterror %swift_error*
+  store %swift_error* null, %swift_error** %error_ptr_ref
+  %call = call float @foo(%swift_error** swifterror %error_ptr_ref)
+  %error_from_foo = load %swift_error*, %swift_error** %error_ptr_ref
+  %had_error_from_foo = icmp ne %swift_error* %error_from_foo, null
+  %tmp = bitcast %swift_error* %error_from_foo to i8*
+  br i1 %had_error_from_foo, label %handler, label %cont
+cont:
+  %v1 = getelementptr inbounds %swift_error, %swift_error* %error_from_foo, i64 0, i32 1
+  %t = load i8, i8* %v1
+  store i8 %t, i8* %error_ref
+  br label %handler
+handler:
+  call void @free(i8* %tmp)
+  ret float 1.0
+}
+
+; "caller2" is the caller of "foo", it calls "foo" inside a loop.
+define float @caller2(i8* %error_ref) {
+; CHECK-APPLE-LABEL: caller2:
+; CHECK-APPLE-DAG: mov [[ID:r[0-9]+]], r0
+; CHECK-APPLE-DAG: mov r6, #0
+; CHECK-APPLE: bl {{.*}}foo
+; CHECK-APPLE: cmp r6, #0
+; CHECK-APPLE: bne
+; Access part of the error object and save it to error_ref
+; CHECK-APPLE: ldrb [[CODE:r[0-9]+]], [r6, #8]
+; CHECK-APPLE: strb [[CODE]], [{{.*}}[[ID]]]
+; CHECK-APPLE: mov r0, r6
+; CHECK_APPLE: bl {{.*}}free
+
+; CHECK-O0-LABEL: caller2:
+; spill r0
+; CHECK-O0-DAG: str r0,
+; CHECK-O0-DAG: mov r6, #0
+; CHECK-O0: bl {{.*}}foo
+; CHECK-O0: mov r{{.*}}, r6
+; CHECK-O0: bne
+; CHECK-O0: ble
+; CHECK-O0: ldrb [[CODE:r[0-9]+]], [r0, #8]
+; reload r0
+; CHECK-O0: ldr [[ID:r[0-9]+]],
+; CHECK-O0: strb [[CODE]], [{{.*}}[[ID]]]
+; CHECK-O0: mov r0,
+; CHECK-O0: free
+entry:
+  %error_ptr_ref = alloca swifterror %swift_error*
+  br label %bb_loop
+bb_loop:
+  store %swift_error* null, %swift_error** %error_ptr_ref
+  %call = call float @foo(%swift_error** swifterror %error_ptr_ref)
+  %error_from_foo = load %swift_error*, %swift_error** %error_ptr_ref
+  %had_error_from_foo = icmp ne %swift_error* %error_from_foo, null
+  %tmp = bitcast %swift_error* %error_from_foo to i8*
+  br i1 %had_error_from_foo, label %handler, label %cont
+cont:
+  %cmp = fcmp ogt float %call, 1.000000e+00
+  br i1 %cmp, label %bb_end, label %bb_loop
+bb_end:
+  %v1 = getelementptr inbounds %swift_error, %swift_error* %error_from_foo, i64 0, i32 1
+  %t = load i8, i8* %v1
+  store i8 %t, i8* %error_ref
+  br label %handler
+handler:
+  call void @free(i8* %tmp)
+  ret float 1.0
+}
+
+; "foo_if" is a function that takes a swifterror parameter, it sets swifterror
+; under a certain condition.
+define float @foo_if(%swift_error** swifterror %error_ptr_ref, i32 %cc) {
+; CHECK-APPLE-LABEL: foo_if:
+; CHECK-APPLE: cmp r0, #0
+; CHECK-APPLE: eq
+; CHECK-APPLE: mov r0, #16
+; CHECK-APPLE: malloc
+; CHECK-APPLE: mov [[ID:r[0-9]+]], #1
+; CHECK-APPLE-DAG: mov r6, r{{.*}}
+; CHECK-APPLE-DAG: strb [[ID]], [r{{.*}}, #8]
+
+; CHECK-O0-LABEL: foo_if:
+; CHECK-O0: cmp r0, #0
+; spill to stack
+; CHECK-O0: str r6
+; CHECK-O0: beq
+; CHECK-O0: mov r0, #16
+; CHECK-O0: malloc
+; CHECK-O0: mov [[ID:r[0-9]+]], r0
+; CHECK-O0: mov [[ID2:[a-z0-9]+]], #1
+; CHECK-O0: strb [[ID2]], [r0, #8]
+; CHECK-O0: mov r6, [[ID]]
+; reload from stack
+; CHECK-O0: ldr r6
+entry:
+  %cond = icmp ne i32 %cc, 0
+  br i1 %cond, label %gen_error, label %normal
+
+gen_error:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+  ret float 1.0
+
+normal:
+  ret float 0.0
+}
+
+; "foo_loop" is a function that takes a swifterror parameter, it sets swifterror
+; under a certain condition inside a loop.
+define float @foo_loop(%swift_error** swifterror %error_ptr_ref, i32 %cc, float %cc2) {
+; CHECK-APPLE-LABEL: foo_loop:
+; CHECK-APPLE: mov [[CODE:r[0-9]+]], r0
+; swifterror is kept in a register
+; CHECK-APPLE: mov [[ID:r[0-9]+]], r6
+; CHECK-APPLE: cmp [[CODE]], #0
+; CHECK-APPLE: beq
+; CHECK-APPLE: mov r0, #16
+; CHECK-APPLE: malloc
+; CHECK-APPLE: strb r{{.*}}, [{{.*}}[[ID]], #8]
+; CHECK-APPLE: ble
+; CHECK-APPLE: mov r6, [[ID]]
+
+; CHECK-O0-LABEL: foo_loop:
+; CHECK-O0: mov r{{.*}}, r6
+; CHECK-O0: cmp r{{.*}}, #0
+; CHECK-O0: beq
+; CHECK-O0-DAG: movw r{{.*}}, #1
+; CHECK-O0-DAG: mov r{{.*}}, #16
+; CHECK-O0: malloc
+; CHECK-O0-DAG: mov [[ID:r[0-9]+]], r0
+; CHECK-O0-DAG: ldr [[ID2:r[0-9]+]], [sp{{.*}}]
+; CHECK-O0: strb [[ID2]], [{{.*}}[[ID]], #8]
+; spill r0
+; CHECK-O0: str r0, [sp{{.*}}]
+; CHECK-O0: vcmpe
+; CHECK-O0: ble
+; reload from stack
+; CHECK-O0: ldr r6
+entry:
+  br label %bb_loop
+
+bb_loop:
+  %cond = icmp ne i32 %cc, 0
+  br i1 %cond, label %gen_error, label %bb_cont
+
+gen_error:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+  br label %bb_cont
+
+bb_cont:
+  %cmp = fcmp ogt float %cc2, 1.000000e+00
+  br i1 %cmp, label %bb_end, label %bb_loop
+bb_end:
+  ret float 0.0
+}
+
+; "foo_sret" is a function that takes a swifterror parameter, it also has a sret
+; parameter.
+define void @foo_sret(%struct.S* sret %agg.result, i32 %val1, %swift_error** swifterror %error_ptr_ref) {
+; CHECK-APPLE-LABEL: foo_sret:
+; CHECK-APPLE: mov [[SRET:r[0-9]+]], r0
+; CHECK-APPLE: mov r0, #16
+; CHECK-APPLE: malloc
+; CHECK-APPLE: mov [[REG:r[0-9]+]], #1
+; CHECK-APPLE-DAG: mov r6, r0
+; CHECK-APPLE-DAG: strb [[REG]], [r0, #8]
+; CHECK-APPLE-DAG: str r{{.*}}, [{{.*}}[[SRET]], #4]
+
+; CHECK-O0-LABEL: foo_sret:
+; CHECK-O0: mov r{{.*}}, #16
+; spill to stack: sret and val1
+; CHECK-O0-DAG: str r0
+; CHECK-O0-DAG: str r1
+; CHECK-O0: malloc
+; CHECK-O0: mov [[ID:r[0-9]+]], #1
+; CHECK-O0: strb [[ID]], [r0, #8]
+; reload from stack: sret and val1
+; CHECK-O0: ldr
+; CHECK-O0: ldr
+; CHECK-O0: str r{{.*}}, [{{.*}}, #4]
+; CHECK-O0: mov r6
+entry:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+  %v2 = getelementptr inbounds %struct.S, %struct.S* %agg.result, i32 0, i32 1
+  store i32 %val1, i32* %v2
+  ret void
+}
+
+; "caller3" calls "foo_sret" that takes a swifterror parameter.
+define float @caller3(i8* %error_ref) {
+; CHECK-APPLE-LABEL: caller3:
+; CHECK-APPLE: mov [[ID:r[0-9]+]], r0
+; CHECK-APPLE: mov r6, #0
+; CHECK-APPLE: bl {{.*}}foo_sret
+; CHECK-APPLE: cmp r6, #0
+; Access part of the error object and save it to error_ref
+; CHECK-APPLE: ldrbeq [[CODE:r[0-9]+]], [r6, #8]
+; CHECK-APPLE: strbeq [[CODE]], [{{.*}}[[ID]]]
+; CHECK-APPLE: mov r0, r6
+; CHECK_APPLE: bl {{.*}}free
+
+; CHECK-O0-LABEL: caller3:
+; CHECK-O0-DAG: mov r6, #0
+; CHECK-O0-DAG: mov r0
+; CHECK-O0-DAG: mov r1
+; CHECK-O0: bl {{.*}}foo_sret
+; CHECK-O0: mov [[ID2:r[0-9]+]], r6
+; CHECK-O0: cmp [[ID2]]
+; CHECK-O0: bne
+; Access part of the error object and save it to error_ref
+; CHECK-O0: ldrb [[CODE:r[0-9]+]]
+; CHECK-O0: ldr [[ID:r[0-9]+]]
+; CHECK-O0: strb [[CODE]], [{{.*}}[[ID]]]
+; CHECK-O0: mov r0,
+; CHECK_O0: bl {{.*}}free
+entry:
+  %s = alloca %struct.S, align 8
+  %error_ptr_ref = alloca swifterror %swift_error*
+  store %swift_error* null, %swift_error** %error_ptr_ref
+  call void @foo_sret(%struct.S* sret %s, i32 1, %swift_error** swifterror %error_ptr_ref)
+  %error_from_foo = load %swift_error*, %swift_error** %error_ptr_ref
+  %had_error_from_foo = icmp ne %swift_error* %error_from_foo, null
+  %tmp = bitcast %swift_error* %error_from_foo to i8*
+  br i1 %had_error_from_foo, label %handler, label %cont
+cont:
+  %v1 = getelementptr inbounds %swift_error, %swift_error* %error_from_foo, i64 0, i32 1
+  %t = load i8, i8* %v1
+  store i8 %t, i8* %error_ref
+  br label %handler
+handler:
+  call void @free(i8* %tmp)
+  ret float 1.0
+}
+
+; "foo_vararg" is a function that takes a swifterror parameter, it also has
+; variable number of arguments.
+declare void @llvm.va_start(i8*) nounwind
+define float @foo_vararg(%swift_error** swifterror %error_ptr_ref, ...) {
+; CHECK-APPLE-LABEL: foo_vararg:
+; CHECK-APPLE: mov r0, #16
+; CHECK-APPLE: malloc
+; CHECK-APPLE: mov [[REG:r[0-9]+]], r0
+; CHECK-APPLE: mov [[ID:r[0-9]+]], #1
+; CHECK-APPLE-DAG: strb [[ID]], [{{.*}}[[REG]], #8]
+; CHECK-APPLE-DAG: mov r6, [[REG]]
+
+entry:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+
+  %args = alloca i8*, align 8
+  %a10 = alloca i32, align 4
+  %a11 = alloca i32, align 4
+  %a12 = alloca i32, align 4
+  %v10 = bitcast i8** %args to i8*
+  call void @llvm.va_start(i8* %v10)
+  %v11 = va_arg i8** %args, i32
+  store i32 %v11, i32* %a10, align 4
+  %v12 = va_arg i8** %args, i32
+  store i32 %v12, i32* %a11, align 4
+  %v13 = va_arg i8** %args, i32
+  store i32 %v13, i32* %a12, align 4
+
+  ret float 1.0
+}
+
+; "caller4" calls "foo_vararg" that takes a swifterror parameter.
+define float @caller4(i8* %error_ref) {
+; CHECK-APPLE-LABEL: caller4:
+; CHECK-APPLE: mov [[ID:r[0-9]+]], r0
+; CHECK-APPLE: mov r6, #0
+; CHECK-APPLE: bl {{.*}}foo_vararg
+; CHECK-APPLE: cmp r6, #0
+; Access part of the error object and save it to error_ref
+; CHECK-APPLE: ldrbeq [[CODE:r[0-9]+]], [r6, #8]
+; CHECK-APPLE: strbeq [[CODE]], [{{.*}}[[ID]]]
+; CHECK-APPLE: mov r0, r6
+; CHECK_APPLE: bl {{.*}}free
+entry:
+  %error_ptr_ref = alloca swifterror %swift_error*
+  store %swift_error* null, %swift_error** %error_ptr_ref
+
+  %a10 = alloca i32, align 4
+  %a11 = alloca i32, align 4
+  %a12 = alloca i32, align 4
+  store i32 10, i32* %a10, align 4
+  store i32 11, i32* %a11, align 4
+  store i32 12, i32* %a12, align 4
+  %v10 = load i32, i32* %a10, align 4
+  %v11 = load i32, i32* %a11, align 4
+  %v12 = load i32, i32* %a12, align 4
+
+  %call = call float (%swift_error**, ...) @foo_vararg(%swift_error** swifterror %error_ptr_ref, i32 %v10, i32 %v11, i32 %v12)
+  %error_from_foo = load %swift_error*, %swift_error** %error_ptr_ref
+  %had_error_from_foo = icmp ne %swift_error* %error_from_foo, null
+  %tmp = bitcast %swift_error* %error_from_foo to i8*
+  br i1 %had_error_from_foo, label %handler, label %cont
+
+cont:
+  %v1 = getelementptr inbounds %swift_error, %swift_error* %error_from_foo, i64 0, i32 1
+  %t = load i8, i8* %v1
+  store i8 %t, i8* %error_ref
+  br label %handler
+handler:
+  call void @free(i8* %tmp)
+  ret float 1.0
+}

llvm/trunk/test/CodeGen/X86/swifterror.ll

+; RUN: llc -verify-machineinstrs < %s -mtriple=x86_64-apple-darwin | FileCheck --check-prefix=CHECK-APPLE %s
+; RUN: llc -verify-machineinstrs -O0 < %s -mtriple=x86_64-apple-darwin | FileCheck --check-prefix=CHECK-O0 %s
+
+declare i8* @malloc(i64)
+declare void @free(i8*)
+%swift_error = type {i64, i8}
+
+; This tests the basic usage of a swifterror parameter. "foo" is the function
+; that takes a swifterror parameter and "caller" is the caller of "foo".
+define float @foo(%swift_error** swifterror %error_ptr_ref) {
+; CHECK-APPLE-LABEL: foo:
+; CHECK-APPLE: movl $16, %edi
+; CHECK-APPLE: malloc
+; CHECK-APPLE: movb $1, 8(%rax)
+; CHECK-APPLE: movq %rax, %r12
+
+; CHECK-O0-LABEL: foo:
+; CHECK-O0: movl $16
+; CHECK-O0: malloc
+; CHECK-O0: movb $1, 8(%rax)
+; CHECK-O0: movq %{{.*}}, %r12
+entry:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+  ret float 1.0
+}
+
+; "caller" calls "foo" that takes a swifterror parameter.
+define float @caller(i8* %error_ref) {
+; CHECK-APPLE-LABEL: caller:
+; CHECK-APPLE: xorl %r12d, %r12d
+; CHECK-APPLE: callq {{.*}}foo
+; CHECK-APPLE: testq %r12, %r12
+; CHECK-APPLE: jne
+; Access part of the error object and save it to error_ref
+; CHECK-APPLE: movb 8(%r12)
+; CHECK-APPLE: movq %r12, %rdi
+; CHECK_APPLE: callq {{.*}}free
+
+; CHECK-O0-LABEL: caller:
+; CHECK-O0: xorl
+; CHECK-O0: movl %{{.*}}, %r12d
+; CHECK-O0: callq {{.*}}foo
+; CHECK-O0: jne
+entry:
+  %error_ptr_ref = alloca swifterror %swift_error*
+  store %swift_error* null, %swift_error** %error_ptr_ref
+  %call = call float @foo(%swift_error** swifterror %error_ptr_ref)
+  %error_from_foo = load %swift_error*, %swift_error** %error_ptr_ref
+  %had_error_from_foo = icmp ne %swift_error* %error_from_foo, null
+  %tmp = bitcast %swift_error* %error_from_foo to i8*
+  br i1 %had_error_from_foo, label %handler, label %cont
+cont:
+  %v1 = getelementptr inbounds %swift_error, %swift_error* %error_from_foo, i64 0, i32 1
+  %t = load i8, i8* %v1
+  store i8 %t, i8* %error_ref
+  br label %handler
+handler:
+  call void @free(i8* %tmp)
+  ret float 1.0
+}
+
+; "caller2" is the caller of "foo", it calls "foo" inside a loop.
+define float @caller2(i8* %error_ref) {
+; CHECK-APPLE-LABEL: caller2:
+; CHECK-APPLE: xorl %r12d, %r12d
+; CHECK-APPLE: callq {{.*}}foo
+; CHECK-APPLE: testq %r12, %r12
+; CHECK-APPLE: jne
+; CHECK-APPLE: ucomiss
+; CHECK-APPLE: jbe
+; Access part of the error object and save it to error_ref
+; CHECK-APPLE: movb 8(%r12)
+; CHECK-APPLE: movq %r12, %rdi
+; CHECK_APPLE: callq {{.*}}free
+
+; CHECK-O0-LABEL: caller2:
+; CHECK-O0: xorl
+; CHECK-O0: movl %{{.*}}, %r12d
+; CHECK-O0: callq {{.*}}foo
+; CHECK-O0: movq %r12, [[ID:%[a-z]+]]
+; CHECK-O0: cmpq $0, [[ID]]
+; CHECK-O0: jne
+entry:
+  %error_ptr_ref = alloca swifterror %swift_error*
+  br label %bb_loop
+bb_loop:
+  store %swift_error* null, %swift_error** %error_ptr_ref
+  %call = call float @foo(%swift_error** swifterror %error_ptr_ref)
+  %error_from_foo = load %swift_error*, %swift_error** %error_ptr_ref
+  %had_error_from_foo = icmp ne %swift_error* %error_from_foo, null
+  %tmp = bitcast %swift_error* %error_from_foo to i8*
+  br i1 %had_error_from_foo, label %handler, label %cont
+cont:
+  %cmp = fcmp ogt float %call, 1.000000e+00
+  br i1 %cmp, label %bb_end, label %bb_loop
+bb_end:
+  %v1 = getelementptr inbounds %swift_error, %swift_error* %error_from_foo, i64 0, i32 1
+  %t = load i8, i8* %v1
+  store i8 %t, i8* %error_ref
+  br label %handler
+handler:
+  call void @free(i8* %tmp)
+  ret float 1.0
+}
+
+; "foo_if" is a function that takes a swifterror parameter, it sets swifterror
+; under a certain condition.
+define float @foo_if(%swift_error** swifterror %error_ptr_ref, i32 %cc) {
+; CHECK-APPLE-LABEL: foo_if:
+; CHECK-APPLE: testl %edi, %edi
+; CHECK-APPLE: je
+; CHECK-APPLE: movl $16, %edi
+; CHECK-APPLE: malloc
+; CHECK-APPLE: movb $1, 8(%rax)
+; CHECK-APPLE: movq %rax, %r12
+; CHECK-APPLE-NOT: %r12
+; CHECK-APPLE: ret
+
+; CHECK-O0-LABEL: foo_if:
+; CHECK-O0: cmpl $0
+; spill to stack
+; CHECK-O0: movq %r12, {{.*}}(%rsp)
+; CHECK-O0: je
+; CHECK-O0: movl $16,
+; CHECK-O0: malloc
+; CHECK-O0: movq %rax, [[ID:%[a-z]+]]
+; CHECK-O0-DAG: movb $1, 8(%rax)
+; CHECK-O0-DAG: movq [[ID]], %r12
+; CHECK-O0: ret
+; reload from stack
+; CHECK-O0: movq {{.*}}(%rsp), %r12
+; CHECK-O0: ret
+entry:
+  %cond = icmp ne i32 %cc, 0
+  br i1 %cond, label %gen_error, label %normal
+
+gen_error:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+  ret float 1.0
+
+normal:
+  ret float 0.0
+}
+
+; "foo_loop" is a function that takes a swifterror parameter, it sets swifterror
+; under a certain condition inside a loop.
+define float @foo_loop(%swift_error** swifterror %error_ptr_ref, i32 %cc, float %cc2) {
+; CHECK-APPLE-LABEL: foo_loop:
+; CHECK-APPLE: movq %r12, %rax
+; CHECK-APPLE: testl
+; CHECK-APPLE: je
+; CHECK-APPLE: movl $16, %edi
+; CHECK-APPLE: malloc
+; CHECK-APPLE: movb $1, 8(%rax)
+; CHECK-APPLE: ucomiss
+; CHECK-APPLE: jbe
+; CHECK-APPLE: movq %rax, %r12
+; CHECK-APPLE: ret
+
+; CHECK-O0-LABEL: foo_loop:
+; spill to stack
+; CHECK-O0: movq %r12, {{.*}}(%rsp)
+; CHECK-O0: cmpl $0
+; CHECK-O0: je
+; CHECK-O0: movl $16,
+; CHECK-O0: malloc
+; CHECK-O0: movq %rax, [[ID:%[a-z]+]]
+; CHECK-O0: movb $1, 8([[ID]])
+; CHECK-O0: jbe
+; reload from stack
+; CHECK-O0: movq {{.*}}(%rsp), %r12
+; CHECK-O0: ret
+entry:
+  br label %bb_loop
+
+bb_loop:
+  %cond = icmp ne i32 %cc, 0
+  br i1 %cond, label %gen_error, label %bb_cont
+
+gen_error:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+  br label %bb_cont
+
+bb_cont:
+  %cmp = fcmp ogt float %cc2, 1.000000e+00
+  br i1 %cmp, label %bb_end, label %bb_loop
+bb_end:
+  ret float 0.0
+}
+
+%struct.S = type { i32, i32, i32, i32, i32, i32 }
+
+; "foo_sret" is a function that takes a swifterror parameter, it also has a sret
+; parameter.
+define void @foo_sret(%struct.S* sret %agg.result, i32 %val1, %swift_error** swifterror %error_ptr_ref) {
+; CHECK-APPLE-LABEL: foo_sret:
+; CHECK-APPLE: movq %rdi, %{{.*}}
+; CHECK-APPLE: movl $16, %edi
+; CHECK-APPLE: malloc
+; CHECK-APPLE: movb $1, 8(%rax)
+; CHECK-APPLE: movl %{{.*}}, 4(%{{.*}})
+; CHECK-APPLE: movq %rax, %r12
+; CHECK-APPLE: movq %{{.*}}, %rax
+; CHECK-APPLE-NOT: x19
+
+; CHECK-O0-LABEL: foo_sret:
+; CHECK-O0: movl $16,
+; spill sret to stack
+; CHECK-O0: movq %rdi,
+; CHECK-O0: movq {{.*}}, %rdi
+; CHECK-O0: malloc
+; CHECK-O0: movb $1, 8(%rax)
+; CHECK-O0: movl %{{.*}}, 4(%{{.*}})
+; CHECK-O0: movq %{{.*}}, %r12
+; reload sret from stack
+; CHECK-O0: movq {{.*}}(%rsp), %rax
+; CHECK-O0: ret
+entry:
+  %call = call i8* @malloc(i64 16)
+  %call.0 = bitcast i8* %call to %swift_error*
+  store %swift_error* %call.0, %swift_error** %error_ptr_ref
+  %tmp = getelementptr inbounds i8, i8* %call, i64 8
+  store i8 1, i8* %tmp
+  %v2 = getelementptr inbounds %struct.S, %struct.S* %agg.result, i32 0, i32 1
+  store i32 %val1, i32* %v2
+  ret void
+}
+
+; "caller3" calls "foo_sret" that takes a swifterror parameter.
+define float @caller3(i8* %error_ref) {
+; CHECK-APPLE-LABEL: caller3:
+; CHECK-APPLE: movl $1, %esi
+; CHECK-APPLE: xorl %r12d, %r12d
+; CHECK-APPLE: callq {{.*}}foo_sret
+; CHECK-APPLE: testq %r12, %r12
+; CHECK-APPLE: jne
+; Access part of the error object and save it to error_ref
+; CHECK-APPLE: movb 8(%r12),
+; CHECK-APPLE: movb %{{.*}},
+; CHECK-APPLE: movq %r12, %rdi
+; CHECK_APPLE: callq {{.*}}free
+
+; CHECK-O0-LABEL: caller3:
+; CHECK-O0: xorl
+; CHECK-O0: movl {{.*}}, %r12d
+; CHECK-O0: movl $1, %esi
+; CHECK-O0: movq {{.*}}, %rdi
+; CHECK-O0: callq {{.*}}foo_sret
+; CHECK-O0: movq %r12,
+; CHECK-O0: cmpq $0
+; CHECK-O0: jne
+; Access part of the error object and save it to error_ref
+; CHECK-O0: movb 8(%{{.*}}),
+; CHECK-O0: movb %{{.*}},
+; reload from stack
+; CHECK-O0: movq {{.*}}(%rsp), %rdi
+; CHECK-O0: callq {{.*}}free
+entry:
+  %s = alloca %struct.S, align 8
+  %error_ptr_ref = alloca swifterror %swift_error*
+  store %swift_error* null, %swift_error** %error_ptr_ref
+  call void @foo_sret(%struct.S* sret %s, i32 1, %swift_error** swifterror %error_ptr_ref)
+  %error_from_foo = load %swift_error*, %swift_error** %error_ptr_ref
+  %had_error_from_foo = icmp ne %swift_error* %error_from_foo, null
+  %tmp = bitcast %swift_error* %error_from_foo to i8*
+  br i1 %had_error_from_foo, label %handler, label %cont
+cont:
+  %v1 = getelementptr inbounds %swift_error, %swift_error* %error_from_foo, i64 0, i32 1
+  %t = load i8, i8* %v1
+  store i8 %t, i8* %error_ref
+  br label %handler
+handler:
+  call void @free(i8* %tmp)
+  ret float 1.0
+}
+
+; This is a caller with multiple swifterror values, it calls "foo" twice, each
+; time with a different swifterror value, from "alloca swifterror".
+define float @caller_with_multiple_swifterror_values(i8* %error_ref, i8* %error_ref2) {
+; CHECK-APPLE-LABEL: caller_with_multiple_swifterror_values:
+
+; The first swifterror value:
+; CHECK-APPLE: xorl %r12d, %r12d
+; CHECK-APPLE: callq {{.*}}foo
+; CHECK-APPLE: testq %r12, %r12
+; CHECK-APPLE: jne
+; Access part of the error object and save it to error_ref
+; CHECK-APPLE: movb 8(%r12)
+; CHECK-APPLE: movq %r12, %rdi
+; CHECK_APPLE: callq {{.*}}free
+
+; The second swifterror value:
+; CHECK-APPLE: xorl %r12d, %r12d
+; CHECK-APPLE: callq {{.*}}foo
+; CHECK-APPLE: testq %r12, %r12
+; CHECK-APPLE: jne
+; Access part of the error object and save it to error_ref
+; CHECK-APPLE: movb 8(%r12)
+; CHECK-APPLE: movq %r12, %rdi
+; CHECK_APPLE: callq {{.*}}free
+
+; CHECK-O0-LABEL: caller_with_multiple_swifterror_values:
+
+; The first swifterror value:
+; CHECK-O0: xorl
+; CHECK-O0: movl %{{.*}}, %r12d
+; CHECK-O0: callq {{.*}}foo
+; CHECK-O0: jne
+
+; The second swifterror value:
+; CHECK-O0: xorl
+; CHECK-O0: movl %{{.*}}, %r12d
+; CHECK-O0: callq {{.*}}foo
+; CHECK-O0: jne
+entry:
+  %error_ptr_ref = alloca swifterror %swift_error*
+  store %swift_error* null, %swift_error** %error_ptr_ref
+  %call = call float @foo(%swift_error** swifterror %error_ptr_ref)
+  %error_from_foo = load %swift_error*, %swift_error** %error_ptr_ref
+  %had_error_from_foo = icmp ne %swift_error* %error_from_foo, null
+  %tmp = bitcast %swift_error* %error_from_foo to i8*
+  br i1 %had_error_from_foo, label %handler, label %cont
+cont:
+  %v1 = getelementptr inbounds %swift_error, %swift_error* %error_from_foo, i64 0, i32 1
+  %t = load i8, i8* %v1
+  store i8 %t, i8* %error_ref
+  br label %handler
+handler:
+  call void @free(i8* %tmp)
+
+  %error_ptr_ref2 = alloca swifterror %swift_error*
+  store %swift_error* null, %swift_error** %error_ptr_ref2
+  %call2 = call float @foo(%swift_error** swifterror %error_ptr_ref2)
+  %error_from_foo2 = load %swift_error*, %swift_error** %error_ptr_ref2
+  %had_error_from_foo2 = icmp ne %swift_error* %error_from_foo2, null
+  %bitcast2 = bitcast %swift_error* %error_from_foo2 to i8*
+  br i1 %had_error_from_foo2, label %handler2, label %cont2
+cont2:
+  %v2 = getelementptr inbounds %swift_error, %swift_error* %error_from_foo2, i64 0, i32 1
+  %t2 = load i8, i8* %v2
+  store i8 %t2, i8* %error_ref2
+  br label %handler2
+handler2:
+  call void @free(i8* %bitcast2)
+
+  ret float 1.0
+}