SwiftTailCC: teach verifier musttail rules applicable to this CC.

[t.p.northover]

Summary

SwiftTailCC has a different set of requirements than the C calling convention for a tail call. The exact argument sequence doesn't have to match, but fewer ABI-affecting attributes are allowed (mostly to be conservative and because we have a reasonable idea of what might actually be useful).

Also make sure the musttail diagnostic triggers if a musttail call isn't actually a tail call.

[jroelofs]

Why doesn't the count and sequence of arguments/parameters have to match for swifttail calls?

[efriedma]

Why doesn't the count and sequence of arguments/parameters have to match for swifttail calls?

tailcc/swifttailcc use a callee-pop convention, so we can allocate as much stack space as we need for arguments.

We might want to consider relaxing the rules for tailcc as well as swifttailcc, but I guess it doesn't need to happen in this patch.

llvm/docs/LangRef.rst

11347

There's one source of complexity related to allowing undef here you might not have considered: return type promotion (converting a large return value to an indirect argument). In particular, if we need to promote the return type of the musttail call. I think you can make this work, but not sure if targets generate the right sequence.

11350

The way this is organized is hard to read: can you rearrange it to separate the common, non-swittailcc, and the swifttailcc requirements?

[t.p.northover]

We might want to consider relaxing the rules for tailcc as well as swifttailcc, but I guess it doesn't need to happen in this patch.

I'm happy to do it here, I think it's definitely an improvement worth having.

llvm/docs/LangRef.rst

11347

Return type promotion seems really iffy with tail calls even now. For this module:

declare [16 x i64] @foo()

define [16 x i64] @bar() {
  %res = musttail call [16 x i64] @foo()
  ret [16 x i64] %res
}

AArch64 & ARM don't actually do a tail call but are otherwise correct (no diagnostic because of where the check is; this patch fixes that). X86 does the tail call but tells @foo to use a random bit of @bar's caller's stack for the array.

I don't think the verifier can check that though, it's quite entwined in CodeGen and Verifier.cpp has nothing but a TargetTriple to work with.

Best I can think of is to document that functions with return type promotion probably won't work as expected here.

[t.p.northover]

Added tailcc to the changes, and reworked the documentation to be (hopefully) clearer.

[efriedma]

llvm/docs/LangRef.rst

11347

If the return types match, it's easy to do the "right" thing: we just need to forward the indirect pointer.

It's not high priority to implement, but I'd at least like an appropriate error. And I'm concerned if we don't have any idea how to implement it. I guess one thing we could do is allow passing a null pointer for the promoted return value, and add a null check in the callee before storing the value.

[t.p.northover]

llvm/docs/LangRef.rst

11347

Fair enough, I'll make the X86 backend give an error instead of assuming it knows better than SDAG for now.

[t.p.northover]

Add error message to x86 backend if it can't honour a musttail call. It seems from the (presumably original) musttail.ll test that it's intended to override the disable-tail-calls function attribute, so I had to teach SelectionDAGBuilder about that too.

[efriedma]

llvm/docs/LangRef.rst

11347

I'd also like to see testcases that show we generate the expected "failed to perform tail call elimination" error for the unimplemented cases.

[t.p.northover]

Added test for sret promotion case the verifier can't handle.

[efriedma]

LGTM

[t.p.northover]

Thanks Eli, committed as 9ff2eb1ea596.

llvm/docs/LangRef.rst

#. The optional ``tail`` and ``musttail`` markers indicate that the optimizers
    Both markers imply that the callee does not access allocas from the caller.
    The ``tail`` marker additionally implies that the callee does not access
    varargs from the caller. Calls marked ``musttail`` must obey the following
    additional  rules:
 
    - The call must immediately precede a :ref:`ret <i_ret>` instruction,
      or a pointer bitcast followed by a ret instruction.
    - The ret instruction must return the (possibly bitcasted) value
-     produced by the call or void.
-   - The caller and callee prototypes must match. Pointer types of
-     parameters or return types may differ in pointee type, but not
-     in address space.
+     produced by the call, undef, or void.
    - The calling conventions of the caller and callee must match.
-   - All ABI-impacting function attributes, such as sret, byval, inreg,
-     returned, and inalloca, must match.
    - The callee must be varargs iff the caller is varargs. Bitcasting a
      non-varargs function to the appropriate varargs type is legal so
      long as the non-varargs prefixes obey the other rules.
+   - The return type must not undergo automatic conversion to an `sret` pointer.
+
+  In addition, if the calling convention is not `swifttailcc` or `tailcc`:
+
+   - All ABI-impacting function attributes, such as sret, byval, inreg,
+     returned, and inalloca, must match.
+   - The caller and callee prototypes must match. Pointer types of parameters
+     or return types may differ in pointee type, but not in address space.
+
+  On the other hand, if the calling convention is `swifttailcc` or `swiftcc`:
+
+   - Only these ABI-impacting attributes attributes are allowed: sret, byval,
+     swiftself, and swiftasync.
+   - Prototypes are not required to match.
 
    Tail call optimization for calls marked ``tail`` is guaranteed to occur if
    the following conditions are met:
 
    -  Caller and callee both have the calling convention ``fastcc`` or ``tailcc``.
    -  The call is in tail position (ret immediately follows call and ret
       uses value of call or is void).
    -  Option ``-tailcallopt`` is enabled,

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h

void EmitBranchForMergedCondition(const Value *Cond, MachineBasicBlock *TBB,
                                     MachineBasicBlock *SwitchBB,
                                     BranchProbability TProb, BranchProbability FProb,
                                     bool InvertCond);
   bool ShouldEmitAsBranches(const std::vector<SwitchCG::CaseBlock> &Cases);
   bool isExportableFromCurrentBlock(const Value *V, const BasicBlock *FromBB);
   void CopyToExportRegsIfNeeded(const Value *V);
   void ExportFromCurrentBlock(const Value *V);
   void LowerCallTo(const CallBase &CB, SDValue Callee, bool IsTailCall,
-                   const BasicBlock *EHPadBB = nullptr);
+                   bool IsMustTailCall, const BasicBlock *EHPadBB = nullptr);
 
   // Lower range metadata from 0 to N to assert zext to an integer of nearest
   // floor power of two.
   SDValue lowerRangeToAssertZExt(SelectionDAG &DAG, const Instruction &I,
                                  SDValue Op);
 
   void populateCallLoweringInfo(TargetLowering::CallLoweringInfo &CLI,
                                 const CallBase *Call, unsigned ArgIdx,

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

else if (Fn && Fn->isIntrinsic()) {
     }
   } else if (I.countOperandBundlesOfType(LLVMContext::OB_deopt)) {
     // Currently we do not lower any intrinsic calls with deopt operand bundles.
     // Eventually we will support lowering the @llvm.experimental.deoptimize
     // intrinsic, and right now there are no plans to support other intrinsics
     // with deopt state.
     LowerCallSiteWithDeoptBundle(&I, getValue(Callee), EHPadBB);
   } else {
-    LowerCallTo(I, getValue(Callee), false, EHPadBB);
+    LowerCallTo(I, getValue(Callee), false, false, EHPadBB);
   }
 
   // If the value of the invoke is used outside of its defining block, make it
   // available as a virtual register.
   // We already took care of the exported value for the statepoint instruction
   // during call to the LowerStatepoint.
   if (!isa<GCStatepointInst>(I)) {
     CopyToExportRegsIfNeeded(&I);

 }
 
 void SelectionDAGBuilder::lowerCallToExternalSymbol(const CallInst &I,
                                            const char *FunctionName) {
   assert(FunctionName && "FunctionName must not be nullptr");
   SDValue Callee = DAG.getExternalSymbol(
       FunctionName,
       DAG.getTargetLoweringInfo().getPointerTy(DAG.getDataLayout()));
-  LowerCallTo(I, Callee, I.isTailCall());
+  LowerCallTo(I, Callee, I.isTailCall(), I.isMustTailCall());
 }
 
 /// Given a @llvm.call.preallocated.setup, return the corresponding
 /// preallocated call.
 static const CallBase *FindPreallocatedCall(const Value *PreallocatedSetup) {
   assert(cast<CallBase>(PreallocatedSetup)
                  ->getCalledFunction()
                  ->getIntrinsicID() == Intrinsic::call_preallocated_setup &&
DAG.setRoot(lowerEndEH(getRoot(), cast_or_null<InvokeInst>(CLI.CB), EHPadBB,
                            BeginLabel));
   }
 
   return Result;
 }
 
 void SelectionDAGBuilder::LowerCallTo(const CallBase &CB, SDValue Callee,
                                       bool isTailCall,
+                                      bool isMustTailCall,
                                       const BasicBlock *EHPadBB) {
   auto &DL = DAG.getDataLayout();
   FunctionType *FTy = CB.getFunctionType();
   Type *RetTy = CB.getType();
 
   TargetLowering::ArgListTy Args;
   Args.reserve(CB.arg_size());
 
   const Value *SwiftErrorVal = nullptr;
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
 
   if (isTailCall) {
     // Avoid emitting tail calls in functions with the disable-tail-calls
     // attribute.
     auto *Caller = CB.getParent()->getParent();
     if (Caller->getFnAttribute("disable-tail-calls").getValueAsString() ==
-        "true")
+        "true" && !isMustTailCall)
       isTailCall = false;
 
     // We can't tail call inside a function with a swifterror argument. Lowering
     // does not support this yet. It would have to move into the swifterror
     // register before the call.
     if (TLI.supportSwiftError() &&
         Caller->getAttributes().hasAttrSomewhere(Attribute::SwiftError))
       isTailCall = false;
void SelectionDAGBuilder::visitCall(const CallInst &I) {
   SDValue Callee = getValue(I.getCalledOperand());
 
   if (I.countOperandBundlesOfType(LLVMContext::OB_deopt))
     LowerCallSiteWithDeoptBundle(&I, Callee, nullptr);
   else
     // Check if we can potentially perform a tail call. More detailed checking
     // is be done within LowerCallTo, after more information about the call is
     // known.
-    LowerCallTo(I, Callee, I.isTailCall());
+    LowerCallTo(I, Callee, I.isTailCall(), I.isMustTailCall());
 }
 
 namespace {
 
 /// AsmOperandInfo - This contains information for each constraint that we are
 /// lowering.
 class SDISelAsmOperandInfo : public TargetLowering::AsmOperandInfo {
 public:

llvm/lib/IR/Verifier.cpp

#include "llvm/IR/Metadata.def"
   void visitCatchReturnInst(CatchReturnInst &CatchReturn);
   void visitCleanupPadInst(CleanupPadInst &CPI);
   void visitFuncletPadInst(FuncletPadInst &FPI);
   void visitCatchSwitchInst(CatchSwitchInst &CatchSwitch);
   void visitCleanupReturnInst(CleanupReturnInst &CRI);
 
   void verifySwiftErrorCall(CallBase &Call, const Value *SwiftErrorVal);
   void verifySwiftErrorValue(const Value *SwiftErrorVal);
+  void verifyTailCCMustTailAttrs(AttrBuilder Attrs, StringRef Context);
   void verifyMustTailCall(CallInst &CI);
   bool verifyAttributeCount(AttributeList Attrs, unsigned Params);
   void verifyAttributeTypes(AttributeSet Attrs, bool IsFunction,
                             const Value *V);
   void verifyParameterAttrs(AttributeSet Attrs, Type *Ty, const Value *V);
   void verifyFunctionAttrs(FunctionType *FT, AttributeList Attrs,
                            const Value *V, bool IsIntrinsic);
   void verifyFunctionMetadata(ArrayRef<std::pair<unsigned, MDNode *>> MDs);
if (Call.getFunction()->getSubprogram() && Call.getCalledFunction() &&
     AssertDI(Call.getDebugLoc(),
              "inlinable function call in a function with "
              "debug info must have a !dbg location",
              Call);
 
   visitInstruction(Call);
 }
 
+void Verifier::verifyTailCCMustTailAttrs(AttrBuilder Attrs,
+                                         StringRef Context) {
+  Assert(!Attrs.contains(Attribute::InAlloca),
+         Twine("inalloca attribute not allowed in ") + Context);
+  Assert(!Attrs.contains(Attribute::InReg),
+         Twine("inreg attribute not allowed in ") + Context);
+  Assert(!Attrs.contains(Attribute::SwiftError),
+         Twine("swifterror attribute not allowed in ") + Context);
+  Assert(!Attrs.contains(Attribute::Preallocated),
+         Twine("preallocated attribute not allowed in ") + Context);
+  Assert(!Attrs.contains(Attribute::ByRef),
+         Twine("byref attribute not allowed in ") + Context);
+}
+
 /// Two types are "congruent" if they are identical, or if they are both pointer
 /// types with different pointee types and the same address space.
 static bool isTypeCongruent(Type *L, Type *R) {
   if (L == R)
     return true;
   PointerType *PL = dyn_cast<PointerType>(L);
   PointerType *PR = dyn_cast<PointerType>(R);
   if (!PL || !PR)
if (Attrs.hasParamAttribute(I, Attribute::Alignment) &&
        Attrs.hasParamAttribute(I, Attribute::ByRef)))
     Copy.addAlignmentAttr(Attrs.getParamAlignment(I));
   return Copy;
 }
 
 void Verifier::verifyMustTailCall(CallInst &CI) {
   Assert(!CI.isInlineAsm(), "cannot use musttail call with inline asm", &CI);
 
-  // - The caller and callee prototypes must match.  Pointer types of
-  //   parameters or return types may differ in pointee type, but not
-  //   address space.
   Function *F = CI.getParent()->getParent();
   FunctionType *CallerTy = F->getFunctionType();
   FunctionType *CalleeTy = CI.getFunctionType();
-  if (!CI.getCalledFunction() || !CI.getCalledFunction()->isIntrinsic()) {
-    Assert(CallerTy->getNumParams() == CalleeTy->getNumParams(),
-           "cannot guarantee tail call due to mismatched parameter counts",
-           &CI);
-    for (int I = 0, E = CallerTy->getNumParams(); I != E; ++I) {
-      Assert(
-          isTypeCongruent(CallerTy->getParamType(I), CalleeTy->getParamType(I)),
-          "cannot guarantee tail call due to mismatched parameter types", &CI);
-    }
-  }
   Assert(CallerTy->isVarArg() == CalleeTy->isVarArg(),
          "cannot guarantee tail call due to mismatched varargs", &CI);
   Assert(isTypeCongruent(CallerTy->getReturnType(), CalleeTy->getReturnType()),
          "cannot guarantee tail call due to mismatched return types", &CI);
 
   // - The calling conventions of the caller and callee must match.
   Assert(F->getCallingConv() == CI.getCallingConv(),
          "cannot guarantee tail call due to mismatched calling conv", &CI);
 
-  // - All ABI-impacting function attributes, such as sret, byval, inreg,
-  //   returned, preallocated, and inalloca, must match.
-  AttributeList CallerAttrs = F->getAttributes();
-  AttributeList CalleeAttrs = CI.getAttributes();
-  for (int I = 0, E = CallerTy->getNumParams(); I != E; ++I) {
-    AttrBuilder CallerABIAttrs = getParameterABIAttributes(I, CallerAttrs);
-    AttrBuilder CalleeABIAttrs = getParameterABIAttributes(I, CalleeAttrs);
-    Assert(CallerABIAttrs == CalleeABIAttrs,
-           "cannot guarantee tail call due to mismatched ABI impacting "
-           "function attributes",
-           &CI, CI.getOperand(I));
-  }
-
   // - The call must immediately precede a :ref:`ret <i_ret>` instruction,
   //   or a pointer bitcast followed by a ret instruction.
   // - The ret instruction must return the (possibly bitcasted) value
   //   produced by the call or void.
   Value *RetVal = &CI;
   Instruction *Next = CI.getNextNode();
 
   // Handle the optional bitcast.
   if (BitCastInst *BI = dyn_cast_or_null<BitCastInst>(Next)) {
     Assert(BI->getOperand(0) == RetVal,
            "bitcast following musttail call must use the call", BI);
     RetVal = BI;
     Next = BI->getNextNode();
   }
 
   // Check the return.
   ReturnInst *Ret = dyn_cast_or_null<ReturnInst>(Next);
   Assert(Ret, "musttail call must precede a ret with an optional bitcast",
          &CI);
-  Assert(!Ret->getReturnValue() || Ret->getReturnValue() == RetVal,
+  Assert(!Ret->getReturnValue() || Ret->getReturnValue() == RetVal ||
+             isa<UndefValue>(Ret->getReturnValue()),
          "musttail call result must be returned", Ret);
+
+  AttributeList CallerAttrs = F->getAttributes();
+  AttributeList CalleeAttrs = CI.getAttributes();
+  if (CI.getCallingConv() == CallingConv::SwiftTail ||
+      CI.getCallingConv() == CallingConv::Tail) {
+    StringRef CCName =
+        CI.getCallingConv() == CallingConv::Tail ? "tailcc" : "swifttailcc";
+
+    // - Only sret, byval, swiftself, and swiftasync ABI-impacting attributes
+    //   are allowed in swifttailcc call
+    for (int I = 0, E = CallerTy->getNumParams(); I != E; ++I) {
+      AttrBuilder ABIAttrs = getParameterABIAttributes(I, CallerAttrs);
+      SmallString<32> Context{CCName, StringRef(" musttail caller")};
+      verifyTailCCMustTailAttrs(ABIAttrs, Context);
+    }
+    for (int I = 0, E = CalleeTy->getNumParams(); I != E; ++I) {
+      AttrBuilder ABIAttrs = getParameterABIAttributes(I, CalleeAttrs);
+      SmallString<32> Context{CCName, StringRef(" musttail callee")};
+      verifyTailCCMustTailAttrs(ABIAttrs, Context);
+    }
+    // - Varargs functions are not allowed
+    Assert(!CallerTy->isVarArg(), Twine("cannot guarantee ") + CCName +
+                                      " tail call for varargs function");
+    return;
+  }
+
+  // - The caller and callee prototypes must match.  Pointer types of
+  //   parameters or return types may differ in pointee type, but not
+  //   address space.
+  if (!CI.getCalledFunction() || !CI.getCalledFunction()->isIntrinsic()) {
+    Assert(CallerTy->getNumParams() == CalleeTy->getNumParams(),
+           "cannot guarantee tail call due to mismatched parameter counts",
+           &CI);
+    for (int I = 0, E = CallerTy->getNumParams(); I != E; ++I) {
+      Assert(
+          isTypeCongruent(CallerTy->getParamType(I), CalleeTy->getParamType(I)),
+          "cannot guarantee tail call due to mismatched parameter types", &CI);
+    }
+  }
+
+  // - All ABI-impacting function attributes, such as sret, byval, inreg,
+  //   returned, preallocated, and inalloca, must match.
+  for (int I = 0, E = CallerTy->getNumParams(); I != E; ++I) {
+    AttrBuilder CallerABIAttrs = getParameterABIAttributes(I, CallerAttrs);
+    AttrBuilder CalleeABIAttrs = getParameterABIAttributes(I, CalleeAttrs);
+    Assert(CallerABIAttrs == CalleeABIAttrs,
+           "cannot guarantee tail call due to mismatched ABI impacting "
+           "function attributes",
+           &CI, CI.getOperand(I));
+  }
 }
 
 void Verifier::visitCallInst(CallInst &CI) {
   visitCallBase(CI);
 
   if (CI.isMustTailCall())
     verifyMustTailCall(CI);
 }

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (CallConv == CallingConv::C || CallConv == CallingConv::Fast) {
     if (CalleeInSVE || CalleeOutSVE)
       CallConv = CallingConv::AArch64_SVE_VectorCall;
   }
 
   if (IsTailCall) {
     // Check if it's really possible to do a tail call.
     IsTailCall = isEligibleForTailCallOptimization(
         Callee, CallConv, IsVarArg, Outs, OutVals, Ins, DAG);
-    if (!IsTailCall && CLI.CB && CLI.CB->isMustTailCall())
-      report_fatal_error("failed to perform tail call elimination on a call "
-                         "site marked musttail");
 
     // A sibling call is one where we're under the usual C ABI and not planning
     // to change that but can still do a tail call:
     if (!TailCallOpt && IsTailCall && CallConv != CallingConv::Tail &&
         CallConv != CallingConv::SwiftTail)
       IsSibCall = true;
 
     if (IsTailCall)
       ++NumTailCalls;
   }
 
+  if (!IsTailCall && CLI.CB && CLI.CB->isMustTailCall())
+    report_fatal_error("failed to perform tail call elimination on a call "
+                       "site marked musttail");
+
   // Analyze operands of the call, assigning locations to each operand.
   SmallVector<CCValAssign, 16> ArgLocs;
   CCState CCInfo(CallConv, IsVarArg, DAG.getMachineFunction(), ArgLocs,
                  *DAG.getContext());
 
   if (IsVarArg) {
     // Handle fixed and variable vector arguments differently.
     // Variable vector arguments always go into memory.

llvm/lib/Target/ARM/ARMISelLowering.cpp

if (isa<GlobalAddressSDNode>(Callee)) {
     }
   }
   if (isTailCall) {
     // Check if it's really possible to do a tail call.
     isTailCall = IsEligibleForTailCallOptimization(
         Callee, CallConv, isVarArg, isStructRet,
         MF.getFunction().hasStructRetAttr(), Outs, OutVals, Ins, DAG,
         PreferIndirect);
-    if (!isTailCall && CLI.CB && CLI.CB->isMustTailCall())
-      report_fatal_error("failed to perform tail call elimination on a call "
-                         "site marked musttail");
 
     if (isTailCall && !getTargetMachine().Options.GuaranteedTailCallOpt &&
         CallConv != CallingConv::Tail && CallConv != CallingConv::SwiftTail)
       isSibCall = true;
 
     // We don't support GuaranteedTailCallOpt for ARM, only automatically
     // detected sibcalls.
     if (isTailCall)
       ++NumTailCalls;
   }
 
+  if (!isTailCall && CLI.CB && CLI.CB->isMustTailCall())
+    report_fatal_error("failed to perform tail call elimination on a call "
+                       "site marked musttail");
   // Analyze operands of the call, assigning locations to each operand.
   SmallVector<CCValAssign, 16> ArgLocs;
   CCState CCInfo(CallConv, isVarArg, DAG.getMachineFunction(), ArgLocs,
                  *DAG.getContext());
   CCInfo.AnalyzeCallOperands(Outs, CCAssignFnForCall(CallConv, isVarArg));
 
   // Get a count of how many bytes are to be pushed on the stack.
   unsigned NumBytes = CCInfo.getNextStackOffset();

llvm/lib/Target/X86/X86ISelLowering.cpp

X86TargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI,
   bool IsIndirectCall = (CB && isa<CallInst>(CB) && CB->isIndirectCall());
   const Module *M = MF.getMMI().getModule();
   Metadata *IsCFProtectionSupported = M->getModuleFlag("cf-protection-branch");
 
   MachineFunction::CallSiteInfo CSInfo;
   if (CallConv == CallingConv::X86_INTR)
     report_fatal_error("X86 interrupts may not be called directly");
 
-  if (Subtarget.isPICStyleGOT() && !IsGuaranteeTCO) {
+  bool IsMustTail = CLI.CB && CLI.CB->isMustTailCall();
+  if (Subtarget.isPICStyleGOT() && !IsGuaranteeTCO && !IsMustTail) {
     // If we are using a GOT, disable tail calls to external symbols with
     // default visibility. Tail calling such a symbol requires using a GOT
     // relocation, which forces early binding of the symbol. This breaks code
     // that require lazy function symbol resolution. Using musttail or
     // GuaranteedTailCallOpt will override this.
     GlobalAddressSDNode *G = dyn_cast<GlobalAddressSDNode>(Callee);
     if (!G || (!G->getGlobal()->hasLocalLinkage() &&
                G->getGlobal()->hasDefaultVisibility()))
       isTailCall = false;
   }
 
-  bool IsMustTail = CLI.CB && CLI.CB->isMustTailCall();
-  if (IsMustTail) {
-    // Force this to be a tail call.  The verifier rules are enough to ensure
-    // that we can lower this successfully without moving the return address
-    // around.
-    isTailCall = true;
-  } else if (isTailCall) {
+
+  if (isTailCall && !IsMustTail) {
     // Check if it's really possible to do a tail call.
     isTailCall = IsEligibleForTailCallOptimization(Callee, CallConv,
                     isVarArg, SR != NotStructReturn,
                     MF.getFunction().hasStructRetAttr(), CLI.RetTy,
                     Outs, OutVals, Ins, DAG);
 
     // Sibcalls are automatically detected tailcalls which do not require
     // ABI changes.
     if (!IsGuaranteeTCO && isTailCall)
       IsSibcall = true;
 
     if (isTailCall)
       ++NumTailCalls;
   }
 
+  if (IsMustTail && !isTailCall)
+    report_fatal_error("failed to perform tail call elimination on a call "
+                       "site marked musttail");
+
   assert(!(isVarArg && canGuaranteeTCO(CallConv)) &&
          "Var args not supported with calling convention fastcc, ghc or hipe");
 
   // Analyze operands of the call, assigning locations to each operand.
   SmallVector<CCValAssign, 16> ArgLocs;
   CCState CCInfo(CallConv, isVarArg, MF, ArgLocs, *DAG.getContext());
 
   // Allocate shadow area for Win64.
X86TargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI,
   if (IsSibcall)
     // This is a sibcall. The memory operands are available in caller's
     // own caller's stack.
     NumBytes = 0;
   else if (IsGuaranteeTCO && canGuaranteeTCO(CallConv))
     NumBytes = GetAlignedArgumentStackSize(NumBytes, DAG);
 
   int FPDiff = 0;
-  if (isTailCall && !IsSibcall && !IsMustTail) {
+  if (isTailCall &&
+      shouldGuaranteeTCO(CallConv,
+                         MF.getTarget().Options.GuaranteedTailCallOpt)) {
     // Lower arguments at fp - stackoffset + fpdiff.
     unsigned NumBytesCallerPushed = X86Info->getBytesToPopOnReturn();
 
     FPDiff = NumBytesCallerPushed - NumBytes;
 
     // Set the delta of movement of the returnaddr stackslot.
     // But only set if delta is greater than previous delta.
     if (FPDiff < X86Info->getTCReturnAddrDelta())

llvm/test/CodeGen/AArch64/swifttail-call.ll

 ; RUN: llc -verify-machineinstrs < %s -mtriple=aarch64-none-linux-gnu | FileCheck %s --check-prefixes=SDAG,COMMON
 ; RUN: llc -global-isel -global-isel-abort=1 -verify-machineinstrs < %s -mtriple=aarch64-none-linux-gnu  | FileCheck %s --check-prefixes=GISEL,COMMON
 
 declare swifttailcc void @callee_stack0()
 declare swifttailcc void @callee_stack8([8 x i64], i64)
 declare swifttailcc void @callee_stack16([8 x i64], i64, i64)
 declare extern_weak swifttailcc void @callee_weak()
 
 define swifttailcc void @caller_to0_from0() nounwind {
 ; COMMON-LABEL: caller_to0_from0:
 ; COMMON-NEXT: // %bb.
 
-  tail call swifttailcc void @callee_stack0()
+  musttail call swifttailcc void @callee_stack0()
   ret void
 
 ; COMMON-NEXT: b callee_stack0
 }
 
 define swifttailcc void @caller_to0_from8([8 x i64], i64) {
 ; COMMON-LABEL: caller_to0_from8:
 
-  tail call swifttailcc void @callee_stack0()
+  musttail call swifttailcc void @callee_stack0()
   ret void
 
 ; COMMON: add sp, sp, #16
 ; COMMON-NEXT: b callee_stack0
 }
 
 define swifttailcc void @caller_to8_from0() {
 ; COMMON-LABEL: caller_to8_from0:
 
 ; Key point is that the "42" should go #16 below incoming stack
 ; pointer (we didn't have arg space to reuse).
-  tail call swifttailcc void @callee_stack8([8 x i64] undef, i64 42)
+  musttail call swifttailcc void @callee_stack8([8 x i64] undef, i64 42)
   ret void
 
 ; COMMON: str {{x[0-9]+}}, [sp, #-16]!
 ; COMMON-NEXT: b callee_stack8
 }
 
 define swifttailcc void @caller_to8_from8([8 x i64], i64 %a) {
 ; COMMON-LABEL: caller_to8_from8:
 ; COMMON-NOT: sub sp,
 
 ; Key point is that the "%a" should go where at SP on entry.
-  tail call swifttailcc void @callee_stack8([8 x i64] undef, i64 42)
+  musttail call swifttailcc void @callee_stack8([8 x i64] undef, i64 42)
   ret void
 
 ; COMMON: str {{x[0-9]+}}, [sp]
 ; COMMON-NEXT: b callee_stack8
 }
 
 define swifttailcc void @caller_to16_from8([8 x i64], i64 %a) {
 ; COMMON-LABEL: caller_to16_from8:
 ; COMMON-NOT: sub sp,
 
 ; Important point is that the call reuses the "dead" argument space
 ; above %a on the stack. If it tries to go below incoming-SP then the
 ; callee will not deallocate the space, even in swifttailcc.
-  tail call swifttailcc void @callee_stack16([8 x i64] undef, i64 42, i64 2)
+  musttail call swifttailcc void @callee_stack16([8 x i64] undef, i64 42, i64 2)
 
 ; COMMON: stp {{x[0-9]+}}, {{x[0-9]+}}, [sp]
 ; COMMON-NEXT: b callee_stack16
   ret void
 }
 
 
 define swifttailcc void @caller_to8_from24([8 x i64], i64 %a, i64 %b, i64 %c) {
 ; COMMON-LABEL: caller_to8_from24:
 ; COMMON-NOT: sub sp,
 
 ; Key point is that the "%a" should go where at #16 above SP on entry.
-  tail call swifttailcc void @callee_stack8([8 x i64] undef, i64 42)
+  musttail call swifttailcc void @callee_stack8([8 x i64] undef, i64 42)
   ret void
 
 ; COMMON: str {{x[0-9]+}}, [sp, #16]!
 ; COMMON-NEXT: b callee_stack8
 }
 
 
 define swifttailcc void @caller_to16_from16([8 x i64], i64 %a, i64 %b) {
 ; COMMON-LABEL: caller_to16_from16:
 ; COMMON-NOT: sub sp,
 
 ; Here we want to make sure that both loads happen before the stores:
 ; otherwise either %a or %b will be wrongly clobbered.
-  tail call swifttailcc void @callee_stack16([8 x i64] undef, i64 %b, i64 %a)
+  musttail call swifttailcc void @callee_stack16([8 x i64] undef, i64 %b, i64 %a)
   ret void
 
 ; COMMON: ldp {{x[0-9]+}}, {{x[0-9]+}}, [sp]
 ; COMMON: stp {{x[0-9]+}}, {{x[0-9]+}}, [sp]
 ; COMMON-NEXT: b callee_stack16
 }
 
 define swifttailcc void @disable_tail_calls() nounwind "disable-tail-calls"="true" {

llvm/test/CodeGen/AArch64/tailcc-notail.ll

+; RUN: not --crash llc -mtriple=arm64-apple-ios %s -o - 2>&1 | FileCheck %s
+
+; CHECK: LLVM ERROR: failed to perform tail call elimination on a call site marked musttail
+declare tailcc [16 x i64] @callee()
+define tailcc [16 x i64] @caller() {
+  %res = musttail call tailcc [16 x i64] @callee()
+  ret [16 x i64] %res
+}

llvm/test/CodeGen/ARM/tailcc-notail.ll

+; RUN: not --crash llc -mtriple=thumbv7k-apple-ios %s -o - 2>&1 | FileCheck %s
+
+; CHECK: LLVM ERROR: failed to perform tail call elimination on a call site marked musttail
+declare tailcc [16 x i64] @callee()
+define tailcc [16 x i64] @caller() {
+  %res = musttail call tailcc [16 x i64] @callee()
+  ret [16 x i64] %res
+}

llvm/test/CodeGen/X86/tailcall-swifttailcc.ll

 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s
 
 declare dso_local swifttailcc i32 @tailcallee(i32 %a1, i32 %a2, i32 %a3, i32 %a4)
 
 define dso_local swifttailcc i32 @tailcaller(i32 %in1, i32 %in2) nounwind {
 ; CHECK-LABEL: tailcaller:
 ; CHECK-NOT: subq
 ; CHECK-NOT: addq
 ; CHECK: jmp tailcallee
 entry:
-  %tmp11 = tail call swifttailcc i32 @tailcallee(i32 %in1, i32 %in2, i32 %in1, i32 %in2)
+  %tmp11 = musttail call swifttailcc i32 @tailcallee(i32 %in1, i32 %in2, i32 %in1, i32 %in2)
   ret i32 %tmp11
 }
 
 declare dso_local swifttailcc i8* @alias_callee()
 
 define swifttailcc noalias i8* @noalias_caller() nounwind {
 ; CHECK-LABEL: noalias_caller:
 ; CHECK:    jmp alias_callee
   %p = musttail call swifttailcc i8* @alias_callee()
   ret i8* %p
 }
 
 declare dso_local swifttailcc noalias i8* @noalias_callee()
 
 define dso_local swifttailcc i8* @alias_caller() nounwind {
 ; CHECK-LABEL: alias_caller:
 ; CHECK:    jmp noalias_callee # TAILCALL
-  %p = tail call swifttailcc noalias i8* @noalias_callee()
+  %p = musttail call swifttailcc noalias i8* @noalias_callee()
   ret i8* %p
 }
 
 declare dso_local swifttailcc i32 @i32_callee()
 
 define dso_local swifttailcc i32 @ret_undef() nounwind {
 ; CHECK-LABEL: ret_undef:
 ; CHECK:    jmp i32_callee # TAILCALL
-  %p = tail call swifttailcc i32 @i32_callee()
+  %p = musttail call swifttailcc i32 @i32_callee()
   ret i32 undef
 }
 
 declare dso_local swifttailcc void @does_not_return()
 
 define dso_local swifttailcc i32 @noret() nounwind {
 ; CHECK-LABEL: noret:
 ; CHECK:    jmp does_not_return
   tail call swifttailcc void @does_not_return()
   unreachable
 }
 
 define dso_local swifttailcc void @void_test(i32, i32, i32, i32) {
 ; CHECK-LABEL: void_test:
 ; CHECK:    jmp void_test
   entry:
-   tail call swifttailcc void @void_test( i32 %0, i32 %1, i32 %2, i32 %3)
+   musttail call swifttailcc void @void_test( i32 %0, i32 %1, i32 %2, i32 %3)
    ret void
 }
 
 define dso_local swifttailcc i1 @i1test(i32, i32, i32, i32) {
 ; CHECK-LABEL: i1test:
 ; CHECK:    jmp i1test
   entry:
-  %4 = tail call swifttailcc i1 @i1test( i32 %0, i32 %1, i32 %2, i32 %3)
+  %4 = musttail call swifttailcc i1 @i1test( i32 %0, i32 %1, i32 %2, i32 %3)
   ret i1 %4
 }

llvm/test/CodeGen/X86/tailcc-notail.ll

+; RUN: not --crash llc -mtriple=x86_64-linux-gnu %s -o - 2>&1 | FileCheck %s
+
+; CHECK: LLVM ERROR: failed to perform tail call elimination on a call site marked musttail
+declare tailcc [16 x i64] @callee()
+define tailcc [16 x i64] @caller() {
+  %res = musttail call tailcc [16 x i64] @callee()
+  ret [16 x i64] %res
+}

llvm/test/Verifier/swifttailcc-musttail-valid.ll

+; RUN: opt -verify %s
+
+define swifttailcc void @valid_attrs(i64* sret(i64) %ret, i8* byval(i8) %byval, i8* swiftself %self, i8* swiftasync %ctx) {
+  musttail call swifttailcc void @valid_attrs(i64* sret(i64) %ret, i8* byval(i8) %byval, i8* swiftself %self, i8* swiftasync %ctx)
+  ret void
+}
+
+define swifttailcc void @mismatch_parms() {
+  musttail call swifttailcc void @valid_attrs(i64* sret(i64) undef, i8* byval(i8) undef, i8* swiftself undef, i8* swiftasync  undef)
+  ret void
+}

llvm/test/Verifier/swifttailcc-musttail.ll

+; RUN: not opt -verify %s 2>&1 | FileCheck %s
+
+declare swifttailcc void @simple()
+
+define swifttailcc void @inreg(i8* inreg) {
+; CHECK: inreg attribute not allowed in swifttailcc musttail caller
+  musttail call swifttailcc void @simple()
+  ret void
+}
+
+define swifttailcc void @inalloca(i8* inalloca(i8)) {
+; CHECK: inalloca attribute not allowed in swifttailcc musttail caller
+  musttail call swifttailcc void @simple()
+  ret void
+}
+
+define swifttailcc void @swifterror(i8** swifterror) {
+; CHECK: swifterror attribute not allowed in swifttailcc musttail caller
+  musttail call swifttailcc void @simple()
+  ret void
+}
+
+define swifttailcc void @preallocated(i8* preallocated(i8)) {
+; CHECK: preallocated attribute not allowed in swifttailcc musttail caller
+  musttail call swifttailcc void @simple()
+  ret void
+}
+
+define swifttailcc void @byref(i8* byref(i8)) {
+; CHECK: byref attribute not allowed in swifttailcc musttail caller
+  musttail call swifttailcc void @simple()
+  ret void
+}
+
+define swifttailcc void @call_inreg() {
+; CHECK: inreg attribute not allowed in swifttailcc musttail callee
+  musttail call swifttailcc void @inreg(i8* inreg undef)
+  ret void
+}
+
+define swifttailcc void @call_inalloca() {
+; CHECK: inalloca attribute not allowed in swifttailcc musttail callee
+  musttail call swifttailcc void @inalloca(i8* inalloca(i8) undef)
+  ret void
+}
+
+define swifttailcc void @call_swifterror() {
+; CHECK: swifterror attribute not allowed in swifttailcc musttail callee
+  %err = alloca swifterror i8*
+  musttail call swifttailcc void @swifterror(i8** swifterror %err)
+  ret void
+}
+
+define swifttailcc void @call_preallocated() {
+; CHECK: preallocated attribute not allowed in swifttailcc musttail callee
+  musttail call swifttailcc void @preallocated(i8* preallocated(i8) undef)
+  ret void
+}
+
+define swifttailcc void @call_byref() {
+; CHECK: byref attribute not allowed in swifttailcc musttail callee
+  musttail call swifttailcc void @byref(i8* byref(i8) undef)
+  ret void
+}
+
+
+declare swifttailcc void @varargs(...)
+define swifttailcc void @call_varargs(...) {
+; CHECK: cannot guarantee swifttailcc tail call for varargs function
+  musttail call swifttailcc void(...) @varargs(...)
+  ret void
+}

llvm/test/Verifier/tailcc-musttail.ll

+; RUN: not opt -verify %s 2>&1 | FileCheck %s
+
+declare tailcc void @simple()
+
+define tailcc void @inreg(i8* inreg) {
+; CHECK: inreg attribute not allowed in tailcc musttail caller
+  musttail call tailcc void @simple()
+  ret void
+}
+
+define tailcc void @inalloca(i8* inalloca(i8)) {
+; CHECK: inalloca attribute not allowed in tailcc musttail caller
+  musttail call tailcc void @simple()
+  ret void
+}
+
+define tailcc void @swifterror(i8** swifterror) {
+; CHECK: swifterror attribute not allowed in tailcc musttail caller
+  musttail call tailcc void @simple()
+  ret void
+}
+
+define tailcc void @preallocated(i8* preallocated(i8)) {
+; CHECK: preallocated attribute not allowed in tailcc musttail caller
+  musttail call tailcc void @simple()
+  ret void
+}
+
+define tailcc void @byref(i8* byref(i8)) {
+; CHECK: byref attribute not allowed in tailcc musttail caller
+  musttail call tailcc void @simple()
+  ret void
+}
+
+define tailcc void @call_inreg() {
+; CHECK: inreg attribute not allowed in tailcc musttail callee
+  musttail call tailcc void @inreg(i8* inreg undef)
+  ret void
+}
+
+define tailcc void @call_inalloca() {
+; CHECK: inalloca attribute not allowed in tailcc musttail callee
+  musttail call tailcc void @inalloca(i8* inalloca(i8) undef)
+  ret void
+}
+
+define tailcc void @call_swifterror() {
+; CHECK: swifterror attribute not allowed in tailcc musttail callee
+  %err = alloca swifterror i8*
+  musttail call tailcc void @swifterror(i8** swifterror %err)
+  ret void
+}
+
+define tailcc void @call_preallocated() {
+; CHECK: preallocated attribute not allowed in tailcc musttail callee
+  musttail call tailcc void @preallocated(i8* preallocated(i8) undef)
+  ret void
+}
+
+define tailcc void @call_byref() {
+; CHECK: byref attribute not allowed in tailcc musttail callee
+  musttail call tailcc void @byref(i8* byref(i8) undef)
+  ret void
+}
+
+
+declare tailcc void @varargs(...)
+define tailcc void @call_varargs(...) {
+; CHECK: cannot guarantee tailcc tail call for varargs function
+  musttail call tailcc void(...) @varargs(...)
+  ret void
+}