[ARM] Enable tail calls for all Thumb1 targets.

[efriedma]

Summary

It isn't really profitable in general, but it's profitable in cases where there we don't need to spill LR and the callee is a function pointer.

We don't actually generate a tail-call until after isel: we can't tell whether it will be profitable at that point, so we delay the decision to a separate Thumb1TailCallOptimizer pass.

Depends on D49459.

[chill]

LGTM. test/CodeGen/Thumb/PR35481.ll needs an update though ( and a typo there CHECK-V47)

[dmgreen]

Hello. This has code to prevent it from increasing codesize, right? I'm seeing the opposite happen, at least at -Oz. Perhaps because of an interplay between block layout or jump tables or something like it?

One of the examples I looked at is in csibe, with OpenTCP/smtp/smtp_client.c showing some increase.

[efriedma]

I'm not sure how this could increase size; the only thing I can think of is that there's some bad interaction with tail merge. Maybe we need to do reverse transform earlier? (I'd appreciate a testcase, if you have one.)

[efriedma]

Spent some time looking at David's testcases; looks like there are a few problems which lead to codesize increases:

1. SupportsTailCall = true changes the result of ARMTargetLowering::mayBeEmittedAsTailCall, which makes CodeGenPrepare clone a bunch of "ret" instructions. The result should be cleaned up again by tail merge, but the default tail merge threshold is too high ("bl foo; pop {r4, pc}" is two instructions, and the default threshold is 3 instructions).
2. The reverse transform in Thumb1FrameLowering::restoreCalleeSavedRegisters adds the wrong operands to the tPOP_RET instruction; it's based on the arguments to the tail call, not the returned values. As a consequence, tail merge thinks the instructions don't match. Also, there's a theoretical miscompile, but in practice the compiler probably won't insert code there anyway, so I'm not sure how to trigger it. Not sure how to fix this; I don't think we track the necessary information anywhere. (Tail merge could be fixed to merge instructions with mismatched implicit operands, which is probably a good idea anyway, but that wouldn't really solve the underlying incorrect modeling.)

So this needs more work.

[efriedma]

Updated to use a pass after isel to convert a call in tail position to a tail-call, rather than try to do the opposite in frame lowering. I'm not completely happy with this; the pass is a lot more code than I'd really like. But I don't have any better ideas.

Still need to look more at the CodeGenPrepare tail duplication issue; we might need to suppress the transform for Thumb1, even if that means missing some cases we could tail-call. And I probably need some more testcases.

lib/Target/ARM/ARM.h

 FunctionPass *createARMExpandPseudoPass();
 FunctionPass *createARMCodeGenPreparePass();
 FunctionPass *createARMConstantIslandPass();
 FunctionPass *createMLxExpansionPass();
 FunctionPass *createThumb2ITBlockPass();
 FunctionPass *createARMOptimizeBarriersPass();
 FunctionPass *createThumb2SizeReductionPass(
     std::function<bool(const Function &)> Ftor = nullptr);
+FunctionPass *createThumb1TailCallOptimizerPass();
+
 InstructionSelector *
 createARMInstructionSelector(const ARMBaseTargetMachine &TM, const ARMSubtarget &STI,
                              const ARMRegisterBankInfo &RBI);
 
 void LowerARMMachineInstrToMCInst(const MachineInstr *MI, MCInst &OutMI,
                                   ARMAsmPrinter &AP);
 
 void computeBlockSize(MachineFunction *MF, MachineBasicBlock *MBB,

lib/Target/ARM/ARMISelLowering.h

enum NodeType : unsigned {
       WrapperJT,    // WrapperJT - A wrapper node for TargetJumpTable
 
       // Add pseudo op to model memcpy for struct byval.
       COPY_STRUCT_BYVAL,
 
       CALL,         // Function call.
       CALL_PRED,    // Function call that's predicable.
       CALL_NOLINK,  // Function call with branch not branch-and-link.
+      CALL_TAIL,    // Function call that can be transformed to a tail call.
       BRCOND,       // Conditional branch.
       BR_JT,        // Jumptable branch.
       BR2_JT,       // Jumptable branch (2 level - jumptable entry is a jump).
       RET_FLAG,     // Return with a flag operand.
       INTRET_FLAG,  // Interrupt return with an LR-offset and a flag operand.
 
       PIC_ADD,      // Add with a PC operand and a PIC label.
 

lib/Target/ARM/ARMISelLowering.cpp

 const char *ARMTargetLowering::getTargetNodeName(unsigned Opcode) const {
   switch ((ARMISD::NodeType)Opcode) {
   case ARMISD::FIRST_NUMBER:  break;
   case ARMISD::Wrapper:       return "ARMISD::Wrapper";
   case ARMISD::WrapperPIC:    return "ARMISD::WrapperPIC";
   case ARMISD::WrapperJT:     return "ARMISD::WrapperJT";
   case ARMISD::COPY_STRUCT_BYVAL: return "ARMISD::COPY_STRUCT_BYVAL";
   case ARMISD::CALL:          return "ARMISD::CALL";
+  case ARMISD::CALL_TAIL:     return "ARMISD::CALL_TAIL";
   case ARMISD::CALL_PRED:     return "ARMISD::CALL_PRED";
   case ARMISD::CALL_NOLINK:   return "ARMISD::CALL_NOLINK";
   case ARMISD::BRCOND:        return "ARMISD::BRCOND";
   case ARMISD::BR_JT:         return "ARMISD::BR_JT";
   case ARMISD::BR2_JT:        return "ARMISD::BR2_JT";
   case ARMISD::RET_FLAG:      return "ARMISD::RET_FLAG";
   case ARMISD::INTRET_FLAG:   return "ARMISD::INTRET_FLAG";
   case ARMISD::PIC_ADD:       return "ARMISD::PIC_ADD";
ARMTargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI,
   CallingConv::ID CallConv              = CLI.CallConv;
   bool doesNotRet                       = CLI.DoesNotReturn;
   bool isVarArg                         = CLI.IsVarArg;
 
   MachineFunction &MF = DAG.getMachineFunction();
   bool isStructRet    = (Outs.empty()) ? false : Outs[0].Flags.isSRet();
   bool isThisReturn   = false;
   bool isSibCall      = false;
+  bool isThumb1TailCall = false;
   auto Attr = MF.getFunction().getFnAttribute("disable-tail-calls");
 
   // Disable tail calls if they're not supported.
   if (!Subtarget->supportsTailCall() || Attr.getValueAsString() == "true")
     isTailCall = false;
 
   if (isTailCall) {
     // Check if it's really possible to do a tail call.
     isTailCall = IsEligibleForTailCallOptimization(Callee, CallConv,
                     isVarArg, isStructRet, MF.getFunction().hasStructRetAttr(),
                                                    Outs, OutVals, Ins, DAG);
-    if (!isTailCall && CLI.CS && CLI.CS.isMustTailCall())
+    if ((!isTailCall || Subtarget->isThumb1Only()) &&
+        CLI.CS && CLI.CS.isMustTailCall())
       report_fatal_error("failed to perform tail call elimination on a call "
                          "site marked musttail");
+    if (isTailCall && Subtarget->isThumb1Only()) {
+      // For Thumb1, we make a "tail call" which is sort of like an LLVM IR
+      // tail call; it's not a terminator, just a marker indicating that we
+      // could transform it to a terminator later.
+      ++NumTailCalls;
+      isThumb1TailCall = true;
+      isTailCall = false;
+    }
     // We don't support GuaranteedTailCallOpt for ARM, only automatically
     // detected sibcalls.
     if (isTailCall) {
       ++NumTailCalls;
       isSibCall = true;
     }
   }
 
if (Subtarget->genLongCalls()) {
     }
   }
 
   // FIXME: handle tail calls differently.
   unsigned CallOpc;
   if (Subtarget->isThumb()) {
     if ((!isDirect || isARMFunc) && !Subtarget->hasV5TOps())
       CallOpc = ARMISD::CALL_NOLINK;
+    else if (isThumb1TailCall)
+      CallOpc = ARMISD::CALL_TAIL;
     else
       CallOpc = ARMISD::CALL;
   } else {
     if (!isDirect && !Subtarget->hasV5TOps())
       CallOpc = ARMISD::CALL_NOLINK;
     else if (doesNotRet && isDirect && Subtarget->hasRetAddrStack() &&
              // Emit regular call when code size is the priority
              !MF.getFunction().optForMinSize())
ARMTargetLowering::IsEligibleForTailCallOptimization(SDValue Callee,
                                     const SmallVectorImpl<ISD::InputArg> &Ins,
                                                      SelectionDAG& DAG) const {
   MachineFunction &MF = DAG.getMachineFunction();
   const Function &CallerF = MF.getFunction();
   CallingConv::ID CallerCC = CallerF.getCallingConv();
 
   assert(Subtarget->supportsTailCall());
 
-  // Tail calls to function pointers cannot be optimized for Thumb1 if the args
-  // to the call take up r0-r3. The reason is that there are no legal registers
-  // left to hold the pointer to the function to be called.
-  if (Subtarget->isThumb1Only() && Outs.size() >= 4 &&
-      !isa<GlobalAddressSDNode>(Callee.getNode()))
+  if (Subtarget->isThumb1Only()) {
+    // FIXME: This approximation isn't right for non-ELF targets.
+    if (!Subtarget->isTargetELF())
+      return false;
+    bool IsImmediateCall = isa<GlobalAddressSDNode>(Callee.getNode()) ||
+                           isa<ExternalSymbolSDNode>(Callee.getNode());
+
+    // Tail calls to function pointers cannot be optimized for Thumb1 if the
+    // args to the call take up r0-r3. The reason is that there are no legal
+    // registers left to hold the pointer to the function to be called. r12 is
+    // free, but it would be tricky to emit the right sequence because LLVM
+    // doesn't treat it as allocatable.
+    if (Outs.size() >= 4 && (!Subtarget->hasV8MBaselineOps() ||
+                             !IsImmediateCall))
+      return false;
+
+    // Don't try to emit a tail call on Thumb1 if the callee is a known global;
+    // we would be forced to load the address to a GPR. v8m is the exception: it
+    // supports the required immediate branch. (This restriction shouldn't be
+    // necessary for correctness; it's just a codesize optimization.)
+    if (!Subtarget->hasV8MBaselineOps() && IsImmediateCall)
       return false;
+  }
 
   // Look for obvious safe cases to perform tail call optimization that do not
   // require ABI changes. This is what gcc calls sibcall.
 
   // Exception-handling functions need a special set of instructions to indicate
   // a return to the hardware. Tail-calling another function would probably
   // break this.
   if (CallerF.hasFnAttribute("interrupt"))
ARMTargetLowering::IsEligibleForTailCallOptimization(SDValue Callee,
   // call.
   if (!Outs.empty()) {
     // Check if stack adjustment is needed. For now, do not do this if any
     // argument is passed on the stack.
     SmallVector<CCValAssign, 16> ArgLocs;
     CCState CCInfo(CalleeCC, isVarArg, MF, ArgLocs, C);
     CCInfo.AnalyzeCallOperands(Outs, CCAssignFnForCall(CalleeCC, isVarArg));
     if (CCInfo.getNextStackOffset()) {
+      // On Thumb1, don't tail call functions which pass data on the stack;
+      // we need to perform the transform after isel, and the transform
+      // doesn't currently reason about the stack.
+      if (Subtarget->isThumb1Only())
+        return false;
+
       // Check if the arguments are already laid out in the right way as
       // the caller's fixed stack objects.
       MachineFrameInfo &MFI = MF.getFrameInfo();
       const MachineRegisterInfo *MRI = &MF.getRegInfo();
       const TargetInstrInfo *TII = Subtarget->getInstrInfo();
       for (unsigned i = 0, realArgIdx = 0, e = ArgLocs.size();
            i != e;
            ++i, ++realArgIdx) {

lib/Target/ARM/ARMInstrInfo.td

 def ARMcopystructbyval : SDNode<"ARMISD::COPY_STRUCT_BYVAL" ,
                                 SDT_ARMStructByVal,
                                 [SDNPHasChain, SDNPInGlue, SDNPOutGlue,
                                  SDNPMayStore, SDNPMayLoad]>;
 
 def ARMcall          : SDNode<"ARMISD::CALL", SDT_ARMcall,
                               [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue,
                                SDNPVariadic]>;
+def ARMcall_tail     : SDNode<"ARMISD::CALL_TAIL", SDT_ARMcall,
+                              [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue,
+                               SDNPVariadic]>;
 def ARMcall_pred    : SDNode<"ARMISD::CALL_PRED", SDT_ARMcall,
                               [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue,
                                SDNPVariadic]>;
 def ARMcall_nolink   : SDNode<"ARMISD::CALL_NOLINK", SDT_ARMcall,
                               [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue,
                                SDNPVariadic]>;
 
 def ARMretflag       : SDNode<"ARMISD::RET_FLAG", SDTNone,

lib/Target/ARM/ARMInstrThumb.td

let Uses = [SP] in {
     def tTAILJMPdND : tPseudoExpand<(outs),
                    (ins t_brtarget:$dst, pred:$p),
                    4, IIC_Br, [],
                    (tB t_brtarget:$dst, pred:$p)>,
                  Requires<[IsThumb, IsNotMachO]>, Sched<[WriteBr]>;
   }
 }
 
+let isCall = 1, Defs = [LR], Uses = [SP] in {
+  def tCALL_TAILr : tPseudoExpand<(outs), (ins tcGPR:$func, pred:$p),
+                    2, IIC_Br, [(ARMcall_tail tcGPR:$func)],
+                   (tBLXr pred:$p, GPR:$func)>,
+                   Requires<[IsThumb, HasV5T]>, Sched<[WriteBrL]>;
+  def tCALL_TAILi : tPseudoExpand<(outs), (ins thumb_bl_target:$func, pred:$p),
+                    2, IIC_Br, [(ARMcall_tail tglobaladdr:$func)],
+                   (tBL pred:$p, thumb_bl_target:$func)>,
+                   Requires<[IsThumb, HasV8MBaseline]>, Sched<[WriteBr]>;
+}
+
 
 // A8.6.218 Supervisor Call (Software Interrupt)
 // A8.6.16 B: Encoding T1
 // If Inst{11-8} == 0b1111 then SEE SVC
 let isCall = 1, Uses = [SP] in
 def tSVC : T1pI<(outs), (ins imm0_255:$imm), IIC_Br,
                 "svc", "\t$imm", []>, Encoding16, Sched<[WriteBr]> {
   bits<8> imm;

 
 // JumpTable
 def : T1Pat<(ARMWrapperJT tjumptable:$dst),
             (tLEApcrelJT tjumptable:$dst)>;
 
 // Direct calls
 def : T1Pat<(ARMcall texternalsym:$func), (tBL texternalsym:$func)>,
       Requires<[IsThumb]>;
+def : T1Pat<(ARMcall_tail texternalsym:$func),
+            (tCALL_TAILi texternalsym:$func)>,
+      Requires<[IsThumb]>;
 
 // zextload i1 -> zextload i8
 def : T1Pat<(zextloadi1 t_addrmode_is1:$addr),
             (tLDRBi t_addrmode_is1:$addr)>;
 def : T1Pat<(zextloadi1 t_addrmode_rr:$addr),
             (tLDRBr t_addrmode_rr:$addr)>;
 
 // extload from the stack -> word load from the stack, as it avoids having to

lib/Target/ARM/ARMSubtarget.cpp

void ARMSubtarget::initSubtargetFeatures(StringRef CPU, StringRef FS) {
   if (isTargetWindows())
     NoARM = true;
 
   if (isAAPCS_ABI())
     stackAlignment = 8;
   if (isTargetNaCl() || isAAPCS16_ABI())
     stackAlignment = 16;
 
-  // FIXME: Completely disable sibcall for Thumb1 since ThumbRegisterInfo::
-  // emitEpilogue is not ready for them. Thumb tail calls also use t2B, as
-  // the Thumb1 16-bit unconditional branch doesn't have sufficient relocation
-  // support in the assembler and linker to be used. This would need to be
-  // fixed to fully support tail calls in Thumb1.
-  //
-  // For ARMv8-M, we /do/ implement tail calls.  Doing this is tricky for v8-M
-  // baseline, since the LDM/POP instruction on Thumb doesn't take LR.  This
-  // means if we need to reload LR, it takes extra instructions, which outweighs
-  // the value of the tail call; but here we don't know yet whether LR is going
-  // to be used. We take the optimistic approach of generating the tail call and
-  // perhaps taking a hit if we need to restore the LR.
-
-  // Thumb1 PIC calls to external symbols use BX, so they can be tail calls,
-  // but we need to make sure there are enough registers; the only valid
-  // registers are the 4 used for parameters.  We don't currently do this
-  // case.
-
-  SupportsTailCall = !isThumb() || hasV8MBaselineOps();
-
   if (isTargetMachO() && isTargetIOS() && getTargetTriple().isOSVersionLT(5, 0))
     SupportsTailCall = false;
+  else
+    SupportsTailCall = true;
 
   switch (IT) {
   case DefaultIT:
     RestrictIT = hasV8Ops();
     break;
   case RestrictedIT:
     RestrictIT = true;
     break;

lib/Target/ARM/ARMTargetMachine.cpp

void ARMPassConfig::addPreRegAlloc() {
   if (getOptLevel() != CodeGenOpt::None) {
     addPass(createMLxExpansionPass());
 
     if (EnableARMLoadStoreOpt)
       addPass(createARMLoadStoreOptimizationPass(/* pre-register alloc */ true));
 
     if (!DisableA15SDOptimization)
       addPass(createA15SDOptimizerPass());
+
+    addPass(createThumb1TailCallOptimizerPass());
   }
 }
 
 void ARMPassConfig::addPreSched2() {
   if (getOptLevel() != CodeGenOpt::None) {
     if (EnableARMLoadStoreOpt)
       addPass(createARMLoadStoreOptimizationPass());
 

lib/Target/ARM/CMakeLists.txt

add_llvm_target(ARMCodeGen
   ARMSelectionDAGInfo.cpp
   ARMSubtarget.cpp
   ARMTargetMachine.cpp
   ARMTargetObjectFile.cpp
   ARMTargetTransformInfo.cpp
   MLxExpansionPass.cpp
   Thumb1FrameLowering.cpp
   Thumb1InstrInfo.cpp
+  Thumb1TailCallOptimizer.cpp
   ThumbRegisterInfo.cpp
   Thumb2ITBlockPass.cpp
   Thumb2InstrInfo.cpp
   Thumb2SizeReduction.cpp
   ARMComputeBlockSize.cpp
   )
 
 add_subdirectory(AsmParser)
 add_subdirectory(Disassembler)
 add_subdirectory(InstPrinter)
 add_subdirectory(MCTargetDesc)
 add_subdirectory(TargetInfo)
 add_subdirectory(Utils)

lib/Target/ARM/Thumb1TailCallOptimizer.cpp

+//=== Thumb1TailCallOptimizer.cpp - Optimize Thumb1 tail calls-------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// On Thumb1, tail calls are restricted due to the limited instruction set.
+// The key restriction that we need to deal with here, rather than during
+// isel, is that restoring lr from the stack is expensive: it adds at least
+// four bytes to the epilogue.  So if we're optimizing for size, we must avoid
+// avoid emitting any tail calls in a function that contains any non-tail
+// call.
+//
+// This pass scans each Thumb1 function for non-tail calls; if it finds any,
+// it does nothing.  Otherwise, it takes each call with a tail marking,
+// and transforms it to a branch.
+//
+//===----------------------------------------------------------------------===//
+
+#include "ARM.h"
+#include "ARMSubtarget.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "thumb1-tail-call-optimizer"
+
+namespace {
+  struct Thumb1TailCallOptimizer : public MachineFunctionPass {
+    static char ID;
+    Thumb1TailCallOptimizer() : MachineFunctionPass(ID) {}
+
+    bool runOnMachineFunction(MachineFunction &Fn) override;
+
+    StringRef getPassName() const override { return "Thumb1 tail call optimizer"; }
+  };
+  char Thumb1TailCallOptimizer::ID = 0;
+} // end anonymous namespace
+
+bool Thumb1TailCallOptimizer::runOnMachineFunction(MachineFunction &MF) {
+  if (skipFunction(MF.getFunction()))
+    return false;
+
+  const ARMSubtarget &STI = MF.getSubtarget<ARMSubtarget>();
+  if (!STI.isThumb1Only() || !MF.getFrameInfo().hasCalls())
+    return false;
+
+  LLVM_DEBUG(dbgs() << "Tail call optimizer analyzing func: " << MF.getName()
+                    << "\n");
+  const MachineFrameInfo &MFI = MF.getFrameInfo();
+  if (!MFI.hasCalls())
+    return false;
+
+  for (auto &MBB : MF) {
+    bool AfterCall = false;
+    for (auto &MI: MBB) {
+      // Make sure no unexpected instruction snuck in after a tail call.
+      if (AfterCall) {
+        if (MI.getOpcode() == ARM::COPY) {
+          // FIXME: Check that the COPY doesn't affect some unexpected
+          // register? Unlikely to matter.
+        } else if (MI.getOpcode() != ARM::ADJCALLSTACKUP &&
+                   MI.getOpcode() != ARM::tBX_RET) {
+          return false;
+        }
+        continue;
+      }
+
+      if (MI.getOpcode() == ARM::tCALL_TAILr ||
+          MI.getOpcode() == ARM::tCALL_TAILi)
+        AfterCall = true;
+      // Reject other calls because the transform would increase codesize.
+      else if (MI.definesRegister(ARM::LR))
+        return false;
+    }
+  }
+
+  bool Changed = false;
+  for (auto &MBB : MF) {
+    bool HasTailCall = false;
+    for (auto &MI: MBB)
+      HasTailCall |= MI.getOpcode() == ARM::tCALL_TAILr ||
+                     MI.getOpcode() == ARM::tCALL_TAILi;
+    if (!HasTailCall)
+      continue;
+
+    const ARMBaseInstrInfo *TII = STI.getInstrInfo();
+    auto MII = std::prev(MBB.end());
+    while (MII->getOpcode() != ARM::tCALL_TAILr &&
+           MII->getOpcode() != ARM::tCALL_TAILi) {
+      if (MII->getOpcode() == ARM::ADJCALLSTACKUP) {
+        --MII;
+        continue;
+      }
+      --MII;
+      std::next(MII)->eraseFromParent();
+    }
+
+    MachineInstr *CallInst = &*MII;
+    LLVM_DEBUG(dbgs() << "Replacing Thumb1 tail call\n" << *CallInst);
+    DebugLoc Loc = CallInst->getDebugLoc();
+    if (CallInst->getOpcode() == ARM::tCALL_TAILr) {
+      unsigned DstReg = CallInst->getOperand(0).getReg();
+      BuildMI(MBB, MBB.end(), Loc, TII->get(ARM::TCRETURNri))
+        .addUse(DstReg, RegState::Kill);
+    } else {
+      BuildMI(MBB, MBB.end(), Loc, TII->get(ARM::TCRETURNdi))
+        .add(CallInst->getOperand(0));
+    }
+    auto NewMI = std::prev(MBB.end());
+
+    for (unsigned i = 3, e = CallInst->getNumOperands(); i != e; ++i) {
+      MachineOperand &Op = CallInst->getOperand(i);
+      if (Op.isReg() && !Op.isDef() && Op.getReg() != ARM::SP)
+        NewMI->addOperand(CallInst->getOperand(i));
+    }
+    CallInst->eraseFromParent();
+    Changed = true;
+  }
+  return Changed;
+}
+
+FunctionPass *llvm::createThumb1TailCallOptimizerPass() {
+  return new Thumb1TailCallOptimizer();
+}
+

test/CodeGen/ARM/thumb_indirect_calls.ll

 ; RUN: llc -mtriple=thumbv4t-eabi %s -o - | FileCheck ---check-prefix=CHECK -check-prefix=CHECK-V4T %s
 ; RUN: llc -mtriple=thumbv5t-eabi %s -o - | FileCheck ---check-prefix=CHECK -check-prefix=CHECK-V5T %s
 
 @f = common global void (i32)* null, align 4
 
 ; CHECK-LABEL: foo:
-define void @foo(i32 %x) {
+define void @foo(i32 %x) "disable-tail-calls"="true" {
 entry:
   %0 = load void (i32)*, void (i32)** @f, align 4
   tail call void %0(i32 %x)
   ret void
 
 ; CHECK: ldr [[TMP:r[0-3]]], [[F:\.[A-Z0-9_]+]]
 ; CHECK: ldr [[CALLEE:r[0-3]]], {{\[}}[[TMP]]{{\]}}
 
 ; CHECK-V4T-NOT: blx
 ; CHECK-V4T: bl [[INDIRECT_PAD:\.Ltmp[0-9]+]]
 ; CHECK-V4T: [[F]]:
 ; CHECK-V4T: [[INDIRECT_PAD]]:
 ; CHECK-V4T-NEXT: bx [[CALLEE]]
 ; CHECK-V5T: blx [[CALLEE]]
 }
 
 ; CHECK-LABEL: bar:
-define void @bar(void (i32)* nocapture %g, i32 %x, void (i32)* nocapture %h) {
+define void @bar(void (i32)* nocapture %g, i32 %x, void (i32)* nocapture %h) "disable-tail-calls"="true" {
 entry:
   tail call void %g(i32 %x)
   tail call void %h(i32 %x)
   ret void
 
 ; CHECK-V4T: bl [[INDIRECT_PAD1:\.Ltmp[0-9]+]]
 ; CHECK-V4T: bl [[INDIRECT_PAD2:\.Ltmp[0-9]+]]
 ; CHECK-V4T: [[INDIRECT_PAD1]]:
 ; CHECK-V4T-NEXT: bx
 ; CHECK-V4T: [[INDIRECT_PAD2]]:
 ; CHECK-V4T-NEXT: bx
 ; CHECK-V5T: blx
 ; CHECK-V5T: blx
 }

test/CodeGen/ARM/v8m-tail-call.ll

-; RUN: llc %s -o - -mtriple=thumbv8m.base | FileCheck %s
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc %s -o - -mtriple=thumbv8m.base -verify-machineinstrs | FileCheck %s
 
 declare i32 @g(...)
 
 declare i32 @h0(i32, i32, i32, i32)
 define hidden i32 @f0() {
+; CHECK-LABEL: f0:
+; CHECK:       @ %bb.0:
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    bl g
+; CHECK-NEXT:    movs r1, #1
+; CHECK-NEXT:    movs r2, #2
+; CHECK-NEXT:    movs r3, #3
+; CHECK-NEXT:    bl h0
+; CHECK-NEXT:    pop {r7, pc}
   %1 = tail call i32 bitcast (i32 (...)* @g to i32 ()*)()
   %2 = tail call i32 @h0(i32 %1, i32 1, i32 2, i32 3)
   ret i32 %2
-; CHECK-LABEL: f0
-; CHECK:      ldr     [[POP:r[4567]]], [sp, #4]
-; CHECK-NEXT: mov     lr, [[POP]]
-; CHECK-NEXT: pop     {{.*}}[[POP]]
-; CHECK-NEXT: add     sp, #4
-; CHECK-NEXT: b       h0
 }
 
 declare i32 @h1(i32)
 define hidden i32 @f1() {
+; CHECK-LABEL: f1:
+; CHECK:       @ %bb.0:
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    bl g
+; CHECK-NEXT:    bl h1
+; CHECK-NEXT:    pop {r7, pc}
   %1 = tail call i32 bitcast (i32 (...)* @g to i32 ()*)()
   %2 = tail call i32 @h1(i32 %1)
   ret i32 %2
-; CHECK-LABEL: f1
-; CHECK: pop     {r7}
-; CHECK: pop     {r1}
-; CHECK: mov     lr, r1
-; CHECK: b       h1
 }
 
 declare i32 @h2(i32, i32, i32, i32, i32)
 define hidden i32 @f2(i32, i32, i32, i32, i32) {
+; CHECK-LABEL: f2:
+; CHECK:       @ %bb.0:
+; CHECK-NEXT:    push {r4, r5, r6, lr}
+; CHECK-NEXT:    sub sp, #8
+; CHECK-NEXT:    mov r4, r3
+; CHECK-NEXT:    mov r5, r2
+; CHECK-NEXT:    mov r6, r1
+; CHECK-NEXT:    bl g
+; CHECK-NEXT:    cbz r0, .LBB2_2
+; CHECK-NEXT:  @ %bb.1:
+; CHECK-NEXT:    ldr r1, [sp, #24]
+; CHECK-NEXT:    mov r2, sp
+; CHECK-NEXT:    str r1, [r2]
+; CHECK-NEXT:    mov r1, r6
+; CHECK-NEXT:    mov r2, r5
+; CHECK-NEXT:    mov r3, r4
+; CHECK-NEXT:    bl h2
+; CHECK-NEXT:    add sp, #8
+; CHECK-NEXT:    pop {r4, r5, r6, pc}
+; CHECK-NEXT:  .LBB2_2:
+; CHECK-NEXT:    movs r0, #0
+; CHECK-NEXT:    mvns r0, r0
+; CHECK-NEXT:    add sp, #8
+; CHECK-NEXT:    pop {r4, r5, r6, pc}
   %6 = tail call i32 bitcast (i32 (...)* @g to i32 ()*)()
   %7 = icmp eq i32 %6, 0
   br i1 %7, label %10, label %8
 
   %9 = tail call i32 @h2(i32 %6, i32 %1, i32 %2, i32 %3, i32 %4)
   br label %10
 
   %11 = phi i32 [ %9, %8 ], [ -1, %5 ]
   ret i32 %11
-; CHECK-LABEL: f2
-; CHECK:      ldr     [[POP:r[4567]]], [sp, #12]
-; CHECK-NEXT: mov     lr, [[POP]]
-; CHECK-NEXT: pop     {{.*}}[[POP]]
-; CHECK-NEXT: add     sp, #4
-; CHECK-NEXT: b       h2
 }
 
 ; Make sure that tail calls to function pointers that require r0-r3 for argument
 ; passing do not break the compiler.
 @fnptr = global i32 (i32, i32, i32, i32)* null
 define i32 @test3() {
 ; CHECK-LABEL: test3:
-; CHECK: blx {{r[0-9]+}}
+; CHECK:       @ %bb.0:
+; CHECK-NEXT:    push {r4, lr}
+; CHECK-NEXT:    movw r0, :lower16:fnptr
+; CHECK-NEXT:    movt r0, :upper16:fnptr
+; CHECK-NEXT:    ldr r4, [r0]
+; CHECK-NEXT:    movs r0, #1
+; CHECK-NEXT:    movs r1, #2
+; CHECK-NEXT:    movs r2, #3
+; CHECK-NEXT:    movs r3, #4
+; CHECK-NEXT:    blx r4
+; CHECK-NEXT:    pop {r4, pc}
   %1 = load i32 (i32, i32, i32, i32)*, i32 (i32, i32, i32, i32)** @fnptr
   %2 = tail call i32 %1(i32 1, i32 2, i32 3, i32 4)
   ret i32 %2
 }
 
 @fnptr2 = global i32 (i32, i32, i64)* null
 define i32 @test4() {
 ; CHECK-LABEL: test4:
-; CHECK: blx {{r[0-9]+}}
+; CHECK:       @ %bb.0:
+; CHECK-NEXT:    push {r4, lr}
+; CHECK-NEXT:    movw r0, :lower16:fnptr2
+; CHECK-NEXT:    movt r0, :upper16:fnptr2
+; CHECK-NEXT:    ldr r4, [r0]
+; CHECK-NEXT:    movs r0, #1
+; CHECK-NEXT:    movs r1, #2
+; CHECK-NEXT:    movs r2, #3
+; CHECK-NEXT:    movs r3, #0
+; CHECK-NEXT:    blx r4
+; CHECK-NEXT:    pop {r4, pc}
   %1 = load i32 (i32, i32, i64)*, i32 (i32, i32, i64)** @fnptr2
   %2 = tail call i32 %1(i32 1, i32 2, i64 3)
   ret i32 %2
 }
 
 ; Check that tail calls to function pointers where not all of r0-r3 are used for
 ; parameter passing are tail-call optimized.
 ; test5: params in r0, r1. r2 & r3 are free.
 @fnptr3 = global i32 (i32, i32)* null
 define i32 @test5() {
 ; CHECK-LABEL: test5:
-; CHECK: ldr [[REG:r[0-9]+]]
-; CHECK: bx [[REG]]
-; CHECK-NOT: blx [[REG]]
+; CHECK:       @ %bb.0:
+; CHECK-NEXT:    movw r0, :lower16:fnptr3
+; CHECK-NEXT:    movt r0, :upper16:fnptr3
+; CHECK-NEXT:    ldr r2, [r0]
+; CHECK-NEXT:    movs r0, #1
+; CHECK-NEXT:    movs r1, #2
+; CHECK-NEXT:    bx r2
   %1 = load i32 (i32, i32)*, i32 (i32, i32)** @fnptr3
   %2 = tail call i32 %1(i32 1, i32 2)
   ret i32 %2
 }
 
 ; test6: params in r0 and r2-r3. r1 is free.
 @fnptr4 = global i32 (i32, i64)* null
 define i32 @test6() {
 ; CHECK-LABEL: test6:
-; CHECK: ldr [[REG:r[0-9]+]]
-; CHECK: bx [[REG]]
-; CHECK-NOT: blx [[REG]]
+; CHECK:       @ %bb.0:
+; CHECK-NEXT:    movw r0, :lower16:fnptr4
+; CHECK-NEXT:    movt r0, :upper16:fnptr4
+; CHECK-NEXT:    ldr r1, [r0]
+; CHECK-NEXT:    movs r0, #1
+; CHECK-NEXT:    movs r2, #2
+; CHECK-NEXT:    movs r3, #0
+; CHECK-NEXT:    bx r1
   %1 = load i32 (i32, i64)*, i32 (i32, i64)** @fnptr4
   %2 = tail call i32 %1(i32 1, i64 2)
   ret i32 %2
 }
 
 ; Check that tail calls to functions other than function pointers are
 ; tail-call optimized.
 define i32 @test7() {
 ; CHECK-LABEL: test7:
-; CHECK: b bar
-; CHECK-NOT: bl bar
+; CHECK:       @ %bb.0:
+; CHECK-NEXT:    movs r0, #1
+; CHECK-NEXT:    movs r1, #2
+; CHECK-NEXT:    movs r2, #3
+; CHECK-NEXT:    movs r3, #4
+; CHECK-NEXT:    b bar
   %tail = tail call i32 @bar(i32 1, i32 2, i32 3, i32 4)
   ret i32 %tail
 }
 
 declare i32 @bar(i32, i32, i32, i32)
 
 ; Regression test for failure to load indirect branch target (class tcGPR) from
 ; a stack slot.
 %struct.S = type { i32 }
 
 define void @test8(i32 (i32, i32, i32)* nocapture %fn, i32 %x) local_unnamed_addr {
+; CHECK-LABEL: test8:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    push {r4, r5, r6, r7, lr}
+; CHECK-NEXT:    sub sp, #4
+; CHECK-NEXT:    mov r4, r1
+; CHECK-NEXT:    str r0, [sp] @ 4-byte Spill
+; CHECK-NEXT:    bl test8_u
+; CHECK-NEXT:    mov r5, r0
+; CHECK-NEXT:    ldr r6, [r0]
+; CHECK-NEXT:    movs r7, #0
+; CHECK-NEXT:    mov r0, r7
+; CHECK-NEXT:    bl test8_h
+; CHECK-NEXT:    mov r1, r0
+; CHECK-NEXT:    mov r0, r6
+; CHECK-NEXT:    mov r2, r7
+; CHECK-NEXT:    bl test8_g
+; CHECK-NEXT:    str r4, [r5]
+; CHECK-NEXT:    movs r0, #1
+; CHECK-NEXT:    movs r1, #2
+; CHECK-NEXT:    movs r2, #3
+; CHECK-NEXT:    ldr r3, [sp] @ 4-byte Reload
+; CHECK-NEXT:    blx r3
+; CHECK-NEXT:    add sp, #4
+; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
 entry:
   %call = tail call %struct.S* bitcast (%struct.S* (...)* @test8_u to %struct.S* ()*)()
   %a = getelementptr inbounds %struct.S, %struct.S* %call, i32 0, i32 0
   %0 = load i32, i32* %a, align 4
   %call1 = tail call i32 @test8_h(i32 0)
   %call2 = tail call i32 @test8_g(i32 %0, i32 %call1, i32 0)
   store i32 %x, i32* %a, align 4
   %call4 = tail call i32 %fn(i32 1, i32 2, i32 3)
   ret void
 }
 
 declare %struct.S* @test8_u(...)
 
 declare i32 @test8_g(i32, i32, i32)
 
 declare i32 @test8_h(i32)
-; CHECK: str r0, [sp] @ 4-byte Spill
-; CHECK: ldr r3, [sp] @ 4-byte Reload
-; CHECK: bx r3