[RISCV] Implement MC layer support for the tail pseudoinstruction

[mgrang]

Summary

This patch implements MC support for tail psuedo instruction.
A follow-up patch implements the codegen support as well as handling of the indirect tail pseudo instruction.

[mgrang]

Here's the patch to generate tail pseudo instruction: https://reviews.llvm.org/D45395

[asb]

This patch would be better named 'Implement MC layer support for the tail pseudoinstruction' or similar. It should apply to the current head ,with D45395 rebased to apply on top of it. This is because MC layer support is a useful incremental change, and can be cleanly separated from the codegen changes (meaning we can revert one without affecting the other).

It looks like you'd just need to add the PseudoTAIL definition to RISCVInstrInfo.td and remove the PseudoTAILIndirect reference from this patch (it can be introduced in D45935).

lib/Target/RISCV/MCTargetDesc/RISCVMCCodeEmitter.cpp

153–155

This line is probably going to be deleted in this patch anyway, but at least for the follow-up patch that includes PseudoTAILIndirect it might be better to have expandFunctionCall return the number of MIs inserted.

[mgrang]

Thanks for your comments @asb. There seems to be some problem with long double function arguments which I wanted to clarify.

The RISCV Calling Convention (https://riscv.org/wp-content/uploads/2015/01/riscv-calling.pdf) says that for soft float abi, long doubles are passed in registers via reference:

In RV32, for example, the function double foo(int, double, long double) is passed its first argument in a0, its second argument in a2 and a3, and its third argument by reference via a4

However, consider this code:

__attribute__((noinline))
int bar(long double a) {
  printf("### Val L: %LF\n", a);
  return a;
}

__attribute__((noinline))
int foo() {
  long double a = 1;
  return bar(1);
}

Here's the assembly for function foo:

Without tail call:
_Z3foov:                                # @_Z3foov
# %bb.0:                                # %entry
	addi	sp, sp, -32
	sw	ra, 28(sp)
	lui	a0, 262128
	sw	a0, 12(sp)
	sw	zero, 8(sp)
	sw	zero, 4(sp)
	sw	zero, 0(sp)
	mv	a0, sp
	call	_Z3bare
	lw	ra, 28(sp)
	addi	sp, sp, 32
	ret

With tail call:
_Z3foov:                                # @_Z3foov
# %bb.0:                                # %entry
	addi	sp, sp, -16
	lui	a0, 262128
	sw	a0, 12(sp)
	sw	zero, 8(sp)
	sw	zero, 4(sp)
	sw	zero, 0(sp)
	mv	a0, sp
	addi	sp, sp, 16
	tail	_Z3bare

I do not understand why the long double argument is passed via stack. For tail call, this results in an incorrect value being printed inside function bar:

### Val: 0.000000

For double arguments, the assembly looks fine:

_Z3foov:                                # @_Z3foov
# %bb.0:                                # %entry
	lui	a1, 261888
	mv	a0, zero
	tail	_Z3bard

[asb]

These are the possibilities for parameter passing on RV32I/RV64I:

1. Directly in a single register
2. Directly in a pair of registers
3. Directly in a register stack location
4. Indirectly in a register
5. Indirectly in a stack location

Because long double (fp128) and i128 are larger than 2*XLEN, they are passed indirectly and options 1-3 don't apply. Therefore the address of the value will be passed in a register, or if not available then the address is put on the stack. In order to pass indirectly, space on the stack often needs to be allocated in order to store the value. You're seeing this in the example code you pasted. You can see the logic for handling this in RISCVISelLowering.cpp if you grep for CCValAssign::Indirect.

Thanks for catching this issue. Rejecting tailcall optimisation when CCInfo.getNextStackOffset() != 0 isn't quit enough, we need to also reject it if any of the CCValAssign ArgsLocs are passed CCValAssign::Indirect (I can see the ARM's isEligibleForTailCall does that check too).

[mgrang]

Thanks Alex for the explanation on indirect parameter passing. I have updated the condition in my patch and added a unit test.

[mgrang]

test/CodeGen/RISCV/tail-calls.ll

1

Note: I have limited this test only to riscv32. With riscv64 I get the following error for fp128 parameter:
LLVM ERROR: Cannot select: t17: i64 = Constant<4611404543450677248>

I guess I can restrict only the failing test to riscv32 but I felt it was cleaner this way. We could enable this for riscv64 once the support for fp128 in riscv64 is more mature.

@asb What do you suggest?

[mgrang]

test/CodeGen/RISCV/tail-calls.ll

1

Please disregard the above comment. It was meant to be added to D45395.

[asb]

Thanks Mandeep, just a few minor cleanups

lib/Target/RISCV/RISCVISelLowering.cpp

1546–1547

This change is needed for codegen rather than MC and should go in D45395

lib/Target/RISCV/RISCVISelLowering.h

32

This change is needed for codegen rather than MC and should go in D45395

lib/Target/RISCV/RISCVInstrInfo.td

41–43

This change is needed for codegen rather than MC and should go in D45395

666

Do we actually need Uses = [X2]?

672–675

This change is needed for codegen rather than MC and should go in D45395

test/MC/RISCV/tail-call-invalid.s

1 ↗

(On Diff #145065)

Please add a riscv64 RUN line

test/MC/RISCV/tail-call.s

7 ↗

(On Diff #145065)

Please add riscv64 RUN lines

[mgrang]

Addressed comments.

[mgrang]

lib/Target/RISCV/RISCVInstrInfo.td

666

I decided to align this with ARM which has Uses = [SP] in its definition of Tail.

[mgrang]

lib/Target/RISCV/RISCVInstrInfo.td

667

If I remove the definition for the Tail node then it cannot determine hasSideEfects, mayStore and mayLoad properties:

def Tail         : SDNode<"RISCVISD::TAIL", SDT_RISCVCall,
                          [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue,
                           SDNPVariadic]>;

So I had to explicitly specify them here. We can remove these in the next patch which includes the definition for Tail.

[asb]

Thanks, looks good to me.

lib/Target/RISCV/MCTargetDesc/RISCVMCCodeEmitter.cpp

RISCVMCCodeEmitter(MCContext &ctx, MCInstrInfo const &MCII)
       : Ctx(ctx), MCII(MCII) {}
 
   ~RISCVMCCodeEmitter() override {}
 
   void encodeInstruction(const MCInst &MI, raw_ostream &OS,
                          SmallVectorImpl<MCFixup> &Fixups,
                          const MCSubtargetInfo &STI) const override;
 
-  void expandFunctionCall(const MCInst &MI, raw_ostream &OS,
+  unsigned expandFunctionCall(const MCInst &MI, raw_ostream &OS,
                               SmallVectorImpl<MCFixup> &Fixups,
                               const MCSubtargetInfo &STI) const;
 
   /// TableGen'erated function for getting the binary encoding for an
   /// instruction.
   uint64_t getBinaryCodeForInstr(const MCInst &MI,
                                  SmallVectorImpl<MCFixup> &Fixups,
                                  const MCSubtargetInfo &STI) const;
 
   /// Return binary encoding of operand. If the machine operand requires

 
 // Expand PseudoCALL to AUIPC and JALR with relocation types.
 // We expand PseudoCALL while encoding, meaning AUIPC and JALR won't go through
 // RISCV MC to MC compressed instruction transformation. This is acceptable
 // because AUIPC has no 16-bit form and C_JALR have no immediate operand field.
 // We let linker relaxation deal with it. When linker relaxation enabled,
 // AUIPC and JALR have chance relax to JAL. If C extension is enabled,
 // JAL has chance relax to C_JAL.
-void RISCVMCCodeEmitter::expandFunctionCall(const MCInst &MI, raw_ostream &OS,
-                                            SmallVectorImpl<MCFixup> &Fixups,
-                                            const MCSubtargetInfo &STI) const {
+unsigned
+RISCVMCCodeEmitter::expandFunctionCall(const MCInst &MI, raw_ostream &OS,
   MCInst TmpInst;
   MCOperand Func = MI.getOperand(0);
-  unsigned Ra = (MI.getOpcode() == RISCV::PseudoTAIL) ? RISCV::X6 : RISCV::X1;
   uint32_t Binary;
 
+  // For an indirect tail call, emit JALR X0, 0(Reg).
+  // TODO: We had to custom lower this here since for tail calls we cannot
+  // allow JALR with any GPR; we need to restrict the register class to only
+  // caller-saved regs [x5-x7]. Hence we had to create a new regclass GPRTC
+  // which is a subclass of GPR. However, currently tablegen does not support
+  // the use of the registers of a subclass of the allowed regclass for an
+  // instruction. Once this is supported, we can replace the custom lowering
+  // here with a PseudoInstExpansion rule in RISCVInstrInfo.td.
+  if (MI.getOpcode() == RISCV::PseudoTAILIndirect) {
+    TmpInst = MCInstBuilder(RISCV::JALR)
+                .addReg(RISCV::X0)
+                .addReg(Func.getReg())
+                .addImm(0);
+    Binary = getBinaryCodeForInstr(TmpInst, Fixups, STI);
+    support::endian::Writer<support::little>(OS).write(Binary);
+    return 1 /* MCNumEmitted */;
+  }
+
+  unsigned Ra = (MI.getOpcode() == RISCV::PseudoTAIL) ? RISCV::X6 : RISCV::X1;
+
   assert(Func.isExpr() && "Expected expression");
 
   const MCExpr *Expr = Func.getExpr();
 
   // Create function call expression CallExpr for AUIPC.
   const MCExpr *CallExpr =
       RISCVMCExpr::create(Expr, RISCVMCExpr::VK_RISCV_CALL, Ctx);
 
   // Emit AUIPC Ra, Func with R_RISCV_CALL relocation type.
   TmpInst = MCInstBuilder(RISCV::AUIPC)
                 .addReg(Ra)
                 .addOperand(MCOperand::createExpr(CallExpr));
   Binary = getBinaryCodeForInstr(TmpInst, Fixups, STI);
   support::endian::Writer<support::little>(OS).write(Binary);
 
   // Emit JALR Ra, Ra, 0
   TmpInst = MCInstBuilder(RISCV::JALR).addReg(Ra).addReg(Ra).addImm(0);
   Binary = getBinaryCodeForInstr(TmpInst, Fixups, STI);
   support::endian::Writer<support::little>(OS).write(Binary);
+  return 2 /* MCNumEmitted */;
 }
 
 void RISCVMCCodeEmitter::encodeInstruction(const MCInst &MI, raw_ostream &OS,
                                            SmallVectorImpl<MCFixup> &Fixups,
                                            const MCSubtargetInfo &STI) const {
   const MCInstrDesc &Desc = MCII.get(MI.getOpcode());
   // Get byte count of instruction.
   unsigned Size = Desc.getSize();
 
   if (MI.getOpcode() == RISCV::PseudoCALL ||
-      MI.getOpcode() == RISCV::PseudoTAIL) {
-    expandFunctionCall(MI, OS, Fixups, STI);
-    MCNumEmitted += 2;
+      MI.getOpcode() == RISCV::PseudoTAIL ||
+      MI.getOpcode() == RISCV::PseudoTAILIndirect) {
+    MCNumEmitted += expandFunctionCall(MI, OS, Fixups, STI);
     return;
   }
 
   switch (Size) {
   default:
     llvm_unreachable("Unhandled encodeInstruction length!");
   case 2: {
     uint16_t Bits = getBinaryCodeForInstr(MI, Fixups, STI);

lib/Target/RISCV/RISCVISelLowering.h

 namespace RISCVISD {
 enum NodeType : unsigned {
   FIRST_NUMBER = ISD::BUILTIN_OP_END,
   RET_FLAG,
   CALL,
   SELECT_CC,
   BuildPairF64,
   SplitF64,
   TAIL
 };
 }
 
 class RISCVTargetLowering : public TargetLowering {
   const RISCVSubtarget &Subtarget;
 
 public:
   explicit RISCVTargetLowering(const TargetMachine &TM,
private:
   SDValue lowerGlobalAddress(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerBlockAddress(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerConstantPool(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerExternalSymbol(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerSELECT(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerVASTART(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFRAMEADDR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerRETURNADDR(SDValue Op, SelectionDAG &DAG) const;
+
+  bool IsEligibleForTailCallOptimization(CCState &CCInfo,
+    CallLoweringInfo &CLI, MachineFunction &MF,
+    const SmallVector<CCValAssign, 16> &ArgLocs) const;
 };
 }
 
 #endif

lib/Target/RISCV/RISCVISelLowering.cpp

 //===----------------------------------------------------------------------===//
 
 #include "RISCVISelLowering.h"
 #include "RISCV.h"
 #include "RISCVMachineFunctionInfo.h"
 #include "RISCVRegisterInfo.h"
 #include "RISCVSubtarget.h"
 #include "RISCVTargetMachine.h"
+#include "llvm/ADT/Statistic.h"
 #include "llvm/CodeGen/CallingConvLower.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/SelectionDAGISel.h"
 #include "llvm/CodeGen/TargetLoweringObjectFileImpl.h"
 #include "llvm/CodeGen/ValueTypes.h"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/DiagnosticPrinter.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/raw_ostream.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "riscv-lower"
 
+STATISTIC(NumTailCalls, "Number of tail calls");
+
 RISCVTargetLowering::RISCVTargetLowering(const TargetMachine &TM,
                                          const RISCVSubtarget &STI)
     : TargetLowering(TM), Subtarget(STI) {
 
   MVT XLenVT = Subtarget.getXLenVT();
 
   // Set up the register classes.
   addRegisterClass(XLenVT, &RISCV::GPRRegClass);
SDValue RISCVTargetLowering::LowerFormalArguments(
   if (!OutChains.empty()) {
     OutChains.push_back(Chain);
     Chain = DAG.getNode(ISD::TokenFactor, DL, MVT::Other, OutChains);
   }
 
   return Chain;
 }
 
+/// IsEligibleForTailCallOptimization - Check whether the call is eligible
+/// for tail call optimization.
+/// Note: This is modelled after ARM's IsEligibleForTailCallOptimization.
+bool RISCVTargetLowering::IsEligibleForTailCallOptimization(
+  CCState &CCInfo, CallLoweringInfo &CLI, MachineFunction &MF,
+  const SmallVector<CCValAssign, 16> &ArgLocs) const {
+
+  auto &Callee = CLI.Callee;
+  auto CalleeCC = CLI.CallConv;
+  auto IsVarArg = CLI.IsVarArg;
+  auto &Outs = CLI.Outs;
+  auto &Caller = MF.getFunction();
+  auto CallerCC = Caller.getCallingConv();
+
+  // Do not tail call opt functions with "disable-tail-calls" attribute.
+  if (Caller.getFnAttribute("disable-tail-calls").getValueAsString() == "true")
+    return false;
+
+  // Exception-handling functions need a special set of instructions to
+  // indicate a return to the hardware. Tail-calling another function would
+  // probably break this.
+  // TODO: The "interrupt" attribute isn't currently defined by RISC-V. This
+  // should be expanded as new function attributes are introduced.
+  if (Caller.hasFnAttribute("interrupt"))
+    return false;
+
+  // Do not tail call opt functions with varargs.
+  if (IsVarArg)
+    return false;
+
+  // Do not tail call opt if the stack is used to pass parameters.
+  if (CCInfo.getNextStackOffset() != 0)
+    return false;
+
+  // Do not tail call opt if any parameters need to be passed indirectly.
+  // Since long doubles (fp128) and i128 are larger than 2*XLEN, they are
+  // passed indirectly. So the address of the value will be passed in a
+  // register, or if not available, then the address is put on the stack. In
+  // order to pass indirectly, space on the stack often needs to be allocated
+  // in order to store the value. In this case the CCInfo.getNextStackOffset()
+  // != 0 check is not enough and we need to check if any CCValAssign ArgsLocs
+  // are passed CCValAssign::Indirect.
+  for (auto &VA : ArgLocs)
+    if (VA.getLocInfo() == CCValAssign::Indirect)
+      return false;
+
+  // Do not tail call opt if either caller or callee uses struct return
+  // semantics.
+  auto IsCallerStructRet = Caller.hasStructRetAttr();
+  auto IsCalleeStructRet = Outs.empty() ? false : Outs[0].Flags.isSRet();
+  if (IsCallerStructRet || IsCalleeStructRet)
+    return false;
+
+  // Externally-defined functions with weak linkage should not be
+  // tail-called. The behaviour of branch instructions in this situation (as
+  // used for tail calls) is implementation-defined, so we cannot rely on the
+  // linker replacing the tail call with a return.
+  if (GlobalAddressSDNode *G = dyn_cast<GlobalAddressSDNode>(Callee)) {
+    const GlobalValue *GV = G->getGlobal();
+    if (GV->hasExternalWeakLinkage())
+      return false;
+  }
+
+  // The callee has to preserve all registers the caller needs to preserve.
+  const RISCVRegisterInfo *TRI = Subtarget.getRegisterInfo();
+  const uint32_t *CallerPreserved = TRI->getCallPreservedMask(MF, CallerCC);
+  if (CalleeCC != CallerCC) {
+    const uint32_t *CalleePreserved = TRI->getCallPreservedMask(MF, CalleeCC);
+    if (!TRI->regmaskSubsetEqual(CallerPreserved, CalleePreserved))
+      return false;
+  }
+
+  // Byval parameters hand the function a pointer directly into the stack area
+  // we want to reuse during a tail call. Working around this *is* possible
+  // but less efficient and uglier in LowerCall.
+  for (auto &Arg : Outs)
+    if (Arg.Flags.isByVal())
+      return false;
+
+  return true;
+}
+
 // Lower a call to a callseq_start + CALL + callseq_end chain, and add input
 // and output parameter nodes.
 SDValue RISCVTargetLowering::LowerCall(CallLoweringInfo &CLI,
                                        SmallVectorImpl<SDValue> &InVals) const {
   SelectionDAG &DAG = CLI.DAG;
   SDLoc &DL = CLI.DL;
   SmallVectorImpl<ISD::OutputArg> &Outs = CLI.Outs;
   SmallVectorImpl<SDValue> &OutVals = CLI.OutVals;
   SmallVectorImpl<ISD::InputArg> &Ins = CLI.Ins;
   SDValue Chain = CLI.Chain;
   SDValue Callee = CLI.Callee;
-  CLI.IsTailCall = false;
+  bool &IsTailCall = CLI.IsTailCall;
   CallingConv::ID CallConv = CLI.CallConv;
   bool IsVarArg = CLI.IsVarArg;
   EVT PtrVT = getPointerTy(DAG.getDataLayout());
   MVT XLenVT = Subtarget.getXLenVT();
 
   MachineFunction &MF = DAG.getMachineFunction();
 
   // Analyze the operands of the call, assigning locations to each operand.
   SmallVector<CCValAssign, 16> ArgLocs;
   CCState ArgCCInfo(CallConv, IsVarArg, MF, ArgLocs, *DAG.getContext());
   analyzeOutputArgs(MF, ArgCCInfo, Outs, /*IsRet=*/false, &CLI);
 
+  // Check if it's really possible to do a tail call.
+  if (IsTailCall)
+    IsTailCall = IsEligibleForTailCallOptimization(ArgCCInfo, CLI, MF,
+                                                   ArgLocs);
+
+  if (IsTailCall)
+    ++NumTailCalls;
+  else if (CLI.CS && CLI.CS.isMustTailCall())
+    report_fatal_error("failed to perform tail call elimination on a call "
+                       "site marked musttail");
+
   // Get a count of how many bytes are to be pushed on the stack.
   unsigned NumBytes = ArgCCInfo.getNextStackOffset();
 
   // Create local copies for byval args
   SmallVector<SDValue, 8> ByValArgs;
   for (unsigned i = 0, e = Outs.size(); i != e; ++i) {
     ISD::ArgFlagsTy Flags = Outs[i].Flags;
     if (!Flags.isByVal())
       continue;
 
     SDValue Arg = OutVals[i];
     unsigned Size = Flags.getByValSize();
     unsigned Align = Flags.getByValAlign();
 
     int FI = MF.getFrameInfo().CreateStackObject(Size, Align, /*isSS=*/false);
     SDValue FIPtr = DAG.getFrameIndex(FI, getPointerTy(DAG.getDataLayout()));
     SDValue SizeNode = DAG.getConstant(Size, DL, XLenVT);
 
     Chain = DAG.getMemcpy(Chain, DL, FIPtr, Arg, SizeNode, Align,
                           /*IsVolatile=*/false,
                           /*AlwaysInline=*/false,
-                          /*isTailCall=*/false, MachinePointerInfo(),
+                          IsTailCall, MachinePointerInfo(),
                           MachinePointerInfo());
     ByValArgs.push_back(FIPtr);
   }
 
+  if (!IsTailCall)
     Chain = DAG.getCALLSEQ_START(Chain, NumBytes, 0, CLI.DL);
 
   // Copy argument values to their designated locations.
   SmallVector<std::pair<unsigned, SDValue>, 8> RegsToPass;
   SmallVector<SDValue, 8> MemOpChains;
   SDValue StackPtr;
   for (unsigned i = 0, j = 0, e = ArgLocs.size(); i != e; ++i) {
     CCValAssign &VA = ArgLocs[i];
     SDValue ArgValue = OutVals[i];
if (VA.isRegLoc()) {
       // Queue up the argument copies and emit them at the end.
       RegsToPass.push_back(std::make_pair(VA.getLocReg(), ArgValue));
     } else {
       assert(VA.isMemLoc() && "Argument not register or memory");
 
       // Work out the address of the stack slot.
       if (!StackPtr.getNode())
         StackPtr = DAG.getCopyFromReg(Chain, DL, RISCV::X2, PtrVT);
+
+      if (!IsTailCall) {
         SDValue Address =
             DAG.getNode(ISD::ADD, DL, PtrVT, StackPtr,
                         DAG.getIntPtrConstant(VA.getLocMemOffset(), DL));
 
         // Emit the store.
         MemOpChains.push_back(
             DAG.getStore(Chain, DL, ArgValue, Address, MachinePointerInfo()));
       }
     }
+  }
 
   // Join the stores, which are independent of one another.
   if (!MemOpChains.empty())
     Chain = DAG.getNode(ISD::TokenFactor, DL, MVT::Other, MemOpChains);
 
   SDValue Glue;
 
+  // Tail call byval lowering might overwrite argument registers so in case of
+  // tail call optimization the copies to registers are lowered to 'real' stack
+  // slot.
+  // Force all the incoming stack arguments to be loaded from the stack
+  // before any new outgoing arguments are stored to the stack, because the
+  // outgoing stack slots may alias the incoming argument stack slots, and
+  // the alias isn't otherwise explicit. This is slightly more conservative
+  // than necessary, because it means that each store effectively depends
+  // on every argument instead of just those arguments it would clobber.
+  if (IsTailCall)
+    Glue = SDValue();
+
   // Build a sequence of copy-to-reg nodes, chained and glued together.
   for (auto &Reg : RegsToPass) {
     Chain = DAG.getCopyToReg(Chain, DL, Reg.first, Reg.second, Glue);
     Glue = Chain.getValue(1);
   }
 
+  if (IsTailCall)
+    Glue = SDValue();
+
   // If the callee is a GlobalAddress/ExternalSymbol node, turn it into a
   // TargetGlobalAddress/TargetExternalSymbol node so that legalize won't
   // split it and then direct call can be matched by PseudoCALL.
   if (GlobalAddressSDNode *S = dyn_cast<GlobalAddressSDNode>(Callee)) {
     Callee = DAG.getTargetGlobalAddress(S->getGlobal(), DL, PtrVT, 0, 0);
   } else if (ExternalSymbolSDNode *S = dyn_cast<ExternalSymbolSDNode>(Callee)) {
     Callee = DAG.getTargetExternalSymbol(S->getSymbol(), PtrVT, 0);
   }
 
   // The first call operand is the chain and the second is the target address.
   SmallVector<SDValue, 8> Ops;
   Ops.push_back(Chain);
   Ops.push_back(Callee);
 
   // Add argument registers to the end of the list so that they are
   // known live into the call.
   for (auto &Reg : RegsToPass)
     Ops.push_back(DAG.getRegister(Reg.first, Reg.second.getValueType()));
 
+  if (!IsTailCall) {
     // Add a register mask operand representing the call-preserved registers.
     const TargetRegisterInfo *TRI = Subtarget.getRegisterInfo();
     const uint32_t *Mask = TRI->getCallPreservedMask(MF, CallConv);
     assert(Mask && "Missing call preserved mask for calling convention");
     Ops.push_back(DAG.getRegisterMask(Mask));
+  }
 
   // Glue the call to the argument copies, if any.
   if (Glue.getNode())
     Ops.push_back(Glue);
 
   // Emit the call.
   SDVTList NodeTys = DAG.getVTList(MVT::Other, MVT::Glue);
+
+  if (IsTailCall) {
+    MF.getFrameInfo().setHasTailCall();
+    return DAG.getNode(RISCVISD::TAIL, DL, NodeTys, Ops);
+  }
+
   Chain = DAG.getNode(RISCVISD::CALL, DL, NodeTys, Ops);
   Glue = Chain.getValue(1);
 
   // Mark the end of the call, which is glued to the call itself.
   Chain = DAG.getCALLSEQ_END(Chain,
                              DAG.getConstant(NumBytes, DL, PtrVT, true),
                              DAG.getConstant(0, DL, PtrVT, true),
                              Glue, DL);
const char *RISCVTargetLowering::getTargetNodeName(unsigned Opcode) const {
   case RISCVISD::CALL:
     return "RISCVISD::CALL";
   case RISCVISD::SELECT_CC:
     return "RISCVISD::SELECT_CC";
   case RISCVISD::BuildPairF64:
     return "RISCVISD::BuildPairF64";
   case RISCVISD::SplitF64:
     return "RISCVISD::SplitF64";
   case RISCVISD::TAIL:
     return "RISCVISD::TAIL";
   }
   return nullptr;
 }
 
 std::pair<unsigned, const TargetRegisterClass *>
 RISCVTargetLowering::getRegForInlineAsmConstraint(const TargetRegisterInfo *TRI,
                                                   StringRef Constraint,
                                                   MVT VT) const {

lib/Target/RISCV/RISCVInstrInfo.cpp

unsigned RISCVInstrInfo::getInstSizeInBytes(const MachineInstr &MI) const {
 
   switch (Opcode) {
   default: { return get(Opcode).getSize(); }
   case TargetOpcode::EH_LABEL:
   case TargetOpcode::IMPLICIT_DEF:
   case TargetOpcode::KILL:
   case TargetOpcode::DBG_VALUE:
     return 0;
+  case RISCV::PseudoTAILIndirect:
+    return 4;
   case RISCV::PseudoCALL:
+  case RISCV::PseudoTAIL:
     return 8;
   case TargetOpcode::INLINEASM: {
     const MachineFunction &MF = *MI.getParent()->getParent();
     const auto &TM = static_cast<const RISCVTargetMachine &>(MF.getTarget());
     return getInlineAsmLength(MI.getOperand(0).getSymbolName(),
                               *TM.getMCAsmInfo());
   }
   }
 }

lib/Target/RISCV/RISCVInstrInfo.td

 def CallSeqStart : SDNode<"ISD::CALLSEQ_START", SDT_RISCVCallSeqStart,
                           [SDNPHasChain, SDNPOutGlue]>;
 def CallSeqEnd   : SDNode<"ISD::CALLSEQ_END", SDT_RISCVCallSeqEnd,
                           [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue]>;
 def RetFlag      : SDNode<"RISCVISD::RET_FLAG", SDTNone,
                           [SDNPHasChain, SDNPOptInGlue, SDNPVariadic]>;
 def SelectCC     : SDNode<"RISCVISD::SELECT_CC", SDT_RISCVSelectCC,
                           [SDNPInGlue]>;
 def Tail         : SDNode<"RISCVISD::TAIL", SDT_RISCVCall,
                           [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue,
                            SDNPVariadic]>;
 
 //===----------------------------------------------------------------------===//
 // Operand and SDNode transformation definitions.
 //===----------------------------------------------------------------------===//
 
 class ImmAsmOperand<string prefix, int width, string suffix> : AsmOperandClass {
   let Name = prefix # "Imm" # width # suffix;
   let RenderMethod = "addImmOperands";

 
 let isBarrier = 1, isReturn = 1, isTerminator = 1 in
 def PseudoRET : Pseudo<(outs), (ins), [(RetFlag)]>,
                 PseudoInstExpansion<(JALR X0, X1, 0)>;
 
 // PseudoTAIL is a pseudo instruction similar to PseudoCALL and will eventually
 // expand to auipc and jalr while encoding.
 // Define AsmString to print "tail" when compile with -S flag.
 let isCall = 1, isTerminator = 1, isReturn = 1, isBarrier = 1, Uses = [X2],
     isCodeGenOnly = 0 in
 def PseudoTAIL : Pseudo<(outs), (ins bare_symbol:$dst), []> {
   let AsmString = "tail\t$dst";
 }
 
+let isCall = 1, isTerminator = 1, isReturn = 1, isBarrier = 1, Uses = [X2] in
+def PseudoTAILIndirect : Pseudo<(outs), (ins GPRTC:$dst), []>;
+
 def : Pat<(Tail (iPTR tglobaladdr:$dst)),
           (PseudoTAIL texternalsym:$dst)>;
 def : Pat<(Tail (iPTR texternalsym:$dst)),
           (PseudoTAIL texternalsym:$dst)>;
+def : Pat<(Tail GPRTC:$dst),
+          (PseudoTAILIndirect GPRTC:$dst)>;
 
 /// Loads
 
 multiclass LdPat<PatFrag LoadOp, RVInst Inst> {
   def : Pat<(LoadOp GPR:$rs1), (Inst GPR:$rs1, 0)>;
   def : Pat<(LoadOp AddrFI:$rs1), (Inst AddrFI:$rs1, 0)>;
   def : Pat<(LoadOp (add GPR:$rs1, simm12:$imm12)),
             (Inst GPR:$rs1, simm12:$imm12)>;

lib/Target/RISCV/RISCVRegisterInfo.td

def GPRC : RegisterClass<"RISCV", [XLenVT], 32, (add
     (sequence "X%u", 10, 15),
     (sequence "X%u", 8, 9)
   )> {
   let RegInfos = RegInfoByHwMode<
       [RV32,              RV64,              DefaultMode],
       [RegInfo<32,32,32>, RegInfo<64,64,64>, RegInfo<32,32,32>]>;
 }
 
+// For tail calls, we can't use callee-saved registers, as they are restored to
+// the saved value before the tail call, which would clobber a call address.
+def GPRTC : RegisterClass<"RISCV", [XLenVT], 32, (add
+    (sequence "X%u", 5, 7),
+    (sequence "X%u", 10, 17),
+    (sequence "X%u", 28, 31)
+  )> {
+  let RegInfos = RegInfoByHwMode<
+      [RV32,              RV64,              DefaultMode],
+      [RegInfo<32,32,32>, RegInfo<64,64,64>, RegInfo<32,32,32>]>;
+}
+
 def SP : RegisterClass<"RISCV", [XLenVT], 32, (add X2)> {
   let RegInfos = RegInfoByHwMode<
       [RV32,              RV64,              DefaultMode],
       [RegInfo<32,32,32>, RegInfo<64,64,64>, RegInfo<32,32,32>]>;
 }
 
 // Floating point registers
 let RegAltNameIndices = [ABIRegAltName] in {

test/CodeGen/RISCV/disable-tail-calls.ll

+; Check that command line option "-disable-tail-calls" overrides function
+; attribute "disable-tail-calls".
+
+; RUN: llc < %s -mtriple=riscv32-unknown-elf \
+; RUN: | FileCheck %s --check-prefixes=CALLER1,NOTAIL
+; RUN: llc < %s -mtriple=riscv32-unknown-elf -disable-tail-calls \
+; RUN: | FileCheck %s --check-prefixes=CALLER1,NOTAIL
+; RUN: llc < %s -mtriple=riscv32-unknown-elf -disable-tail-calls=false \
+; RUN: | FileCheck %s --check-prefixes=CALLER1,TAIL
+
+; RUN: llc < %s -mtriple=riscv32-unknown-elf \
+; RUN: | FileCheck %s --check-prefixes=CALLER2,TAIL
+; RUN: llc < %s -mtriple=riscv32-unknown-elf -disable-tail-calls \
+; RUN: | FileCheck %s --check-prefixes=CALLER2,NOTAIL
+; RUN: llc < %s -mtriple=riscv32-unknown-elf -disable-tail-calls=false \
+; RUN: | FileCheck %s --check-prefixes=CALLER2,TAIL
+
+; CALLER1-LABEL: {{\_?}}caller1
+; CALLER2-LABEL: {{\_?}}caller2
+; NOTAIL-NOT: tail callee
+; NOTAIL: call callee
+; TAIL: tail callee
+; TAIL-NOT: call callee
+
+; Function with attribute #0 = { "disable-tail-calls"="true" }
+define i32 @caller1(i32 %a) #0 {
+entry:
+  %call = tail call i32 @callee(i32 %a)
+  ret i32 %call
+}
+
+define i32 @caller2(i32 %a) {
+entry:
+  %call = tail call i32 @callee(i32 %a)
+  ret i32 %call
+}
+
+declare i32 @callee(i32)
+
+attributes #0 = { "disable-tail-calls"="true" }

test/CodeGen/RISCV/tail-calls.ll

+; RUN: llc -mtriple riscv32-unknown-linux-gnu -o - %s | FileCheck %s
+; RUN: llc -mtriple riscv32-unknown-elf       -o - %s | FileCheck %s
+
+; Perform tail call optimization.
+declare i32 @callee(i32 %i)
+define i32 @caller(i32 %i) {
+; CHECK-LABEL: caller
+; CHECK: tail callee
+entry:
+  %r = tail call i32 @callee(i32 %i)
+  ret i32 %r
+}
+
+; Do not tail call optimize functions with varargs.
+declare i32 @callee_varargs(i32, ...)
+define void @caller_varargs(i32 %a, i32 %b) {
+; CHECK-LABEL: caller_varargs
+; CHECK-NOT: tail callee_varargs
+; CHECK: call callee_varargs
+entry:
+  %call = tail call i32 (i32, ...) @callee_varargs(i32 %a, i32 %b, i32 %b, i32 %a)
+  ret void
+}
+
+; Do not tail call optimize if stack is used to pass parameters.
+declare i32 @callee_args(i32 %a, i32 %b, i32 %c, i32 %dd, i32 %e, i32 %ff, i32 %g, i32 %h, i32 %i, i32 %j, i32 %k, i32 %l, i32 %m, i32 %n)
+define i32 @caller_args(i32 %a, i32 %b, i32 %c, i32 %dd, i32 %e, i32 %ff, i32 %g, i32 %h, i32 %i, i32 %j, i32 %k, i32 %l, i32 %m, i32 %n) {
+; CHECK-LABEL: caller_args
+; CHECK-NOT: tail callee_args
+; CHECK: call callee_args
+entry:
+  %r = tail call i32 @callee_args(i32 %a, i32 %b, i32 %c, i32 %dd, i32 %e, i32 %ff, i32 %g, i32 %h, i32 %i, i32 %j, i32 %k, i32 %l, i32 %m, i32 %n)
+  ret i32 %r
+}
+
+; Do not tail call optimize if parameters need to be passed indirectly.
+declare i32 @callee_indirect(fp128 %a)
+define void @caller_indirect() {
+; CHECK-LABEL: caller_indirect
+; CHECK-NOT: tail callee_indirect
+; CHECK: call callee_indirect
+entry:
+  %call = tail call i32 @callee_indirect(fp128 0xL00000000000000003FFF000000000000)
+  ret void
+}
+
+; Externally-defined functions with weak linkage should not be tail-called.
+; The behaviour of branch instructions in this situation (as used for tail
+; calls) is implementation-defined, so we cannot rely on the linker replacing
+; the tail call with a return.
+declare extern_weak void @callee_weak()
+define void @caller_weak() {
+; CHECK-LABEL: caller_weak
+; CHECK-NOT: tail callee_weak
+; CHECK: call callee_weak
+entry:
+  tail call void @callee_weak()
+  ret void
+}
+
+; Exception-handling functions need a special set of instructions to indicate a
+; return to the hardware. Tail-calling another function would probably break
+; this.
+declare void @callee_irq()
+define void @caller_irq() #0 {
+; CHECK-LABEL: caller_irq
+; CHECK-NOT: tail callee_irq
+; CHECK: call callee_irq
+entry:
+  tail call void @callee_irq()
+  ret void
+}
+attributes #0 = { "interrupt" }
+
+; Byval parameters hand the function a pointer directly into the stack area
+; we want to reuse during a tail call. Do not tail call optimize functions with
+; byval parameters.
+declare i32 @callee_byval(i32** byval %a)
+define i32 @caller_byval() {
+; CHECK-LABEL: caller_byval
+; CHECK-NOT: tail callee_byval
+; CHECK: call callee_byval
+entry:
+  %a = alloca i32*
+  %r = tail call i32 @callee_byval(i32** byval %a)
+  ret i32 %r
+}