[RISCV] Add support for using -msave-restore with tailcalls

[edward-jones]

Summary

This introduces the use of a new compiler runtime library function which behaves the same as riscv_restore_N, however instead of returning to the callee these functions do a tailcall to the function whose pointer is stored in t1. This allows the -msave-restore optimization to be applied even in a tailcall context. The new runtime library functions are called riscv_restore_tailcall_X where X is the number of callee-saved registers which will be restored.

Original patch by Simon Cook.

[jrtc27]

Where is this specified? I only see the non-tailcall versions in GCC. I don't think we should merge anything until it's at least a draft specification in an official place.

[jrtc27]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

1133

We should not need to duplicate all this code surely? Certainly these need to honour the code model and not assume medlow.

[edward-jones]

I have updated this so that the generation of the address passed to the restore functions is done in a separate method RISCVFrameLowering::buildAddr. This handles both the code models and mirrors RISCVISelLowering::getAddr - it would be nice if these functions could share code, or at least be in a common place.

I've also opened a pull request to document the existing -msave-restore behaviour in riscv-toolchain-conventions, and I have a follow up patch to document the tail call versions of the entry points too.

I've expanded the test to cover the different code models, and pic mode too.

[edward-jones]

Documentation for -msave-restore has been committed to riscv-toolchain-conventions, I have an additional pull request to document this proposed tail-call version here

[edward-jones]

On the RISC-V sync-up call I said I would try and provide some code size numbers for these changes. First I mentioned that I saw a code size regression when compiled Embench, here's per-benchmark numbers:

-Oz -msave-restore (no tailcall version)

 Benchmark            size
---------            ----
aha-mont64          1,964
crc32               1,140
cubic              55,942
edn                 2,364
huffbench           2,456
matmult-int         1,420
minver             11,160
nbody              10,748
nettle-aes          3,852
nettle-sha256       8,552
nsichneu           14,268
picojpeg            9,476
qrduino             7,256
sglib-combined      3,528
slre                3,652
st                 11,056
statemate           4,908
ud                  1,620
wikisort           15,060
---------           -----
Geometric mean      5,347
Geometric SD            2.65
Geometric range  12,143.345300622888

-Oz -msave-restore (with tailcall version)

Benchmark            size
---------            ----
aha-mont64          1,968
crc32               1,340
cubic              55,942
edn                 2,372
huffbench           2,656
matmult-int         1,476
minver             11,164
nbody              10,752
nettle-aes          3,896
nettle-sha256       8,588
nsichneu           14,272
picojpeg            9,476
qrduino             7,428
sglib-combined      6,512
slre                3,656
st                 11,116
statemate           4,912
ud                  1,620
wikisort           15,060
---------           -----
Geometric mean      5,620
Geometric SD            2.59
Geometric range  12,365.236826034434

The Embench benchmarks are very small, so the constant cost of the additional __riscv_restore_tailcall entry points exceeds the benefit of the optimization.

To try something larger I also grabbed sqlite 3.35.5 and tried building that (again -Oz -msave-restore). Below are results for that experiment.

  text    data     bss     dec     hex filename
652816    5604     903  659323   a0f7b sqlite3-save-restore.o
645328    5604     903  651835   9f23b sqlite3-save-restore-tailcall.o

So for this larger codebase where the constant cost of the additional functions is minor, we see approximately a 1% reduction in code size (note that this only compiling to an object file, so it doesn't include the the cost of the save-restore compiler-rt functions)

[edward-jones]

I've rebased and added an option so that the tailcall version of save/restore is only used if -mllvm -riscv-save-restore-tailcall is provided.

[asb]

In D91719#2843627, @edward-jones wrote:

I've rebased and added an option so that the tailcall version of save/restore is only used if -mllvm -riscv-save-restore-tailcall is provided.

Thanks Ed, I think making it opt-in like this makes the decision to merge or not slightly easier, but ideally we can get at least some confirmation in https://github.com/riscv/riscv-toolchain-conventions/pull/10 as to whether anybody has major concerns about going in this direction.

[edward-jones]

I'm not entirely sure if there's anything that needs to be agreed on still for this. It feels like this has soft agreement, but I don't know whether I would get push back if I get this committed

[edward-jones]

Rebased

[edward-jones]

To summarize where I think this stands. We don't want to implement something in LLVM which is undocumented or not likely to be implemented for other tools, so I also opened a pull request to document this change (https://github.com/riscv-non-isa/riscv-toolchain-conventions/pull/10) in the hope that it could be later implemented for GCC. From the discussion on that pull request there's an understandable reluctance to introduce a feature which would stop libgcc/compiler-rt from being drop in replacements for each other.

So I think to unstick this a patch to libgcc needs to be submitted.

[edward-jones]

The documentation changes to the riscv-toolchain-conventions repository have been merged now, (https://github.com/riscv-non-isa/riscv-toolchain-conventions/pull/10). In addition, there's a pending patch to implement this in GCC (https://gcc.gnu.org/pipermail/gcc-patches/2022-March/592467.html). I think this means there's nothing blocking this except a rebase.

[asb]

In D91719#3471226, @edward-jones wrote:

The documentation changes to the riscv-toolchain-conventions repository have been merged now, (https://github.com/riscv-non-isa/riscv-toolchain-conventions/pull/10). In addition, there's a pending patch to implement this in GCC (https://gcc.gnu.org/pipermail/gcc-patches/2022-March/592467.html). I think this means there's nothing blocking this except a rebase.

I think that's right - thanks for your perseverance with this Ed.

[edward-jones]

Rebased!

llvm/lib/Target/RISCV/RISCVFrameLowering.h

public:
   bool hasFP(const MachineFunction &MF) const override;
 
   bool hasBP(const MachineFunction &MF) const;
 
   bool hasReservedCallFrame(const MachineFunction &MF) const override;
   MachineBasicBlock::iterator
   eliminateCallFramePseudoInstr(MachineFunction &MF, MachineBasicBlock &MBB,
                                 MachineBasicBlock::iterator MI) const override;
+
   bool spillCalleeSavedRegisters(MachineBasicBlock &MBB,
                                  MachineBasicBlock::iterator MI,
                                  ArrayRef<CalleeSavedInfo> CSI,
                                  const TargetRegisterInfo *TRI) const override;
+  void buildAddr(MachineBasicBlock &MBB, MachineBasicBlock::iterator InsertPos,
+                 const DebugLoc &DL, Register DestReg, const MachineOperand &Op,
+                 bool IsLocal, MachineInstr::MIFlag MIFlag) const;
   bool
   restoreCalleeSavedRegisters(MachineBasicBlock &MBB,
                               MachineBasicBlock::iterator MI,
                               MutableArrayRef<CalleeSavedInfo> CSI,
                               const TargetRegisterInfo *TRI) const override;
 
   // Get the first stack adjustment amount for SplitSPAdjust.
   // Return 0 if we don't want to to split the SP adjustment in prologue and

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

if (LibCallID == -1)
     return nullptr;
   return SpillLibCalls[LibCallID];
 }
 
 // Get the name of the libcall used for restoring callee saved registers.
 // If this function will not use save/restore libcalls, then return a nullptr.
 static const char *
 getRestoreLibCallName(const MachineFunction &MF,
-                      const std::vector<CalleeSavedInfo> &CSI) {
+                      const std::vector<CalleeSavedInfo> &CSI,
+                      bool TailCallSite) {
   static const char *const RestoreLibCalls[] = {
     "__riscv_restore_0",
     "__riscv_restore_1",
     "__riscv_restore_2",
     "__riscv_restore_3",
     "__riscv_restore_4",
     "__riscv_restore_5",
     "__riscv_restore_6",
     "__riscv_restore_7",
     "__riscv_restore_8",
     "__riscv_restore_9",
     "__riscv_restore_10",
     "__riscv_restore_11",
     "__riscv_restore_12"
   };
+  static const char *const RestoreLibCallsTC[] = {
+    "__riscv_restore_tailcall_0",
+    "__riscv_restore_tailcall_1",
+    "__riscv_restore_tailcall_2",
+    "__riscv_restore_tailcall_3",
+    "__riscv_restore_tailcall_4",
+    "__riscv_restore_tailcall_5",
+    "__riscv_restore_tailcall_6",
+    "__riscv_restore_tailcall_7",
+    "__riscv_restore_tailcall_8",
+    "__riscv_restore_tailcall_9",
+    "__riscv_restore_tailcall_10",
+    "__riscv_restore_tailcall_11",
+    "__riscv_restore_tailcall_12"
+  };
 
   int LibCallID = getLibCallID(MF, CSI);
   if (LibCallID == -1)
     return nullptr;
+  if (TailCallSite)
+    return RestoreLibCallsTC[LibCallID];
   return RestoreLibCalls[LibCallID];
 }
 
 bool RISCVFrameLowering::hasFP(const MachineFunction &MF) const {
   const TargetRegisterInfo *RegInfo = MF.getSubtarget().getRegisterInfo();
 
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   return MF.getTarget().Options.DisableFramePointerElim(MF) ||
for (auto &CS : NonLibcallCSI) {
     Register Reg = CS.getReg();
     const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(Reg);
     TII.storeRegToStackSlot(MBB, MI, Reg, true, CS.getFrameIdx(), RC, TRI);
   }
 
   return true;
 }
 
+void RISCVFrameLowering::buildAddr(MachineBasicBlock &MBB,
+                                   MachineBasicBlock::iterator InsertPos,
+                                   const DebugLoc &DL, Register DestReg,
+                                   const MachineOperand &Op, bool IsLocal,
+                                   MachineInstr::MIFlag MIFlag) const {
+  assert(DestReg.isPhysical());
+  assert((Op.getType() == MachineOperand::MO_GlobalAddress ||
+          Op.getType() == MachineOperand::MO_ExternalSymbol) &&
+         "Unhandled operand type");
+
+  MachineFunction &MF = *MBB.getParent();
+  const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
+
+  if (MF.getTarget().isPositionIndependent()) {
+    if (IsLocal)
+      BuildMI(MBB, InsertPos, DL, TII.get(RISCV::PseudoLLA), DestReg).add(Op);
+    else
+      BuildMI(MBB, InsertPos, DL, TII.get(RISCV::PseudoLA), DestReg).add(Op);
+    return;
+  }
+
+  switch (MF.getTarget().getCodeModel()) {
+  default:
+    report_fatal_error("Unsupported code model for lowering");
+  case CodeModel::Small: {
+    MachineInstr *NewMI =
+        BuildMI(MBB, InsertPos, DL, TII.get(RISCV::LUI), DestReg)
+            .add(Op)
+            .setMIFlag(MIFlag);
+    NewMI->getOperand(1).setTargetFlags(RISCVII::MO_HI);
+
+    NewMI = BuildMI(MBB, InsertPos, DL, TII.get(RISCV::ADDI), DestReg)
+                .addReg(DestReg)
+                .add(Op)
+                .setMIFlag(MIFlag);
+    NewMI->getOperand(2).setTargetFlags(RISCVII::MO_LO);
+    break;
+  }
+  case CodeModel::Medium: {
+    BuildMI(MBB, InsertPos, DL, TII.get(RISCV::PseudoLLA), DestReg)
+        .add(Op)
+        .setMIFlag(MIFlag);
+    break;
+  }
+  }
+}
+
 bool RISCVFrameLowering::restoreCalleeSavedRegisters(
     MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
     MutableArrayRef<CalleeSavedInfo> CSI, const TargetRegisterInfo *TRI) const {
   if (CSI.empty())
     return true;
 
   MachineFunction *MF = MBB.getParent();
   const TargetInstrInfo &TII = *MF->getSubtarget().getInstrInfo();
   DebugLoc DL;
   if (MI != MBB.end() && !MI->isDebugInstr())
     DL = MI->getDebugLoc();
 
   // Manually restore values not restored by libcall. Insert in reverse order.
   // loadRegFromStackSlot can insert multiple instructions.
   const auto &NonLibcallCSI = getNonLibcallCSI(*MF, CSI);
   for (auto &CS : reverse(NonLibcallCSI)) {
     Register Reg = CS.getReg();
     const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(Reg);
     TII.loadRegFromStackSlot(MBB, MI, Reg, CS.getFrameIdx(), RC, TRI);
     assert(MI != MBB.begin() && "loadRegFromStackSlot didn't insert any code!");
   }
 
-  const char *RestoreLibCall = getRestoreLibCallName(*MF, CSI);
+  bool IsTailCall =
+      MI != MBB.end() && (MI->getOpcode() == RISCV::PseudoTAIL ||
+                          MI->getOpcode() == RISCV::PseudoTAILIndirect);
+  const char *RestoreLibCall = getRestoreLibCallName(*MF, CSI, IsTailCall);
   if (RestoreLibCall) {
+    if (IsTailCall) {
+      assert (MF->getSubtarget<RISCVSubtarget>().enableSaveRestoreTailCall());
+
+      // Add restore libcall via tail call. X6/T1 is added as use as this holds
+      // the pointer to tailcall to.
+      MachineBasicBlock::iterator NewMI =
+          BuildMI(MBB, MI, DL, TII.get(RISCV::PseudoTAIL))
+              .addExternalSymbol(RestoreLibCall, RISCVII::MO_CALL)
+              .addReg(RISCV::X6 /*T1*/, RegState::Implicit)
+              .setMIFlag(MachineInstr::FrameDestroy);
+
+      if (MI->getOperand(0).getType() == MachineOperand::MO_Register) {
+        if (MI->getOperand(0).getReg() != RISCV::X6 /*T1*/) {
+          BuildMI(MBB, NewMI, DL, TII.get(RISCV::ADDI), RISCV::X6 /*T1*/)
+              .addReg(MI->getOperand(0).getReg())
+              .addImm(0)
+              .setMIFlag(MachineInstr::FrameDestroy);
+        }
+      } else {
+        const MachineOperand &Op = MI->getOperand(0);
+        assert(Op.isGlobal() || Op.isSymbol());
+        bool IsLocal = MF->getTarget().shouldAssumeDSOLocal(
+            *MF->getFunction().getParent(),
+            Op.isGlobal() ? Op.getGlobal() : nullptr);
+        buildAddr(MBB, NewMI, DL, RISCV::X6, MI->getOperand(0), IsLocal,
+                  MachineInstr::FrameDestroy);
+      }
+
+      NewMI->copyImplicitOps(*MF, *MI);
+      MI->eraseFromParent();
+    } else {
       // Add restore libcall via tail call.
       MachineBasicBlock::iterator NewMI =
           BuildMI(MBB, MI, DL, TII.get(RISCV::PseudoTAIL))
               .addExternalSymbol(RestoreLibCall, RISCVII::MO_CALL)
               .setMIFlag(MachineInstr::FrameDestroy);
 
       // Remove trailing returns, since the terminator is now a tail call to the
       // restore function.
       if (MI != MBB.end() && MI->getOpcode() == RISCV::PseudoRET) {
         NewMI->copyImplicitOps(*MF, *MI);
         MI->eraseFromParent();
       }
     }
+  }
 
   return true;
 }
 
 bool RISCVFrameLowering::canUseAsPrologue(const MachineBasicBlock &MBB) const {
   MachineBasicBlock *TmpMBB = const_cast<MachineBasicBlock *>(&MBB);
   const MachineFunction *MF = MBB.getParent();
   const auto *RVFI = MF->getInfo<RISCVMachineFunctionInfo>();

llvm/lib/Target/RISCV/RISCVMachineFunctionInfo.h

public:
   }
 
   unsigned getLibCallStackSize() const { return LibCallStackSize; }
   void setLibCallStackSize(unsigned Size) { LibCallStackSize = Size; }
 
   bool useSaveRestoreLibCalls(const MachineFunction &MF) const {
     // We cannot use fixed locations for the callee saved spill slots if the
     // function uses a varargs save area, or is an interrupt handler.
-    return MF.getSubtarget<RISCVSubtarget>().enableSaveRestore() &&
-           VarArgsSaveSize == 0 && !MF.getFrameInfo().hasTailCall() &&
+    const auto &Subtarget = MF.getSubtarget<RISCVSubtarget>();
+    bool IsTailCall = MF.getFrameInfo().hasTailCall();
+    return Subtarget.enableSaveRestore() &&
+           (!IsTailCall || Subtarget.enableSaveRestoreTailCall()) &&
+           VarArgsSaveSize == 0 &&
            !MF.getFunction().hasFnAttribute("interrupt");
   }
 
   uint64_t getRVVStackSize() const { return RVVStackSize; }
   void setRVVStackSize(uint64_t Size) { RVVStackSize = Size; }
 
   uint64_t getRVVPadding() const { return RVVPadding; }
   void setRVVPadding(uint64_t Padding) { RVVPadding = Padding; }
 
   unsigned getCalleeSavedStackSize() const { return CalleeSavedStackSize; }
   void setCalleeSavedStackSize(unsigned Size) { CalleeSavedStackSize = Size; }
 };
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_RISCV_RISCVMACHINEFUNCTIONINFO_H

llvm/lib/Target/RISCV/RISCVSubtarget.h

protected:
   std::unique_ptr<RegisterBankInfo> RegBankInfo;
 
 public:
   const CallLowering *getCallLowering() const override;
   InstructionSelector *getInstructionSelector() const override;
   const LegalizerInfo *getLegalizerInfo() const override;
   const RegisterBankInfo *getRegBankInfo() const override;
 
+  // Returns true if the special tailcall versions of the save-restore
+  // entry points can also be used.
+  bool enableSaveRestoreTailCall() const;
+
   // Return the known range for the bit length of RVV data registers. A value
   // of 0 means nothing is known about that particular limit beyond what's
   // implied by the architecture.
   unsigned getMaxRVVVectorSizeInBits() const;
   unsigned getMinRVVVectorSizeInBits() const;
   unsigned getMaxLMULForFixedLengthVectors() const;
   bool useRVVForFixedLengthVectors() const;
 };
 } // End llvm namespace
 
 #endif

llvm/lib/Target/RISCV/RISCVSubtarget.cpp

static cl::opt<unsigned> RVVVectorBitsMin(
     cl::init(0), cl::Hidden);
 
 static cl::opt<unsigned> RVVVectorLMULMax(
     "riscv-v-fixed-length-vector-lmul-max",
     cl::desc("The maximum LMUL value to use for fixed length vectors. "
              "Fractional LMUL values are not supported."),
     cl::init(8), cl::Hidden);
 
+static cl::opt<bool> SaveRestoreTailCall(
+    "riscv-save-restore-tailcall",
+    cl::desc("Enable save-restore optimization in a tailcall context."),
+    cl::init(false), cl::Hidden);
+
 void RISCVSubtarget::anchor() {}
 
 RISCVSubtarget &
 RISCVSubtarget::initializeSubtargetDependencies(const Triple &TT, StringRef CPU,
                                                 StringRef TuneCPU, StringRef FS,
                                                 StringRef ABIName) {
   // Determine default and user-specified characteristics
   bool Is64Bit = TT.isArch64Bit();

 const LegalizerInfo *RISCVSubtarget::getLegalizerInfo() const {
   return Legalizer.get();
 }
 
 const RegisterBankInfo *RISCVSubtarget::getRegBankInfo() const {
   return RegBankInfo.get();
 }
 
+bool RISCVSubtarget::enableSaveRestoreTailCall() const {
+  return SaveRestoreTailCall;
+}
+
 unsigned RISCVSubtarget::getMaxRVVVectorSizeInBits() const {
   assert(hasStdExtV() && "Tried to get vector length without V support!");
   if (RVVVectorBitsMax == 0)
     return 0;
   assert(RVVVectorBitsMax >= 128 && isPowerOf2_32(RVVVectorBitsMax) &&
          "V extension requires vector length to be at least 128 and a power of "
          "2!");
   assert(RVVVectorBitsMax >= RVVVectorBitsMin &&

llvm/test/CodeGen/RISCV/saverestore.ll

 ; RV64I-FP-SR-NEXT:    fsd fs0, 8(sp)
 ; RV64I-FP-SR:         fld fs0, 8(sp)
 ; RV64I-FP-SR-NEXT:    addi sp, sp, 16
 ; RV64I-FP-SR-NEXT:    tail __riscv_restore_7
   call void asm sideeffect "", "~{f8},~{x9},~{x18},~{x19},~{x20},~{x21},~{x22}"()
   ret void
 }
 
-; Check that preserving tail calls is preferred over save/restore
-
-declare i32 @tail_callee(i32 %i)
-
-define i32 @tail_call(i32 %i) nounwind {
-; RV32I-LABEL: tail_call:
-; RV32I-NOT:     call t0, __riscv_save
-; RV32I:         tail tail_callee
-; RV32I-NOT:     tail __riscv_restore
-;
-; RV64I-LABEL: tail_call:
-; RV64I-NOT:     call t0, __riscv_save
-; RV64I:         tail tail_callee
-; RV64I-NOT:     tail __riscv_restore
-;
-; RV32I-SR-LABEL: tail_call:
-; RV32I-SR-NOT:     call t0, __riscv_save
-; RV32I-SR:         tail tail_callee
-; RV32I-SR-NOT:     tail __riscv_restore
-;
-; RV64I-SR-LABEL: tail_call:
-; RV64I-SR-NOT:     call t0, __riscv_save
-; RV64I-SR:         tail tail_callee
-; RV64I-SR-NOT:     tail __riscv_restore
-;
-; RV32I-FP-SR-LABEL: tail_call:
-; RV32I-FP-SR-NOT:     call t0, __riscv_save
-; RV32I-FP-SR:         tail tail_callee
-; RV32I-FP-SR-NOT:     tail __riscv_restore
-;
-; RV64I-FP-SR-LABEL: tail_call:
-; RV64I-FP-SR-NOT:     call t0, __riscv_save
-; RV64I-FP-SR:         tail tail_callee
-; RV64I-FP-SR-NOT:     tail __riscv_restore
-entry:
-  %val = load [18 x i32], [18 x i32]* @var0
-  store volatile [18 x i32] %val, [18 x i32]* @var0
-  %r = tail call i32 @tail_callee(i32 %i)
-  ret i32 %r
-}
-
 ; Check that functions with varargs do not use save/restore code
 
 declare void @llvm.va_start(i8*)
 declare void @llvm.va_end(i8*)
 
 define i32 @varargs(i8* %fmt, ...) nounwind {
 ; RV32I-LABEL: varargs:
 ; RV32I-NOT:     call t0, __riscv_save

llvm/test/CodeGen/RISCV/saverestore_tailcall.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 < %s | FileCheck %s -check-prefix=RV32I
+; RUN: llc -mtriple=riscv64 < %s | FileCheck %s -check-prefix=RV64I
+; RUN: llc -mtriple=riscv32 --code-model=small -riscv-save-restore-tailcall -mattr=+save-restore < %s | FileCheck %s -check-prefix=RV32I-SR-SMALL
+; RUN: llc -mtriple=riscv64 --code-model=small -riscv-save-restore-tailcall -mattr=+save-restore < %s | FileCheck %s -check-prefix=RV64I-SR-SMALL
+; RUN: llc -mtriple=riscv32 --code-model=small -riscv-save-restore-tailcall -mattr=+save-restore < %s | FileCheck %s -check-prefix=RV32I-SR-SMALL
+; RUN: llc -mtriple=riscv64 --code-model=small -riscv-save-restore-tailcall -mattr=+save-restore < %s | FileCheck %s -check-prefix=RV64I-SR-SMALL
+; RUN: llc -mtriple=riscv32 --code-model=small -riscv-save-restore-tailcall -mattr=+f,+save-restore -target-abi=ilp32f < %s | FileCheck %s -check-prefix=RV32I-FP-SR-SMALL
+; RUN: llc -mtriple=riscv64 --code-model=small -riscv-save-restore-tailcall -mattr=+f,+d,+save-restore -target-abi=lp64d < %s | FileCheck %s -check-prefix=RV64I-FP-SR-SMALL
+; RUN: llc -mtriple=riscv32 --code-model=medium -riscv-save-restore-tailcall -mattr=+save-restore < %s | FileCheck %s -check-prefix=RV32I-SR-MEDIUM
+; RUN: llc -mtriple=riscv64 --code-model=medium -riscv-save-restore-tailcall -mattr=+save-restore < %s | FileCheck %s -check-prefix=RV64I-SR-MEDIUM
+; RUN: llc -mtriple=riscv32 --relocation-model=pic -riscv-save-restore-tailcall -mattr=+save-restore < %s | FileCheck %s -check-prefix=RV32I-SR-PIC
+; RUN: llc -mtriple=riscv64 --relocation-model=pic -riscv-save-restore-tailcall -mattr=+save-restore < %s | FileCheck %s -check-prefix=RV64I-SR-PIC
+; RUN: llc -mtriple=riscv32 --code-model=small -mattr=+save-restore < %s | FileCheck %s -check-prefix=RV32I-SR-SMALL-NO-TAILCALL
+; RUN: llc -mtriple=riscv64 --code-model=small -mattr=+save-restore < %s | FileCheck %s -check-prefix=RV64I-SR-SMALL-NO-TAILCALL
+
+; Check that the correct save/restore libcalls are generated for tail calls
+
+@var0 = global [18 x i32] zeroinitializer
+@var1 = global [24 x i32] zeroinitializer
+@var2 = global [30 x i32] zeroinitializer
+
+declare i32 @tail_callee(i32 %i)
+
+define i32 @tail_call(i32 %i) nounwind {
+; RV32I-LABEL: tail_call:
+; RV32I-NOT:     call t0, __riscv_save
+; RV32I:         tail tail_callee
+; RV32I-NOT:     tail __riscv_restore
+;
+; RV64I-LABEL: tail_call:
+; RV64I-NOT:     call t0, __riscv_save
+; RV64I:         tail tail_callee
+; RV64I-NOT:     tail __riscv_restore
+;
+; RV32I-SR-SMALL-LABEL: tail_call:
+; RV32I-SR-SMALL:    call t0, __riscv_save_6
+; RV32I-SR-SMALL:    lui t1, %hi(tail_callee)
+; RV32I-SR-SMALL:    addi t1, t1, %lo(tail_callee)
+; RV32I-SR-SMALL:    tail __riscv_restore_tailcall_6
+;
+; RV64I-SR-SMALL-LABEL: tail_call:
+; RV64I-SR-SMALL:    call t0, __riscv_save_6
+; RV64I-SR-SMALL:    lui t1, %hi(tail_callee)
+; RV64I-SR-SMALL:    addi t1, t1, %lo(tail_callee)
+; RV64I-SR-SMALL:    tail __riscv_restore_tailcall_6
+;
+; RV32I-FP-SR-SMALL-LABEL: tail_call:
+; RV32I-FP-SR-SMALL:    call t0, __riscv_save_6
+; RV32I-FP-SR-SMALL:    lui t1, %hi(tail_callee)
+; RV32I-FP-SR-SMALL:    addi t1, t1, %lo(tail_callee)
+; RV32I-FP-SR-SMALL:    tail __riscv_restore_tailcall_6
+;
+; RV64I-FP-SR-SMALL-LABEL: tail_call:
+; RV64I-FP-SR-SMALL:    call t0, __riscv_save_6
+; RV64I-FP-SR-SMALL:    lui t1, %hi(tail_callee)
+; RV64I-FP-SR-SMALL:    addi t1, t1, %lo(tail_callee)
+; RV64I-FP-SR-SMALL:    tail __riscv_restore_tailcall_6
+;
+; RV32I-SR-MEDIUM-LABEL: tail_call:
+; RV32I-SR-MEDIUM:    call t0, __riscv_save_5
+; RV32I-SR-MEDIUM:  .LBB0_2: # %entry
+; RV32I-SR-MEDIUM:    # Label of block must be emitted
+; RV32I-SR-MEDIUM:    auipc t1, %pcrel_hi(tail_callee)
+; RV32I-SR-MEDIUM:    addi t1, t1, %pcrel_lo(.LBB0_2)
+; RV32I-SR-MEDIUM:    tail __riscv_restore_tailcall_5
+;
+; RV64I-SR-MEDIUM-LABEL: tail_call:
+; RV64I-SR-MEDIUM:    call t0, __riscv_save_5
+; RV64I-SR-MEDIUM:  .LBB0_2: # %entry
+; RV64I-SR-MEDIUM:    # Label of block must be emitted
+; RV64I-SR-MEDIUM:    auipc t1, %pcrel_hi(tail_callee)
+; RV64I-SR-MEDIUM:    addi t1, t1, %pcrel_lo(.LBB0_2)
+; RV64I-SR-MEDIUM:    tail __riscv_restore_tailcall_5
+;
+; RV32I-SR-PIC-LABEL: tail_call:
+; RV32I-SR-PIC:    call t0, __riscv_save_5
+; RV32I-SR-PIC:  .LBB0_2: # %entry
+; RV32I-SR-PIC:    # Label of block must be emitted
+; RV32I-SR-PIC:    auipc t1, %got_pcrel_hi(tail_callee)
+; RV32I-SR-PIC:    lw t1, %pcrel_lo(.LBB0_2)(t1)
+; RV32I-SR-PIC:    tail __riscv_restore_tailcall_5
+;
+; RV64I-SR-PIC-LABEL: tail_call:
+; RV64I-SR-PIC:    call t0, __riscv_save_5
+; RV64I-SR-PIC:  .LBB0_2: # %entry
+; RV64I-SR-PIC:    # Label of block must be emitted
+; RV64I-SR-PIC:    auipc t1, %got_pcrel_hi(tail_callee)
+; RV64I-SR-PIC:    ld t1, %pcrel_lo(.LBB0_2)(t1)
+; RV64I-SR-PIC:    tail __riscv_restore_tailcall_5
+;
+; RV32I-SR-SMALL-NO-TAILCALL-LABEL: tail_call:
+; RV32I-SR-SMALL-NO-TAILCALL-NOT:     call t0, __riscv_save
+; RV32I-SR-SMALL-NO-TAILCALL:         tail tail_callee
+; RV32I-SR-SMALL-NO-TAILCALL-NOT:     tail __riscv_restore
+;
+; RV64I-SR-SMALL-NO-TAILCALL-LABEL: tail_call:
+; RV64I-SR-SMALL-NO-TAILCALL-NOT:     call t0, __riscv_save
+; RV64I-SR-SMALL-NO-TAILCALL:         tail tail_callee
+; RV64I-SR-SMALL-NO-TAILCALL-NOT:     tail __riscv_restore
+entry:
+  %val = load [18 x i32], [18 x i32]* @var0
+  store volatile [18 x i32] %val, [18 x i32]* @var0
+  %r = tail call i32 @tail_callee(i32 %i)
+  ret i32 %r
+}
+
+declare dso_local i32 @dso_local_tail_callee(i32 %i)
+
+define i32 @tail_call_to_dso_local(i32 %i) nounwind {
+; RV32I-LABEL: tail_call_to_dso_local:
+; RV32I-NOT:     call t0, __riscv_save
+; RV32I:         tail dso_local_tail_callee
+; RV32I-NOT:     tail __riscv_restore
+;
+; RV64I-LABEL: tail_call_to_dso_local:
+; RV64I-NOT:     call t0, __riscv_save
+; RV64I:         tail dso_local_tail_callee
+; RV64I-NOT:     tail __riscv_restore
+;
+; RV32I-SR-SMALL-LABEL: tail_call_to_dso_local:
+; RV32I-SR-SMALL:    call t0, __riscv_save_6
+; RV32I-SR-SMALL:    lui t1, %hi(dso_local_tail_callee)
+; RV32I-SR-SMALL:    addi t1, t1, %lo(dso_local_tail_callee)
+; RV32I-SR-SMALL:    tail __riscv_restore_tailcall_6
+;
+; RV64I-SR-SMALL-LABEL: tail_call_to_dso_local:
+; RV64I-SR-SMALL:    call t0, __riscv_save_6
+; RV64I-SR-SMALL:    lui t1, %hi(dso_local_tail_callee)
+; RV64I-SR-SMALL:    addi t1, t1, %lo(dso_local_tail_callee)
+; RV64I-SR-SMALL:    tail __riscv_restore_tailcall_6
+;
+; RV32I-FP-SR-SMALL-LABEL: tail_call_to_dso_local:
+; RV32I-FP-SR-SMALL:    call t0, __riscv_save_6
+; RV32I-FP-SR-SMALL:    lui t1, %hi(dso_local_tail_callee)
+; RV32I-FP-SR-SMALL:    addi t1, t1, %lo(dso_local_tail_callee)
+; RV32I-FP-SR-SMALL:    tail __riscv_restore_tailcall_6
+;
+; RV64I-FP-SR-SMALL-LABEL: tail_call_to_dso_local:
+; RV64I-FP-SR-SMALL:    call t0, __riscv_save_6
+; RV64I-FP-SR-SMALL:    lui t1, %hi(dso_local_tail_callee)
+; RV64I-FP-SR-SMALL:    addi t1, t1, %lo(dso_local_tail_callee)
+; RV64I-FP-SR-SMALL:    tail __riscv_restore_tailcall_6
+;
+; RV32I-SR-MEDIUM-LABEL: tail_call_to_dso_local:
+; RV32I-SR-MEDIUM:    call t0, __riscv_save_5
+; RV32I-SR-MEDIUM:  .LBB1_2: # %entry
+; RV32I-SR-MEDIUM:    # Label of block must be emitted
+; RV32I-SR-MEDIUM:    auipc t1, %pcrel_hi(dso_local_tail_callee)
+; RV32I-SR-MEDIUM:    addi t1, t1, %pcrel_lo(.LBB1_2)
+; RV32I-SR-MEDIUM:    tail __riscv_restore_tailcall_5
+;
+; RV64I-SR-MEDIUM-LABEL: tail_call_to_dso_local:
+; RV64I-SR-MEDIUM:    call t0, __riscv_save_5
+; RV64I-SR-MEDIUM:  .LBB1_2: # %entry
+; RV64I-SR-MEDIUM:    # Label of block must be emitted
+; RV64I-SR-MEDIUM:    auipc t1, %pcrel_hi(dso_local_tail_callee)
+; RV64I-SR-MEDIUM:    addi t1, t1, %pcrel_lo(.LBB1_2)
+; RV64I-SR-MEDIUM:    tail __riscv_restore_tailcall_5
+;
+; RV32I-SR-PIC-LABEL: tail_call_to_dso_local:
+; RV32I-SR-PIC:    call t0, __riscv_save_5
+; RV32I-SR-PIC:  .LBB1_2: # %entry
+; RV32I-SR-PIC:    # Label of block must be emitted
+; RV32I-SR-PIC:    auipc t1, %pcrel_hi(dso_local_tail_callee)
+; RV32I-SR-PIC:    addi t1, t1, %pcrel_lo(.LBB1_2)
+; RV32I-SR-PIC:    tail __riscv_restore_tailcall_5
+;
+; RV64I-SR-PIC-LABEL: tail_call_to_dso_local:
+; RV64I-SR-PIC:    call t0, __riscv_save_5
+; RV64I-SR-PIC:  .LBB1_2: # %entry
+; RV64I-SR-PIC:    # Label of block must be emitted
+; RV64I-SR-PIC:    auipc t1, %pcrel_hi(dso_local_tail_callee)
+; RV64I-SR-PIC:    addi t1, t1, %pcrel_lo(.LBB1_2)
+; RV64I-SR-PIC:    tail __riscv_restore_tailcall_5
+;
+; RV32I-SR-SMALL-NO-TAILCALL-LABEL: tail_call_to_dso_local:
+; RV32I-SR-SMALL-NO-TAILCALL-NOT:     call t0, __riscv_save
+; RV32I-SR-SMALL-NO-TAILCALL:         tail dso_local_tail_callee
+; RV32I-SR-SMALL-NO-TAILCALL-NOT:     tail __riscv_restore
+;
+; RV64I-SR-SMALL-NO-TAILCALL-LABEL: tail_call_to_dso_local:
+; RV64I-SR-SMALL-NO-TAILCALL-NOT:     call t0, __riscv_save
+; RV64I-SR-SMALL-NO-TAILCALL:         tail dso_local_tail_callee
+; RV64I-SR-SMALL-NO-TAILCALL-NOT:     tail __riscv_restore
+entry:
+  %val = load [18 x i32], [18 x i32]* @var0
+  store volatile [18 x i32] %val, [18 x i32]* @var0
+  %r = tail call i32 @dso_local_tail_callee(i32 %i)
+  ret i32 %r
+}