[RISCV] Add support for save/restore of callee-saved registers via libcalls

[lewis-revill]

Summary

This patch adds the support required for using the __riscv_save and __riscv_restore libcalls to implement a size-optimization for prologue and epilogue code, whereby the spill and restore code of callee-saved registers is implemented by common functions to reduce code duplication.

The initial prototype for this patch was implemented by @simoncook

[lewis-revill]

I might need to go through the variable/function names to replace 'CSR' with CalleeSavedRegister or something that doesn't cause confusion with RISC-V 'CSR' meaning 'Control & Status Register'...

lib/Target/RISCV/RISCVMachineFunctionInfo.h

53

I have put these two functions here just to try to deduplicate this common operation of distinguishing between fixed and non-fixed callee saved registers. I'd appreciate advice on a neater approach.

lib/Target/RISCV/RISCVRegisterInfo.cpp

84

This should probably appear somewhere else, there ideally needs to be a Clang flag for this.

[lewis-revill]

lib/Target/RISCV/RISCVMachineFunctionInfo.h

53

I've realized that this is likely completely unneccessary anyway. I should think that since either all callee-saved registers are listed as having reserved spill slots or none are, then these two vectors are mutually exclusive. I'll test to see whether simply storing a boolean and interpreting the entire CSI vector differently gives the same behaviour.

[lewis-revill]

• Replaced split fixed/nonfixed vectors by simply interpreting the CSI vector differently depending if save/restore calls will be used or not.

• Add UseSaveRestoreLibCalls to RISCVMachineFunctionInfo, along with a modifying getUseSaveRestoreLibCalls accessor and a non-modifying useSaveRestoreLibCalls accessor.

• Add 'RISCVMachineFunctionInfo.cpp' to avoid having a cl::opt declared in a header file.

• Simplify spillCalleeSavedRegisters and restoreCalleeSavedRegisters.

[lewis-revill]

Added a basic check to avoid using save/restore libcalls if we know it will make the code size worse.

[lewis-revill]

Revert previous change as the information needed to check the size cost was not available.

[edward-jones]

Since CFI emission has been added this probably needs to be rebased. I don't think CFI should materially affect the implementation though.

Overall the patch looks good. As well as a test for varargs I would also suggest adding a non-varargs test for a function with arguments passed via the stack.

I noticed that GCC only emits riscv_save_* and riscv_restore_* when there are >4 stack callee saved registers to be saved. So it might be worth making the threshold for when to apply this optimization dependent on whether you are compiling for size or speed (though that may be more appropriate in a future patch)

[lewis-revill]

Rebased to incorporate the recent CFI additions.

Thanks for the suggestions Ed, I added the test for arguments passed on the stack (many_args), and it exposed a bug whereby I disabled save/restore incorrectly for that function. I've changed the enable/disable logic to just simply check for the presence of var args, and now this case works.

[edward-jones]

LGTM now

@asb , what do you think?

[lewis-revill]

lib/Target/RISCV/RISCVRegisterInfo.cpp

105

I have a feeling it may be beneficial here to instead cast away const to get a modifiable RVFI, so that we can call getUseSaveRestoreLibCalls instead and never end up with the situation where we call useSaveRestoreLibCalls here but get a different result when we call getUseSaveRestoreLibCalls at a later stage.

This way we'd know that every time we check for whether lib calls can be used we get the same result, since we're always calling the lazy getter.

[lewis-revill]

Rebase, and only use the getUseSaveRestoreLibCalls getter method rather than the useSaveRestoreLibCalls method which does not save its result.

[lenary]

lib/Target/RISCV/RISCVFrameLowering.cpp

401

Won't we hit this case if (after the hard float ABI lands) someone uses any of fs0-fs11?

Would it not be better to return nullptr, so the code can bail out, rather than asserting?

The same applies in getRestoreLibCallName

[MaskRay]

lib/Target/RISCV/RISCVMachineFunctionInfo.cpp

27

return EnableSaveRestore && VarArgsSaveSize == 0; ?

[MaskRay]

lib/Target/RISCV/RISCVMachineFunctionInfo.cpp

27

Oh, why can't this stub just be inlined into getUseSaveRestoreLibCalls? It can be trivially computed every time its value is used. You can expand it later if this function ever becomes more complex.

[MaskRay]

lib/Target/RISCV/RISCVFrameLowering.cpp

394

Delete using std::max;

399

switch (*std::max_element(CSI.begin(), CSI.end())) {

428

Delete this

433

switch (*std::max_element(CSI.begin(), CSI.end())) {

[MaskRay]

lib/Target/RISCV/RISCVFrameLowering.cpp

399

Sorry, please ignore the max_element comment.

[lewis-revill]

lib/Target/RISCV/RISCVFrameLowering.cpp

401

I think the assertion is valid behaviour, since something has definitely gone wrong if any register not present in the list of callee-saved registers below is used as a callee-saved register. We would currently hit this if float ABI callee-saved registers are used, which would indicate that more logic needs to be added to explicitly handle that case.

lib/Target/RISCV/RISCVMachineFunctionInfo.cpp

27

So this getUseSaveRestoreLibCalls call gets called at multiple points in the pipeline. When I first implemented this function I wanted to be absolutely sure that at any stage in the pipeline the function returns the same result. Now that I think about it that would indicate incorrect behaviour anyway, so I'll make this change and see if I can find anything that breaks.

[asb]

Hi Lewis, did you evaluate implementing the machine outliner as a more general approach for getting code size savings in this sort of way?

You should have tests for at least a subset of target-abi {ilp32,lp64}{f,d} and the compiler shouldn't assert in those cases.

[lewis-revill]

In D62686#1571192, @asb wrote:

Hi Lewis, did you evaluate implementing the machine outliner as a more general approach for getting code size savings in this sort of way?

I haven't, is there an open patch for this?

You should have tests for at least a subset of target-abi {ilp32,lp64}{f,d} and the compiler shouldn't assert in those cases.

Will do, I can rebase this patch on top of your hard float patch.

[lewis-revill]

In D62686#1571360, @lewis-revill wrote:

Will do, I can rebase this patch on top of your hard float patch.

Ah, I wasn't aware it had landed!

[lewis-revill]

lib/Target/RISCV/RISCVFrameLowering.cpp

401

GCC's behaviour is to add manual floating point save/restore code in addition to any libcall that has been inserted for non-fp registers. I realize that this would break a lot of the logic throughout this patch where I assume that either all callee-save registers would be save/restored by libcall or none of them would, so perhaps the approach for now should be to bail out of using libcalls if any of the fp registers require save/restoring.

[lewis-revill]

Add support for a split save/restore approach when callee saved registers from the floating point ABIs are used.

[lewis-revill]

In D62686#1571192, @asb wrote:

Hi Lewis, did you evaluate implementing the machine outliner as a more general approach for getting code size savings in this sort of way?

You should have tests for at least a subset of target-abi {ilp32,lp64}{f,d} and the compiler shouldn't assert in those cases.

Would the existence of save/restore in the codebase conflict with the machine outliner approach? Otherwise it should be a case of simply disallowing enabling both flags at once if this leads to invalid behaviour.

[lewis-revill]

In D62686#1571192, @asb wrote:

Hi Lewis, did you evaluate implementing the machine outliner as a more general approach for getting code size savings in this sort of way?

I've added D66210 as a very basic implementation of the machine outliner, the results are currently slightly better than this save/restore approach, but I'm not certain the implementation isn't too relaxed about what can be outlined. It's worth noting that enabling both the optimizations at once doesn't diminish the gains you get from the machine outliner.

[lewis-revill]

Rebased.

[lewis-revill]

Fix git error whereby logic from this patch was mistakenly included in D62190

[lewis-revill]

There is an option to Clang for '-msave-restore' which should be utilized to enable this.

[lenary]

In D62686#1671123, @lewis-revill wrote:

There is an option to Clang for '-msave-restore' which should be utilized to enable this.

I don't think we'd have a problem if clang support for this came in a separate commit.

[lewis-revill]

In D62686#1672346, @lenary wrote:

In D62686#1671123, @lewis-revill wrote:

There is an option to Clang for '-msave-restore' which should be utilized to enable this.

I don't think we'd have a problem if clang support for this came in a separate commit.

There needs to be an LLVM component too, to add a target feature such that we can do Features.push_back("+save-restore") from Clang.

[lewis-revill]

Replace internal -mllvm option with target feature enabled through the clang frontend using -msave-restore.

[lenary]

We discussed this in the RISC-V meeting on 19 Sept 2019. @apazos says there are some SPEC failures in both 2006 and 2017, which would be good to triage.

[lenary]

Two nits, that I wanted to submit before the meeting, but didn't get around to.

llvm/lib/Target/RISCV/RISCV.td

72 ↗

(On Diff #220677)

Given the clang option defaults to false, I think it should here too, to avoid confusion in other frontends.

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

278 ↗

(On Diff #220677)

Nit: You shouldn't need this if statement - the for loop just won't execute if CSI is empty, surely.

[lewis-revill]

llvm/lib/Target/RISCV/RISCV.td

72 ↗

(On Diff #220677)

This is simply a case of a confusing parameter of the SubtargetFeature class.

SubtargetFeature interprets this "true" string as 'Value the attribute to be set to by feature'. IE: when the feature is enabled, what value should the corresponding RISCVSubtarget variable be set to? Rather than a default value for that variable.

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

278 ↗

(On Diff #220677)

Good catch, thanks!

[lenary]

llvm/lib/Target/RISCV/RISCV.td

72 ↗

(On Diff #220677)

Ah, apologies for the confusion.

[lewis-revill]

In D62686#1675347, @lenary wrote:

We discussed this in the RISC-V meeting on 19 Sept 2019. @apazos says there are some SPEC failures in both 2006 and 2017, which would be good to triage.

I've uncovered some execution failures in our annotated GCC testsuite as well, so I will look into those as a starting point.

[lewis-revill]

It seems like the regressions I'm seeing are due to the fact that calculating offsets for fixed objects at the top of the frame didn't account for extra stack size adjustment from the libcalls. I'm trying to find a neat way around this without breaking other calculations.

[lewis-revill]

Rewrote logic to calculate stack sizes, frame indexes and frame pointer offsets. This was necessary to take into account the fact that the save/restore lib calls are essentially an opaque section of the stack that is inserted at the beginning of the frame, after positive offset objects such as fixed stack arguments, but before additional callee saved registers (IE FP registers) and negative offset (dynamic) objects. So calculations of the actual offsets of these objects must be adjusted accordingly for the stack to function correctly.

[apazos]

Thanks for the patch update. I will launch some new correctness runs.

[apazos]

Lewis, with this patch we see less failures. But still some tests in SPEC and perennial test suites are failing.

Pengxuan and I are trying to triage the failures.

Here is what we see in one of the failed tests:

Code right before rologue/Epilogue Insertion & Frame Finalization:

%call39 = tail call i32 @somefunction()
ret i32 %call39

Code after Prologue/Epilogue Insertion & Frame Finalization:

PseudoCALL target-flags(<unknown>) @somefunction, implicit-def $x1, implicit $x2
$x10 = frame-destroy LUI 1
$x10 = frame-destroy ADDI killed $x10, -1712
$x2 = frame-destroy ADD $x2, killed $x10
CFI_INSTRUCTION def_cfa_offset 0
frame-destroy PseudoTAIL target-flags(<unknown>) &__riscv_restore_12, implicit $x2

x10 is the return value and it is being overwritten.

Is this patch self contained? Or does it depend on another patch under review?

For the tests that are passing, I see better code size reduction with C extension enabled (testing RV32IMAC) than with the machine outliner.

[apazos]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

631 ↗

(On Diff #222612)

Where are we making sure the PseudoCALL result in a0 is alive through the riscv_restore call?

[apazos]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

631 ↗

(On Diff #222612)

I did not find any other target that transforms a tail call back into a regular call.

The issue with doing this is that we don't have info about the return value of the original call.

If anyone knows how to do it, please give me some pointers.

I tried to fix this problem by adding implicit operands to the riscv_restore tail call that are all the possible return value registers:
BuildMI(MBB, MI, DL, TII.get(RISCV::PseudoTAIL))
.addExternalSymbol(RestoreLibCall, RISCVII::MO_CALL)
-.setMIFlag(MachineInstr::FrameDestroy);
+ .setMIFlag(MachineInstr::FrameDestroy)
+ .addReg(RISCV::X10, RegState::Implicit)
+ .addReg(RISCV::X11, RegState::Implicit)
+ .addReg(RISCV::F10_F, RegState::Implicit)
+ .addReg(RISCV::F11_F, RegState::Implicit)

With this change, some perennial tests now pass.

But there are still failures to be analyzed in SPEC.

[lewis-revill]

Rebased to fix conflicts with recent split SP adjustments

[luismarques]

The priority for this patch is to address the issues reported by @apazos but after that please check the clang-format output. There are some cases in this patch where it might make sense to use a different formatting than clang-format indicates, but the remaining should be addressed.

@apazos Have you considered tweaking the patch code to not do a tail call, just to check if that's what's causing the remaining failures? I'm not sure if that's too hard, but it could eventually be easier than drilling into the failing cases.

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

27 ↗

(On Diff #223855)

The return value isn't used as just an opaque index, it also reflects the frame size and is used for that purpose. The function comment should probably reflect that.

34 ↗

(On Diff #223855)

Use Register and RISCV::NoRegister. (You'll have to use MaxReg.id() instead in the call to max).

36 ↗

(On Diff #223855)

Might be worth adding a small comment explaining how this serves as a filters for the registers we are interested in. Or point to a later relevant comment?

39 ↗

(On Diff #223855)

Ditto NoRegister.

66 ↗

(On Diff #223855)

Check LLVM naming convention capitalization. Ditto other vars here.

93 ↗

(On Diff #223855)

Check LLVM naming convention capitalization. Ditto other vars here.

190 ↗

(On Diff #223855)

This could probably use SmallVector.

706 ↗

(On Diff #223855)

Ditto Register.

llvm/lib/Target/RISCV/RISCVRegisterInfo.cpp

95 ↗

(On Diff #223855)

Use IndexedMap instead?

llvm/test/CodeGen/RISCV/saverestore.ll

348 ↗

(On Diff #223855)

Maybe for these tests just put a -NOT check that __riscv_save_ isn't called?

[apazos]

Here is the bugpoint-reduced test case for the SPEC failure when enabling -msave-restore and allowing tail calls:

Run the command llc test.ll -mattr=+save-restore -o out.s

You will see the code generated is wrong:

tail    __riscv_restore_2
 jr      a5

target datalayout = "e-m:e-p:32:32-i64:64-n32-S128"
target triple = "riscv32-unknown-linux-gnu"

%struct.1_s = type { i8*, i8*, i8*}
%struct.2_s = type { i8*, i8*, i8*}

declare dso_local void @test2() local_unnamed_addr
declare dso_local i32 @test3(%struct.2_s* nonnull %a, %struct.1_s** %b) local_unnamed_addr

define dso_local void @test1(%struct.2_s* %a, %struct.1_s** %b) local_unnamed_addr {
entry:

br i1 undef, label %if.end2, label %if.then

if.then: ; preds = %entry

tail call void @test2()
br label %if.end2

if.end2: ; preds = %if.then, %entry

%call3 = tail call i32 null(%struct.2_s* nonnull %a, %struct.1_s** %b)
ret void

}

To summarize the issues we have detected so far:

1. Bug1: When transforming a tail call back into a regular call, the transformation is not correctly preserving the original call's return values.

I fixed this by conservatively adding RegState::Implicit definitions for each possible integer and FP return registers, when defining the riscv_restore tail call.

2. Bug2: test case provided above. Please Lewis take a look at how this case can be fixed.

3. I have also run the tests with "-fno-optimize-sibling-calls -msave-restore", i.e., disabling tail calls when m-save-restore is enabled.

With this config, issues (1) and (2) do not happen.
There is some loss in code size savings when using fno-optimize-sibling-calls.
But still we see code size savings benefit from -msave-restore.

[efriedma]

Bug2: test case provided above. Please Lewis take a look at how this case can be fixed.

Isn't the issue just that the code is checking for PseudoTAIL, and not PseudoTAILIndirect?

---

What happens if a tail call has more than 8 arguments?

What happens for a musttail call? A tailcc call? (Not sure if either of those are actually supported on RISCV at the moment?)

[apazos]

Yes Eli thanks for pointing out there are more scenarios that can fail.
It looks like the best solution is to permit both flags on, but then bail out from doing this transformation if there is any type of tail call already in the function.
This way we avoid messing with reverting tail calls back to regular calls.

[lewis-revill]

In D62686#1708792, @apazos wrote:

Yes Eli thanks for pointing out there are more scenarios that can fail.
It looks like the best solution is to permit both flags on, but then bail out from doing this transformation if there is any type of tail call already in the function.
This way we avoid messing with reverting tail calls back to regular calls.

Is it worth trying to disallow tail call optimization completely if this flag is enabled? I'm not sure what GCC does exactly. but this seems to be the behaviour.

[lewis-revill]

llvm/test/CodeGen/RISCV/saverestore.ll

348 ↗

(On Diff #223855)

Thanks, I realise it makes much more sense to do this for all the other 'non-save-restore' checks too, since they don't actually check any save/restore behaviour.

[apazos]

Is it worth trying to disallow tail call optimization completely if this flag is enabled? I'm not sure what GCC does exactly. but this seems to be the behaviour.

I had reported above that I have already run that test: with "-fno-optimize-sibling-calls -msave-restore", i.e., disabling tail calls when m-save-restore is enabled.

But it seems a better solution is to optimistically apply -msave-restore when there are no tail calls of any type in a function, instead of disabling tail calls completed. Let tail call optimization prevail over msave-restore. No LLVM target is disabling tail calls.

So you can update the patch according to this solution plan.

[lewis-revill]

Disable the save/restore optimization when a function contains tail calls. Address various miscellaneous concerns with the patch. Update tests to include less redundant code when checking cases where save/restore libcalls are not used.

[apazos]

Thanks Lewis, the runs are looking good, no failures, and good code size savings (average 3%)

[lewis-revill]

Rebase on top of shrink wrapping patch.

[lewis-revill]

Rebased and merged D68644 into this patch - this patch already assumes shrink wrapping support anyway.

[pzheng]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

278 ↗

(On Diff #220677)

Any chance this comment from @lenary is missed?

[lewis-revill]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

278 ↗

(On Diff #220677)

It appears this CFI code was removed since I last rebased.
But yes, just for posterity I did miss this comment!

[lewis-revill]

Rebased

[apazos]

Lewis, try rebasing it, not applying cleanly nor https://reviews.llvm.org/D62190

[lewis-revill]

Rebased.

[shiva0217]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

289 ↗

(On Diff #232702)

Should the -StackSize be -RealStackSize?

667 ↗

(On Diff #232702)

There is a case may trigger an assertion when compile with -O3 -g -msave-restore if the libcall has FrameSetup flag.

int main(int a, char* argv[]) {
  exit(0);
  return 0;
}

668 ↗

(On Diff #232702)

GCC will generate stack adjustment and GPR callee saved CFIs for the save libcalls. Should we do the same?

llvm/lib/Target/RISCV/RISCVRegisterInfo.h

40 ↗

(On Diff #232702)

An alternative of defining hasReservedSpillSlot could be set SaveRegs for the registers will be pushed by the libcalls in determineCalleeSaves. So the StackSize calculation will take the callee saved registers will be pushed by the libcalls into account.

[lewis-revill]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

667 ↗

(On Diff #232702)

Think I've found the cause of this. When 'DIFlagAllCallsDescribed' is set for the module LLVM attempts to provide call entry info DIEs for all calls, including this one that has been added by us. One detail of this is attempting to calculate the return PC value or offset for the call using a label that was assumed to be inserted after the call instruction.

However, while the attempt to add the call entry info doesn't check for the FrameSetup flag, the label was never inserted, since that code does check for the flag. Adding the missing check to avoid adding call entry info for calls marked as FrameSetup fixed this issue.

[lewis-revill]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

289 ↗

(On Diff #232702)

Looks like it, good catch.

668 ↗

(On Diff #232702)

I'm not familiar with the use of the CFI offset stuff, though just to be sure, you're saying that in addition to the manually-added .cfi_offsets in the libcalls themselves we would want to add them in our frame too?

[jrtc27]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

668 ↗

(On Diff #232702)

Yes. The CFI use is completely ignorant of any control flow. It goes to the start of the function, and walks forward, executing every .cfi_* directive until reaching the current point in the code. Since the libcalls (and thus their CFI directives) are not inline in the function, they are completely invisible to the CFI machinery. Thus, their side-effects must be duplicated straight after the calls to them.

[lewis-revill]

Rebased and addressed StackSize vs RealStackSize error.

[lewis-revill]

Fixed existing .cfi_offset offsets. Since these are frame-pointer based they also need to account for the libcall stack adjustment.

Currently working on adding .cfi_offset instructions for the registers saved by the libcall.

[lewis-revill]

Added .cfi_offset directives for registers saved by libcalls.

[apazos]

Lewis, is the patch final? It would be good to merge it before the 10.0 release branch creation on Jan 15th

[pzheng]

I see the following .cfi_offset directives generated using @shiva0217's test case. Any idea why the offset for ra is 536870908?

call    t0, __riscv_save_0
.cfi_def_cfa_offset 16
.cfi_offset ra, 536870908

[lewis-revill]

In D62686#1815158, @pzheng wrote:

I see the following .cfi_offset directives generated using @shiva0217's test case. Any idea why the offset for ra is 536870908?

call    t0, __riscv_save_0
.cfi_def_cfa_offset 16
.cfi_offset ra, 536870908

Hmm I see this too I'll look into it.

[lewis-revill]

Fix .cfi_offset signedness error.

[lewis-revill]

In D62686#1808041, @apazos wrote:

Lewis, is the patch final? It would be good to merge it before the 10.0 release branch creation on Jan 15th

I would say so now.

[pzheng]

In D62686#1816681, @lewis-revill wrote:

Fix .cfi_offset signedness error.

Thanks, @lewis-revill. It looks correct now.

[lewis-revill]

Should I wait for the comments to be resolved on D71593 before I commit this patch? Ideally if this patch makes it into a release then that bug fix should be there too.

[apazos]

Lewis, your latest patch looks good, we just had another run with no new failures. But we know it will have issues with -g. So I think we should not merge it yet. Do you have a version of the patch that creates the labels for the compiler-generated save/restore lib calls, so that this optimization does not depend on D71593? We could merge that version then, and when D71593 is accepted, you just have to rework/remove the label generation part of the patch.

[lewis-revill]

In D62686#1820816, @apazos wrote:

Lewis, your latest patch looks good, we just had another run with no new failures. But we know it will have issues with -g. So I think we should not merge it yet. Do you have a version of the patch that creates the labels for the compiler-generated save/restore lib calls, so that this optimization does not depend on D71593? We could merge that version then, and when D71593 is accepted, you just have to rework/remove the label generation part of the patch.

I don't expect that would be possible without making changes to generic code anyway. Removing the framesetup flag from the libcalls when generating them would allow labels to be produced, but that would require making modifications to the logic of this patch which relies on the libcalls being annotated as such.

[lewis-revill]

Rebased

[lewis-revill]

Since the DebugInfo fix has been accepted, I'm looking to get this patch and that fix committed shortly if there are no problems caused by rebasing.

lib/Target/RISCV/CMakeLists.txt

 add_llvm_target(RISCVCodeGen
   RISCVAsmPrinter.cpp
   RISCVExpandPseudoInsts.cpp
   RISCVFrameLowering.cpp
   RISCVInstrInfo.cpp
   RISCVISelDAGToDAG.cpp
   RISCVISelLowering.cpp
   RISCVMCInstLower.cpp
+  RISCVMachineFunctionInfo.cpp
   RISCVMergeBaseOffset.cpp
   RISCVRegisterInfo.cpp
   RISCVSubtarget.cpp
   RISCVTargetMachine.cpp
   RISCVTargetObjectFile.cpp
   RISCVTargetTransformInfo.cpp
   )
 
 add_subdirectory(AsmParser)
 add_subdirectory(Disassembler)
 add_subdirectory(MCTargetDesc)
 add_subdirectory(TargetInfo)
 add_subdirectory(Utils)

lib/Target/RISCV/RISCVFrameLowering.h

void processFunctionBeforeFrameFinalized(MachineFunction &MF,
                                            RegScavenger *RS) const override;
 
   bool hasFP(const MachineFunction &MF) const override;
 
   bool hasReservedCallFrame(const MachineFunction &MF) const override;
   MachineBasicBlock::iterator
   eliminateCallFramePseudoInstr(MachineFunction &MF, MachineBasicBlock &MBB,
                                 MachineBasicBlock::iterator MI) const override;
+  bool spillCalleeSavedRegisters(MachineBasicBlock &MBB,
+                                 MachineBasicBlock::iterator MI,
+                                 const std::vector<CalleeSavedInfo> &CSI,
+                                 const TargetRegisterInfo *TRI) const override;
+  bool
+  restoreCalleeSavedRegisters(MachineBasicBlock &MBB,
+                              MachineBasicBlock::iterator MI,
+                              std::vector<CalleeSavedInfo> &CSI,
+                              const TargetRegisterInfo *TRI) const override;
 
 protected:
   const RISCVSubtarget &STI;
 
 private:
   void determineFrameLayout(MachineFunction &MF) const;
   void adjustReg(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
                  const DebugLoc &DL, unsigned DestReg, unsigned SrcReg,
                  int64_t Val, MachineInstr::MIFlag Flag) const;
 };
 }
 #endif

lib/Target/RISCV/RISCVFrameLowering.cpp

void RISCVFrameLowering::emitPrologue(MachineFunction &MF,
   auto *RVFI = MF.getInfo<RISCVMachineFunctionInfo>();
   const RISCVRegisterInfo *RI = STI.getRegisterInfo();
   const RISCVInstrInfo *TII = STI.getInstrInfo();
   MachineBasicBlock::iterator MBBI = MBB.begin();
 
   unsigned FPReg = getFPReg(STI);
   unsigned SPReg = getSPReg(STI);
 
+  // Since spillCalleeSavedRegisters may have inserted a libcall, skip past
+  // any instructions marked as FrameSetup
+  while (MBBI != MBB.end() && MBBI->getFlag(MachineInstr::FrameSetup))
+    ++MBBI;
+
   // Debug location must be unknown since the first debug location is used
   // to determine the end of the prologue.
   DebugLoc DL;
 
   // Determine the correct frame layout
   determineFrameLayout(MF);
 
   // FIXME (note copied from Lanai): This appears to be overallocating.  Needs
   // investigation. Get the number of bytes to allocate from the FrameInfo.
   uint64_t StackSize = MFI.getStackSize();
 
   // Early exit if there is no need to allocate on the stack
   if (StackSize == 0 && !MFI.adjustsStack())
     return;
 
   // Allocate space on the stack if necessary.
   adjustReg(MBB, MBBI, DL, SPReg, SPReg, -StackSize, MachineInstr::FrameSetup);
 
   // Emit ".cfi_def_cfa_offset StackSize"
   unsigned CFIIndex = MF.addFrameInst(
       MCCFIInstruction::createDefCfaOffset(nullptr, -StackSize));
   BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
       .addCFIIndex(CFIIndex);
 
+  if (!RVFI->getUseSaveRestoreLibCalls()) {
     // The frame pointer is callee-saved, and code has been generated for us to
     // save it to the stack. We need to skip over the storing of callee-saved
     // registers as the frame pointer must be modified after it has been saved
     // to the stack, not before.
     // FIXME: assumes exactly one instruction is used to save each callee-saved
     // register.
     const std::vector<CalleeSavedInfo> &CSI = MFI.getCalleeSavedInfo();
     std::advance(MBBI, CSI.size());
 
     // Iterate over list of callee-saved registers and emit .cfi_offset
     // directives.
     for (const auto &Entry : CSI) {
       int64_t Offset = MFI.getObjectOffset(Entry.getFrameIdx());
       unsigned Reg = Entry.getReg();
       unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::createOffset(
           nullptr, RI->getDwarfRegNum(Reg, true), Offset));
       BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
           .addCFIIndex(CFIIndex);
     }
+  }
 
   // Generate new FP.
   if (hasFP(MF)) {
     adjustReg(MBB, MBBI, DL, FPReg, SPReg,
               StackSize - RVFI->getVarArgsSaveSize(), MachineInstr::FrameSetup);
 
     // Emit ".cfi_def_cfa $fp, 0"
     unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::createDefCfa(
void RISCVFrameLowering::emitEpilogue(MachineFunction &MF,
   const RISCVRegisterInfo *RI = STI.getRegisterInfo();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   auto *RVFI = MF.getInfo<RISCVMachineFunctionInfo>();
   DebugLoc DL = MBBI->getDebugLoc();
   const RISCVInstrInfo *TII = STI.getInstrInfo();
   unsigned FPReg = getFPReg(STI);
   unsigned SPReg = getSPReg(STI);
 
+  // If callee-saved registers are saved via libcall, place stack adjustment
+  // before this call.
+  while (MBBI != MBB.begin() &&
+         std::prev(MBBI)->getFlag(MachineInstr::FrameDestroy))
+    --MBBI;
+
+  const std::vector<CalleeSavedInfo> &CSI = MFI.getCalleeSavedInfo();
+
   // Skip to before the restores of callee-saved registers
   // FIXME: assumes exactly one instruction is used to restore each
   // callee-saved register.
-  auto LastFrameDestroy = std::prev(MBBI, MFI.getCalleeSavedInfo().size());
+  auto LastFrameDestroy = MBBI;
+  if (!RVFI->getUseSaveRestoreLibCalls())
+    LastFrameDestroy = std::prev(MBBI, CSI.size());
 
   uint64_t StackSize = MFI.getStackSize();
   uint64_t FPOffset = StackSize - RVFI->getVarArgsSaveSize();
 
   // Restore the stack pointer using the value of the frame pointer. Only
   // necessary if the stack pointer was modified, meaning the stack size is
   // unknown.
   if (RI->needsStackRealignment(MF) || MFI.hasVarSizedObjects()) {
for (auto &I = LastFrameDestroy; I != MBBI; ++I) {
               .addCFIIndex(CFIIndex);
           break;
         }
       }
     }
   }
 
   // Add CFI directives for callee-saved registers.
-  const std::vector<CalleeSavedInfo> &CSI = MFI.getCalleeSavedInfo();
+  if (!RVFI->getUseSaveRestoreLibCalls()) {
     // Iterate over list of callee-saved registers and emit .cfi_restore
     // directives.
     for (const auto &Entry : CSI) {
       unsigned Reg = Entry.getReg();
       unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::createRestore(
           nullptr, RI->getDwarfRegNum(Reg, true)));
       BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
           .addCFIIndex(CFIIndex);
     }
+  }
 
   // Deallocate stack
   adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackSize, MachineInstr::FrameDestroy);
 
   // After restoring $sp, we need to adjust CFA to $(sp + 0)
   // Emit ".cfi_def_cfa_offset 0"
   unsigned CFIIndex =
       MF.addFrameInst(MCCFIInstruction::createDefCfaOffset(nullptr, 0));
if (Amount != 0) {
         Amount = -Amount;
 
       adjustReg(MBB, MI, DL, SPReg, SPReg, Amount, MachineInstr::NoFlags);
     }
   }
 
   return MBB.erase(MI);
 }
+
+// Get the name of the libcall used for spilling callee saved registers.
+// If this function will not use save/restore libcalls, then return a nullptr.
+static const char *
+getSpillLibCallName(MachineFunction &MF,
+                    const std::vector<CalleeSavedInfo> &CSI) {
+  auto *RVFI = MF.getInfo<RISCVMachineFunctionInfo>();
+
+  if (CSI.empty() || !RVFI->getUseSaveRestoreLibCalls())
+    return nullptr;
+
+  using std::max;
+  unsigned MaxReg = 0;
+  for (auto &CS : CSI)
+    MaxReg = std::max(MaxReg, CS.getReg());
+
+  switch (MaxReg) {
+  default:
+    llvm_unreachable("Something has gone wrong!");
+  case /*s11*/ RISCV::X27: return "__riscv_save_12";
+  case /*s10*/ RISCV::X26: return "__riscv_save_11";
+  case /*s9*/  RISCV::X25: return "__riscv_save_10";
+  case /*s8*/  RISCV::X24: return "__riscv_save_9";
+  case /*s7*/  RISCV::X23: return "__riscv_save_8";
+  case /*s6*/  RISCV::X22: return "__riscv_save_7";
+  case /*s5*/  RISCV::X21: return "__riscv_save_6";
+  case /*s4*/  RISCV::X20: return "__riscv_save_5";
+  case /*s3*/  RISCV::X19: return "__riscv_save_4";
+  case /*s2*/  RISCV::X18: return "__riscv_save_3";
+  case /*s1*/  RISCV::X9:  return "__riscv_save_2";
+  case /*s0*/  RISCV::X8:  return "__riscv_save_1";
+  case /*ra*/  RISCV::X1:  return "__riscv_save_0";
+  }
+}
+
+// Get the name of the libcall used for restoring callee saved registers.
+// If this function will not use save/restore libcalls, then return a nullptr.
+static const char *
+getRestoreLibCallName(MachineFunction &MF,
+                      const std::vector<CalleeSavedInfo> &CSI) {
+  auto *RVFI = MF.getInfo<RISCVMachineFunctionInfo>();
+
+  if (CSI.empty() || !RVFI->getUseSaveRestoreLibCalls())
+    return nullptr;
+
+  using std::max;
+  unsigned MaxReg = 0;
+  for (auto &CS : CSI)
+    MaxReg = std::max(MaxReg, CS.getReg());
+
+  switch (MaxReg) {
+  default:
+    llvm_unreachable("Something has gone wrong!");
+  case /*s11*/ RISCV::X27: return "__riscv_restore_12";
+  case /*s10*/ RISCV::X26: return "__riscv_restore_11";
+  case /*s9*/  RISCV::X25: return "__riscv_restore_10";
+  case /*s8*/  RISCV::X24: return "__riscv_restore_9";
+  case /*s7*/  RISCV::X23: return "__riscv_restore_8";
+  case /*s6*/  RISCV::X22: return "__riscv_restore_7";
+  case /*s5*/  RISCV::X21: return "__riscv_restore_6";
+  case /*s4*/  RISCV::X20: return "__riscv_restore_5";
+  case /*s3*/  RISCV::X19: return "__riscv_restore_4";
+  case /*s2*/  RISCV::X18: return "__riscv_restore_3";
+  case /*s1*/  RISCV::X9:  return "__riscv_restore_2";
+  case /*s0*/  RISCV::X8:  return "__riscv_restore_1";
+  case /*ra*/  RISCV::X1:  return "__riscv_restore_0";
+  }
+}
+
+bool RISCVFrameLowering::spillCalleeSavedRegisters(
+    MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
+    const std::vector<CalleeSavedInfo> &CSI,
+    const TargetRegisterInfo *TRI) const {
+  if (CSI.empty())
+    return true;
+
+  MachineFunction *MF = MBB.getParent();
+  const TargetInstrInfo &TII = *MF->getSubtarget().getInstrInfo();
+  DebugLoc DL;
+  if (MI != MBB.end() && !MI->isDebugInstr())
+    DL = MI->getDebugLoc();
+
+  const char *SpillLibCall = getSpillLibCallName(*MF, CSI);
+  if (SpillLibCall) {
+    // Add spill libcall via non-callee-saved register t0.
+    BuildMI(MBB, MI, DL, TII.get(RISCV::PseudoCALLReg), RISCV::X5)
+        .addExternalSymbol(SpillLibCall, RISCVII::MO_CALL)
+        .setMIFlag(MachineInstr::FrameSetup);
+
+    // Add registers spilled in libcall as liveins.
+    for (auto &CS : CSI)
+      MBB.addLiveIn(CS.getReg());
+  } else {
+    // Manually spill values not spilled by libcall.
+    for (auto &CS : CSI) {
+      // Insert the spill to the stack frame.
+      unsigned Reg = CS.getReg();
+      const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(Reg);
+      TII.storeRegToStackSlot(MBB, MI, Reg, true, CS.getFrameIdx(), RC, TRI);
+    }
+  }
+
+  return true;
+}
+
+bool RISCVFrameLowering::restoreCalleeSavedRegisters(
+    MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
+    std::vector<CalleeSavedInfo> &CSI, const TargetRegisterInfo *TRI) const {
+  if (CSI.empty())
+    return true;
+
+  MachineFunction *MF = MBB.getParent();
+  const TargetInstrInfo &TII = *MF->getSubtarget().getInstrInfo();
+  DebugLoc DL;
+  if (MI != MBB.end() && !MI->isDebugInstr())
+    DL = MI->getDebugLoc();
+
+  const char *RestoreLibCall = getRestoreLibCallName(*MF, CSI);
+  if (RestoreLibCall) {
+    // Replace terminating tail calls with a simple call. This is valid because
+    // the return address register is always callee saved as part of the
+    // save/restore libcalls.
+    if (MI != MBB.end() && MI->getOpcode() == RISCV::PseudoTAIL) {
+      MachineBasicBlock::iterator NewMI =
+          BuildMI(MBB, MI, DL, TII.get(RISCV::PseudoCALL))
+              .add(MI->getOperand(0));
+      NewMI->copyImplicitOps(*MF, *MI);
+      MI->eraseFromParent();
+      MI = ++NewMI;
+    }
+
+    // Add restore libcall via tail call.
+    MachineBasicBlock::iterator NewMI =
+        BuildMI(MBB, MI, DL, TII.get(RISCV::PseudoTAIL))
+            .addExternalSymbol(RestoreLibCall, RISCVII::MO_CALL)
+            .setMIFlag(MachineInstr::FrameDestroy);
+
+    // Remove trailing returns, since the terminator is now a tail call to the
+    // restore function.
+    if (MI != MBB.end() && MI->getOpcode() == RISCV::PseudoRET) {
+      NewMI->copyImplicitOps(*MF, *MI);
+      MI->eraseFromParent();
+    }
+  } else {
+    // Manually restore values not restored by libcall. Insert in reverse order.
+    // loadRegFromStackSlot can insert multiple instructions.
+    for (auto &CS : reverse(CSI)) {
+      unsigned Reg = CS.getReg();
+      const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(Reg);
+      TII.loadRegFromStackSlot(MBB, MI, Reg, CS.getFrameIdx(), RC, TRI);
+      assert(MI != MBB.begin() &&
+             "loadRegFromStackSlot didn't insert any code!");
+    }
+  }
+
+  return true;
+}

lib/Target/RISCV/RISCVMachineFunctionInfo.h

private:
   MachineFunction &MF;
   /// FrameIndex for start of varargs area
   int VarArgsFrameIndex = 0;
   /// Size of the save area used for varargs
   int VarArgsSaveSize = 0;
   /// FrameIndex used for transferring values between 64-bit FPRs and a pair
   /// of 32-bit GPRs via the stack.
   int MoveF64FrameIndex = -1;
+  /// Store whether libcalls are used to save/restore callee-saved registers.
+  Optional<bool> UseSaveRestoreLibCalls = None;
 
 public:
   //  RISCVMachineFunctionInfo() = default;
 
   RISCVMachineFunctionInfo(MachineFunction &MF) : MF(MF) {}
 
   int getVarArgsFrameIndex() const { return VarArgsFrameIndex; }
   void setVarArgsFrameIndex(int Index) { VarArgsFrameIndex = Index; }
 
   unsigned getVarArgsSaveSize() const { return VarArgsSaveSize; }
   void setVarArgsSaveSize(int Size) { VarArgsSaveSize = Size; }
 
   int getMoveF64FrameIndex() {
     if (MoveF64FrameIndex == -1)
       MoveF64FrameIndex = MF.getFrameInfo().CreateStackObject(8, 8, false);
     return MoveF64FrameIndex;
   }
+
+  bool useSaveRestoreLibCalls() const;
+
+  bool getUseSaveRestoreLibCalls() {
+    if (!UseSaveRestoreLibCalls)
+      UseSaveRestoreLibCalls = useSaveRestoreLibCalls();
+    return *UseSaveRestoreLibCalls;
+  }
 };
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_RISCV_RISCVMACHINEFUNCTIONINFO_H

lib/Target/RISCV/RISCVMachineFunctionInfo.cpp

+//=- RISCVMachineFunctionInfo.cpp - RISCV machine function info ---*- C++ -*-=//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines functionality for RISCVMachineFunctionInfo.
+//
+//===----------------------------------------------------------------------===//
+
+#include "RISCVMachineFunctionInfo.h"
+
+using namespace llvm;
+
+static cl::opt<bool> EnableSaveRestore(
+    "enable-save-restore", cl::init(false),
+    cl::desc("Enable save/restore of callee-saved registers via libcalls"));
+
+bool RISCVMachineFunctionInfo::useSaveRestoreLibCalls() const {
+  if (!EnableSaveRestore)
+    return false;
+
+  // We cannot use fixed locations for the callee saved spill slots if the
+  // function uses a varargs save area.
+  if (VarArgsSaveSize != 0)
+    return false;
+
+  return true;
+}

lib/Target/RISCV/RISCVRegisterInfo.h

struct RISCVRegisterInfo : public RISCVGenRegisterInfo {
   const MCPhysReg *getCalleeSavedRegs(const MachineFunction *MF) const override;
 
   BitVector getReservedRegs(const MachineFunction &MF) const override;
 
   bool isConstantPhysReg(unsigned PhysReg) const override;
 
   const uint32_t *getNoPreservedMask() const override;
 
+  bool hasReservedSpillSlot(const MachineFunction &MF, unsigned Reg,
+                            int &FrameIdx) const override;
+
   void eliminateFrameIndex(MachineBasicBlock::iterator MI, int SPAdj,
                            unsigned FIOperandNum,
                            RegScavenger *RS = nullptr) const override;
 
   Register getFrameRegister(const MachineFunction &MF) const override;
 
   bool requiresRegisterScavenging(const MachineFunction &MF) const override {
     return true;

lib/Target/RISCV/RISCVRegisterInfo.cpp

 //===-- RISCVRegisterInfo.cpp - RISCV Register Information ------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file contains the RISCV implementation of the TargetRegisterInfo class.
 //
 //===----------------------------------------------------------------------===//
 
 #include "RISCVRegisterInfo.h"
 #include "RISCV.h"
+#include "RISCVMachineFunctionInfo.h"
 #include "RISCVSubtarget.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/RegisterScavenging.h"
 #include "llvm/CodeGen/TargetFrameLowering.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/Support/ErrorHandling.h"

 bool RISCVRegisterInfo::isConstantPhysReg(unsigned PhysReg) const {
   return PhysReg == RISCV::X0;
 }
 
 const uint32_t *RISCVRegisterInfo::getNoPreservedMask() const {
   return CSR_NoRegs_RegMask;
 }
 
+// Frame indexes representing locations of CSRs which are given a fixed location
+// by save/restore libcalls.
+static std::map<unsigned, int> FixedCSRFIMap = {
+  {/*ra*/  RISCV::X1,   -1},
+  {/*s0*/  RISCV::X8,   -2},
+  {/*s1*/  RISCV::X9,   -3},
+  {/*s2*/  RISCV::X18,  -4},
+  {/*s3*/  RISCV::X19,  -5},
+  {/*s4*/  RISCV::X20,  -6},
+  {/*s5*/  RISCV::X21,  -7},
+  {/*s6*/  RISCV::X22,  -8},
+  {/*s7*/  RISCV::X23,  -9},
+  {/*s8*/  RISCV::X24,  -10},
+  {/*s9*/  RISCV::X25,  -11},
+  {/*s10*/ RISCV::X26,  -12},
+  {/*s11*/ RISCV::X27,  -13}};
+
+bool RISCVRegisterInfo::hasReservedSpillSlot(const MachineFunction &MF,
+                                             unsigned Reg,
+                                             int &FrameIdx) const {
+  auto *RVFI =
+      const_cast<MachineFunction *>(&MF)->getInfo<RISCVMachineFunctionInfo>();
+  if (!RVFI->getUseSaveRestoreLibCalls())
+    return false;
+
+  auto FII = FixedCSRFIMap.find(Reg);
+  if (FII == FixedCSRFIMap.end())
+    return false;
+
+  FrameIdx = FII->second;
+  return true;
+}
+
 void RISCVRegisterInfo::eliminateFrameIndex(MachineBasicBlock::iterator II,
                                             int SPAdj, unsigned FIOperandNum,
                                             RegScavenger *RS) const {
   assert(SPAdj == 0 && "Unexpected non-zero SPAdj value");
 
   MachineInstr &MI = *II;
   MachineFunction &MF = *MI.getParent()->getParent();
   MachineRegisterInfo &MRI = MF.getRegInfo();

test/CodeGen/RISCV/saverestore.ll

+; RUN: llc -mtriple=riscv32 < %s | FileCheck %s -check-prefix=RV32I
+; RUN: llc -mtriple=riscv64 < %s | FileCheck %s -check-prefix=RV64I
+; RUN: llc -mtriple=riscv32 -enable-save-restore < %s | FileCheck %s -check-prefix=RV32I-SR
+; RUN: llc -mtriple=riscv64 -enable-save-restore < %s | FileCheck %s -check-prefix=RV64I-SR
+
+
+; Check that the correct save/restore libcalls are generated.
+
+@var0 = global [18 x i32] zeroinitializer
+@var1 = global [24 x i32] zeroinitializer
+@var2 = global [30 x i32] zeroinitializer
+
+define void @callee_saved0() nounwind {
+; RV32I-LABEL: callee_saved0:
+; RV32I:         addi sp, sp, -32
+; RV32I-NEXT:    sw s0, 28(sp)
+; RV32I-NEXT:    sw s1, 24(sp)
+; RV32I-NEXT:    sw s2, 20(sp)
+; RV32I-NEXT:    sw s3, 16(sp)
+; RV32I-NEXT:    sw s4, 12(sp)
+; RV32I:         lw s4, 12(sp)
+; RV32I-NEXT:    lw s3, 16(sp)
+; RV32I-NEXT:    lw s2, 20(sp)
+; RV32I-NEXT:    lw s1, 24(sp)
+; RV32I-NEXT:    lw s0, 28(sp)
+; RV32I-NEXT:    addi sp, sp, 32
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: callee_saved0:
+; RV64I:         addi sp, sp, -48
+; RV64I-NEXT:    sd s0, 40(sp)
+; RV64I-NEXT:    sd s1, 32(sp)
+; RV64I-NEXT:    sd s2, 24(sp)
+; RV64I-NEXT:    sd s3, 16(sp)
+; RV64I:         ld s4, 8(sp)
+; RV64I-NEXT:    ld s3, 16(sp)
+; RV64I-NEXT:    ld s2, 24(sp)
+; RV64I-NEXT:    ld s1, 32(sp)
+; RV64I-NEXT:    ld s0, 40(sp)
+; RV64I-NEXT:    addi sp, sp, 48
+; RV64I-NEXT:    ret
+;
+; RV32I-SR-LABEL: callee_saved0:
+; RV32I-SR:         call t0, __riscv_save_5
+; RV32I-SR:         tail __riscv_restore_5
+;
+; RV64I-SR-LABEL: callee_saved0:
+; RV64I-SR:         call t0, __riscv_save_5
+; RV64I-SR:         tail __riscv_restore_5
+  %val = load [18 x i32], [18 x i32]* @var0
+  store volatile [18 x i32] %val, [18 x i32]* @var0
+  ret void
+}
+
+define void @callee_saved1() nounwind {
+; RV32I-LABEL: callee_saved1:
+; RV32I:         addi sp, sp, -48
+; RV32I-NEXT:    sw s0, 44(sp)
+; RV32I-NEXT:    sw s1, 40(sp)
+; RV32I-NEXT:    sw s2, 36(sp)
+; RV32I-NEXT:    sw s3, 32(sp)
+; RV32I-NEXT:    sw s4, 28(sp)
+; RV32I-NEXT:    sw s5, 24(sp)
+; RV32I-NEXT:    sw s6, 20(sp)
+; RV32I-NEXT:    sw s7, 16(sp)
+; RV32I-NEXT:    sw s8, 12(sp)
+; RV32I-NEXT:    sw s9, 8(sp)
+; RV32I-NEXT:    sw s10, 4(sp)
+; RV32I:         lw s10, 4(sp)
+; RV32I-NEXT:    lw s9, 8(sp)
+; RV32I-NEXT:    lw s8, 12(sp)
+; RV32I-NEXT:    lw s7, 16(sp)
+; RV32I-NEXT:    lw s6, 20(sp)
+; RV32I-NEXT:    lw s5, 24(sp)
+; RV32I-NEXT:    lw s4, 28(sp)
+; RV32I-NEXT:    lw s3, 32(sp)
+; RV32I-NEXT:    lw s2, 36(sp)
+; RV32I-NEXT:    lw s1, 40(sp)
+; RV32I-NEXT:    lw s0, 44(sp)
+; RV32I-NEXT:    addi sp, sp, 48
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: callee_saved1:
+; RV64I:         addi sp, sp, -96
+; RV64I-NEXT:    sd s0, 88(sp)
+; RV64I-NEXT:    sd s1, 80(sp)
+; RV64I-NEXT:    sd s2, 72(sp)
+; RV64I-NEXT:    sd s3, 64(sp)
+; RV64I-NEXT:    sd s4, 56(sp)
+; RV64I-NEXT:    sd s5, 48(sp)
+; RV64I-NEXT:    sd s6, 40(sp)
+; RV64I-NEXT:    sd s7, 32(sp)
+; RV64I-NEXT:    sd s8, 24(sp)
+; RV64I-NEXT:    sd s9, 16(sp)
+; RV64I-NEXT:    sd s10, 8(sp)
+; RV64I:         ld s10, 8(sp)
+; RV64I-NEXT:    ld s9, 16(sp)
+; RV64I-NEXT:    ld s8, 24(sp)
+; RV64I-NEXT:    ld s7, 32(sp)
+; RV64I-NEXT:    ld s6, 40(sp)
+; RV64I-NEXT:    ld s5, 48(sp)
+; RV64I-NEXT:    ld s4, 56(sp)
+; RV64I-NEXT:    ld s3, 64(sp)
+; RV64I-NEXT:    ld s2, 72(sp)
+; RV64I-NEXT:    ld s1, 80(sp)
+; RV64I-NEXT:    ld s0, 88(sp)
+; RV64I-NEXT:    addi sp, sp, 96
+; RV64I-NEXT:    ret
+;
+; RV32I-SR-LABEL: callee_saved1:
+; RV32I-SR:         call t0, __riscv_save_11
+; RV32I-SR:         tail __riscv_restore_11
+;
+; RV64I-SR-LABEL: callee_saved1:
+; RV64I-SR:         call t0, __riscv_save_11
+; RV64I-SR:         tail __riscv_restore_11
+  %val = load [24 x i32], [24 x i32]* @var1
+  store volatile [24 x i32] %val, [24 x i32]* @var1
+  ret void
+}
+
+define void @callee_saved2() nounwind {
+; RV32I-LABEL: callee_saved2:
+; RV32I:         addi sp, sp, -64
+; RV32I-NEXT:    sw s0, 60(sp)
+; RV32I-NEXT:    sw s1, 56(sp)
+; RV32I-NEXT:    sw s2, 52(sp)
+; RV32I-NEXT:    sw s3, 48(sp)
+; RV32I-NEXT:    sw s4, 44(sp)
+; RV32I-NEXT:    sw s5, 40(sp)
+; RV32I-NEXT:    sw s6, 36(sp)
+; RV32I-NEXT:    sw s7, 32(sp)
+; RV32I-NEXT:    sw s8, 28(sp)
+; RV32I-NEXT:    sw s9, 24(sp)
+; RV32I-NEXT:    sw s10, 20(sp)
+; RV32I-NEXT:    sw s11, 16(sp)
+; RV32I:         lw s11, 16(sp)
+; RV32I-NEXT:    lw s10, 20(sp)
+; RV32I-NEXT:    lw s9, 24(sp)
+; RV32I-NEXT:    lw s8, 28(sp)
+; RV32I-NEXT:    lw s7, 32(sp)
+; RV32I-NEXT:    lw s6, 36(sp)
+; RV32I-NEXT:    lw s5, 40(sp)
+; RV32I-NEXT:    lw s4, 44(sp)
+; RV32I-NEXT:    lw s3, 48(sp)
+; RV32I-NEXT:    lw s2, 52(sp)
+; RV32I-NEXT:    lw s1, 56(sp)
+; RV32I-NEXT:    lw s0, 60(sp)
+; RV32I-NEXT:    addi sp, sp, 64
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: callee_saved2:
+; RV64I:         addi sp, sp, -128
+; RV64I-NEXT:    sd s0, 120(sp)
+; RV64I-NEXT:    sd s1, 112(sp)
+; RV64I-NEXT:    sd s2, 104(sp)
+; RV64I-NEXT:    sd s3, 96(sp)
+; RV64I-NEXT:    sd s4, 88(sp)
+; RV64I-NEXT:    sd s5, 80(sp)
+; RV64I-NEXT:    sd s6, 72(sp)
+; RV64I-NEXT:    sd s7, 64(sp)
+; RV64I-NEXT:    sd s8, 56(sp)
+; RV64I-NEXT:    sd s9, 48(sp)
+; RV64I-NEXT:    sd s10, 40(sp)
+; RV64I-NEXT:    sd s11, 32(sp)
+; RV64I:         ld s11, 32(sp)
+; RV64I-NEXT:    ld s10, 40(sp)
+; RV64I-NEXT:    ld s9, 48(sp)
+; RV64I-NEXT:    ld s8, 56(sp)
+; RV64I-NEXT:    ld s7, 64(sp)
+; RV64I-NEXT:    ld s6, 72(sp)
+; RV64I-NEXT:    ld s5, 80(sp)
+; RV64I-NEXT:    ld s4, 88(sp)
+; RV64I-NEXT:    ld s3, 96(sp)
+; RV64I-NEXT:    ld s2, 104(sp)
+; RV64I-NEXT:    ld s1, 112(sp)
+; RV64I-NEXT:    ld s0, 120(sp)
+; RV64I-NEXT:    addi sp, sp, 128
+; RV64I-NEXT:    ret
+;
+; RV32I-SR-LABEL: callee_saved2:
+; RV32I-SR:         call t0, __riscv_save_12
+; RV32I-SR:         tail __riscv_restore_12
+;
+; RV64I-SR-LABEL: callee_saved2:
+; RV64I-SR:         call t0, __riscv_save_12
+; RV64I-SR:         tail __riscv_restore_12
+  %val = load [30 x i32], [30 x i32]* @var2
+  store volatile [30 x i32] %val, [30 x i32]* @var2
+  ret void
+}
+
+; Check that tail calls are updated correctly by save/restore
+
+declare i32 @tail_callee(i32 %i)
+
+define i32 @tail_call(i32 %i) nounwind {
+; RV32I-LABEL: tail_call:
+; RV32I:         addi sp, sp, -32
+; RV32I-NEXT:    sw s0, 28(sp)
+; RV32I-NEXT:    sw s1, 24(sp)
+; RV32I-NEXT:    sw s2, 20(sp)
+; RV32I-NEXT:    sw s3, 16(sp)
+; RV32I-NEXT:    sw s4, 12(sp)
+; RV32I-NEXT:    sw s5, 8(sp)
+; RV32I:         lw s5, 8(sp)
+; RV32I-NEXT:    lw s4, 12(sp)
+; RV32I-NEXT:    lw s3, 16(sp)
+; RV32I-NEXT:    lw s2, 20(sp)
+; RV32I-NEXT:    lw s1, 24(sp)
+; RV32I-NEXT:    lw s0, 28(sp)
+; RV32I-NEXT:    addi sp, sp, 32
+; RV32I-NEXT:    tail tail_callee
+;
+; RV64I-LABEL: tail_call:
+; RV64I:         addi sp, sp, -48
+; RV64I-NEXT:    sd s0, 40(sp)
+; RV64I-NEXT:    sd s1, 32(sp)
+; RV64I-NEXT:    sd s2, 24(sp)
+; RV64I-NEXT:    sd s3, 16(sp)
+; RV64I-NEXT:    sd s4, 8(sp)
+; RV64I-NEXT:    sd s5, 0(sp)
+; RV64I:         ld s5, 0(sp)
+; RV64I-NEXT:    ld s4, 8(sp)
+; RV64I-NEXT:    ld s3, 16(sp)
+; RV64I-NEXT:    ld s2, 24(sp)
+; RV64I-NEXT:    ld s1, 32(sp)
+; RV64I-NEXT:    ld s0, 40(sp)
+; RV64I-NEXT:    addi sp, sp, 48
+; RV64I-NEXT:    tail tail_callee
+;
+; RV32I-SR-LABEL: tail_call:
+; RV32I-SR:         call t0, __riscv_save_6
+; RV32I-SR:         call tail_callee
+; RV32I-SR-NEXT:    tail __riscv_restore_6
+;
+; RV64I-SR-LABEL: tail_call:
+; RV64I-SR:         call t0, __riscv_save_6
+; RV64I-SR:         call tail_callee
+; RV64I-SR-NEXT:    tail __riscv_restore_6
+entry:
+  %val = load [18 x i32], [18 x i32]* @var0
+  store volatile [18 x i32] %val, [18 x i32]* @var0
+  %r = tail call i32 @tail_callee(i32 %i)
+  ret i32 %r
+}
+
+; Check that functions with varargs do not use save/restore code
+
+declare void @llvm.va_start(i8*)
+declare void @llvm.va_end(i8*)
+
+define i32 @varargs(i8* %fmt, ...) nounwind {
+; RV32I-LABEL: varargs:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi sp, sp, -48
+; RV32I-NEXT:    mv a0, a1
+; RV32I-NEXT:    sw a7, 44(sp)
+; RV32I-NEXT:    sw a6, 40(sp)
+; RV32I-NEXT:    sw a5, 36(sp)
+; RV32I-NEXT:    sw a4, 32(sp)
+; RV32I-NEXT:    sw a3, 28(sp)
+; RV32I-NEXT:    sw a2, 24(sp)
+; RV32I-NEXT:    addi a1, sp, 24
+; RV32I-NEXT:    sw a1, 12(sp)
+; RV32I-NEXT:    sw a0, 20(sp)
+; RV32I-NEXT:    addi sp, sp, 48
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: varargs:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi sp, sp, -80
+; RV64I-NEXT:    sd a1, 24(sp)
+; RV64I-NEXT:    sd a7, 72(sp)
+; RV64I-NEXT:    sd a6, 64(sp)
+; RV64I-NEXT:    sd a5, 56(sp)
+; RV64I-NEXT:    sd a4, 48(sp)
+; RV64I-NEXT:    sd a3, 40(sp)
+; RV64I-NEXT:    sd a2, 32(sp)
+; RV64I-NEXT:    addi a0, sp, 24
+; RV64I-NEXT:    ori a0, a0, 4
+; RV64I-NEXT:    sd a0, 8(sp)
+; RV64I-NEXT:    lw a0, 24(sp)
+; RV64I-NEXT:    addi sp, sp, 80
+; RV64I-NEXT:    ret
+;
+; RV32I-SR-LABEL: varargs:
+; RV32I-SR:       # %bb.0:
+; RV32I-SR-NEXT:    addi sp, sp, -48
+; RV32I-SR-NEXT:    mv a0, a1
+; RV32I-SR-NEXT:    sw a7, 44(sp)
+; RV32I-SR-NEXT:    sw a6, 40(sp)
+; RV32I-SR-NEXT:    sw a5, 36(sp)
+; RV32I-SR-NEXT:    sw a4, 32(sp)
+; RV32I-SR-NEXT:    sw a3, 28(sp)
+; RV32I-SR-NEXT:    sw a2, 24(sp)
+; RV32I-SR-NEXT:    addi a1, sp, 24
+; RV32I-SR-NEXT:    sw a1, 12(sp)
+; RV32I-SR-NEXT:    sw a0, 20(sp)
+; RV32I-SR-NEXT:    addi sp, sp, 48
+; RV32I-SR-NEXT:    ret
+;
+; RV64I-SR-LABEL: varargs:
+; RV64I-SR:       # %bb.0:
+; RV64I-SR-NEXT:    addi sp, sp, -80
+; RV64I-SR-NEXT:    sd a1, 24(sp)
+; RV64I-SR-NEXT:    sd a7, 72(sp)
+; RV64I-SR-NEXT:    sd a6, 64(sp)
+; RV64I-SR-NEXT:    sd a5, 56(sp)
+; RV64I-SR-NEXT:    sd a4, 48(sp)
+; RV64I-SR-NEXT:    sd a3, 40(sp)
+; RV64I-SR-NEXT:    sd a2, 32(sp)
+; RV64I-SR-NEXT:    addi a0, sp, 24
+; RV64I-SR-NEXT:    ori a0, a0, 4
+; RV64I-SR-NEXT:    sd a0, 8(sp)
+; RV64I-SR-NEXT:    lw a0, 24(sp)
+; RV64I-SR-NEXT:    addi sp, sp, 80
+; RV64I-SR-NEXT:    ret
+  %va = alloca i8*, align 4
+  %1 = bitcast i8** %va to i8*
+  call void @llvm.va_start(i8* %1)
+  %argp.cur = load i8*, i8** %va, align 4
+  %argp.next = getelementptr inbounds i8, i8* %argp.cur, i32 4
+  store i8* %argp.next, i8** %va, align 4
+  %2 = bitcast i8* %argp.cur to i32*
+  %3 = load i32, i32* %2, align 4
+  call void @llvm.va_end(i8* %1)
+  ret i32 %3
+}
+
+define void @many_args(i32, i32, i32, i32, i32, i32, i32, i32, i32) nounwind {
+; RV32I-LABEL: many_args:
+; RV32I:         addi sp, sp, -32
+; RV32I-NEXT:    sw s0, 28(sp)
+; RV32I-NEXT:    sw s1, 24(sp)
+; RV32I-NEXT:    sw s2, 20(sp)
+; RV32I-NEXT:    sw s3, 16(sp)
+; RV32I-NEXT:    sw s4, 12(sp)
+; RV32I:         lw s4, 12(sp)
+; RV32I-NEXT:    lw s3, 16(sp)
+; RV32I-NEXT:    lw s2, 20(sp)
+; RV32I-NEXT:    lw s1, 24(sp)
+; RV32I-NEXT:    lw s0, 28(sp)
+; RV32I-NEXT:    addi sp, sp, 32
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: many_args:
+; RV64I:         addi sp, sp, -48
+; RV64I-NEXT:    sd s0, 40(sp)
+; RV64I-NEXT:    sd s1, 32(sp)
+; RV64I-NEXT:    sd s2, 24(sp)
+; RV64I-NEXT:    sd s3, 16(sp)
+; RV64I-NEXT:    sd s4, 8(sp)
+; RV64I:         ld s4, 8(sp)
+; RV64I-NEXT:    ld s3, 16(sp)
+; RV64I-NEXT:    ld s2, 24(sp)
+; RV64I-NEXT:    ld s1, 32(sp)
+; RV64I-NEXT:    ld s0, 40(sp)
+; RV64I-NEXT:    addi sp, sp, 48
+; RV64I-NEXT:    ret
+;
+; RV32I-SR-LABEL: many_args:
+; RV32I-SR:         call t0, __riscv_save_5
+; RV32I-SR:         tail __riscv_restore_5
+;
+; RV64I-SR-LABEL: many_args:
+; RV64I-SR:         call t0, __riscv_save_5
+; RV64I-SR:         tail __riscv_restore_5
+entry:
+  %val = load [18 x i32], [18 x i32]* @var0
+  store volatile [18 x i32] %val, [18 x i32]* @var0
+  ret void
+}
+
+; Check that dynamic allocation calculations remain correct
+
+declare i8* @llvm.stacksave()
+declare void @llvm.stackrestore(i8*)
+declare void @notdead(i8*)
+
+define void @alloca(i32 %n) nounwind {
+; RV32I-LABEL: alloca:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi sp, sp, -16
+; RV32I-NEXT:    sw ra, 12(sp)
+; RV32I-NEXT:    sw s0, 8(sp)
+; RV32I-NEXT:    sw s1, 4(sp)
+; RV32I-NEXT:    addi s0, sp, 16
+; RV32I-NEXT:    mv s1, sp
+; RV32I-NEXT:    addi a0, a0, 15
+; RV32I-NEXT:    andi a0, a0, -16
+; RV32I-NEXT:    sub a0, sp, a0
+; RV32I-NEXT:    mv sp, a0
+; RV32I-NEXT:    call notdead
+; RV32I-NEXT:    mv sp, s1
+; RV32I-NEXT:    addi sp, s0, -16
+; RV32I-NEXT:    lw s1, 4(sp)
+; RV32I-NEXT:    lw s0, 8(sp)
+; RV32I-NEXT:    lw ra, 12(sp)
+; RV32I-NEXT:    addi sp, sp, 16
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: alloca:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi sp, sp, -32
+; RV64I-NEXT:    sd ra, 24(sp)
+; RV64I-NEXT:    sd s0, 16(sp)
+; RV64I-NEXT:    sd s1, 8(sp)
+; RV64I-NEXT:    addi s0, sp, 32
+; RV64I-NEXT:    addi a1, zero, 1
+; RV64I-NEXT:    slli a1, a1, 33
+; RV64I-NEXT:    addi a1, a1, -16
+; RV64I-NEXT:    slli a0, a0, 32
+; RV64I-NEXT:    srli a0, a0, 32
+; RV64I-NEXT:    addi a0, a0, 15
+; RV64I-NEXT:    and a0, a0, a1
+; RV64I-NEXT:    mv s1, sp
+; RV64I-NEXT:    sub a0, sp, a0
+; RV64I-NEXT:    mv sp, a0
+; RV64I-NEXT:    call notdead
+; RV64I-NEXT:    mv sp, s1
+; RV64I-NEXT:    addi sp, s0, -32
+; RV64I-NEXT:    ld s1, 8(sp)
+; RV64I-NEXT:    ld s0, 16(sp)
+; RV64I-NEXT:    ld ra, 24(sp)
+; RV64I-NEXT:    addi sp, sp, 32
+; RV64I-NEXT:    ret
+;
+; RV32I-SR-LABEL: alloca:
+; RV32I-SR:       # %bb.0:
+; RV32I-SR-NEXT:    call t0, __riscv_save_2
+; RV32I-SR-NEXT:    mv s0, sp
+; RV32I-SR-NEXT:    mv s1, sp
+; RV32I-SR-NEXT:    addi a0, a0, 15
+; RV32I-SR-NEXT:    andi a0, a0, -16
+; RV32I-SR-NEXT:    sub a0, sp, a0
+; RV32I-SR-NEXT:    mv sp, a0
+; RV32I-SR-NEXT:    call notdead
+; RV32I-SR-NEXT:    mv sp, s1
+; RV32I-SR-NEXT:    mv sp, s0
+; RV32I-SR-NEXT:    tail __riscv_restore_2
+;
+; RV64I-SR-LABEL: alloca:
+; RV64I-SR:       # %bb.0:
+; RV64I-SR-NEXT:    call t0, __riscv_save_2
+; RV64I-SR-NEXT:    mv s0, sp
+; RV64I-SR-NEXT:    addi a1, zero, 1
+; RV64I-SR-NEXT:    slli a1, a1, 33
+; RV64I-SR-NEXT:    addi a1, a1, -16
+; RV64I-SR-NEXT:    slli a0, a0, 32
+; RV64I-SR-NEXT:    srli a0, a0, 32
+; RV64I-SR-NEXT:    addi a0, a0, 15
+; RV64I-SR-NEXT:    and a0, a0, a1
+; RV64I-SR-NEXT:    mv s1, sp
+; RV64I-SR-NEXT:    sub a0, sp, a0
+; RV64I-SR-NEXT:    mv sp, a0
+; RV64I-SR-NEXT:    call notdead
+; RV64I-SR-NEXT:    mv sp, s1
+; RV64I-SR-NEXT:    mv sp, s0
+; RV64I-SR-NEXT:    tail __riscv_restore_2
+  %sp = call i8* @llvm.stacksave()
+  %addr = alloca i8, i32 %n
+  call void @notdead(i8* %addr)
+  call void @llvm.stackrestore(i8* %sp)
+  ret void
+}