[RISCV] Readjusting the framestack for Zcmp

[VincentWu]

Summary

This patch readjusts the frame stack for the push and pop instructions

co-author: @Lukacma

[VincentWu]

Do not emit Push/Pop when FP is in used

Fixed the problem mentioned in https://github.com/plctlab/llvm-project/issues/58

For the reason that the Spec of the ZCMP extension is not currently modified for this purpose, the LLVM implementation is modified first to avoid this problem. When DisableFramePointerElim is set, the generation of Push/Pop instructions is stopped.

[KYG]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

267

Seems like the def PGPR : RegisterClass ... is missed in RISCVRegisterInfo.td?

llvm/lib/Target/RISCV/RISCVPushPopOptimizer.cpp

106

Um...where is the implementation of this function? (Or use isCopyInstrImpl() instead?)

[VincentWu]

rebase & update to 1.0

[VincentWu]

rebase

[jrtc27]

llvm/test/CodeGen/RISCV/cm_mvas_mvsa.ll

2

Please no manual CHECK lines, use update_mir_test_checks.py

3

Indent, and why VALID rather than CHECK?

8

Don't just dump Clang-produced IR into a test, clean it up properly to be minimal and comprehensible. And fix your formatting issues while you're at it (*nocapture is incorrect)

llvm/test/CodeGen/RISCV/push-pop-popret.ll

3

?

15

Again, clean up your IR. Though this attribute isn't even there, so the comment is worse than unnecessary, it's wrong.

[• pcwang-thead]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

393

Any reason why we change it to STL vector?

[VincentWu]

address comments

[kito-cheng]

llvm/lib/Target/RISCV/RISCVRegisterInfo.td

186

Zcmp?

[VincentWu]

rebase

[VincentWu]

address comments

[kito-cheng]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

280

Zcmp

llvm/lib/Target/RISCV/RISCVMoveOptimizer.cpp

20

Zcmp

llvm/lib/Target/RISCV/RISCVPushPopOptimizer.cpp

21

Zcmp

[kito-cheng]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

300

Zcmp

[KYG]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

257

reallocPushStackFream -> reallocPushStackFrame ?
Is it necessary to setObjectOffset for non-pushed registers? I though their offset has already been set in
PEI::calculateFrameObjectOffsets (correct me if I'm wrong).

284

total_sp_adj = base_sp_adj (rlist) + additional_sp_adj (Spimm)
I think it's better to get base_sp_adj through rlist, instead of assuming that the init Spimm value is actually base_sp_adj then correct it to additional_sp_adj here.

565

Already checked in isCSIpushable().

1396

I think we should have something like getNonLibcallCSI(), maybe getNonPushPopCSI()?

1405–1410

Better just addImm(0) then update it in adjSPInPushPop(), since the Spimm field never mean to store the base sp adjustment.

1457–1470

Is it possible to cache the result (isPushalbe, and maybe rlist)?
Since push/pop uses a reversed order, using different rlist between prologue and epilogue will cause issue, so maybe just check isCSIpushable once then cache the rlist value.

1472–1498

Need to notify CM_POP kill (def) callee-saved registers, otherwise MIScheduler could not see the dependency, but I'm not sure how to do this (addDef to each poped register?).

[kito-cheng]

Could you split this patch into 3 different revision for easier review? it also easier to demo what you improved for individual part.

1. move merging pass
2. Push/Pop optimization pass
3. FrameLowering related stuffs

My understating is: 1 can be standalone patch, and 2 has some dependency on 3, but it should be able to split that into separated patch?

[VincentWu]

split the patch

https://reviews.llvm.org/D150415

https://reviews.llvm.org/D150416

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

257

IMHO, I think it might be necessary in some cases.
Please refer to the callee_with_irq in llvm/test/CodeGen/RISCV/push-pop-popret.ll.
It will generate following code in RV32I

addi    sp, sp, -144
sw      ra, 140(sp)                     # 4-byte Folded Spill
sw      t0, 136(sp)                     # 4-byte Folded Spill
sw      t1, 132(sp)                     # 4-byte Folded Spill
sw      t2, 128(sp)                     # 4-byte Folded Spill
sw      s0, 124(sp)                     # 4-byte Folded Spill
sw      s1, 120(sp)                     # 4-byte Folded Spill
sw      a0, 116(sp)                     # 4-byte Folded Spill
sw      a1, 112(sp)                     # 4-byte Folded Spill
sw      a2, 108(sp)                     # 4-byte Folded Spill
sw      a3, 104(sp)                     # 4-byte Folded Spill
sw      a4, 100(sp)                     # 4-byte Folded Spill
sw      a5, 96(sp)                      # 4-byte Folded Spill
sw      a6, 92(sp)                      # 4-byte Folded Spill
sw      a7, 88(sp)                      # 4-byte Folded Spill
sw      s2, 84(sp)                      # 4-byte Folded Spill
sw      s3, 80(sp)                      # 4-byte Folded Spill
sw      s4, 76(sp)                      # 4-byte Folded Spill
sw      s5, 72(sp)                      # 4-byte Folded Spill
sw      s6, 68(sp)                      # 4-byte Folded Spill
sw      s7, 64(sp)                      # 4-byte Folded Spill
sw      s8, 60(sp)                      # 4-byte Folded Spill
sw      s9, 56(sp)                      # 4-byte Folded Spill
sw      s10, 52(sp)                     # 4-byte Folded Spill
sw      s11, 48(sp)                     # 4-byte Folded Spill
sw      t3, 44(sp)                      # 4-byte Folded Spill
sw      t4, 40(sp)                      # 4-byte Folded Spill
sw      t5, 36(sp)                      # 4-byte Folded Spill
sw      t6, 32(sp)

if we don't realloc non-pushed registers in RV32IZcmp, it will generate code like

cm.push {ra, s0-s11}, -112
addi    sp, sp, -32
sw      t0, 136(sp)                     # 4-byte Folded Spill
sw      t1, 132(sp)                     # 4-byte Folded Spill
sw      t2, 128(sp)                     # 4-byte Folded Spill
sw      a0, 116(sp)                     # 4-byte Folded Spill
sw      a1, 112(sp)                     # 4-byte Folded Spill
sw      a2, 108(sp)                     # 4-byte Folded Spill
sw      a3, 104(sp)                     # 4-byte Folded Spill
sw      a4, 100(sp)                     # 4-byte Folded Spill
sw      a5, 96(sp)                      # 4-byte Folded Spill
sw      a6, 92(sp)                      # 4-byte Folded Spill
sw      a7, 88(sp)                      # 4-byte Folded Spill
sw      t3, 44(sp)                      # 4-byte Folded Spill
sw      t4, 40(sp)                      # 4-byte Folded Spill
sw      t5, 36(sp)                      # 4-byte Folded Spill
sw      t6, 32(sp)                      # 4-byte Folded Spill

meanwhile, accronding to Operation of cm.push defiend in spec

You may find both s0 and t0 will be saved to sp+136. It will cause problems.
So I think I have to re-alloc non-pushed registers. please correct me if I'm wrong or there is a better solution )

1472–1498

I think I have got what you mean, but can you tell me which function I should use to kill the callee-saved register?
Or where the LW/LD instruction does？

[KYG]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

257

Ah yes, you're right.
However, seems like it's only necessary when push/pop change any of the offset, which happens only in irq handler.
So maybe it's better to reallocPushStackFrame only for functions with interrupt attribute.

1472–1498

Normally we should add those registers to the "Defs" list in .td, but I don't know how to do this since the rlist is a compile time variable.
So I just add implicit def to the MI as a convenient workaround.

for (int x = 0; RegList[x] <= MaxReg && RegList[x] > 0; ++x)
  PopBuilder.addDef(RegList[x], RegState::Implicit);

[fakepaper56]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

405

Is it better to support Zcmp for RISCVRegisterInfo::hasReservedSpillSlot and use a more general name for getNonLibcallCSI.

llvm/lib/Target/RISCV/RISCVRegisterInfo.td

187

I suggest to add X26 to PGPR to make Zcmp work for spill register are {ra, s0 - 10}.

[fakepaper56]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

280

How about rewriting to Spimm = std::min(StackAdjust - StackAdjBase, 48); ?

300

Could the for loop be refined to any_of ?

1395

IMHO, is it possible the callee save register needed spilled are not successive, like {ra, s0, s3}?

[fakepaper56]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

257

Is the problem caused by not adjust RISCVRegisterInfo::hasReservedSpillSlot? The frame indexes of the wrong asm looks like that it uses the {ra, s0 - s11} frames.

[fakepaper56]

Should we provide the Zcmp stack size information into getFrameIndexReference and getFirstSPAdjustAmount like lib call ?

[craig.topper]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

229

Unexpected*

261

Can this be a const reference? I think that's what getCalleedSavedInfo returns

298

Why can't we called RISCV::PGPRRegClass.contains(Reg)?

414

RISCV::PGPRRegClass.contains(Reg)?

606

Use curly braces here since there is a long comment.

1390

Posh -> Push

1390

unsigned should be sufficient here right?

llvm/lib/Target/RISCV/RISCVMachineFunctionInfo.h

74

zcmp -> Zcmp

[VincentWu]

address comments
refactor Push/Pop reference to save-restore Libcall
better name for NonLibcallCSI

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

1395

I think it should not support this syntax.
In spec, it is valid only in the following cases:

1472–1498

is it RegState::Implicit? did you mean RegState::Kill ?

or I have add .setMIFlag(MachineInstr::FrameDestroy), does it useful for this point?

[VincentWu]

fix compile error

[fakepaper56]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

1395

My idea is your code may raise problem when we need to spill s3 but not spill s1 and s2. The stack size you calculated is register number is 3 but indeed you spill 5 register.

1395

you could use else if (const char *SpillLibCall = getSpillLibCallName(*MF, CSI)) here.

[fakepaper56]

llvm/lib/Target/RISCV/RISCVRegisterInfo.td

189–190

Fold the two lines to (sequence "X%u", 18, 27) ?

[fakepaper56]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

1007

Support getRVPushStackSize() here.

1016

Support getRVPushStackSize() here.

1349

Support getRVPushStackSize() here?

1383

Maybe align the size for accurate stack size. And it can help you not to align for getRVPushStackSize().

[KYG]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

1472–1498

It should be RegState::ImplicitDefine or RegState::Define, since CM_POP is defining new values to registers rather than using their values (RegState::Kill means the last use of a register).
I don't think .setMIFlag(MachineInstr::FrameDestroy) is enough for this purpose, it doesn't contain information about which registers are being assigned.

[VincentWu]

address comments

[fakepaper56]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

1472–1498

Why not use ImplicitDefine here? I think using ImplicitDefine can avoid using variadic operand for CM_PUSH and CM_POP.

[fakepaper56]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

1393

Should we implicit use spill register like CM.POP?

1472–1498

I am sorry that I forgot to mention you this idea.

[VincentWu]

use ImplicitDefine and set Implicit for push

[VincentWu]

use PushPopRegs directly

[fakepaper56]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

1400

CM.PUSH does "use" the callee save. I think you should use addUse instead of addDef.

[VincentWu]

address comments

[fakepaper56]

LGTM. But wait for other reviewers' approve.

[craig.topper]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

230

Put the register name comments at the end of the line instead of in the middle

276

Can we count PushPopRegs as we go through the loop and only have a special case for RISCV::X26 after the loop?

277

Can we replace this with

MaxPushPopReg = RISCV::X27;
break;

to avoid a return when every other case breaks.

483

Why is this blank line removed?

1377–1378

Emmit -> Emit

1483

This list is repeated in two places. Should it be a global variable?

[VincentWu]

address comments

[craig.topper]

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

226

Why does this return`int`? Aren't all the enum values positive?

[VincentWu]

address comments

[craig.topper]

LGTM

[VincentWu]

rebase & update test case

[VincentWu]

In D134599#4454410, @craig.topper wrote:

LGTM

the status of this patch is sitll Needs Review.
should I land this patch?
or waiting for @jrtc27 comments?

llvm/lib/Target/RISCV/CMakeLists.txt

add_llvm_target(RISCVCodeGen
   RISCVISelDAGToDAG.cpp
   RISCVISelLowering.cpp
   RISCVMachineFunctionInfo.cpp
   RISCVMacroFusion.cpp
   RISCVMCInstLower.cpp
   RISCVMergeBaseOffset.cpp
   RISCVOptWInstrs.cpp
   RISCVRedundantCopyElimination.cpp
+  RISCVMoveOptimizer.cpp
+  RISCVPushPopOptimizer.cpp
   RISCVRegisterInfo.cpp
   RISCVRVVInitUndef.cpp
   RISCVSubtarget.cpp
   RISCVTargetMachine.cpp
   RISCVTargetObjectFile.cpp
   RISCVTargetTransformInfo.cpp
   GISel/RISCVCallLowering.cpp
   GISel/RISCVInstructionSelector.cpp

llvm/lib/Target/RISCV/RISCV.h

 
 FunctionPass *createRISCVRedundantCopyEliminationPass();
 void initializeRISCVRedundantCopyEliminationPass(PassRegistry &);
 
 FunctionPass *createRISCVInitUndefPass();
 void initializeRISCVInitUndefPass(PassRegistry &);
 extern char &RISCVInitUndefID;
 
+FunctionPass *createRISCVMoveOptimizationPass();
+void initializeRISCVMoveOptPass(PassRegistry &);
+
+FunctionPass *createRISCVPushPopOptimizationPass();
+void initializeRISCVPushPopOptPass(PassRegistry &);
+
 InstructionSelector *createRISCVInstructionSelector(const RISCVTargetMachine &,
                                                     RISCVSubtarget &,
                                                     RISCVRegisterBankInfo &);
 void initializeRISCVDAGToDAGISelPass(PassRegistry &);
 } // namespace llvm
 
 #endif

llvm/lib/Target/RISCV/RISCVFrameLowering.h

public:
 
   void processFunctionBeforeFrameFinalized(MachineFunction &MF,
                                            RegScavenger *RS) const override;
 
   bool hasFP(const MachineFunction &MF) const override;
 
   bool hasBP(const MachineFunction &MF) const;
 
+  bool isCSIpushable(const MachineFunction &MF,
+                     const std::vector<CalleeSavedInfo> &CSI) const;
+
   bool hasReservedCallFrame(const MachineFunction &MF) const override;
   MachineBasicBlock::iterator
   eliminateCallFramePseudoInstr(MachineFunction &MF, MachineBasicBlock &MBB,
                                 MachineBasicBlock::iterator MI) const override;
   bool spillCalleeSavedRegisters(MachineBasicBlock &MBB,
                                  MachineBasicBlock::iterator MI,
                                  ArrayRef<CalleeSavedInfo> CSI,
                                  const TargetRegisterInfo *TRI) const override;

llvm/lib/Target/RISCV/RISCVFrameLowering.cpp

getRestoreLibCallName(const MachineFunction &MF,
   };
 
   int LibCallID = getLibCallID(MF, CSI);
   if (LibCallID == -1)
     return nullptr;
   return RestoreLibCalls[LibCallID];
 }
 
+// Return encoded value for PUSH/POP instruction, representing
+// registers to store/load.
+static int getPushPopEncoding(const Register MaxReg) {
+  switch (MaxReg) {
+  default:
+    llvm_unreachable("Unexcetped Reg for Push/Pop Inst");
+  case /*s11*/ RISCV::X27:
+    return llvm::RISCVZC::RLISTENCODE::RA_S0_S11;
+  case /*s9*/ RISCV::X25:
+    return llvm::RISCVZC::RLISTENCODE::RA_S0_S9;
+  case /*s8*/ RISCV::X24:
+    return llvm::RISCVZC::RLISTENCODE::RA_S0_S8;
+  case /*s7*/ RISCV::X23:
+    return llvm::RISCVZC::RLISTENCODE::RA_S0_S7;
+  case /*s6*/ RISCV::X22:
+    return llvm::RISCVZC::RLISTENCODE::RA_S0_S6;
+  case /*s5*/ RISCV::X21:
+    return llvm::RISCVZC::RLISTENCODE::RA_S0_S5;
+  case /*s4*/ RISCV::X20:
+    return llvm::RISCVZC::RLISTENCODE::RA_S0_S4;
+  case /*s3*/ RISCV::X19:
+    return llvm::RISCVZC::RLISTENCODE::RA_S0_S3;
+  case /*s2*/ RISCV::X18:
+    return llvm::RISCVZC::RLISTENCODE::RA_S0_S2;
+  case /*s1*/ RISCV::X9:
+    return llvm::RISCVZC::RLISTENCODE::RA_S0_S1;
+  case /*s0*/ RISCV::X8:
+    return llvm::RISCVZC::RLISTENCODE::RA_S0;
+  case /*ra*/ RISCV::X1:
+    return llvm::RISCVZC::RLISTENCODE::RA;
+  }
+}
+
+void reallocPushStackFream(MachineFunction &MF) {
+  auto *RVFI = MF.getInfo<RISCVMachineFunctionInfo>();
+  MachineFrameInfo &MFI = MF.getFrameInfo();
+
+  std::vector<CalleeSavedInfo> CSI = MFI.getCalleeSavedInfo();
+  // realloc stack frame for PUSH
+  size_t NonePushStackOffset = -RVFI->getRVPushStackSize();
+  for (const auto &Entry : CSI) {
+    int FrameIdx = Entry.getFrameIdx();
+    Register Reg = Entry.getReg();
+    if (!(Reg == RISCV::X26 || RISCV::PGPRRegClass.contains(Reg))) {
+      NonePushStackOffset -= MFI.getObjectSize(Entry.getFrameIdx());
+      MFI.setObjectOffset(FrameIdx, NonePushStackOffset);
+    }
+  }
+}
+
+static uint64_t adjSPInPushPop(MachineBasicBlock::iterator MBBI,
+                               uint64_t StackAdj, bool isPop) {
+  // The spec allocates 2 bits to specify number of extra 16 byte blocks.
+  uint32_t AvailableAdj = 48;
+  uint64_t RequiredAdj = StackAdj;
+
+  // Use available stack adjustment in Zc PUSH/POP instruction
+  // to allocate/deallocate space on stack.
+  int OpNum = MBBI->getNumOperands();
+  auto &Operand = MBBI->getOperand(OpNum - 1);
+  int RegisterOffset = Operand.getImm();
+  RequiredAdj -= RegisterOffset;
+
+  if (RequiredAdj >= AvailableAdj) {
+    RequiredAdj -= AvailableAdj;
+    StackAdj = AvailableAdj;
+  } else {
+    // Round to the nearest 16 byte block able to fit RequiredAdj.
+    StackAdj = alignTo(RequiredAdj, 16);
+    RequiredAdj = 0;
+  }
+  Operand.setImm(StackAdj);
+  MBBI->setFlag(isPop ? MachineInstr::FrameDestroy : MachineInstr::FrameSetup);
+  return RequiredAdj;
+}
+
+// Checks if Zc PUSH/POP instructions can be used with the given CSI.
+bool RISCVFrameLowering::isCSIpushable(
+    const MachineFunction &MF, const std::vector<CalleeSavedInfo> &CSI) const {
+  if (!STI.hasStdExtZcmp() || CSI.empty() ||
+      MF.getTarget().Options.DisableFramePointerElim(MF))
+    return false;
+  for (auto &CS : CSI) {
+    Register Reg = CS.getReg();
+    const TargetRegisterClass *RC =
+        STI.getRegisterInfo()->getMinimalPhysRegClass(Reg);
+    if (RISCV::PGPRRegClass.hasSubClassEq(RC))
+      return true;
+  }
+  return false;
+}
+
 // Return true if the specified function should have a dedicated frame
 // pointer register.  This is true if frame pointer elimination is
 // disabled, if it needs dynamic stack realignment, if the function has
 // variable sized allocas, or if the frame address is taken.
 bool RISCVFrameLowering::hasFP(const MachineFunction &MF) const {
   const TargetRegisterInfo *RegInfo = MF.getSubtarget().getRegisterInfo();
 
   const MachineFrameInfo &MFI = MF.getFrameInfo();

 
 // Returns the register used to hold the stack pointer.
 static Register getSPReg(const RISCVSubtarget &STI) { return RISCV::X2; }
 
 static SmallVector<CalleeSavedInfo, 8>
 getNonLibcallCSI(const MachineFunction &MF,
                  const std::vector<CalleeSavedInfo> &CSI) {
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   SmallVector<CalleeSavedInfo, 8> NonLibcallCSI;
 
   for (auto &CS : CSI) {
     int FI = CS.getFrameIdx();
     if (FI >= 0 && MFI.getStackID(FI) == TargetStackID::Default)
       NonLibcallCSI.push_back(CS);
   }
 
   return NonLibcallCSI;
 }
 
 void RISCVFrameLowering::adjustStackForRVV(MachineFunction &MF,
                                            MachineBasicBlock &MBB,
                                            MachineBasicBlock::iterator MBBI,
                                            const DebugLoc &DL, int64_t Amount,
                                            MachineInstr::MIFlag Flag) const {
   assert(Amount != 0 && "Did not need to adjust stack pointer for RVV.");
 
   const Register SPReg = getSPReg(STI);
 
   // Optimize compile time offset case
   StackOffset Offset = StackOffset::getScalable(Amount);
   if (STI.getRealMinVLen() == STI.getRealMaxVLen()) {
     // 1. Multiply the number of v-slots by the (constant) length of register
     const int64_t VLENB = STI.getRealMinVLen() / 8;
     assert(Amount % 8 == 0 &&
            "Reserve the stack by the multiple of one vector size.");
     const int64_t NumOfVReg = Amount / 8;
     const int64_t FixedOffset = NumOfVReg * VLENB;
     if (!isInt<32>(FixedOffset)) {
void RISCVFrameLowering::emitPrologue(MachineFunction &MF,
 
   uint64_t FirstSPAdjustAmount = getFirstSPAdjustAmount(MF);
   // Split the SP adjustment to reduce the offsets of callee saved spill.
   if (FirstSPAdjustAmount) {
     StackSize = FirstSPAdjustAmount;
     RealStackSize = FirstSPAdjustAmount;
   }
 
-  // Allocate space on the stack if necessary.
-  RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackOffset::getFixed(-StackSize),
+  const auto &CSI = MFI.getCalleeSavedInfo();
+  bool PushEnabled = isCSIpushable(MF, CSI);
+  if (PushEnabled && (CSI.size() != 0)) {
+    // Check at what offset spilling of registers starts and allocate space
+    // before it.
+    int64_t preAdjustStack = 0;
+    for (auto CS : CSI) {
+      preAdjustStack =
+          std::min(preAdjustStack, -(MFI.getObjectOffset(CS.getFrameIdx()) +
+                                     MFI.getObjectSize(CS.getFrameIdx())));
+    }
+    if (preAdjustStack != 0)
+      RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg,
+                    StackOffset::getFixed(-preAdjustStack),
                     MachineInstr::FrameSetup, getStackAlign());
+    StackSize -= preAdjustStack;
+
+    // Use available stack adjustment in push instruction to allocate additional
+    // stack space.
+    StackSize = adjSPInPushPop(MBBI, StackSize, false);
+    if (StackSize != 0) {
+      RI->adjustReg(MBB, next_nodbg(MBBI, MBB.end()), DL, SPReg, SPReg,
+                    StackOffset::getFixed(-StackSize), MachineInstr::FrameSetup,
+                    getStackAlign());
+      MBBI = next_nodbg(MBBI, MBB.end());
+      reallocPushStackFream(MF);
+    }
+  } else {
+    // Allocate space on the stack if necessary.
+    RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg,
+                  StackOffset::getFixed(-StackSize), MachineInstr::FrameSetup,
+                  getStackAlign());
+  }
 
   // Emit ".cfi_def_cfa_offset RealStackSize"
   unsigned CFIIndex = MF.addFrameInst(
       MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize));
   BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
       .addCFIIndex(CFIIndex)
       .setMIFlag(MachineInstr::FrameSetup);
 
-  const auto &CSI = MFI.getCalleeSavedInfo();
-
+  if (PushEnabled)
+    std::advance(MBBI, 1);
+  else
     // The frame pointer is callee-saved, and code has been generated for us to
     // save it to the stack. We need to skip over the storing of callee-saved
     // registers as the frame pointer must be modified after it has been saved
     // to the stack, not before.
     // FIXME: assumes exactly one instruction is used to save each callee-saved
     // register.
     std::advance(MBBI, getNonLibcallCSI(MF, CSI).size());
 
   // Iterate over list of callee-saved registers and emit .cfi_offset
   // directives.
   for (const auto &Entry : CSI) {
     int FrameIdx = Entry.getFrameIdx();
     int64_t Offset;
     // Offsets for objects with fixed locations (IE: those saved by libcall) are
     // simply calculated from the frame index.
void RISCVFrameLowering::emitEpilogue(MachineFunction &MF,
   }
 
   const auto &CSI = getNonLibcallCSI(MF, MFI.getCalleeSavedInfo());
 
   // Skip to before the restores of callee-saved registers
   // FIXME: assumes exactly one instruction is used to restore each
   // callee-saved register.
   auto LastFrameDestroy = MBBI;
-  if (!CSI.empty())
+  bool PopEnabled = isCSIpushable(
+      MF, std::vector<llvm::CalleeSavedInfo>(CSI.begin(), CSI.end()));
+  if (PopEnabled)
+    LastFrameDestroy = prev_nodbg(MBBI, MBB.begin());
+  else if (!CSI.empty())
     LastFrameDestroy = std::prev(MBBI, CSI.size());
 
   uint64_t StackSize = getStackSizeWithRVVPadding(MF);
   uint64_t RealStackSize = StackSize + RVFI->getLibCallStackSize();
   uint64_t FPOffset = RealStackSize - RVFI->getVarArgsSaveSize();
   uint64_t RVVStackSize = RVFI->getRVVStackSize();
 
   // Restore the stack pointer using the value of the frame pointer. Only
RI->adjustReg(MBB, LastFrameDestroy, DL, SPReg, SPReg,
                   StackOffset::getFixed(SecondSPAdjustAmount),
                   MachineInstr::FrameDestroy, getStackAlign());
   }
 
   if (FirstSPAdjustAmount)
     StackSize = FirstSPAdjustAmount;
 
   // Deallocate stack
+  if (PopEnabled) {
+    // Check at what offset spilling of registers starts and calculate space
+    // before it.
+    int64_t preAdjustSize = 0;
+    for (auto CS : CSI) {
+      preAdjustSize =
+          std::min(preAdjustSize, -(MFI.getObjectOffset(CS.getFrameIdx()) +
+                                    MFI.getObjectSize(CS.getFrameIdx())));
+    }
+    RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg,
+                  StackOffset::getFixed(preAdjustSize),
+                  MachineInstr::FrameDestroy, getStackAlign());
+    StackSize -= preAdjustSize;
+    if (preAdjustSize != 0)
+      MBBI = prev_nodbg(MBBI, MBB.begin());
+
+    // Use available stack adjustment in pop instruction to deallocate stack
+    // space.
+    StackSize = adjSPInPushPop(prev_nodbg(MBBI, MBB.begin()), StackSize, true);
+    if (StackSize != 0) {
+      RI->adjustReg(MBB, prev_nodbg(MBBI, MBB.begin()), DL, SPReg, SPReg,
+                    StackOffset::getFixed(StackSize),
+                    MachineInstr::FrameDestroy, getStackAlign());
+    }
+  } else
     RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackOffset::getFixed(StackSize),
                   MachineInstr::FrameDestroy, getStackAlign());
 
   // Emit epilogue for shadow call stack.
   emitSCSEpilogue(MF, MBB, MBBI, DL);
 }
 
 StackOffset
 RISCVFrameLowering::getFrameIndexReference(const MachineFunction &MF, int FI,
                                            Register &FrameReg) const {
RISCVFrameLowering::getFrameIndexReference(const MachineFunction &MF, int FI,
   // The total amount of padding surrounding RVV objects is described by
   // RVV->getRVVPadding() and it can be zero. It allows us to align the RVV
   // objects to the required alignment.
   if (MFI.getStackID(FI) == TargetStackID::Default) {
     if (MFI.isFixedObjectIndex(FI)) {
       assert(!RI->hasStackRealignment(MF) &&
              "Can't index across variable sized realign");
       Offset += StackOffset::get(getStackSizeWithRVVPadding(MF) +
                                  RVFI->getLibCallStackSize(),
                                  RVFI->getRVVStackSize());
     } else {
       Offset += StackOffset::getFixed(MFI.getStackSize());
     }
   } else if (MFI.getStackID(FI) == TargetStackID::ScalableVector) {
     // Ensure the base of the RVV stack is correctly aligned: add on the
     // alignment padding.
     int ScalarLocalVarSize =
       MFI.getStackSize() - RVFI->getCalleeSavedStackSize() -
       RVFI->getVarArgsSaveSize() + RVFI->getRVVPadding();
     Offset += StackOffset::get(ScalarLocalVarSize, RVFI->getRVVStackSize());
   }
   return Offset;
 }
 
 void RISCVFrameLowering::determineCalleeSaves(MachineFunction &MF,
                                               BitVector &SavedRegs,
RISCVFrameLowering::getFirstSPAdjustAmount(const MachineFunction &MF) const {
   const auto *RVFI = MF.getInfo<RISCVMachineFunctionInfo>();
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   const std::vector<CalleeSavedInfo> &CSI = MFI.getCalleeSavedInfo();
   uint64_t StackSize = getStackSizeWithRVVPadding(MF);
 
   // Disable SplitSPAdjust if save-restore libcall is used. The callee-saved
   // registers will be pushed by the save-restore libcalls, so we don't have to
   // split the SP adjustment in this case.
   if (RVFI->getLibCallStackSize())
     return 0;
 
   // Return the FirstSPAdjustAmount if the StackSize can not fit in a signed
   // 12-bit and there exists a callee-saved register needing to be pushed.
   if (!isInt<12>(StackSize) && (CSI.size() > 0)) {
     // FirstSPAdjustAmount is chosen as (2048 - StackAlign) because 2048 will
     // cause sp = sp + 2048 in the epilogue to be split into multiple
     // instructions. Offsets smaller than 2048 can fit in a single load/store
bool RISCVFrameLowering::spillCalleeSavedRegisters(
   if (CSI.empty())
     return true;
 
   MachineFunction *MF = MBB.getParent();
   const TargetInstrInfo &TII = *MF->getSubtarget().getInstrInfo();
   DebugLoc DL;
   if (MI != MBB.end() && !MI->isDebugInstr())
     DL = MI->getDebugLoc();
 
+  // Emmit CM.PUSH with base SPimm & evaluate Push stack
+  if (isCSIpushable(*MF, CSI.vec())) {
+    auto *RVFI = MF->getInfo<RISCVMachineFunctionInfo>();
+    uint64_t PushStackSize = 0;
+    std::vector<CalleeSavedInfo> NonePushCSI;
+    Register MaxReg = RISCV::NoRegister;
+
+    for (auto &CS : CSI) {
+      Register Reg = CS.getReg();
+      const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(Reg);
+      if (RISCV::PGPRRegClass.hasSubClassEq(RC)) {
+        if (Reg != RISCV::X27)
+          PushStackSize += 4;
+        MaxReg = std::max(MaxReg.id(), Reg.id());
+      } else if (Reg.id() == RISCV::X26) {
+        PushStackSize += 8;
+        MaxReg = RISCV::X27;
+      } else
+        NonePushCSI.push_back(CS);
+    }
+    RVFI->setRVPushStackSize(PushStackSize);
+
+    MachineInstrBuilder PushBuilder =
+        BuildMI(MBB, MI, DL, TII.get(RISCV::CM_PUSH));
+    // Use encoded number to represent registers to spill.
+    int RegEnc = getPushPopEncoding(MaxReg);
+    PushBuilder.addImm(RegEnc);
+    // Calculate SpImm Base adjustment, and SpImm field will be updated
+    // through adjSPInPushPop.
+    bool isRV64 = STI.getFeatureBits()[RISCV::Feature64Bit];
+    bool isEABI = false; // Reserved for future implementation
+    uint32_t SpImmBase = RISCVZC::getStackAdjBase(RegEnc, isRV64, isEABI);
+    PushBuilder.addImm(SpImmBase);
+
+    for (auto &CS : NonePushCSI) {
+      Register Reg = CS.getReg();
+      TII.storeRegToStackSlot(MBB, MI, Reg, true, CS.getFrameIdx(),
+                              TRI->getMinimalPhysRegClass(Reg), TRI,
+                              Register());
+    }
+  } else {
     const char *SpillLibCall = getSpillLibCallName(*MF, CSI);
     if (SpillLibCall) {
       // Add spill libcall via non-callee-saved register t0.
       BuildMI(MBB, MI, DL, TII.get(RISCV::PseudoCALLReg), RISCV::X5)
           .addExternalSymbol(SpillLibCall, RISCVII::MO_CALL)
           .setMIFlag(MachineInstr::FrameSetup);
 
       // Add registers spilled in libcall as liveins.
       for (auto &CS : CSI)
         MBB.addLiveIn(CS.getReg());
     }
 
     // Manually spill values not spilled by libcall.
     const auto &NonLibcallCSI = getNonLibcallCSI(*MF, CSI);
     for (auto &CS : NonLibcallCSI) {
       // Insert the spill to the stack frame.
       Register Reg = CS.getReg();
       const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(Reg);
-    TII.storeRegToStackSlot(MBB, MI, Reg, !MBB.isLiveIn(Reg), CS.getFrameIdx(),
-                            RC, TRI, Register());
+      TII.storeRegToStackSlot(MBB, MI, Reg, !MBB.isLiveIn(Reg),
+                              CS.getFrameIdx(), RC, TRI, Register());
+    }
   }
 
   return true;
 }
 
 bool RISCVFrameLowering::restoreCalleeSavedRegisters(
     MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
     MutableArrayRef<CalleeSavedInfo> CSI, const TargetRegisterInfo *TRI) const {
   if (CSI.empty())
     return true;
 
   MachineFunction *MF = MBB.getParent();
   const TargetInstrInfo &TII = *MF->getSubtarget().getInstrInfo();
   DebugLoc DL;
   if (MI != MBB.end() && !MI->isDebugInstr())
     DL = MI->getDebugLoc();
 
+  if (isCSIpushable(*MF, CSI.vec())) {
+    Register MaxReg = RISCV::NoRegister;
+
+    for (auto &CS : reverse(CSI)) {
+      Register Reg = CS.getReg();
+      const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(Reg);
+      if (RISCV::PGPRRegClass.hasSubClassEq(RC))
+        MaxReg = std::max(MaxReg.id(), Reg.id());
+      else if (Reg.id() == RISCV::X26) {
+        MaxReg = RISCV::X27;
+      } else
+        TII.loadRegFromStackSlot(MBB, MI, Reg, CS.getFrameIdx(), RC, TRI,
+                                 Register());
+    }
+
+    MachineInstrBuilder PopBuilder =
+        BuildMI(MBB, MI, DL, TII.get(RISCV::CM_POP));
+    // Use encoded number to represent registers to restore.
+    int RegEnc = getPushPopEncoding(MaxReg);
+    PopBuilder.addImm(RegEnc);
+    // Calculate SpImm Base adjustment, and SpImm field will be updated
+    // through adjSPInPushPop.
+    bool isRV64 = STI.getFeatureBits()[RISCV::Feature64Bit];
+    bool isEABI = false; // Reserved for future implementation
+    uint32_t SpImmBase = RISCVZC::getStackAdjBase(RegEnc, isRV64, isEABI);
+    PopBuilder.addImm(SpImmBase);
+  } else {
     // Manually restore values not restored by libcall.
     // Keep the same order as in the prologue. There is no need to reverse the
     // order in the epilogue. In addition, the return address will be restored
     // first in the epilogue. It increases the opportunity to avoid the
     // load-to-use data hazard between loading RA and return by RA.
     // loadRegFromStackSlot can insert multiple instructions.
     const auto &NonLibcallCSI = getNonLibcallCSI(*MF, CSI);
     for (auto &CS : NonLibcallCSI) {
       Register Reg = CS.getReg();
       const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(Reg);
       TII.loadRegFromStackSlot(MBB, MI, Reg, CS.getFrameIdx(), RC, TRI,
                                Register());
-    assert(MI != MBB.begin() && "loadRegFromStackSlot didn't insert any code!");
+      assert(MI != MBB.begin() &&
+             "loadRegFromStackSlot didn't insert any code!");
     }
 
     const char *RestoreLibCall = getRestoreLibCallName(*MF, CSI);
     if (RestoreLibCall) {
       // Add restore libcall via tail call.
       MachineBasicBlock::iterator NewMI =
           BuildMI(MBB, MI, DL, TII.get(RISCV::PseudoTAIL))
               .addExternalSymbol(RestoreLibCall, RISCVII::MO_CALL)
               .setMIFlag(MachineInstr::FrameDestroy);
 
       // Remove trailing returns, since the terminator is now a tail call to the
       // restore function.
       if (MI != MBB.end() && MI->getOpcode() == RISCV::PseudoRET) {
         NewMI->copyImplicitOps(*MF, *MI);
         MI->eraseFromParent();
       }
     }
-
+  }
   return true;
 }
 
 bool RISCVFrameLowering::enableShrinkWrapping(const MachineFunction &MF) const {
   // Keep the conventional code flow when not optimizing.
   if (MF.getFunction().hasOptNone())
     return false;
 

llvm/lib/Target/RISCV/RISCVInstrInfo.h

bool analyzeSelect(const MachineInstr &MI,
                      unsigned &FalseOp, bool &Optimizable) const override;
 
   MachineInstr *optimizeSelect(MachineInstr &MI,
                                SmallPtrSetImpl<MachineInstr *> &SeenMIs,
                                bool) const override;
 
   bool isAsCheapAsAMove(const MachineInstr &MI) const override;
 
+  std::optional<DestSourcePair> isLoadImmImpl(const MachineInstr &MI) const;
+
   std::optional<DestSourcePair>
   isCopyInstrImpl(const MachineInstr &MI) const override;
 
   bool verifyInstruction(const MachineInstr &MI,
                          StringRef &ErrInfo) const override;
 
   bool getMemOperandWithOffsetWidth(const MachineInstr &LdSt,
                                     const MachineOperand *&BaseOp,

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

case RISCV::XORI:
     return (MI.getOperand(1).isReg() &&
             MI.getOperand(1).getReg() == RISCV::X0) ||
            (MI.getOperand(2).isImm() && MI.getOperand(2).getImm() == 0);
   }
   return MI.isAsCheapAsAMove();
 }
 
 std::optional<DestSourcePair>
+RISCVInstrInfo::isLoadImmImpl(const MachineInstr &MI) const {
+  if (MI.isMoveImmediate())
+    return DestSourcePair{MI.getOperand(0), MI.getOperand(2)};
+  switch (MI.getOpcode()) {
+  default:
+    break;
+  case RISCV::ADDIW:
+  case RISCV::ADDI:
+    // Operand 1 can be a frameindex but callers expect registers
+    if (MI.getOperand(1).isReg() && MI.getOperand(2).isImm() &&
+        MI.getOperand(1).getReg() == RISCV::X0)
+      return DestSourcePair{MI.getOperand(0), MI.getOperand(2)};
+    break;
+  }
+  return std::nullopt;
+}
+
+std::optional<DestSourcePair>
 RISCVInstrInfo::isCopyInstrImpl(const MachineInstr &MI) const {
   if (MI.isMoveReg())
     return DestSourcePair{MI.getOperand(0), MI.getOperand(1)};
   switch (MI.getOpcode()) {
   default:
     break;
+  case RISCV::ADD:
+    if (!MI.getOperand(1).isReg() || !MI.getOperand(2).isReg())
+      break;
+    if ((MI.getOperand(1).getReg() == RISCV::X0) &&
+        (MI.getOperand(2).getReg() != RISCV::X0))
+      return DestSourcePair{MI.getOperand(0), MI.getOperand(2)};
+    if ((MI.getOperand(1).getReg() != RISCV::X0) &&
+        (MI.getOperand(2).getReg() == RISCV::X0))
+      return DestSourcePair{MI.getOperand(0), MI.getOperand(1)};
+    break;
   case RISCV::ADDI:
     // Operand 1 can be a frameindex but callers expect registers
     if (MI.getOperand(1).isReg() && MI.getOperand(2).isImm() &&
         MI.getOperand(2).getImm() == 0)
       return DestSourcePair{MI.getOperand(0), MI.getOperand(1)};
     break;
   case RISCV::FSGNJ_D:
   case RISCV::FSGNJ_S:

llvm/lib/Target/RISCV/RISCVMachineFunctionInfo.h

private:
   /// Size of stack frame to save callee saved registers
   unsigned CalleeSavedStackSize = 0;
   /// Is there any vector argument or return?
   bool IsVectorCall = false;
 
   /// Registers that have been sign extended from i32.
   SmallVector<Register, 8> SExt32Registers;
 
+  /// Size of stack frame for zcmp PUSH/POP
+  unsigned RVPushStackSize = 0;
+
 public:
   RISCVMachineFunctionInfo(const Function &F, const TargetSubtargetInfo *STI) {}
 
   MachineFunctionInfo *
   clone(BumpPtrAllocator &Allocator, MachineFunction &DestMF,
         const DenseMap<MachineBasicBlock *, MachineBasicBlock *> &Src2DstMBB)
       const override;
 
public:
   unsigned getLibCallStackSize() const { return LibCallStackSize; }
   void setLibCallStackSize(unsigned Size) { LibCallStackSize = Size; }
 
   bool useSaveRestoreLibCalls(const MachineFunction &MF) const {
     // We cannot use fixed locations for the callee saved spill slots if the
     // function uses a varargs save area, or is an interrupt handler.
     return MF.getSubtarget<RISCVSubtarget>().enableSaveRestore() &&
            VarArgsSaveSize == 0 && !MF.getFrameInfo().hasTailCall() &&
-           !MF.getFunction().hasFnAttribute("interrupt");
+           !MF.getFunction().hasFnAttribute("interrupt") &&
+           !MF.getSubtarget<RISCVSubtarget>().hasStdExtZcmp();
   }
 
   uint64_t getRVVStackSize() const { return RVVStackSize; }
   void setRVVStackSize(uint64_t Size) { RVVStackSize = Size; }
 
   Align getRVVStackAlign() const { return RVVStackAlign; }
   void setRVVStackAlign(Align StackAlign) { RVVStackAlign = StackAlign; }
 
   uint64_t getRVVPadding() const { return RVVPadding; }
   void setRVVPadding(uint64_t Padding) { RVVPadding = Padding; }
 
   unsigned getCalleeSavedStackSize() const { return CalleeSavedStackSize; }
   void setCalleeSavedStackSize(unsigned Size) { CalleeSavedStackSize = Size; }
 
+  uint64_t getRVPushStackSize() const { return RVPushStackSize; }
+  void setRVPushStackSize(uint64_t Size) { RVPushStackSize = Size; }
+
   void initializeBaseYamlFields(const yaml::RISCVMachineFunctionInfo &YamlMFI);
 
   void addSExt32Register(Register Reg);
   bool isSExt32Register(Register Reg) const;
 
   bool isVectorCall() const { return IsVectorCall; }
   void setIsVectorCall() { IsVectorCall = true; }
 };
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_RISCV_RISCVMACHINEFUNCTIONINFO_H

llvm/lib/Target/RISCV/RISCVMoveOptimizer.cpp

+//===---------- RISCVMoveOptimizer.cpp - RISCV move opt. pass -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains a pass that performs move related peephole
+// optimizations. This pass should be run after register allocation.
+//
+//===----------------------------------------------------------------------===//
+
+#include "RISCVInstrInfo.h"
+#include "RISCVMachineFunctionInfo.h"
+
+using namespace llvm;
+
+#define RISCV_MOVE_OPT_NAME "RISC-V Zc move merging pass"
+
+namespace {
+struct RISCVMoveOpt : public MachineFunctionPass {
+  static char ID;
+
+  RISCVMoveOpt() : MachineFunctionPass(ID) {
+    initializeRISCVMoveOptPass(*PassRegistry::getPassRegistry());
+  }
+
+  const RISCVInstrInfo *TII;
+  const TargetRegisterInfo *TRI;
+  const RISCVSubtarget *Subtarget;
+
+  // Track which register units have been modified and used.
+  LiveRegUnits ModifiedRegUnits, UsedRegUnits;
+
+  bool isCandidateToMergeMVA01S(DestSourcePair &RegPair);
+  bool isCandidateToMergeMVSA01(DestSourcePair &RegPair);
+  // Merge the two instructions indicated into a single pair instruction.
+  MachineBasicBlock::iterator
+  mergePairedInsns(MachineBasicBlock::iterator I,
+                   MachineBasicBlock::iterator Paired, unsigned Opcode);
+
+  // Look for C.MV instruction that can be combined with
+  // the given instruction into CM.MVA01S or CM.MVSA01. Return the matching
+  // instruction if one exists.
+  MachineBasicBlock::iterator
+  findMatchingInst(MachineBasicBlock::iterator &MBBI, unsigned InstOpcode);
+  bool MovOpt(MachineBasicBlock &MBB);
+  bool runOnMachineFunction(MachineFunction &Fn) override;
+
+  StringRef getPassName() const override { return RISCV_MOVE_OPT_NAME; }
+};
+
+char RISCVMoveOpt::ID = 0;
+
+} // end of anonymous namespace
+
+INITIALIZE_PASS(RISCVMoveOpt, "riscv-move-opt", RISCV_MOVE_OPT_NAME, false,
+                false)
+
+// Check if registers meet CM.MVA01S constraints.
+bool RISCVMoveOpt::isCandidateToMergeMVA01S(DestSourcePair &RegPair) {
+  Register Destination = RegPair.Destination->getReg();
+  Register Source = RegPair.Source->getReg();
+  const TargetRegisterClass *SourceRC = TRI->getMinimalPhysRegClass(Source);
+  // If destination is not a0 or a1.
+  if (Destination == RISCV::X10 || Destination == RISCV::X11)
+    if (RISCV::SR07RegClass.hasSubClassEq(SourceRC))
+      return true;
+  return false;
+}
+
+// Check if registers meet CM.MVSA01 constraints.
+bool RISCVMoveOpt::isCandidateToMergeMVSA01(DestSourcePair &RegPair) {
+  Register Destination = RegPair.Destination->getReg();
+  Register Source = RegPair.Source->getReg();
+  const TargetRegisterClass *DestinationRC =
+      TRI->getMinimalPhysRegClass(Destination);
+  // If Source is s0 - s7.
+  if (RISCV::SR07RegClass.hasSubClassEq(DestinationRC))
+    if (Source == RISCV::X10 || Source == RISCV::X11)
+      return true;
+  return false;
+}
+
+MachineBasicBlock::iterator
+RISCVMoveOpt::mergePairedInsns(MachineBasicBlock::iterator I,
+                               MachineBasicBlock::iterator Paired,
+                               unsigned Opcode) {
+  const MachineOperand *Sreg1, *Sreg2;
+  MachineBasicBlock::iterator E = I->getParent()->end();
+  MachineBasicBlock::iterator NextI = next_nodbg(I, E);
+  DestSourcePair FirstPair = TII->isCopyInstrImpl(*I).value();
+  DestSourcePair PairedRegs = TII->isCopyInstrImpl(*Paired).value();
+  Register ARegInFirstPair = Opcode == RISCV::CM_MVA01S
+                                 ? FirstPair.Destination->getReg()
+                                 : FirstPair.Source->getReg();
+
+  if (NextI == Paired)
+    NextI = next_nodbg(NextI, E);
+  DebugLoc DL = I->getDebugLoc();
+
+  // The order of S-reg depends on which instruction holds A0, instead of
+  // the order of register pair.
+  // e,g.
+  //   mv a1, s1
+  //   mv a0, s2    =>  cm.mva01s s2,s1
+  //
+  //   mv a0, s2
+  //   mv a1, s1    =>  cm.mva01s s2,s1
+  if (Opcode == RISCV::CM_MVA01S) {
+    Sreg1 =
+        ARegInFirstPair == RISCV::X10 ? FirstPair.Source : PairedRegs.Source;
+    Sreg2 =
+        ARegInFirstPair == RISCV::X10 ? PairedRegs.Source : FirstPair.Source;
+  } else {
+    Sreg1 = ARegInFirstPair == RISCV::X10 ? FirstPair.Destination
+                                          : PairedRegs.Destination;
+    Sreg2 = ARegInFirstPair == RISCV::X10 ? PairedRegs.Destination
+                                          : FirstPair.Destination;
+  }
+
+  BuildMI(*I->getParent(), I, DL, TII->get(Opcode)).add(*Sreg1).add(*Sreg2);
+
+  I->eraseFromParent();
+  Paired->eraseFromParent();
+  return NextI;
+}
+
+MachineBasicBlock::iterator
+RISCVMoveOpt::findMatchingInst(MachineBasicBlock::iterator &MBBI,
+                               unsigned InstOpcode) {
+  MachineBasicBlock::iterator E = MBBI->getParent()->end();
+  DestSourcePair FirstPair = TII->isCopyInstrImpl(*MBBI).value();
+
+  // Track which register units have been modified and used between the first
+  // insn and the second insn.
+  ModifiedRegUnits.clear();
+  UsedRegUnits.clear();
+
+  for (MachineBasicBlock::iterator I = next_nodbg(MBBI, E); I != E;
+       I = next_nodbg(I, E)) {
+
+    MachineInstr &MI = *I;
+
+    if (auto SecondPair = TII->isCopyInstrImpl(MI)) {
+      Register SourceReg = SecondPair->Source->getReg();
+      Register DestReg = SecondPair->Destination->getReg();
+
+      if (InstOpcode == RISCV::CM_MVA01S &&
+          isCandidateToMergeMVA01S(*SecondPair)) {
+        // If register pair is valid and destination registers are different.
+        if ((FirstPair.Destination->getReg() == DestReg))
+          return E;
+
+        //  If paired destination register was modified or used, there is no
+        //  possibility of finding matching instruction so exit early.
+        if (!ModifiedRegUnits.available(DestReg) ||
+            !UsedRegUnits.available(DestReg))
+          return E;
+
+        // We need to check if the source register in the second paired
+        // instruction is defined in between.
+        if (ModifiedRegUnits.available(SourceReg))
+          return I;
+
+      } else if (InstOpcode == RISCV::CM_MVSA01 &&
+                 isCandidateToMergeMVSA01(*SecondPair)) {
+        if ((FirstPair.Source->getReg() == SourceReg) ||
+            (FirstPair.Destination->getReg() == DestReg))
+          return E;
+
+        if (!ModifiedRegUnits.available(SourceReg) ||
+            !UsedRegUnits.available(SourceReg))
+          return E;
+
+        // As for mvsa01, we need to make sure the dest register of the second
+        // paired instruction is not used in between, since we would move its
+        // definition ahead.
+        if (UsedRegUnits.available(DestReg))
+          return I;
+      }
+    }
+    // Update modified / used register units.
+    LiveRegUnits::accumulateUsedDefed(MI, ModifiedRegUnits, UsedRegUnits, TRI);
+  }
+  return E;
+}
+
+// Finds instructions, which could be represented as C.MV instructions and
+// merged into CM.MVA01S or CM.MVSA01.
+bool RISCVMoveOpt::MovOpt(MachineBasicBlock &MBB) {
+  bool Modified = false;
+
+  for (MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
+       MBBI != E;) {
+    // Check if the instruction can be compressed to C.MV instruction. If it
+    // can, return Dest/Src register pair.
+    auto RegPair = TII->isCopyInstrImpl(*MBBI);
+    if (RegPair.has_value()) {
+      unsigned Opcode = 0;
+
+      if (isCandidateToMergeMVA01S(*RegPair))
+        Opcode = RISCV::CM_MVA01S;
+      else if (isCandidateToMergeMVSA01(*RegPair))
+        Opcode = RISCV::CM_MVSA01;
+      else {
+        ++MBBI;
+        continue;
+      }
+
+      MachineBasicBlock::iterator Paired = findMatchingInst(MBBI, Opcode);
+      // If matching instruction could be found merge them.
+      if (Paired != E) {
+        MBBI = mergePairedInsns(MBBI, Paired, Opcode);
+        Modified = true;
+        continue;
+      }
+    }
+    ++MBBI;
+  }
+  return Modified;
+}
+
+bool RISCVMoveOpt::runOnMachineFunction(MachineFunction &Fn) {
+  if (skipFunction(Fn.getFunction()))
+    return false;
+
+  Subtarget = &static_cast<const RISCVSubtarget &>(Fn.getSubtarget());
+  if (!Subtarget->hasStdExtZcmp()) {
+    return false;
+  }
+
+  TII = static_cast<const RISCVInstrInfo *>(Subtarget->getInstrInfo());
+  TRI = Subtarget->getRegisterInfo();
+  // Resize the modified and used register unit trackers.  We do this once
+  // per function and then clear the register units each time we optimize a
+  // move.
+  ModifiedRegUnits.init(*TRI);
+  UsedRegUnits.init(*TRI);
+  bool Modified = false;
+  for (auto &MBB : Fn) {
+    Modified |= MovOpt(MBB);
+  }
+  return Modified;
+}
+
+/// createRISCVMoveOptimizationPass - returns an instance of the
+/// move optimization pass.
+FunctionPass *llvm::createRISCVMoveOptimizationPass() {
+  return new RISCVMoveOpt();
+}

llvm/lib/Target/RISCV/RISCVPushPopOptimizer.cpp

+//===------- RISCVPushPopOptimizer.cpp - RISCV Push/Pop opt. pass ---------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains a pass that modifies PUSH/POP instructions from Zca
+// standard to use their non prolog/epilog related functionalities
+// and generates POPRET instruction.
+//
+//===----------------------------------------------------------------------===//
+
+#include "RISCVInstrInfo.h"
+#include "RISCVMachineFunctionInfo.h"
+
+using namespace llvm;
+
+#define RISCV_PUSH_POP_OPT_NAME "RISC-V Zc Push/Pop optimization pass"
+
+namespace {
+struct RISCVPushPopOpt : public MachineFunctionPass {
+  static char ID;
+
+  RISCVPushPopOpt() : MachineFunctionPass(ID) {
+    initializeRISCVPushPopOptPass(*PassRegistry::getPassRegistry());
+  }
+
+  const RISCVInstrInfo *TII;
+  const TargetRegisterInfo *TRI;
+  const RISCVSubtarget *Subtarget;
+
+  // Track which register units have been modified and used.
+  LiveRegUnits ModifiedRegUnits, UsedRegUnits;
+
+  bool usePopRet(MachineBasicBlock::iterator &MBBI);
+  bool adjustRetVal(MachineBasicBlock::iterator &MBBI);
+  bool runOnMachineFunction(MachineFunction &Fn) override;
+
+  std::map<MachineInstr *, int> retValMap;
+
+  StringRef getPassName() const override { return RISCV_PUSH_POP_OPT_NAME; }
+};
+
+char RISCVPushPopOpt::ID = 0;
+
+} // end of anonymous namespace
+
+INITIALIZE_PASS(RISCVPushPopOpt, "riscv-push-pop-opt", RISCV_PUSH_POP_OPT_NAME,
+                false, false)
+
+// Check if POP instruction was inserted into the MBB and return iterator to it.
+static MachineBasicBlock::iterator containsPop(MachineBasicBlock &MBB) {
+  for (MachineBasicBlock::iterator MBBI = MBB.begin(); MBBI != MBB.end();
+       MBBI = next_nodbg(MBBI, MBB.end()))
+    if (MBBI->getOpcode() == RISCV::CM_POP)
+      return MBBI;
+
+  return MBB.end();
+}
+
+bool RISCVPushPopOpt::usePopRet(MachineBasicBlock::iterator &MBBI) {
+  MachineBasicBlock::iterator E = MBBI->getParent()->end();
+  MachineBasicBlock::iterator NextI = next_nodbg(MBBI, E);
+  // Since Pseudo instruction lowering happen later in the pipeline,
+  // this will detect all ret instruction.
+  if (NextI->getOpcode() == RISCV::PseudoRET) {
+    DebugLoc DL = NextI->getDebugLoc();
+    auto retValInfo = retValMap.find(&(*MBBI));
+    if (retValInfo == retValMap.end())
+      BuildMI(*NextI->getParent(), NextI, DL, TII->get(RISCV::CM_POPRET))
+          .add(MBBI->getOperand(0))
+          .add(MBBI->getOperand(1));
+    else if (retValInfo->second == 0)
+      BuildMI(*NextI->getParent(), NextI, DL, TII->get(RISCV::CM_POPRETZ))
+          .add(MBBI->getOperand(0))
+          .add(MBBI->getOperand(1));
+    // If the return value is not 0 then POPRETZ is not used.
+    else
+      return false;
+    MBBI->eraseFromParent();
+    NextI->eraseFromParent();
+    return true;
+  }
+  return false;
+}
+
+// Search for last assignment to a0 and if possible use ret_val slot of POP to
+// store return value.
+bool RISCVPushPopOpt::adjustRetVal(MachineBasicBlock::iterator &MBBI) {
+  MachineBasicBlock::reverse_iterator RE = MBBI->getParent()->rend();
+  // Track which register units have been modified and used between the POP
+  // insn and the last assignment to register a0.
+  ModifiedRegUnits.clear();
+  UsedRegUnits.clear();
+  retValMap.clear();
+  Register A0(RISCV::X10);
+  // Since POP instruction is in Epilogue no normal instructions will follow
+  // after it. Therefore search only previous ones to find the return value.
+  for (MachineBasicBlock::reverse_iterator I =
+           next_nodbg(MBBI.getReverse(), RE);
+       I != RE; I = next_nodbg(I, RE)) {
+    MachineInstr &MI = *I;
+    if (auto OperandPair = TII->isLoadImmImpl(MI)) {
+      Register DestReg = OperandPair->Destination->getReg();
+      int64_t RetVal = OperandPair->Source->getImm();
+      if (DestReg == RISCV::X10) {
+        switch (RetVal) {
+        default:
+          return false;
+        case 0:
+          retValMap[&(*MBBI)] = 0;
+        }
+        MI.removeFromParent();
+        return true;
+      }
+    }
+    // Update modified / used register units.
+    LiveRegUnits::accumulateUsedDefed(MI, ModifiedRegUnits, UsedRegUnits, TRI);
+    // If a0 was modified or used, there is no possibility
+    // of using ret_val slot of POP instruction.
+    if (!ModifiedRegUnits.available(A0) || !UsedRegUnits.available(A0))
+      return false;
+  }
+  return false;
+}
+
+bool RISCVPushPopOpt::runOnMachineFunction(MachineFunction &Fn) {
+  if (skipFunction(Fn.getFunction()))
+    return false;
+
+  // If Zcmp extension is not supported, abort.
+  Subtarget = &static_cast<const RISCVSubtarget &>(Fn.getSubtarget());
+  if (!Subtarget->hasStdExtZcmp()) {
+    return false;
+  }
+
+  // If frame pointer elimination has been disabled,
+  // abort to avoid breaking the ABI.
+  if (Fn.getTarget().Options.DisableFramePointerElim(Fn)) {
+    return false;
+  }
+
+  TII = static_cast<const RISCVInstrInfo *>(Subtarget->getInstrInfo());
+  TRI = Subtarget->getRegisterInfo();
+  // Resize the modified and used register unit trackers.  We do this once
+  // per function and then clear the register units each time we determine
+  // correct return value for the POP.
+  ModifiedRegUnits.init(*TRI);
+  UsedRegUnits.init(*TRI);
+  bool Modified = false;
+  for (auto &MBB : Fn) {
+    auto MBBI = containsPop(MBB);
+    if (MBBI != MBB.end()) {
+      Modified |= adjustRetVal(MBBI);
+      if (MBB.isReturnBlock())
+        Modified |= usePopRet(MBBI);
+    }
+  }
+  return Modified;
+}
+
+/// createRISCVPushPopOptimizationPass - returns an instance of the
+/// Push/Pop optimization pass.
+FunctionPass *llvm::createRISCVPushPopOptimizationPass() {
+  return new RISCVPushPopOpt();
+}

llvm/lib/Target/RISCV/RISCVRegisterInfo.td

 // Saved Registers from s0 to s7, for C.MVA01S07 instruction in Zcmp extension
 def SR07 : RegisterClass<"RISCV", [XLenVT], 32, (add
     (sequence "X%u", 8, 9),
     (sequence "X%u", 18, 23)
   )> {
   let RegInfos = XLenRI;
 }
 
+// Registers saveable by PUSH instruction in Zce extension
+def PGPR : RegisterClass<"RISCV", [XLenVT], 32, (add
+    (sequence "X%u", 8, 9),
+    (sequence "X%u", 18, 25),
+    X27,
+    X1
+  )> {
+  let RegInfos = XLenRI;
+}
+
 // Floating point registers
 let RegAltNameIndices = [ABIRegAltName] in {
   def F0_H  : RISCVReg16<0, "f0", ["ft0"]>, DwarfRegNum<[32]>;
   def F1_H  : RISCVReg16<1, "f1", ["ft1"]>, DwarfRegNum<[33]>;
   def F2_H  : RISCVReg16<2, "f2", ["ft2"]>, DwarfRegNum<[34]>;
   def F3_H  : RISCVReg16<3, "f3", ["ft3"]>, DwarfRegNum<[35]>;
   def F4_H  : RISCVReg16<4, "f4", ["ft4"]>, DwarfRegNum<[36]>;
   def F5_H  : RISCVReg16<5, "f5", ["ft5"]>, DwarfRegNum<[37]>;

llvm/lib/Target/RISCV/RISCVTargetMachine.cpp

extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeRISCVTarget() {
   initializeRISCVMergeBaseOffsetOptPass(*PR);
   initializeRISCVOptWInstrsPass(*PR);
   initializeRISCVPreRAExpandPseudoPass(*PR);
   initializeRISCVExpandPseudoPass(*PR);
   initializeRISCVInsertNTLHInstsPass(*PR);
   initializeRISCVInsertVSETVLIPass(*PR);
   initializeRISCVDAGToDAGISelPass(*PR);
   initializeRISCVInitUndefPass(*PR);
+  initializeRISCVMoveOptPass(*PR);
+  initializeRISCVPushPopOptPass(*PR);
 }
 
 static StringRef computeDataLayout(const Triple &TT) {
   if (TT.isArch64Bit())
     return "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128";
   assert(TT.isArch32Bit() && "only RV32 and RV64 are currently supported");
   return "e-m:e-p:32:32-i64:64-n32-S128";
 }
void RISCVPassConfig::addPreEmitPass() {
   // propagation after the machine outliner (which runs after addPreEmitPass)
   // currently leads to incorrect code-gen, where copies to registers within
   // outlined functions are removed erroneously.
   if (TM->getOptLevel() >= CodeGenOpt::Default && EnableRISCVCopyPropagation)
     addPass(createMachineCopyPropagationPass(true));
 }
 
 void RISCVPassConfig::addPreEmitPass2() {
+  addPass(createRISCVMoveOptimizationPass());
+  // Schedule PushPop Optimization before expansion of Pseudo instruction,
+  // ensuring return instruction is detected correctly.
+  addPass(createRISCVPushPopOptimizationPass());
   addPass(createRISCVExpandPseudoPass());
   addPass(createRISCVInsertNTLHInstsPass());
 
   // Schedule the expansion of AMOs at the last possible moment, avoiding the
   // possibility for other passes to break the requirements for forward
   // progress in the LR/SC block.
   addPass(createRISCVExpandAtomicPseudoPass());
 }

llvm/test/CodeGen/RISCV/O0-pipeline.ll

 ; CHECK-NEXT:       RISC-V Make Compressible
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Stack Frame Layout Analysis
+; CHECK-NEXT:       RISC-V Zc move merging pass 
+; CHECK-NEXT:       RISC-V Zc Push/Pop optimization pass 
 ; CHECK-NEXT:       RISC-V pseudo instruction expansion pass
 ; CHECK-NEXT:       RISC-V insert NTLH instruction pass
 ; CHECK-NEXT:       RISC-V atomic pseudo instruction expansion pass
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       RISC-V Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction

llvm/test/CodeGen/RISCV/O3-pipeline.ll

 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:     Machine Outliner
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Stack Frame Layout Analysis
+; CHECK-NEXT:       RISC-V Zc move merging pass 
+; CHECK-NEXT:       RISC-V Zc Push/Pop optimization pass 
 ; CHECK-NEXT:       RISC-V pseudo instruction expansion pass
 ; CHECK-NEXT:       RISC-V insert NTLH instruction pass
 ; CHECK-NEXT:       RISC-V atomic pseudo instruction expansion pass
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       RISC-V Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction

llvm/test/CodeGen/RISCV/attributes.ll

 ; RUN: llc -mtriple=riscv32 -mattr=+xtheadmac %s -o - | FileCheck --check-prefixes=CHECK,RV32XTHEADMAC %s
 ; RUN: llc -mtriple=riscv32 -mattr=+xtheadmemidx %s -o - | FileCheck --check-prefix=RV32XTHEADMEMIDX %s
 ; RUN: llc -mtriple=riscv32 -mattr=+xtheadmempair %s -o - | FileCheck --check-prefix=RV32XTHEADMEMPAIR %s
 ; RUN: llc -mtriple=riscv32 -mattr=+xtheadsync %s -o - | FileCheck --check-prefix=RV32XTHEADSYNC %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-zca %s -o - | FileCheck --check-prefixes=CHECK,RV32ZCA %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-zcb %s -o - | FileCheck --check-prefixes=CHECK,RV32ZCB %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-zcd %s -o - | FileCheck --check-prefixes=CHECK,RV32ZCD %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-zcf %s -o - | FileCheck --check-prefixes=CHECK,RV32ZCF %s
+; RUN: llc -mtriple=riscv32 -mattr=+experimental-zcmp %s -o - | FileCheck --check-prefixes=CHECK,RV32ZCMP %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zicsr %s -o - | FileCheck --check-prefixes=CHECK,RV32ZICSR %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zifencei %s -o - | FileCheck --check-prefixes=CHECK,RV32ZIFENCEI %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-zfa %s -o - | FileCheck --check-prefixes=CHECK,RV32ZFA %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zve32x -mattr=+experimental-zvbb %s -o - | FileCheck --check-prefix=RV32ZVBB %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zve64x -mattr=+experimental-zvbc %s -o - | FileCheck --check-prefix=RV32ZVBC %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zve32x -mattr=+experimental-zvkg %s -o - | FileCheck --check-prefix=RV32ZVKG %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zve64x -mattr=+experimental-zvkn %s -o - | FileCheck --check-prefix=RV32ZVKN %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zve32x -mattr=+experimental-zvkned %s -o - | FileCheck --check-prefix=RV32ZVKNED %s

 ; RUN: llc -mtriple=riscv64 -mattr=+xtheadmempair %s -o - | FileCheck --check-prefix=RV64XTHEADMEMPAIR %s
 ; RUN: llc -mtriple=riscv64 -mattr=+xtheadsync %s -o - | FileCheck --check-prefix=RV64XTHEADSYNC %s
 ; RUN: llc -mtriple=riscv64 -mattr=+xtheadvdot %s -o - | FileCheck --check-prefixes=CHECK,RV64XTHEADVDOT %s
 ; RUN: llc -mtriple=riscv64 -mattr=+zawrs %s -o - | FileCheck --check-prefixes=CHECK,RV64ZAWRS %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-ztso %s -o - | FileCheck --check-prefixes=CHECK,RV64ZTSO %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-zca %s -o - | FileCheck --check-prefixes=CHECK,RV64ZCA %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-zcb %s -o - | FileCheck --check-prefixes=CHECK,RV64ZCB %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-zcd %s -o - | FileCheck --check-prefixes=CHECK,RV64ZCD %s
+; RUN: llc -mtriple=riscv64 -mattr=+experimental-zcmp %s -o - | FileCheck --check-prefixes=CHECK,RV64ZCMP %s
 ; RUN: llc -mtriple=riscv64 -mattr=+zicsr %s -o - | FileCheck --check-prefixes=CHECK,RV64ZICSR %s
 ; RUN: llc -mtriple=riscv64 -mattr=+zifencei %s -o - | FileCheck --check-prefixes=CHECK,RV64ZIFENCEI %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-zfa %s -o - | FileCheck --check-prefixes=CHECK,RV64ZFA %s
 ; RUN: llc -mtriple=riscv64 -mattr=+zve32x -mattr=+experimental-zvbb %s -o - | FileCheck --check-prefix=RV64ZVBB %s
 ; RUN: llc -mtriple=riscv64 -mattr=+zve64x -mattr=+experimental-zvbc %s -o - | FileCheck --check-prefix=RV64ZVBC %s
 ; RUN: llc -mtriple=riscv64 -mattr=+zve32x -mattr=+experimental-zvkg %s -o - | FileCheck --check-prefix=RV64ZVKG %s
 ; RUN: llc -mtriple=riscv64 -mattr=+zve64x -mattr=+experimental-zvkn %s -o - | FileCheck --check-prefix=RV64ZVKN %s
 ; RUN: llc -mtriple=riscv64 -mattr=+zve32x -mattr=+experimental-zvkned %s -o - | FileCheck --check-prefix=RV64ZVKNED %s

 ; RV32XTHEADMAC: .attribute 5, "rv32i2p1_xtheadmac1p0"
 ; RV32XTHEADMEMIDX: .attribute 5, "rv32i2p1_xtheadmemidx1p0"
 ; RV32XTHEADMEMPAIR: .attribute 5, "rv32i2p1_xtheadmempair1p0"
 ; RV32XTHEADSYNC: .attribute 5, "rv32i2p1_xtheadsync1p0"
 ; RV32ZCA: .attribute 5, "rv32i2p1_zca1p0"
 ; RV32ZCB: .attribute 5, "rv32i2p1_zca1p0_zcb1p0"
 ; RV32ZCD: .attribute 5, "rv32i2p1_zcd1p0"
 ; RV32ZCF: .attribute 5, "rv32i2p1_zcf1p0"
+; RV32ZCMP: .attribute 5, "rv32i2p1_zca1p0_zcmp1p0"
 ; RV32ZICSR: .attribute 5, "rv32i2p1_zicsr2p0"
 ; RV32ZIFENCEI: .attribute 5, "rv32i2p1_zifencei2p0"
 ; RV32ZFA: .attribute 5, "rv32i2p1_f2p2_zicsr2p0_zfa0p2"
 ; RV32ZVBB: .attribute 5, "rv32i2p1_zicsr2p0_zvbb0p5_zve32x1p0_zvl32b1p0"
 ; RV32ZVBC: .attribute 5, "rv32i2p1_zicsr2p0_zvbc0p5_zve32x1p0_zve64x1p0_zvl32b1p0_zvl64b1p0"
 ; RV32ZVKG: .attribute 5, "rv32i2p1_zicsr2p0_zve32x1p0_zvkg0p5_zvl32b1p0"
 ; RV32ZVKN: .attribute 5, "rv32i2p1_zicsr2p0_zvbb0p5_zvbc0p5_zve32x1p0_zve64x1p0_zvkn0p5_zvkned0p5_zvknha0p5_zvknhb0p5_zvkt0p5_zvl32b1p0_zvl64b1p0"
 ; RV32ZVKNED: .attribute 5, "rv32i2p1_zicsr2p0_zve32x1p0_zvkned0p5_zvl32b1p0"

 ; RV64XTHEADMEMIDX: .attribute 5, "rv64i2p1_xtheadmemidx1p0"
 ; RV64XTHEADMEMPAIR: .attribute 5, "rv64i2p1_xtheadmempair1p0"
 ; RV64XTHEADSYNC: .attribute 5, "rv64i2p1_xtheadsync1p0"
 ; RV64XTHEADVDOT: .attribute 5, "rv64i2p1_f2p2_d2p2_v1p0_zicsr2p0_zve32f1p0_zve32x1p0_zve64d1p0_zve64f1p0_zve64x1p0_zvl128b1p0_zvl32b1p0_zvl64b1p0_xtheadvdot1p0"
 ; RV64ZTSO: .attribute 5, "rv64i2p1_ztso0p1"
 ; RV64ZCA: .attribute 5, "rv64i2p1_zca1p0"
 ; RV64ZCB: .attribute 5, "rv64i2p1_zca1p0_zcb1p0"
 ; RV64ZCD: .attribute 5, "rv64i2p1_zcd1p0"
+; RV64ZCMP: .attribute 5, "rv64i2p1_zca1p0_zcmp1p0"
 ; RV64ZICSR: .attribute 5, "rv64i2p1_zicsr2p0"
 ; RV64ZIFENCEI: .attribute 5, "rv64i2p1_zifencei2p0"
 ; RV64ZFA: .attribute 5, "rv64i2p1_f2p2_zicsr2p0_zfa0p2"
 ; RV64ZVBB: .attribute 5, "rv64i2p1_zicsr2p0_zvbb0p5_zve32x1p0_zvl32b1p0"
 ; RV64ZVBC: .attribute 5, "rv64i2p1_zicsr2p0_zvbc0p5_zve32x1p0_zve64x1p0_zvl32b1p0_zvl64b1p0"
 ; RV64ZVKG: .attribute 5, "rv64i2p1_zicsr2p0_zve32x1p0_zvkg0p5_zvl32b1p0"
 ; RV64ZVKN: .attribute 5, "rv64i2p1_zicsr2p0_zvbb0p5_zvbc0p5_zve32x1p0_zve64x1p0_zvkn0p5_zvkned0p5_zvknha0p5_zvknhb0p5_zvkt0p5_zvl32b1p0_zvl64b1p0"
 ; RV64ZVKNED: .attribute 5, "rv64i2p1_zicsr2p0_zve32x1p0_zvkned0p5_zvl32b1p0"

llvm/test/CodeGen/RISCV/callee-saved-gprs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=riscv32 -verify-machineinstrs < %s \
 ; RUN:   | FileCheck %s -check-prefix=RV32I
 ; RUN: llc -mtriple=riscv32 -mattr=+f -target-abi ilp32f -verify-machineinstrs < %s \
 ; RUN:   | FileCheck %s -check-prefix=RV32I
 ; RUN: llc -mtriple=riscv32 -mattr=+d -target-abi ilp32f -verify-machineinstrs < %s \
 ; RUN:   | FileCheck %s -check-prefix=RV32I
 ; RUN: llc -mtriple=riscv32 -mattr=+d -target-abi ilp32d -verify-machineinstrs < %s \
 ; RUN:   | FileCheck %s -check-prefix=RV32I
 ; RUN: llc -mtriple=riscv32 -verify-machineinstrs -frame-pointer=all < %s \
 ; RUN:   | FileCheck %s -check-prefix=RV32I-WITH-FP
+; RUN: llc -mtriple=riscv32 -mattr=+experimental-zcmp -verify-machineinstrs < %s \
+; RUN: | FileCheck %s -check-prefixes=RV32IZCMP
+; RUN: llc -mtriple=riscv32 -mattr=+experimental-zcmp -verify-machineinstrs -frame-pointer=all < %s \
+; RUN: | FileCheck %s -check-prefixes=RV32IZCMP-WITH-FP
 ; RUN: llc -mtriple=riscv64 -verify-machineinstrs < %s \
 ; RUN:   | FileCheck %s -check-prefix=RV64I
 ; RUN: llc -mtriple=riscv64 -mattr=+f -target-abi lp64f -verify-machineinstrs < %s \
 ; RUN:   | FileCheck %s -check-prefix=RV64I
 ; RUN: llc -mtriple=riscv64 -mattr=+d -target-abi lp64f -verify-machineinstrs < %s \
 ; RUN:   | FileCheck %s -check-prefix=RV64I
 ; RUN: llc -mtriple=riscv64 -mattr=+d -target-abi lp64d -verify-machineinstrs < %s \
 ; RUN:   | FileCheck %s -check-prefix=RV64I
 ; RUN: llc -mtriple=riscv64 -verify-machineinstrs -frame-pointer=all < %s \
 ; RUN:   | FileCheck %s -check-prefix=RV64I-WITH-FP
+; RUN: llc -mtriple=riscv64 -mattr=+experimental-zcmp -verify-machineinstrs < %s \
+; RUN: | FileCheck %s -check-prefixes=RV64IZCMP
+; RUN: llc -mtriple=riscv64 -mattr=+experimental-zcmp -verify-machineinstrs -frame-pointer=all < %s \
+; RUN: | FileCheck %s -check-prefixes=RV64IZCMP-WITH-FP
 
 @var = global [32 x i32] zeroinitializer
 
 ; This function tests that RISCVRegisterInfo::getCalleeSavedRegs returns
 ; something appropriate.
 
 define void @callee() nounwind {
 ; RV32I-LABEL: callee:

 ; RV32I-WITH-FP-NEXT:    lw s7, 44(sp) # 4-byte Folded Reload
 ; RV32I-WITH-FP-NEXT:    lw s8, 40(sp) # 4-byte Folded Reload
 ; RV32I-WITH-FP-NEXT:    lw s9, 36(sp) # 4-byte Folded Reload
 ; RV32I-WITH-FP-NEXT:    lw s10, 32(sp) # 4-byte Folded Reload
 ; RV32I-WITH-FP-NEXT:    lw s11, 28(sp) # 4-byte Folded Reload
 ; RV32I-WITH-FP-NEXT:    addi sp, sp, 80
 ; RV32I-WITH-FP-NEXT:    ret
 ;
+; RV32IZCMP-LABEL: callee:
+; RV32IZCMP:       # %bb.0:
+; RV32IZCMP-NEXT:    cm.push {ra, s0-s11}, -80
+; RV32IZCMP-NEXT:    lui a7, %hi(var)
+; RV32IZCMP-NEXT:    lw a0, %lo(var)(a7)
+; RV32IZCMP-NEXT:    sw a0, 24(sp) # 4-byte Folded Spill
+; RV32IZCMP-NEXT:    lw a0, %lo(var+4)(a7)
+; RV32IZCMP-NEXT:    sw a0, 20(sp) # 4-byte Folded Spill
+; RV32IZCMP-NEXT:    lw a0, %lo(var+8)(a7)
+; RV32IZCMP-NEXT:    sw a0, 16(sp) # 4-byte Folded Spill
+; RV32IZCMP-NEXT:    lw a0, %lo(var+12)(a7)
+; RV32IZCMP-NEXT:    sw a0, 12(sp) # 4-byte Folded Spill
+; RV32IZCMP-NEXT:    addi a5, a7, %lo(var)
+; RV32IZCMP-NEXT:    lw a0, 16(a5)
+; RV32IZCMP-NEXT:    sw a0, 8(sp) # 4-byte Folded Spill
+; RV32IZCMP-NEXT:    lw a0, 20(a5)
+; RV32IZCMP-NEXT:    sw a0, 4(sp) # 4-byte Folded Spill
+; RV32IZCMP-NEXT:    lw t4, 24(a5)
+; RV32IZCMP-NEXT:    lw t5, 28(a5)
+; RV32IZCMP-NEXT:    lw t6, 32(a5)
+; RV32IZCMP-NEXT:    lw s2, 36(a5)
+; RV32IZCMP-NEXT:    lw s3, 40(a5)
+; RV32IZCMP-NEXT:    lw s4, 44(a5)
+; RV32IZCMP-NEXT:    lw s5, 48(a5)
+; RV32IZCMP-NEXT:    lw s6, 52(a5)
+; RV32IZCMP-NEXT:    lw s7, 56(a5)
+; RV32IZCMP-NEXT:    lw s8, 60(a5)
+; RV32IZCMP-NEXT:    lw s9, 64(a5)
+; RV32IZCMP-NEXT:    lw s10, 68(a5)
+; RV32IZCMP-NEXT:    lw s11, 72(a5)
+; RV32IZCMP-NEXT:    lw ra, 76(a5)
+; RV32IZCMP-NEXT:    lw s1, 80(a5)
+; RV32IZCMP-NEXT:    lw t3, 84(a5)
+; RV32IZCMP-NEXT:    lw t2, 88(a5)
+; RV32IZCMP-NEXT:    lw t1, 92(a5)
+; RV32IZCMP-NEXT:    lw t0, 96(a5)
+; RV32IZCMP-NEXT:    lw s0, 100(a5)
+; RV32IZCMP-NEXT:    lw a6, 104(a5)
+; RV32IZCMP-NEXT:    lw a4, 108(a5)
+; RV32IZCMP-NEXT:    lw a0, 124(a5)
+; RV32IZCMP-NEXT:    lw a1, 120(a5)
+; RV32IZCMP-NEXT:    lw a2, 116(a5)
+; RV32IZCMP-NEXT:    lw a3, 112(a5)
+; RV32IZCMP-NEXT:    sw a0, 124(a5)
+; RV32IZCMP-NEXT:    sw a1, 120(a5)
+; RV32IZCMP-NEXT:    sw a2, 116(a5)
+; RV32IZCMP-NEXT:    sw a3, 112(a5)
+; RV32IZCMP-NEXT:    sw a4, 108(a5)
+; RV32IZCMP-NEXT:    sw a6, 104(a5)
+; RV32IZCMP-NEXT:    sw s0, 100(a5)
+; RV32IZCMP-NEXT:    sw t0, 96(a5)
+; RV32IZCMP-NEXT:    sw t1, 92(a5)
+; RV32IZCMP-NEXT:    sw t2, 88(a5)
+; RV32IZCMP-NEXT:    sw t3, 84(a5)
+; RV32IZCMP-NEXT:    sw s1, 80(a5)
+; RV32IZCMP-NEXT:    sw ra, 76(a5)
+; RV32IZCMP-NEXT:    sw s11, 72(a5)
+; RV32IZCMP-NEXT:    sw s10, 68(a5)
+; RV32IZCMP-NEXT:    sw s9, 64(a5)
+; RV32IZCMP-NEXT:    sw s8, 60(a5)
+; RV32IZCMP-NEXT:    sw s7, 56(a5)
+; RV32IZCMP-NEXT:    sw s6, 52(a5)
+; RV32IZCMP-NEXT:    sw s5, 48(a5)
+; RV32IZCMP-NEXT:    sw s4, 44(a5)
+; RV32IZCMP-NEXT:    sw s3, 40(a5)
+; RV32IZCMP-NEXT:    sw s2, 36(a5)
+; RV32IZCMP-NEXT:    sw t6, 32(a5)
+; RV32IZCMP-NEXT:    sw t5, 28(a5)
+; RV32IZCMP-NEXT:    sw t4, 24(a5)
+; RV32IZCMP-NEXT:    lw a0, 4(sp) # 4-byte Folded Reload
+; RV32IZCMP-NEXT:    sw a0, 20(a5)
+; RV32IZCMP-NEXT:    lw a0, 8(sp) # 4-byte Folded Reload
+; RV32IZCMP-NEXT:    sw a0, 16(a5)
+; RV32IZCMP-NEXT:    lw a0, 12(sp) # 4-byte Folded Reload
+; RV32IZCMP-NEXT:    sw a0, %lo(var+12)(a7)
+; RV32IZCMP-NEXT:    lw a0, 16(sp) # 4-byte Folded Reload
+; RV32IZCMP-NEXT:    sw a0, %lo(var+8)(a7)
+; RV32IZCMP-NEXT:    lw a0, 20(sp) # 4-byte Folded Reload
+; RV32IZCMP-NEXT:    sw a0, %lo(var+4)(a7)
+; RV32IZCMP-NEXT:    lw a0, 24(sp) # 4-byte Folded Reload
+; RV32IZCMP-NEXT:    sw a0, %lo(var)(a7)
+; RV32IZCMP-NEXT:    cm.popret {ra, s0-s11}, 80
+;
+; RV32IZCMP-WITH-FP-LABEL: callee:
+; RV32IZCMP-WITH-FP:       # %bb.0:
+; RV32IZCMP-WITH-FP-NEXT:    addi sp, sp, -80
+; RV32IZCMP-WITH-FP-NEXT:    sw ra, 76(sp) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    sw s0, 72(sp) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    sw s1, 68(sp) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    sw s2, 64(sp) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    sw s3, 60(sp) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    sw s4, 56(sp) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    sw s5, 52(sp) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    sw s6, 48(sp) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    sw s7, 44(sp) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    sw s8, 40(sp) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    sw s9, 36(sp) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    sw s10, 32(sp) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    sw s11, 28(sp) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    addi s0, sp, 80
+; RV32IZCMP-WITH-FP-NEXT:    lui a7, %hi(var)
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, %lo(var)(a7)
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, -56(s0) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, %lo(var+4)(a7)
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, -60(s0) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, %lo(var+8)(a7)
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, -64(s0) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, %lo(var+12)(a7)
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, -68(s0) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    addi a5, a7, %lo(var)
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, 16(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, -72(s0) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, 20(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, -76(s0) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, 24(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, -80(s0) # 4-byte Folded Spill
+; RV32IZCMP-WITH-FP-NEXT:    lw t5, 28(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw t6, 32(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw s2, 36(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw s3, 40(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw s4, 44(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw s5, 48(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw s6, 52(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw s7, 56(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw s8, 60(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw s9, 64(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw s10, 68(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw s11, 72(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw ra, 76(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw t4, 80(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw t3, 84(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw t2, 88(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw s1, 92(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw t1, 96(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw t0, 100(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw a6, 104(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw a4, 108(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, 124(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw a1, 120(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw a2, 116(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw a3, 112(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, 124(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw a1, 120(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw a2, 116(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw a3, 112(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw a4, 108(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw a6, 104(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw t0, 100(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw t1, 96(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw s1, 92(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw t2, 88(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw t3, 84(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw t4, 80(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw ra, 76(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw s11, 72(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw s10, 68(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw s9, 64(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw s8, 60(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw s7, 56(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw s6, 52(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw s5, 48(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw s4, 44(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw s3, 40(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw s2, 36(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw t6, 32(a5)
+; RV32IZCMP-WITH-FP-NEXT:    sw t5, 28(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, -80(s0) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, 24(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, -76(s0) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, 20(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, -72(s0) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, 16(a5)
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, -68(s0) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, %lo(var+12)(a7)
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, -64(s0) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, %lo(var+8)(a7)
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, -60(s0) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, %lo(var+4)(a7)
+; RV32IZCMP-WITH-FP-NEXT:    lw a0, -56(s0) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    sw a0, %lo(var)(a7)
+; RV32IZCMP-WITH-FP-NEXT:    lw ra, 76(sp) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    lw s0, 72(sp) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    lw s1, 68(sp) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    lw s2, 64(sp) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    lw s3, 60(sp) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    lw s4, 56(sp) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    lw s5, 52(sp) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    lw s6, 48(sp) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    lw s7, 44(sp) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    lw s8, 40(sp) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    lw s9, 36(sp) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    lw s10, 32(sp) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    lw s11, 28(sp) # 4-byte Folded Reload
+; RV32IZCMP-WITH-FP-NEXT:    addi sp, sp, 80
+; RV32IZCMP-WITH-FP-NEXT:    ret
+;
 ; RV64I-LABEL: callee:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -160
 ; RV64I-NEXT:    sd ra, 152(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    sd s0, 144(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    sd s1, 136(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    sd s2, 128(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    sd s3, 120(sp) # 8-byte Folded Spill

 ; RV64I-WITH-FP-NEXT:    ld s6, 96(sp) # 8-byte Folded Reload
 ; RV64I-WITH-FP-NEXT:    ld s7, 88(sp) # 8-byte Folded Reload
 ; RV64I-WITH-FP-NEXT:    ld s8, 80(sp) # 8-byte Folded Reload
 ; RV64I-WITH-FP-NEXT:    ld s9, 72(sp) # 8-byte Folded Reload
 ; RV64I-WITH-FP-NEXT:    ld s10, 64(sp) # 8-byte Folded Reload
 ; RV64I-WITH-FP-NEXT:    ld s11, 56(sp) # 8-byte Folded Reload
 ; RV64I-WITH-FP-NEXT:    addi sp, sp, 160
 ; RV64I-WITH-FP-NEXT:    ret
+;
+; RV64IZCMP-LABEL: callee:
+; RV64IZCMP:       # %bb.0:
+; RV64IZCMP-NEXT:    cm.push {ra, s0-s11}, -160
+; RV64IZCMP-NEXT:    lui a7, %hi(var)
+; RV64IZCMP-NEXT:    lw a0, %lo(var)(a7)
+; RV64IZCMP-NEXT:    sd a0, 48(sp) # 8-byte Folded Spill
+; RV64IZCMP-NEXT:    lw a0, %lo(var+4)(a7)
+; RV64IZCMP-NEXT:    sd a0, 40(sp) # 8-byte Folded Spill
+; RV64IZCMP-NEXT:    lw a0, %lo(var+8)(a7)
+; RV64IZCMP-NEXT:    sd a0, 32(sp) # 8-byte Folded Spill
+; RV64IZCMP-NEXT:    lw a0, %lo(var+12)(a7)
+; RV64IZCMP-NEXT:    sd a0, 24(sp) # 8-byte Folded Spill
+; RV64IZCMP-NEXT:    addi a5, a7, %lo(var)
+; RV64IZCMP-NEXT:    lw a0, 16(a5)
+; RV64IZCMP-NEXT:    sd a0, 16(sp) # 8-byte Folded Spill
+; RV64IZCMP-NEXT:    lw a0, 20(a5)
+; RV64IZCMP-NEXT:    sd a0, 8(sp) # 8-byte Folded Spill
+; RV64IZCMP-NEXT:    lw t4, 24(a5)
+; RV64IZCMP-NEXT:    lw t5, 28(a5)
+; RV64IZCMP-NEXT:    lw t6, 32(a5)
+; RV64IZCMP-NEXT:    lw s2, 36(a5)
+; RV64IZCMP-NEXT:    lw s3, 40(a5)
+; RV64IZCMP-NEXT:    lw s4, 44(a5)
+; RV64IZCMP-NEXT:    lw s5, 48(a5)
+; RV64IZCMP-NEXT:    lw s6, 52(a5)
+; RV64IZCMP-NEXT:    lw s7, 56(a5)
+; RV64IZCMP-NEXT:    lw s8, 60(a5)
+; RV64IZCMP-NEXT:    lw s9, 64(a5)
+; RV64IZCMP-NEXT:    lw s10, 68(a5)
+; RV64IZCMP-NEXT:    lw s11, 72(a5)
+; RV64IZCMP-NEXT:    lw ra, 76(a5)
+; RV64IZCMP-NEXT:    lw s1, 80(a5)
+; RV64IZCMP-NEXT:    lw t3, 84(a5)
+; RV64IZCMP-NEXT:    lw t2, 88(a5)
+; RV64IZCMP-NEXT:    lw t1, 92(a5)
+; RV64IZCMP-NEXT:    lw t0, 96(a5)
+; RV64IZCMP-NEXT:    lw s0, 100(a5)
+; RV64IZCMP-NEXT:    lw a6, 104(a5)
+; RV64IZCMP-NEXT:    lw a4, 108(a5)
+; RV64IZCMP-NEXT:    lw a0, 124(a5)
+; RV64IZCMP-NEXT:    lw a1, 120(a5)
+; RV64IZCMP-NEXT:    lw a2, 116(a5)
+; RV64IZCMP-NEXT:    lw a3, 112(a5)
+; RV64IZCMP-NEXT:    sw a0, 124(a5)
+; RV64IZCMP-NEXT:    sw a1, 120(a5)
+; RV64IZCMP-NEXT:    sw a2, 116(a5)
+; RV64IZCMP-NEXT:    sw a3, 112(a5)
+; RV64IZCMP-NEXT:    sw a4, 108(a5)
+; RV64IZCMP-NEXT:    sw a6, 104(a5)
+; RV64IZCMP-NEXT:    sw s0, 100(a5)
+; RV64IZCMP-NEXT:    sw t0, 96(a5)
+; RV64IZCMP-NEXT:    sw t1, 92(a5)
+; RV64IZCMP-NEXT:    sw t2, 88(a5)
+; RV64IZCMP-NEXT:    sw t3, 84(a5)
+; RV64IZCMP-NEXT:    sw s1, 80(a5)
+; RV64IZCMP-NEXT:    sw ra, 76(a5)
+; RV64IZCMP-NEXT:    sw s11, 72(a5)
+; RV64IZCMP-NEXT:    sw s10, 68(a5)
+; RV64IZCMP-NEXT:    sw s9, 64(a5)
+; RV64IZCMP-NEXT:    sw s8, 60(a5)
+; RV64IZCMP-NEXT:    sw s7, 56(a5)
+; RV64IZCMP-NEXT:    sw s6, 52(a5)
+; RV64IZCMP-NEXT:    sw s5, 48(a5)
+; RV64IZCMP-NEXT:    sw s4, 44(a5)
+; RV64IZCMP-NEXT:    sw s3, 40(a5)
+; RV64IZCMP-NEXT:    sw s2, 36(a5)
+; RV64IZCMP-NEXT:    sw t6, 32(a5)
+; RV64IZCMP-NEXT:    sw t5, 28(a5)
+; RV64IZCMP-NEXT:    sw t4, 24(a5)
+; RV64IZCMP-NEXT:    ld a0, 8(sp) # 8-byte Folded Reload
+; RV64IZCMP-NEXT:    sw a0, 20(a5)
+; RV64IZCMP-NEXT:    ld a0, 16(sp) # 8-byte Folded Reload
+; RV64IZCMP-NEXT:    sw a0, 16(a5)
+; RV64IZCMP-NEXT:    ld a0, 24(sp) # 8-byte Folded Reload
+; RV64IZCMP-NEXT:    sw a0, %lo(var+12)(a7)
+; RV64IZCMP-NEXT:    ld a0, 32(sp) # 8-byte Folded Reload
+; RV64IZCMP-NEXT:    sw a0, %lo(var+8)(a7)
+; RV64IZCMP-NEXT:    ld a0, 40(sp) # 8-byte Folded Reload
+; RV64IZCMP-NEXT:    sw a0, %lo(var+4)(a7)
+; RV64IZCMP-NEXT:    ld a0, 48(sp) # 8-byte Folded Reload
+; RV64IZCMP-NEXT:    sw a0, %lo(var)(a7)
+; RV64IZCMP-NEXT:    cm.popret {ra, s0-s11}, 160
+;
+; RV64IZCMP-WITH-FP-LABEL: callee:
+; RV64IZCMP-WITH-FP:       # %bb.0:
+; RV64IZCMP-WITH-FP-NEXT:    addi sp, sp, -160
+; RV64IZCMP-WITH-FP-NEXT:    sd ra, 152(sp) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    sd s0, 144(sp) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    sd s1, 136(sp) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    sd s2, 128(sp) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    sd s3, 120(sp) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    sd s4, 112(sp) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    sd s5, 104(sp) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    sd s6, 96(sp) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    sd s7, 88(sp) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    sd s8, 80(sp) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    sd s9, 72(sp) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    sd s10, 64(sp) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    sd s11, 56(sp) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    addi s0, sp, 160
+; RV64IZCMP-WITH-FP-NEXT:    lui a7, %hi(var)
+; RV64IZCMP-WITH-FP-NEXT:    lw a0, %lo(var)(a7)
+; RV64IZCMP-WITH-FP-NEXT:    sd a0, -112(s0) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    lw a0, %lo(var+4)(a7)
+; RV64IZCMP-WITH-FP-NEXT:    sd a0, -120(s0) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    lw a0, %lo(var+8)(a7)
+; RV64IZCMP-WITH-FP-NEXT:    sd a0, -128(s0) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    lw a0, %lo(var+12)(a7)
+; RV64IZCMP-WITH-FP-NEXT:    sd a0, -136(s0) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    addi a5, a7, %lo(var)
+; RV64IZCMP-WITH-FP-NEXT:    lw a0, 16(a5)
+; RV64IZCMP-WITH-FP-NEXT:    sd a0, -144(s0) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    lw a0, 20(a5)
+; RV64IZCMP-WITH-FP-NEXT:    sd a0, -152(s0) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    lw a0, 24(a5)
+; RV64IZCMP-WITH-FP-NEXT:    sd a0, -160(s0) # 8-byte Folded Spill
+; RV64IZCMP-WITH-FP-NEXT:    lw t5, 28(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw t6, 32(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw s2, 36(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw s3, 40(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw s4, 44(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw s5, 48(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw s6, 52(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw s7, 56(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw s8, 60(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw s9, 64(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw s10, 68(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw s11, 72(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw ra, 76(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw t4, 80(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw t3, 84(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw t2, 88(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw s1, 92(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw t1, 96(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw t0, 100(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw a6, 104(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw a4, 108(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw a0, 124(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw a1, 120(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw a2, 116(a5)
+; RV64IZCMP-WITH-FP-NEXT:    lw a3, 112(a5)
+; RV64IZCMP-WITH-FP-NEXT:    sw a0, 124(a5)</