[RISCV] Add a pass to combine `cm.pop` and `ret` insts

[VincentWu]

Summary

RISCVPushPopOptimizer.cpp combine cm.pop and ret to generates cm.popretz or cm.popret .

[KYG]

llvm/lib/Target/RISCV/RISCVInstrInfo.h

134 ↗

(On Diff #521526)

Could we just use isCopyInstrImpl instead?
Since it's doing almost the same thing, and there's no prototype in TargetInstrInfo.h.

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfo.h

134 ↗

(On Diff #521526)

Agreed. Can we use isCopyInstrImpl and check for a copy from X0?

llvm/lib/Target/RISCV/RISCVPushPopOptimizer.cpp

42

RetVal map should be capitalized.

And this should probably be a DenseMap.

135

use Fn.getSubtarget<RISCVSubtarget>

136

Drop curly braces

140

The line break in this comment should be closer to column 80.

141

How does it break the ABI?

142

Drop curly braces

158

If it's not a return block, and adjustRetVal deletes the LI instruction, where does the return value go?

llvm/lib/Target/RISCV/RISCVTargetMachine.cpp

351

Should this be skipped at -O0?

[VincentWu]

address comments

[craig.topper]

llvm/lib/Target/RISCV/RISCVPushPopOptimizer.cpp

33

Subtarget doesn't need to be a member. It's only used in runOnMachineFunction

42

Capitalize variable names

71

Why does this need to be map? This function and adjustRetVal are called on the same basic block. There can't be values for any other basic block in the map. You could probably just pass the bool result from adjustRetVal to this function?

76

If the value exists, it must be zero right?

109

Isn't MI.getOperand(2).getImm() == 0 guaranteed by TII->isCopyInstrImpl(MI)?

141

This question was not answered.

151

I think you need to know the terminator is PseudoRET before you can call adjustRetVal. PseudoTAIL, PseudoTAILIndirect, SRET, and MRET also have isReturn but are not eligible for usePopRet.

[jrtc27]

llvm/lib/Target/RISCV/RISCVPushPopOptimizer.cpp

141

https://github.com/riscv/riscv-code-size-reduction/issues/194 and https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/18#issuecomment-1304496707 onwards

[craig.topper]

llvm/lib/Target/RISCV/RISCVPushPopOptimizer.cpp

141

As far as I can tell, this patch just fuses the cm.pop with ret. Wouldn't the ABI break occur at the creation of cm.pop not at the formation of cm.popret?

[jrtc27]

llvm/lib/Target/RISCV/RISCVPushPopOptimizer.cpp

141

Yes, using cm.push/pop is the issue, not fusing ones that already exist. Haven't looked at the content of the patch other than the few lines of context here.

[VincentWu]

llvm/lib/Target/RISCV/RISCVPushPopOptimizer.cpp

141

it was mentioned here https://github.com/plctlab/llvm-project/issues/58 ,
but I'm not sure whether it is the best solution

141

yes, So you can find the same code in the patch D134599#change-ZdluzfeA7hef

This code here is just a quick exit if it doesn't match the condition

151

Yeah you are right, that's why I have whether NextI->getOpcode() == RISCV::PseudoRET at func adjustRetVal.

I will move it to here )

[VincentWu]

llvm/lib/Target/RISCV/RISCVPushPopOptimizer.cpp

141

it was mentioned here https://github.com/plctlab/llvm-project/issues/58 ,
but I'm not sure whether it is the best solution

Sorry for late reply, I forgot to submit it

[craig.topper]

I see comments are marked done, but the patch wasn't updated?

[VincentWu]

address comments

[craig.topper]

LGTM

[craig.topper]

llvm/lib/Target/RISCV/RISCVPushPopOptimizer.cpp

68

Can we do?

unsigned Opc = IsReturnZero ? RISCV::CM_POPRETZ : RISCV::CM_POPRET;
BuildMI(*NextI->getParent(), NextI, DL, TII->get(Opc))
        .add(MBBI->getOperand(0))
        .add(MBBI->getOperand(1));

[VincentWu]

address comments

[VincentWu]

rebase & update testcase
ready to land

[VincentWu]

format

llvm/lib/Target/RISCV/CMakeLists.txt

add_llvm_target(RISCVCodeGen
   RISCVISelDAGToDAG.cpp
   RISCVISelLowering.cpp
   RISCVMachineFunctionInfo.cpp
   RISCVMacroFusion.cpp
   RISCVMCInstLower.cpp
   RISCVMergeBaseOffset.cpp
   RISCVOptWInstrs.cpp
   RISCVRedundantCopyElimination.cpp
+  RISCVPushPopOptimizer.cpp
   RISCVRegisterInfo.cpp
   RISCVRVVInitUndef.cpp
   RISCVSubtarget.cpp
   RISCVTargetMachine.cpp
   RISCVTargetObjectFile.cpp
   RISCVTargetTransformInfo.cpp
   GISel/RISCVCallLowering.cpp
   GISel/RISCVInstructionSelector.cpp

llvm/lib/Target/RISCV/RISCV.h

 
 FunctionPass *createRISCVRedundantCopyEliminationPass();
 void initializeRISCVRedundantCopyEliminationPass(PassRegistry &);
 
 FunctionPass *createRISCVInitUndefPass();
 void initializeRISCVInitUndefPass(PassRegistry &);
 extern char &RISCVInitUndefID;
 
+FunctionPass *createRISCVPushPopOptimizationPass();
+void initializeRISCVPushPopOptPass(PassRegistry &);
+
 InstructionSelector *createRISCVInstructionSelector(const RISCVTargetMachine &,
                                                     RISCVSubtarget &,
                                                     RISCVRegisterBankInfo &);
 void initializeRISCVDAGToDAGISelPass(PassRegistry &);
 } // namespace llvm
 
 #endif

llvm/lib/Target/RISCV/RISCVPushPopOptimizer.cpp

+//===------- RISCVPushPopOptimizer.cpp - RISCV Push/Pop opt. pass ---------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains a pass that modifies PUSH/POP instructions from Zca
+// standard to use their non prolog/epilog related functionalities
+// and generates POPRET instruction.
+//
+//===----------------------------------------------------------------------===//
+
+#include "RISCVInstrInfo.h"
+#include "RISCVMachineFunctionInfo.h"
+
+using namespace llvm;
+
+#define RISCV_PUSH_POP_OPT_NAME "RISC-V Zcmp Push/Pop optimization pass"
+
+namespace {
+struct RISCVPushPopOpt : public MachineFunctionPass {
+  static char ID;
+
+  RISCVPushPopOpt() : MachineFunctionPass(ID) {
+    initializeRISCVPushPopOptPass(*PassRegistry::getPassRegistry());
+  }
+
+  const RISCVInstrInfo *TII;
+  const TargetRegisterInfo *TRI;
+
+  // Track which register units have been modified and used.
+  LiveRegUnits ModifiedRegUnits, UsedRegUnits;
+
+  bool usePopRet(MachineBasicBlock::iterator &MBBI,
+                 MachineBasicBlock::iterator &NextI, bool IsReturnZero);
+  bool adjustRetVal(MachineBasicBlock::iterator &MBBI);
+  bool runOnMachineFunction(MachineFunction &Fn) override;
+
+  StringRef getPassName() const override { return RISCV_PUSH_POP_OPT_NAME; }
+};
+
+char RISCVPushPopOpt::ID = 0;
+
+} // end of anonymous namespace
+
+INITIALIZE_PASS(RISCVPushPopOpt, "riscv-push-pop-opt", RISCV_PUSH_POP_OPT_NAME,
+                false, false)
+
+// Check if POP instruction was inserted into the MBB and return iterator to it.
+static MachineBasicBlock::iterator containsPop(MachineBasicBlock &MBB) {
+  for (MachineBasicBlock::iterator MBBI = MBB.begin(); MBBI != MBB.end();
+       MBBI = next_nodbg(MBBI, MBB.end()))
+    if (MBBI->getOpcode() == RISCV::CM_POP)
+      return MBBI;
+
+  return MBB.end();
+}
+
+bool RISCVPushPopOpt::usePopRet(MachineBasicBlock::iterator &MBBI,
+                                MachineBasicBlock::iterator &NextI,
+                                bool IsReturnZero) {
+  // Since Pseudo instruction lowering happen later in the pipeline,
+  // this will detect all ret instruction.
+  DebugLoc DL = NextI->getDebugLoc();
+  unsigned Opc = IsReturnZero ? RISCV::CM_POPRETZ : RISCV::CM_POPRET;
+  BuildMI(*NextI->getParent(), NextI, DL, TII->get(Opc))
+      .add(MBBI->getOperand(0))
+      .add(MBBI->getOperand(1));
+
+  MBBI->eraseFromParent();
+  NextI->eraseFromParent();
+  return true;
+}
+
+// Search for last assignment to a0 and if possible use ret_val slot of POP to
+// store return value.
+bool RISCVPushPopOpt::adjustRetVal(MachineBasicBlock::iterator &MBBI) {
+  MachineBasicBlock::reverse_iterator RE = MBBI->getParent()->rend();
+  // Track which register units have been modified and used between the POP
+  // insn and the last assignment to register a0.
+  ModifiedRegUnits.clear();
+  UsedRegUnits.clear();
+  // Since POP instruction is in Epilogue no normal instructions will follow
+  // after it. Therefore search only previous ones to find the return value.
+  for (MachineBasicBlock::reverse_iterator I =
+           next_nodbg(MBBI.getReverse(), RE);
+       I != RE; I = next_nodbg(I, RE)) {
+    MachineInstr &MI = *I;
+    if (auto OperandPair = TII->isCopyInstrImpl(MI)) {
+      Register DestReg = OperandPair->Destination->getReg();
+      Register Source = OperandPair->Source->getReg();
+      if (DestReg == RISCV::X10 && Source == RISCV::X0) {
+        MI.removeFromParent();
+        return true;
+      }
+    }
+    // Update modified / used register units.
+    LiveRegUnits::accumulateUsedDefed(MI, ModifiedRegUnits, UsedRegUnits, TRI);
+    // If a0 was modified or used, there is no possibility
+    // of using ret_val slot of POP instruction.
+    if (!ModifiedRegUnits.available(RISCV::X10) ||
+        !UsedRegUnits.available(RISCV::X10))
+      return false;
+  }
+  return false;
+}
+
+bool RISCVPushPopOpt::runOnMachineFunction(MachineFunction &Fn) {
+  if (skipFunction(Fn.getFunction()))
+    return false;
+
+  // If Zcmp extension is not supported, abort.
+  const RISCVSubtarget *Subtarget = &Fn.getSubtarget<RISCVSubtarget>();
+  if (!Subtarget->hasStdExtZcmp())
+    return false;
+
+  // If frame pointer elimination has been disabled, abort to avoid breaking the
+  // ABI.
+  if (Fn.getTarget().Options.DisableFramePointerElim(Fn))
+    return false;
+
+  TII = static_cast<const RISCVInstrInfo *>(Subtarget->getInstrInfo());
+  TRI = Subtarget->getRegisterInfo();
+  // Resize the modified and used register unit trackers.  We do this once
+  // per function and then clear the register units each time we determine
+  // correct return value for the POP.
+  ModifiedRegUnits.init(*TRI);
+  UsedRegUnits.init(*TRI);
+  bool Modified = false;
+  for (auto &MBB : Fn) {
+    MachineBasicBlock::iterator MBBI = containsPop(MBB);
+    MachineBasicBlock::iterator NextI = next_nodbg(MBBI, MBB.end());
+    if (MBBI != MBB.end() && NextI->getOpcode() == RISCV::PseudoRET)
+      Modified |= usePopRet(MBBI, NextI, adjustRetVal(MBBI));
+  }
+  return Modified;
+}
+
+/// createRISCVPushPopOptimizationPass - returns an instance of the
+/// Push/Pop optimization pass.
+FunctionPass *llvm::createRISCVPushPopOptimizationPass() {
+  return new RISCVPushPopOpt();
+}

llvm/lib/Target/RISCV/RISCVTargetMachine.cpp

extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeRISCVTarget() {
   initializeRISCVCodeGenPreparePass(*PR);
   initializeRISCVMergeBaseOffsetOptPass(*PR);
   initializeRISCVOptWInstrsPass(*PR);
   initializeRISCVPreRAExpandPseudoPass(*PR);
   initializeRISCVExpandPseudoPass(*PR);
   initializeRISCVInsertVSETVLIPass(*PR);
   initializeRISCVDAGToDAGISelPass(*PR);
   initializeRISCVInitUndefPass(*PR);
+  initializeRISCVPushPopOptPass(*PR);
 }
 
 static StringRef computeDataLayout(const Triple &TT) {
   if (TT.isArch64Bit())
     return "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128";
   assert(TT.isArch32Bit() && "only RV32 and RV64 are currently supported");
   return "e-m:e-p:32:32-i64:64-n32-S128";
 }
void RISCVPassConfig::addPreEmitPass() {
   // propagation after the machine outliner (which runs after addPreEmitPass)
   // currently leads to incorrect code-gen, where copies to registers within
   // outlined functions are removed erroneously.
   if (TM->getOptLevel() >= CodeGenOpt::Default && EnableRISCVCopyPropagation)
     addPass(createMachineCopyPropagationPass(true));
 }
 
 void RISCVPassConfig::addPreEmitPass2() {
+  // Schedule PushPop Optimization before expansion of Pseudo instruction,
+  // ensuring return instruction is detected correctly.
+  if (TM->getOptLevel() != CodeGenOpt::None)
+    addPass(createRISCVPushPopOptimizationPass());
   addPass(createRISCVExpandPseudoPass());
 
   // Schedule the expansion of AMOs at the last possible moment, avoiding the
   // possibility for other passes to break the requirements for forward
   // progress in the LR/SC block.
   addPass(createRISCVExpandAtomicPseudoPass());
 }
 

llvm/test/CodeGen/RISCV/O3-pipeline.ll

 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:     Machine Outliner
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Stack Frame Layout Analysis
+; CHECK-NEXT:       RISC-V Zcmp Push/Pop optimization pass 
 ; CHECK-NEXT:       RISC-V pseudo instruction expansion pass
 ; CHECK-NEXT:       RISC-V atomic pseudo instruction expansion pass
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       RISC-V Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction

llvm/test/CodeGen/RISCV/callee-saved-gprs.ll

 ; RV32IZCMP-NEXT:    lw a0, 12(sp) # 4-byte Folded Reload
 ; RV32IZCMP-NEXT:    sw a0, %lo(var+12)(a7)
 ; RV32IZCMP-NEXT:    lw a0, 16(sp) # 4-byte Folded Reload
 ; RV32IZCMP-NEXT:    sw a0, %lo(var+8)(a7)
 ; RV32IZCMP-NEXT:    lw a0, 20(sp) # 4-byte Folded Reload
 ; RV32IZCMP-NEXT:    sw a0, %lo(var+4)(a7)
 ; RV32IZCMP-NEXT:    lw a0, 24(sp) # 4-byte Folded Reload
 ; RV32IZCMP-NEXT:    sw a0, %lo(var)(a7)
-; RV32IZCMP-NEXT:    cm.pop {ra, s0-s11}, 80
-; RV32IZCMP-NEXT:    ret
+; RV32IZCMP-NEXT:    cm.popret {ra, s0-s11}, 80
 ;
 ; RV32IZCMP-WITH-FP-LABEL: callee:
 ; RV32IZCMP-WITH-FP:       # %bb.0:
 ; RV32IZCMP-WITH-FP-NEXT:    addi sp, sp, -80
 ; RV32IZCMP-WITH-FP-NEXT:    sw ra, 76(sp) # 4-byte Folded Spill
 ; RV32IZCMP-WITH-FP-NEXT:    sw s0, 72(sp) # 4-byte Folded Spill
 ; RV32IZCMP-WITH-FP-NEXT:    sw s1, 68(sp) # 4-byte Folded Spill
 ; RV32IZCMP-WITH-FP-NEXT:    sw s2, 64(sp) # 4-byte Folded Spill

 ; RV64IZCMP-NEXT:    ld a0, 24(sp) # 8-byte Folded Reload
 ; RV64IZCMP-NEXT:    sw a0, %lo(var+12)(a7)
 ; RV64IZCMP-NEXT:    ld a0, 32(sp) # 8-byte Folded Reload
 ; RV64IZCMP-NEXT:    sw a0, %lo(var+8)(a7)
 ; RV64IZCMP-NEXT:    ld a0, 40(sp) # 8-byte Folded Reload
 ; RV64IZCMP-NEXT:    sw a0, %lo(var+4)(a7)
 ; RV64IZCMP-NEXT:    ld a0, 48(sp) # 8-byte Folded Reload
 ; RV64IZCMP-NEXT:    sw a0, %lo(var)(a7)
-; RV64IZCMP-NEXT:    cm.pop {ra, s0-s11}, 160
-; RV64IZCMP-NEXT:    ret
+; RV64IZCMP-NEXT:    cm.popret {ra, s0-s11}, 160
 ;
 ; RV64IZCMP-WITH-FP-LABEL: callee:
 ; RV64IZCMP-WITH-FP:       # %bb.0:
 ; RV64IZCMP-WITH-FP-NEXT:    addi sp, sp, -160
 ; RV64IZCMP-WITH-FP-NEXT:    sd ra, 152(sp) # 8-byte Folded Spill
 ; RV64IZCMP-WITH-FP-NEXT:    sd s0, 144(sp) # 8-byte Folded Spill
 ; RV64IZCMP-WITH-FP-NEXT:    sd s1, 136(sp) # 8-byte Folded Spill
 ; RV64IZCMP-WITH-FP-NEXT:    sd s2, 128(sp) # 8-byte Folded Spill

 ; RV32IZCMP-NEXT:    sw a0, %lo(var+12)(s0)
 ; RV32IZCMP-NEXT:    lw a0, 80(sp) # 4-byte Folded Reload
 ; RV32IZCMP-NEXT:    sw a0, %lo(var+8)(s0)
 ; RV32IZCMP-NEXT:    lw a0, 84(sp) # 4-byte Folded Reload
 ; RV32IZCMP-NEXT:    sw a0, %lo(var+4)(s0)
 ; RV32IZCMP-NEXT:    lw a0, 88(sp) # 4-byte Folded Reload
 ; RV32IZCMP-NEXT:    sw a0, %lo(var)(s0)
 ; RV32IZCMP-NEXT:    addi sp, sp, 32
-; RV32IZCMP-NEXT:    cm.pop {ra, s0-s11}, 112
-; RV32IZCMP-NEXT:    ret
+; RV32IZCMP-NEXT:    cm.popret {ra, s0-s11}, 112
 ;
 ; RV32IZCMP-WITH-FP-LABEL: caller:
 ; RV32IZCMP-WITH-FP:       # %bb.0:
 ; RV32IZCMP-WITH-FP-NEXT:    addi sp, sp, -144
 ; RV32IZCMP-WITH-FP-NEXT:    sw ra, 140(sp) # 4-byte Folded Spill
 ; RV32IZCMP-WITH-FP-NEXT:    sw s0, 136(sp) # 4-byte Folded Spill
 ; RV32IZCMP-WITH-FP-NEXT:    sw s1, 132(sp) # 4-byte Folded Spill
 ; RV32IZCMP-WITH-FP-NEXT:    sw s2, 128(sp) # 4-byte Folded Spill

 ; RV64IZCMP-NEXT:    sw a0, %lo(var+12)(s0)
 ; RV64IZCMP-NEXT:    ld a0, 160(sp) # 8-byte Folded Reload
 ; RV64IZCMP-NEXT:    sw a0, %lo(var+8)(s0)
 ; RV64IZCMP-NEXT:    ld a0, 168(sp) # 8-byte Folded Reload
 ; RV64IZCMP-NEXT:    sw a0, %lo(var+4)(s0)
 ; RV64IZCMP-NEXT:    ld a0, 176(sp) # 8-byte Folded Reload
 ; RV64IZCMP-NEXT:    sw a0, %lo(var)(s0)
 ; RV64IZCMP-NEXT:    addi sp, sp, 128
-; RV64IZCMP-NEXT:    cm.pop {ra, s0-s11}, 160
-; RV64IZCMP-NEXT:    ret
+; RV64IZCMP-NEXT:    cm.popret {ra, s0-s11}, 160
 ;
 ; RV64IZCMP-WITH-FP-LABEL: caller:
 ; RV64IZCMP-WITH-FP:       # %bb.0:
 ; RV64IZCMP-WITH-FP-NEXT:    addi sp, sp, -288
 ; RV64IZCMP-WITH-FP-NEXT:    sd ra, 280(sp) # 8-byte Folded Spill
 ; RV64IZCMP-WITH-FP-NEXT:    sd s0, 272(sp) # 8-byte Folded Spill
 ; RV64IZCMP-WITH-FP-NEXT:    sd s1, 264(sp) # 8-byte Folded Spill
 ; RV64IZCMP-WITH-FP-NEXT:    sd s2, 256(sp) # 8-byte Folded Spill

llvm/test/CodeGen/RISCV/push-pop-popret.ll

 ; RV32IZCMP-LABEL: foo:
 ; RV32IZCMP:       # %bb.0:
 ; RV32IZCMP-NEXT:    cm.push {ra}, -64
 ; RV32IZCMP-NEXT:    .cfi_def_cfa_offset 528
 ; RV32IZCMP-NEXT:    addi sp, sp, -464
 ; RV32IZCMP-NEXT:    .cfi_offset ra, -4
 ; RV32IZCMP-NEXT:    addi a0, sp, 12
 ; RV32IZCMP-NEXT:    call test@plt
-; RV32IZCMP-NEXT:    li a0, 0
 ; RV32IZCMP-NEXT:    addi sp, sp, 464
-; RV32IZCMP-NEXT:    cm.pop {ra}, 64
-; RV32IZCMP-NEXT:    ret
+; RV32IZCMP-NEXT:    cm.popretz {ra}, 64
 ;
 ; RV64IZCMP-LABEL: foo:
 ; RV64IZCMP:       # %bb.0:
 ; RV64IZCMP-NEXT:    cm.push {ra}, -64
 ; RV64IZCMP-NEXT:    .cfi_def_cfa_offset 528
 ; RV64IZCMP-NEXT:    addi sp, sp, -464
 ; RV64IZCMP-NEXT:    .cfi_offset ra, -8
 ; RV64IZCMP-NEXT:    addi a0, sp, 8
 ; RV64IZCMP-NEXT:    call test@plt
-; RV64IZCMP-NEXT:    li a0, 0
 ; RV64IZCMP-NEXT:    addi sp, sp, 464
-; RV64IZCMP-NEXT:    cm.pop {ra}, 64
-; RV64IZCMP-NEXT:    ret
+; RV64IZCMP-NEXT:    cm.popretz {ra}, 64
 ;
 ; RV32I-LABEL: foo:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -528
 ; RV32I-NEXT:    .cfi_def_cfa_offset 528
 ; RV32I-NEXT:    sw ra, 524(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    .cfi_offset ra, -4
 ; RV32I-NEXT:    addi a0, sp, 12

 ; RV32IZCMP-NEXT:    .cfi_offset s0, -8
 ; RV32IZCMP-NEXT:    addi s0, sp, 16
 ; RV32IZCMP-NEXT:    .cfi_def_cfa s0, 0
 ; RV32IZCMP-NEXT:    addi a0, a0, 15
 ; RV32IZCMP-NEXT:    andi a0, a0, -16
 ; RV32IZCMP-NEXT:    sub a0, sp, a0
 ; RV32IZCMP-NEXT:    mv sp, a0
 ; RV32IZCMP-NEXT:    call callee_void@plt
-; RV32IZCMP-NEXT:    li a0, 0
 ; RV32IZCMP-NEXT:    addi sp, s0, -16
-; RV32IZCMP-NEXT:    cm.pop {ra, s0}, 16
-; RV32IZCMP-NEXT:    ret
+; RV32IZCMP-NEXT:    cm.popretz {ra, s0}, 16
 ;
 ; RV64IZCMP-LABEL: pushpopret0:
 ; RV64IZCMP:       # %bb.0: # %entry
 ; RV64IZCMP-NEXT:    .cfi_def_cfa_offset 16
 ; RV64IZCMP-NEXT:    cm.push {ra, s0}, -16
 ; RV64IZCMP-NEXT:    .cfi_offset ra, -8
 ; RV64IZCMP-NEXT:    .cfi_offset s0, -16
 ; RV64IZCMP-NEXT:    addi s0, sp, 16
 ; RV64IZCMP-NEXT:    .cfi_def_cfa s0, 0
 ; RV64IZCMP-NEXT:    slli a0, a0, 32
 ; RV64IZCMP-NEXT:    srli a0, a0, 32
 ; RV64IZCMP-NEXT:    addi a0, a0, 15
 ; RV64IZCMP-NEXT:    andi a0, a0, -16
 ; RV64IZCMP-NEXT:    sub a0, sp, a0
 ; RV64IZCMP-NEXT:    mv sp, a0
 ; RV64IZCMP-NEXT:    call callee_void@plt
-; RV64IZCMP-NEXT:    li a0, 0
 ; RV64IZCMP-NEXT:    addi sp, s0, -16
-; RV64IZCMP-NEXT:    cm.pop {ra, s0}, 16
-; RV64IZCMP-NEXT:    ret
+; RV64IZCMP-NEXT:    cm.popretz {ra, s0}, 16
 ;
 ; RV32I-LABEL: pushpopret0:
 ; RV32I:       # %bb.0: # %entry
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    .cfi_def_cfa_offset 16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    sw s0, 8(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    .cfi_offset ra, -4

 ; RV32IZCMP-NEXT:    .cfi_def_cfa s0, 0
 ; RV32IZCMP-NEXT:    addi a0, a0, 15
 ; RV32IZCMP-NEXT:    andi a0, a0, -16
 ; RV32IZCMP-NEXT:    sub a0, sp, a0
 ; RV32IZCMP-NEXT:    mv sp, a0
 ; RV32IZCMP-NEXT:    call callee_void@plt
 ; RV32IZCMP-NEXT:    li a0, 1
 ; RV32IZCMP-NEXT:    addi sp, s0, -16
-; RV32IZCMP-NEXT:    cm.pop {ra, s0}, 16
-; RV32IZCMP-NEXT:    ret
+; RV32IZCMP-NEXT:    cm.popret {ra, s0}, 16
 ;
 ; RV64IZCMP-LABEL: pushpopret1:
 ; RV64IZCMP:       # %bb.0: # %entry
 ; RV64IZCMP-NEXT:    .cfi_def_cfa_offset 16
 ; RV64IZCMP-NEXT:    cm.push {ra, s0}, -16
 ; RV64IZCMP-NEXT:    .cfi_offset ra, -8
 ; RV64IZCMP-NEXT:    .cfi_offset s0, -16
 ; RV64IZCMP-NEXT:    addi s0, sp, 16
 ; RV64IZCMP-NEXT:    .cfi_def_cfa s0, 0
 ; RV64IZCMP-NEXT:    slli a0, a0, 32
 ; RV64IZCMP-NEXT:    srli a0, a0, 32
 ; RV64IZCMP-NEXT:    addi a0, a0, 15
 ; RV64IZCMP-NEXT:    andi a0, a0, -16
 ; RV64IZCMP-NEXT:    sub a0, sp, a0
 ; RV64IZCMP-NEXT:    mv sp, a0
 ; RV64IZCMP-NEXT:    call callee_void@plt
 ; RV64IZCMP-NEXT:    li a0, 1
 ; RV64IZCMP-NEXT:    addi sp, s0, -16
-; RV64IZCMP-NEXT:    cm.pop {ra, s0}, 16
-; RV64IZCMP-NEXT:    ret
+; RV64IZCMP-NEXT:    cm.popret {ra, s0}, 16
 ;
 ; RV32I-LABEL: pushpopret1:
 ; RV32I:       # %bb.0: # %entry
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    .cfi_def_cfa_offset 16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    sw s0, 8(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    .cfi_offset ra, -4

 ; RV32IZCMP-NEXT:    .cfi_def_cfa s0, 0
 ; RV32IZCMP-NEXT:    addi a0, a0, 15
 ; RV32IZCMP-NEXT:    andi a0, a0, -16
 ; RV32IZCMP-NEXT:    sub a0, sp, a0
 ; RV32IZCMP-NEXT:    mv sp, a0
 ; RV32IZCMP-NEXT:    call callee_void@plt
 ; RV32IZCMP-NEXT:    li a0, -1
 ; RV32IZCMP-NEXT:    addi sp, s0, -16
-; RV32IZCMP-NEXT:    cm.pop {ra, s0}, 16
-; RV32IZCMP-NEXT:    ret
+; RV32IZCMP-NEXT:    cm.popret {ra, s0}, 16
 ;
 ; RV64IZCMP-LABEL: pushpopretneg1:
 ; RV64IZCMP:       # %bb.0: # %entry
 ; RV64IZCMP-NEXT:    .cfi_def_cfa_offset 16
 ; RV64IZCMP-NEXT:    cm.push {ra, s0}, -16
 ; RV64IZCMP-NEXT:    .cfi_offset ra, -8
 ; RV64IZCMP-NEXT:    .cfi_offset s0, -16
 ; RV64IZCMP-NEXT:    addi s0, sp, 16
 ; RV64IZCMP-NEXT:    .cfi_def_cfa s0, 0
 ; RV64IZCMP-NEXT:    slli a0, a0, 32
 ; RV64IZCMP-NEXT:    srli a0, a0, 32
 ; RV64IZCMP-NEXT:    addi a0, a0, 15
 ; RV64IZCMP-NEXT:    andi a0, a0, -16
 ; RV64IZCMP-NEXT:    sub a0, sp, a0
 ; RV64IZCMP-NEXT:    mv sp, a0
 ; RV64IZCMP-NEXT:    call callee_void@plt
 ; RV64IZCMP-NEXT:    li a0, -1
 ; RV64IZCMP-NEXT:    addi sp, s0, -16
-; RV64IZCMP-NEXT:    cm.pop {ra, s0}, 16
-; RV64IZCMP-NEXT:    ret
+; RV64IZCMP-NEXT:    cm.popret {ra, s0}, 16
 ;
 ; RV32I-LABEL: pushpopretneg1:
 ; RV32I:       # %bb.0: # %entry
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    .cfi_def_cfa_offset 16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    sw s0, 8(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    .cfi_offset ra, -4

 ; RV32IZCMP-NEXT:    .cfi_def_cfa s0, 0
 ; RV32IZCMP-NEXT:    addi a0, a0, 15
 ; RV32IZCMP-NEXT:    andi a0, a0, -16
 ; RV32IZCMP-NEXT:    sub a0, sp, a0
 ; RV32IZCMP-NEXT:    mv sp, a0
 ; RV32IZCMP-NEXT:    call callee_void@plt
 ; RV32IZCMP-NEXT:    li a0, 2
 ; RV32IZCMP-NEXT:    addi sp, s0, -16
-; RV32IZCMP-NEXT:    cm.pop {ra, s0}, 16
-; RV32IZCMP-NEXT:    ret
+; RV32IZCMP-NEXT:    cm.popret {ra, s0}, 16
 ;
 ; RV64IZCMP-LABEL: pushpopret2:
 ; RV64IZCMP:       # %bb.0: # %entry
 ; RV64IZCMP-NEXT:    .cfi_def_cfa_offset 16
 ; RV64IZCMP-NEXT:    cm.push {ra, s0}, -16
 ; RV64IZCMP-NEXT:    .cfi_offset ra, -8
 ; RV64IZCMP-NEXT:    .cfi_offset s0, -16
 ; RV64IZCMP-NEXT:    addi s0, sp, 16
 ; RV64IZCMP-NEXT:    .cfi_def_cfa s0, 0
 ; RV64IZCMP-NEXT:    slli a0, a0, 32
 ; RV64IZCMP-NEXT:    srli a0, a0, 32
 ; RV64IZCMP-NEXT:    addi a0, a0, 15
 ; RV64IZCMP-NEXT:    andi a0, a0, -16
 ; RV64IZCMP-NEXT:    sub a0, sp, a0
 ; RV64IZCMP-NEXT:    mv sp, a0
 ; RV64IZCMP-NEXT:    call callee_void@plt
 ; RV64IZCMP-NEXT:    li a0, 2
 ; RV64IZCMP-NEXT:    addi sp, s0, -16
-; RV64IZCMP-NEXT:    cm.pop {ra, s0}, 16
-; RV64IZCMP-NEXT:    ret
+; RV64IZCMP-NEXT:    cm.popret {ra, s0}, 16
 ;
 ; RV32I-LABEL: pushpopret2:
 ; RV32I:       # %bb.0: # %entry
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    .cfi_def_cfa_offset 16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    sw s0, 8(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    .cfi_offset ra, -4
; RV64I-NEXT:    mret
   ret void
 }
 
 define void @foo_no_irq() nounwind{
 ; RV32IZCMP-LABEL: foo_no_irq:
 ; RV32IZCMP:       # %bb.0:
 ; RV32IZCMP-NEXT:    cm.push {ra}, -16
 ; RV32IZCMP-NEXT:    call foo_test_irq@plt
-; RV32IZCMP-NEXT:    cm.pop {ra}, 16
-; RV32IZCMP-NEXT:    ret
+; RV32IZCMP-NEXT:    cm.popret {ra}, 16
 ;
 ; RV64IZCMP-LABEL: foo_no_irq:
 ; RV64IZCMP:       # %bb.0:
 ; RV64IZCMP-NEXT:    cm.push {ra}, -16
 ; RV64IZCMP-NEXT:    call foo_test_irq@plt
-; RV64IZCMP-NEXT:    cm.pop {ra}, 16
-; RV64IZCMP-NEXT:    ret
+; RV64IZCMP-NEXT:    cm.popret {ra}, 16
 ;
 ; RV32I-LABEL: foo_no_irq:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    call foo_test_irq@plt
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16

 ; RV32IZCMP-NEXT:    lw a0, 12(sp) # 4-byte Folded Reload
 ; RV32IZCMP-NEXT:    sw a0, %lo(var_test_irq+12)(a7)
 ; RV32IZCMP-NEXT:    lw a0, 16(sp) # 4-byte Folded Reload
 ; RV32IZCMP-NEXT:    sw a0, %lo(var_test_irq+8)(a7)
 ; RV32IZCMP-NEXT:    lw a0, 20(sp) # 4-byte Folded Reload
 ; RV32IZCMP-NEXT:    sw a0, %lo(var_test_irq+4)(a7)
 ; RV32IZCMP-NEXT:    lw a0, 24(sp) # 4-byte Folded Reload
 ; RV32IZCMP-NEXT:    sw a0, %lo(var_test_irq)(a7)
-; RV32IZCMP-NEXT:    cm.pop {ra, s0-s11}, 80
-; RV32IZCMP-NEXT:    ret
+; RV32IZCMP-NEXT:    cm.popret {ra, s0-s11}, 80
 ;
 ; RV64IZCMP-LABEL: callee_no_irq:
 ; RV64IZCMP:       # %bb.0:
 ; RV64IZCMP-NEXT:    cm.push {ra, s0-s11}, -160
 ; RV64IZCMP-NEXT:    lui a7, %hi(var_test_irq)
 ; RV64IZCMP-NEXT:    lw a0, %lo(var_test_irq)(a7)
 ; RV64IZCMP-NEXT:    sd a0, 48(sp) # 8-byte Folded Spill
 ; RV64IZCMP-NEXT:    lw a0, %lo(var_test_irq+4)(a7)

 ; RV64IZCMP-NEXT:    ld a0, 24(sp) # 8-byte Folded Reload
 ; RV64IZCMP-NEXT:    sw a0, %lo(var_test_irq+12)(a7)
 ; RV64IZCMP-NEXT:    ld a0, 32(sp) # 8-byte Folded Reload
 ; RV64IZCMP-NEXT:    sw a0, %lo(var_test_irq+8)(a7)
 ; RV64IZCMP-NEXT:    ld a0, 40(sp) # 8-byte Folded Reload
 ; RV64IZCMP-NEXT:    sw a0, %lo(var_test_irq+4)(a7)
 ; RV64IZCMP-NEXT:    ld a0, 48(sp) # 8-byte Folded Reload
 ; RV64IZCMP-NEXT:    sw a0, %lo(var_test_irq)(a7)
-; RV64IZCMP-NEXT:    cm.pop {ra, s0-s11}, 160
-; RV64IZCMP-NEXT:    ret
+; RV64IZCMP-NEXT:    cm.popret {ra, s0-s11}, 160
 ;
 ; RV32I-LABEL: callee_no_irq:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -80
 ; RV32I-NEXT:    sw ra, 76(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    sw s0, 72(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    sw s1, 68(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    sw s2, 64(sp) # 4-byte Folded Spill