[RISCV] Implement KCFI operand bundle lowering

[samitolvanen]

Summary

With -fsanitize=kcfi (Kernel Control-Flow Integrity), Clang emits
"kcfi" operand bundles to indirect call instructions. Similarly to
the target-specific lowering added in D119296, implement KCFI operand
bundle lowering for RISC-V.

This patch disables the generic KCFI pass for RISC-V in Clang, and
adds the KCFI machine function pass in RISCVPassConfig::addPreSched
to emit target-specific KCFI_CHECK pseudo instructions before calls
that have KCFI operand bundles. The machine function pass also bundles
the instructions to ensure we emit the checks immediately before the
calls, which is not possible with the generic pass.

KCFI_CHECK instructions are lowered in RISCVAsmPrinter to a
contiguous code sequence that traps if the expected hash in the
operand bundle doesn't match the hash before the target function
address. This patch emits an ebreak instruction for error handling
to match the Linux kernel's BUG() implementation. Just like for X86,
we also emit trap locations to a .kcfi_traps section to support
error handling, as we cannot embed additional information to the trap
instruction itself.

[nickdesaulniers]

llvm/lib/Target/RISCV/RISCVKCFI.cpp

106–107 ↗

(On Diff #513763)

for (MachineInstr &MI : MBB)

[nickdesaulniers]

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

336–337

if (++NextReg > RISCV::X31)

[jrtc27]

I can't help but feel the core of this should be target-independent, with TLI or similar hooks to actually do the target-specific bits?

llvm/test/CodeGen/RISCV/kcfi.ll

2

Please try not to introduce tests with hand-written CHECK lines, but if you have to, try and make them look similar in style to the auto-generated lines.

[jrtc27]

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

314

What about this code relies on it being 64-bit?

llvm/lib/Target/RISCV/RISCVInstrInfo.td

1894

This formatting clearly doesn't match the surrounding code

1897

Don't introduce two blank lines

[craig.topper]

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

347

Are there two many operands here?

347

Err. "too many"

[craig.topper]

llvm/utils/gn/secondary/llvm/lib/Target/RISCV/BUILD.gn

81 ↗

(On Diff #513763)

Isn't this done automatically by the GN Syncbot when CMakeLists.txt is updated?

[samitolvanen]

In D148385#4275617, @jrtc27 wrote:

I can't help but feel the core of this should be target-independent, with TLI or similar hooks to actually do the target-specific bits?

I agree, that might be a nice improvement. The AArch64 version at least is nearly identical to this one, although there's a bit more going on in the X86 version. I'll take a look.

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

314

Good point, this doesn't require much tweaking to also work on RV32. We just need to avoid ADDIW.

336–337

That's not quite the same. We want to check the current register, not the next one, which we might not even need. I can certainly not post-increment the NextReg value if it's confusing.

347

Yes there are, thanks for noticing.

llvm/lib/Target/RISCV/RISCVKCFI.cpp

106–107 ↗

(On Diff #513763)

We don't want to skip bundles here, so we have to use instr_iterator explicitly.

llvm/test/CodeGen/RISCV/kcfi.ll

2

update_llc_test_checks.py doesn't play well with testing function prefix data, but I updated the style to match its output, added RV32 (without duplicating identical code), and moved the MIR tests to separate files with auto-generated checks. Please let me know if there's something specific I'm missing here.

llvm/utils/gn/secondary/llvm/lib/Target/RISCV/BUILD.gn

81 ↗

(On Diff #513763)

I remember updating this manually before, but if that's the case, I'm happy to drop this line.

[samitolvanen]

Addressed most of the feedback.

[samitolvanen]

Uploaded D149915 to remove the arch-specific KCFI passes. I'll update this patch once we clean that up.

[samitolvanen]

Dropped the arch-specific KCFI machine function pass.

[MaskRay]

Sorry for the very slow response.

KCFI_CHECK lowering has some complexity to allocate a temporary register. This needs to follow the calling convention which can be modified by many compiler options and function attributes.

I wonder whether we can move the if-condition part of the expanded code sequence (i.e. if type-hashes mismatch; crash) to ClangCodeGen (more like`-fsanitize=function`), and change the "kcfi" operand bundle to focus on expanding to a desired trap instruction (ud2 on x86-64).
On the plus side, this gives optimizers more opportunities to place trap basic blocks to cold regions.
On the downside, we cannot assume the code sequence is contiguous but that may be fine.

---

Some notice for other reviewers:

To emit .kcfi_traps and a preferred trap instruction, the instrumentation cannot be done purely in ClangCodeGen. Instead, the following scheme is used:

• ClangCodeGen emits "kcfi" operand bundles.
• In a target-overridden TargetPassConfig::addPreSched, add a KCFI pass to emit KCFI_CHECK pseudo instructions.
• In a target AsmPrinter, lower KCFI_CHECK pseudo instructions to a code sequence that crash if the type hashes don't match.

If a target doesn't implement KCFI_CHECK lowering, llvm/lib/Transforms/Instrumentation/KCFI.cpp is used to replace "kcfi" operand bundles with if conditions, then no TargetPassConfig::addPreSched or KCFI_CHECK lowering will be needed.
In this case, llvm.debugtrap is used to generate trap instructions.

[samitolvanen]

In D148385#4397995, @MaskRay wrote:

KCFI_CHECK lowering has some complexity to allocate a temporary register. This needs to follow the calling convention which can be modified by many compiler options and function attributes.

Correct, although in the RISC-V case, finding two temporary registers shouldn't be an issue. The latest Zisslpcfi specification also switched to using a temporary register (x7) for the expected type hash instead of specifying a dedicated register, so I don't believe we have issues with calling conventions in this case.

I wonder whether we can move the if-condition part of the expanded code sequence (i.e. if type-hashes mismatch; crash) to ClangCodeGen (more like`-fsanitize=function`), and change the "kcfi" operand bundle to focus on expanding to a desired trap instruction (ud2 on x86-64).
On the plus side, this gives optimizers more opportunities to place trap basic blocks to cold regions.
On the downside, we cannot assume the code sequence is contiguous but that may be fine.

The initial versions of KCFI actually had the codegen implemented in Clang, but Linux maintainers wanted the exact same instruction sequence for each check, which is why the lowering was moved to the back-end. It also wasn't just about the specific trap instruction, we spent considerable amount of time fine tuning the code that's generated and ensured it's emitted immediately before the call instructions (e.g. see the thread starting at https://lore.kernel.org/lkml/87o7xmup5t.ffs@tglx/).

That being said, on AArch64 we actually do encode register information to the immediate in the trap instruction (so the kernel can produce a useful error message), but on X86 (and in future, RISC-V) the kernel's trap handling code instead needs to look at the instructions precending the trap and decode register information from them, which again relies on the expected instructions being there:

https://elixir.bootlin.com/linux/v6.4-rc5/source/arch/x86/kernel/cfi.c#L16

The kernel also relies on a stable instruction sequence for runtime patching of alternative CFI schemes (FineIBT on X86 systems that support IBT) and runtime re-randomization of the hashes, for example:

https://elixir.bootlin.com/linux/v6.4-rc5/source/arch/x86/kernel/alternative.c#L702

While Zisslpcfi hasn't yet been ratified, a similar runtime patching scheme with KCFI may be preferable there in future so it's possible to ship a single kernel binary that still can use the potentially stronger hardware-assisted CFI scheme when hardware support is available.

[MaskRay]

Thanks for the description. The code generally looks good.

If the kernel wants a specific code sequence, implementing conditions in the LLVMCodeGen looks good to me. Consider adding a link to the discussion.

I wonder whether it will be more useful to update the summary/commit message to give more context to those who are unaware of -fsanitize=kcfi:

This patch implements KCFI operand bundle lowering to generate contiguous code sequences and produce .kcfi_traps sections.

• clang emits "kcfi" operand bundles
• SelectionDAGBuilder::visitCall visits the indirect CallInst and builds a SDNode using XXXTargetLowering::LowerCall.
• In RISCVPassConfig::addPreSched, add a KCFI pass to emit a KCFI_CHECK target-specific pseudo instruction before an indirect call instruction.
• In RISCVAsmPrinter, lower the KCFI_CHECK pseudo instructions to a contiguous code sequence that crash if the type hashes don't match.

Without the patch, there is no .kcfi_traps and the type hash comparison code sequence may not be contiguous due to basic block placement.

Needs some thoughts to merge the above paragraphs with the existing one "Similarly to the target-specific lowering added in D119296 [...]"

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

328

The prevailing style doesn't add parens for return.

333

As a minor optimization to skip one redundant isRegAvailable call: while (!isRegAvailable(++NextReg));

339

This assert seems redundant as Reg != AddrReg is very close.

353

Just check STI.hasStdExtCOrZca()

369

int64_t is the return type of getImm, not the original KCFI_CHECK operand type. Casting this to a uint32_t is perhaps clearer.

372

Add {} if the then body has more than one lines.

376

Is Hi20 == 0 tested?

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

16881

This switch should probably be turned to an assert. You can use llvm::is_contained if you don't want to write MBBI->getOpcode() twice.

assert(is_contained({RISCV::PseudoCALLIndirect, RISCV::PseudoTAILIndirect}, MBBI->getOpcode()));

16890

This assert seems redundant since we immediately call getReg with an internal assert.

llvm/test/CodeGen/RISCV/kcfi-patchable-function-prefix.ll

5

f1/f2/f3/f4 have the same shape. Without increasing the number of functions, we can make some disturbance (e.g. add one extra indirect call, use inline asm to clobber some registers) to check various cases without using .mir tests. .mir tests are fine, but the signal-to-noise ratio is not great and readers can be lost in the paramemters...

[MaskRay]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

15158

If CLI.CFIType != 0, is CLI.CB->isIndirectCall() possible? If not, add an assert?
This is an opportunity to save a redundant check.

[samitolvanen]

Addressed feedback.

[samitolvanen]

Needs some thoughts to merge the above paragraphs with the existing one

Great points. I updated the commit message and tried to capture most of these, PTAL.

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

333

Reg != NextReg here, and using while (!isRegAvailable(++NextReg)) would always skip the first alternative register by first incrementing NextReg before calling isRegsAvailable. What am I missing?

369

I don't think casting this to uint32_t make sense since we'd have to cast it back to a larger type immediately below (to prevent + 0x800 from overflowing). I changed the comment to indicate this is a 32-bit value though.

376

Good catch, added a test.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

15158

Sure. CFIType should be set only for indirect calls when we reach this point.

llvm/test/CodeGen/RISCV/kcfi-patchable-function-prefix.ll

5

There are four functions because we're testing type hash locations in relation to patchable-function-prefix nops in four different cases, in addition to the code generated for KCFI checks. I'm not sure there's a way to do this with fewer functions.

[MaskRay]

Will be good to wait a few days to give regular RISC-V backend developers a chance to leave any final comments.

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

333

Ah you are right. Didn't notice the Reg/nextReg difference.

clang/lib/CodeGen/BackendUtil.cpp

case 3:
     return OptimizationLevel::O3;
   }
 }
 
 static void addKCFIPass(const Triple &TargetTriple, const LangOptions &LangOpts,
                         PassBuilder &PB) {
   // If the back-end supports KCFI operand bundle lowering, skip KCFIPass.
   if (TargetTriple.getArch() == llvm::Triple::x86_64 ||
-      TargetTriple.isAArch64(64))
+      TargetTriple.isAArch64(64) || TargetTriple.isRISCV())
     return;
 
   // Ensure we lower KCFI operand bundles with -O0.
   PB.registerOptimizerLastEPCallback(
       [&](ModulePassManager &MPM, OptimizationLevel Level) {
         if (Level == OptimizationLevel::O0 &&
             LangOpts.Sanitize.has(SanitizerKind::KCFI))
           MPM.addPass(createModuleToFunctionPassAdaptor(KCFIPass()));

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

 #include "MCTargetDesc/RISCVBaseInfo.h"
 #include "MCTargetDesc/RISCVInstPrinter.h"
 #include "MCTargetDesc/RISCVMCExpr.h"
 #include "MCTargetDesc/RISCVTargetStreamer.h"
 #include "RISCV.h"
 #include "RISCVMachineFunctionInfo.h"
 #include "RISCVTargetMachine.h"
 #include "TargetInfo/RISCVTargetInfo.h"
+#include "llvm/ADT/APInt.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/BinaryFormat/ELF.h"
 #include "llvm/CodeGen/AsmPrinter.h"
 #include "llvm/CodeGen/MachineConstantPool.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/MC/MCAsmInfo.h"
public:
 
   void EmitToStreamer(MCStreamer &S, const MCInst &Inst);
   bool emitPseudoExpansionLowering(MCStreamer &OutStreamer,
                                    const MachineInstr *MI);
 
   typedef std::tuple<unsigned, uint32_t> HwasanMemaccessTuple;
   std::map<HwasanMemaccessTuple, MCSymbol *> HwasanMemaccessSymbols;
   void LowerHWASAN_CHECK_MEMACCESS(const MachineInstr &MI);
+  void LowerKCFI_CHECK(const MachineInstr &MI);
   void EmitHwasanMemaccessSymbols(Module &M);
 
   // Wrapper needed for tblgenned pseudo lowering.
   bool lowerOperand(const MachineOperand &MO, MCOperand &MCOp) const;
 
   void emitStartOfAsmFile(Module &M) override;
   void emitEndOfAsmFile(Module &M) override;
 
void RISCVAsmPrinter::emitInstruction(const MachineInstr *MI) {
   if (emitPseudoExpansionLowering(*OutStreamer, MI))
     return;
 
 
   switch (MI->getOpcode()) {
   case RISCV::HWASAN_CHECK_MEMACCESS_SHORTGRANULES:
     LowerHWASAN_CHECK_MEMACCESS(*MI);
     return;
+  case RISCV::KCFI_CHECK:
+    LowerKCFI_CHECK(*MI);
+    return;
   case RISCV::PseudoRVVInitUndefM1:
   case RISCV::PseudoRVVInitUndefM2:
   case RISCV::PseudoRVVInitUndefM4:
   case RISCV::PseudoRVVInitUndefM8:
     return;
   }
 
   MCInst OutInst;
if (!Sym) {
     Sym = OutContext.getOrCreateSymbol(SymName);
   }
   auto Res = MCSymbolRefExpr::create(Sym, MCSymbolRefExpr::VK_None, OutContext);
   auto Expr = RISCVMCExpr::create(Res, RISCVMCExpr::VK_RISCV_CALL, OutContext);
 
   EmitToStreamer(*OutStreamer, MCInstBuilder(RISCV::PseudoCALL).addExpr(Expr));
 }
 
+void RISCVAsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) {
+  Register AddrReg = MI.getOperand(0).getReg();
+  assert(std::next(MI.getIterator())->isCall() &&
+         "KCFI_CHECK not followed by a call instruction");
+  assert(std::next(MI.getIterator())->getOperand(0).getReg() == AddrReg &&
+         "KCFI_CHECK call target doesn't match call operand");
+
+  // Temporary registers for comparing the hashes. If a register is used
+  // for the call target, or reserved by the user, we can clobber another
+  // temporary register as the check is immediately followed by the
+  // call. The check defaults to X6/X7, but can fall back to X28-X31 if
+  // needed.
+  unsigned ScratchRegs[] = {RISCV::X6, RISCV::X7};
+  unsigned NextReg = RISCV::X28;
+  auto isRegAvailable = [&](unsigned Reg) {
+    return Reg != AddrReg && !STI->isRegisterReservedByUser(Reg);
+  };
+  for (auto &Reg : ScratchRegs) {
+    if (isRegAvailable(Reg))
+      continue;
+    while (!isRegAvailable(NextReg))
+      ++NextReg;
+    Reg = NextReg++;
+    if (Reg > RISCV::X31)
+      report_fatal_error("Unable to find scratch registers for KCFI_CHECK");
+  }
+
+  if (AddrReg == RISCV::X0) {
+    // Checking X0 makes no sense. Instead of emitting a load, zero
+    // ScratchRegs[0].
+    EmitToStreamer(*OutStreamer, MCInstBuilder(RISCV::ADDI)
+                                     .addReg(ScratchRegs[0])
+                                     .addReg(RISCV::X0)
+                                     .addImm(0));
+  } else {
+    // Adjust the offset for patchable-function-prefix. This assumes that
+    // patchable-function-prefix is the same for all functions.
+    int NopSize = STI->hasStdExtCOrZca() ? 2 : 4;
+    int64_t PrefixNops = 0;
+    (void)MI.getMF()
+        ->getFunction()
+        .getFnAttribute("patchable-function-prefix")
+        .getValueAsString()
+        .getAsInteger(10, PrefixNops);
+
+    // Load the target function type hash.
+    EmitToStreamer(*OutStreamer, MCInstBuilder(RISCV::LW)
+                                     .addReg(ScratchRegs[0])
+                                     .addReg(AddrReg)
+                                     .addImm(-(PrefixNops * NopSize + 4)));
+  }
+
+  // Load the expected 32-bit type hash.
+  const int64_t Type = MI.getOperand(1).getImm();
+  const int64_t Hi20 = ((Type + 0x800) >> 12) & 0xFFFFF;
+  const int64_t Lo12 = SignExtend64<12>(Type);
+  if (Hi20) {
+    EmitToStreamer(
+        *OutStreamer,
+        MCInstBuilder(RISCV::LUI).addReg(ScratchRegs[1]).addImm(Hi20));
+  }
+  if (Lo12 || Hi20 == 0) {
+    EmitToStreamer(*OutStreamer,
+                   MCInstBuilder((STI->hasFeature(RISCV::Feature64Bit) && Hi20)
+                                     ? RISCV::ADDIW
+                                     : RISCV::ADDI)
+                       .addReg(ScratchRegs[1])
+                       .addReg(ScratchRegs[1])
+                       .addImm(Lo12));
+  }
+
+  // Compare the hashes and trap if there's a mismatch.
+  MCSymbol *Pass = OutContext.createTempSymbol();
+  EmitToStreamer(*OutStreamer,
+                 MCInstBuilder(RISCV::BEQ)
+                     .addReg(ScratchRegs[0])
+                     .addReg(ScratchRegs[1])
+                     .addExpr(MCSymbolRefExpr::create(Pass, OutContext)));
+
+  MCSymbol *Trap = OutContext.createTempSymbol();
+  OutStreamer->emitLabel(Trap);
+  EmitToStreamer(*OutStreamer, MCInstBuilder(RISCV::EBREAK));
+  emitKCFITrapEntry(*MI.getMF(), Trap);
+  OutStreamer->emitLabel(Pass);
+}
+
 void RISCVAsmPrinter::EmitHwasanMemaccessSymbols(Module &M) {
   if (HwasanMemaccessSymbols.empty())
     return;
 
   assert(TM.getTargetTriple().isOSBinFormatELF());
   // Use MCSubtargetInfo from TargetMachine. Individual functions may have
   // attributes that differ from other functions in the module and we have no
   // way to know which function is correct.

llvm/lib/Target/RISCV/RISCVISelLowering.h

public:
   bool lowerInterleavedLoad(LoadInst *LI,
                             ArrayRef<ShuffleVectorInst *> Shuffles,
                             ArrayRef<unsigned> Indices,
                             unsigned Factor) const override;
 
   bool lowerInterleavedStore(StoreInst *SI, ShuffleVectorInst *SVI,
                              unsigned Factor) const override;
 
+  bool supportKCFIBundles() const override { return true; }
+
+  MachineInstr *EmitKCFICheck(MachineBasicBlock &MBB,
+                              MachineBasicBlock::instr_iterator &MBBI,
+                              const TargetInstrInfo *TII) const override;
+
   /// RISCVCCAssignFn - This target-specific function extends the default
   /// CCValAssign with additional information used to lower RISC-V calling
   /// conventions.
   typedef bool RISCVCCAssignFn(const DataLayout &DL, RISCVABI::ABI,
                                unsigned ValNo, MVT ValVT, MVT LocVT,
                                CCValAssign::LocInfo LocInfo,
                                ISD::ArgFlagsTy ArgFlags, CCState &State,
                                bool IsFixed, bool IsRet, Type *OrigTy,

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

SDValue RISCVTargetLowering::LowerCall(CallLoweringInfo &CLI,
                                        SmallVectorImpl<SDValue> &InVals) const {
   SelectionDAG &DAG = CLI.DAG;
   SDLoc &DL = CLI.DL;
   SmallVectorImpl<ISD::OutputArg> &Outs = CLI.Outs;
   SmallVectorImpl<SDValue> &OutVals = CLI.OutVals;
   SmallVectorImpl<ISD::InputArg> &Ins = CLI.Ins;
   SDValue Chain = CLI.Chain;
   SDValue Callee = CLI.Callee;
   bool &IsTailCall = CLI.IsTailCall;
   CallingConv::ID CallConv = CLI.CallConv;
   bool IsVarArg = CLI.IsVarArg;
   EVT PtrVT = getPointerTy(DAG.getDataLayout());
   MVT XLenVT = Subtarget.getXLenVT();
 
   MachineFunction &MF = DAG.getMachineFunction();
 
   // Analyze the operands of the call, assigning locations to each operand.
if (!IsTailCall) {
     assert(Mask && "Missing call preserved mask for calling convention");
     Ops.push_back(DAG.getRegisterMask(Mask));
   }
 
   // Glue the call to the argument copies, if any.
   if (Glue.getNode())
     Ops.push_back(Glue);
 
+  assert((!CLI.CFIType || CLI.CB->isIndirectCall()) &&
+         "Unexpected CFI type for a direct call");
+
   // Emit the call.
   SDVTList NodeTys = DAG.getVTList(MVT::Other, MVT::Glue);
 
   if (IsTailCall) {
     MF.getFrameInfo().setHasTailCall();
     SDValue Ret = DAG.getNode(RISCVISD::TAIL, DL, NodeTys, Ops);
+    if (CLI.CFIType)
+      Ret.getNode()->setCFIType(CLI.CFIType->getZExtValue());
     DAG.addNoMergeSiteInfo(Ret.getNode(), CLI.NoMerge);
     return Ret;
   }
 
   Chain = DAG.getNode(RISCVISD::CALL, DL, NodeTys, Ops);
+  if (CLI.CFIType)
+    Chain.getNode()->setCFIType(CLI.CFIType->getZExtValue());
   DAG.addNoMergeSiteInfo(Chain.getNode(), CLI.NoMerge);
   Glue = Chain.getValue(1);
 
   // Mark the end of the call, which is glued to the call itself.
   Chain = DAG.getCALLSEQ_END(Chain, NumBytes, 0, Glue, DL);
   Glue = Chain.getValue(1);
 
   // Assign locations to each value returned by this call.
bool RISCVTargetLowering::lowerInterleavedStore(StoreInst *SI,
   Value *VL = ConstantInt::get(XLenTy, VTy->getNumElements());
   Ops.append({SI->getPointerOperand(), VL});
 
   Builder.CreateCall(VssegNFunc, Ops);
 
   return true;
 }
 
+MachineInstr *
+RISCVTargetLowering::EmitKCFICheck(MachineBasicBlock &MBB,
+                                   MachineBasicBlock::instr_iterator &MBBI,
+                                   const TargetInstrInfo *TII) const {
+  assert(MBBI->isCall() && MBBI->getCFIType() &&
+         "Invalid call instruction for a KCFI check");
+  assert(is_contained({RISCV::PseudoCALLIndirect, RISCV::PseudoTAILIndirect},
+                      MBBI->getOpcode()));
+
+  MachineOperand &Target = MBBI->getOperand(0);
+  Target.setIsRenamable(false);
+
+  return BuildMI(MBB, MBBI, MBBI->getDebugLoc(), TII->get(RISCV::KCFI_CHECK))
+      .addReg(Target.getReg())
+      .addImm(MBBI->getCFIType())
+      .getInstr();
+}
+
 #define GET_REGISTER_MATCHER
 #include "RISCVGenAsmMatcher.inc"
 
 Register
 RISCVTargetLowering::getRegisterByName(const char *RegName, LLT VT,
                                        const MachineFunction &MF) const {
   Register Reg = MatchRegisterAltName(RegName);
   if (Reg == RISCV::NoRegister)

llvm/lib/Target/RISCV/RISCVInstrInfo.h

public:
 
   std::optional<unsigned> getInverseOpcode(unsigned Opcode) const override;
 
   ArrayRef<std::pair<MachineMemOperand::Flags, const char *>>
   getSerializableMachineMemOperandTargetFlags() const override;
 
 protected:
   const RISCVSubtarget &STI;
+
+private:
+  unsigned getInstBundleLength(const MachineInstr &MI) const;
 };
 
 namespace RISCV {
 
 // Returns true if this is the sext.w pattern, addiw rd, rs1, 0.
 bool isSEXT_W(const MachineInstr &MI);
 bool isZEXT_W(const MachineInstr &MI);
 bool isZEXT_B(const MachineInstr &MI);

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

if (ST.hasStdExtZihintntl() && MMO->isNonTemporal()) {
         if (isCompressibleInst(MI, STI))
           return 4; // c.ntl.all + c.load/c.store
         return 6;   // c.ntl.all + load/store
       }
       return 8; // ntl.all + load/store
     }
   }
 
+  if (Opcode == TargetOpcode::BUNDLE)
+    return getInstBundleLength(MI);
+
   if (MI.getParent() && MI.getParent()->getParent()) {
     if (isCompressibleInst(MI, STI))
       return 2;
   }
   return get(Opcode).getSize();
 }
 
+unsigned RISCVInstrInfo::getInstBundleLength(const MachineInstr &MI) const {
+  unsigned Size = 0;
+  MachineBasicBlock::const_instr_iterator I = MI.getIterator();
+  MachineBasicBlock::const_instr_iterator E = MI.getParent()->instr_end();
+  while (++I != E && I->isInsideBundle()) {
+    assert(!I->isBundle() && "No nested bundle!");
+    Size += getInstSizeInBytes(*I);
+  }
+  return Size;
+}
+
 bool RISCVInstrInfo::isAsCheapAsAMove(const MachineInstr &MI) const {
   const unsigned Opcode = MI.getOpcode();
   switch (Opcode) {
   default:
     break;
   case RISCV::FSGNJ_D:
   case RISCV::FSGNJ_S:
   case RISCV::FSGNJ_H:

llvm/lib/Target/RISCV/RISCVInstrInfo.td

 
 let Predicates = [IsRV64], Uses = [X5],
     Defs = [X1, X6, X7, X28, X29, X30, X31] in
 def HWASAN_CHECK_MEMACCESS_SHORTGRANULES 
   : Pseudo<(outs), (ins GPRJALR:$ptr, i32imm:$accessinfo),
            [(int_hwasan_check_memaccess_shortgranules X5, GPRJALR:$ptr,
                                                       (i32 timm:$accessinfo))]>;
 
+// This gets lowered into a 20-byte instruction sequence (at most)
+let hasSideEffects = 0, mayLoad = 1, mayStore = 0,
+    Defs = [ X6, X7, X28, X29, X30, X31 ], Size = 20 in {
+def KCFI_CHECK
+  : Pseudo<(outs), (ins GPRJALR:$ptr, i32imm:$type), []>, Sched<[]>;
+}
+
 /// Simple optimization
 def : Pat<(XLenVT (add GPR:$rs1, (AddiPair:$rs2))),
           (ADDI (ADDI GPR:$rs1, (AddiPairImmLarge AddiPair:$rs2)),
                 (AddiPairImmSmall GPR:$rs2))>;
 
 let Predicates = [IsRV64] in {
 // Select W instructions if only the lower 32-bits of the result are used.
 def : Pat<(binop_allwusers<add> GPR:$rs1, (AddiPair:$rs2)),
           (ADDIW (ADDIW GPR:$rs1, (AddiPairImmLarge AddiPair:$rs2)),

llvm/lib/Target/RISCV/RISCVTargetMachine.cpp

static cl::opt<bool> EnableRISCVCopyPropagation(
     cl::desc("Enable the copy propagation with RISC-V copy instr"),
     cl::init(true), cl::Hidden);
 
 extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeRISCVTarget() {
   RegisterTargetMachine<RISCVTargetMachine> X(getTheRISCV32Target());
   RegisterTargetMachine<RISCVTargetMachine> Y(getTheRISCV64Target());
   auto *PR = PassRegistry::getPassRegistry();
   initializeGlobalISel(*PR);
+  initializeKCFIPass(*PR);
   initializeRISCVMakeCompressibleOptPass(*PR);
   initializeRISCVGatherScatterLoweringPass(*PR);
   initializeRISCVCodeGenPreparePass(*PR);
   initializeRISCVMergeBaseOffsetOptPass(*PR);
   initializeRISCVOptWInstrsPass(*PR);
   initializeRISCVPreRAExpandPseudoPass(*PR);
   initializeRISCVExpandPseudoPass(*PR);
   initializeRISCVInsertVSETVLIPass(*PR);
bool RISCVPassConfig::addRegBankSelect() {
   return false;
 }
 
 bool RISCVPassConfig::addGlobalInstructionSelect() {
   addPass(new InstructionSelect(getOptLevel()));
   return false;
 }
 
-void RISCVPassConfig::addPreSched2() {}
+void RISCVPassConfig::addPreSched2() {
+  // Emit KCFI checks for indirect calls.
+  addPass(createKCFIPass());
+}
 
 void RISCVPassConfig::addPreEmitPass() {
   addPass(&BranchRelaxationPassID);
   addPass(createRISCVMakeCompressibleOptPass());
 
   // TODO: It would potentially be better to schedule copy propagation after
   // expanding pseudos (in addPreEmitPass2). However, performing copy
   // propagation after the machine outliner (which runs after addPreEmitPass)
   // currently leads to incorrect code-gen, where copies to registers within
   // outlined functions are removed erroneously.
   if (TM->getOptLevel() >= CodeGenOpt::Default && EnableRISCVCopyPropagation)
     addPass(createMachineCopyPropagationPass(true));
 }
 
 void RISCVPassConfig::addPreEmitPass2() {
   if (TM->getOptLevel() != CodeGenOpt::None)
     addPass(createRISCVMoveMergePass());
   addPass(createRISCVExpandPseudoPass());
 
   // Schedule the expansion of AMOs at the last possible moment, avoiding the
   // possibility for other passes to break the requirements for forward
   // progress in the LR/SC block.
   addPass(createRISCVExpandAtomicPseudoPass());
+
+  // KCFI indirect call checks are lowered to a bundle.
+  addPass(createUnpackMachineBundles([&](const MachineFunction &MF) {
+    return MF.getFunction().getParent()->getModuleFlag("kcfi");
+  }));
 }
 
 void RISCVPassConfig::addMachineSSAOptimization() {
   TargetPassConfig::addMachineSSAOptimization();
   if (EnableMachineCombiner)
     addPass(&MachineCombinerID);
 
   if (TM->getTargetTriple().getArch() == Triple::riscv64) {

llvm/test/CodeGen/RISCV/O0-pipeline.ll

 ; CHECK-NEXT:       Two-Address instruction pass
 ; CHECK-NEXT:       Fast Register Allocator
 ; CHECK-NEXT:       Remove Redundant DEBUG_VALUE analysis
 ; CHECK-NEXT:       Fixup Statepoint Caller Saved
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Prologue/Epilogue Insertion & Frame Finalization
 ; CHECK-NEXT:       Post-RA pseudo instruction expansion pass
+; CHECK-NEXT:       Insert KCFI indirect call checks
 ; CHECK-NEXT:       Analyze Machine Code For Garbage Collection
 ; CHECK-NEXT:       Insert fentry calls
 ; CHECK-NEXT:       Insert XRay ops
 ; CHECK-NEXT:       Implement the 'patchable-function' attribute
 ; CHECK-NEXT:       Branch relaxation pass
 ; CHECK-NEXT:       RISC-V Make Compressible
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Stack Frame Layout Analysis
 ; CHECK-NEXT:       RISC-V pseudo instruction expansion pass
 ; CHECK-NEXT:       RISC-V atomic pseudo instruction expansion pass
+; CHECK-NEXT:       Unpack machine instruction bundles
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       RISC-V Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction

llvm/test/CodeGen/RISCV/O3-pipeline.ll

 ; CHECK-NEXT:       Shrink Wrapping analysis
 ; CHECK-NEXT:       Prologue/Epilogue Insertion & Frame Finalization
 ; CHECK-NEXT:       Machine Late Instructions Cleanup Pass
 ; CHECK-NEXT:       Control Flow Optimizer
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Tail Duplication
 ; CHECK-NEXT:       Machine Copy Propagation Pass
 ; CHECK-NEXT:       Post-RA pseudo instruction expansion pass
+; CHECK-NEXT:       Insert KCFI indirect call checks
 ; CHECK-NEXT:       MachineDominator Tree Construction
 ; CHECK-NEXT:       Machine Natural Loop Construction
 ; CHECK-NEXT:       Post RA top-down list latency scheduler
 ; CHECK-NEXT:       Analyze Machine Code For Garbage Collection
 ; CHECK-NEXT:       Machine Block Frequency Analysis
 ; CHECK-NEXT:       MachinePostDominator Tree Construction
 ; CHECK-NEXT:       Branch Probability Basic Block Placement
 ; CHECK-NEXT:       Insert fentry calls

 ; CHECK-NEXT:     Machine Outliner
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Stack Frame Layout Analysis
 ; CHECK-NEXT:       RISC-V Zcmp move merging pass
 ; CHECK-NEXT:       RISC-V pseudo instruction expansion pass
 ; CHECK-NEXT:       RISC-V atomic pseudo instruction expansion pass
+; CHECK-NEXT:       Unpack machine instruction bundles
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       RISC-V Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction

llvm/test/CodeGen/RISCV/kcfi-isel.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 2
+# RUN: llc -mtriple=riscv64 -stop-after=finalize-isel -o - %s | FileCheck %s
+--- |
+  ; ModuleID = '<stdin>'
+  source_filename = "<stdin>"
+  target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128"
+  target triple = "riscv64"
+
+  define void @f1(ptr noundef %x) !kcfi_type !1 {
+    call void %x() [ "kcfi"(i32 12345678) ]
+    ret void
+  }
+
+  define void @f2(ptr noundef %x) #0 {
+    tail call void %x() [ "kcfi"(i32 12345678) ]
+    ret void
+  }
+
+  attributes #0 = { "patchable-function-entry"="2" }
+
+  !llvm.module.flags = !{!0}
+
+  !0 = !{i32 4, !"kcfi", i32 1}
+  !1 = !{i32 12345678}
+
+...
+---
+name:            f1
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+callsEHReturn:   false
+callsUnwindInit: false
+hasEHCatchret:   false
+hasEHScopes:     false
+hasEHFunclets:   false
+isOutlined:      false
+debugInstrRef:   false
+failsVerification: false
+tracksDebugUserValues: false
+registers:
+  - { id: 0, class: gprjalr, preferred-register: '' }
+liveins:
+  - { reg: '$x10', virtual-reg: '%0' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1
+  adjustsStack:    false
+  hasCalls:        true
+  stackProtector:  ''
+  functionContext: ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  hasTailCall:     false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo:
+  varArgsFrameIndex: 0
+  varArgsSaveSize: 0
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $x10
+
+    ; CHECK-LABEL: name: f1
+    ; CHECK: liveins: $x10, $x10, $x10
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gprjalr = COPY $x10
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gprjalr = COPY $x10
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gprjalr = COPY $x10
+    ; CHECK-NEXT: ADJCALLSTACKDOWN 0, 0, implicit-def dead $x2, implicit $x2
+    ; CHECK-NEXT: PseudoCALLIndirect [[COPY2]], csr_ilp32_lp64, implicit-def dead $x1, implicit-def $x2, cfi-type 12345678
+    ; CHECK-NEXT: ADJCALLSTACKUP 0, 0, implicit-def dead $x2, implicit $x2
+    ; CHECK-NEXT: PseudoRET
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: .1 (%ir-block.0):
+    ; CHECK-NEXT: ADJCALLSTACKDOWN 0, 0, implicit-def dead $x2, implicit $x2
+    ; CHECK-NEXT: PseudoCALLIndirect [[COPY]], csr_ilp32_lp64, implicit-def dead $x1, implicit-def $x2, cfi-type 12345678
+    ; CHECK-NEXT: ADJCALLSTACKUP 0, 0, implicit-def dead $x2, implicit $x2
+    ; CHECK-NEXT: PseudoRET
+    %0:gprjalr = COPY $x10
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $x2, implicit $x2
+    PseudoCALLIndirect %0, csr_ilp32_lp64, implicit-def dead $x1, implicit-def $x2, cfi-type 12345678
+    ADJCALLSTACKUP 0, 0, implicit-def dead $x2, implicit $x2
+    PseudoRET
+
+...
+---
+name:            f2
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+callsEHReturn:   false
+callsUnwindInit: false
+hasEHCatchret:   false
+hasEHScopes:     false
+hasEHFunclets:   false
+isOutlined:      false
+debugInstrRef:   false
+failsVerification: false
+tracksDebugUserValues: false
+registers:
+  - { id: 0, class: gprtc, preferred-register: '' }
+liveins:
+  - { reg: '$x10', virtual-reg: '%0' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  functionContext: ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  hasTailCall:     true
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo:
+  varArgsFrameIndex: 0
+  varArgsSaveSize: 0
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $x10
+
+    ; CHECK-LABEL: name: f2
+    ; CHECK: liveins: $x10, $x10, $x10
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gprtc = COPY $x10
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gprtc = COPY $x10
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gprtc = COPY $x10
+    ; CHECK-NEXT: PseudoTAILIndirect [[COPY2]], implicit $x2, cfi-type 12345678
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: .1 (%ir-block.0):
+    ; CHECK-NEXT: PseudoTAILIndirect [[COPY]], implicit $x2, cfi-type 12345678
+    %0:gprtc = COPY $x10
+    PseudoTAILIndirect %0, implicit $x2, cfi-type 12345678
+
+...

llvm/test/CodeGen/RISCV/kcfi-patchable-function-prefix.ll

+; RUN: llc -mtriple=riscv64 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,NOC
+; RUN: llc -mtriple=riscv64 -mattr=+c -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,C
+
+; NOC:            .p2align 2
+; C:              .p2align 1
+; CHECK-NOT:        nop
+; CHECK:          .word   12345678
+; CHECK-LABEL:    f1:
+define void @f1(ptr noundef %x) !kcfi_type !1 {
+; CHECK:            lw      t1, -4(a0)
+  call void %x() [ "kcfi"(i32 12345678) ]
+  ret void
+}
+
+; NOC:            .p2align 2
+; C:              .p2align 1
+; CHECK-NOT:       .word
+; CHECK-NOT:        nop
+; CHECK-LABEL:    f2:
+define void @f2(ptr noundef %x) {
+; CHECK:            lw      t1, -4(a0)
+  call void %x() [ "kcfi"(i32 12345678) ]
+  ret void
+}
+
+; NOC:            .p2align 2
+; C:              .p2align 1
+; CHECK:          .word   12345678
+; CHECK-COUNT-11:   nop
+; CHECK-LABEL:    f3:
+define void @f3(ptr noundef %x) #0 !kcfi_type !1 {
+; NOC:              lw      t1, -48(a0)
+; C:                lw      t1, -26(a0)
+  call void %x() [ "kcfi"(i32 12345678) ]
+  ret void
+}
+
+; NOC:            .p2align 2
+; C:              .p2align 1
+; CHECK-NOT:      .word
+; CHECK-COUNT-11:   nop
+; CHECK-LABEL:    f4:
+define void @f4(ptr noundef %x) #0 {
+; NOC:            lw      t1, -48(a0)
+; C:              lw      t1, -26(a0)
+  call void %x() [ "kcfi"(i32 12345678) ]
+  ret void
+}
+
+attributes #0 = { "patchable-function-prefix"="11" }
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 4, !"kcfi", i32 1}
+!1 = !{i32 12345678}

llvm/test/CodeGen/RISCV/kcfi.ll

+; RUN: llc -mtriple=riscv32 -verify-machineinstrs -riscv-no-aliases < %s \
+; RUN:      | FileCheck %s --check-prefixes=CHECK,RV32
+; RUN: llc -mtriple=riscv64 -verify-machineinstrs -riscv-no-aliases < %s \
+; RUN:      | FileCheck %s --check-prefixes=CHECK,RV64
+
+; CHECK:       .word 12345678
+define void @f1(ptr noundef %x) !kcfi_type !1 {
+; CHECK-LABEL: f1:
+; CHECK:       # %bb.0:
+; CHECK:         lw t1, -4(a0)
+; CHECK-NEXT:    lui t2, 3014
+; RV32-NEXT:     addi t2, t2, 334
+; RV64-NEXT:     addiw t2, t2, 334
+; CHECK-NEXT:    beq t1, t2, .Ltmp0
+; CHECK-NEXT:  .Ltmp1:
+; CHECK-NEXT:    ebreak
+; CHECK-NEXT:    .section .kcfi_traps,"ao",@progbits,.text
+; CHECK-NEXT:  .Ltmp2:
+; CHECK-NEXT:    .word .Ltmp1-.Ltmp2
+; CHECK-NEXT:    .text
+; CHECK-NEXT:  .Ltmp0:
+; CHECK-NEXT:    jalr ra, 0(a0)
+  call void %x() [ "kcfi"(i32 12345678) ]
+; CHECK:         lw t1, -4(s0)
+; CHECK-NEXT:    addi t2, t2, 1234
+; CHECK-NEXT:    beq t1, t2, .Ltmp3
+; CHECK-NEXT:  .Ltmp4:
+; CHECK-NEXT:    ebreak
+; CHECK-NEXT:    .section .kcfi_traps,"ao",@progbits,.text
+; CHECK-NEXT:  .Ltmp5:
+; CHECK-NEXT:    .word .Ltmp4-.Ltmp5
+; CHECK-NEXT:    .text
+; CHECK-NEXT:  .Ltmp3:
+; CHECK-NEXT:    jalr ra, 0(s0)
+  call void %x() [ "kcfi"(i32 1234) ]
+  ret void
+}
+
+; CHECK-NOT:   .word:
+define void @f2(ptr noundef %x) #0 {
+; CHECK-LABEL: f2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi zero, zero, 0
+; CHECK-NEXT:    addi zero, zero, 0
+; CHECK-NEXT:    lw t1, -4(a0)
+; CHECK-NEXT:    lui t2, 3014
+; RV32-NEXT:     addi t2, t2, 334
+; RV64-NEXT:     addiw t2, t2, 334
+; CHECK-NEXT:    beq t1, t2, .Ltmp6
+; CHECK-NEXT:  .Ltmp7:
+; CHECK-NEXT:    ebreak
+; CHECK-NEXT:    .section .kcfi_traps,"ao",@progbits,.text
+; CHECK-NEXT:  .Ltmp8:
+; CHECK-NEXT:    .word .Ltmp7-.Ltmp8
+; CHECK-NEXT:    .text
+; CHECK-NEXT:  .Ltmp6:
+; CHECK-NEXT:    jalr zero, 0(a0)
+  tail call void %x() [ "kcfi"(i32 12345678) ]
+  ret void
+}
+
+attributes #0 = { "patchable-function-entry"="2" }
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 4, !"kcfi", i32 1}
+!1 = !{i32 12345678}

llvm/test/CodeGen/RISCV/kcfi.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 2
+# RUN: llc -mtriple=riscv64 -stop-after=kcfi -o - %s | FileCheck %s
+--- |
+  ; ModuleID = '<stdin>'
+  source_filename = "<stdin>"
+  target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128"
+  target triple = "riscv64"
+
+  define void @f1(ptr noundef %x) !kcfi_type !1 {
+    call void %x() [ "kcfi"(i32 12345678) ]
+    ret void
+  }
+
+  define void @f2(ptr noundef %x) #0 {
+    tail call void %x() [ "kcfi"(i32 12345678) ]
+    ret void
+  }
+
+  attributes #0 = { "patchable-function-entry"="2" }
+
+  !llvm.module.flags = !{!0}
+
+  !0 = !{i32 4, !"kcfi", i32 1}
+  !1 = !{i32 12345678}
+
+...
+---
+name:            f1
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+callsEHReturn:   false
+callsUnwindInit: false
+hasEHCatchret:   false
+hasEHScopes:     false
+hasEHFunclets:   false
+isOutlined:      false
+debugInstrRef:   false
+failsVerification: false
+tracksDebugUserValues: true
+registers:       []
+liveins:
+  - { reg: '$x10', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       16
+  offsetAdjustment: 0
+  maxAlignment:    8
+  adjustsStack:    true
+  hasCalls:        true
+  stackProtector:  ''
+  functionContext: ''
+  maxCallFrameSize: 0
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  hasTailCall:     false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:
+  - { id: 0, name: '', type: spill-slot, offset: -8, size: 8, alignment: 8,
+      stack-id: default, callee-saved-register: '$x1', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo:
+  varArgsFrameIndex: 0
+  varArgsSaveSize: 0
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $x10, $x1
+
+    ; CHECK-LABEL: name: f1
+    ; CHECK: liveins: $x1, $x10
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x2 = frame-setup ADDI $x2, -16
+    ; CHECK-NEXT: frame-setup CFI_INSTRUCTION def_cfa_offset 16
+    ; CHECK-NEXT: SD $x1, $x2, 8 :: (store (s64) into %stack.1)
+    ; CHECK-NEXT: frame-setup CFI_INSTRUCTION offset $x1, -8
+    ; CHECK-NEXT: $x2 = frame-setup ADDI $x2, -16
+    ; CHECK-NEXT: frame-setup CFI_INSTRUCTION def_cfa_offset 16
+    ; CHECK-NEXT: SD $x1, $x2, 8 :: (store (s64) into %stack.0)
+    ; CHECK-NEXT: frame-setup CFI_INSTRUCTION offset $x1, -8
+    ; CHECK-NEXT: BUNDLE implicit-def dead $x6, implicit-def dead $x7, implicit-def dead $x28, implicit-def dead $x29, implicit-def dead $x30, implicit-def dead $x31, implicit-def dead $x1, implicit-def $x2, implicit $x10 {
+    ; CHECK-NEXT:   KCFI_CHECK $x10, 12345678, implicit-def $x6, implicit-def $x7, implicit-def $x28, implicit-def $x29, implicit-def $x30, implicit-def $x31
+    ; CHECK-NEXT:   PseudoCALLIndirect $x10, csr_ilp32_lp64, implicit-def dead $x1, implicit-def $x2
+    ; CHECK-NEXT: }
+    ; CHECK-NEXT: dead $x1 = LD $x2, 8 :: (load (s64) from %stack.0)
+    ; CHECK-NEXT: $x2 = frame-destroy ADDI $x2, 16
+    ; CHECK-NEXT: $x1 = LD $x2, 8 :: (load (s64) from %stack.1)
+    ; CHECK-NEXT: $x2 = frame-destroy ADDI $x2, 16
+    ; CHECK-NEXT: PseudoRET
+    $x2 = frame-setup ADDI $x2, -16
+    frame-setup CFI_INSTRUCTION def_cfa_offset 16
+    SD killed $x1, $x2, 8 :: (store (s64) into %stack.0)
+    frame-setup CFI_INSTRUCTION offset $x1, -8
+    BUNDLE implicit-def $x6, implicit-def $x7, implicit-def $x28, implicit-def $x29, implicit-def $x30, implicit-def $x31, implicit-def dead $x1, implicit-def $x2, implicit killed $x10 {
+      KCFI_CHECK $x10, 12345678, implicit-def $x6, implicit-def $x7, implicit-def $x28, implicit-def $x29, implicit-def $x30, implicit-def $x31
+      PseudoCALLIndirect killed $x10, csr_ilp32_lp64, implicit-def dead $x1, implicit-def $x2
+    }
+    $x1 = LD $x2, 8 :: (load (s64) from %stack.0)
+    $x2 = frame-destroy ADDI $x2, 16
+    PseudoRET
+
+...
+---
+name:            f2
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+callsEHReturn:   false
+callsUnwindInit: false
+hasEHCatchret:   false
+hasEHScopes:     false
+hasEHFunclets:   false
+isOutlined:      false
+debugInstrRef:   false
+failsVerification: false
+tracksDebugUserValues: true
+registers:       []
+liveins:
+  - { reg: '$x10', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  functionContext: ''
+  maxCallFrameSize: 0
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  hasTailCall:     true
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo:
+  varArgsFrameIndex: 0
+  varArgsSaveSize: 0
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $x10
+
+    ; CHECK-LABEL: name: f2
+    ; CHECK: liveins: $x10
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: BUNDLE implicit-def dead $x6, implicit-def dead $x7, implicit-def dead $x28, implicit-def dead $x29, implicit-def dead $x30, implicit-def dead $x31, implicit $x10, implicit $x2 {
+    ; CHECK-NEXT:   KCFI_CHECK $x10, 12345678, implicit-def $x6, implicit-def $x7, implicit-def $x28, implicit-def $x29, implicit-def $x30, implicit-def $x31
+    ; CHECK-NEXT:   PseudoTAILIndirect $x10, implicit $x2
+    ; CHECK-NEXT: }
+    BUNDLE implicit-def $x6, implicit-def $x7, implicit-def $x28, implicit-def $x29, implicit-def $x30, implicit-def $x31, implicit killed $x10, implicit $x2 {
+      KCFI_CHECK $x10, 12345678, implicit-def $x6, implicit-def $x7, implicit-def $x28, implicit-def $x29, implicit-def $x30, implicit-def $x31
+      PseudoTAILIndirect killed $x10, implicit $x2
+    }
+
+...