Add generic KCFI operand bundle lowering

[samitolvanen]

Summary

The KCFI sanitizer emits "kcfi" operand bundles to indirect
call instructions, which the LLVM back-end lowers into an
architecture-specific type check with a known machine instruction
sequence. Currently, KCFI operand bundle lowering is supported only
on 64-bit X86 and AArch64 architectures.

As a lightweight forward-edge CFI implementation that doesn't
require LTO is also useful for non-Linux low-level targets on
other machine architectures, add a generic KCFI operand bundle
lowering pass that's only used when back-end lowering support is not
available and allows -fsanitize=kcfi to be enabled in Clang on all
architectures.

[kees]

What's the best way to test this in the kernel? I assume add ARCH_SUPPORTS_CFI_CLANG to an arch, and see what blows up? :) Have you tried this on any other arch yet?

[samitolvanen]

In D135411#3841693, @kees wrote:

What's the best way to test this in the kernel? I assume add ARCH_SUPPORTS_CFI_CLANG to an arch, and see what blows up? :)

Yes, I think that's the best way.

Have you tried this on any other arch yet?

The generic lowering pass is not ideal for kernel use as-is, because it doesn't generate a consistent machine instruction sequence, which makes error handling more difficult. The primary target audience here is firmware running on less common architectures.

That being said, I did compile a 64-bit MIPS kernel using this pass, but I didn't try booting it. I would expect to run into quite a few issues initially.

[nickdesaulniers]

llvm/lib/Transforms/Instrumentation/KCFI.cpp

49–50

prefer a range-for here

for (Instruction &I : F)
  if (auto &CI = dyn_cast<CallInst>(I);

78

CallBase *Call = ...

llvm/test/Transforms/KCFI/kcfi-supported.ll

4 ↗

(On Diff #465914)

COM?

[kees]

In D135411#3841841, @samitolvanen wrote:

That being said, I did compile a 64-bit MIPS kernel using this pass, but I didn't try booting it. I would expect to run into quite a few issues initially.

I've done a test build with riscv64, and I can see the expected loads and literal compares. Nice!

[samitolvanen]

Addressed feedback.

llvm/lib/Transforms/Instrumentation/KCFI.cpp

49–50

Function doesn't directly support iterating through instructions. I used inst_iterator here to avoid looping through basic blocks as well. Your example results in:

error: non-const lvalue reference to type 'llvm::Instruction' cannot bind to a value of unrelated type 'llvm::BasicBlock'

llvm/test/Transforms/KCFI/kcfi-supported.ll

4 ↗

(On Diff #465914)

https://llvm.org/docs/CommandGuide/FileCheck.html#the-com-directive

But yes, not needed here.

[nickdesaulniers]

llvm/lib/Transforms/Instrumentation/KCFI.cpp

49–50

That's right, sorry, you'd need either a double-nested for loop

for BB : F

for I : BB

or

for I : instructions(F)

Please use that ^.

[samitolvanen]

Switched to instructions(F).

[MaskRay]

clang/lib/Driver/ToolChain.cpp

1091

This member function is overridden by various toolchains, so adding KCFI here isn't sufficient, e.g. --target=riscv64

[MaskRay]

clang/lib/Driver/ToolChain.cpp

1091

Ignore me. This works.

[MaskRay]

llvm/lib/Transforms/Instrumentation/KCFI.cpp

48

You can omit , 8 to use the default (also 8).

62

Try avoiding report_fatal_error. Use a proper error handling mechanism like Ctx.diagnose(...) with a DiagnosticInfo subclass. Remove the trailing period for the message.

73

Add const if not modified

98

pre-increment https://llvm.org/docs/CodingStandards.html#prefer-preincrement

llvm/test/Transforms/KCFI/kcfi-supported.ll

4 ↗

(On Diff #466635)

llvm/test/Transforms/KCFI/kcfi.ll

4

Appending ( to prevent match with another function with the same prefix (not in this test, but this is a good general practice)

[samitolvanen]

Addressed Fangrui's feedback.

llvm/lib/Transforms/Instrumentation/KCFI.cpp

62

Try avoiding report_fatal_error. Use a proper error handling mechanism like Ctx.diagnose(...) with a DiagnosticInfo subclass.

I was looking at this, but I feel like adding a new class might be a bit of an overkill for this situation. Looking at the code in lib/Transforms, there are 70 uses of report_fatal_error and 17 uses of Ctx.diagnose(...). The latter seem to all be more complex diagnostics than just printing out an error message. Do you think I should nevertheless use Ctx.diagnose for this message?

Remove the trailing period for the message.

Done.

[MaskRay]

llvm/lib/Transforms/Instrumentation/KCFI.cpp

62

A lot of report_fatal_error uses are used similar to unreachable for both assertion and non-assertion builds. The code is pretty much reachable (if a user has a patchable_function_entry attribute and use -fsanitize=kcfi). It's bad interface to surface a crash-style diagnostic.

You may add a namespace { class ... : public DiagnosticInfo } in this file. It just needs to overload print which is not much boilerplate.

[samitolvanen]

Added KCFIDiagnosticInfo and switched from report_fatal_error to LLVMContext::diagnose.

[MaskRay]

llvm/lib/Transforms/Instrumentation/KCFI.cpp

43

If you look at existing DiagnosticInfo subclasses, they are mostly named DiagnosticInfo*

llvm/test/Transforms/KCFI/kcfi-patchable-function-prefix.ll

2

You can omit <

[samitolvanen]

Addressed more feedback.

[MaskRay]

llvm/lib/Transforms/Instrumentation/KCFI.cpp

17

Sorry, I just noticed this include. lib/Transforms/ files cannot depend on CodeGen for layering. You can see that no other Instrumentation uses CodeGen/

[samitolvanen]

Dropped the CodeGen dependency. Note that Clang already only adds KCFIPass for targets that don't support back-end lowering, so the check in the pass itself was a bit redundant. The existing CodeGen/kcfi.c test also ensures we continue to emit kcfi operand bundles.

[MaskRay]

llvm/include/llvm/Transforms/Instrumentation/KCFI.h

25

remove default constructor?

[samitolvanen]

Dropped the unneeded constructor.

[samitolvanen]

Fixed clang-format warnings.

[MaskRay]

clang/lib/CodeGen/BackendUtil.cpp

975

To my self: I'll need to dig up why if (!IsThinLTOPostLink) { is here. I think it's unnecessary. But it's ok to keep KCFI in the same place of sanitizers.

llvm/include/llvm/Transforms/Instrumentation/KCFI.h

20

the declaration is unused now.

clang/lib/CodeGen/BackendUtil.cpp

 #include "llvm/Bitcode/BitcodeReader.h"
 #include "llvm/Bitcode/BitcodeWriter.h"
 #include "llvm/Bitcode/BitcodeWriterPass.h"
 #include "llvm/CodeGen/RegAllocRegistry.h"
 #include "llvm/CodeGen/SchedulerRegistry.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DebugInfo.h"
-#include "llvm/IRPrinter/IRPrintingPasses.h"
 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/ModuleSummaryIndex.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/IR/Verifier.h"
+#include "llvm/IRPrinter/IRPrintingPasses.h"
 #include "llvm/LTO/LTOBackend.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/SubtargetFeature.h"
 #include "llvm/MC/TargetRegistry.h"
 #include "llvm/Object/OffloadBinary.h"
 #include "llvm/Passes/PassBuilder.h"
 #include "llvm/Passes/PassPlugin.h"
 #include "llvm/Passes/StandardInstrumentations.h"

 #include "llvm/Transforms/Instrumentation.h"
 #include "llvm/Transforms/Instrumentation/AddressSanitizer.h"
 #include "llvm/Transforms/Instrumentation/AddressSanitizerOptions.h"
 #include "llvm/Transforms/Instrumentation/BoundsChecking.h"
 #include "llvm/Transforms/Instrumentation/DataFlowSanitizer.h"
 #include "llvm/Transforms/Instrumentation/GCOVProfiler.h"
 #include "llvm/Transforms/Instrumentation/HWAddressSanitizer.h"
 #include "llvm/Transforms/Instrumentation/InstrProfiling.h"
+#include "llvm/Transforms/Instrumentation/KCFI.h"
 #include "llvm/Transforms/Instrumentation/MemProfiler.h"
 #include "llvm/Transforms/Instrumentation/MemorySanitizer.h"
 #include "llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h"
 #include "llvm/Transforms/Instrumentation/SanitizerCoverage.h"
 #include "llvm/Transforms/Instrumentation/ThreadSanitizer.h"
 #include "llvm/Transforms/ObjCARC.h"
 #include "llvm/Transforms/Scalar.h"
 #include "llvm/Transforms/Scalar/EarlyCSE.h"
case 2:
       return OptimizationLevel::Oz;
     }
 
   case 3:
     return OptimizationLevel::O3;
   }
 }
 
+static void addKCFIPass(const Triple &TargetTriple, const LangOptions &LangOpts,
+                        PassBuilder &PB) {
+  // If the back-end supports KCFI operand bundle lowering, skip KCFIPass.
+  if (TargetTriple.getArch() == llvm::Triple::x86_64 ||
+      TargetTriple.isAArch64(64))
+    return;
+
+  // Ensure we lower KCFI operand bundles with -O0.
+  PB.registerOptimizerLastEPCallback(
+      [&](ModulePassManager &MPM, OptimizationLevel Level) {
+        if (Level == OptimizationLevel::O0 &&
+            LangOpts.Sanitize.has(SanitizerKind::KCFI))
+          MPM.addPass(createModuleToFunctionPassAdaptor(KCFIPass()));
+      });
+
+  // When optimizations are requested, run KCIFPass after InstCombine to
+  // avoid unnecessary checks.
+  PB.registerPeepholeEPCallback(
+      [&](FunctionPassManager &FPM, OptimizationLevel Level) {
+        if (Level != OptimizationLevel::O0 &&
+            LangOpts.Sanitize.has(SanitizerKind::KCFI))
+          FPM.addPass(KCFIPass());
+      });
+}
+
 static void addSanitizers(const Triple &TargetTriple,
                           const CodeGenOptions &CodeGenOpts,
                           const LangOptions &LangOpts, PassBuilder &PB) {
   auto SanitizersCallback = [&](ModulePassManager &MPM,
                                 OptimizationLevel Level) {
     if (CodeGenOpts.hasSanitizeCoverage()) {
       auto SancovOpts = getSancovOptsFromCGOpts(CodeGenOpts);
       MPM.addPass(SanitizerCoveragePass(
if (!CodeGenOpts.DisableLLVMPasses) {
     if (LangOpts.Sanitize.has(SanitizerKind::LocalBounds))
       PB.registerScalarOptimizerLateEPCallback(
           [](FunctionPassManager &FPM, OptimizationLevel Level) {
             FPM.addPass(BoundsCheckingPass());
           });
 
     // Don't add sanitizers if we are here from ThinLTO PostLink. That already
     // done on PreLink stage.
-    if (!IsThinLTOPostLink)
+    if (!IsThinLTOPostLink) {
       addSanitizers(TargetTriple, CodeGenOpts, LangOpts, PB);
+      addKCFIPass(TargetTriple, LangOpts, PB);
+    }
 
     if (Optional<GCOVOptions> Options = getGCOVOptions(CodeGenOpts, LangOpts))
       PB.registerPipelineStartEPCallback(
           [Options](ModulePassManager &MPM, OptimizationLevel Level) {
             MPM.addPass(GCOVProfilerPass(*Options));
           });
     if (Optional<InstrProfOptions> Options =
             getInstrProfOptions(CodeGenOpts, LangOpts))

clang/lib/Driver/ToolChain.cpp

SanitizerMask ToolChain::getSupportedSanitizers() const {
   // Return sanitizers which don't require runtime support and are not
   // platform dependent.
 
   SanitizerMask Res =
       (SanitizerKind::Undefined & ~SanitizerKind::Vptr &
        ~SanitizerKind::Function) |
       (SanitizerKind::CFI & ~SanitizerKind::CFIICall) |
       SanitizerKind::CFICastStrict | SanitizerKind::FloatDivideByZero |
-      SanitizerKind::UnsignedIntegerOverflow |
+      SanitizerKind::KCFI | SanitizerKind::UnsignedIntegerOverflow |
       SanitizerKind::UnsignedShiftBase | SanitizerKind::ImplicitConversion |
       SanitizerKind::Nullability | SanitizerKind::LocalBounds;
   if (getTriple().getArch() == llvm::Triple::x86 ||
       getTriple().getArch() == llvm::Triple::x86_64 ||
       getTriple().getArch() == llvm::Triple::arm || getTriple().isWasm() ||
       getTriple().isAArch64() || getTriple().isRISCV())
     Res |= SanitizerKind::CFIICall;
   if (getTriple().getArch() == llvm::Triple::x86_64 ||
-      getTriple().isAArch64(64))
-    Res |= SanitizerKind::KCFI;
-  if (getTriple().getArch() == llvm::Triple::x86_64 ||
       getTriple().isAArch64(64) || getTriple().isRISCV())
     Res |= SanitizerKind::ShadowCallStack;
   if (getTriple().isAArch64(64))
     Res |= SanitizerKind::MemTag;
   return Res;
 }
 
 void ToolChain::AddCudaIncludeArgs(const ArgList &DriverArgs,

llvm/include/llvm/InitializePasses.h

 void initializeLowerGlobalDtorsLegacyPassPass(PassRegistry &);
 void initializeLowerGuardIntrinsicLegacyPassPass(PassRegistry&);
 void initializeLowerWidenableConditionLegacyPassPass(PassRegistry&);
 void initializeLowerIntrinsicsPass(PassRegistry&);
 void initializeLowerInvokeLegacyPassPass(PassRegistry&);
 void initializeLowerSwitchLegacyPassPass(PassRegistry &);
 void initializeLowerMatrixIntrinsicsLegacyPassPass(PassRegistry &);
 void initializeLowerMatrixIntrinsicsMinimalLegacyPassPass(PassRegistry &);
+void initializeKCFIPass(PassRegistry &);
 void initializeMIRAddFSDiscriminatorsPass(PassRegistry &);
 void initializeMIRCanonicalizerPass(PassRegistry &);
 void initializeMIRNamerPass(PassRegistry &);
 void initializeMIRPrintingPassPass(PassRegistry&);
 void initializeMachineBlockFrequencyInfoPass(PassRegistry&);
 void initializeMachineBlockPlacementPass(PassRegistry&);
 void initializeMachineBlockPlacementStatsPass(PassRegistry&);
 void initializeMachineBranchProbabilityInfoPass(PassRegistry&);

llvm/include/llvm/Transforms/Instrumentation/KCFI.h

+//===-- KCFI.h - Generic KCFI operand bundle lowering -----------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass emits generic KCFI indirect call checks for targets that don't
+// support lowering KCFI operand bundles in the back-end.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TRANSFORMS_INSTRUMENTATION_KCFI_H
+#define LLVM_TRANSFORMS_INSTRUMENTATION_KCFI_H
+
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+class TargetMachine;
+
+class KCFIPass : public PassInfoMixin<KCFIPass> {
+public:
+  static bool isRequired() { return true; }
+  PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
+};
+} // namespace llvm
+#endif // LLVM_TRANSFORMS_INSTRUMENTATION_KCFI_H

llvm/lib/Passes/PassBuilder.cpp

 #include "llvm/Analysis/ScopedNoAliasAA.h"
 #include "llvm/Analysis/StackLifetime.h"
 #include "llvm/Analysis/StackSafetyAnalysis.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/Analysis/TypeBasedAliasAnalysis.h"
 #include "llvm/IR/DebugInfo.h"
 #include "llvm/IR/Dominators.h"
-#include "llvm/IRPrinter/IRPrintingPasses.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/IR/PrintPasses.h"
 #include "llvm/IR/SafepointIRVerifier.h"
 #include "llvm/IR/Verifier.h"
+#include "llvm/IRPrinter/IRPrintingPasses.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/Regex.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
 #include "llvm/Transforms/Coroutines/CoroCleanup.h"

 #include "llvm/Transforms/Instrumentation/BoundsChecking.h"
 #include "llvm/Transforms/Instrumentation/CGProfile.h"
 #include "llvm/Transforms/Instrumentation/ControlHeightReduction.h"
 #include "llvm/Transforms/Instrumentation/DataFlowSanitizer.h"
 #include "llvm/Transforms/Instrumentation/GCOVProfiler.h"
 #include "llvm/Transforms/Instrumentation/HWAddressSanitizer.h"
 #include "llvm/Transforms/Instrumentation/InstrOrderFile.h"
 #include "llvm/Transforms/Instrumentation/InstrProfiling.h"
+#include "llvm/Transforms/Instrumentation/KCFI.h"
 #include "llvm/Transforms/Instrumentation/MemProfiler.h"
 #include "llvm/Transforms/Instrumentation/MemorySanitizer.h"
 #include "llvm/Transforms/Instrumentation/PGOInstrumentation.h"
 #include "llvm/Transforms/Instrumentation/PoisonChecking.h"
 #include "llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h"
 #include "llvm/Transforms/Instrumentation/SanitizerCoverage.h"
 #include "llvm/Transforms/Instrumentation/ThreadSanitizer.h"
 #include "llvm/Transforms/ObjCARC.h"

llvm/lib/Passes/PassRegistry.def

 FUNCTION_PASS("mem2reg", PromotePass())
 FUNCTION_PASS("memcpyopt", MemCpyOptPass())
 FUNCTION_PASS("mergeicmps", MergeICmpsPass())
 FUNCTION_PASS("mergereturn", UnifyFunctionExitNodesPass())
 FUNCTION_PASS("nary-reassociate", NaryReassociatePass())
 FUNCTION_PASS("newgvn", NewGVNPass())
 FUNCTION_PASS("jump-threading", JumpThreadingPass())
 FUNCTION_PASS("partially-inline-libcalls", PartiallyInlineLibCallsPass())
+FUNCTION_PASS("kcfi", KCFIPass())
 FUNCTION_PASS("lcssa", LCSSAPass())
 FUNCTION_PASS("loop-data-prefetch", LoopDataPrefetchPass())
 FUNCTION_PASS("loop-load-elim", LoopLoadEliminationPass())
 FUNCTION_PASS("loop-fusion", LoopFusePass())
 FUNCTION_PASS("loop-distribute", LoopDistributePass())
 FUNCTION_PASS("loop-versioning", LoopVersioningPass())
 FUNCTION_PASS("objc-arc", ObjCARCOptPass())
 FUNCTION_PASS("objc-arc-contract", ObjCARCContractPass())

llvm/lib/Transforms/Instrumentation/CMakeLists.txt

 add_llvm_component_library(LLVMInstrumentation
   AddressSanitizer.cpp
   BoundsChecking.cpp
   CGProfile.cpp
   ControlHeightReduction.cpp
   DataFlowSanitizer.cpp
   GCOVProfiling.cpp
   MemProfiler.cpp
   MemorySanitizer.cpp
   IndirectCallPromotion.cpp
   Instrumentation.cpp
   InstrOrderFile.cpp
   InstrProfiling.cpp
+  KCFI.cpp
   PGOInstrumentation.cpp
   PGOMemOPSizeOpt.cpp
   PoisonChecking.cpp
   SanitizerCoverage.cpp
   SanitizerBinaryMetadata.cpp
   ValueProfileCollector.cpp
   ThreadSanitizer.cpp
   HWAddressSanitizer.cpp

llvm/lib/Transforms/Instrumentation/KCFI.cpp

+//===-- KCFI.cpp - Generic KCFI operand bundle lowering ---------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass emits generic KCFI indirect call checks for targets that don't
+// support lowering KCFI operand bundles in the back-end.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Transforms/Instrumentation/KCFI.h"
+#include "llvm/ADT/Statistic.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/DiagnosticInfo.h"
+#include "llvm/IR/DiagnosticPrinter.h"
+#include "llvm/IR/Function.h"
+#include "llvm/IR/GlobalObject.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/InstIterator.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/Intrinsics.h"
+#include "llvm/IR/MDBuilder.h"
+#include "llvm/IR/Module.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+#include "llvm/Target/TargetMachine.h"
+#include "llvm/Transforms/Instrumentation.h"
+#include "llvm/Transforms/Utils/BasicBlockUtils.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "kcfi"
+
+STATISTIC(NumKCFIChecks, "Number of kcfi operands transformed into checks");
+
+namespace {
+class DiagnosticInfoKCFI : public DiagnosticInfo {
+  const Twine &Msg;
+
+public:
+  DiagnosticInfoKCFI(const Twine &DiagMsg,
+                     DiagnosticSeverity Severity = DS_Error)
+      : DiagnosticInfo(DK_Linker, Severity), Msg(DiagMsg) {}
+  void print(DiagnosticPrinter &DP) const override { DP << Msg; }
+};
+} // namespace
+
+PreservedAnalyses KCFIPass::run(Function &F, FunctionAnalysisManager &AM) {
+  Module &M = *F.getParent();
+  if (!M.getModuleFlag("kcfi"))
+    return PreservedAnalyses::all();
+
+  // Find call instructions with KCFI operand bundles.
+  SmallVector<CallInst *> KCFICalls;
+  for (Instruction &I : instructions(F)) {
+    if (auto *CI = dyn_cast<CallInst>(&I))
+      if (CI->getOperandBundle(LLVMContext::OB_kcfi))
+        KCFICalls.push_back(CI);
+  }
+
+  if (KCFICalls.empty())
+    return PreservedAnalyses::all();
+
+  LLVMContext &Ctx = M.getContext();
+  // patchable-function-prefix emits nops between the KCFI type identifier
+  // and the function start. As we don't know the size of the emitted nops,
+  // don't allow this attribute with generic lowering.
+  if (F.hasFnAttribute("patchable-function-prefix"))
+    Ctx.diagnose(
+        DiagnosticInfoKCFI("-fpatchable-function-entry=N,M, where M>0 is not "
+                           "compatible with -fsanitize=kcfi on this target"));
+
+  IntegerType *Int32Ty = Type::getInt32Ty(Ctx);
+  MDNode *VeryUnlikelyWeights =
+      MDBuilder(Ctx).createBranchWeights(1, (1U << 20) - 1);
+
+  for (CallInst *CI : KCFICalls) {
+    // Get the expected hash value.
+    const uint32_t ExpectedHash =
+        cast<ConstantInt>(CI->getOperandBundle(LLVMContext::OB_kcfi)->Inputs[0])
+            ->getZExtValue();
+
+    // Drop the KCFI operand bundle.
+    CallBase *Call =
+        CallBase::removeOperandBundle(CI, LLVMContext::OB_kcfi, CI);
+    assert(Call != CI);
+    Call->copyMetadata(*CI);
+    CI->replaceAllUsesWith(Call);
+    CI->eraseFromParent();
+
+    if (!Call->isIndirectCall())
+      continue;
+
+    // Emit a check and trap if the target hash doesn't match.
+    IRBuilder<> Builder(Call);
+    Value *HashPtr = Builder.CreateConstInBoundsGEP1_32(
+        Int32Ty, Call->getCalledOperand(), -1);
+    Value *Test = Builder.CreateICmpNE(Builder.CreateLoad(Int32Ty, HashPtr),
+                                       ConstantInt::get(Int32Ty, ExpectedHash));
+    Instruction *ThenTerm =
+        SplitBlockAndInsertIfThen(Test, Call, false, VeryUnlikelyWeights);
+    Builder.SetInsertPoint(ThenTerm);
+    Builder.CreateCall(Intrinsic::getDeclaration(&M, Intrinsic::trap));
+    ++NumKCFIChecks;
+  }
+
+  return PreservedAnalyses::none();
+}

llvm/test/Transforms/KCFI/kcfi-patchable-function-prefix.ll

+; RUN: not opt -S -passes=kcfi %s 2>&1 | FileCheck %s
+
+; CHECK: error: -fpatchable-function-entry=N,M, where M>0 is not compatible with -fsanitize=kcfi on this target
+define void @f1(ptr noundef %x) #0 {
+  call void %x() [ "kcfi"(i32 12345678) ]
+  ret void
+}
+
+attributes #0 = { "patchable-function-prefix"="1" }
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 4, !"kcfi", i32 1}

llvm/test/Transforms/KCFI/kcfi.ll

+; RUN: opt -S -passes=kcfi %s | FileCheck %s
+
+; CHECK-LABEL: define void @f1(
+define void @f1(ptr noundef %x) {
+  ; CHECK:      %[[#GEPI:]] = getelementptr inbounds i32, ptr %x, i32 -1
+  ; CHECK-NEXT: %[[#LOAD:]] = load i32, ptr %[[#GEPI]], align 4
+  ; CHECK-NEXT: %[[#ICMP:]] = icmp ne i32 %[[#LOAD]], 12345678
+  ; CHECK-NEXT: br i1 %[[#ICMP]], label %[[#TRAP:]], label %[[#CALL:]], !prof ![[#WEIGHTS:]]
+  ; CHECK:      [[#TRAP]]:
+  ; CHECK-NEXT: call void @llvm.trap()
+  ; CHECK-NEXT: br label %[[#CALL]]
+  ; CHECK:      [[#CALL]]:
+  ; CHECK-NEXT: call void %x()
+  ; CHECK-NOT:  [ "kcfi"(i32 12345678) ]
+  call void %x() [ "kcfi"(i32 12345678) ]
+  ret void
+}
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 4, !"kcfi", i32 1}
+; CHECK: ![[#WEIGHTS]] = !{!"branch_weights", i32 1, i32 1048575}

llvm/utils/gn/secondary/llvm/lib/Transforms/Instrumentation/BUILD.gn

sources = [
     "ControlHeightReduction.cpp",
     "DataFlowSanitizer.cpp",
     "GCOVProfiling.cpp",
     "HWAddressSanitizer.cpp",
     "IndirectCallPromotion.cpp",
     "InstrOrderFile.cpp",
     "InstrProfiling.cpp",
     "Instrumentation.cpp",
+    "KCFI.cpp",
     "MemProfiler.cpp",
     "MemorySanitizer.cpp",
     "PGOInstrumentation.cpp",
     "PGOMemOPSizeOpt.cpp",
     "PoisonChecking.cpp",
     "SanitizerBinaryMetadata.cpp",
     "SanitizerCoverage.cpp",
     "ThreadSanitizer.cpp",
     "ValueProfileCollector.cpp",
   ]
 }