[llvm][IR] Add no_cfi constant

[samitolvanen]

Summary

With Control-Flow Integrity (CFI), the LowerTypeTests pass replaces
function references with CFI jump table references, which is a problem
for low-level code that needs the address of the actual function body.

For example, in the Linux kernel, the code that sets up interrupt
handlers needs to take the address of the interrupt handler function
instead of the CFI jump table, as the jump table may not even be mapped
into memory when an interrupt is triggered.

This change adds the no_cfi constant type, which wraps function
references in a value that LowerTypeTestsModule::replaceCfiUses does not
replace.

Link: https://github.com/ClangBuiltLinux/linux/issues/1353

[samitolvanen]

Here's a PoC of the "no cfi" constant type suggested by Peter. PTAL.

[nickdesaulniers]

I think you might want to update the syntax files for the various text editors. See:

• llvm/utils/vim/syntax/llvm.vim
• llvm/utils/emacs/llvm-mode.el
• llvm/utils/kate/llvm.xml
• llvm/utils/vscode/llvm/syntaxes/ll.tmLanguage.yaml

llvm/include/llvm/IR/Constants.h

936

plz fix linter warning

llvm/lib/Transforms/IPO/LowerTypeTests.cpp

1778

Neat! I didn't know isa supported multiple types like this!

llvm/test/Bitcode/nocfivalue.ll

2–3

If the < is not necessary, let's drop it!

43

If @f1, @f2, @f4, and @f5 are "empty", want to simply use declare rather than define?

llvm/test/CodeGen/X86/nocfivalue.ll

2

Do we need <? If not, let's remove.

How does -asm-verbose=false change this test?

16–26

Can we change these to be simply declarations rather then empty function definitions?

llvm/test/Transforms/LowerTypeTests/nocfivalue.ll

10–23

declare?

[samitolvanen]

Fixed clang-tidy warnings, cleaned up tests.

[samitolvanen]

In D108478#2967870, @nickdesaulniers wrote:

I think you might want to update the syntax files for the various text editors. See:

• llvm/utils/vim/syntax/llvm.vim
• llvm/utils/emacs/llvm-mode.el
• llvm/utils/kate/llvm.xml
• llvm/utils/vscode/llvm/syntaxes/ll.tmLanguage.yaml

I was looking at what was done with dso_local_equivalent, which is similar constant type, and it looks like that's only in emacs/llvm-mode.el. I'm not sure how kate or vscode would display this keyword, but I added it to vim and emacs for now.

llvm/test/CodeGen/X86/nocfivalue.ll

2

How does -asm-verbose=false change this test?

It removes some comments from the output, which doesn't affect this test, but I suppose might reduce the probability of future breakages. I followed the example of other similar tests here.

16–26

Can we change these to be simply declarations rather then empty function definitions?

Only @f1 it seems. Using declare for an internal function results in an error, and the "cfi-canonical-jump-table" attribute is also ignored for a declaration.

llvm/test/Transforms/LowerTypeTests/nocfivalue.ll

10–23

declare?

Only @f1 again.

[samitolvanen]

Fixed bitcast handling in NoCFIValue::handleOperandChangeImpl.

[pcc]

llvm/lib/IR/Constants.cpp

1993

Shouldn't this use stripPointerCasts instead? Otherwise you're potentially replacing this with a different global.

1997

Can this ever be true if we're asserting that this is a GlobalValue?

llvm/lib/Transforms/IPO/LowerTypeTests.cpp

1777–1779

Prefer to say why, not what, in this comment.

[samitolvanen]

Addressed pcc's comments.

[samitolvanen]

Rebased.

[samitolvanen]

Rebased.

[nickdesaulniers]

LGTM. @pcc do you have any additional feedback?

llvm/lib/IR/Constants.cpp

1993

I wouldn't use auto here. If we don't reuse To, then perhaps simply

To = To->stripPointerCasts();

would do.

1994–1997

or you could do:

auto *GV =  dyn_cast<GlobalValue>(To);
assert(GV && "Can only replace the operands with a global value");

[samitolvanen]

Addressed Nick's comments about NoCFIValue::handleOperandChangeImpl.

[pcc]

LGTM

llvm/docs/LangRef.rst

 this can be used is for static offset calculations between a function and some other
 ``dso_local`` symbol. This is especially useful for the Relative VTables C++ ABI,
 where dynamic relocations for function pointers in VTables can be replaced with
 static relocations for offsets between the VTable and virtual functions which
 may not be ``dso_local``.
 
 This is currently only supported for ELF binary formats.
 
+.. _no_cfi:
+
+No CFI
+------
+
+``no_cfi @func``
+
+With `Control-Flow Integrity (CFI)
+<https://clang.llvm.org/docs/ControlFlowIntegrity.html>`_, a '``no_cfi``'
+constant represents a function reference that does not get replaced with a
+reference to the CFI jump table in the ``LowerTypeTests`` pass. These constants
+may be useful in low-level programs, such as operating system kernels, which
+need to refer to the actual function body.
+
 .. _constantexprs:
 
 Constant Expressions
 --------------------
 
 Constant expressions are used to allow expressions involving other
 constants to be used as constants. Constant expressions may be of any
 :ref:`first class <t_firstclass>` type and may involve any LLVM operation

llvm/include/llvm/AsmParser/LLParser.h

struct ValID {
     LLLexer::LocTy Loc;
     unsigned UIntVal;
     FunctionType *FTy = nullptr;
     std::string StrVal, StrVal2;
     APSInt APSIntVal;
     APFloat APFloatVal{0.0};
     Constant *ConstantVal;
     std::unique_ptr<Constant *[]> ConstantStructElts;
+    bool NoCFI = false;
 
     ValID() = default;
     ValID(const ValID &RHS)
         : Kind(RHS.Kind), Loc(RHS.Loc), UIntVal(RHS.UIntVal), FTy(RHS.FTy),
           StrVal(RHS.StrVal), StrVal2(RHS.StrVal2), APSIntVal(RHS.APSIntVal),
-          APFloatVal(RHS.APFloatVal), ConstantVal(RHS.ConstantVal) {
+          APFloatVal(RHS.APFloatVal), ConstantVal(RHS.ConstantVal),
+          NoCFI(RHS.NoCFI) {
       assert(!RHS.ConstantStructElts);
     }
 
     bool operator<(const ValID &RHS) const {
       if (Kind == t_LocalID || Kind == t_GlobalID)
         return UIntVal < RHS.UIntVal;
       assert((Kind == t_LocalName || Kind == t_GlobalName ||
               Kind == t_ConstantStruct || Kind == t_PackedConstantStruct) &&

llvm/include/llvm/AsmParser/LLToken.h

enum Kind {
 
   kw_extractelement,
   kw_insertelement,
   kw_shufflevector,
   kw_extractvalue,
   kw_insertvalue,
   kw_blockaddress,
   kw_dso_local_equivalent,
+  kw_no_cfi,
 
   kw_freeze,
 
   // Metadata types.
   kw_distinct,
 
   // Use-list order directives.
   kw_uselistorder,

llvm/include/llvm/Bitcode/LLVMBitCodes.h

enum ConstantsCodes {
   CST_CODE_BLOCKADDRESS = 21,    // CST_CODE_BLOCKADDRESS [fnty, fnval, bb#]
   CST_CODE_DATA = 22,            // DATA:          [n x elements]
   CST_CODE_INLINEASM_OLD2 = 23,  // INLINEASM:     [sideeffect|alignstack|
                                  //                 asmdialect,asmstr,conststr]
   CST_CODE_CE_GEP_WITH_INRANGE_INDEX = 24, //      [opty, flags, n x operands]
   CST_CODE_CE_UNOP = 25,                   // CE_UNOP:      [opcode, opval]
   CST_CODE_POISON = 26,                    // POISON
   CST_CODE_DSO_LOCAL_EQUIVALENT = 27,      // DSO_LOCAL_EQUIVALENT [gvty, gv]
-  CST_CODE_INLINEASM = 28, // INLINEASM:     [sideeffect|alignstack|
+  CST_CODE_INLINEASM = 28,    // INLINEASM:     [sideeffect|alignstack|
                               //                 asmdialect|unwind,
                               //                 asmstr,conststr]
+  CST_CODE_NO_CFI_VALUE = 29, // NO_CFI [ fty, f ]
 };
 
 /// CastOpcodes - These are values used in the bitcode files to encode which
 /// cast a CST_CODE_CE_CAST or a XXX refers to.  The values of these enums
 /// have no fixed relation to the LLVM IR enum values.  Changing these will
 /// break compatibility with old files.
 enum CastOpcodes {
   CAST_TRUNC = 0,

llvm/include/llvm/IR/Constants.h

 };
 
 template <>
 struct OperandTraits<DSOLocalEquivalent>
     : public FixedNumOperandTraits<DSOLocalEquivalent, 1> {};
 
 DEFINE_TRANSPARENT_OPERAND_ACCESSORS(DSOLocalEquivalent, Value)
 
+/// Wrapper for a value that won't be replaced with a CFI jump table
+/// pointer in LowerTypeTestsModule.
+class NoCFIValue final : public Constant {
+  friend class Constant;
+
+  NoCFIValue(GlobalValue *GV);
+
+  void *operator new(size_t S) { return User::operator new(S, 1); }
+
+  void destroyConstantImpl();
+  Value *handleOperandChangeImpl(Value *From, Value *To);
+
+public:
+  /// Return a NoCFIValue for the specified function.
+  static NoCFIValue *get(GlobalValue *GV);
+
+  /// Transparently provide more efficient getOperand methods.
+  DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
+
+  GlobalValue *getGlobalValue() const {
+    return cast<GlobalValue>(Op<0>().get());
+  }
+
+  /// Methods for support type inquiry through isa, cast, and dyn_cast:
+  static bool classof(const Value *V) {
+    return V->getValueID() == NoCFIValueVal;
+  }
+};
+
+template <>
+struct OperandTraits<NoCFIValue> : public FixedNumOperandTraits<NoCFIValue, 1> {
+};
+
+DEFINE_TRANSPARENT_OPERAND_ACCESSORS(NoCFIValue, Value)
+
 //===----------------------------------------------------------------------===//
 /// A constant value that is initialized with an expression using
 /// other constant values.
 ///
 /// This class uses the standard Instruction opcodes to define the various
 /// constant expressions.  The Opcode field for the ConstantExpr class is
 /// maintained in the Value::SubclassData field.
 class ConstantExpr : public Constant {

llvm/include/llvm/IR/Value.def

 // Constant
 HANDLE_GLOBAL_VALUE(Function)
 HANDLE_GLOBAL_VALUE(GlobalAlias)
 HANDLE_GLOBAL_VALUE(GlobalIFunc)
 HANDLE_GLOBAL_VALUE(GlobalVariable)
 HANDLE_CONSTANT(BlockAddress)
 HANDLE_CONSTANT(ConstantExpr)
 HANDLE_CONSTANT_EXCLUDE_LLVM_C_API(DSOLocalEquivalent)
+HANDLE_CONSTANT_EXCLUDE_LLVM_C_API(NoCFIValue)
 
 // ConstantAggregate.
 HANDLE_CONSTANT(ConstantArray)
 HANDLE_CONSTANT(ConstantStruct)
 HANDLE_CONSTANT(ConstantVector)
 
 // ConstantData.
 HANDLE_CONSTANT(UndefValue)

llvm/lib/AsmParser/LLLexer.cpp

#define KEYWORD(STR)                                                           \
 
   KEYWORD(xchg); KEYWORD(nand); KEYWORD(max); KEYWORD(min); KEYWORD(umax);
   KEYWORD(umin);
 
   KEYWORD(vscale);
   KEYWORD(x);
   KEYWORD(blockaddress);
   KEYWORD(dso_local_equivalent);
+  KEYWORD(no_cfi);
 
   // Metadata types.
   KEYWORD(distinct);
 
   // Use-list order directives.
   KEYWORD(uselistorder);
   KEYWORD(uselistorder_bb);
 

llvm/lib/AsmParser/LLParser.cpp

if (!GV->getValueType()->isFunctionTy())
       return error(Fn.Loc, "expected a function, alias to function, or ifunc "
                            "in dso_local_equivalent");
 
     ID.ConstantVal = DSOLocalEquivalent::get(GV);
     ID.Kind = ValID::t_Constant;
     return false;
   }
 
+  case lltok::kw_no_cfi: {
+    // ValID ::= 'no_cfi' @foo
+    Lex.Lex();
+
+    if (parseValID(ID, PFS))
+      return true;
+
+    if (ID.Kind != ValID::t_GlobalID && ID.Kind != ValID::t_GlobalName)
+      return error(ID.Loc, "expected global value name in no_cfi");
+
+    ID.NoCFI = true;
+    return false;
+  }
+
   case lltok::kw_trunc:
   case lltok::kw_zext:
   case lltok::kw_sext:
   case lltok::kw_fptrunc:
   case lltok::kw_fpext:
   case lltok::kw_bitcast:
   case lltok::kw_addrspacecast:
   case lltok::kw_uitofp:
if (!ID.FTy || !InlineAsm::Verify(ID.FTy, ID.StrVal2))
       return error(ID.Loc, "invalid type for inline asm constraint string");
     V = InlineAsm::get(
         ID.FTy, ID.StrVal, ID.StrVal2, ID.UIntVal & 1, (ID.UIntVal >> 1) & 1,
         InlineAsm::AsmDialect((ID.UIntVal >> 2) & 1), (ID.UIntVal >> 3) & 1);
     return false;
   }
   case ValID::t_GlobalName:
     V = getGlobalVal(ID.StrVal, Ty, ID.Loc);
+    if (V && ID.NoCFI)
+      V = NoCFIValue::get(cast<GlobalValue>(V));
     return V == nullptr;
   case ValID::t_GlobalID:
     V = getGlobalVal(ID.UIntVal, Ty, ID.Loc);
+    if (V && ID.NoCFI)
+      V = NoCFIValue::get(cast<GlobalValue>(V));
     return V == nullptr;
   case ValID::t_APSInt:
     if (!Ty->isIntegerTy())
       return error(ID.Loc, "integer constant must have integer type");
     ID.APSIntVal = ID.APSIntVal.extOrTrunc(Ty->getPrimitiveSizeInBits());
     V = ConstantInt::get(Context, ID.APSIntVal);
     return false;
   case ValID::t_APFloat:

llvm/lib/Bitcode/Reader/BitcodeAnalyzer.cpp

default:
       STRINGIFY_CODE(CST_CODE, CE_EXTRACTELT)
       STRINGIFY_CODE(CST_CODE, CE_INSERTELT)
       STRINGIFY_CODE(CST_CODE, CE_SHUFFLEVEC)
       STRINGIFY_CODE(CST_CODE, CE_CMP)
       STRINGIFY_CODE(CST_CODE, INLINEASM)
       STRINGIFY_CODE(CST_CODE, CE_SHUFVEC_EX)
       STRINGIFY_CODE(CST_CODE, CE_UNOP)
       STRINGIFY_CODE(CST_CODE, DSO_LOCAL_EQUIVALENT)
+      STRINGIFY_CODE(CST_CODE, NO_CFI_VALUE)
     case bitc::CST_CODE_BLOCKADDRESS:
       return "CST_CODE_BLOCKADDRESS";
       STRINGIFY_CODE(CST_CODE, DATA)
     }
   case bitc::FUNCTION_BLOCK_ID:
     switch (CodeID) {
     default:
       return None;

llvm/lib/Bitcode/Reader/BitcodeReader.cpp

case bitc::CST_CODE_DSO_LOCAL_EQUIVALENT: {
       GlobalValue *GV = dyn_cast_or_null<GlobalValue>(
           ValueList.getConstantFwdRef(Record[1], GVTy));
       if (!GV)
         return error("Invalid record");
 
       V = DSOLocalEquivalent::get(GV);
       break;
     }
+    case bitc::CST_CODE_NO_CFI_VALUE: {
+      if (Record.size() < 2)
+        return error("Invalid record");
+      Type *GVTy = getTypeByID(Record[0]);
+      if (!GVTy)
+        return error("Invalid record");
+      GlobalValue *GV = dyn_cast_or_null<GlobalValue>(
+          ValueList.getConstantFwdRef(Record[1], GVTy));
+      if (!GV)
+        return error("Invalid record");
+      V = NoCFIValue::get(GV);
+      break;
+    }
     }
 
     ValueList.assignValue(V, NextCstNo);
     ++NextCstNo;
   }
 }
 
 Error BitcodeReader::parseUseLists() {

llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

if (C->isNullValue()) {
       Code = bitc::CST_CODE_BLOCKADDRESS;
       Record.push_back(VE.getTypeID(BA->getFunction()->getType()));
       Record.push_back(VE.getValueID(BA->getFunction()));
       Record.push_back(VE.getGlobalBasicBlockID(BA->getBasicBlock()));
     } else if (const auto *Equiv = dyn_cast<DSOLocalEquivalent>(C)) {
       Code = bitc::CST_CODE_DSO_LOCAL_EQUIVALENT;
       Record.push_back(VE.getTypeID(Equiv->getGlobalValue()->getType()));
       Record.push_back(VE.getValueID(Equiv->getGlobalValue()));
+    } else if (const auto *NC = dyn_cast<NoCFIValue>(C)) {
+      Code = bitc::CST_CODE_NO_CFI_VALUE;
+      Record.push_back(VE.getTypeID(NC->getGlobalValue()->getType()));
+      Record.push_back(VE.getValueID(NC->getGlobalValue()));
     } else {
 #ifndef NDEBUG
       C->dump();
 #endif
       llvm_unreachable("Unknown constant!");
     }
     Stream.EmitRecord(Code, Record, AbbrevToUse);
     Record.clear();

llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp

if (const GlobalValue *GV = dyn_cast<GlobalValue>(CV))
     return MCSymbolRefExpr::create(getSymbol(GV), Ctx);
 
   if (const BlockAddress *BA = dyn_cast<BlockAddress>(CV))
     return MCSymbolRefExpr::create(GetBlockAddressSymbol(BA), Ctx);
 
   if (const auto *Equiv = dyn_cast<DSOLocalEquivalent>(CV))
     return getObjFileLowering().lowerDSOLocalEquivalent(Equiv, TM);
 
+  if (const NoCFIValue *NC = dyn_cast<NoCFIValue>(CV))
+    return MCSymbolRefExpr::create(getSymbol(NC->getGlobalValue()), Ctx);
+
   const ConstantExpr *CE = dyn_cast<ConstantExpr>(CV);
   if (!CE) {
     llvm_unreachable("Unknown constant value to lower!");
   }
 
   switch (CE->getOpcode()) {
   case Instruction::AddrSpaceCast: {
     const Constant *Op = CE->getOperand(0);

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

if (const Constant *C = dyn_cast<Constant>(V)) {
     }
 
     if (const BlockAddress *BA = dyn_cast<BlockAddress>(C))
       return DAG.getBlockAddress(BA, VT);
 
     if (const auto *Equiv = dyn_cast<DSOLocalEquivalent>(C))
       return getValue(Equiv->getGlobalValue());
 
+    if (const auto *NC = dyn_cast<NoCFIValue>(C))
+      return getValue(NC->getGlobalValue());
+
     VectorType *VecTy = cast<VectorType>(V->getType());
 
     // Now that we know the number and type of the elements, get that number of
     // elements into the Ops array based on what kind of constant it is.
     if (const ConstantVector *CV = dyn_cast<ConstantVector>(C)) {
       SmallVector<SDValue, 16> Ops;
       unsigned NumElements = cast<FixedVectorType>(VecTy)->getNumElements();
       for (unsigned i = 0; i != NumElements; ++i)

llvm/lib/IR/AsmWriter.cpp

static void WriteConstantInternal(raw_ostream &Out, const Constant *CV,
   }
 
   if (const auto *Equiv = dyn_cast<DSOLocalEquivalent>(CV)) {
     Out << "dso_local_equivalent ";
     WriteAsOperandInternal(Out, Equiv->getGlobalValue(), WriterCtx);
     return;
   }
 
+  if (const auto *NC = dyn_cast<NoCFIValue>(CV)) {
+    Out << "no_cfi ";
+    WriteAsOperandInternal(Out, NC->getGlobalValue(), WriterCtx);
+    return;
+  }
+
   if (const ConstantArray *CA = dyn_cast<ConstantArray>(CV)) {
     Type *ETy = CA->getType()->getElementType();
     Out << '[';
     WriterCtx.TypePrinter->print(ETy, Out);
     Out << ' ';
     WriteAsOperandInternal(Out, CA->getOperand(0), WriterCtx);
     for (unsigned i = 1, e = CA->getNumOperands(); i != e; ++i) {
       Out << ", ";

llvm/lib/IR/Constants.cpp

case Constant::ConstantTokenNoneVal:
     delete static_cast<ConstantTokenNone *>(C);
     break;
   case Constant::BlockAddressVal:
     delete static_cast<BlockAddress *>(C);
     break;
   case Constant::DSOLocalEquivalentVal:
     delete static_cast<DSOLocalEquivalent *>(C);
     break;
+  case Constant::NoCFIValueVal:
+    delete static_cast<NoCFIValue *>(C);
+    break;
   case Constant::UndefValueVal:
     delete static_cast<UndefValue *>(C);
     break;
   case Constant::PoisonValueVal:
     delete static_cast<PoisonValue *>(C);
     break;
   case Constant::ConstantExprVal:
     if (isa<UnaryConstantExpr>(C))
Value *DSOLocalEquivalent::handleOperandChangeImpl(Value *From, Value *To) {
   if (Func->getType() != getType()) {
     // It is ok to mutate the type here because this constant should always
     // reflect the type of the function it's holding.
     mutateType(Func->getType());
   }
   return nullptr;
 }
 
+NoCFIValue *NoCFIValue::get(GlobalValue *GV) {
+  NoCFIValue *&NC = GV->getContext().pImpl->NoCFIValues[GV];
+  if (!NC)
+    NC = new NoCFIValue(GV);
+
+  assert(NC->getGlobalValue() == GV &&
+         "NoCFIValue does not match the expected global value");
+  return NC;
+}
+
+NoCFIValue::NoCFIValue(GlobalValue *GV)
+    : Constant(GV->getType(), Value::NoCFIValueVal, &Op<0>(), 1) {
+  setOperand(0, GV);
+}
+
+/// Remove the constant from the constant table.
+void NoCFIValue::destroyConstantImpl() {
+  const GlobalValue *GV = getGlobalValue();
+  GV->getContext().pImpl->NoCFIValues.erase(GV);
+}
+
+Value *NoCFIValue::handleOperandChangeImpl(Value *From, Value *To) {
+  assert(From == getGlobalValue() && "Changing value does not match operand.");
+
+  GlobalValue *GV = dyn_cast<GlobalValue>(To->stripPointerCasts());
+  assert(GV && "Can only replace the operands with a global value");
+
+  NoCFIValue *&NewNC = getContext().pImpl->NoCFIValues[GV];
+  if (NewNC)
+    return llvm::ConstantExpr::getBitCast(NewNC, getType());
+
+  getContext().pImpl->NoCFIValues.erase(getGlobalValue());
+  NewNC = this;
+  setOperand(0, GV);
+
+  if (GV->getType() != getType())
+    mutateType(GV->getType());
+
+  return nullptr;
+}
+
 //---- ConstantExpr::get() implementations.
 //
 
 /// This is a utility function to handle folding of casts and lookup of the
 /// cast in the ExprConstants map. It is used by the various get* methods below.
 static Constant *getFoldedCast(Instruction::CastOps opc, Constant *C, Type *Ty,
                                bool OnlyIfReduced = false) {
   assert(Ty->isFirstClassType() && "Cannot cast to an aggregate type!");

llvm/lib/IR/LLVMContextImpl.h

#include "llvm/IR/Metadata.def"
 
   StringMap<std::unique_ptr<ConstantDataSequential>> CDSConstants;
 
   DenseMap<std::pair<const Function *, const BasicBlock *>, BlockAddress *>
       BlockAddresses;
 
   DenseMap<const GlobalValue *, DSOLocalEquivalent *> DSOLocalEquivalents;
 
+  DenseMap<const GlobalValue *, NoCFIValue *> NoCFIValues;
+
   ConstantUniqueMap<ConstantExpr> ExprConstants;
 
   ConstantUniqueMap<InlineAsm> InlineAsms;
 
   ConstantInt *TheTrueVal = nullptr;
   ConstantInt *TheFalseVal = nullptr;
 
   std::unique_ptr<ConstantTokenNone> TheNoneToken;

llvm/lib/Transforms/IPO/LowerTypeTests.cpp

static bool isDirectCall(Use& U) {
   }
   return false;
 }
 
 void LowerTypeTestsModule::replaceCfiUses(Function *Old, Value *New,
                                           bool IsJumpTableCanonical) {
   SmallSetVector<Constant *, 4> Constants;
   for (Use &U : llvm::make_early_inc_range(Old->uses())) {
-    // Skip block addresses
-    if (isa<BlockAddress>(U.getUser()))
+    // Skip block addresses and no_cfi values, which refer to the function
+    // body instead of the jump table.
+    if (isa<BlockAddress, NoCFIValue>(U.getUser()))
       continue;
 
     // Skip direct calls to externally defined or non-dso_local functions
     if (isDirectCall(U) && (Old->isDSOLocal() || !IsJumpTableCanonical))
       continue;
 
     // Must handle Constants specially, we cannot call replaceUsesOfWith on a
     // constant because they are uniqued.

llvm/lib/Transforms/Utils/ValueMapper.cpp

if (const auto *E = dyn_cast<DSOLocalEquivalent>(C)) {
     auto *Func = cast<Function>(Val->stripPointerCastsAndAliases());
     Type *NewTy = E->getType();
     if (TypeMapper)
       NewTy = TypeMapper->remapType(NewTy);
     return getVM()[E] = llvm::ConstantExpr::getBitCast(
                DSOLocalEquivalent::get(Func), NewTy);
   }
 
+  if (const auto *NC = dyn_cast<NoCFIValue>(C)) {
+    auto *Val = mapValue(NC->getGlobalValue());
+    GlobalValue *GV = cast<GlobalValue>(Val);
+    return getVM()[NC] = NoCFIValue::get(GV);
+  }
+
   auto mapValueOrNull = [this](Value *V) {
     auto Mapped = mapValue(V);
     assert((Mapped || (Flags & RF_NullMapMissingGlobalValues)) &&
            "Unexpected null mapping for constant operand without "
            "NullMapMissingGlobalValues flag");
     return Mapped;
   };
 

llvm/test/Bitcode/nocfivalue.ll

+; RUN: llvm-as < %s | llvm-dis | FileCheck %s
+; RUN: verify-uselistorder %s
+
+; CHECK: @a = global [4 x void ()*] [void ()* no_cfi @f1, void ()* @f1, void ()* @f2, void ()* no_cfi @f2]
+@a = global [4 x void ()*] [void ()* no_cfi @f1, void ()* @f1, void ()* @f2, void ()* no_cfi @f2]
+; CHECK: @b = constant void ()* no_cfi @f3
+@b = constant void ()* no_cfi @f3
+; CHECK: @c = constant void ()* @f3
+@c = constant void ()* @f3
+
+; CHECK: declare void @f1()
+declare void @f1()
+
+; CHECK: declare void @f2()
+declare void @f2()
+
+; CHECK: define void @f3()
+define void @f3() {
+  ; CHECK: call void no_cfi @f4()
+  call void no_cfi @f4()
+  ; CHECK: call void @f4()
+  call void @f4()
+  ; CHECK: call void no_cfi @f5()
+  call void no_cfi @f5()
+  ; CHECK: call void @f5()
+  call void @f5()
+  ret void
+}
+
+; CHECK: declare void @f4()
+declare void @f4()
+
+; CHECK: declare void @f5()
+declare void @f5()
+
+define void @g() {
+  %n = alloca void ()*, align 8
+  ; CHECK: store void ()* no_cfi @f5, void ()** %n, align 8
+  store void ()* no_cfi @f5, void ()** %n, align 8
+  %1 = load void ()*, void ()** %n
+  call void %1()
+  ret void
+}

llvm/test/CodeGen/X86/nocfivalue.ll

+; RUN: opt -S -lowertypetests %s | llc -asm-verbose=false | FileCheck %s
+
+target datalayout = "e-p:64:64"
+target triple = "x86_64-unknown-linux-gnu"
+
+; CHECK: a:
+; CHECK-NEXT: .quad   f1
+; CHECK-NEXT: .quad   .L.cfi.jumptable
+; CHECK-NEXT: .quad   .L.cfi.jumptable
+; CHECK-NEXT: .quad   f2
+; CHECK-NEXT: .quad   f3
+; CHECK-NEXT: .quad   f3.cfi
+@a = global [6 x void ()*] [void ()* no_cfi @f1, void ()* @f1, void ()* @f2, void ()* no_cfi @f2, void ()* @f3, void ()* no_cfi @f3]
+
+declare !type !0 void @f1()
+
+define internal void @f2() !type !0 {
+  ret void
+}
+
+define void @f3() #0 !type !0 {
+  ret void
+}
+
+declare i1 @llvm.type.test(i8* %ptr, metadata %bitset) nounwind readnone
+
+define i1 @foo(i8* %p) {
+  %x = call i1 @llvm.type.test(i8* %p, metadata !"typeid1")
+  ret i1 %x
+}
+
+!llvm.module.flags = !{!1}
+
+attributes #0 = { "cfi-canonical-jump-table" }
+
+!0 = !{i32 0, !"typeid1"}
+!1 = !{i32 4, !"CFI Canonical Jump Tables", i32 0}

llvm/test/Transforms/LowerTypeTests/nocfivalue.ll

+; RUN: opt -S -lowertypetests < %s | FileCheck %s
+
+target datalayout = "e-p:64:64"
+target triple = "x86_64-unknown-linux-gnu"
+
+; CHECK: @a = global [6 x void ()*] [void ()* no_cfi @f1, void ()* @.cfi.jumptable, void ()* bitcast ([8 x i8]* getelementptr inbounds ([3 x [8 x i8]], [3 x [8 x i8]]* bitcast (void ()* @.cfi.jumptable to [3 x [8 x i8]]*), i64 0, i64 1) to void ()*), void ()* no_cfi @f2, void ()* @f3, void ()* no_cfi @f3.cfi]
+@a = global [6 x void ()*] [void ()* no_cfi @f1, void ()* @f1, void ()* @f2, void ()* no_cfi @f2, void ()* @f3, void ()* no_cfi @f3]
+
+; CHECK: define void @f1()
+define void @f1() !type !0 {
+  ret void
+}
+
+; CHECK: define internal void @f2()
+define internal void @f2() !type !0 {
+  ret void
+}
+
+; CHECK: define hidden void @f3.cfi()
+define void @f3() #0 !type !0 {
+  ret void
+}
+
+declare i1 @llvm.type.test(i8* %ptr, metadata %bitset) nounwind readnone
+
+define i1 @foo(i8* %p) {
+  %x = call i1 @llvm.type.test(i8* %p, metadata !"typeid1")
+  ret i1 %x
+}
+
+!llvm.module.flags = !{!1}
+
+attributes #0 = { "cfi-canonical-jump-table" }
+
+!0 = !{i32 0, !"typeid1"}
+!1 = !{i32 4, !"CFI Canonical Jump Tables", i32 0}

llvm/utils/emacs/llvm-mode.el

`(,(regexp-opt
          "default" "hidden" "protected"
          ;; DLL storages
          "dllimport" "dllexport"
          ;; Thread local
          "thread_local" "localdynamic" "initialexec" "localexec"
          ;; Runtime preemption specifiers
          "dso_preemptable" "dso_local" "dso_local_equivalent"
 
-         "gc" "atomic" "volatile" "personality" "prologue" "section") 'symbols) . font-lock-keyword-face)
+         "gc" "atomic" "no_cfi" "volatile" "personality" "prologue" "section") 'symbols) . font-lock-keyword-face)
    ;; Arithmetic and Logical Operators
    `(,(regexp-opt '("add" "sub" "mul" "sdiv" "udiv" "urem" "srem" "and" "or" "xor"
                     "setne" "seteq" "setlt" "setgt" "setle" "setge") 'symbols) . font-lock-keyword-face)
    ;; Floating-point operators
    `(,(regexp-opt '("fadd" "fsub" "fneg" "fmul" "fdiv" "frem") 'symbols) . font-lock-keyword-face)
    ;; Special instructions
    `(,(regexp-opt '("phi" "tail" "call" "select" "to" "shl" "lshr" "ashr" "fcmp" "icmp" "va_arg" "landingpad" "freeze") 'symbols) . font-lock-keyword-face)
    ;; Control instructions

llvm/utils/vim/syntax/llvm.vim

syn keyword llvmKeyword
       \ musttail
       \ naked
       \ nest
       \ noalias
       \ nobuiltin
       \ nocallback
       \ nocapture
       \ nocf_check
+      \ no_cfi
       \ noduplicate
       \ nofree
       \ noimplicitfloat
       \ noinline
       \ nomerge
       \ nonlazybind
       \ nonnull
       \ noprofile