[CodeGen] Introduce a FAULTING_LOAD_OP pseudo-op.

[sanjoy]

Summary

This instruction encodes a loading operation that may fault, and a label
to branch to if the load page-faults. The locations of potentially
faulting loads and their "handler" destinations are recorded in a
FaultMap section, meant to be consumed by LLVM's clients.

Nothing generates FAULTING_LOAD_OP instructions yet, but they will be
used in a future change.

The documentation (FaultMaps.rst) needs improvement and I will update
this diff with a more expanded version shortly.

Depends on D10196

[sanjoy]

ping

[pgavlin]

include/llvm/CodeGen/FaultMaps.h

2 ↗

(On Diff #27006)

Extra blank line.

15 ↗

(On Diff #27006)

Extra blank line.

54 ↗

(On Diff #27006)

Nit: s/FunctionFaultingInfo/FunctionFaultInfos/

lib/CodeGen/FaultMaps.cpp

11 ↗

(On Diff #27006)

Extra blank line.

75–86 ↗

(On Diff #27006)

What specifically about testing are you trying to make easier here? Are you just looking for determinism in the output order? If so, perhaps it would be better if FunctionInfos was a std::map, which should give you this property without the extra work.

FWIW, I think determinism here is useful beyond the scope of testing, so it would be nice to be able to provide it relatively cheaply.

lib/MC/MCObjectFileInfo.cpp

523 ↗

(On Diff #27006)

Could you also add code to initialize FaultMapSection for COFF objects?

lib/Target/X86/X86AsmPrinter.cpp

686 ↗

(On Diff #27006)

Please serialize the fault map section for COFF objects as well.

[atrick]

Looks good.

docs/FaultMaps.rst

20–26 ↗

(On Diff #27006)

Should this be an admonition?

52 ↗

(On Diff #27006)

"Handler"

[sanjoy]

• address review

docs/FaultMaps.rst

20–26 ↗

(On Diff #27006)

I don't think I understand -- what should be an admonition?

52 ↗

(On Diff #27006)

Fixed.

include/llvm/CodeGen/FaultMaps.h

2 ↗

(On Diff #27006)

Fixed.

15 ↗

(On Diff #27006)

This one was intentional, to visually separate the llvm/ includes from the stdlib includes. :)

54 ↗

(On Diff #27006)

Fixed.

lib/CodeGen/FaultMaps.cpp

11 ↗

(On Diff #27006)

That's intentional, to visually separate the header this file implements from the rest.

75–86 ↗

(On Diff #27006)

What specifically about testing are you trying to make easier here?

FileCheck based testing, I don't want tests to spuriously fail just because the order in which we emitted function infos changed. I'll change the data structure to an std::map and remove the sorting step.

lib/MC/MCObjectFileInfo.cpp

523 ↗

(On Diff #27006)

I don't know enough about COFF to confidently do that, and I don't have a stackmap example to copy from -- do you mind adding that bit of code once this change has landed?

[pgavlin]

I don't know enough about COFF to confidently do that, and I don't have a stackmap example to copy from -- do you mind adding that bit of code once this change has landed?

Sure, that sounds fine.

LGTM otherwise.

llvm/trunk/docs/FaultMaps.rst

+==============================
+FaultMaps and implicit checks
+==============================
+
+.. contents::
+   :local:
+   :depth: 2
+
+Motivation
+==========
+
+Code generated by managed language runtimes tend to have checks that
+are required for safety but never fail in practice.  In such cases, it
+is profitable to make the non-failing case cheaper even if it makes
+the failing case significantly more expensive.  This asymmetry can be
+exploited by folding such safety checks into operations that can be
+made to fault reliably if the check would have failed, and recovering
+from such a fault by using a signal handler.
+
+For example, Java requires null checks on objects before they are read
+from or written to.  If the object is ``null`` then a
+``NullPointerException`` has to be thrown, interrupting normal
+execution.  In practice, however, dereferencing a ``null`` pointer is
+extremely rare in well-behaved Java programs, and typically the null
+check can be folded into a nearby memory operation that operates on
+the same memory location.
+
+The Fault Map Section
+=====================
+
+Information about implicit checks generated by LLVM are put in a
+special "fault map" section.  On Darwin this section is named
+``__llvm_faultmaps``.
+
+The format of this section is
+
+.. code-block:: none
+
+  Header {
+    uint8  : Fault Map Version (current version is 1)
+    uint8  : Reserved (expected to be 0)
+    uint16 : Reserved (expected to be 0)
+  }
+  uint32 : NumFunctions
+  FunctionInfo[NumFunctions] {
+    uint64 : FunctionAddress
+    uint32 : NumFaultingPCs
+    uint32 : Reserved (expected to be 0)
+    FunctionFaultInfo[NumFaultingPCs] {
+      uint32  : FaultType = FaultMaps::FaultingLoad (only legal value currently)
+      uint32  : FaultingPCOffset
+      uint32  : handlerPCOffset
+    }
+  }

llvm/trunk/include/llvm/CodeGen/FaultMaps.h

+//===------------------- FaultMaps.h - StackMaps ----------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CODEGEN_FAULTMAPS_H
+#define LLVM_CODEGEN_FAULTMAPS_H
+
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/MC/MCSymbol.h"
+
+#include <vector>
+#include <map>
+
+namespace llvm {
+
+class AsmPrinter;
+class MCExpr;
+class MCSymbol;
+class MCStreamer;
+
+class FaultMaps {
+public:
+  enum FaultType { FaultingLoad = 1, FaultTypeMax };
+
+  static const char *faultTypeToString(FaultType);
+
+  explicit FaultMaps(AsmPrinter &AP);
+
+  void recordFaultingOp(FaultType FaultTy, const MCSymbol *HandlerLabel);
+  void serializeToFaultMapSection();
+
+private:
+  static const char *WFMP;
+
+  struct FaultInfo {
+    FaultType FaultType;
+    const MCExpr *FaultingOffsetExpr;
+    const MCExpr *HandlerOffsetExpr;
+
+    FaultInfo()
+        : FaultType(FaultTypeMax), FaultingOffsetExpr(nullptr),
+          HandlerOffsetExpr(nullptr) {}
+
+    explicit FaultInfo(FaultMaps::FaultType FType, const MCExpr *FaultingOffset,
+                       const MCExpr *HandlerOffset)
+        : FaultType(FType), FaultingOffsetExpr(FaultingOffset),
+          HandlerOffsetExpr(HandlerOffset) {}
+  };
+
+  typedef std::vector<FaultInfo> FunctionFaultInfos;
+
+  // We'd like to keep a stable iteration order for FunctionInfos to help
+  // FileCheck based testing.
+  struct MCSymbolComparator {
+    bool operator()(const MCSymbol *LHS, const MCSymbol *RHS) const {
+      return LHS->getName() < RHS->getName();
+    }
+  };
+
+  std::map<const MCSymbol *, FunctionFaultInfos, MCSymbolComparator>
+      FunctionInfos;
+  AsmPrinter &AP;
+
+  void emitFunctionInfo(const MCSymbol *FnLabel, const FunctionFaultInfos &FFI);
+};
+}
+
+#endif

llvm/trunk/include/llvm/MC/MCObjectFileInfo.h

protected:
   /// Section directive for Thread Local uninitialized data.
   ///
   /// Null if this target doesn't support a BSS section. ELF and MachO only.
   MCSection *TLSBSSSection; // Defaults to ".tbss".
 
   /// StackMap section.
   MCSection *StackMapSection;
 
+  /// FaultMap section.
+  MCSection *FaultMapSection;
+
   /// EH frame section.
   ///
   /// It is initialized on demand so it can be overwritten (with uniquing).
   MCSection *EHFrameSection;
 
   // ELF specific sections.
   MCSection *DataRelSection;
   const MCSection *DataRelLocalSection;
MCSection *getCOFFDebugSymbolsSection() const {
     return COFFDebugSymbolsSection;
   }
 
   MCSection *getTLSExtraDataSection() const { return TLSExtraDataSection; }
   const MCSection *getTLSDataSection() const { return TLSDataSection; }
   MCSection *getTLSBSSSection() const { return TLSBSSSection; }
 
   MCSection *getStackMapSection() const { return StackMapSection; }
+  MCSection *getFaultMapSection() const { return FaultMapSection; }
 
   // ELF specific sections.
   MCSection *getDataRelSection() const { return DataRelSection; }
   const MCSection *getDataRelLocalSection() const {
     return DataRelLocalSection;
   }
   MCSection *getDataRelROSection() const { return DataRelROSection; }
   MCSection *getDataRelROLocalSection() const { return DataRelROLocalSection; }

llvm/trunk/include/llvm/Target/Target.td

def FRAME_ALLOC : Instruction {
   // This instruction is really just a label. It has to be part of the chain so
   // that it doesn't get dropped from the DAG, but it produces nothing and has
   // no side effects.
   let OutOperandList = (outs);
   let InOperandList = (ins ptr_rc:$symbol, i32imm:$id);
   let hasSideEffects = 0;
   let hasCtrlDep = 1;
 }
+def FAULTING_LOAD_OP : Instruction {
+  let OutOperandList = (outs unknown:$dst);
+  let InOperandList = (ins variable_ops);
+  let usesCustomInserter = 1;
+  let mayLoad = 1;
+}
 }
 
 //===----------------------------------------------------------------------===//
 // AsmParser - This class can be implemented by targets that wish to implement
 // .s file parsing.
 //
 // Subtargets can have multiple different assembly parsers (e.g. AT&T vs Intel
 // syntax on X86 for example).

llvm/trunk/include/llvm/Target/TargetOpcodes.h

enum {
   /// intended to support garbage collection with fully precise relocating
   /// collectors and deoptimizations in either the callee or caller.
   STATEPOINT = 20,
 
   /// Instruction that records the offset of a function's frame allocation in a
   /// label. Created by the llvm.frameallocate intrinsic. It has two arguments:
   /// the symbol for the label and the frame index of the stack allocation.
   FRAME_ALLOC = 21,
+
+  /// Loading instruction that may page fault, bundled with associated
+  /// information on how to handle such a page fault.  It is intended to support
+  /// "zero cost" null checks in managed languages by allowing LLVM to fold
+  /// comparisions into existing memory operations.
+  FAULTING_LOAD_OP = 22,
 };
 } // end namespace TargetOpcode
 } // end namespace llvm
 
 #endif

llvm/trunk/lib/CodeGen/CMakeLists.txt

add_llvm_library(LLVMCodeGen
   DeadMachineInstructionElim.cpp
   DwarfEHPrepare.cpp
   EarlyIfConversion.cpp
   EdgeBundles.cpp
   ErlangGC.cpp
   ExecutionDepsFix.cpp
   ExpandISelPseudos.cpp
   ExpandPostRAPseudos.cpp
+  FaultMaps.cpp
   GCMetadata.cpp
   GCMetadataPrinter.cpp
   GCRootLowering.cpp
   GCStrategy.cpp
   GlobalMerge.cpp
   IfConversion.cpp
   InlineSpiller.cpp
   InterferenceCache.cpp

llvm/trunk/lib/CodeGen/FaultMaps.cpp

+//===---------------------------- FaultMaps.cpp ---------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/CodeGen/FaultMaps.h"
+
+#include "llvm/CodeGen/AsmPrinter.h"
+#include "llvm/MC/MCContext.h"
+#include "llvm/MC/MCExpr.h"
+#include "llvm/MC/MCObjectFileInfo.h"
+#include "llvm/MC/MCStreamer.h"
+#include "llvm/Support/Debug.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "faultmaps"
+
+static const int FaultMapVersion = 1;
+const char *FaultMaps::WFMP = "Fault Maps: ";
+
+FaultMaps::FaultMaps(AsmPrinter &AP) : AP(AP) {}
+
+void FaultMaps::recordFaultingOp(FaultType FaultTy,
+                                 const MCSymbol *HandlerLabel) {
+  MCContext &OutContext = AP.OutStreamer->getContext();
+  MCSymbol *FaultingLabel = OutContext.createTempSymbol();
+
+  AP.OutStreamer->EmitLabel(FaultingLabel);
+
+  const MCExpr *FaultingOffset = MCBinaryExpr::createSub(
+      MCSymbolRefExpr::create(FaultingLabel, OutContext),
+      MCSymbolRefExpr::create(AP.CurrentFnSymForSize, OutContext), OutContext);
+
+  const MCExpr *HandlerOffset = MCBinaryExpr::createSub(
+      MCSymbolRefExpr::create(HandlerLabel, OutContext),
+      MCSymbolRefExpr::create(AP.CurrentFnSymForSize, OutContext), OutContext);
+
+  FunctionInfos[AP.CurrentFnSym].emplace_back(FaultTy, FaultingOffset,
+                                              HandlerOffset);
+}
+
+void FaultMaps::serializeToFaultMapSection() {
+  if (FunctionInfos.empty())
+    return;
+
+  MCContext &OutContext = AP.OutStreamer->getContext();
+  MCStreamer &OS = *AP.OutStreamer;
+
+  // Create the section.
+  MCSection *FaultMapSection =
+      OutContext.getObjectFileInfo()->getFaultMapSection();
+  OS.SwitchSection(FaultMapSection);
+
+  // Emit a dummy symbol to force section inclusion.
+  OS.EmitLabel(OutContext.getOrCreateSymbol(Twine("__LLVM_FaultMaps")));
+
+  DEBUG(dbgs() << "********** Fault Map Output **********\n");
+
+  // Header
+  OS.EmitIntValue(FaultMapVersion, 1); // Version.
+  OS.EmitIntValue(0, 1);               // Reserved.
+  OS.EmitIntValue(0, 2);               // Reserved.
+
+  DEBUG(dbgs() << WFMP << "#functions = " << FunctionInfos.size() << "\n");
+  OS.EmitIntValue(FunctionInfos.size(), 4);
+
+  DEBUG(dbgs() << WFMP << "functions:\n");
+
+  for (const auto &FFI : FunctionInfos)
+    emitFunctionInfo(FFI.first, FFI.second);
+}
+
+void FaultMaps::emitFunctionInfo(const MCSymbol *FnLabel,
+                                 const FunctionFaultInfos &FFI) {
+  MCStreamer &OS = *AP.OutStreamer;
+
+  DEBUG(dbgs() << WFMP << "  function addr: " << *FnLabel << "\n");
+  OS.EmitSymbolValue(FnLabel, 8);
+
+  DEBUG(dbgs() << WFMP << "  #faulting PCs: " << FFI.size() << "\n");
+  OS.EmitIntValue(FFI.size(), 4);
+
+  OS.EmitIntValue(0, 4); // Reserved
+
+  for (auto &Fault : FFI) {
+    DEBUG(dbgs() << WFMP << "    fault type: "
+          << faultTypeToString(Fault.FaultType) << "\n");
+    OS.EmitIntValue(Fault.FaultType, 4);
+
+    DEBUG(dbgs() << WFMP << "    faulting PC offset: "
+          << *Fault.FaultingOffsetExpr << "\n");
+    OS.EmitValue(Fault.FaultingOffsetExpr, 4);
+
+    DEBUG(dbgs() << WFMP << "    fault handler PC offset: "
+          << *Fault.HandlerOffsetExpr << "\n");
+    OS.EmitValue(Fault.HandlerOffsetExpr, 4);
+  }
+}
+
+
+const char *FaultMaps::faultTypeToString(FaultMaps::FaultType FT) {
+  switch (FT) {
+  default:
+    llvm_unreachable("unhandled fault type!");
+
+  case FaultMaps::FaultingLoad:
+    return "FaultingLoad";
+  }
+}

llvm/trunk/lib/MC/MCObjectFileInfo.cpp

DwarfRangesSection =
       Ctx->getMachOSection("__DWARF", "__debug_ranges", MachO::S_ATTR_DEBUG,
                            SectionKind::getMetadata(), "debug_range");
   DwarfDebugInlineSection =
       Ctx->getMachOSection("__DWARF", "__debug_inlined", MachO::S_ATTR_DEBUG,
                            SectionKind::getMetadata());
   StackMapSection = Ctx->getMachOSection("__LLVM_STACKMAPS", "__llvm_stackmaps",
                                          0, SectionKind::getMetadata());
 
+  FaultMapSection = Ctx->getMachOSection("__LLVM_FAULTMAPS", "__llvm_faultmaps",
+                                         0, SectionKind::getMetadata());
+
   TLSExtraDataSection = TLSTLVSection;
 }
 
 void MCObjectFileInfo::initELFMCObjectFileInfo(Triple T) {
   switch (T.getArch()) {
   case Triple::mips:
   case Triple::mipsel:
     FDECFIEncoding = dwarf::DW_EH_PE_sdata4;
DwarfLocDWOSection =
       Ctx->getELFSection(".debug_loc.dwo", ELF::SHT_PROGBITS, 0, "skel_loc");
   DwarfStrOffDWOSection =
       Ctx->getELFSection(".debug_str_offsets.dwo", ELF::SHT_PROGBITS, 0);
   DwarfAddrSection =
       Ctx->getELFSection(".debug_addr", ELF::SHT_PROGBITS, 0, "addr_sec");
 
   StackMapSection =
       Ctx->getELFSection(".llvm_stackmaps", ELF::SHT_PROGBITS, ELF::SHF_ALLOC);
+
+  FaultMapSection =
+      Ctx->getELFSection(".llvm_faultmaps", ELF::SHT_PROGBITS, ELF::SHF_ALLOC);
 }
 
 void MCObjectFileInfo::initCOFFMCObjectFileInfo(Triple T) {
   bool IsWoA = T.getArch() == Triple::arm || T.getArch() == Triple::thumb;
 
   CommDirectiveSupportsAlignment = true;
 
   // COFF

llvm/trunk/lib/Target/X86/X86AsmPrinter.h

 //===-- X86AsmPrinter.h - X86 implementation of AsmPrinter ------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LIB_TARGET_X86_X86ASMPRINTER_H
 #define LLVM_LIB_TARGET_X86_X86ASMPRINTER_H
 
 #include "X86Subtarget.h"
 #include "llvm/CodeGen/AsmPrinter.h"
+#include "llvm/CodeGen/FaultMaps.h"
 #include "llvm/CodeGen/StackMaps.h"
 #include "llvm/Target/TargetMachine.h"
 
 // Implemented in X86MCInstLower.cpp
 namespace {
   class X86MCInstLower;
 }
 
 namespace llvm {
 class MCStreamer;
 class MCSymbol;
 
 class LLVM_LIBRARY_VISIBILITY X86AsmPrinter : public AsmPrinter {
   const X86Subtarget *Subtarget;
   StackMaps SM;
+  FaultMaps FM;
 
   void GenerateExportDirective(const MCSymbol *Sym, bool IsData);
 
   // This utility class tracks the length of a stackmap instruction's 'shadow'.
   // It is used by the X86AsmPrinter to ensure that the stackmap shadow
   // invariants (i.e. no other stackmaps, patchpoints, or control flow within
   // the shadow) are met, while outputting a minimal number of NOPs for padding.
   //
class LLVM_LIBRARY_VISIBILITY X86AsmPrinter : public AsmPrinter {
   // outputting it to the OutStream. This allows the shadow tracker to minimise
   // the number of NOPs used for stackmap padding.
   void EmitAndCountInstruction(MCInst &Inst);
 
   void InsertStackMapShadows(MachineFunction &MF);
   void LowerSTACKMAP(const MachineInstr &MI);
   void LowerPATCHPOINT(const MachineInstr &MI, X86MCInstLower &MCIL);
   void LowerSTATEPOINT(const MachineInstr &MI, X86MCInstLower &MCIL);
+  void LowerFAULTING_LOAD_OP(const MachineInstr &MI, X86MCInstLower &MCIL);
 
   void LowerTlsAddr(X86MCInstLower &MCInstLowering, const MachineInstr &MI);
 
  public:
    explicit X86AsmPrinter(TargetMachine &TM,
                           std::unique_ptr<MCStreamer> Streamer)
-       : AsmPrinter(TM, std::move(Streamer)), SM(*this), SMShadowTracker(TM) {}
+       : AsmPrinter(TM, std::move(Streamer)), SM(*this), FM(*this),
+         SMShadowTracker(TM) {}
 
   const char *getPassName() const override {
     return "X86 Assembly / Object Emitter";
   }
 
   const X86Subtarget &getSubtarget() const { return *Subtarget; }
 
   void EmitStartOfAsmFile(Module &M) override;

llvm/trunk/lib/Target/X86/X86AsmPrinter.cpp

if (!Stubs.empty()) {
       for (auto &Stub : Stubs)
         emitNonLazySymbolPointer(*OutStreamer, Stub.first, Stub.second);
 
       Stubs.clear();
       OutStreamer->AddBlankLine();
     }
 
     SM.serializeToStackMapSection();
+    FM.serializeToFaultMapSection();
 
     // Funny Darwin hack: This flag tells the linker that no global symbols
     // contain code that falls through to other global symbols (e.g. the obvious
     // implementation of multiple entry points).  If this doesn't occur, the
     // linker can safely perform dead code stripping.  Since LLVM never
     // generates code that does this, it is always safe to set.
     OutStreamer->EmitAssemblerFlag(MCAF_SubsectionsViaSymbols);
   }
if (!DLLExportedGlobals.empty() || !DLLExportedFns.empty()) {
 
       for (auto & Symbol : DLLExportedGlobals)
         GenerateExportDirective(Symbol, /*IsData=*/true);
       for (auto & Symbol : DLLExportedFns)
         GenerateExportDirective(Symbol, /*IsData=*/false);
     }
   }
 
-  if (TT.isOSBinFormatELF())
+  if (TT.isOSBinFormatELF()) {
     SM.serializeToStackMapSection();
+    FM.serializeToFaultMapSection();
+  }
 }
 
 //===----------------------------------------------------------------------===//
 // Target Registry Stuff
 //===----------------------------------------------------------------------===//
 
 // Force static initialization.
 extern "C" void LLVMInitializeX86AsmPrinter() {
   RegisterAsmPrinter<X86AsmPrinter> X(TheX86_32Target);
   RegisterAsmPrinter<X86AsmPrinter> Y(TheX86_64Target);
 }

llvm/trunk/lib/Target/X86/X86MCInstLower.cpp

if (unsigned PatchBytes = SOpers.getNumPatchBytes()) {
     OutStreamer->EmitInstruction(CallInst, getSubtargetInfo());
   }
 
   // Record our statepoint node in the same section used by STACKMAP
   // and PATCHPOINT
   SM.recordStatepoint(MI);
 }
 
+void X86AsmPrinter::LowerFAULTING_LOAD_OP(const MachineInstr &MI,
+                                       X86MCInstLower &MCIL) {
+  // FAULTING_LOAD_OP <def>, <handler label>, <load opcode>, <load operands>
+
+  unsigned LoadDefRegister = MI.getOperand(0).getReg();
+  MCSymbol *HandlerLabel = MI.getOperand(1).getMCSymbol();
+  unsigned LoadOpcode = MI.getOperand(2).getImm();
+  unsigned LoadOperandsBeginIdx = 3;
+
+  FM.recordFaultingOp(FaultMaps::FaultingLoad, HandlerLabel);
+
+  MCInst LoadMI;
+  LoadMI.setOpcode(LoadOpcode);
+  LoadMI.addOperand(MCOperand::createReg(LoadDefRegister));
+  for (auto I = MI.operands_begin() + LoadOperandsBeginIdx,
+            E = MI.operands_end();
+       I != E; ++I)
+    if (auto MaybeOperand = MCIL.LowerMachineOperand(&MI, *I))
+      LoadMI.addOperand(MaybeOperand.getValue());
+
+  OutStreamer->EmitInstruction(LoadMI, getSubtargetInfo());
+}
 
 // Lower a stackmap of the form:
 // <id>, <shadowBytes>, ...
 void X86AsmPrinter::LowerSTACKMAP(const MachineInstr &MI) {
   SMShadowTracker.emitShadowPadding(*OutStreamer, getSubtargetInfo());
   SM.recordStackMap(MI);
   unsigned NumShadowBytes = MI.getOperand(1).getImm();
   SMShadowTracker.reset(NumShadowBytes);
EmitAndCountInstruction(MCInstBuilder(X86::ADD32ri)
       .addReg(MI->getOperand(0).getReg())
       .addReg(MI->getOperand(1).getReg())
       .addExpr(DotExpr));
     return;
   }
   case TargetOpcode::STATEPOINT:
     return LowerSTATEPOINT(*MI, MCInstLowering);
 
+  case TargetOpcode::FAULTING_LOAD_OP:
+    return LowerFAULTING_LOAD_OP(*MI, MCInstLowering);
+
   case TargetOpcode::STACKMAP:
     return LowerSTACKMAP(*MI);
 
   case TargetOpcode::PATCHPOINT:
     return LowerPATCHPOINT(*MI, MCInstLowering);
 
   case X86::MORESTACK_RET:
     EmitAndCountInstruction(MCInstBuilder(getRetOpcode(*Subtarget)));

llvm/trunk/utils/TableGen/CodeGenTarget.cpp

 void CodeGenTarget::ComputeInstrsByEnum() const {
   // The ordering here must match the ordering in TargetOpcodes.h.
   static const char *const FixedInstrs[] = {
       "PHI",          "INLINEASM",     "CFI_INSTRUCTION",  "EH_LABEL",
       "GC_LABEL",     "KILL",          "EXTRACT_SUBREG",   "INSERT_SUBREG",
       "IMPLICIT_DEF", "SUBREG_TO_REG", "COPY_TO_REGCLASS", "DBG_VALUE",
       "REG_SEQUENCE", "COPY",          "BUNDLE",           "LIFETIME_START",
       "LIFETIME_END", "STACKMAP",      "PATCHPOINT",       "LOAD_STACK_GUARD",
-      "STATEPOINT",   "FRAME_ALLOC",
+      "STATEPOINT",   "FRAME_ALLOC",   "FAULTING_LOAD_OP",
       nullptr};
   const auto &Insts = getInstructions();
   for (const char *const *p = FixedInstrs; *p; ++p) {
     const CodeGenInstruction *Instr = GetInstByName(*p, Insts, Records);
     assert(Instr && "Missing target independent instruction");
     assert(Instr->Namespace == "TargetOpcode" && "Bad namespace");
     InstrsByEnum.push_back(Instr);
   }