[ImplicitNullCheck] Extend Implicit Null Check scope by using stores

[skatkov]

Summary

This change allows usage of store instruction for implicit null check.

Memory Aliasing Analisys is not used and change conservatively supposes
that any store and load may access the same memory. As a result
re-ordering of store-store, store-load and load-store is prohibited.

[sanjoy]

If it is not too complicated, I'd rather have a single FAULTING_OP pseudo-instruction, and wrap loads, stores and load-stores into it. We can then later re-use it for divisions as well (implicit division by zero checking).

Also, please update docs/FaultMaps.rst.

lib/Target/X86/X86MCInstLower.cpp

976

When is StoreDefRegister not X86::NoRegister?

[skatkov]

Please take a look.

[sanjoy]

docs/FaultMaps.rst

49 ↗

(On Diff #86775)

Please document the two possible values of this FaultKind field.

include/llvm/CodeGen/FaultMaps.h

29–30

I found the earlier classification system better, with a third FaultingLoadStore category. Right now we don't have a place to put addq (%rax), %rbx type instructions, which both load and store.

lib/CodeGen/ImplicitNullChecks.cpp

302

s/no be/not be/

303

I'd s/usage of memory aliasing analysis/using alias analysis/

307

How about SeenLoad |= MI.mayLoad()?

309

It is more idiomatic to say "alias analysis" instead of "Memory Aliasing Analysis".

311

Please call this something more descriptive than cont.

586

s/instr/instruction/

lib/Target/X86/X86MCInstLower.cpp

914

Later on (in a separate change), perhaps we can rewrite this as:

for (auto &MO : make_range(FaultingMI.operands_begin() + OperandsBeginIdx, FaultingMI.operands_end())) {
  // Use MO instead of I
}

[skatkov]

Thank you, for review. I will update a change.

include/llvm/CodeGen/FaultMaps.h

29–30

I considered that for that kind of instruction we first read and failed on this :) But ok, I will return it back.

lib/CodeGen/ImplicitNullChecks.cpp

307

Sanjoy, as I understand in my case mayLoad will be einvoked only if SeenLoad == false, in |= form mayLoad will be invoked always and there is no ||= operator.

[sanjoy]

lgtm, will check this in for you now after running clang-format

(Please ask Chris Lattner for commit access -- http://llvm.org/docs/DeveloperPolicy.html#obtaining-commit-access :) )

include/llvm/CodeGen/FaultMaps.h

29–30

Okay, if all of the load-and-store instructions load before storing then I'm fine changing it back in a later change (but let's not waste another review cycle on it right now).

lib/CodeGen/ImplicitNullChecks.cpp

307

I'd normally assume that the host compiler would be smart enough to optimize SeenLoad || MI.MayLoad() into SeenLoad | MI.MayLoad(), but your decision is fine by me.

[skatkov]

In D29400#669471, @sanjoy wrote:

lgtm, will check this in for you now after running clang-format

(Please ask Chris Lattner for commit access -- http://llvm.org/docs/DeveloperPolicy.html#obtaining-commit-access :) )

Actually I decided that I will do that after my 10-th patch - some kind of anniversary :)
Thank you, Sanjoy for review and assistance in landing my patches.

include/llvm/CodeGen/FaultMaps.h

 
 class AsmPrinter;
 class MCExpr;
 class MCSymbol;
 class MCStreamer;
 
 class FaultMaps {
 public:
-  enum FaultKind { FaultingLoad = 1, FaultKindMax };
+  enum FaultKind { FaultingLoad = 1, FaultingLoadStore,
+                   FaultingStore, FaultKindMax };
 
   static const char *faultTypeToString(FaultKind);
 
   explicit FaultMaps(AsmPrinter &AP);
 
   void recordFaultingOp(FaultKind FaultTy, const MCSymbol *HandlerLabel);
   void serializeToFaultMapSection();
 

include/llvm/Target/Target.td

 def FAULTING_LOAD_OP : Instruction {
   let OutOperandList = (outs unknown:$dst);
   let InOperandList = (ins variable_ops);
   let usesCustomInserter = 1;
   let mayLoad = 1;
   let isTerminator = 1;
   let isBranch = 1;
 }
+def FAULTING_LOAD_STORE_OP : Instruction {
+  let OutOperandList = (outs unknown:$dst);
+  let InOperandList = (ins variable_ops);
+  let usesCustomInserter = 1;
+  let mayLoad = 1;
+  let mayStore = 1;
+  let isTerminator = 1;
+  let isBranch = 1;
+}
+def FAULTING_STORE_OP : Instruction {
+  let OutOperandList = (outs unknown:$dst);
+  let InOperandList = (ins variable_ops);
+  let usesCustomInserter = 1;
+  let mayStore = 1;
+  let isTerminator = 1;
+  let isBranch = 1;
+}
 def PATCHABLE_OP : Instruction {
   let OutOperandList = (outs unknown:$dst);
   let InOperandList = (ins variable_ops);
   let usesCustomInserter = 1;
   let mayLoad = 1;
   let mayStore = 1;
   let hasSideEffects = 1;
 }

include/llvm/Target/TargetOpcodes.def

 HANDLE_TARGET_OPCODE(LOCAL_ESCAPE)
 
 /// Loading instruction that may page fault, bundled with associated
 /// information on how to handle such a page fault.  It is intended to support
 /// "zero cost" null checks in managed languages by allowing LLVM to fold
 /// comparisons into existing memory operations.
 HANDLE_TARGET_OPCODE(FAULTING_LOAD_OP)
 
+/// Loading and storing at once instruction that may page fault,
+/// bundled with associated information on how to handle such a page fault.
+/// It is intended to support "zero cost" null checks in managed languages
+/// by allowing LLVM to fold comparisons into existing memory operations.
+HANDLE_TARGET_OPCODE(FAULTING_LOAD_STORE_OP)
+
+/// Storing instruction that may page fault, bundled with associated
+/// information on how to handle such a page fault.  It is intended to support
+/// "zero cost" null checks in managed languages by allowing LLVM to fold
+/// comparisons into existing memory operations.
+HANDLE_TARGET_OPCODE(FAULTING_STORE_OP)
+
 /// Wraps a machine instruction to add patchability constraints.  An
 /// instruction wrapped in PATCHABLE_OP has to either have a minimum
 /// size or be preceded with a nop of that size.  The first operand is
 /// an immediate denoting the minimum size of the instruction, the
 /// second operand is an immediate denoting the opcode of the original
 /// instruction.  The rest of the operands are the operands of the
 /// original instruction.
 HANDLE_TARGET_OPCODE(PATCHABLE_OP)

lib/CodeGen/FaultMaps.cpp

 
 const char *FaultMaps::faultTypeToString(FaultMaps::FaultKind FT) {
   switch (FT) {
   default:
     llvm_unreachable("unhandled fault type!");
 
   case FaultMaps::FaultingLoad:
     return "FaultingLoad";
+  case FaultMaps::FaultingLoadStore:
+    return "FaultingLoadStore";
+  case FaultMaps::FaultingStore:
+    return "FaultingStore";
   }
 }
 
 raw_ostream &llvm::
 operator<<(raw_ostream &OS,
            const FaultMapParser::FunctionFaultInfoAccessor &FFI) {
   OS << "Fault kind: "
      << FaultMaps::faultTypeToString((FaultMaps::FaultKind)FFI.getFaultKind())

lib/CodeGen/ImplicitNullChecks.cpp

 // to
 //
 //   faulting_load_op("movl (%r10), %esi", throw_npe)
 //   ...
 //
 // With the help of a runtime that understands the .fault_maps section,
 // faulting_load_op branches to throw_npe if executing movl (%r10), %esi incurs
 // a page fault.
+// Store is also supported.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/AliasAnalysis.h"
 #include "llvm/CodeGen/Passes.h"
class ImplicitNullChecks : public MachineFunctionPass {
 
   const TargetInstrInfo *TII = nullptr;
   const TargetRegisterInfo *TRI = nullptr;
   AliasAnalysis *AA = nullptr;
   MachineModuleInfo *MMI = nullptr;
 
   bool analyzeBlockForNullChecks(MachineBasicBlock &MBB,
                                  SmallVectorImpl<NullCheck> &NullCheckList);
-  MachineInstr *insertFaultingLoad(MachineInstr *LoadMI, MachineBasicBlock *MBB,
+  MachineInstr *insertFaultingInstr(MachineInstr *MI, MachineBasicBlock *MBB,
                                    MachineBasicBlock *HandlerMBB);
   void rewriteNullChecks(ArrayRef<NullCheck> NullCheckList);
 
   enum SuitabilityResult { SR_Suitable, SR_Unsuitable, SR_Impossible };
 
   /// Return SR_Suitable if \p MI a memory operation that can be used to
   /// implicitly null check the value in \p PointerReg, SR_Unsuitable if
   /// \p MI cannot be used to null check and SR_Impossible if there is
   /// no sense to continue lookup due to any other instruction will not be able
   /// to be used. \p PrevInsts is the set of instruction seen since
-  /// the explicit null check on \p PointerReg.
+  /// the explicit null check on \p PointerReg. \p SeenLoad means that load
+  /// instruction has been observed in \PrevInsts set.
   SuitabilityResult isSuitableMemoryOp(MachineInstr &MI, unsigned PointerReg,
-                                       ArrayRef<MachineInstr *> PrevInsts);
+                                       ArrayRef<MachineInstr *> PrevInsts,
+                                       bool &SeenLoad);
 
   /// Return true if \p FaultingMI can be hoisted from after the the
   /// instructions in \p InstsSeenSoFar to before them.  Set \p Dependence to a
   /// non-null value if we also need to (and legally can) hoist a depedency.
-  bool canHoistLoadInst(MachineInstr *FaultingMI, unsigned PointerReg,
+  bool canHoistInst(MachineInstr *FaultingMI, unsigned PointerReg,
                         ArrayRef<MachineInstr *> InstsSeenSoFar,
                         MachineBasicBlock *NullSucc, MachineInstr *&Dependence);
 
 public:
   static char ID;
 
   ImplicitNullChecks() : MachineFunctionPass(ID) {
     initializeImplicitNullChecksPass(*PassRegistry::getPassRegistry());
MachineFunctionProperties getRequiredProperties() const override {
     return MachineFunctionProperties().set(
         MachineFunctionProperties::Property::NoVRegs);
   }
 };
 
 }
 
 bool ImplicitNullChecks::canHandle(const MachineInstr *MI) {
-  if (MI->isCall() || MI->mayStore() || MI->hasUnmodeledSideEffects())
+  if (MI->isCall() || MI->hasUnmodeledSideEffects())
     return false;
   auto IsRegMask = [](const MachineOperand &MO) { return MO.isRegMask(); };
   (void)IsRegMask;
 
   assert(!llvm::any_of(MI->operands(), IsRegMask) &&
          "Calls were filtered out above!");
 
   auto IsUnordered = [](MachineMemOperand *MMO) { return MMO->isUnordered(); };
for (MCRegAliasIterator AR(Reg, TRI, /*IncludeSelf*/ true); AR.isValid();
        ++AR)
     if (MBB->isLiveIn(*AR))
       return true;
   return false;
 }
 
 ImplicitNullChecks::SuitabilityResult
 ImplicitNullChecks::isSuitableMemoryOp(MachineInstr &MI, unsigned PointerReg,
-                                       ArrayRef<MachineInstr *> PrevInsts) {
+                                       ArrayRef<MachineInstr *> PrevInsts,
+                                       bool &SeenLoad) {
   int64_t Offset;
   unsigned BaseReg;
 
+  // First, if it is a store and we saw load before we bail out
+  // because we will no be able to re-order load-store without
+  // usage of memory aliasing analysis.
+  if (SeenLoad && MI.mayStore())
+    return SR_Impossible;
+
+  SeenLoad = SeenLoad || MI.mayLoad();
+
+  // Without Memory Aliasing Analysis we cannot re-order store with anything.
+  // so if this instruction is not a candidate we should stop.
+  SuitabilityResult cont = MI.mayStore() ? SR_Impossible : SR_Unsuitable;
+
   if (!TII->getMemOpBaseRegImmOfs(MI, BaseReg, Offset, TRI) ||
       BaseReg != PointerReg)
-    return SR_Unsuitable;
+    return cont;
 
-  // We want the load to be issued at a sane offset from PointerReg, so that
-  // if PointerReg is null then the load reliably page faults.
-  if (!(MI.mayLoad() && !MI.isPredicable() && Offset < PageSize))
-    return SR_Unsuitable;
-
-  // Finally, we need to make sure that the load instruction actually is
-  // loading from PointerReg, and there isn't some re-definition of PointerReg
-  // between the compare and the load.
+  // We want the mem access to be issued at a sane offset from PointerReg,
+  // so that if PointerReg is null then the access reliably page faults.
+  if (!((MI.mayLoad() || MI.mayStore()) &&
+        !MI.isPredicable() && Offset < PageSize))
+    return cont;
+
+  // Finally, we need to make sure that the access instruction actually is
+  // accessing from PointerReg, and there isn't some re-definition of PointerReg
+  // between the compare and the memory access.
   // If PointerReg has been redefined before then there is no sense to continue
   // lookup due to this condition will fail for any further instruction.
   for (auto *PrevMI : PrevInsts)
     for (auto &PrevMO : PrevMI->operands())
       if (PrevMO.isReg() && PrevMO.getReg() && PrevMO.isDef() &&
           TRI->regsOverlap(PrevMO.getReg(), PointerReg))
         return SR_Impossible;
 
   return SR_Suitable;
 }
 
-bool ImplicitNullChecks::canHoistLoadInst(
+bool ImplicitNullChecks::canHoistInst(
     MachineInstr *FaultingMI, unsigned PointerReg,
     ArrayRef<MachineInstr *> InstsSeenSoFar, MachineBasicBlock *NullSucc,
     MachineInstr *&Dependence) {
   auto DepResult = computeDependence(FaultingMI, InstsSeenSoFar);
   if (!DepResult.CanReorder)
     return false;
 
   if (!DepResult.PotentialDependence) {
bool ImplicitNullChecks::analyzeBlockForNullChecks(
   // ptr -- clause (2) above does not apply in this case.  In the above program
   // the safety of ptr->field can be dependent on some_cond; and, for instance,
   // ptr could be some non-null invalid reference that never gets loaded from
   // because some_cond is always true.
 
   const unsigned PointerReg = MBP.LHS.getReg();
 
   SmallVector<MachineInstr *, 8> InstsSeenSoFar;
+  bool SeenLoad = false;
 
   for (auto &MI : *NotNullSucc) {
     if (!canHandle(&MI) || InstsSeenSoFar.size() >= MaxInstsToConsider)
       return false;
 
     MachineInstr *Dependence;
-    SuitabilityResult SR = isSuitableMemoryOp(MI, PointerReg, InstsSeenSoFar);
+    SuitabilityResult SR = isSuitableMemoryOp(MI, PointerReg,
+                                              InstsSeenSoFar, SeenLoad);
     if (SR == SR_Impossible)
       return false;
-    if (SR == SR_Suitable && canHoistLoadInst(&MI, PointerReg, InstsSeenSoFar,
+    if (SR == SR_Suitable && canHoistInst(&MI, PointerReg, InstsSeenSoFar,
                                               NullSucc, Dependence)) {
       NullCheckList.emplace_back(&MI, MBP.ConditionDef, &MBB, NotNullSucc,
                                  NullSucc, Dependence);
       return true;
     }
 
     InstsSeenSoFar.push_back(&MI);
   }
 
   return false;
 }
 
-/// Wrap a machine load instruction, LoadMI, into a FAULTING_LOAD_OP machine
-/// instruction.  The FAULTING_LOAD_OP instruction does the same load as LoadMI
-/// (defining the same register), and branches to HandlerMBB if the load
-/// faults.  The FAULTING_LOAD_OP instruction is inserted at the end of MBB.
+/// Wrap a machine instruction, MI, into a FAULTING machine instruction.
+/// The FAULTING instruction does the same load/store as MI
+/// (defining the same register), and branches to HandlerMBB if the mem access
+/// faults.  The FAULTING instruction is inserted at the end of MBB.
 MachineInstr *
-ImplicitNullChecks::insertFaultingLoad(MachineInstr *LoadMI,
+ImplicitNullChecks::insertFaultingInstr(MachineInstr *MI,
                                        MachineBasicBlock *MBB,
                                        MachineBasicBlock *HandlerMBB) {
   const unsigned NoRegister = 0; // Guaranteed to be the NoRegister value for
                                  // all targets.
 
   DebugLoc DL;
-  unsigned NumDefs = LoadMI->getDesc().getNumDefs();
+  unsigned NumDefs = MI->getDesc().getNumDefs();
   assert(NumDefs <= 1 && "other cases unhandled!");
 
   unsigned DefReg = NoRegister;
   if (NumDefs != 0) {
-    DefReg = LoadMI->defs().begin()->getReg();
-    assert(std::distance(LoadMI->defs().begin(), LoadMI->defs().end()) == 1 &&
+    DefReg = MI->defs().begin()->getReg();
+    assert(std::distance(MI->defs().begin(), MI->defs().end()) == 1 &&
            "expected exactly one def!");
   }
 
-  auto MIB = BuildMI(MBB, DL, TII->get(TargetOpcode::FAULTING_LOAD_OP), DefReg)
+  unsigned Opcode;
+  if (MI->mayLoad())
+    Opcode = MI->mayStore() ?
+             TargetOpcode::FAULTING_LOAD_STORE_OP :
+             TargetOpcode::FAULTING_LOAD_OP;
+  else
+    Opcode = TargetOpcode::FAULTING_STORE_OP;
+
+  auto MIB = BuildMI(MBB, DL, TII->get(Opcode), DefReg)
                  .addMBB(HandlerMBB)
-                 .addImm(LoadMI->getOpcode());
+                 .addImm(MI->getOpcode());
 
-  for (auto &MO : LoadMI->uses())
+  for (auto &MO : MI->uses())
     MIB.add(MO);
 
-  MIB.setMemRefs(LoadMI->memoperands_begin(), LoadMI->memoperands_end());
+  MIB.setMemRefs(MI->memoperands_begin(), MI->memoperands_end());
 
   return MIB;
 }
 
 /// Rewrite the null checks in NullCheckList into implicit null checks.
 void ImplicitNullChecks::rewriteNullChecks(
     ArrayRef<ImplicitNullChecks::NullCheck> NullCheckList) {
   DebugLoc DL;
 
   for (auto &NC : NullCheckList) {
     // Remove the conditional branch dependent on the null check.
     unsigned BranchesRemoved = TII->removeBranch(*NC.getCheckBlock());
     (void)BranchesRemoved;
     assert(BranchesRemoved > 0 && "expected at least one branch!");
 
     if (auto *DepMI = NC.getOnlyDependency()) {
       DepMI->removeFromParent();
       NC.getCheckBlock()->insert(NC.getCheckBlock()->end(), DepMI);
     }
 
-    // Insert a faulting load where the conditional branch was originally.  We
+    // Insert a faulting instr where the conditional branch was originally. We
     // check earlier ensures that this bit of code motion is legal.  We do not
     // touch the successors list for any basic block since we haven't changed
     // control flow, we've just made it implicit.
-    MachineInstr *FaultingLoad = insertFaultingLoad(
+    MachineInstr *FaultingInstr = insertFaultingInstr(
         NC.getMemOperation(), NC.getCheckBlock(), NC.getNullSucc());
     // Now the values defined by MemOperation, if any, are live-in of
     // the block of MemOperation.
-    // The original load operation may define implicit-defs alongside
-    // the loaded value.
+    // The original operation may define implicit-defs alongside
+    // the value.
     MachineBasicBlock *MBB = NC.getMemOperation()->getParent();
-    for (const MachineOperand &MO : FaultingLoad->operands()) {
+    for (const MachineOperand &MO : FaultingInstr->operands()) {
       if (!MO.isReg() || !MO.isDef())
         continue;
       unsigned Reg = MO.getReg();
       if (!Reg || MBB->isLiveIn(Reg))
         continue;
       MBB->addLiveIn(Reg);
     }
 

lib/Target/X86/X86AsmPrinter.h

class LLVM_LIBRARY_VISIBILITY X86AsmPrinter : public AsmPrinter {
   // This helper function invokes the SMShadowTracker on each instruction before
   // outputting it to the OutStream. This allows the shadow tracker to minimise
   // the number of NOPs used for stackmap padding.
   void EmitAndCountInstruction(MCInst &Inst);
   void LowerSTACKMAP(const MachineInstr &MI);
   void LowerPATCHPOINT(const MachineInstr &MI, X86MCInstLower &MCIL);
   void LowerSTATEPOINT(const MachineInstr &MI, X86MCInstLower &MCIL);
   void LowerFAULTING_LOAD_OP(const MachineInstr &MI, X86MCInstLower &MCIL);
+  void LowerFAULTING_LOAD_STORE_OP(const MachineInstr &MI, X86MCInstLower &MCIL);
+  void LowerFAULTING_STORE_OP(const MachineInstr &MI, X86MCInstLower &MCIL);
   void LowerPATCHABLE_OP(const MachineInstr &MI, X86MCInstLower &MCIL);
 
   void LowerTlsAddr(X86MCInstLower &MCInstLowering, const MachineInstr &MI);
 
   // XRay-specific lowering for X86.
   void LowerPATCHABLE_FUNCTION_ENTER(const MachineInstr &MI,
                                      X86MCInstLower &MCIL);
   void LowerPATCHABLE_RET(const MachineInstr &MI, X86MCInstLower &MCIL);

lib/Target/X86/X86MCInstLower.cpp

void X86AsmPrinter::LowerFAULTING_LOAD_OP(const MachineInstr &MI,
 
   MCInst LoadMI;
   LoadMI.setOpcode(LoadOpcode);
 
   if (LoadDefRegister != X86::NoRegister)
     LoadMI.addOperand(MCOperand::createReg(LoadDefRegister));
 
   for (auto I = MI.operands_begin() + LoadOperandsBeginIdx,
             E = MI.operands_end();
        I != E; ++I)
     if (auto MaybeOperand = MCIL.LowerMachineOperand(&MI, *I))
       LoadMI.addOperand(MaybeOperand.getValue());
 
   OutStreamer->EmitInstruction(LoadMI, getSubtargetInfo());
 }
 
 void X86AsmPrinter::LowerFENTRY_CALL(const MachineInstr &MI,
                                      X86MCInstLower &MCIL) {
   bool Is64Bits = Subtarget->is64Bit();
   MCContext &Ctx = OutStreamer->getContext();
   MCSymbol *fentry = Ctx.getOrCreateSymbol("__fentry__");
   const MCSymbolRefExpr *Op =
       MCSymbolRefExpr::create(fentry, MCSymbolRefExpr::VK_None, Ctx);
 
   EmitAndCountInstruction(
       MCInstBuilder(Is64Bits ? X86::CALL64pcrel32 : X86::CALLpcrel32)
           .addExpr(Op));
 }
 
+void X86AsmPrinter::LowerFAULTING_LOAD_STORE_OP(const MachineInstr &MI,
+                                       X86MCInstLower &MCIL) {
+  // FAULTING_LOAD_STORE_OP <def>, <MBB handler>, <load opcode>, <load operands>
+
+  unsigned LoadStoreDefRegister = MI.getOperand(0).getReg();
+  MCSymbol *HandlerLabel = MI.getOperand(1).getMBB()->getSymbol();
+  unsigned LoadStoreOpcode = MI.getOperand(2).getImm();
+  unsigned LoadStoreOperandsBeginIdx = 3;
+
+  FM.recordFaultingOp(FaultMaps::FaultingLoadStore, HandlerLabel);
+
+  MCInst LoadStoreMI;
+  LoadStoreMI.setOpcode(LoadStoreOpcode);
+
+  if (LoadStoreDefRegister != X86::NoRegister)
+    LoadStoreMI.addOperand(MCOperand::createReg(LoadStoreDefRegister));
+
+  for (auto I = MI.operands_begin() + LoadStoreOperandsBeginIdx,
+            E = MI.operands_end();
+       I != E; ++I)
+    if (auto MaybeOperand = MCIL.LowerMachineOperand(&MI, *I))
+      LoadStoreMI.addOperand(MaybeOperand.getValue());
+
+  OutStreamer->EmitInstruction(LoadStoreMI, getSubtargetInfo());
+}
+
+void X86AsmPrinter::LowerFAULTING_STORE_OP(const MachineInstr &MI,
+                                       X86MCInstLower &MCIL) {
+  // FAULTING_STORE_OP <def>, <MBB handler>, <store opcode>, <store operands>
+
+  unsigned StoreDefRegister = MI.getOperand(0).getReg();
+  MCSymbol *HandlerLabel = MI.getOperand(1).getMBB()->getSymbol();
+  unsigned StoreOpcode = MI.getOperand(2).getImm();
+  unsigned StoreOperandsBeginIdx = 3;
+
+  FM.recordFaultingOp(FaultMaps::FaultingStore, HandlerLabel);
+
+  MCInst StoreMI;
+  StoreMI.setOpcode(StoreOpcode);
+
+  if (StoreDefRegister != X86::NoRegister)
+    StoreMI.addOperand(MCOperand::createReg(StoreDefRegister));
+
+  for (auto I = MI.operands_begin() + StoreOperandsBeginIdx,
+            E = MI.operands_end();
+       I != E; ++I)
+    if (auto MaybeOperand = MCIL.LowerMachineOperand(&MI, *I))
+      StoreMI.addOperand(MaybeOperand.getValue());
+
+  OutStreamer->EmitInstruction(StoreMI, getSubtargetInfo());
+}
+
 void X86AsmPrinter::LowerPATCHABLE_OP(const MachineInstr &MI,
                                       X86MCInstLower &MCIL) {
   // PATCHABLE_OP minsize, opcode, operands
 
   unsigned MinSize = MI.getOperand(0).getImm();
   unsigned Opcode = MI.getOperand(1).getImm();
 
   MCInst MCI;
case TargetOpcode::STATEPOINT:
     return LowerSTATEPOINT(*MI, MCInstLowering);
 
   case TargetOpcode::FAULTING_LOAD_OP:
     return LowerFAULTING_LOAD_OP(*MI, MCInstLowering);
 
   case TargetOpcode::FENTRY_CALL:
     return LowerFENTRY_CALL(*MI, MCInstLowering);
 
+  case TargetOpcode::FAULTING_LOAD_STORE_OP:
+    return LowerFAULTING_LOAD_STORE_OP(*MI, MCInstLowering);
+
+  case TargetOpcode::FAULTING_STORE_OP:
+    return LowerFAULTING_STORE_OP(*MI, MCInstLowering);
+
   case TargetOpcode::PATCHABLE_OP:
     return LowerPATCHABLE_OP(*MI, MCInstLowering);
 
   case TargetOpcode::STACKMAP:
     return LowerSTACKMAP(*MI);
 
   case TargetOpcode::PATCHPOINT:
     return LowerPATCHPOINT(*MI, MCInstLowering);

test/CodeGen/X86/implicit-null-checks.mir

--- |
 
   not_null:
     ret i32 undef
 
   is_null:
     ret i32 0
   }
 
-  attributes #0 = { "target-features"="+bmi,+bmi2" }
-
   define i32 @imp_null_check_gep_load_with_use_dep(i32* %x, i32 %a) {
   entry:
     %c = icmp eq i32* %x, null
     br i1 %c, label %is_null, label %not_null, !make.implicit !0
   
   is_null:                                          ; preds = %entry
     ret i32 42
   
--- |
   
   is_null:                                          ; preds = %entry
     ret i32 42
   
   not_null:                                         ; preds = %entry
     ret i32 undef
   }
 
+  define void @inc_store(i32* %ptr, i32 %val) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    ret void
+
+  is_null:
+    ret void
+  }
+
+  define void @inc_store_plus_offset(i32* %ptr, i32 %val) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    ret void
+
+  is_null:
+    ret void
+  }
+
+  define void @inc_store_with_dep(i32* %ptr, i32 %val) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    ret void
+
+  is_null:
+    ret void
+  }
+
+  define i32 @inc_store_with_dep_in_null(i32* %ptr, i32 %val) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    ret i32 undef
+
+  is_null:
+    ret i32 undef
+  }
+
+  define void @inc_store_with_volatile(i32* %ptr, i32 %val) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    ret void
+
+  is_null:
+    ret void
+  }
+
+  define void @inc_store_with_two_dep(i32* %ptr, i32 %val) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    ret void
+
+  is_null:
+    ret void
+  }
+
+  define void @inc_store_with_redefined_base(i32* %ptr, i32 %val) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    ret void
+
+  is_null:
+    ret void
+  }
+
+  define i32 @inc_store_with_reused_base(i32* %ptr, i32 %val) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    ret i32 undef
+
+  is_null:
+    ret i32 undef
+  }
+
+  define i32 @inc_store_across_call(i32* %ptr) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    call void @f()
+    ret i32 undef
+
+  is_null:
+    ret i32 undef
+  }
+
+  define i32 @inc_store_with_dep_in_dep(i32* %ptr, i32 %val) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    ret i32 undef
+
+  is_null:
+    ret i32 undef
+  }
+
+  define i32 @inc_store_with_load_over_store(i32* %ptr, i32* %ptr2) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    ret i32 undef
+
+  is_null:
+    ret i32 undef
+  }
+
+  define i32 @inc_store_with_store_over_load(i32* %ptr, i32* %ptr2) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    ret i32 undef
+
+  is_null:
+    ret i32 undef
+  }
+
+  define void @inc_store_with_store_over_store(i32* %ptr, i32* %ptr2) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    ret void
+
+  is_null:
+    ret void
+  }
+
+  define void @inc_store_with_load_and_store(i32* %ptr, i32* %ptr2) {
+  entry:
+    %ptr_is_null = icmp eq i32* %ptr, null
+    br i1 %ptr_is_null, label %is_null, label %not_null, !make.implicit !0
+
+  not_null:
+    ret void
+
+  is_null:
+    ret void
+  }
+
+  attributes #0 = { "target-features"="+bmi,+bmi2" }
+
   !0 = !{}
 ...
 ---
 name:            imp_null_check_with_bitwise_op_0
 # CHECK-LABEL: name:            imp_null_check_with_bitwise_op_0
 alignment:       4
 tracksRegLiveness: true
 liveins:
body:             |
 
   bb.2.is_null:
     %eax = XOR32rr undef %eax, undef %eax, implicit-def dead %eflags
     RETQ %eax
 
 ...
 ---
 name:            use_alternate_load_op
-# CHECK-LABEL: use_alternate_load_op
+# CHECK-LABEL: name:            use_alternate_load_op
 # CHECK: bb.0.entry:
 # CHECK: %r10 = FAULTING_LOAD_OP %bb.2.is_null, {{[0-9]+}}, killed %rdi, 1, _, 0, _
 # CHECK-NEXT: JMP_1 %bb.1.not_null
 # CHECK: bb.1.not_null
 
 alignment:       4
 tracksRegLiveness: true
 liveins:
body:             |
 
   bb.2.is_null:
     %eax = XOR32rr undef %eax, undef %eax, implicit-def dead %eflags
     RETQ %eax
 
 ...
 ---
 name:            imp_null_check_gep_load_with_use_dep
+# CHECK-LABEL: name:            imp_null_check_gep_load_with_use_dep
 # CHECK:  bb.0.entry:
 # CHECK:    %eax = FAULTING_LOAD_OP %bb.2.is_null, {{[0-9]+}}, killed %rdi, 1, _, 0, _, implicit-def %rax :: (load 4 from %ir.x)
 # CHECK-NEXT:    JMP_1 %bb.1.not_null
 alignment:       4
 tracksRegLiveness: true
 liveins:         
   - { reg: '%rdi' }
   - { reg: '%rsi' }
body:             |
   
   bb.1.is_null:
     %eax = MOV32ri 42
     RETQ %eax
 
 ...
 ---
 name:            imp_null_check_load_with_base_sep
+# CHECK-LABEL: name:            imp_null_check_load_with_base_sep
 # CHECK:  bb.0.entry:
 # CHECK:     %rsi = ADD64rr %rsi, %rdi, implicit-def dead %eflags
 # CHECK-NEXT:    %esi = FAULTING_LOAD_OP %bb.2.is_null, {{[0-9]+}}, killed %esi, %rdi, 1, _, 0, _, implicit-def dead %eflags
 # CHECK-NEXT:    JMP_1 %bb.1.not_null
 alignment:       4
 tracksRegLiveness: true
 liveins:         
   - { reg: '%rdi' }
bb.2.not_null:
     %eax = MOV32rr %esi
     RETQ %eax
   
   bb.1.is_null:
     %eax = MOV32ri 42
     RETQ %eax
 
 ...
+---
+name:            inc_store
+# CHECK-LABEL: name:            inc_store
+# CHECK: bb.0.entry:
+# CHECK:  _ = FAULTING_STORE_OP %bb.2.is_null, {{[0-9]+}}, killed %rdi, 1, _, 0, _, killed %rsi
+# CHECK-NEXT: JMP_1 %bb.1.not_null
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+  - { reg: '%rsi' }
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rsi
+
+    TEST64rr %rdi, %rdi, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rdi, %rsi
+
+    MOV64mr killed %rdi, 1, _, 0, _, killed %rsi
+    RETQ
+
+  bb.2.is_null:
+    RETQ
+
+...
+---
+name:            inc_store_plus_offset
+# CHECK-LABEL: inc_store_plus_offset
+# CHECK: bb.0.entry:
+# CHECK:  _ = FAULTING_STORE_OP %bb.2.is_null, {{[0-9]+}}, killed %rdi, 1, _, 16, _, killed %rsi
+# CHECK-NEXT: JMP_1 %bb.1.not_null
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+  - { reg: '%rsi' }
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rsi
+
+    TEST64rr %rdi, %rdi, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rdi, %rsi
+
+    MOV64mr killed %rdi, 1, _, 16, _, killed %rsi
+    RETQ
+
+  bb.2.is_null:
+    RETQ
+
+...
+---
+name:            inc_store_with_dep
+# CHECK-LABEL: inc_store_with_dep
+# CHECK: bb.0.entry:
+# CHECK:  %esi = ADD32rr killed %esi, killed %esi, implicit-def dead %eflags
+# CHECK-NEXT:  _ = FAULTING_STORE_OP %bb.2.is_null, {{[0-9]+}}, killed %rdi, 1, _, 16, _, killed %esi
+# CHECK-NEXT: JMP_1 %bb.1.not_null
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+  - { reg: '%rsi' }
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rsi
+
+    TEST64rr %rdi, %rdi, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rdi, %rsi
+
+    %esi = ADD32rr killed %esi, killed %esi, implicit-def dead %eflags
+    MOV32mr killed %rdi, 1, _, 16, _, killed %esi
+    RETQ
+
+  bb.2.is_null:
+    RETQ
+
+...
+---
+name:            inc_store_with_dep_in_null
+# CHECK-LABEL: inc_store_with_dep_in_null
+# CHECK: bb.0.entry:
+# CHECK:    TEST64rr %rdi, %rdi, implicit-def %eflags
+# CHECK-NEXT:    JE_1 %bb.2.is_null, implicit killed %eflags
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+  - { reg: '%rsi' }
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rsi
+
+    TEST64rr %rdi, %rdi, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rdi, %rsi
+
+    %esi = ADD32rr %esi, %esi, implicit-def dead %eflags
+    MOV32mr killed %rdi, 1, _, 0, _, %esi
+    %eax = MOV32rr killed %esi
+    RETQ %eax
+
+  bb.2.is_null:
+    liveins: %rsi
+    
+    %eax = MOV32rr killed %esi
+    RETQ %eax
+
+...
+---
+name:            inc_store_with_volatile
+# CHECK-LABEL: inc_store_with_volatile
+# CHECK: bb.0.entry:
+# CHECK:    TEST64rr %rdi, %rdi, implicit-def %eflags
+# CHECK-NEXT:    JE_1 %bb.2.is_null, implicit killed %eflags
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+  - { reg: '%rsi' }
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rsi
+
+    TEST64rr %rdi, %rdi, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rdi, %rsi
+
+    MOV32mr killed %rdi, 1, _, 0, _, killed %esi :: (volatile store 4 into %ir.ptr)
+    RETQ
+
+  bb.2.is_null:
+    RETQ
+
+...
+---
+name:            inc_store_with_two_dep
+# CHECK-LABEL: inc_store_with_two_dep
+# CHECK: bb.0.entry:
+# CHECK:    TEST64rr %rdi, %rdi, implicit-def %eflags
+# CHECK-NEXT:    JE_1 %bb.2.is_null, implicit killed %eflags
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+  - { reg: '%rsi' }
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rsi
+
+    TEST64rr %rdi, %rdi, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rdi, %rsi
+
+    %esi = ADD32rr killed %esi, killed %esi, implicit-def dead %eflags
+    %esi = ADD32ri killed %esi, 15, implicit-def dead %eflags
+    MOV32mr killed %rdi, 1, _, 16, _, killed %esi
+    RETQ
+
+  bb.2.is_null:
+    RETQ
+
+...
+---
+name:            inc_store_with_redefined_base
+# CHECK-LABEL: inc_store_with_redefined_base
+# CHECK: bb.0.entry:
+# CHECK:    TEST64rr %rdi, %rdi, implicit-def %eflags
+# CHECK-NEXT:    JE_1 %bb.2.is_null, implicit killed %eflags
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+  - { reg: '%rsi' }
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rsi
+
+    TEST64rr %rdi, %rdi, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rdi, %rsi
+
+    %rdi = ADD64rr killed %rdi, killed %rdi, implicit-def dead %eflags
+    MOV32mr killed %rdi, 1, _, 16, _, killed %esi
+    RETQ
+
+  bb.2.is_null:
+    RETQ
+
+...
+---
+name:            inc_store_with_reused_base
+# CHECK-LABEL: inc_store_with_reused_base
+# CHECK: bb.0.entry:
+# CHECK:  _ = FAULTING_STORE_OP %bb.2.is_null, {{[0-9]+}}, killed %rdi, 1, _, 16, _, killed %esi
+# CHECK-NEXT: JMP_1 %bb.1.not_null
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+  - { reg: '%rsi' }
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rsi
+
+    TEST64rr %rdi, %rdi, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rdi, %rsi
+
+    %rax = MOV64rr %rdi
+    MOV32mr killed %rdi, 1, _, 16, _, killed %esi
+    RETQ %eax
+
+  bb.2.is_null:
+    %rax = XOR64rr undef %rax, undef %rax, implicit-def dead %eflags
+    RETQ %eax
+
+...
+---
+name:            inc_store_across_call
+# CHECK-LABEL: inc_store_across_call
+# CHECK: bb.0.entry:
+# CHECK:    TEST64rr %rbx, %rbx, implicit-def %eflags
+# CHECK-NEXT:    JE_1 %bb.2.is_null, implicit killed %eflags
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+calleeSavedRegisters: [ '%bh', '%bl', '%bp', '%bpl', '%bx', '%ebp', '%ebx',
+                        '%rbp', '%rbx', '%r12', '%r13', '%r14', '%r15',
+                        '%r12b', '%r13b', '%r14b', '%r15b', '%r12d', '%r13d',
+                        '%r14d', '%r15d', '%r12w', '%r13w', '%r14w', '%r15w' ]
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rbx
+
+    frame-setup PUSH64r killed %rbx, implicit-def %rsp, implicit %rsp
+    CFI_INSTRUCTION def_cfa_offset 16
+    CFI_INSTRUCTION offset %rbx, -16
+    %rbx = MOV64rr killed %rdi
+    TEST64rr %rbx, %rbx, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rbx
+
+    CALL64pcrel32 @f, csr_64, implicit %rsp, implicit-def %rsp
+    MOV32mi %rbx, 1, _, 0, _, 20
+    %rax = MOV64rr killed %rbx
+    %rbx = POP64r implicit-def %rsp, implicit %rsp
+    RETQ %eax
+
+  bb.2.is_null:
+    %eax = XOR32rr undef %eax, undef %eax, implicit-def dead %eflags
+    %rbx = POP64r implicit-def %rsp, implicit %rsp
+    RETQ %eax
+
+...
+---
+name:            inc_store_with_dep_in_dep
+# CHECK-LABEL: inc_store_with_dep_in_dep
+# CHECK: bb.0.entry:
+# CHECK:    TEST64rr %rdi, %rdi, implicit-def %eflags
+# CHECK-NEXT:    JE_1 %bb.2.is_null, implicit killed %eflags
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+  - { reg: '%rsi' }
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rsi
+
+    TEST64rr %rdi, %rdi, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rdi, %rsi
+
+    %eax = MOV32rr %esi
+    %esi = ADD32ri killed %esi, 15, implicit-def dead %eflags
+    MOV32mr killed %rdi, 1, _, 0, _, killed %esi
+    RETQ %eax
+
+  bb.2.is_null:
+    %eax = XOR32rr undef %eax, undef %eax, implicit-def dead %eflags
+    RETQ %eax
+
+...
+---
+name:            inc_store_with_load_over_store
+# CHECK-LABEL: inc_store_with_load_over_store
+# CHECK: bb.0.entry:
+# CHECK:    TEST64rr %rdi, %rdi, implicit-def %eflags
+# CHECK-NEXT:    JE_1 %bb.2.is_null, implicit killed %eflags
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+  - { reg: '%rsi' }
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rsi
+
+    TEST64rr %rdi, %rdi, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rdi, %rsi
+
+    MOV32mi killed %rsi, 1, _, 0, _, 2
+    %eax = MOV32rm killed %rdi, 1, _, 0, _ 
+    RETQ %eax
+
+  bb.2.is_null:
+    %eax = XOR32rr undef %eax, undef %eax, implicit-def dead %eflags
+    RETQ %eax
+
+...
+---
+name:            inc_store_with_store_over_load
+# CHECK-LABEL: inc_store_with_store_over_load
+# CHECK: bb.0.entry:
+# CHECK:    TEST64rr %rdi, %rdi, implicit-def %eflags
+# CHECK-NEXT:    JE_1 %bb.2.is_null, implicit killed %eflags
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+  - { reg: '%rsi' }
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rsi
+
+    TEST64rr %rdi, %rdi, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rdi, %rsi
+
+    %eax = MOV32rm killed %rsi, 1, _, 0, _ 
+    MOV32mi killed %rdi, 1, _, 0, _, 2
+    RETQ %eax
+
+  bb.2.is_null:
+    %eax = XOR32rr undef %eax, undef %eax, implicit-def dead %eflags
+    RETQ %eax
+
+...
+---
+name:            inc_store_with_store_over_store
+# CHECK-LABEL: inc_store_with_store_over_store
+# CHECK: bb.0.entry:
+# CHECK:    TEST64rr %rdi, %rdi, implicit-def %eflags
+# CHECK-NEXT:    JE_1 %bb.2.is_null, implicit killed %eflags
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+  - { reg: '%rsi' }
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rsi
+
+    TEST64rr %rdi, %rdi, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rdi, %rsi
+
+    MOV32mi killed %rsi, 1, _, 0, _, 3 
+    MOV32mi killed %rdi, 1, _, 0, _, 2
+    RETQ
+
+  bb.2.is_null:
+    RETQ
+
+...
+---
+name:            inc_store_with_load_and_store
+# CHECK-LABEL: inc_store_with_load_and_store
+# CHECK: bb.0.entry:
+# CHECK:  _ = FAULTING_LOAD_STORE_OP %bb.2.is_null, {{[0-9]+}}, killed %rdi, 1, _, 0, _, killed %esi, implicit-def dead %eflags
+# CHECK-NEXT: JMP_1 %bb.1.not_null
+# CHECK: bb.1.not_null
+
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '%rdi' }
+  - { reg: '%rsi' }
+body:             |
+  bb.0.entry:
+    successors: %bb.2.is_null, %bb.1.not_null
+    liveins: %rdi, %rsi
+
+    TEST64rr %rdi, %rdi, implicit-def %eflags
+    JE_1 %bb.2.is_null, implicit killed %eflags
+
+  bb.1.not_null:
+    liveins: %rdi, %rsi
+
+    %esi = ADD32rr %esi, %esi, implicit-def dead %eflags
+    ADD32mr killed %rdi, 1, _, 0, _, killed %esi, implicit-def dead %eflags
+    RETQ
+
+  bb.2.is_null:
+    RETQ
+
+...