Introduce infrastructure for an incremental port of SelectionDAG atomic load/store handling

[reames]

Summary

This is the first patch in a large sequence. The eventual goal is to have unordered atomic loads and stores - and possibly ordered atomics as well - handled through the normal ISEL codepaths for loads and stores. Today, there handled w/instances of AtomicSDNodes. The result of which is that all transforms need to be duplicated to work for unordered atomics. The benefit of the current design is that it's harder to introduce a silent miscompile by adding an transform which forgets about atomicity.

Note that this patch is NFC unless the experimental flag is set.

The basic strategy I plan on taking is:

1. introduce infrastructure and a flag for testing (this patch)
2. Audit uses of isVolatile, and apply isAtomic conservatively*
3. piecemeal conservative* update generic code and x86 backedge code in individual reviews w/tests for cases which didn't check volatile, but can be found with inspection
4. flip the flag at the end (with minimal diffs)
5. Work through todo list identified in (2) and (3) exposing performance ops

(*) The "conservative" bit here is aimed at minimizing the number of diffs involved in (4). Ideally, there'd be none. In practice, getting it down to something reviewable by a human is the actual goal. Note that there are (currently) no paths which produce LoadSDNode or StoreSDNode with atomic MMOs, so we don't need to worry about preserving any behaviour there.

We've taken a very similar strategy twice before with success - once at IR level, and once at the MI level (post ISEL). I'll probably need some help with some of the ISEL patterns since that's the part I'm not familiar with.

Next two patches for (2) above are posted as: D66318 (generic CodeGen) and D66322 (X86). I'll stop there until some of these make it through review.

[arsenm]

This will help atomic store handling for global isel. Right now there's a complication because ISD::ATOMIC_STORE swaps the operand order from a normal store

[reames]

ping

[efriedma]

Is there an llvm-dev thread for the general project? It looks fine to me, but maybe it should have a wider audience.

Are you planning to move all atomic load/store operations to this framework, eventually, or just unordered ones?

Do you have any test coverage for code using the flags from an unordered load folded into arithmetic, or a folded RMW operation?

[craig.topper]

test/CodeGen/X86/atomic-unordered.ll

4 ↗

(On Diff #215458)

Can we add a common prefix to share with the two O3 run lines? Looks like our codegen is the same for the majority of the test cases.

[reames]

In D66309#1643783, @efriedma wrote:

Is there an llvm-dev thread for the general project? It looks fine to me, but maybe it should have a wider audience.

Will do. Sent as "FYI: proposed changes to atomic load/store in SelectionDAG" on Aug 28th.

Are you planning to move all atomic load/store operations to this framework, eventually, or just unordered ones?

Honestly, haven't decided. My near term needs are for unordered only.

Do you have any test coverage for code using the flags from an unordered load folded into arithmetic, or a folded RMW operation?

See test/CodeGen/X86/atomic-unordered.ll @rmw_fold_* for the later.
Will add the flag checks, I don't have those (I think).

[reames]

test/CodeGen/X86/atomic-unordered.ll

4 ↗

(On Diff #215458)

Not quite following the why behind this request? Is there some (undocumented) feature of FileCheck which merges the check lines or something?

[xbolva00]

test/CodeGen/X86/atomic-unordered.ll

4 ↗

(On Diff #215458)

Possibly something like
-check-prefixes=CHECK,CHECK-O3
-check-prefixes=CHECK,CHECK-EX

(not tested, just saw it many times)

[craig.topper]

test/CodeGen/X86/atomic-unordered.ll

4 ↗

(On Diff #215458)

Yeah that's it. The test update script understands this. FileCheck and the update script also understand "--check-prefix=CHECK --check-prefix=CHECK-O3" on the same command line as meaning the same thing. FileCheck will check any lines with the listed prefix. For the script to work, the common prefix should be placed first and more specific prefix placed last. Then it will favor the common prefix when possible.

[reames]

test/CodeGen/X86/atomic-unordered.ll

4 ↗

(On Diff #215458)

Did not know about this, cool.

[reames]

Ping. I'd like to get this landed if possible.

[craig.topper]

test/CodeGen/X86/atomic-unordered.ll

4 ↗

(On Diff #215458)

This test didn't get rebased and updated to merge the CHECK prefixes

[reames]

Rebase tests...

[RKSimon]

include/llvm/CodeGen/TargetLowering.h

3719 ↗

(On Diff #218788)

missing assertion message

3728 ↗

(On Diff #218788)

missing assertion message

[reames]

include/llvm/CodeGen/TargetLowering.h

3719 ↗

(On Diff #218788)

Happy to add something, but given the naming of the functions, I don't really think (... && "violated precondition") adds anything valuable.

This also feels like a very small nit. Is there any chance I can get an LGTM and just fix this before the commit? Refreshing this is a real pain since I have to rebuild the world.

[craig.topper]

LGTM

[RKSimon]

LGTM, sorry for sidetracking the patch

llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h

 class LSBaseSDNode : public MemSDNode {
 public:
   LSBaseSDNode(ISD::NodeType NodeTy, unsigned Order, const DebugLoc &dl,
                SDVTList VTs, ISD::MemIndexedMode AM, EVT MemVT,
                MachineMemOperand *MMO)
       : MemSDNode(NodeTy, Order, dl, VTs, MemVT, MMO) {
     LSBaseSDNodeBits.AddressingMode = AM;
     assert(getAddressingMode() == AM && "Value truncated");
-    assert((!MMO->isAtomic() || MMO->isVolatile()) &&
-           "use an AtomicSDNode instead for non-volatile atomics");
   }
 
   const SDValue &getOffset() const {
     return getOperand(getOpcode() == ISD::LOAD ? 2 : 3);
   }
 
   /// Return the addressing mode for this load or store:
   /// unindexed, pre-inc, pre-dec, post-inc, or post-dec.

llvm/trunk/include/llvm/CodeGen/TargetLowering.h

public:
 
   /// This callback is used to inspect load/store instructions and add
   /// target-specific MachineMemOperand flags to them.  The default
   /// implementation does nothing.
   virtual MachineMemOperand::Flags getMMOFlags(const Instruction &I) const {
     return MachineMemOperand::MONone;
   }
 
+  /// Should SelectionDAG lower an atomic store of the given kind as a normal
+  /// StoreSDNode (as opposed to an AtomicSDNode)?  NOTE: The intention is to
+  /// eventually migrate all targets to the using StoreSDNodes, but porting is
+  /// being done target at a time.  
+  virtual bool lowerAtomicStoreAsStoreSDNode(const StoreInst &SI) const {
+    assert(SI.isAtomic() && "violated precondition");
+    return false;
+  }
+
+  /// Should SelectionDAG lower an atomic load of the given kind as a normal
+  /// LoadSDNode (as opposed to an AtomicSDNode)?  NOTE: The intention is to
+  /// eventually migrate all targets to the using LoadSDNodes, but porting is
+  /// being done target at a time.  
+  virtual bool lowerAtomicLoadAsLoadSDNode(const LoadInst &LI) const {
+    assert(LI.isAtomic() && "violated precondition");
+    return false;
+  }
+
+
   /// This callback is invoked by the type legalizer to legalize nodes with an
   /// illegal operand type but legal result types.  It replaces the
   /// LowerOperation callback in the type Legalizer.  The reason we can not do
   /// away with LowerOperation entirely is that LegalizeDAG isn't yet ready to
   /// use this callback.
   ///
   /// TODO: Consider merging with ReplaceNodeResults.
   ///

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

MachineMemOperand *MMO =
       DAG.getMachineFunction().
       getMachineMemOperand(MachinePointerInfo(I.getPointerOperand()),
                            Flags, MemVT.getStoreSize(),
                            I.getAlignment() ? I.getAlignment() :
                                               DAG.getEVTAlignment(MemVT),
                            AAMDNodes(), nullptr, SSID, Order);
 
   InChain = TLI.prepareVolatileOrAtomicLoad(InChain, dl, DAG);
-  SDValue L =
-      DAG.getAtomic(ISD::ATOMIC_LOAD, dl, MemVT, MemVT, InChain,
-                    getValue(I.getPointerOperand()), MMO);
+
+  SDValue Ptr = getValue(I.getPointerOperand());
+
+  if (TLI.lowerAtomicLoadAsLoadSDNode(I)) {
+    // TODO: Once this is better exercised by tests, it should be merged with
+    // the normal path for loads to prevent future divergence.
+    SDValue L = DAG.getLoad(MemVT, dl, InChain, Ptr, MMO);
+    if (MemVT != VT)
+      L = DAG.getPtrExtOrTrunc(L, dl, VT);
+
+    setValue(&I, L);
+    if (!I.isUnordered()) {
+      SDValue OutChain = L.getValue(1);
+      DAG.setRoot(OutChain);
+    }
+    return;
+  }
+  
+  SDValue L = DAG.getAtomic(ISD::ATOMIC_LOAD, dl, MemVT, MemVT, InChain,
+                            Ptr, MMO);
 
   SDValue OutChain = L.getValue(1);
   if (MemVT != VT)
     L = DAG.getPtrExtOrTrunc(L, dl, VT);
 
   setValue(&I, L);
   DAG.setRoot(OutChain);
 }
void SelectionDAGBuilder::visitAtomicStore(const StoreInst &I) {
   MachineMemOperand *MMO =
     MF.getMachineMemOperand(MachinePointerInfo(I.getPointerOperand()), Flags,
                             MemVT.getStoreSize(), I.getAlignment(), AAMDNodes(),
                             nullptr, SSID, Ordering);
 
   SDValue Val = getValue(I.getValueOperand());
   if (Val.getValueType() != MemVT)
     Val = DAG.getPtrExtOrTrunc(Val, dl, MemVT);
+  SDValue Ptr = getValue(I.getPointerOperand());
 
+  if (TLI.lowerAtomicStoreAsStoreSDNode(I)) {
+    // TODO: Once this is better exercised by tests, it should be merged with
+    // the normal path for stores to prevent future divergence.
+    SDValue S = DAG.getStore(InChain, dl, Val, Ptr, MMO);
+    DAG.setRoot(S);
+    return;
+  }
   SDValue OutChain = DAG.getAtomic(ISD::ATOMIC_STORE, dl, MemVT, InChain,
-                                   getValue(I.getPointerOperand()), Val, MMO);
+                                   Ptr, Val, MMO);
 
 
   DAG.setRoot(OutChain);
 }
 
 /// visitTargetIntrinsic - Lower a call of a target intrinsic to an INTRINSIC
 /// node.
 void SelectionDAGBuilder::visitTargetIntrinsic(const CallInst &I,

llvm/trunk/lib/Target/X86/X86ISelLowering.h

private:
     shouldExpandAtomicLoadInIR(LoadInst *SI) const override;
     bool shouldExpandAtomicStoreInIR(StoreInst *SI) const override;
     TargetLoweringBase::AtomicExpansionKind
     shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
 
     LoadInst *
     lowerIdempotentRMWIntoFencedLoad(AtomicRMWInst *AI) const override;
 
+    bool lowerAtomicStoreAsStoreSDNode(const StoreInst &SI) const override;
+    bool lowerAtomicLoadAsLoadSDNode(const LoadInst &LI) const override;
+
     bool needsCmpXchgNb(Type *MemType) const;
 
     void SetupEntryBlockForSjLj(MachineInstr &MI, MachineBasicBlock *MBB,
                                 MachineBasicBlock *DispatchBB, int FI) const;
 
     // Utility function to emit the low-level va_arg code for X86-64.
     MachineBasicBlock *
     EmitVAARG64WithCustomInserter(MachineInstr &MI,

llvm/trunk/lib/Target/X86/X86ISelLowering.cpp

static cl::opt<bool> EnableOldKNLABI(
     cl::Hidden);
 
 static cl::opt<bool> MulConstantOptimization(
     "mul-constant-optimization", cl::init(true),
     cl::desc("Replace 'mul x, Const' with more effective instructions like "
              "SHIFT, LEA, etc."),
     cl::Hidden);
 
+static cl::opt<bool> ExperimentalUnorderedISEL(
+    "x86-experimental-unordered-atomic-isel", cl::init(false),
+    cl::desc("Use LoadSDNode and StoreSDNode instead of "
+             "AtomicSDNode for unordered atomic loads and "
+             "stores respectively."),
+    cl::Hidden);
+
 /// Call this when the user attempts to do something unsupported, like
 /// returning a double without SSE2 enabled on x86_64. This is not fatal, unlike
 /// report_fatal_error, so calling code should attempt to recover without
 /// crashing.
 static void errorUnsupported(SelectionDAG &DAG, const SDLoc &dl,
                              const char *Msg) {
   MachineFunction &MF = DAG.getMachineFunction();
   DAG.getContext()->diagnose(
LoadInst *Loaded =
       Builder.CreateAlignedLoad(AI->getType(), AI->getPointerOperand(),
                                 AI->getType()->getPrimitiveSizeInBits());
   Loaded->setAtomic(Order, SSID);
   AI->replaceAllUsesWith(Loaded);
   AI->eraseFromParent();
   return Loaded;
 }
 
+bool X86TargetLowering::lowerAtomicStoreAsStoreSDNode(const StoreInst &SI) const {
+  if (!SI.isUnordered())
+    return false;
+  return ExperimentalUnorderedISEL;
+}
+bool X86TargetLowering::lowerAtomicLoadAsLoadSDNode(const LoadInst &LI) const {
+  if (!LI.isUnordered())
+    return false;
+  return ExperimentalUnorderedISEL;
+}
+
+
 /// Emit a locked operation on a stack location which does not change any
 /// memory location, but does involve a lock prefix.  Location is chosen to be
 /// a) very likely accessed only by a single thread to minimize cache traffic,
 /// and b) definitely dereferenceable.  Returns the new Chain result.  
 static SDValue emitLockedStackOp(SelectionDAG &DAG,
                                  const X86Subtarget &Subtarget,
                                  SDValue Chain, SDLoc DL) {
   // Implementation notes:

llvm/trunk/test/CodeGen/X86/atomic-unordered.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -O0 < %s -mtriple=x86_64-linux-generic -verify-machineinstrs -mcpu=skylake | FileCheck --check-prefixes=CHECK,CHECK-O0 %s
-; RUN: llc -O3 < %s -mtriple=x86_64-linux-generic -verify-machineinstrs -mcpu=skylake | FileCheck --check-prefixes=CHECK,CHECK-O3 %s
+; RUN: llc -O0 < %s -mtriple=x86_64-linux-generic -verify-machineinstrs -mcpu=skylake | FileCheck --check-prefixes=CHECK,CHECK-NOX,CHECK-O0 %s
+; RUN: llc -O3 < %s -mtriple=x86_64-linux-generic -verify-machineinstrs -mcpu=skylake | FileCheck --check-prefixes=CHECK,CHECK-NOX,CHECK-O3 %s
+; RUN: llc -O3 < %s -mtriple=x86_64-linux-generic -verify-machineinstrs -mcpu=skylake -x86-experimental-unordered-atomic-isel | FileCheck --check-prefixes=CHECK,CHECK-EX %s
 
 define i8 @load_i8(i8* %ptr) {
 ; CHECK-LABEL: load_i8:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movb (%rdi), %al
 ; CHECK-NEXT:    retq
   %v = load atomic i8, i8* %ptr unordered, align 1
   ret i8 %v
 }
 
 define void @store_i8(i8* %ptr, i8 %v) {
 ; CHECK-O0-LABEL: store_i8:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    # kill: def $sil killed $sil killed $esi
 ; CHECK-O0-NEXT:    movb %sil, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: store_i8:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movb %sil, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: store_i8:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movb %sil, (%rdi)
+; CHECK-EX-NEXT:    retq
   store atomic i8 %v, i8* %ptr unordered, align 1
   ret void
 }
 
 define i16 @load_i16(i16* %ptr) {
 ; CHECK-O0-LABEL: load_i16:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movw (%rdi), %ax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_i16:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movzwl (%rdi), %eax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_i16:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movzwl (%rdi), %eax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i16, i16* %ptr unordered, align 2
   ret i16 %v
 }
 
 
 define void @store_i16(i16* %ptr, i16 %v) {
 ; CHECK-O0-LABEL: store_i16:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    # kill: def $si killed $si killed $esi
 ; CHECK-O0-NEXT:    movw %si, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: store_i16:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movw %si, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: store_i16:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movw %si, (%rdi)
+; CHECK-EX-NEXT:    retq
   store atomic i16 %v, i16* %ptr unordered, align 2
   ret void
 }
 
 define i32 @load_i32(i32* %ptr) {
 ; CHECK-LABEL: load_i32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movl (%rdi), %eax

 ; CHECK-O0-NEXT:    orq $7, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: narrow_writeback_or:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    orq $7, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: narrow_writeback_or:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    orq $7, (%rdi)
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %ptr unordered, align 8
   %v.new = or i64 %v, 7
   store atomic i64 %v.new, i64* %ptr unordered, align 8
   ret void
 }
 
 ; Must use a full width op, not a byte op
 define void @narrow_writeback_and(i64* %ptr) {
 ; CHECK-O0-LABEL: narrow_writeback_and:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O0-NEXT:    andl $-256, %eax
 ; CHECK-O0-NEXT:    movl %eax, %ecx
 ; CHECK-O0-NEXT:    movq %rcx, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: narrow_writeback_and:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movl $4294967040, %eax # imm = 0xFFFFFF00
 ; CHECK-O3-NEXT:    andq %rax, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: narrow_writeback_and:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movl $4294967040, %eax # imm = 0xFFFFFF00
+; CHECK-EX-NEXT:    andq %rax, (%rdi)
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %ptr unordered, align 8
   %v.new = and i64 %v, 4294967040 ;; 0xFFFF_FF00
   store atomic i64 %v.new, i64* %ptr unordered, align 8
   ret void
 }
 
 ; Must use a full width op, not a byte op
 define void @narrow_writeback_xor(i64* %ptr) {
 ; CHECK-O0-LABEL: narrow_writeback_xor:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    xorq $7, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: narrow_writeback_xor:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    xorq $7, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: narrow_writeback_xor:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    xorq $7, (%rdi)
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %ptr unordered, align 8
   %v.new = xor i64 %v, 7
   store atomic i64 %v.new, i64* %ptr unordered, align 8
   ret void
 }
 
 ;; Next batch of tests are exercising cases where store widening would
 ;; improve codegeneration.  Note that widening is only legal if the

 ; CHECK-O3-NEXT:    xorl %eax, %eax
 ; CHECK-O3-NEXT:    xorl %edx, %edx
 ; CHECK-O3-NEXT:    xorl %ecx, %ecx
 ; CHECK-O3-NEXT:    xorl %ebx, %ebx
 ; CHECK-O3-NEXT:    lock cmpxchg16b (%rdi)
 ; CHECK-O3-NEXT:    popq %rbx
 ; CHECK-O3-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_i128:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    pushq %rbx
+; CHECK-EX-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-EX-NEXT:    .cfi_offset %rbx, -16
+; CHECK-EX-NEXT:    xorl %eax, %eax
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    xorl %ecx, %ecx
+; CHECK-EX-NEXT:    xorl %ebx, %ebx
+; CHECK-EX-NEXT:    lock cmpxchg16b (%rdi)
+; CHECK-EX-NEXT:    popq %rbx
+; CHECK-EX-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-EX-NEXT:    retq
   %v = load atomic i128, i128* %ptr unordered, align 16
   ret i128 %v
 }
 
 define void @store_i128(i128* %ptr, i128 %v) {
 ; CHECK-O0-LABEL: store_i128:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    pushq %rbx

 ; CHECK-O3-NEXT:  .LBB16_1: # %atomicrmw.start
 ; CHECK-O3-NEXT:    # =>This Inner Loop Header: Depth=1
 ; CHECK-O3-NEXT:    lock cmpxchg16b (%rdi)
 ; CHECK-O3-NEXT:    jne .LBB16_1
 ; CHECK-O3-NEXT:  # %bb.2: # %atomicrmw.end
 ; CHECK-O3-NEXT:    popq %rbx
 ; CHECK-O3-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: store_i128:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    pushq %rbx
+; CHECK-EX-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-EX-NEXT:    .cfi_offset %rbx, -16
+; CHECK-EX-NEXT:    movq %rdx, %rcx
+; CHECK-EX-NEXT:    movq %rsi, %rbx
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq 8(%rdi), %rdx
+; CHECK-EX-NEXT:    .p2align 4, 0x90
+; CHECK-EX-NEXT:  .LBB16_1: # %atomicrmw.start
+; CHECK-EX-NEXT:    # =>This Inner Loop Header: Depth=1
+; CHECK-EX-NEXT:    lock cmpxchg16b (%rdi)
+; CHECK-EX-NEXT:    jne .LBB16_1
+; CHECK-EX-NEXT:  # %bb.2: # %atomicrmw.end
+; CHECK-EX-NEXT:    popq %rbx
+; CHECK-EX-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-EX-NEXT:    retq
   store atomic i128 %v, i128* %ptr unordered, align 16
   ret void
 }
 
 define i256 @load_i256(i256* %ptr) {
 ; CHECK-O0-LABEL: load_i256:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    subq $56, %rsp

 ; CHECK-O3-NEXT:    vmovups %ymm0, (%rbx)
 ; CHECK-O3-NEXT:    movq %rbx, %rax
 ; CHECK-O3-NEXT:    addq $32, %rsp
 ; CHECK-O3-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-O3-NEXT:    popq %rbx
 ; CHECK-O3-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-O3-NEXT:    vzeroupper
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_i256:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    pushq %rbx
+; CHECK-EX-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-EX-NEXT:    subq $32, %rsp
+; CHECK-EX-NEXT:    .cfi_def_cfa_offset 48
+; CHECK-EX-NEXT:    .cfi_offset %rbx, -16
+; CHECK-EX-NEXT:    movq %rdi, %rbx
+; CHECK-EX-NEXT:    movq %rsp, %rdx
+; CHECK-EX-NEXT:    movl $32, %edi
+; CHECK-EX-NEXT:    xorl %ecx, %ecx
+; CHECK-EX-NEXT:    callq __atomic_load
+; CHECK-EX-NEXT:    vmovups (%rsp), %ymm0
+; CHECK-EX-NEXT:    vmovups %ymm0, (%rbx)
+; CHECK-EX-NEXT:    movq %rbx, %rax
+; CHECK-EX-NEXT:    addq $32, %rsp
+; CHECK-EX-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-EX-NEXT:    popq %rbx
+; CHECK-EX-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-EX-NEXT:    vzeroupper
+; CHECK-EX-NEXT:    retq
   %v = load atomic i256, i256* %ptr unordered, align 16
   ret i256 %v
 }
 
 define void @store_i256(i256* %ptr, i256 %v) {
 ; CHECK-O0-LABEL: store_i256:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    subq $40, %rsp

 ; CHECK-O3-NEXT:    leaq {{[0-9]+}}(%rsp), %rdx
 ; CHECK-O3-NEXT:    movl $32, %edi
 ; CHECK-O3-NEXT:    movq %rax, %rsi
 ; CHECK-O3-NEXT:    xorl %ecx, %ecx
 ; CHECK-O3-NEXT:    callq __atomic_store
 ; CHECK-O3-NEXT:    addq $40, %rsp
 ; CHECK-O3-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: store_i256:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    subq $40, %rsp
+; CHECK-EX-NEXT:    .cfi_def_cfa_offset 48
+; CHECK-EX-NEXT:    movq %rdi, %rax
+; CHECK-EX-NEXT:    movq %r8, {{[0-9]+}}(%rsp)
+; CHECK-EX-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
+; CHECK-EX-NEXT:    movq %rdx, {{[0-9]+}}(%rsp)
+; CHECK-EX-NEXT:    movq %rsi, {{[0-9]+}}(%rsp)
+; CHECK-EX-NEXT:    leaq {{[0-9]+}}(%rsp), %rdx
+; CHECK-EX-NEXT:    movl $32, %edi
+; CHECK-EX-NEXT:    movq %rax, %rsi
+; CHECK-EX-NEXT:    xorl %ecx, %ecx
+; CHECK-EX-NEXT:    callq __atomic_store
+; CHECK-EX-NEXT:    addq $40, %rsp
+; CHECK-EX-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-EX-NEXT:    retq
   store atomic i256 %v, i256* %ptr unordered, align 16
   ret void
 }
 
 ; Legal if wider type is also atomic (TODO)
 define void @vec_store(i32* %p0, <2 x i32> %vec) {
-; CHECK-LABEL: vec_store:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovd %xmm0, %eax
-; CHECK-NEXT:    vpextrd $1, %xmm0, %ecx
-; CHECK-NEXT:    movl %eax, (%rdi)
-; CHECK-NEXT:    movl %ecx, 4(%rdi)
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: vec_store:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    vmovd %xmm0, %eax
+; CHECK-NOX-NEXT:    vpextrd $1, %xmm0, %ecx
+; CHECK-NOX-NEXT:    movl %eax, (%rdi)
+; CHECK-NOX-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: vec_store:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    vmovss %xmm0, (%rdi)
+; CHECK-EX-NEXT:    vextractps $1, %xmm0, 4(%rdi)
+; CHECK-EX-NEXT:    retq
   %v1 = extractelement <2 x i32> %vec, i32 0
   %v2 = extractelement <2 x i32> %vec, i32 1
   %p1 = getelementptr i32, i32* %p0, i64 1
   store atomic i32 %v1, i32* %p0 unordered, align 8
   store atomic i32 %v2, i32* %p1 unordered, align 4
   ret void
 }
 
 ; Not legal to widen due to alignment restriction
 define void @vec_store_unaligned(i32* %p0, <2 x i32> %vec) {
-; CHECK-LABEL: vec_store_unaligned:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovd %xmm0, %eax
-; CHECK-NEXT:    vpextrd $1, %xmm0, %ecx
-; CHECK-NEXT:    movl %eax, (%rdi)
-; CHECK-NEXT:    movl %ecx, 4(%rdi)
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: vec_store_unaligned:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    vmovd %xmm0, %eax
+; CHECK-NOX-NEXT:    vpextrd $1, %xmm0, %ecx
+; CHECK-NOX-NEXT:    movl %eax, (%rdi)
+; CHECK-NOX-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: vec_store_unaligned:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    vmovss %xmm0, (%rdi)
+; CHECK-EX-NEXT:    vextractps $1, %xmm0, 4(%rdi)
+; CHECK-EX-NEXT:    retq
   %v1 = extractelement <2 x i32> %vec, i32 0
   %v2 = extractelement <2 x i32> %vec, i32 1
   %p1 = getelementptr i32, i32* %p0, i64 1
   store atomic i32 %v1, i32* %p0 unordered, align 4
   store atomic i32 %v2, i32* %p1 unordered, align 4
   ret void
 }
 
 
 
 ; Legal if wider type is also atomic (TODO)
 ; Also, can avoid register move from xmm to eax (TODO)
 define void @widen_broadcast2(i32* %p0, <2 x i32> %vec) {
-; CHECK-LABEL: widen_broadcast2:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovd %xmm0, %eax
-; CHECK-NEXT:    movl %eax, (%rdi)
-; CHECK-NEXT:    movl %eax, 4(%rdi)
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: widen_broadcast2:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    vmovd %xmm0, %eax
+; CHECK-NOX-NEXT:    movl %eax, (%rdi)
+; CHECK-NOX-NEXT:    movl %eax, 4(%rdi)
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: widen_broadcast2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    vmovss %xmm0, (%rdi)
+; CHECK-EX-NEXT:    vmovss %xmm0, 4(%rdi)
+; CHECK-EX-NEXT:    retq
   %v1 = extractelement <2 x i32> %vec, i32 0
   %p1 = getelementptr i32, i32* %p0, i64 1
   store atomic i32 %v1, i32* %p0 unordered, align 8
   store atomic i32 %v1, i32* %p1 unordered, align 4
   ret void
 }
 
 ; Not legal to widen due to alignment restriction
 define void @widen_broadcast2_unaligned(i32* %p0, <2 x i32> %vec) {
-; CHECK-LABEL: widen_broadcast2_unaligned:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovd %xmm0, %eax
-; CHECK-NEXT:    movl %eax, (%rdi)
-; CHECK-NEXT:    movl %eax, 4(%rdi)
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: widen_broadcast2_unaligned:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    vmovd %xmm0, %eax
+; CHECK-NOX-NEXT:    movl %eax, (%rdi)
+; CHECK-NOX-NEXT:    movl %eax, 4(%rdi)
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: widen_broadcast2_unaligned:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    vmovss %xmm0, (%rdi)
+; CHECK-EX-NEXT:    vmovss %xmm0, 4(%rdi)
+; CHECK-EX-NEXT:    retq
   %v1 = extractelement <2 x i32> %vec, i32 0
   %p1 = getelementptr i32, i32* %p0, i64 1
   store atomic i32 %v1, i32* %p0 unordered, align 4
   store atomic i32 %v1, i32* %p1 unordered, align 4
   ret void
 }
 
 ; Legal if wider type is also atomic (TODO)
 define void @widen_zero_init(i32* %p0, i32 %v1, i32 %v2) {
-; CHECK-LABEL: widen_zero_init:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl $0, (%rdi)
-; CHECK-NEXT:    movl $0, 4(%rdi)
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: widen_zero_init:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movl $0, (%rdi)
+; CHECK-NOX-NEXT:    movl $0, 4(%rdi)
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: widen_zero_init:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq $0, (%rdi)
+; CHECK-EX-NEXT:    retq
   %p1 = getelementptr i32, i32* %p0, i64 1
   store atomic i32 0, i32* %p0 unordered, align 8
   store atomic i32 0, i32* %p1 unordered, align 4
   ret void
 }
 
 ; Not legal to widen due to alignment restriction
 define void @widen_zero_init_unaligned(i32* %p0, i32 %v1, i32 %v2) {
-; CHECK-LABEL: widen_zero_init_unaligned:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl $0, (%rdi)
-; CHECK-NEXT:    movl $0, 4(%rdi)
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: widen_zero_init_unaligned:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movl $0, (%rdi)
+; CHECK-NOX-NEXT:    movl $0, 4(%rdi)
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: widen_zero_init_unaligned:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq $0, (%rdi)
+; CHECK-EX-NEXT:    retq
   %p1 = getelementptr i32, i32* %p0, i64 1
   store atomic i32 0, i32* %p0 unordered, align 4
   store atomic i32 0, i32* %p1 unordered, align 4
   ret void
 }
 
 ;; The next batch of tests are stressing load folding. Folding is legal
 ;; on x86, so these are simply checking optimization quality.

 ; CHECK-O0-NEXT:    movq %rsi, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_add2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq %rsi, %rax
 ; CHECK-O3-NEXT:    addq (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_add2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rsi, %rax
+; CHECK-EX-NEXT:    addq (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = add i64 %v, %v2
   ret i64 %ret
 }
 
 define i64 @load_fold_add3(i64* %p1, i64* %p2) {
 ; CHECK-O0-LABEL: load_fold_add3:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    addq (%rsi), %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_add3:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rsi), %rax
 ; CHECK-O3-NEXT:    addq (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_add3:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    addq (%rsi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p1 unordered, align 8
   %v2 = load atomic i64, i64* %p2 unordered, align 8
   %ret = add i64 %v, %v2
   ret i64 %ret
 }
 
 ; Legal, as expected
 define i64 @load_fold_sub1(i64* %p) {
 ; CHECK-O0-LABEL: load_fold_sub1:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    subq $15, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_sub1:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rdi), %rax
 ; CHECK-O3-NEXT:    addq $-15, %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_sub1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    addq $-15, %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = sub i64 %v, 15
   ret i64 %ret
 }
 
 define i64 @load_fold_sub2(i64* %p, i64 %v2) {
 ; CHECK-LABEL: load_fold_sub2:
 ; CHECK:       # %bb.0:

 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_mul1:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rdi), %rax
 ; CHECK-O3-NEXT:    leaq (%rax,%rax,4), %rax
 ; CHECK-O3-NEXT:    leaq (%rax,%rax,2), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_mul1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    leaq (%rax,%rax,4), %rax
+; CHECK-EX-NEXT:    leaq (%rax,%rax,2), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = mul i64 %v, 15
   ret i64 %ret
 }
 
 define i64 @load_fold_mul2(i64* %p, i64 %v2) {
 ; CHECK-O0-LABEL: load_fold_mul2:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    imulq (%rdi), %rsi
 ; CHECK-O0-NEXT:    movq %rsi, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_mul2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq %rsi, %rax
 ; CHECK-O3-NEXT:    imulq (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_mul2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rsi, %rax
+; CHECK-EX-NEXT:    imulq (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = mul i64 %v, %v2
   ret i64 %ret
 }
 
 define i64 @load_fold_mul3(i64* %p1, i64* %p2) {
 ; CHECK-O0-LABEL: load_fold_mul3:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    imulq (%rsi), %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_mul3:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rsi), %rax
 ; CHECK-O3-NEXT:    imulq (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_mul3:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    imulq (%rsi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p1 unordered, align 8
   %v2 = load atomic i64, i64* %p2 unordered, align 8
   %ret = mul i64 %v, %v2
   ret i64 %ret
 }
 
 ; Legal to fold (TODO)
 define i64 @load_fold_sdiv1(i64* %p) {

 ; CHECK-O3-NEXT:    movq %rcx, %rax
 ; CHECK-O3-NEXT:    imulq %rdx
 ; CHECK-O3-NEXT:    addq %rcx, %rdx
 ; CHECK-O3-NEXT:    movq %rdx, %rax
 ; CHECK-O3-NEXT:    shrq $63, %rax
 ; CHECK-O3-NEXT:    sarq $3, %rdx
 ; CHECK-O3-NEXT:    addq %rdx, %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_sdiv1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rcx
+; CHECK-EX-NEXT:    movabsq $-8608480567731124087, %rdx # imm = 0x8888888888888889
+; CHECK-EX-NEXT:    movq %rcx, %rax
+; CHECK-EX-NEXT:    imulq %rdx
+; CHECK-EX-NEXT:    addq %rcx, %rdx
+; CHECK-EX-NEXT:    movq %rdx, %rax
+; CHECK-EX-NEXT:    shrq $63, %rax
+; CHECK-EX-NEXT:    sarq $3, %rdx
+; CHECK-EX-NEXT:    addq %rdx, %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = sdiv i64 %v, 15
   ret i64 %ret
 }
 
 ; Legal to fold (TODO)
 define i64 @load_fold_sdiv2(i64* %p, i64 %v2) {
 ; CHECK-O0-LABEL: load_fold_sdiv2:

 ; CHECK-O3-NEXT:    idivq %rsi
 ; CHECK-O3-NEXT:    retq
 ; CHECK-O3-NEXT:  .LBB35_1:
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    xorl %edx, %edx
 ; CHECK-O3-NEXT:    divl %esi
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax def $rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_sdiv2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq %rax, %rcx
+; CHECK-EX-NEXT:    orq %rsi, %rcx
+; CHECK-EX-NEXT:    shrq $32, %rcx
+; CHECK-EX-NEXT:    je .LBB35_1
+; CHECK-EX-NEXT:  # %bb.2:
+; CHECK-EX-NEXT:    cqto
+; CHECK-EX-NEXT:    idivq %rsi
+; CHECK-EX-NEXT:    retq
+; CHECK-EX-NEXT:  .LBB35_1:
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divl %esi
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax def $rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = sdiv i64 %v, %v2
   ret i64 %ret
 }
 
 define i64 @load_fold_sdiv3(i64* %p1, i64* %p2) {
 ; CHECK-O0-LABEL: load_fold_sdiv3:
 ; CHECK-O0:       # %bb.0:

 ; CHECK-O3-NEXT:    idivq %rcx
 ; CHECK-O3-NEXT:    retq
 ; CHECK-O3-NEXT:  .LBB36_1:
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    xorl %edx, %edx
 ; CHECK-O3-NEXT:    divl %ecx
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax def $rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_sdiv3:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq (%rsi), %rcx
+; CHECK-EX-NEXT:    movq %rax, %rdx
+; CHECK-EX-NEXT:    orq %rcx, %rdx
+; CHECK-EX-NEXT:    shrq $32, %rdx
+; CHECK-EX-NEXT:    je .LBB36_1
+; CHECK-EX-NEXT:  # %bb.2:
+; CHECK-EX-NEXT:    cqto
+; CHECK-EX-NEXT:    idivq %rcx
+; CHECK-EX-NEXT:    retq
+; CHECK-EX-NEXT:  .LBB36_1:
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divl %ecx
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax def $rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p1 unordered, align 8
   %v2 = load atomic i64, i64* %p2 unordered, align 8
   %ret = sdiv i64 %v, %v2
   ret i64 %ret
 }
 
 ; Legal to fold (TODO)
 define i64 @load_fold_udiv1(i64* %p) {

 ; CHECK-O3-LABEL: load_fold_udiv1:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rdi), %rax
 ; CHECK-O3-NEXT:    movabsq $-8608480567731124087, %rcx # imm = 0x8888888888888889
 ; CHECK-O3-NEXT:    mulq %rcx
 ; CHECK-O3-NEXT:    movq %rdx, %rax
 ; CHECK-O3-NEXT:    shrq $3, %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_udiv1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movabsq $-8608480567731124087, %rax # imm = 0x8888888888888889
+; CHECK-EX-NEXT:    mulq (%rdi)
+; CHECK-EX-NEXT:    movq %rdx, %rax
+; CHECK-EX-NEXT:    shrq $3, %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = udiv i64 %v, 15
   ret i64 %ret
 }
 
 define i64 @load_fold_udiv2(i64* %p, i64 %v2) {
 ; CHECK-O0-LABEL: load_fold_udiv2:
 ; CHECK-O0:       # %bb.0:

 ; CHECK-O3-NEXT:    divq %rsi
 ; CHECK-O3-NEXT:    retq
 ; CHECK-O3-NEXT:  .LBB38_1:
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    xorl %edx, %edx
 ; CHECK-O3-NEXT:    divl %esi
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax def $rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_udiv2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq %rax, %rcx
+; CHECK-EX-NEXT:    orq %rsi, %rcx
+; CHECK-EX-NEXT:    shrq $32, %rcx
+; CHECK-EX-NEXT:    je .LBB38_1
+; CHECK-EX-NEXT:  # %bb.2:
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divq %rsi
+; CHECK-EX-NEXT:    retq
+; CHECK-EX-NEXT:  .LBB38_1:
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divl %esi
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax def $rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = udiv i64 %v, %v2
   ret i64 %ret
 }
 
 define i64 @load_fold_udiv3(i64* %p1, i64* %p2) {
 ; CHECK-O0-LABEL: load_fold_udiv3:
 ; CHECK-O0:       # %bb.0:

 ; CHECK-O3-NEXT:    divq %rcx
 ; CHECK-O3-NEXT:    retq
 ; CHECK-O3-NEXT:  .LBB39_1:
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    xorl %edx, %edx
 ; CHECK-O3-NEXT:    divl %ecx
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax def $rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_udiv3:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq (%rsi), %rcx
+; CHECK-EX-NEXT:    movq %rax, %rdx
+; CHECK-EX-NEXT:    orq %rcx, %rdx
+; CHECK-EX-NEXT:    shrq $32, %rdx
+; CHECK-EX-NEXT:    je .LBB39_1
+; CHECK-EX-NEXT:  # %bb.2:
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divq %rcx
+; CHECK-EX-NEXT:    retq
+; CHECK-EX-NEXT:  .LBB39_1:
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divl %ecx
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax def $rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p1 unordered, align 8
   %v2 = load atomic i64, i64* %p2 unordered, align 8
   %ret = udiv i64 %v, %v2
   ret i64 %ret
 }
 
 ; Legal to fold (TODO)
 define i64 @load_fold_srem1(i64* %p) {

 ; CHECK-O3-NEXT:    shrq $63, %rax
 ; CHECK-O3-NEXT:    sarq $3, %rdx
 ; CHECK-O3-NEXT:    addq %rax, %rdx
 ; CHECK-O3-NEXT:    leaq (%rdx,%rdx,4), %rax
 ; CHECK-O3-NEXT:    leaq (%rax,%rax,2), %rax
 ; CHECK-O3-NEXT:    subq %rax, %rcx
 ; CHECK-O3-NEXT:    movq %rcx, %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_srem1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rcx
+; CHECK-EX-NEXT:    movabsq $-8608480567731124087, %rdx # imm = 0x8888888888888889
+; CHECK-EX-NEXT:    movq %rcx, %rax
+; CHECK-EX-NEXT:    imulq %rdx
+; CHECK-EX-NEXT:    addq %rcx, %rdx
+; CHECK-EX-NEXT:    movq %rdx, %rax
+; CHECK-EX-NEXT:    shrq $63, %rax
+; CHECK-EX-NEXT:    sarq $3, %rdx
+; CHECK-EX-NEXT:    addq %rax, %rdx
+; CHECK-EX-NEXT:    leaq (%rdx,%rdx,4), %rax
+; CHECK-EX-NEXT:    leaq (%rax,%rax,2), %rax
+; CHECK-EX-NEXT:    subq %rax, %rcx
+; CHECK-EX-NEXT:    movq %rcx, %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = srem i64 %v, 15
   ret i64 %ret
 }
 
 ; Legal, as expected
 define i64 @load_fold_srem2(i64* %p, i64 %v2) {
 ; CHECK-O0-LABEL: load_fold_srem2:

 ; CHECK-O3-NEXT:    movq %rdx, %rax
 ; CHECK-O3-NEXT:    retq
 ; CHECK-O3-NEXT:  .LBB41_1:
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    xorl %edx, %edx
 ; CHECK-O3-NEXT:    divl %esi
 ; CHECK-O3-NEXT:    movl %edx, %eax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_srem2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq %rax, %rcx
+; CHECK-EX-NEXT:    orq %rsi, %rcx
+; CHECK-EX-NEXT:    shrq $32, %rcx
+; CHECK-EX-NEXT:    je .LBB41_1
+; CHECK-EX-NEXT:  # %bb.2:
+; CHECK-EX-NEXT:    cqto
+; CHECK-EX-NEXT:    idivq %rsi
+; CHECK-EX-NEXT:    movq %rdx, %rax
+; CHECK-EX-NEXT:    retq
+; CHECK-EX-NEXT:  .LBB41_1:
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divl %esi
+; CHECK-EX-NEXT:    movl %edx, %eax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = srem i64 %v, %v2
   ret i64 %ret
 }
 
 define i64 @load_fold_srem3(i64* %p1, i64* %p2) {
 ; CHECK-O0-LABEL: load_fold_srem3:
 ; CHECK-O0:       # %bb.0:

 ; CHECK-O3-NEXT:    movq %rdx, %rax
 ; CHECK-O3-NEXT:    retq
 ; CHECK-O3-NEXT:  .LBB42_1:
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    xorl %edx, %edx
 ; CHECK-O3-NEXT:    divl %ecx
 ; CHECK-O3-NEXT:    movl %edx, %eax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_srem3:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq (%rsi), %rcx
+; CHECK-EX-NEXT:    movq %rax, %rdx
+; CHECK-EX-NEXT:    orq %rcx, %rdx
+; CHECK-EX-NEXT:    shrq $32, %rdx
+; CHECK-EX-NEXT:    je .LBB42_1
+; CHECK-EX-NEXT:  # %bb.2:
+; CHECK-EX-NEXT:    cqto
+; CHECK-EX-NEXT:    idivq %rcx
+; CHECK-EX-NEXT:    movq %rdx, %rax
+; CHECK-EX-NEXT:    retq
+; CHECK-EX-NEXT:  .LBB42_1:
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divl %ecx
+; CHECK-EX-NEXT:    movl %edx, %eax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p1 unordered, align 8
   %v2 = load atomic i64, i64* %p2 unordered, align 8
   %ret = srem i64 %v, %v2
   ret i64 %ret
 }
 
 ; Legal to fold (TODO)
 define i64 @load_fold_urem1(i64* %p) {

 ; CHECK-O3-NEXT:    movq %rcx, %rax
 ; CHECK-O3-NEXT:    mulq %rdx
 ; CHECK-O3-NEXT:    shrq $3, %rdx
 ; CHECK-O3-NEXT:    leaq (%rdx,%rdx,4), %rax
 ; CHECK-O3-NEXT:    leaq (%rax,%rax,2), %rax
 ; CHECK-O3-NEXT:    subq %rax, %rcx
 ; CHECK-O3-NEXT:    movq %rcx, %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_urem1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rcx
+; CHECK-EX-NEXT:    movabsq $-8608480567731124087, %rdx # imm = 0x8888888888888889
+; CHECK-EX-NEXT:    movq %rcx, %rax
+; CHECK-EX-NEXT:    mulq %rdx
+; CHECK-EX-NEXT:    shrq $3, %rdx
+; CHECK-EX-NEXT:    leaq (%rdx,%rdx,4), %rax
+; CHECK-EX-NEXT:    leaq (%rax,%rax,2), %rax
+; CHECK-EX-NEXT:    subq %rax, %rcx
+; CHECK-EX-NEXT:    movq %rcx, %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = urem i64 %v, 15
   ret i64 %ret
 }
 
 ; Legal, as expected
 define i64 @load_fold_urem2(i64* %p, i64 %v2) {
 ; CHECK-O0-LABEL: load_fold_urem2:

 ; CHECK-O3-NEXT:    movq %rdx, %rax
 ; CHECK-O3-NEXT:    retq
 ; CHECK-O3-NEXT:  .LBB44_1:
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    xorl %edx, %edx
 ; CHECK-O3-NEXT:    divl %esi
 ; CHECK-O3-NEXT:    movl %edx, %eax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_urem2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq %rax, %rcx
+; CHECK-EX-NEXT:    orq %rsi, %rcx
+; CHECK-EX-NEXT:    shrq $32, %rcx
+; CHECK-EX-NEXT:    je .LBB44_1
+; CHECK-EX-NEXT:  # %bb.2:
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divq %rsi
+; CHECK-EX-NEXT:    movq %rdx, %rax
+; CHECK-EX-NEXT:    retq
+; CHECK-EX-NEXT:  .LBB44_1:
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divl %esi
+; CHECK-EX-NEXT:    movl %edx, %eax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = urem i64 %v, %v2
   ret i64 %ret
 }
 
 define i64 @load_fold_urem3(i64* %p1, i64* %p2) {
 ; CHECK-O0-LABEL: load_fold_urem3:
 ; CHECK-O0:       # %bb.0:

 ; CHECK-O3-NEXT:    movq %rdx, %rax
 ; CHECK-O3-NEXT:    retq
 ; CHECK-O3-NEXT:  .LBB45_1:
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    xorl %edx, %edx
 ; CHECK-O3-NEXT:    divl %ecx
 ; CHECK-O3-NEXT:    movl %edx, %eax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_urem3:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq (%rsi), %rcx
+; CHECK-EX-NEXT:    movq %rax, %rdx
+; CHECK-EX-NEXT:    orq %rcx, %rdx
+; CHECK-EX-NEXT:    shrq $32, %rdx
+; CHECK-EX-NEXT:    je .LBB45_1
+; CHECK-EX-NEXT:  # %bb.2:
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divq %rcx
+; CHECK-EX-NEXT:    movq %rdx, %rax
+; CHECK-EX-NEXT:    retq
+; CHECK-EX-NEXT:  .LBB45_1:
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divl %ecx
+; CHECK-EX-NEXT:    movl %edx, %eax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p1 unordered, align 8
   %v2 = load atomic i64, i64* %p2 unordered, align 8
   %ret = urem i64 %v, %v2
   ret i64 %ret
 }
 
 ; Legal, as expected
 define i64 @load_fold_shl1(i64* %p) {

 ; CHECK-O0-NEXT:    # kill: def $cl killed $rcx
 ; CHECK-O0-NEXT:    shlq %cl, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_shl2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    shlxq %rsi, (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_shl2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    shlxq %rsi, (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = shl i64 %v, %v2
   ret i64 %ret
 }
 
 define i64 @load_fold_shl3(i64* %p1, i64* %p2) {
 ; CHECK-O0-LABEL: load_fold_shl3:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    movq (%rsi), %rcx
 ; CHECK-O0-NEXT:    # kill: def $cl killed $rcx
 ; CHECK-O0-NEXT:    shlq %cl, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_shl3:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rsi), %rax
 ; CHECK-O3-NEXT:    shlxq %rax, (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_shl3:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movb (%rsi), %al
+; CHECK-EX-NEXT:    shlxq %rax, (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p1 unordered, align 8
   %v2 = load atomic i64, i64* %p2 unordered, align 8
   %ret = shl i64 %v, %v2
   ret i64 %ret
 }
 
 ; Legal, as expected
 define i64 @load_fold_lshr1(i64* %p) {

 ; CHECK-O0-NEXT:    # kill: def $cl killed $rcx
 ; CHECK-O0-NEXT:    shrq %cl, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_lshr2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    shrxq %rsi, (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_lshr2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    shrxq %rsi, (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = lshr i64 %v, %v2
   ret i64 %ret
 }
 
 define i64 @load_fold_lshr3(i64* %p1, i64* %p2) {
 ; CHECK-O0-LABEL: load_fold_lshr3:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    movq (%rsi), %rcx
 ; CHECK-O0-NEXT:    # kill: def $cl killed $rcx
 ; CHECK-O0-NEXT:    shrq %cl, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_lshr3:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rsi), %rax
 ; CHECK-O3-NEXT:    shrxq %rax, (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_lshr3:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movb (%rsi), %al
+; CHECK-EX-NEXT:    shrxq %rax, (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p1 unordered, align 8
   %v2 = load atomic i64, i64* %p2 unordered, align 8
   %ret = lshr i64 %v, %v2
   ret i64 %ret
 }
 
 ; Legal, as expected
 define i64 @load_fold_ashr1(i64* %p) {

 ; CHECK-O0-NEXT:    # kill: def $cl killed $rcx
 ; CHECK-O0-NEXT:    sarq %cl, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_ashr2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    sarxq %rsi, (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_ashr2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    sarxq %rsi, (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = ashr i64 %v, %v2
   ret i64 %ret
 }
 
 define i64 @load_fold_ashr3(i64* %p1, i64* %p2) {
 ; CHECK-O0-LABEL: load_fold_ashr3:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    movq (%rsi), %rcx
 ; CHECK-O0-NEXT:    # kill: def $cl killed $rcx
 ; CHECK-O0-NEXT:    sarq %cl, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_ashr3:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rsi), %rax
 ; CHECK-O3-NEXT:    sarxq %rax, (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_ashr3:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movb (%rsi), %al
+; CHECK-EX-NEXT:    sarxq %rax, (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p1 unordered, align 8
   %v2 = load atomic i64, i64* %p2 unordered, align 8
   %ret = ashr i64 %v, %v2
   ret i64 %ret
 }
 
 ; Legal, as expected
 define i64 @load_fold_and1(i64* %p) {
 ; CHECK-O0-LABEL: load_fold_and1:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    andq $15, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_and1:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rdi), %rax
 ; CHECK-O3-NEXT:    andl $15, %eax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_and1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    andl $15, %eax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = and i64 %v, 15
   ret i64 %ret
 }
 
 define i64 @load_fold_and2(i64* %p, i64 %v2) {
 ; CHECK-O0-LABEL: load_fold_and2:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    andq (%rdi), %rsi
 ; CHECK-O0-NEXT:    movq %rsi, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_and2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq %rsi, %rax
 ; CHECK-O3-NEXT:    andq (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_and2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rsi, %rax
+; CHECK-EX-NEXT:    andq (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = and i64 %v, %v2
   ret i64 %ret
 }
 
 define i64 @load_fold_and3(i64* %p1, i64* %p2) {
 ; CHECK-O0-LABEL: load_fold_and3:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    andq (%rsi), %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_and3:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rsi), %rax
 ; CHECK-O3-NEXT:    andq (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_and3:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    andq (%rsi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p1 unordered, align 8
   %v2 = load atomic i64, i64* %p2 unordered, align 8
   %ret = and i64 %v, %v2
   ret i64 %ret
 }
 
 ; Legal, as expected
 define i64 @load_fold_or1(i64* %p) {

 ; CHECK-O0-NEXT:    movq %rsi, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_or2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq %rsi, %rax
 ; CHECK-O3-NEXT:    orq (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_or2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rsi, %rax
+; CHECK-EX-NEXT:    orq (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = or i64 %v, %v2
   ret i64 %ret
 }
 
 define i64 @load_fold_or3(i64* %p1, i64* %p2) {
 ; CHECK-O0-LABEL: load_fold_or3:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    orq (%rsi), %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_or3:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rsi), %rax
 ; CHECK-O3-NEXT:    orq (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_or3:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    orq (%rsi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p1 unordered, align 8
   %v2 = load atomic i64, i64* %p2 unordered, align 8
   %ret = or i64 %v, %v2
   ret i64 %ret
 }
 
 ; Legal, as expected
 define i64 @load_fold_xor1(i64* %p) {

 ; CHECK-O0-NEXT:    movq %rsi, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_xor2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq %rsi, %rax
 ; CHECK-O3-NEXT:    xorq (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_xor2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rsi, %rax
+; CHECK-EX-NEXT:    xorq (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = xor i64 %v, %v2
   ret i64 %ret
 }
 
 define i64 @load_fold_xor3(i64* %p1, i64* %p2) {
 ; CHECK-O0-LABEL: load_fold_xor3:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    xorq (%rsi), %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_xor3:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rsi), %rax
 ; CHECK-O3-NEXT:    xorq (%rdi), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_xor3:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    xorq (%rsi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p1 unordered, align 8
   %v2 = load atomic i64, i64* %p2 unordered, align 8
   %ret = xor i64 %v, %v2
   ret i64 %ret
 }
 
 define i1 @load_fold_icmp1(i64* %p) {
 ; CHECK-O0-LABEL: load_fold_icmp1:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    subq $15, %rax
 ; CHECK-O0-NEXT:    sete %cl
 ; CHECK-O0-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; CHECK-O0-NEXT:    movb %cl, %al
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_icmp1:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    cmpq $15, (%rdi)
 ; CHECK-O3-NEXT:    sete %al
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_icmp1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    cmpq $15, (%rdi)
+; CHECK-EX-NEXT:    sete %al
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = icmp eq i64 %v, 15
   ret i1 %ret
 }
 
 define i1 @load_fold_icmp2(i64* %p, i64 %v2) {
 ; CHECK-O0-LABEL: load_fold_icmp2:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    subq %rsi, %rax
 ; CHECK-O0-NEXT:    sete %cl
 ; CHECK-O0-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; CHECK-O0-NEXT:    movb %cl, %al
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_icmp2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    cmpq %rsi, (%rdi)
 ; CHECK-O3-NEXT:    sete %al
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_icmp2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    cmpq %rsi, (%rdi)
+; CHECK-EX-NEXT:    sete %al
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = icmp eq i64 %v, %v2
   ret i1 %ret
 }
 
 define i1 @load_fold_icmp3(i64* %p1, i64* %p2) {
 ; CHECK-O0-LABEL: load_fold_icmp3:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    movq (%rsi), %rcx
 ; CHECK-O0-NEXT:    subq %rcx, %rax
 ; CHECK-O0-NEXT:    sete %dl
 ; CHECK-O0-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; CHECK-O0-NEXT:    movb %dl, %al
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: load_fold_icmp3:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rsi), %rax
 ; CHECK-O3-NEXT:    cmpq %rax, (%rdi)
 ; CHECK-O3-NEXT:    sete %al
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_fold_icmp3:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    cmpq (%rsi), %rax
+; CHECK-EX-NEXT:    sete %al
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p1 unordered, align 8
   %v2 = load atomic i64, i64* %p2 unordered, align 8
   %ret = icmp eq i64 %v, %v2
   ret i1 %ret
 }
 
 
 ;; The next batch of tests check for read-modify-write patterns

 ; CHECK-O0-NEXT:    addq $15, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_add1:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    addq $15, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_add1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    addq $15, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = add i64 %prev, 15
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_add2(i64* %p, i64 %v) {
 ; CHECK-O0-LABEL: rmw_fold_add2:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    addq %rsi, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_add2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    addq %rsi, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_add2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    addq %rsi, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = add i64 %prev, %v
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_sub1(i64* %p, i64 %v) {
 ; CHECK-O0-LABEL: rmw_fold_sub1:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    addq $-15, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_sub1:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    addq $-15, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_sub1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    addq $-15, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = sub i64 %prev, 15
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_sub2(i64* %p, i64 %v) {
 ; CHECK-O0-LABEL: rmw_fold_sub2:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    subq %rsi, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_sub2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    subq %rsi, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_sub2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    subq %rsi, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = sub i64 %prev, %v
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_mul1(i64* %p, i64 %v) {

 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_mul2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    imulq (%rdi), %rsi
 ; CHECK-O3-NEXT:    movq %rsi, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_mul2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    imulq (%rdi), %rsi
+; CHECK-EX-NEXT:    movq %rsi, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = mul i64 %prev, %v
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_sdiv1(i64* %p, i64 %v) {

 ; CHECK-O3-NEXT:    imulq %rdx
 ; CHECK-O3-NEXT:    addq %rcx, %rdx
 ; CHECK-O3-NEXT:    movq %rdx, %rax
 ; CHECK-O3-NEXT:    shrq $63, %rax
 ; CHECK-O3-NEXT:    sarq $3, %rdx
 ; CHECK-O3-NEXT:    addq %rax, %rdx
 ; CHECK-O3-NEXT:    movq %rdx, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_sdiv1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rcx
+; CHECK-EX-NEXT:    movabsq $-8608480567731124087, %rdx # imm = 0x8888888888888889
+; CHECK-EX-NEXT:    movq %rcx, %rax
+; CHECK-EX-NEXT:    imulq %rdx
+; CHECK-EX-NEXT:    addq %rcx, %rdx
+; CHECK-EX-NEXT:    movq %rdx, %rax
+; CHECK-EX-NEXT:    shrq $63, %rax
+; CHECK-EX-NEXT:    sarq $3, %rdx
+; CHECK-EX-NEXT:    addq %rax, %rdx
+; CHECK-EX-NEXT:    movq %rdx, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = sdiv i64 %prev, 15
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_sdiv2(i64* %p, i64 %v) {

 ; CHECK-O3-NEXT:    retq
 ; CHECK-O3-NEXT:  .LBB74_1:
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    xorl %edx, %edx
 ; CHECK-O3-NEXT:    divl %esi
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax def $rax
 ; CHECK-O3-NEXT:    movq %rax, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_sdiv2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq %rax, %rcx
+; CHECK-EX-NEXT:    orq %rsi, %rcx
+; CHECK-EX-NEXT:    shrq $32, %rcx
+; CHECK-EX-NEXT:    je .LBB74_1
+; CHECK-EX-NEXT:  # %bb.2:
+; CHECK-EX-NEXT:    cqto
+; CHECK-EX-NEXT:    idivq %rsi
+; CHECK-EX-NEXT:    movq %rax, (%rdi)
+; CHECK-EX-NEXT:    retq
+; CHECK-EX-NEXT:  .LBB74_1:
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divl %esi
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax def $rax
+; CHECK-EX-NEXT:    movq %rax, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = sdiv i64 %prev, %v
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_udiv1(i64* %p, i64 %v) {
-; CHECK-LABEL: rmw_fold_udiv1:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq (%rdi), %rax
-; CHECK-NEXT:    movabsq $-8608480567731124087, %rcx # imm = 0x8888888888888889
-; CHECK-NEXT:    mulq %rcx
-; CHECK-NEXT:    shrq $3, %rdx
-; CHECK-NEXT:    movq %rdx, (%rdi)
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: rmw_fold_udiv1:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movq (%rdi), %rax
+; CHECK-NOX-NEXT:    movabsq $-8608480567731124087, %rcx # imm = 0x8888888888888889
+; CHECK-NOX-NEXT:    mulq %rcx
+; CHECK-NOX-NEXT:    shrq $3, %rdx
+; CHECK-NOX-NEXT:    movq %rdx, (%rdi)
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_udiv1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movabsq $-8608480567731124087, %rax # imm = 0x8888888888888889
+; CHECK-EX-NEXT:    mulq (%rdi)
+; CHECK-EX-NEXT:    shrq $3, %rdx
+; CHECK-EX-NEXT:    movq %rdx, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = udiv i64 %prev, 15
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_udiv2(i64* %p, i64 %v) {

 ; CHECK-O3-NEXT:    retq
 ; CHECK-O3-NEXT:  .LBB76_1:
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    xorl %edx, %edx
 ; CHECK-O3-NEXT:    divl %esi
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax def $rax
 ; CHECK-O3-NEXT:    movq %rax, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_udiv2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq %rax, %rcx
+; CHECK-EX-NEXT:    orq %rsi, %rcx
+; CHECK-EX-NEXT:    shrq $32, %rcx
+; CHECK-EX-NEXT:    je .LBB76_1
+; CHECK-EX-NEXT:  # %bb.2:
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divq %rsi
+; CHECK-EX-NEXT:    movq %rax, (%rdi)
+; CHECK-EX-NEXT:    retq
+; CHECK-EX-NEXT:  .LBB76_1:
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divl %esi
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax def $rax
+; CHECK-EX-NEXT:    movq %rax, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = udiv i64 %prev, %v
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_srem1(i64* %p, i64 %v) {

 ; CHECK-O3-NEXT:    shrq $63, %rax
 ; CHECK-O3-NEXT:    sarq $3, %rdx
 ; CHECK-O3-NEXT:    addq %rax, %rdx
 ; CHECK-O3-NEXT:    leaq (%rdx,%rdx,4), %rax
 ; CHECK-O3-NEXT:    leaq (%rax,%rax,2), %rax
 ; CHECK-O3-NEXT:    subq %rax, %rcx
 ; CHECK-O3-NEXT:    movq %rcx, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_srem1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rcx
+; CHECK-EX-NEXT:    movabsq $-8608480567731124087, %rdx # imm = 0x8888888888888889
+; CHECK-EX-NEXT:    movq %rcx, %rax
+; CHECK-EX-NEXT:    imulq %rdx
+; CHECK-EX-NEXT:    addq %rcx, %rdx
+; CHECK-EX-NEXT:    movq %rdx, %rax
+; CHECK-EX-NEXT:    shrq $63, %rax
+; CHECK-EX-NEXT:    sarq $3, %rdx
+; CHECK-EX-NEXT:    addq %rax, %rdx
+; CHECK-EX-NEXT:    leaq (%rdx,%rdx,4), %rax
+; CHECK-EX-NEXT:    leaq (%rax,%rax,2), %rax
+; CHECK-EX-NEXT:    subq %rax, %rcx
+; CHECK-EX-NEXT:    movq %rcx, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = srem i64 %prev, 15
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_srem2(i64* %p, i64 %v) {

 ; CHECK-O3-NEXT:    retq
 ; CHECK-O3-NEXT:  .LBB78_1:
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    xorl %edx, %edx
 ; CHECK-O3-NEXT:    divl %esi
 ; CHECK-O3-NEXT:    # kill: def $edx killed $edx def $rdx
 ; CHECK-O3-NEXT:    movq %rdx, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_srem2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq %rax, %rcx
+; CHECK-EX-NEXT:    orq %rsi, %rcx
+; CHECK-EX-NEXT:    shrq $32, %rcx
+; CHECK-EX-NEXT:    je .LBB78_1
+; CHECK-EX-NEXT:  # %bb.2:
+; CHECK-EX-NEXT:    cqto
+; CHECK-EX-NEXT:    idivq %rsi
+; CHECK-EX-NEXT:    movq %rdx, (%rdi)
+; CHECK-EX-NEXT:    retq
+; CHECK-EX-NEXT:  .LBB78_1:
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divl %esi
+; CHECK-EX-NEXT:    # kill: def $edx killed $edx def $rdx
+; CHECK-EX-NEXT:    movq %rdx, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = srem i64 %prev, %v
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_urem1(i64* %p, i64 %v) {

 ; CHECK-O3-NEXT:    movq %rcx, %rax
 ; CHECK-O3-NEXT:    mulq %rdx
 ; CHECK-O3-NEXT:    shrq $3, %rdx
 ; CHECK-O3-NEXT:    leaq (%rdx,%rdx,4), %rax
 ; CHECK-O3-NEXT:    leaq (%rax,%rax,2), %rax
 ; CHECK-O3-NEXT:    subq %rax, %rcx
 ; CHECK-O3-NEXT:    movq %rcx, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_urem1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rcx
+; CHECK-EX-NEXT:    movabsq $-8608480567731124087, %rdx # imm = 0x8888888888888889
+; CHECK-EX-NEXT:    movq %rcx, %rax
+; CHECK-EX-NEXT:    mulq %rdx
+; CHECK-EX-NEXT:    shrq $3, %rdx
+; CHECK-EX-NEXT:    leaq (%rdx,%rdx,4), %rax
+; CHECK-EX-NEXT:    leaq (%rax,%rax,2), %rax
+; CHECK-EX-NEXT:    subq %rax, %rcx
+; CHECK-EX-NEXT:    movq %rcx, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = urem i64 %prev, 15
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_urem2(i64* %p, i64 %v) {

 ; CHECK-O3-NEXT:    retq
 ; CHECK-O3-NEXT:  .LBB80_1:
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    xorl %edx, %edx
 ; CHECK-O3-NEXT:    divl %esi
 ; CHECK-O3-NEXT:    # kill: def $edx killed $edx def $rdx
 ; CHECK-O3-NEXT:    movq %rdx, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_urem2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq %rax, %rcx
+; CHECK-EX-NEXT:    orq %rsi, %rcx
+; CHECK-EX-NEXT:    shrq $32, %rcx
+; CHECK-EX-NEXT:    je .LBB80_1
+; CHECK-EX-NEXT:  # %bb.2:
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divq %rsi
+; CHECK-EX-NEXT:    movq %rdx, (%rdi)
+; CHECK-EX-NEXT:    retq
+; CHECK-EX-NEXT:  .LBB80_1:
+; CHECK-EX-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-EX-NEXT:    xorl %edx, %edx
+; CHECK-EX-NEXT:    divl %esi
+; CHECK-EX-NEXT:    # kill: def $edx killed $edx def $rdx
+; CHECK-EX-NEXT:    movq %rdx, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = urem i64 %prev, %v
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal to fold (TODO)
 define void @rmw_fold_shl1(i64* %p, i64 %v) {
-; CHECK-LABEL: rmw_fold_shl1:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq (%rdi), %rax
-; CHECK-NEXT:    shlq $15, %rax
-; CHECK-NEXT:    movq %rax, (%rdi)
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: rmw_fold_shl1:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movq (%rdi), %rax
+; CHECK-NOX-NEXT:    shlq $15, %rax
+; CHECK-NOX-NEXT:    movq %rax, (%rdi)
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_shl1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    shlq $15, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = shl i64 %prev, 15
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal to fold (TODO)
 define void @rmw_fold_shl2(i64* %p, i64 %v) {
 ; CHECK-O0-LABEL: rmw_fold_shl2:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    # kill: def $sil killed $sil killed $rsi
 ; CHECK-O0-NEXT:    # implicit-def: $rcx
 ; CHECK-O0-NEXT:    movb %sil, %cl
 ; CHECK-O0-NEXT:    shlxq %rcx, %rax, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_shl2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    shlxq %rsi, (%rdi), %rax
 ; CHECK-O3-NEXT:    movq %rax, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_shl2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rsi, %rcx
+; CHECK-EX-NEXT:    # kill: def $cl killed $cl killed $rcx
+; CHECK-EX-NEXT:    shlq %cl, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = shl i64 %prev, %v
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal to fold (TODO)
 define void @rmw_fold_lshr1(i64* %p, i64 %v) {
-; CHECK-LABEL: rmw_fold_lshr1:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq (%rdi), %rax
-; CHECK-NEXT:    shrq $15, %rax
-; CHECK-NEXT:    movq %rax, (%rdi)
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: rmw_fold_lshr1:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movq (%rdi), %rax
+; CHECK-NOX-NEXT:    shrq $15, %rax
+; CHECK-NOX-NEXT:    movq %rax, (%rdi)
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_lshr1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    shrq $15, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = lshr i64 %prev, 15
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal to fold (TODO)
 define void @rmw_fold_lshr2(i64* %p, i64 %v) {
 ; CHECK-O0-LABEL: rmw_fold_lshr2:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    # kill: def $sil killed $sil killed $rsi
 ; CHECK-O0-NEXT:    # implicit-def: $rcx
 ; CHECK-O0-NEXT:    movb %sil, %cl
 ; CHECK-O0-NEXT:    shrxq %rcx, %rax, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_lshr2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    shrxq %rsi, (%rdi), %rax
 ; CHECK-O3-NEXT:    movq %rax, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_lshr2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rsi, %rcx
+; CHECK-EX-NEXT:    # kill: def $cl killed $cl killed $rcx
+; CHECK-EX-NEXT:    shrq %cl, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = lshr i64 %prev, %v
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal to fold (TODO)
 define void @rmw_fold_ashr1(i64* %p, i64 %v) {
-; CHECK-LABEL: rmw_fold_ashr1:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq (%rdi), %rax
-; CHECK-NEXT:    sarq $15, %rax
-; CHECK-NEXT:    movq %rax, (%rdi)
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: rmw_fold_ashr1:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movq (%rdi), %rax
+; CHECK-NOX-NEXT:    sarq $15, %rax
+; CHECK-NOX-NEXT:    movq %rax, (%rdi)
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_ashr1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    sarq $15, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = ashr i64 %prev, 15
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal to fold (TODO)
 define void @rmw_fold_ashr2(i64* %p, i64 %v) {
 ; CHECK-O0-LABEL: rmw_fold_ashr2:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    # kill: def $sil killed $sil killed $rsi
 ; CHECK-O0-NEXT:    # implicit-def: $rcx
 ; CHECK-O0-NEXT:    movb %sil, %cl
 ; CHECK-O0-NEXT:    sarxq %rcx, %rax, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_ashr2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    sarxq %rsi, (%rdi), %rax
 ; CHECK-O3-NEXT:    movq %rax, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_ashr2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rsi, %rcx
+; CHECK-EX-NEXT:    # kill: def $cl killed $cl killed $rcx
+; CHECK-EX-NEXT:    sarq %cl, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = ashr i64 %prev, %v
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_and1(i64* %p, i64 %v) {
 ; CHECK-O0-LABEL: rmw_fold_and1:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O0-NEXT:    andl $15, %eax
 ; CHECK-O0-NEXT:    movl %eax, %ecx
 ; CHECK-O0-NEXT:    movq %rcx, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_and1:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    andq $15, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_and1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    andq $15, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = and i64 %prev, 15
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_and2(i64* %p, i64 %v) {
 ; CHECK-O0-LABEL: rmw_fold_and2:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    andq %rsi, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_and2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    andq %rsi, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_and2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    andq %rsi, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = and i64 %prev, %v
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_or1(i64* %p, i64 %v) {
 ; CHECK-O0-LABEL: rmw_fold_or1:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    orq $15, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_or1:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    orq $15, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_or1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    orq $15, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = or i64 %prev, 15
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_or2(i64* %p, i64 %v) {
 ; CHECK-O0-LABEL: rmw_fold_or2:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    orq %rsi, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_or2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    orq %rsi, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_or2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    orq %rsi, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = or i64 %prev, %v
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_xor1(i64* %p, i64 %v) {
 ; CHECK-O0-LABEL: rmw_fold_xor1:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    xorq $15, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_xor1:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    xorq $15, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_xor1:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    xorq $15, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = xor i64 %prev, 15
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal, as expected
 define void @rmw_fold_xor2(i64* %p, i64 %v) {
 ; CHECK-O0-LABEL: rmw_fold_xor2:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    xorq %rsi, %rax
 ; CHECK-O0-NEXT:    movq %rax, (%rdi)
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: rmw_fold_xor2:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    xorq %rsi, (%rdi)
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: rmw_fold_xor2:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    xorq %rsi, (%rdi)
+; CHECK-EX-NEXT:    retq
   %prev = load atomic i64, i64* %p unordered, align 8
   %val = xor i64 %prev, %v
   store atomic i64 %val, i64* %p unordered, align 8
   ret void
 }
 
 ;; The next batch test truncations, in combination w/operations which could
 ;; be folded against the memory operation.
 
 ; Legal to reduce the load width (TODO)
 define i32 @fold_trunc(i64* %p) {
-; CHECK-LABEL: fold_trunc:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq (%rdi), %rax
-; CHECK-NEXT:    # kill: def $eax killed $eax killed $rax
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: fold_trunc:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movq (%rdi), %rax
+; CHECK-NOX-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: fold_trunc:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movl (%rdi), %eax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %ret = trunc i64 %v to i32
   ret i32 %ret
 }
 
 ; Legal to reduce the load width and fold the load (TODO)
 define i32 @fold_trunc_add(i64* %p, i32 %v2) {
 ; CHECK-O0-LABEL: fold_trunc_add:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O0-NEXT:    addl %esi, %eax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: fold_trunc_add:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rdi), %rax
 ; CHECK-O3-NEXT:    addl %esi, %eax
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: fold_trunc_add:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movl %esi, %eax
+; CHECK-EX-NEXT:    addl (%rdi), %eax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %trunc = trunc i64 %v to i32
   %ret = add i32 %trunc, %v2
   ret i32 %ret
 }
 
 ; Legal to reduce the load width and fold the load (TODO)
 define i32 @fold_trunc_and(i64* %p, i32 %v2) {
 ; CHECK-O0-LABEL: fold_trunc_and:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O0-NEXT:    andl %esi, %eax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: fold_trunc_and:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rdi), %rax
 ; CHECK-O3-NEXT:    andl %esi, %eax
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: fold_trunc_and:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movl %esi, %eax
+; CHECK-EX-NEXT:    andl (%rdi), %eax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %trunc = trunc i64 %v to i32
   %ret = and i32 %trunc, %v2
   ret i32 %ret
 }
 
 ; Legal to reduce the load width and fold the load (TODO)
 define i32 @fold_trunc_or(i64* %p, i32 %v2) {
 ; CHECK-O0-LABEL: fold_trunc_or:
 ; CHECK-O0:       # %bb.0:
 ; CHECK-O0-NEXT:    movq (%rdi), %rax
 ; CHECK-O0-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O0-NEXT:    orl %esi, %eax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: fold_trunc_or:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rdi), %rax
 ; CHECK-O3-NEXT:    orl %esi, %eax
 ; CHECK-O3-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: fold_trunc_or:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movl %esi, %eax
+; CHECK-EX-NEXT:    orl (%rdi), %eax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %trunc = trunc i64 %v to i32
   %ret = or i32 %trunc, %v2
   ret i32 %ret
 }
 
 ; It's tempting to split the wide load into two smaller byte loads
 ; to reduce memory traffic, but this would be illegal for a atomic load

 ; CHECK-O3-LABEL: split_load:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq (%rdi), %rax
 ; CHECK-O3-NEXT:    movq %rax, %rcx
 ; CHECK-O3-NEXT:    shrq $32, %rcx
 ; CHECK-O3-NEXT:    orl %eax, %ecx
 ; CHECK-O3-NEXT:    movzbl %cl, %eax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: split_load:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    movq %rax, %rcx
+; CHECK-EX-NEXT:    shrq $32, %rcx
+; CHECK-EX-NEXT:    orl %eax, %ecx
+; CHECK-EX-NEXT:    movzbl %cl, %eax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %b1 = trunc i64 %v to i8
   %v.shift = lshr i64 %v, 32
   %b2 = trunc i64 %v.shift to i8
   %or = or i8 %b1, %b2
   %ret = zext i8 %or to i32
   ret i32 %ret
 }

 ; CHECK-NEXT:    movq (%rdi), %rax
 ; CHECK-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   ret i64 %v
 }
 
 ; Legal to forward and fold (TODO)
 define i64 @load_forwarding(i64* %p) {
-; CHECK-LABEL: load_forwarding:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq (%rdi), %rax
-; CHECK-NEXT:    orq (%rdi), %rax
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: load_forwarding:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movq (%rdi), %rax
+; CHECK-NOX-NEXT:    orq (%rdi), %rax
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: load_forwarding:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   %v2 = load atomic i64, i64* %p unordered, align 8
   %ret = or i64 %v, %v2
   ret i64 %ret
 }
 
 ; Legal to forward (TODO)
 define i64 @store_forward(i64* %p, i64 %v) {
-; CHECK-LABEL: store_forward:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq %rsi, (%rdi)
-; CHECK-NEXT:    movq (%rdi), %rax
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: store_forward:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movq %rsi, (%rdi)
+; CHECK-NOX-NEXT:    movq (%rdi), %rax
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: store_forward:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rsi, %rax
+; CHECK-EX-NEXT:    movq %rsi, (%rdi)
+; CHECK-EX-NEXT:    retq
   store atomic i64 %v, i64* %p unordered, align 8
   %ret = load atomic i64, i64* %p unordered, align 8
   ret i64 %ret
 }
 
 ; Legal to kill (TODO)
 define void @dead_writeback(i64* %p) {
 ; CHECK-LABEL: dead_writeback:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movq (%rdi), %rax
 ; CHECK-NEXT:    movq %rax, (%rdi)
 ; CHECK-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8
   store atomic i64 %v, i64* %p unordered, align 8
   ret void
 }
 
 ; Legal to kill (TODO)
 define void @dead_store(i64* %p, i64 %v) {
-; CHECK-LABEL: dead_store:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq $0, (%rdi)
-; CHECK-NEXT:    movq %rsi, (%rdi)
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: dead_store:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movq $0, (%rdi)
+; CHECK-NOX-NEXT:    movq %rsi, (%rdi)
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: dead_store:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rsi, (%rdi)
+; CHECK-EX-NEXT:    retq
   store atomic i64 0, i64* %p unordered, align 8
   store atomic i64 %v, i64* %p unordered, align 8
   ret void
 }
 
 ;; The next batch of tests ensure that we don't try to fold a load into a
 ;; use where the code motion implied for the load is prevented by a fence.
 ;; Note: We're checking that the load doesn't get moved below the fence as

 ; CHECK-O0-NEXT:    movq %rdi, %rax
 ; CHECK-O0-NEXT:    retq
 ;
 ; CHECK-O3-LABEL: fold_constant:
 ; CHECK-O3:       # %bb.0:
 ; CHECK-O3-NEXT:    movq %rdi, %rax
 ; CHECK-O3-NEXT:    addq {{.*}}(%rip), %rax
 ; CHECK-O3-NEXT:    retq
+;
+; CHECK-EX-LABEL: fold_constant:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rdi, %rax
+; CHECK-EX-NEXT:    addq {{.*}}(%rip), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* @Constant unordered, align 8
   %ret = add i64 %v, %arg
   ret i64 %ret
 }
 
 define i64 @fold_constant_clobber(i64* %p, i64 %arg) {
-; CHECK-LABEL: fold_constant_clobber:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq {{.*}}(%rip), %rax
-; CHECK-NEXT:    movq $5, (%rdi)
-; CHECK-NEXT:    addq %rsi, %rax
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: fold_constant_clobber:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movq {{.*}}(%rip), %rax
+; CHECK-NOX-NEXT:    movq $5, (%rdi)
+; CHECK-NOX-NEXT:    addq %rsi, %rax
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: fold_constant_clobber:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rsi, %rax
+; CHECK-EX-NEXT:    movq $5, (%rdi)
+; CHECK-EX-NEXT:    addq {{.*}}(%rip), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* @Constant unordered, align 8
   store i64 5, i64* %p
   %ret = add i64 %v, %arg
   ret i64 %ret
 }
 
 define i64 @fold_constant_fence(i64 %arg) {
-; CHECK-LABEL: fold_constant_fence:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq {{.*}}(%rip), %rax
-; CHECK-NEXT:    mfence
-; CHECK-NEXT:    addq %rdi, %rax
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: fold_constant_fence:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movq {{.*}}(%rip), %rax
+; CHECK-NOX-NEXT:    mfence
+; CHECK-NOX-NEXT:    addq %rdi, %rax
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: fold_constant_fence:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rdi, %rax
+; CHECK-EX-NEXT:    mfence
+; CHECK-EX-NEXT:    addq {{.*}}(%rip), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* @Constant unordered, align 8
   fence seq_cst
   %ret = add i64 %v, %arg
   ret i64 %ret
 }
 
 define i64 @fold_invariant_clobber(i64* dereferenceable(8) %p, i64 %arg) {
-; CHECK-LABEL: fold_invariant_clobber:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq (%rdi), %rax
-; CHECK-NEXT:    movq $5, (%rdi)
-; CHECK-NEXT:    addq %rsi, %rax
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: fold_invariant_clobber:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movq (%rdi), %rax
+; CHECK-NOX-NEXT:    movq $5, (%rdi)
+; CHECK-NOX-NEXT:    addq %rsi, %rax
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: fold_invariant_clobber:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rsi, %rax
+; CHECK-EX-NEXT:    movq $5, (%rdi)
+; CHECK-EX-NEXT:    addq (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8, !invariant.load !{}
   store i64 5, i64* %p
   %ret = add i64 %v, %arg
   ret i64 %ret
 }
 
 
 define i64 @fold_invariant_fence(i64* dereferenceable(8) %p, i64 %arg) {
-; CHECK-LABEL: fold_invariant_fence:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movq (%rdi), %rax
-; CHECK-NEXT:    mfence
-; CHECK-NEXT:    addq %rsi, %rax
-; CHECK-NEXT:    retq
+; CHECK-NOX-LABEL: fold_invariant_fence:
+; CHECK-NOX:       # %bb.0:
+; CHECK-NOX-NEXT:    movq (%rdi), %rax
+; CHECK-NOX-NEXT:    mfence
+; CHECK-NOX-NEXT:    addq %rsi, %rax
+; CHECK-NOX-NEXT:    retq
+;
+; CHECK-EX-LABEL: fold_invariant_fence:
+; CHECK-EX:       # %bb.0:
+; CHECK-EX-NEXT:    movq %rsi, %rax
+; CHECK-EX-NEXT:    mfence
+; CHECK-EX-NEXT:    addq (%rdi), %rax
+; CHECK-EX-NEXT:    retq
   %v = load atomic i64, i64* %p unordered, align 8, !invariant.load !{}
   fence seq_cst
   %ret = add i64 %v, %arg
   ret i64 %ret
 }