[mips] Fix atomic operations at O0, v3

[sdardis]

Summary

Similar to PR/25526, fast-regalloc introduces spills at the end of basic
blocks. When this occurs in between an ll and sc, the stores can cause the
atomic sequence to fail.

This patch fixes the issue by introducing more pseudos to represent atomic
operations and moving their lowering to after the expansion of postRA
pseudos.

This version addresses issues with the initial implementation and covers
all atomic operations.

This resolves PR/32020.

Thanks to James Cowgill for reporting the issue!

[slthakur]

LGTM

test/CodeGen/Mips/atomic.ll

20 ↗

(On Diff #92801)

Nit: Spelling should be "tests"

[sdardis]

I'm going to upload a new version of this diff, I found problems with the register allocation in the posted version.

[sdardis]

Updated the liveness tracking for new basic blocks produced by MipsExpandPseudo.
Reduced the number of basic blocks created by MipsISelLowering for atomic operations - this helps optimize the code operated on by FastRegAlloc, i.e. less spurious spills and reloads.
Updated the lowering of atomic operations not to produce spurious copies, the better liveness tracking handles values being live.
Updated the register flags used for atomic pseudos, my initial implementation marked the inputs as early clobber when it was the output that needed to be marked early clobber.

[sdardis]

Sagar, would you mind taking a second look at this?

Thanks,
Simon

[slthakur]

LGTM

[sdardis]

I have a new version of this patch which should hopefully address this atomics issue.

[sdardis]

Rebase to master, and address some of the register allocation issues.

[arichardson]

LGTM. Also looks like it should be easy to adjust our CHERI ll/sc instructions to use the same mechanism when I do the next upstream merge.

[smaksimovic]

Mostly comment fix suggestions

lib/Target/Mips/MipsExpandPseudo.cpp

218–219 ↗

(On Diff #147772)

Lowercase i here: hasMIps32r6()

487 ↗

(On Diff #147772)

BNE and MOVE seem to be unused in any of the BuildMI calls in this function.

491–494 ↗

(On Diff #147772)

STI->hasMips32r6()

lib/Target/Mips/MipsISelLowering.cpp

1483–1484 ↗

(On Diff #147772)

Put | Implicit before flags, registers -> register

1487 ↗

(On Diff #147772)

amoung -> among

1491 ↗

(On Diff #147772)

amoung -> among

1527–1528 ↗

(On Diff #147772)

In accordance with the comment above, shouldn't the scratch register have the accompanying dead flag as well?

1691 ↗

(On Diff #147772)

Change to unique among registers

1700–1704 ↗

(On Diff #147772)

Indentation

1760 ↗

(On Diff #147772)

Change to unique among registers

1768 ↗

(On Diff #147772)

Indentation

1809 ↗

(On Diff #147772)

is -> are, add Implicit to the list of flags too since it is used in the addReg() for Scratch and Scratch2 below.

1814 ↗

(On Diff #147772)

amoung -> among

1877 ↗

(On Diff #147772)

Either purposes -> purpose or is -> are

1879 ↗

(On Diff #147772)

Change to unique among registers

1890–1893 ↗

(On Diff #147772)

Indentation, RegState::EarlyClobber appears twice for Scratch.

test/CodeGen/Mips/atomic.ll

25 ↗

(On Diff #147772)

tets -> tests

[sdardis]

Address review comments.

[smaksimovic]

LGTM.

[asb]

lib/Target/Mips/MipsTargetMachine.cpp

288–290 ↗

(On Diff #149456)

Would it be more robust to have this pass under addPreEmit2? This guarantees it runs after potentially troublesome passes such as the machineoutliner (obviously not yet enabled for Mips anyway), and you'd no longer need to worry about machine block placement.

[sdardis]

lib/Target/Mips/MipsISelLowering.cpp

1700–1704 ↗

(On Diff #147772)

This is weird, but it is the output of clang-format.

1768 ↗

(On Diff #147772)

This is clang-formatted.

lib/Target/Mips/MipsTargetMachine.cpp

288–290 ↗

(On Diff #149456)

I hadn't considered that yet. Most of my concerns were about making sure the register allocator didn't break the scratch registers by making them non-unique with the address or data registers for the pseudo instruction.

[smaksimovic]

@sdardis I hadn't checked myself before posting but as you say that is indeed what clang-format outputs.

lib/Target/Mips/MipsISelLowering.cpp

1805 ↗

(On Diff #149456)

Implicit typo

[sdardis]

lib/Target/Mips/MipsTargetMachine.cpp

288–290 ↗

(On Diff #149456)

After looking at the machine outliner, I can see that it's possible to use the getMachineOutlinerMBBFlags function to detect ll/sc sequences and mark those entire blocks as illegal to outline*.

The choice of preEmit2 is more complex for MIPS as we would then need to perform the equivalent of MipsBranchExpansion at the MC layer as we would have invalidated its' assumptions, along with the delay slot filler and the size reduction pass.

PreEmit seems to be the right place.

• ISTM that if the registers used by all the occurrences of a single type of atomic operation were the same, then they could be merged into a single outlined function.

[sdardis]

Moved pass to after the scheduler and machine block placement.
Normalized the probabilities for the successors of modified basic blocks for sanity purposes.

[sdardis]

Actually include the expansion pass.

[jyknight]

lib/Target/Mips/MipsTargetMachine.cpp

288–290 ↗

(On Diff #149456)

The problem being that MipsBranchExpansion expects that every MI (by the time it runs) turns into 4 bytes of output? I think that just means these pseudos must be handled in getInstSizeInBytes(), right?

I think ideally the ll/sc loop should be treated as if it was effectively an inline-asm blob -- no mutations made by any other passes to the contents. So, e.g. if you want a delay slot filled, you place an instruction there yourself.

And, yes, it's perfectly reasonable to outline the LL/SC loop in its entirety, just not _parts_ of it.

But maybe this discussion should be continued on the llvm-dev email thread. Certainly moving to preEmit is closer to the ideal state than before. :)

llvm/trunk/lib/Target/Mips/CMakeLists.txt

add_llvm_target(MipsCodeGen
   Mips16ISelLowering.cpp
   Mips16RegisterInfo.cpp
   MipsAnalyzeImmediate.cpp
   MipsAsmPrinter.cpp
   MipsCallLowering.cpp
   MipsCCState.cpp
   MipsConstantIslandPass.cpp
   MipsDelaySlotFiller.cpp
+  MipsExpandPseudo.cpp
   MipsFastISel.cpp
   MipsInstrInfo.cpp
   MipsInstructionSelector.cpp
   MipsISelDAGToDAG.cpp
   MipsISelLowering.cpp
   MipsFrameLowering.cpp
   MipsLegalizerInfo.cpp
   MipsBranchExpansion.cpp

llvm/trunk/lib/Target/Mips/Mips.h

namespace llvm {
   ModulePass *createMips16HardFloatPass();
 
   FunctionPass *createMipsModuleISelDagPass();
   FunctionPass *createMipsOptimizePICCallPass();
   FunctionPass *createMipsDelaySlotFillerPass();
   FunctionPass *createMipsBranchExpansion();
   FunctionPass *createMipsConstantIslandPass();
   FunctionPass *createMicroMipsSizeReducePass();
+  FunctionPass *createMipsExpandPseudoPass();
 
   InstructionSelector *createMipsInstructionSelector(const MipsTargetMachine &,
                                                      MipsSubtarget &,
                                                      MipsRegisterBankInfo &);
 
   void initializeMipsDelaySlotFillerPass(PassRegistry &);
   void initializeMipsBranchExpansionPass(PassRegistry &);
   void initializeMicroMipsSizeReducePass(PassRegistry &);
 } // end namespace llvm;
 
 #endif

llvm/trunk/lib/Target/Mips/Mips64InstrInfo.td

let usesCustomInserter = 1 in {
   def ATOMIC_LOAD_AND_I64  : Atomic2Ops<atomic_load_and_64, GPR64>;
   def ATOMIC_LOAD_OR_I64   : Atomic2Ops<atomic_load_or_64, GPR64>;
   def ATOMIC_LOAD_XOR_I64  : Atomic2Ops<atomic_load_xor_64, GPR64>;
   def ATOMIC_LOAD_NAND_I64 : Atomic2Ops<atomic_load_nand_64, GPR64>;
   def ATOMIC_SWAP_I64      : Atomic2Ops<atomic_swap_64, GPR64>;
   def ATOMIC_CMP_SWAP_I64  : AtomicCmpSwap<atomic_cmp_swap_64, GPR64>;
 }
 
+def ATOMIC_LOAD_ADD_I64_POSTRA  : Atomic2OpsPostRA<GPR64>;
+def ATOMIC_LOAD_SUB_I64_POSTRA  : Atomic2OpsPostRA<GPR64>;
+def ATOMIC_LOAD_AND_I64_POSTRA  : Atomic2OpsPostRA<GPR64>;
+def ATOMIC_LOAD_OR_I64_POSTRA   : Atomic2OpsPostRA<GPR64>;
+def ATOMIC_LOAD_XOR_I64_POSTRA  : Atomic2OpsPostRA<GPR64>;
+def ATOMIC_LOAD_NAND_I64_POSTRA : Atomic2OpsPostRA<GPR64>;
+
+def ATOMIC_SWAP_I64_POSTRA      : Atomic2OpsPostRA<GPR64>;
+
+def ATOMIC_CMP_SWAP_I64_POSTRA  : AtomicCmpSwapPostRA<GPR64>;
+
 /// Pseudo instructions for loading and storing accumulator registers.
 let isPseudo = 1, isCodeGenOnly = 1, hasNoSchedulingInfo = 1 in {
   def LOAD_ACC128  : Load<"", ACC128>;
   def STORE_ACC128 : Store<"", ACC128>;
 }
 
 //===----------------------------------------------------------------------===//
 // Instruction definition

llvm/trunk/lib/Target/Mips/MipsExpandPseudo.cpp

+//===-- MipsExpandPseudoInsts.cpp - Expand pseudo instructions ------------===//
+//
+// The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains a pass that expands pseudo instructions into target
+// instructions to allow proper scheduling, if-conversion, and other late
+// optimizations. This pass should be run after register allocation but before
+// the post-regalloc scheduling pass.
+//
+// This is currently only used for expanding atomic pseudos after register
+// allocation. We do this to avoid the fast register allocator introducing
+// spills between ll and sc. These stores cause some MIPS implementations to
+// abort the atomic RMW sequence.
+//
+//===----------------------------------------------------------------------===//
+
+#include "Mips.h"
+#include "MipsInstrInfo.h"
+#include "MipsSubtarget.h"
+#include "llvm/CodeGen/LivePhysRegs.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineInstrBuilder.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "mips-pseudo"
+
+namespace {
+  class MipsExpandPseudo : public MachineFunctionPass {
+  public:
+    static char ID;
+    MipsExpandPseudo() : MachineFunctionPass(ID) {}
+
+    const MipsInstrInfo *TII;
+    const MipsSubtarget *STI;
+
+    bool runOnMachineFunction(MachineFunction &Fn) override;
+
+    MachineFunctionProperties getRequiredProperties() const override {
+      return MachineFunctionProperties().set(
+          MachineFunctionProperties::Property::NoVRegs);
+    }
+
+    StringRef getPassName() const override {
+      return "Mips pseudo instruction expansion pass";
+    }
+
+  private:
+    bool expandAtomicCmpSwap(MachineBasicBlock &MBB,
+                             MachineBasicBlock::iterator MBBI,
+                             MachineBasicBlock::iterator &NextMBBI);
+    bool expandAtomicCmpSwapSubword(MachineBasicBlock &MBB,
+                                    MachineBasicBlock::iterator MBBI,
+                                    MachineBasicBlock::iterator &NextMBBI);
+
+    bool expandAtomicBinOp(MachineBasicBlock &BB,
+                           MachineBasicBlock::iterator I,
+                           MachineBasicBlock::iterator &NMBBI, unsigned Size);
+    bool expandAtomicBinOpSubword(MachineBasicBlock &BB,
+                                  MachineBasicBlock::iterator I,
+                                  MachineBasicBlock::iterator &NMBBI);
+
+    bool expandMI(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
+                  MachineBasicBlock::iterator &NMBB);
+    bool expandMBB(MachineBasicBlock &MBB);
+   };
+  char MipsExpandPseudo::ID = 0;
+}
+
+bool MipsExpandPseudo::expandAtomicCmpSwapSubword(
+    MachineBasicBlock &BB, MachineBasicBlock::iterator I,
+    MachineBasicBlock::iterator &NMBBI) {
+
+  MachineFunction *MF = BB.getParent();
+
+  const bool ArePtrs64bit = STI->getABI().ArePtrs64bit();
+  DebugLoc DL = I->getDebugLoc();
+  unsigned LL, SC;
+
+  unsigned ZERO = Mips::ZERO;
+  unsigned BNE = Mips::BNE;
+  unsigned BEQ = Mips::BEQ;
+  unsigned SEOp =
+      I->getOpcode() == Mips::ATOMIC_CMP_SWAP_I8_POSTRA ? Mips::SEB : Mips::SEH;
+
+  if (STI->inMicroMipsMode()) {
+      LL = STI->hasMips32r6() ? Mips::LL_MMR6 : Mips::LL_MM;
+      SC = STI->hasMips32r6() ? Mips::SC_MMR6 : Mips::SC_MM;
+      BNE = STI->hasMips32r6() ? Mips::BNEC_MMR6 : Mips::BNE_MM;
+      BEQ = STI->hasMips32r6() ? Mips::BEQC_MMR6 : Mips::BEQ_MM;
+  } else {
+    LL = STI->hasMips32r6() ? (ArePtrs64bit ? Mips::LL64_R6 : Mips::LL_R6)
+                            : (ArePtrs64bit ? Mips::LL64 : Mips::LL);
+    SC = STI->hasMips32r6() ? (ArePtrs64bit ? Mips::SC64_R6 : Mips::SC_R6)
+                            : (ArePtrs64bit ? Mips::SC64 : Mips::SC);
+  }
+
+  unsigned Dest = I->getOperand(0).getReg();
+  unsigned Ptr = I->getOperand(1).getReg();
+  unsigned Mask = I->getOperand(2).getReg();
+  unsigned ShiftCmpVal = I->getOperand(3).getReg();
+  unsigned Mask2 = I->getOperand(4).getReg();
+  unsigned ShiftNewVal = I->getOperand(5).getReg();
+  unsigned ShiftAmnt = I->getOperand(6).getReg();
+  unsigned Scratch = I->getOperand(7).getReg();
+  unsigned Scratch2 = I->getOperand(8).getReg();
+
+  // insert new blocks after the current block
+  const BasicBlock *LLVM_BB = BB.getBasicBlock();
+  MachineBasicBlock *loop1MBB = MF->CreateMachineBasicBlock(LLVM_BB);
+  MachineBasicBlock *loop2MBB = MF->CreateMachineBasicBlock(LLVM_BB);
+  MachineBasicBlock *sinkMBB = MF->CreateMachineBasicBlock(LLVM_BB);
+  MachineBasicBlock *exitMBB = MF->CreateMachineBasicBlock(LLVM_BB);
+  MachineFunction::iterator It = ++BB.getIterator();
+  MF->insert(It, loop1MBB);
+  MF->insert(It, loop2MBB);
+  MF->insert(It, sinkMBB);
+  MF->insert(It, exitMBB);
+
+  // Transfer the remainder of BB and its successor edges to exitMBB.
+  exitMBB->splice(exitMBB->begin(), &BB,
+                  std::next(MachineBasicBlock::iterator(I)), BB.end());
+  exitMBB->transferSuccessorsAndUpdatePHIs(&BB);
+
+  //  thisMBB:
+  //    ...
+  //    fallthrough --> loop1MBB
+  BB.addSuccessor(loop1MBB, BranchProbability::getOne());
+  loop1MBB->addSuccessor(sinkMBB);
+  loop1MBB->addSuccessor(loop2MBB);
+  loop1MBB->normalizeSuccProbs();
+  loop2MBB->addSuccessor(loop1MBB);
+  loop2MBB->addSuccessor(sinkMBB);
+  loop2MBB->normalizeSuccProbs();
+  sinkMBB->addSuccessor(exitMBB, BranchProbability::getOne());
+
+  // loop1MBB:
+  //   ll dest, 0(ptr)
+  //   and Mask', dest, Mask
+  //   bne Mask', ShiftCmpVal, exitMBB
+  BuildMI(loop1MBB, DL, TII->get(LL), Scratch).addReg(Ptr).addImm(0);
+  BuildMI(loop1MBB, DL, TII->get(Mips::AND), Scratch2)
+      .addReg(Scratch)
+      .addReg(Mask);
+  BuildMI(loop1MBB, DL, TII->get(BNE))
+    .addReg(Scratch2).addReg(ShiftCmpVal).addMBB(sinkMBB);
+
+  // loop2MBB:
+  //   and dest, dest, mask2
+  //   or dest, dest, ShiftNewVal
+  //   sc dest, dest, 0(ptr)
+  //   beq dest, $0, loop1MBB
+  BuildMI(loop2MBB, DL, TII->get(Mips::AND), Scratch)
+      .addReg(Scratch, RegState::Kill)
+      .addReg(Mask2);
+  BuildMI(loop2MBB, DL, TII->get(Mips::OR), Scratch)
+      .addReg(Scratch, RegState::Kill)
+      .addReg(ShiftNewVal);
+  BuildMI(loop2MBB, DL, TII->get(SC), Scratch)
+      .addReg(Scratch, RegState::Kill)
+      .addReg(Ptr)
+      .addImm(0);
+  BuildMI(loop2MBB, DL, TII->get(BEQ))
+      .addReg(Scratch, RegState::Kill)
+      .addReg(ZERO)
+      .addMBB(loop1MBB);
+
+  //  sinkMBB:
+  //    srl     srlres, Mask', shiftamt
+  //    sign_extend dest,srlres
+  BuildMI(sinkMBB, DL, TII->get(Mips::SRLV), Dest)
+      .addReg(Scratch2)
+      .addReg(ShiftAmnt);
+  if (STI->hasMips32r2()) {
+    BuildMI(sinkMBB, DL, TII->get(SEOp), Dest).addReg(Dest);
+  } else {
+    const unsigned ShiftImm =
+        I->getOpcode() == Mips::ATOMIC_CMP_SWAP_I16_POSTRA ? 16 : 24;
+    BuildMI(sinkMBB, DL, TII->get(Mips::SLL), Dest)
+        .addReg(Dest, RegState::Kill)
+        .addImm(ShiftImm);
+    BuildMI(sinkMBB, DL, TII->get(Mips::SRA), Dest)
+        .addReg(Dest, RegState::Kill)
+        .addImm(ShiftImm);
+  }
+
+  LivePhysRegs LiveRegs;
+  computeAndAddLiveIns(LiveRegs, *loop1MBB);
+  computeAndAddLiveIns(LiveRegs, *loop2MBB);
+  computeAndAddLiveIns(LiveRegs, *sinkMBB);
+  computeAndAddLiveIns(LiveRegs, *exitMBB);
+
+  NMBBI = BB.end();
+  I->eraseFromParent();
+  return true;
+}
+
+bool MipsExpandPseudo::expandAtomicCmpSwap(MachineBasicBlock &BB,
+                                           MachineBasicBlock::iterator I,
+                                           MachineBasicBlock::iterator &NMBBI) {
+
+  const unsigned Size =
+      I->getOpcode() == Mips::ATOMIC_CMP_SWAP_I32_POSTRA ? 4 : 8;
+  MachineFunction *MF = BB.getParent();
+
+  const bool ArePtrs64bit = STI->getABI().ArePtrs64bit();
+  DebugLoc DL = I->getDebugLoc();
+
+  unsigned LL, SC, ZERO, BNE, BEQ, MOVE;
+
+  if (Size == 4) {
+    if (STI->inMicroMipsMode()) {
+      LL = STI->hasMips32r6() ? Mips::LL_MMR6 : Mips::LL_MM;
+      SC = STI->hasMips32r6() ? Mips::SC_MMR6 : Mips::SC_MM;
+      BNE = STI->hasMips32r6() ? Mips::BNEC_MMR6 : Mips::BNE_MM;
+      BEQ = STI->hasMips32r6() ? Mips::BEQC_MMR6 : Mips::BEQ_MM;
+    } else {
+      LL = STI->hasMips32r6()
+               ? (ArePtrs64bit ? Mips::LL64_R6 : Mips::LL_R6)
+               : (ArePtrs64bit ? Mips::LL64 : Mips::LL);
+      SC = STI->hasMips32r6()
+               ? (ArePtrs64bit ? Mips::SC64_R6 : Mips::SC_R6)
+               : (ArePtrs64bit ? Mips::SC64 : Mips::SC);
+      BNE = Mips::BNE;
+      BEQ = Mips::BEQ;
+    }
+
+    ZERO = Mips::ZERO;
+    MOVE = Mips::OR;
+  } else {
+    LL = STI->hasMips64r6() ? Mips::LLD_R6 : Mips::LLD;
+    SC = STI->hasMips64r6() ? Mips::SCD_R6 : Mips::SCD;
+    ZERO = Mips::ZERO_64;
+    BNE = Mips::BNE64;
+    BEQ = Mips::BEQ64;
+    MOVE = Mips::OR64;
+  }
+
+  unsigned Dest = I->getOperand(0).getReg();
+  unsigned Ptr = I->getOperand(1).getReg();
+  unsigned OldVal = I->getOperand(2).getReg();
+  unsigned NewVal = I->getOperand(3).getReg();
+  unsigned Scratch = I->getOperand(4).getReg();
+
+  // insert new blocks after the current block
+  const BasicBlock *LLVM_BB = BB.getBasicBlock();
+  MachineBasicBlock *loop1MBB = MF->CreateMachineBasicBlock(LLVM_BB);
+  MachineBasicBlock *loop2MBB = MF->CreateMachineBasicBlock(LLVM_BB);
+  MachineBasicBlock *exitMBB = MF->CreateMachineBasicBlock(LLVM_BB);
+  MachineFunction::iterator It = ++BB.getIterator();
+  MF->insert(It, loop1MBB);
+  MF->insert(It, loop2MBB);
+  MF->insert(It, exitMBB);
+
+  // Transfer the remainder of BB and its successor edges to exitMBB.
+  exitMBB->splice(exitMBB->begin(), &BB,
+                  std::next(MachineBasicBlock::iterator(I)), BB.end());
+  exitMBB->transferSuccessorsAndUpdatePHIs(&BB);
+
+  //  thisMBB:
+  //    ...
+  //    fallthrough --> loop1MBB
+  BB.addSuccessor(loop1MBB, BranchProbability::getOne());
+  loop1MBB->addSuccessor(exitMBB);
+  loop1MBB->addSuccessor(loop2MBB);
+  loop1MBB->normalizeSuccProbs();
+  loop2MBB->addSuccessor(loop1MBB);
+  loop2MBB->addSuccessor(exitMBB);
+  loop2MBB->normalizeSuccProbs();
+
+  // loop1MBB:
+  //   ll dest, 0(ptr)
+  //   bne dest, oldval, exitMBB
+  BuildMI(loop1MBB, DL, TII->get(LL), Dest).addReg(Ptr).addImm(0);
+  BuildMI(loop1MBB, DL, TII->get(BNE))
+    .addReg(Dest, RegState::Kill).addReg(OldVal).addMBB(exitMBB);
+
+  // loop2MBB:
+  //   move scratch, NewVal
+  //   sc Scratch, Scratch, 0(ptr)
+  //   beq Scratch, $0, loop1MBB
+  BuildMI(loop2MBB, DL, TII->get(MOVE), Scratch).addReg(NewVal).addReg(ZERO);
+  BuildMI(loop2MBB, DL, TII->get(SC), Scratch)
+    .addReg(Scratch).addReg(Ptr).addImm(0);
+  BuildMI(loop2MBB, DL, TII->get(BEQ))
+    .addReg(Scratch, RegState::Kill).addReg(ZERO).addMBB(loop1MBB);
+
+  LivePhysRegs LiveRegs;
+  computeAndAddLiveIns(LiveRegs, *loop1MBB);
+  computeAndAddLiveIns(LiveRegs, *loop2MBB);
+  computeAndAddLiveIns(LiveRegs, *exitMBB);
+
+  NMBBI = BB.end();
+  I->eraseFromParent();
+  return true;
+}
+
+bool MipsExpandPseudo::expandAtomicBinOpSubword(
+    MachineBasicBlock &BB, MachineBasicBlock::iterator I,
+    MachineBasicBlock::iterator &NMBBI) {
+
+  MachineFunction *MF = BB.getParent();
+
+  const bool ArePtrs64bit = STI->getABI().ArePtrs64bit();
+  DebugLoc DL = I->getDebugLoc();
+
+  unsigned LL, SC;
+  unsigned BEQ = Mips::BEQ;
+  unsigned SEOp = Mips::SEH;
+
+  if (STI->inMicroMipsMode()) {
+      LL = STI->hasMips32r6() ? Mips::LL_MMR6 : Mips::LL_MM;
+      SC = STI->hasMips32r6() ? Mips::SC_MMR6 : Mips::SC_MM;
+      BEQ = STI->hasMips32r6() ? Mips::BEQC_MMR6 : Mips::BEQ_MM;
+  } else {
+    LL = STI->hasMips32r6() ? (ArePtrs64bit ? Mips::LL64_R6 : Mips::LL_R6)
+                            : (ArePtrs64bit ? Mips::LL64 : Mips::LL);
+    SC = STI->hasMips32r6() ? (ArePtrs64bit ? Mips::SC64_R6 : Mips::SC_R6)
+                            : (ArePtrs64bit ? Mips::SC64 : Mips::SC);
+  }
+
+  bool IsSwap = false;
+  bool IsNand = false;
+
+  unsigned Opcode = 0;
+  switch (I->getOpcode()) {
+  case Mips::ATOMIC_LOAD_NAND_I8_POSTRA:
+    SEOp = Mips::SEB;
+    LLVM_FALLTHROUGH;
+  case Mips::ATOMIC_LOAD_NAND_I16_POSTRA:
+    IsNand = true;
+    break;
+  case Mips::ATOMIC_SWAP_I8_POSTRA:
+    SEOp = Mips::SEB;
+    LLVM_FALLTHROUGH;
+  case Mips::ATOMIC_SWAP_I16_POSTRA:
+    IsSwap = true;
+    break;
+  case Mips::ATOMIC_LOAD_ADD_I8_POSTRA:
+    SEOp = Mips::SEB;
+    LLVM_FALLTHROUGH;
+  case Mips::ATOMIC_LOAD_ADD_I16_POSTRA:
+    Opcode = Mips::ADDu;
+    break;
+  case Mips::ATOMIC_LOAD_SUB_I8_POSTRA:
+    SEOp = Mips::SEB;
+    LLVM_FALLTHROUGH;
+  case Mips::ATOMIC_LOAD_SUB_I16_POSTRA:
+    Opcode = Mips::SUBu;
+    break;
+  case Mips::ATOMIC_LOAD_AND_I8_POSTRA:
+    SEOp = Mips::SEB;
+    LLVM_FALLTHROUGH;
+  case Mips::ATOMIC_LOAD_AND_I16_POSTRA:
+    Opcode = Mips::AND;
+    break;
+  case Mips::ATOMIC_LOAD_OR_I8_POSTRA:
+    SEOp = Mips::SEB;
+    LLVM_FALLTHROUGH;
+  case Mips::ATOMIC_LOAD_OR_I16_POSTRA:
+    Opcode = Mips::OR;
+    break;
+  case Mips::ATOMIC_LOAD_XOR_I8_POSTRA:
+    SEOp = Mips::SEB;
+    LLVM_FALLTHROUGH;
+  case Mips::ATOMIC_LOAD_XOR_I16_POSTRA:
+    Opcode = Mips::XOR;
+    break;
+  default:
+    llvm_unreachable("Unknown subword atomic pseudo for expansion!");
+  }
+
+  unsigned Dest = I->getOperand(0).getReg();
+  unsigned Ptr = I->getOperand(1).getReg();
+  unsigned Incr = I->getOperand(2).getReg();
+  unsigned Mask = I->getOperand(3).getReg();
+  unsigned Mask2 = I->getOperand(4).getReg();
+  unsigned ShiftAmnt = I->getOperand(5).getReg();
+  unsigned OldVal = I->getOperand(6).getReg();
+  unsigned BinOpRes = I->getOperand(7).getReg();
+  unsigned StoreVal = I->getOperand(8).getReg();
+
+  const BasicBlock *LLVM_BB = BB.getBasicBlock();
+  MachineBasicBlock *loopMBB = MF->CreateMachineBasicBlock(LLVM_BB);
+  MachineBasicBlock *sinkMBB = MF->CreateMachineBasicBlock(LLVM_BB);
+  MachineBasicBlock *exitMBB = MF->CreateMachineBasicBlock(LLVM_BB);
+  MachineFunction::iterator It = ++BB.getIterator();
+  MF->insert(It, loopMBB);
+  MF->insert(It, sinkMBB);
+  MF->insert(It, exitMBB);
+
+  exitMBB->splice(exitMBB->begin(), &BB, std::next(I), BB.end());
+  exitMBB->transferSuccessorsAndUpdatePHIs(&BB);
+
+  BB.addSuccessor(loopMBB, BranchProbability::getOne());
+  loopMBB->addSuccessor(sinkMBB);
+  loopMBB->addSuccessor(loopMBB);
+  loopMBB->normalizeSuccProbs();
+
+  BuildMI(loopMBB, DL, TII->get(LL), OldVal).addReg(Ptr).addImm(0);
+  if (IsNand) {
+    //  and andres, oldval, incr2
+    //  nor binopres, $0, andres
+    //  and newval, binopres, mask
+    BuildMI(loopMBB, DL, TII->get(Mips::AND), BinOpRes)
+        .addReg(OldVal)
+        .addReg(Incr);
+    BuildMI(loopMBB, DL, TII->get(Mips::NOR), BinOpRes)
+        .addReg(Mips::ZERO)
+        .addReg(BinOpRes);
+    BuildMI(loopMBB, DL, TII->get(Mips::AND), BinOpRes)
+        .addReg(BinOpRes)
+        .addReg(Mask);
+  } else if (!IsSwap) {
+    //  <binop> binopres, oldval, incr2
+    //  and newval, binopres, mask
+    BuildMI(loopMBB, DL, TII->get(Opcode), BinOpRes)
+        .addReg(OldVal)
+        .addReg(Incr);
+    BuildMI(loopMBB, DL, TII->get(Mips::AND), BinOpRes)
+        .addReg(BinOpRes)
+        .addReg(Mask);
+  } else { // atomic.swap
+    //  and newval, incr2, mask
+    BuildMI(loopMBB, DL, TII->get(Mips::AND), BinOpRes)
+        .addReg(Incr)
+        .addReg(Mask);
+  }
+
+  // and StoreVal, OlddVal, Mask2
+  // or StoreVal, StoreVal, BinOpRes
+  // StoreVal<tied1> = sc StoreVal, 0(Ptr)
+  // beq StoreVal, zero, loopMBB
+  BuildMI(loopMBB, DL, TII->get(Mips::AND), StoreVal)
+    .addReg(OldVal).addReg(Mask2);
+  BuildMI(loopMBB, DL, TII->get(Mips::OR), StoreVal)
+    .addReg(StoreVal).addReg(BinOpRes);
+  BuildMI(loopMBB, DL, TII->get(SC), StoreVal)
+    .addReg(StoreVal).addReg(Ptr).addImm(0);
+  BuildMI(loopMBB, DL, TII->get(BEQ))
+    .addReg(StoreVal).addReg(Mips::ZERO).addMBB(loopMBB);
+
+  //  sinkMBB:
+  //    and     maskedoldval1,oldval,mask
+  //    srl     srlres,maskedoldval1,shiftamt
+  //    sign_extend dest,srlres
+
+  sinkMBB->addSuccessor(exitMBB, BranchProbability::getOne());
+
+  BuildMI(sinkMBB, DL, TII->get(Mips::AND), Dest)
+    .addReg(OldVal).addReg(Mask);
+  BuildMI(sinkMBB, DL, TII->get(Mips::SRLV), Dest)
+      .addReg(Dest).addReg(ShiftAmnt);
+
+  if (STI->hasMips32r2()) {
+    BuildMI(sinkMBB, DL, TII->get(SEOp), Dest).addReg(Dest);
+  } else {
+    const unsigned ShiftImm = SEOp == Mips::SEH ? 16 : 24;
+    BuildMI(sinkMBB, DL, TII->get(Mips::SLL), Dest)
+        .addReg(Dest, RegState::Kill)
+        .addImm(ShiftImm);
+    BuildMI(sinkMBB, DL, TII->get(Mips::SRA), Dest)
+        .addReg(Dest, RegState::Kill)
+        .addImm(ShiftImm);
+  }
+
+  LivePhysRegs LiveRegs;
+  computeAndAddLiveIns(LiveRegs, *loopMBB);
+  computeAndAddLiveIns(LiveRegs, *sinkMBB);
+  computeAndAddLiveIns(LiveRegs, *exitMBB);
+
+  NMBBI = BB.end();
+  I->eraseFromParent();
+
+  return true;
+}
+
+bool MipsExpandPseudo::expandAtomicBinOp(MachineBasicBlock &BB,
+                                         MachineBasicBlock::iterator I,
+                                         MachineBasicBlock::iterator &NMBBI,
+                                         unsigned Size) {
+  MachineFunction *MF = BB.getParent();
+
+  const bool ArePtrs64bit = STI->getABI().ArePtrs64bit();
+  DebugLoc DL = I->getDebugLoc();
+
+  unsigned LL, SC, ZERO, BEQ;
+
+  if (Size == 4) {
+    if (STI->inMicroMipsMode()) {
+      LL = STI->hasMips32r6() ? Mips::LL_MMR6 : Mips::LL_MM;
+      SC = STI->hasMips32r6() ? Mips::SC_MMR6 : Mips::SC_MM;
+      BEQ = STI->hasMips32r6() ? Mips::BEQC_MMR6 : Mips::BEQ_MM;
+    } else {
+      LL = STI->hasMips32r6()
+               ? (ArePtrs64bit ? Mips::LL64_R6 : Mips::LL_R6)
+               : (ArePtrs64bit ? Mips::LL64 : Mips::LL);
+      SC = STI->hasMips32r6()
+               ? (ArePtrs64bit ? Mips::SC64_R6 : Mips::SC_R6)
+               : (ArePtrs64bit ? Mips::SC64 : Mips::SC);
+      BEQ = Mips::BEQ;
+    }
+
+    ZERO = Mips::ZERO;
+  } else {
+    LL = STI->hasMips64r6() ? Mips::LLD_R6 : Mips::LLD;
+    SC = STI->hasMips64r6() ? Mips::SCD_R6 : Mips::SCD;
+    ZERO = Mips::ZERO_64;
+    BEQ = Mips::BEQ64;
+  }
+
+  unsigned OldVal = I->getOperand(0).getReg();
+  unsigned Ptr = I->getOperand(1).getReg();
+  unsigned Incr = I->getOperand(2).getReg();
+  unsigned Scratch = I->getOperand(3).getReg();
+
+  unsigned Opcode = 0;
+  unsigned OR = 0;
+  unsigned AND = 0;
+  unsigned NOR = 0;
+  bool IsNand = false;
+  switch (I->getOpcode()) {
+  case Mips::ATOMIC_LOAD_ADD_I32_POSTRA:
+    Opcode = Mips::ADDu;
+    break;
+  case Mips::ATOMIC_LOAD_SUB_I32_POSTRA:
+    Opcode = Mips::SUBu;
+    break;
+  case Mips::ATOMIC_LOAD_AND_I32_POSTRA:
+    Opcode = Mips::AND;
+    break;
+  case Mips::ATOMIC_LOAD_OR_I32_POSTRA:
+    Opcode = Mips::OR;
+    break;
+  case Mips::ATOMIC_LOAD_XOR_I32_POSTRA:
+    Opcode = Mips::XOR;
+    break;
+  case Mips::ATOMIC_LOAD_NAND_I32_POSTRA:
+    IsNand = true;
+    AND = Mips::AND;
+    NOR = Mips::NOR;
+    break;
+  case Mips::ATOMIC_SWAP_I32_POSTRA:
+    OR = Mips::OR;
+    break;
+  case Mips::ATOMIC_LOAD_ADD_I64_POSTRA:
+    Opcode = Mips::DADDu;
+    break;
+  case Mips::ATOMIC_LOAD_SUB_I64_POSTRA:
+    Opcode = Mips::DSUBu;
+    break;
+  case Mips::ATOMIC_LOAD_AND_I64_POSTRA:
+    Opcode = Mips::AND64;
+    break;
+  case Mips::ATOMIC_LOAD_OR_I64_POSTRA:
+    Opcode = Mips::OR64;
+    break;
+  case Mips::ATOMIC_LOAD_XOR_I64_POSTRA:
+    Opcode = Mips::XOR64;
+    break;
+  case Mips::ATOMIC_LOAD_NAND_I64_POSTRA:
+    IsNand = true;
+    AND = Mips::AND64;
+    NOR = Mips::NOR64;
+    break;
+  case Mips::ATOMIC_SWAP_I64_POSTRA:
+    OR = Mips::OR64;
+    break;
+  default:
+    llvm_unreachable("Unknown pseudo atomic!");
+  }
+
+  const BasicBlock *LLVM_BB = BB.getBasicBlock();
+  MachineBasicBlock *loopMBB = MF->CreateMachineBasicBlock(LLVM_BB);
+  MachineBasicBlock *exitMBB = MF->CreateMachineBasicBlock(LLVM_BB);
+  MachineFunction::iterator It = ++BB.getIterator();
+  MF->insert(It, loopMBB);
+  MF->insert(It, exitMBB);
+
+  exitMBB->splice(exitMBB->begin(), &BB, std::next(I), BB.end());
+  exitMBB->transferSuccessorsAndUpdatePHIs(&BB);
+
+  BB.addSuccessor(loopMBB, BranchProbability::getOne());
+  loopMBB->addSuccessor(exitMBB);
+  loopMBB->addSuccessor(loopMBB);
+  loopMBB->normalizeSuccProbs();
+
+  BuildMI(loopMBB, DL, TII->get(LL), OldVal).addReg(Ptr).addImm(0);
+  assert((OldVal != Ptr) && "Clobbered the wrong ptr reg!");
+  assert((OldVal != Incr) && "Clobbered the wrong reg!");
+  if (Opcode) {
+    BuildMI(loopMBB, DL, TII->get(Opcode), Scratch).addReg(OldVal).addReg(Incr);
+  } else if (IsNand) {
+    assert(AND && NOR &&
+           "Unknown nand instruction for atomic pseudo expansion");
+    BuildMI(loopMBB, DL, TII->get(AND), Scratch).addReg(OldVal).addReg(Incr);
+    BuildMI(loopMBB, DL, TII->get(NOR), Scratch).addReg(ZERO).addReg(Scratch);
+  } else {
+    assert(OR && "Unknown instruction for atomic pseudo expansion!");
+    BuildMI(loopMBB, DL, TII->get(OR), Scratch).addReg(Incr).addReg(ZERO);
+  }
+
+  BuildMI(loopMBB, DL, TII->get(SC), Scratch).addReg(Scratch).addReg(Ptr).addImm(0);
+  BuildMI(loopMBB, DL, TII->get(BEQ)).addReg(Scratch).addReg(ZERO).addMBB(loopMBB);
+
+  NMBBI = BB.end();
+  I->eraseFromParent();
+
+  LivePhysRegs LiveRegs;
+  computeAndAddLiveIns(LiveRegs, *loopMBB);
+  computeAndAddLiveIns(LiveRegs, *exitMBB);
+
+  return true;
+}
+
+bool MipsExpandPseudo::expandMI(MachineBasicBlock &MBB,
+                                MachineBasicBlock::iterator MBBI,
+                                MachineBasicBlock::iterator &NMBB) {
+
+  bool Modified = false;
+
+  switch (MBBI->getOpcode()) {
+  case Mips::ATOMIC_CMP_SWAP_I32_POSTRA:
+  case Mips::ATOMIC_CMP_SWAP_I64_POSTRA:
+    return expandAtomicCmpSwap(MBB, MBBI, NMBB);
+  case Mips::ATOMIC_CMP_SWAP_I8_POSTRA:
+  case Mips::ATOMIC_CMP_SWAP_I16_POSTRA:
+    return expandAtomicCmpSwapSubword(MBB, MBBI, NMBB);
+  case Mips::ATOMIC_SWAP_I8_POSTRA:
+  case Mips::ATOMIC_SWAP_I16_POSTRA:
+  case Mips::ATOMIC_LOAD_NAND_I8_POSTRA:
+  case Mips::ATOMIC_LOAD_NAND_I16_POSTRA:
+  case Mips::ATOMIC_LOAD_ADD_I8_POSTRA:
+  case Mips::ATOMIC_LOAD_ADD_I16_POSTRA:
+  case Mips::ATOMIC_LOAD_SUB_I8_POSTRA:
+  case Mips::ATOMIC_LOAD_SUB_I16_POSTRA:
+  case Mips::ATOMIC_LOAD_AND_I8_POSTRA:
+  case Mips::ATOMIC_LOAD_AND_I16_POSTRA:
+  case Mips::ATOMIC_LOAD_OR_I8_POSTRA:
+  case Mips::ATOMIC_LOAD_OR_I16_POSTRA:
+  case Mips::ATOMIC_LOAD_XOR_I8_POSTRA:
+  case Mips::ATOMIC_LOAD_XOR_I16_POSTRA:
+    return expandAtomicBinOpSubword(MBB, MBBI, NMBB);
+  case Mips::ATOMIC_LOAD_ADD_I32_POSTRA:
+  case Mips::ATOMIC_LOAD_SUB_I32_POSTRA:
+  case Mips::ATOMIC_LOAD_AND_I32_POSTRA:
+  case Mips::ATOMIC_LOAD_OR_I32_POSTRA:
+  case Mips::ATOMIC_LOAD_XOR_I32_POSTRA:
+  case Mips::ATOMIC_LOAD_NAND_I32_POSTRA:
+  case Mips::ATOMIC_SWAP_I32_POSTRA:
+    return expandAtomicBinOp(MBB, MBBI, NMBB, 4);
+  case Mips::ATOMIC_LOAD_ADD_I64_POSTRA:
+  case Mips::ATOMIC_LOAD_SUB_I64_POSTRA:
+  case Mips::ATOMIC_LOAD_AND_I64_POSTRA:
+  case Mips::ATOMIC_LOAD_OR_I64_POSTRA:
+  case Mips::ATOMIC_LOAD_XOR_I64_POSTRA:
+  case Mips::ATOMIC_LOAD_NAND_I64_POSTRA:
+  case Mips::ATOMIC_SWAP_I64_POSTRA:
+    return expandAtomicBinOp(MBB, MBBI, NMBB, 8);
+  default:
+    return Modified;
+  }
+}
+
+bool MipsExpandPseudo::expandMBB(MachineBasicBlock &MBB) {
+  bool Modified = false;
+
+  MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
+  while (MBBI != E) {
+    MachineBasicBlock::iterator NMBBI = std::next(MBBI);
+    Modified |= expandMI(MBB, MBBI, NMBBI);
+    MBBI = NMBBI;
+  }
+
+  return Modified;
+}
+
+bool MipsExpandPseudo::runOnMachineFunction(MachineFunction &MF) {
+  STI = &static_cast<const MipsSubtarget &>(MF.getSubtarget());
+  TII = STI->getInstrInfo();
+
+  bool Modified = false;
+  for (MachineFunction::iterator MFI = MF.begin(), E = MF.end(); MFI != E;
+       ++MFI)
+    Modified |= expandMBB(*MFI);
+
+  if (Modified)
+    MF.RenumberBlocks();
+
+  return Modified;
+}
+
+/// createMipsExpandPseudoPass - returns an instance of the pseudo instruction
+/// expansion pass.
+FunctionPass *llvm::createMipsExpandPseudoPass() {
+  return new MipsExpandPseudo();
+}

llvm/trunk/lib/Target/Mips/MipsISelLowering.h

private:
     }
 
     /// Emit a sign-extension using sll/sra, seb, or seh appropriately.
     MachineBasicBlock *emitSignExtendToI32InReg(MachineInstr &MI,
                                                 MachineBasicBlock *BB,
                                                 unsigned Size, unsigned DstReg,
                                                 unsigned SrcRec) const;
 
-    MachineBasicBlock *emitAtomicBinary(MachineInstr &MI, MachineBasicBlock *BB,
-                                        unsigned Size, unsigned BinOpcode,
-                                        bool Nand = false) const;
+    MachineBasicBlock *emitAtomicBinary(MachineInstr &MI,
+                                        MachineBasicBlock *BB) const;
     MachineBasicBlock *emitAtomicBinaryPartword(MachineInstr &MI,
                                                 MachineBasicBlock *BB,
-                                                unsigned Size,
-                                                unsigned BinOpcode,
-                                                bool Nand = false) const;
-    MachineBasicBlock *emitAtomicCmpSwap(MachineInstr &MI,
-                                         MachineBasicBlock *BB,
                                                 unsigned Size) const;
+    MachineBasicBlock *emitAtomicCmpSwap(MachineInstr &MI,
+                                         MachineBasicBlock *BB) const;
     MachineBasicBlock *emitAtomicCmpSwapPartword(MachineInstr &MI,
                                                  MachineBasicBlock *BB,
                                                  unsigned Size) const;
     MachineBasicBlock *emitSEL_D(MachineInstr &MI, MachineBasicBlock *BB) const;
     MachineBasicBlock *emitPseudoSELECT(MachineInstr &MI, MachineBasicBlock *BB,
                                         bool isFPCmp, unsigned Opc) const;
   };
 

llvm/trunk/lib/Target/Mips/MipsISelLowering.cpp

 
 MachineBasicBlock *
 MipsTargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
                                                 MachineBasicBlock *BB) const {
   switch (MI.getOpcode()) {
   default:
     llvm_unreachable("Unexpected instr type to insert");
   case Mips::ATOMIC_LOAD_ADD_I8:
-    return emitAtomicBinaryPartword(MI, BB, 1, Mips::ADDu);
+    return emitAtomicBinaryPartword(MI, BB, 1);
   case Mips::ATOMIC_LOAD_ADD_I16:
-    return emitAtomicBinaryPartword(MI, BB, 2, Mips::ADDu);
+    return emitAtomicBinaryPartword(MI, BB, 2);
   case Mips::ATOMIC_LOAD_ADD_I32:
-    return emitAtomicBinary(MI, BB, 4, Mips::ADDu);
+    return emitAtomicBinary(MI, BB);
   case Mips::ATOMIC_LOAD_ADD_I64:
-    return emitAtomicBinary(MI, BB, 8, Mips::DADDu);
+    return emitAtomicBinary(MI, BB);
 
   case Mips::ATOMIC_LOAD_AND_I8:
-    return emitAtomicBinaryPartword(MI, BB, 1, Mips::AND);
+    return emitAtomicBinaryPartword(MI, BB, 1);
   case Mips::ATOMIC_LOAD_AND_I16:
-    return emitAtomicBinaryPartword(MI, BB, 2, Mips::AND);
+    return emitAtomicBinaryPartword(MI, BB, 2);
   case Mips::ATOMIC_LOAD_AND_I32:
-    return emitAtomicBinary(MI, BB, 4, Mips::AND);
+    return emitAtomicBinary(MI, BB);
   case Mips::ATOMIC_LOAD_AND_I64:
-    return emitAtomicBinary(MI, BB, 8, Mips::AND64);
+    return emitAtomicBinary(MI, BB);
 
   case Mips::ATOMIC_LOAD_OR_I8:
-    return emitAtomicBinaryPartword(MI, BB, 1, Mips::OR);
+    return emitAtomicBinaryPartword(MI, BB, 1);
   case Mips::ATOMIC_LOAD_OR_I16:
-    return emitAtomicBinaryPartword(MI, BB, 2, Mips::OR);
+    return emitAtomicBinaryPartword(MI, BB, 2);
   case Mips::ATOMIC_LOAD_OR_I32:
-    return emitAtomicBinary(MI, BB, 4, Mips::OR);
+    return emitAtomicBinary(MI, BB);
   case Mips::ATOMIC_LOAD_OR_I64:
-    return emitAtomicBinary(MI, BB, 8, Mips::OR64);
+    return emitAtomicBinary(MI, BB);
 
   case Mips::ATOMIC_LOAD_XOR_I8:
-    return emitAtomicBinaryPartword(MI, BB, 1, Mips::XOR);
+    return emitAtomicBinaryPartword(MI, BB, 1);
   case Mips::ATOMIC_LOAD_XOR_I16:
-    return emitAtomicBinaryPartword(MI, BB, 2, Mips::XOR);
+    return emitAtomicBinaryPartword(MI, BB, 2);
   case Mips::ATOMIC_LOAD_XOR_I32:
-    return emitAtomicBinary(MI, BB, 4, Mips::XOR);
+    return emitAtomicBinary(MI, BB);
   case Mips::ATOMIC_LOAD_XOR_I64:
-    return emitAtomicBinary(MI, BB, 8, Mips::XOR64);
+    return emitAtomicBinary(MI, BB);
 
   case Mips::ATOMIC_LOAD_NAND_I8:
-    return emitAtomicBinaryPartword(MI, BB, 1, 0, true);
+    return emitAtomicBinaryPartword(MI, BB, 1);
   case Mips::ATOMIC_LOAD_NAND_I16:
-    return emitAtomicBinaryPartword(MI, BB, 2, 0, true);
+    return emitAtomicBinaryPartword(MI, BB, 2);
   case Mips::ATOMIC_LOAD_NAND_I32:
-    return emitAtomicBinary(MI, BB, 4, 0, true);
+    return emitAtomicBinary(MI, BB);
   case Mips::ATOMIC_LOAD_NAND_I64:
-    return emitAtomicBinary(MI, BB, 8, 0, true);
+    return emitAtomicBinary(MI, BB);
 
   case Mips::ATOMIC_LOAD_SUB_I8:
-    return emitAtomicBinaryPartword(MI, BB, 1, Mips::SUBu);
+    return emitAtomicBinaryPartword(MI, BB, 1);
   case Mips::ATOMIC_LOAD_SUB_I16:
-    return emitAtomicBinaryPartword(MI, BB, 2, Mips::SUBu);
+    return emitAtomicBinaryPartword(MI, BB, 2);
   case Mips::ATOMIC_LOAD_SUB_I32:
-    return emitAtomicBinary(MI, BB, 4, Mips::SUBu);
+    return emitAtomicBinary(MI, BB);
   case Mips::ATOMIC_LOAD_SUB_I64:
-    return emitAtomicBinary(MI, BB, 8, Mips::DSUBu);
+    return emitAtomicBinary(MI, BB);
 
   case Mips::ATOMIC_SWAP_I8:
-    return emitAtomicBinaryPartword(MI, BB, 1, 0);
+    return emitAtomicBinaryPartword(MI, BB, 1);
   case Mips::ATOMIC_SWAP_I16:
-    return emitAtomicBinaryPartword(MI, BB, 2, 0);
+    return emitAtomicBinaryPartword(MI, BB, 2);
   case Mips::ATOMIC_SWAP_I32:
-    return emitAtomicBinary(MI, BB, 4, 0);
+    return emitAtomicBinary(MI, BB);
   case Mips::ATOMIC_SWAP_I64:
-    return emitAtomicBinary(MI, BB, 8, 0);
+    return emitAtomicBinary(MI, BB);
 
   case Mips::ATOMIC_CMP_SWAP_I8:
     return emitAtomicCmpSwapPartword(MI, BB, 1);
   case Mips::ATOMIC_CMP_SWAP_I16:
     return emitAtomicCmpSwapPartword(MI, BB, 2);
   case Mips::ATOMIC_CMP_SWAP_I32:
-    return emitAtomicCmpSwap(MI, BB, 4);
+    return emitAtomicCmpSwap(MI, BB);
   case Mips::ATOMIC_CMP_SWAP_I64:
-    return emitAtomicCmpSwap(MI, BB, 8);
+    return emitAtomicCmpSwap(MI, BB);
   case Mips::PseudoSDIV:
   case Mips::PseudoUDIV:
   case Mips::DIV:
   case Mips::DIVU:
   case Mips::MOD:
   case Mips::MODU:
     return insertDivByZeroTrap(MI, *BB, *Subtarget.getInstrInfo(), false,
                                false);
MipsTargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
   case Mips::PseudoSELECTFP_T_D32:
   case Mips::PseudoSELECTFP_T_D64:
     return emitPseudoSELECT(MI, BB, true, Mips::BC1T);
   }
 }
 
 // This function also handles Mips::ATOMIC_SWAP_I32 (when BinOpcode == 0), and
 // Mips::ATOMIC_LOAD_NAND_I32 (when Nand == true)
-MachineBasicBlock *MipsTargetLowering::emitAtomicBinary(MachineInstr &MI,
-                                                        MachineBasicBlock *BB,
-                                                        unsigned Size,
-                                                        unsigned BinOpcode,
-                                                        bool Nand) const {
-  assert((Size == 4 || Size == 8) && "Unsupported size for EmitAtomicBinary.");
+MachineBasicBlock *
+MipsTargetLowering::emitAtomicBinary(MachineInstr &MI,
+                                     MachineBasicBlock *BB) const {
 
   MachineFunction *MF = BB->getParent();
   MachineRegisterInfo &RegInfo = MF->getRegInfo();
-  const TargetRegisterClass *RC = getRegClassFor(MVT::getIntegerVT(Size * 8));
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
-  const bool ArePtrs64bit = ABI.ArePtrs64bit();
   DebugLoc DL = MI.getDebugLoc();
-  unsigned LL, SC, AND, NOR, ZERO, BEQ;
 
-  if (Size == 4) {
-    if (isMicroMips) {
-      LL = Subtarget.hasMips32r6() ? Mips::LL_MMR6 : Mips::LL_MM;
-      SC = Subtarget.hasMips32r6() ? Mips::SC_MMR6 : Mips::SC_MM;
-    } else {
-      LL = Subtarget.hasMips32r6()
-               ? (ArePtrs64bit ? Mips::LL64_R6 : Mips::LL_R6)
-               : (ArePtrs64bit ? Mips::LL64 : Mips::LL);
-      SC = Subtarget.hasMips32r6()
-               ? (ArePtrs64bit ? Mips::SC64_R6 : Mips::SC_R6)
-               : (ArePtrs64bit ? Mips::SC64 : Mips::SC);
-    }
-
-    AND = Mips::AND;
-    NOR = Mips::NOR;
-    ZERO = Mips::ZERO;
-    BEQ = Mips::BEQ;
-  } else {
-    LL = Subtarget.hasMips64r6() ? Mips::LLD_R6 : Mips::LLD;
-    SC = Subtarget.hasMips64r6() ? Mips::SCD_R6 : Mips::SCD;
-    AND = Mips::AND64;
-    NOR = Mips::NOR64;
-    ZERO = Mips::ZERO_64;
-    BEQ = Mips::BEQ64;
+  unsigned AtomicOp;
+  switch (MI.getOpcode()) {
+  case Mips::ATOMIC_LOAD_ADD_I32:
+    AtomicOp = Mips::ATOMIC_LOAD_ADD_I32_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_SUB_I32:
+    AtomicOp = Mips::ATOMIC_LOAD_SUB_I32_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_AND_I32:
+    AtomicOp = Mips::ATOMIC_LOAD_AND_I32_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_OR_I32:
+    AtomicOp = Mips::ATOMIC_LOAD_OR_I32_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_XOR_I32:
+    AtomicOp = Mips::ATOMIC_LOAD_XOR_I32_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_NAND_I32:
+    AtomicOp = Mips::ATOMIC_LOAD_NAND_I32_POSTRA;
+    break;
+  case Mips::ATOMIC_SWAP_I32:
+    AtomicOp = Mips::ATOMIC_SWAP_I32_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_ADD_I64:
+    AtomicOp = Mips::ATOMIC_LOAD_ADD_I64_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_SUB_I64:
+    AtomicOp = Mips::ATOMIC_LOAD_SUB_I64_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_AND_I64:
+    AtomicOp = Mips::ATOMIC_LOAD_AND_I64_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_OR_I64:
+    AtomicOp = Mips::ATOMIC_LOAD_OR_I64_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_XOR_I64:
+    AtomicOp = Mips::ATOMIC_LOAD_XOR_I64_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_NAND_I64:
+    AtomicOp = Mips::ATOMIC_LOAD_NAND_I64_POSTRA;
+    break;
+  case Mips::ATOMIC_SWAP_I64:
+    AtomicOp = Mips::ATOMIC_SWAP_I64_POSTRA;
+    break;
+  default:
+    llvm_unreachable("Unknown pseudo atomic for replacement!");
   }
 
   unsigned OldVal = MI.getOperand(0).getReg();
   unsigned Ptr = MI.getOperand(1).getReg();
   unsigned Incr = MI.getOperand(2).getReg();
+  unsigned Scratch = RegInfo.createVirtualRegister(RegInfo.getRegClass(OldVal));
 
-  unsigned StoreVal = RegInfo.createVirtualRegister(RC);
-  unsigned AndRes = RegInfo.createVirtualRegister(RC);
-  unsigned Success = RegInfo.createVirtualRegister(RC);
+  MachineBasicBlock::iterator II(MI);
 
-  // insert new blocks after the current block
-  const BasicBlock *LLVM_BB = BB->getBasicBlock();
-  MachineBasicBlock *loopMBB = MF->CreateMachineBasicBlock(LLVM_BB);
-  MachineBasicBlock *exitMBB = MF->CreateMachineBasicBlock(LLVM_BB);
-  MachineFunction::iterator It = ++BB->getIterator();
-  MF->insert(It, loopMBB);
-  MF->insert(It, exitMBB);
+  // The scratch registers here with the EarlyClobber | Define | Implicit
+  // flags is used to persuade the register allocator and the machine
+  // verifier to accept the usage of this register. This has to be a real
+  // register which has an UNDEF value but is dead after the instruction which
+  // is unique among the registers chosen for the instruction.
+
+  // The EarlyClobber flag has the semantic properties that the operand it is
+  // attached to is clobbered before the rest of the inputs are read. Hence it
+  // must be unique among the operands to the instruction.
+  // The Define flag is needed to coerce the machine verifier that an Undef
+  // value isn't a problem.
+  // The Dead flag is needed as the value in scratch isn't used by any other
+  // instruction. Kill isn't used as Dead is more precise.
+  // The implicit flag is here due to the interaction between the other flags
+  // and the machine verifier.
+
+  // For correctness purpose, a new pseudo is introduced here. We need this
+  // new pseudo, so that FastRegisterAllocator does not see an ll/sc sequence
+  // that is spread over >1 basic blocks. A register allocator which
+  // introduces (or any codegen infact) a store, can violate the expectations
+  // of the hardware.
+  //
+  // An atomic read-modify-write sequence starts with a linked load
+  // instruction and ends with a store conditional instruction. The atomic
+  // read-modify-write sequence fails if any of the following conditions
+  // occur between the execution of ll and sc:
+  //   * A coherent store is completed by another process or coherent I/O
+  //     module into the block of synchronizable physical memory containing
+  //     the word. The size and alignment of the block is
+  //     implementation-dependent.
+  //   * A coherent store is executed between an LL and SC sequence on the
+  //     same processor to the block of synchornizable physical memory
+  //     containing the word.
+  //
 
-  // Transfer the remainder of BB and its successor edges to exitMBB.
-  exitMBB->splice(exitMBB->begin(), BB,
-                  std::next(MachineBasicBlock::iterator(MI)), BB->end());
-  exitMBB->transferSuccessorsAndUpdatePHIs(BB);
+  unsigned PtrCopy = RegInfo.createVirtualRegister(RegInfo.getRegClass(Ptr));
+  unsigned IncrCopy = RegInfo.createVirtualRegister(RegInfo.getRegClass(Incr));
 
-  //  thisMBB:
-  //    ...
-  //    fallthrough --> loopMBB
-  BB->addSuccessor(loopMBB);
-  loopMBB->addSuccessor(loopMBB);
-  loopMBB->addSuccessor(exitMBB);
-
-  //  loopMBB:
-  //    ll oldval, 0(ptr)
-  //    <binop> storeval, oldval, incr
-  //    sc success, storeval, 0(ptr)
-  //    beq success, $0, loopMBB
-  BB = loopMBB;
-  BuildMI(BB, DL, TII->get(LL), OldVal).addReg(Ptr).addImm(0);
-  if (Nand) {
-    //  and andres, oldval, incr
-    //  nor storeval, $0, andres
-    BuildMI(BB, DL, TII->get(AND), AndRes).addReg(OldVal).addReg(Incr);
-    BuildMI(BB, DL, TII->get(NOR), StoreVal).addReg(ZERO).addReg(AndRes);
-  } else if (BinOpcode) {
-    //  <binop> storeval, oldval, incr
-    BuildMI(BB, DL, TII->get(BinOpcode), StoreVal).addReg(OldVal).addReg(Incr);
-  } else {
-    StoreVal = Incr;
-  }
-  BuildMI(BB, DL, TII->get(SC), Success).addReg(StoreVal).addReg(Ptr).addImm(0);
-  BuildMI(BB, DL, TII->get(BEQ)).addReg(Success).addReg(ZERO).addMBB(loopMBB);
+  BuildMI(*BB, II, DL, TII->get(Mips::COPY), IncrCopy).addReg(Incr);
+  BuildMI(*BB, II, DL, TII->get(Mips::COPY), PtrCopy).addReg(Ptr);
 
-  MI.eraseFromParent(); // The instruction is gone now.
+  BuildMI(*BB, II, DL, TII->get(AtomicOp))
+      .addReg(OldVal, RegState::Define | RegState::EarlyClobber)
+      .addReg(PtrCopy)
+      .addReg(IncrCopy)
+      .addReg(Scratch, RegState::Define | RegState::EarlyClobber |
+                           RegState::Implicit | RegState::Dead);
 
-  return exitMBB;
+  MI.eraseFromParent();
+
+  return BB;
 }
 
 MachineBasicBlock *MipsTargetLowering::emitSignExtendToI32InReg(
     MachineInstr &MI, MachineBasicBlock *BB, unsigned Size, unsigned DstReg,
     unsigned SrcReg) const {
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
   const DebugLoc &DL = MI.getDebugLoc();
 
MachineBasicBlock *MipsTargetLowering::emitSignExtendToI32InReg(
 
   BuildMI(BB, DL, TII->get(Mips::SLL), ScrReg).addReg(SrcReg).addImm(ShiftImm);
   BuildMI(BB, DL, TII->get(Mips::SRA), DstReg).addReg(ScrReg).addImm(ShiftImm);
 
   return BB;
 }
 
 MachineBasicBlock *MipsTargetLowering::emitAtomicBinaryPartword(
-    MachineInstr &MI, MachineBasicBlock *BB, unsigned Size, unsigned BinOpcode,
-    bool Nand) const {
+    MachineInstr &MI, MachineBasicBlock *BB, unsigned Size) const {
   assert((Size == 1 || Size == 2) &&
          "Unsupported size for EmitAtomicBinaryPartial.");
 
   MachineFunction *MF = BB->getParent();
   MachineRegisterInfo &RegInfo = MF->getRegInfo();
   const TargetRegisterClass *RC = getRegClassFor(MVT::i32);
   const bool ArePtrs64bit = ABI.ArePtrs64bit();
   const TargetRegisterClass *RCp =
     getRegClassFor(ArePtrs64bit ? MVT::i64 : MVT::i32);
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
   DebugLoc DL = MI.getDebugLoc();
 
   unsigned Dest = MI.getOperand(0).getReg();
   unsigned Ptr = MI.getOperand(1).getReg();
   unsigned Incr = MI.getOperand(2).getReg();
 
   unsigned AlignedAddr = RegInfo.createVirtualRegister(RCp);
   unsigned ShiftAmt = RegInfo.createVirtualRegister(RC);
   unsigned Mask = RegInfo.createVirtualRegister(RC);
   unsigned Mask2 = RegInfo.createVirtualRegister(RC);
-  unsigned NewVal = RegInfo.createVirtualRegister(RC);
-  unsigned OldVal = RegInfo.createVirtualRegister(RC);
   unsigned Incr2 = RegInfo.createVirtualRegister(RC);
   unsigned MaskLSB2 = RegInfo.createVirtualRegister(RCp);
   unsigned PtrLSB2 = RegInfo.createVirtualRegister(RC);
   unsigned MaskUpper = RegInfo.createVirtualRegister(RC);
-  unsigned AndRes = RegInfo.createVirtualRegister(RC);
-  unsigned BinOpRes = RegInfo.createVirtualRegister(RC);
-  unsigned MaskedOldVal0 = RegInfo.createVirtualRegister(RC);
-  unsigned StoreVal = RegInfo.createVirtualRegister(RC);
-  unsigned MaskedOldVal1 = RegInfo.createVirtualRegister(RC);
-  unsigned SrlRes = RegInfo.createVirtualRegister(RC);
-  unsigned Success = RegInfo.createVirtualRegister(RC);
-
-  unsigned LL, SC;
-  if (isMicroMips) {
-    LL = Subtarget.hasMips32r6() ? Mips::LL_MMR6 : Mips::LL_MM;
-    SC = Subtarget.hasMips32r6() ? Mips::SC_MMR6 : Mips::SC_MM;
-  } else {
-    LL = Subtarget.hasMips32r6() ? (ArePtrs64bit ? Mips::LL64_R6 : Mips::LL_R6)
-                                 : (ArePtrs64bit ? Mips::LL64 : Mips::LL);
-    SC = Subtarget.hasMips32r6() ? (ArePtrs64bit ? Mips::SC64_R6 : Mips::SC_R6)
-                                 : (ArePtrs64bit ? Mips::SC64 : Mips::SC);
+  unsigned Scratch = RegInfo.createVirtualRegister(RC);
+  unsigned Scratch2 = RegInfo.createVirtualRegister(RC);
+  unsigned Scratch3 = RegInfo.createVirtualRegister(RC);
+
+  unsigned AtomicOp = 0;
+  switch (MI.getOpcode()) {
+  case Mips::ATOMIC_LOAD_NAND_I8:
+    AtomicOp = Mips::ATOMIC_LOAD_NAND_I8_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_NAND_I16:
+    AtomicOp = Mips::ATOMIC_LOAD_NAND_I16_POSTRA;
+    break;
+  case Mips::ATOMIC_SWAP_I8:
+    AtomicOp = Mips::ATOMIC_SWAP_I8_POSTRA;
+    break;
+  case Mips::ATOMIC_SWAP_I16:
+    AtomicOp = Mips::ATOMIC_SWAP_I16_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_ADD_I8:
+    AtomicOp = Mips::ATOMIC_LOAD_ADD_I8_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_ADD_I16:
+    AtomicOp = Mips::ATOMIC_LOAD_ADD_I16_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_SUB_I8:
+    AtomicOp = Mips::ATOMIC_LOAD_SUB_I8_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_SUB_I16:
+    AtomicOp = Mips::ATOMIC_LOAD_SUB_I16_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_AND_I8:
+    AtomicOp = Mips::ATOMIC_LOAD_AND_I8_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_AND_I16:
+    AtomicOp = Mips::ATOMIC_LOAD_AND_I16_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_OR_I8:
+    AtomicOp = Mips::ATOMIC_LOAD_OR_I8_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_OR_I16:
+    AtomicOp = Mips::ATOMIC_LOAD_OR_I16_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_XOR_I8:
+    AtomicOp = Mips::ATOMIC_LOAD_XOR_I8_POSTRA;
+    break;
+  case Mips::ATOMIC_LOAD_XOR_I16:
+    AtomicOp = Mips::ATOMIC_LOAD_XOR_I16_POSTRA;
+    break;
+  default:
+    llvm_unreachable("Unknown subword atomic pseudo for expansion!");
   }
 
   // insert new blocks after the current block
   const BasicBlock *LLVM_BB = BB->getBasicBlock();
-  MachineBasicBlock *loopMBB = MF->CreateMachineBasicBlock(LLVM_BB);
-  MachineBasicBlock *sinkMBB = MF->CreateMachineBasicBlock(LLVM_BB);
   MachineBasicBlock *exitMBB = MF->CreateMachineBasicBlock(LLVM_BB);
   MachineFunction::iterator It = ++BB->getIterator();
-  MF->insert(It, loopMBB);
-  MF->insert(It, sinkMBB);
   MF->insert(It, exitMBB);
 
   // Transfer the remainder of BB and its successor edges to exitMBB.
   exitMBB->splice(exitMBB->begin(), BB,
                   std::next(MachineBasicBlock::iterator(MI)), BB->end());
   exitMBB->transferSuccessorsAndUpdatePHIs(BB);
 
-  BB->addSuccessor(loopMBB);
-  loopMBB->addSuccessor(loopMBB);
-  loopMBB->addSuccessor(sinkMBB);
-  sinkMBB->addSuccessor(exitMBB);
+  BB->addSuccessor(exitMBB, BranchProbability::getOne());
 
   //  thisMBB:
   //    addiu   masklsb2,$0,-4                # 0xfffffffc
   //    and     alignedaddr,ptr,masklsb2
   //    andi    ptrlsb2,ptr,3
   //    sll     shiftamt,ptrlsb2,3
   //    ori     maskupper,$0,255               # 0xff
   //    sll     mask,maskupper,shiftamt
MachineBasicBlock *MipsTargetLowering::emitAtomicBinaryPartword(
   }
   BuildMI(BB, DL, TII->get(Mips::ORi), MaskUpper)
     .addReg(Mips::ZERO).addImm(MaskImm);
   BuildMI(BB, DL, TII->get(Mips::SLLV), Mask)
     .addReg(MaskUpper).addReg(ShiftAmt);
   BuildMI(BB, DL, TII->get(Mips::NOR), Mask2).addReg(Mips::ZERO).addReg(Mask);
   BuildMI(BB, DL, TII->get(Mips::SLLV), Incr2).addReg(Incr).addReg(ShiftAmt);
 
-  // atomic.load.binop
-  // loopMBB:
-  //   ll      oldval,0(alignedaddr)
-  //   binop   binopres,oldval,incr2
-  //   and     newval,binopres,mask
-  //   and     maskedoldval0,oldval,mask2
-  //   or      storeval,maskedoldval0,newval
-  //   sc      success,storeval,0(alignedaddr)
-  //   beq     success,$0,loopMBB
-
-  // atomic.swap
-  // loopMBB:
-  //   ll      oldval,0(alignedaddr)
-  //   and     newval,incr2,mask
-  //   and     maskedoldval0,oldval,mask2
-  //   or      storeval,maskedoldval0,newval
-  //   sc      success,storeval,0(alignedaddr)
-  //   beq     success,$0,loopMBB
-
-  BB = loopMBB;
-  BuildMI(BB, DL, TII->get(LL), OldVal).addReg(AlignedAddr).addImm(0);
-  if (Nand) {
-    //  and andres, oldval, incr2
-    //  nor binopres, $0, andres
-    //  and newval, binopres, mask
-    BuildMI(BB, DL, TII->get(Mips::AND), AndRes).addReg(OldVal).addReg(Incr2);
-    BuildMI(BB, DL, TII->get(Mips::NOR), BinOpRes)
-      .addReg(Mips::ZERO).addReg(AndRes);
-    BuildMI(BB, DL, TII->get(Mips::AND), NewVal).addReg(BinOpRes).addReg(Mask);
-  } else if (BinOpcode) {
-    //  <binop> binopres, oldval, incr2
-    //  and newval, binopres, mask
-    BuildMI(BB, DL, TII->get(BinOpcode), BinOpRes).addReg(OldVal).addReg(Incr2);
-    BuildMI(BB, DL, TII->get(Mips::AND), NewVal).addReg(BinOpRes).addReg(Mask);
-  } else { // atomic.swap
-    //  and newval, incr2, mask
-    BuildMI(BB, DL, TII->get(Mips::AND), NewVal).addReg(Incr2).addReg(Mask);
-  }
-
-  BuildMI(BB, DL, TII->get(Mips::AND), MaskedOldVal0)
-    .addReg(OldVal).addReg(Mask2);
-  BuildMI(BB, DL, TII->get(Mips::OR), StoreVal)
-    .addReg(MaskedOldVal0).addReg(NewVal);
-  BuildMI(BB, DL, TII->get(SC), Success)
-    .addReg(StoreVal).addReg(AlignedAddr).addImm(0);
-  BuildMI(BB, DL, TII->get(Mips::BEQ))
-    .addReg(Success).addReg(Mips::ZERO).addMBB(loopMBB);
-
-  //  sinkMBB:
-  //    and     maskedoldval1,oldval,mask
-  //    srl     srlres,maskedoldval1,shiftamt
-  //    sign_extend dest,srlres
-  BB = sinkMBB;
 
-  BuildMI(BB, DL, TII->get(Mips::AND), MaskedOldVal1)
-    .addReg(OldVal).addReg(Mask);
-  BuildMI(BB, DL, TII->get(Mips::SRLV), SrlRes)
-      .addReg(MaskedOldVal1).addReg(ShiftAmt);
-  BB = emitSignExtendToI32InReg(MI, BB, Size, Dest, SrlRes);
+  // The purposes of the flags on the scratch registers is explained in
+  // emitAtomicBinary. In summary, we need a scratch register which is going to
+  // be undef, that is unique among registers chosen for the instruction.
+
+  BuildMI(BB, DL, TII->get(AtomicOp))
+      .addReg(Dest, RegState::Define | RegState::EarlyClobber)
+      .addReg(AlignedAddr)
+      .addReg(Incr2)
+      .addReg(Mask)
+      .addReg(Mask2)
+      .addReg(ShiftAmt)
+      .addReg(Scratch, RegState::EarlyClobber | RegState::Define |
+                           RegState::Dead | RegState::Implicit)
+      .addReg(Scratch2, RegState::EarlyClobber | RegState::Define |
+                            RegState::Dead | RegState::Implicit)
+      .addReg(Scratch3, RegState::EarlyClobber | RegState::Define |
+                            RegState::Dead | RegState::Implicit);
 
   MI.eraseFromParent(); // The instruction is gone now.
 
   return exitMBB;
 }
 
-MachineBasicBlock *MipsTargetLowering::emitAtomicCmpSwap(MachineInstr &MI,
-                                                         MachineBasicBlock *BB,
-                                                         unsigned Size) const {
-  assert((Size == 4 || Size == 8) && "Unsupported size for EmitAtomicCmpSwap.");
+// Lower atomic compare and swap to a pseudo instruction, taking care to
+// define a scratch register for the pseudo instruction's expansion. The
+// instruction is expanded after the register allocator as to prevent
+// the insertion of stores between the linked load and the store conditional.
+
+MachineBasicBlock *
+MipsTargetLowering::emitAtomicCmpSwap(MachineInstr &MI,
+                                      MachineBasicBlock *BB) const {
+
+  assert((MI.getOpcode() == Mips::ATOMIC_CMP_SWAP_I32 ||
+          MI.getOpcode() == Mips::ATOMIC_CMP_SWAP_I64) &&
+         "Unsupported atomic psseudo for EmitAtomicCmpSwap.");
+
+  const unsigned Size = MI.getOpcode() == Mips::ATOMIC_CMP_SWAP_I32 ? 4 : 8;
 
   MachineFunction *MF = BB->getParent();
-  MachineRegisterInfo &RegInfo = MF->getRegInfo();
+  MachineRegisterInfo &MRI = MF->getRegInfo();
   const TargetRegisterClass *RC = getRegClassFor(MVT::getIntegerVT(Size * 8));
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
-  const bool ArePtrs64bit = ABI.ArePtrs64bit();
   DebugLoc DL = MI.getDebugLoc();
-  unsigned LL, SC, ZERO, BNE, BEQ;
-
-  if (Size == 4) {
-    if (isMicroMips) {
-      LL = Subtarget.hasMips32r6() ? Mips::LL_MMR6 : Mips::LL_MM;
-      SC = Subtarget.hasMips32r6() ? Mips::SC_MMR6 : Mips::SC_MM;
-    } else {
-      LL = Subtarget.hasMips32r6()
-               ? (ArePtrs64bit ? Mips::LL64_R6 : Mips::LL_R6)
-               : (ArePtrs64bit ? Mips::LL64 : Mips::LL);
-      SC = Subtarget.hasMips32r6()
-               ? (ArePtrs64bit ? Mips::SC64_R6 : Mips::SC_R6)
-               : (ArePtrs64bit ? Mips::SC64 : Mips::SC);
-    }
-
-    ZERO = Mips::ZERO;
-    BNE = Mips::BNE;
-    BEQ = Mips::BEQ;
-  } else {
-    LL = Subtarget.hasMips64r6() ? Mips::LLD_R6 : Mips::LLD;
-    SC = Subtarget.hasMips64r6() ? Mips::SCD_R6 : Mips::SCD;
-    ZERO = Mips::ZERO_64;
-    BNE = Mips::BNE64;
-    BEQ = Mips::BEQ64;
-  }
 
+  unsigned AtomicOp = MI.getOpcode() == Mips::ATOMIC_CMP_SWAP_I32
+                          ? Mips::ATOMIC_CMP_SWAP_I32_POSTRA
+                          : Mips::ATOMIC_CMP_SWAP_I64_POSTRA;
   unsigned Dest = MI.getOperand(0).getReg();
   unsigned Ptr = MI.getOperand(1).getReg();
   unsigned OldVal = MI.getOperand(2).getReg();
   unsigned NewVal = MI.getOperand(3).getReg();
 
-  unsigned Success = RegInfo.createVirtualRegister(RC);
-
-  // insert new blocks after the current block
-  const BasicBlock *LLVM_BB = BB->getBasicBlock();
-  MachineBasicBlock *loop1MBB = MF->CreateMachineBasicBlock(LLVM_BB);
-  MachineBasicBlock *loop2MBB = MF->CreateMachineBasicBlock(LLVM_BB);
-  MachineBasicBlock *exitMBB = MF->CreateMachineBasicBlock(LLVM_BB);
-  MachineFunction::iterator It = ++BB->getIterator();
-  MF->insert(It, loop1MBB);
-  MF->insert(It, loop2MBB);
-  MF->insert(It, exitMBB);
-
-  // Transfer the remainder of BB and its successor edges to exitMBB.
-  exitMBB->splice(exitMBB->begin(), BB,
-                  std::next(MachineBasicBlock::iterator(MI)), BB->end());
-  exitMBB->transferSuccessorsAndUpdatePHIs(BB);
+  unsigned Scratch = MRI.createVirtualRegister(RC);
+  MachineBasicBlock::iterator II(MI);
 
-  //  thisMBB:
-  //    ...
-  //    fallthrough --> loop1MBB
-  BB->addSuccessor(loop1MBB);
-  loop1MBB->addSuccessor(exitMBB);
-  loop1MBB->addSuccessor(loop2MBB);
-  loop2MBB->addSuccessor(loop1MBB);
-  loop2MBB->addSuccessor(exitMBB);
-
-  // loop1MBB:
-  //   ll dest, 0(ptr)
-  //   bne dest, oldval, exitMBB
-  BB = loop1MBB;
-  BuildMI(BB, DL, TII->get(LL), Dest).addReg(Ptr).addImm(0);
-  BuildMI(BB, DL, TII->get(BNE))
-    .addReg(Dest).addReg(OldVal).addMBB(exitMBB);
-
-  // loop2MBB:
-  //   sc success, newval, 0(ptr)
-  //   beq success, $0, loop1MBB
-  BB = loop2MBB;
-  BuildMI(BB, DL, TII->get(SC), Success)
-    .addReg(NewVal).addReg(Ptr).addImm(0);
-  BuildMI(BB, DL, TII->get(BEQ))
-    .addReg(Success).addReg(ZERO).addMBB(loop1MBB);
+  // We need to create copies of the various registers and kill them at the
+  // atomic pseudo. If the copies are not made, when the atomic is expanded
+  // after fast register allocation, the spills will end up outside of the
+  // blocks that their values are defined in, causing livein errors.
+
+  unsigned DestCopy = MRI.createVirtualRegister(MRI.getRegClass(Dest));
+  unsigned PtrCopy = MRI.createVirtualRegister(MRI.getRegClass(Ptr));
+  unsigned OldValCopy = MRI.createVirtualRegister(MRI.getRegClass(OldVal));
+  unsigned NewValCopy = MRI.createVirtualRegister(MRI.getRegClass(NewVal));
+
+  BuildMI(*BB, II, DL, TII->get(Mips::COPY), DestCopy).addReg(Dest);
+  BuildMI(*BB, II, DL, TII->get(Mips::COPY), PtrCopy).addReg(Ptr);
+  BuildMI(*BB, II, DL, TII->get(Mips::COPY), OldValCopy).addReg(OldVal);
+  BuildMI(*BB, II, DL, TII->get(Mips::COPY), NewValCopy).addReg(NewVal);
+
+  // The purposes of the flags on the scratch registers is explained in
+  // emitAtomicBinary. In summary, we need a scratch register which is going to
+  // be undef, that is unique among registers chosen for the instruction.
+
+  BuildMI(*BB, II, DL, TII->get(AtomicOp))
+      .addReg(Dest, RegState::Define | RegState::EarlyClobber)
+      .addReg(PtrCopy, RegState::Kill)
+      .addReg(OldValCopy, RegState::Kill)
+      .addReg(NewValCopy, RegState::Kill)
+      .addReg(Scratch, RegState::EarlyClobber | RegState::Define |
+                           RegState::Dead | RegState::Implicit);
 
   MI.eraseFromParent(); // The instruction is gone now.
 
-  return exitMBB;
+  return BB;
 }
 
 MachineBasicBlock *MipsTargetLowering::emitAtomicCmpSwapPartword(
     MachineInstr &MI, MachineBasicBlock *BB, unsigned Size) const {
   assert((Size == 1 || Size == 2) &&
       "Unsupported size for EmitAtomicCmpSwapPartial.");
 
   MachineFunction *MF = BB->getParent();
MachineBasicBlock *MipsTargetLowering::emitAtomicCmpSwapPartword(
   unsigned CmpVal = MI.getOperand(2).getReg();
   unsigned NewVal = MI.getOperand(3).getReg();
 
   unsigned AlignedAddr = RegInfo.createVirtualRegister(RCp);
   unsigned ShiftAmt = RegInfo.createVirtualRegister(RC);
   unsigned Mask = RegInfo.createVirtualRegister(RC);
   unsigned Mask2 = RegInfo.createVirtualRegister(RC);
   unsigned ShiftedCmpVal = RegInfo.createVirtualRegister(RC);
-  unsigned OldVal = RegInfo.createVirtualRegister(RC);
-  unsigned MaskedOldVal0 = RegInfo.createVirtualRegister(RC);
   unsigned ShiftedNewVal = RegInfo.createVirtualRegister(RC);
   unsigned MaskLSB2 = RegInfo.createVirtualRegister(RCp);
   unsigned PtrLSB2 = RegInfo.createVirtualRegister(RC);
   unsigned MaskUpper = RegInfo.createVirtualRegister(RC);
   unsigned MaskedCmpVal = RegInfo.createVirtualRegister(RC);
   unsigned MaskedNewVal = RegInfo.createVirtualRegister(RC);
-  unsigned MaskedOldVal1 = RegInfo.createVirtualRegister(RC);
-  unsigned StoreVal = RegInfo.createVirtualRegister(RC);
-  unsigned SrlRes = RegInfo.createVirtualRegister(RC);
-  unsigned Success = RegInfo.createVirtualRegister(RC);
-  unsigned LL, SC;
-
-  if (isMicroMips) {
-    LL = Subtarget.hasMips32r6() ? Mips::LL_MMR6 : Mips::LL_MM;
-    SC = Subtarget.hasMips32r6() ? Mips::SC_MMR6 : Mips::SC_MM;
-  } else {
-    LL = Subtarget.hasMips32r6() ? (ArePtrs64bit ? Mips::LL64_R6 : Mips::LL_R6)
-                                 : (ArePtrs64bit ? Mips::LL64 : Mips::LL);
-    SC = Subtarget.hasMips32r6() ? (ArePtrs64bit ? Mips::SC64_R6 : Mips::SC_R6)
-                                 : (ArePtrs64bit ? Mips::SC64 : Mips::SC);
-  }
+  unsigned AtomicOp = MI.getOpcode() == Mips::ATOMIC_CMP_SWAP_I8
+                          ? Mips::ATOMIC_CMP_SWAP_I8_POSTRA
+                          : Mips::ATOMIC_CMP_SWAP_I16_POSTRA;
+
+  // The scratch registers here with the EarlyClobber | Define | Dead | Implicit
+  // flags are used to coerce the register allocator and the machine verifier to
+  // accept the usage of these registers.
+  // The EarlyClobber flag has the semantic properties that the operand it is
+  // attached to is clobbered before the rest of the inputs are read. Hence it
+  // must be unique among the operands to the instruction.
+  // The Define flag is needed to coerce the machine verifier that an Undef
+  // value isn't a problem.
+  // The Dead flag is needed as the value in scratch isn't used by any other
+  // instruction. Kill isn't used as Dead is more precise.
+  unsigned Scratch = RegInfo.createVirtualRegister(RC);
+  unsigned Scratch2 = RegInfo.createVirtualRegister(RC);
 
   // insert new blocks after the current block
   const BasicBlock *LLVM_BB = BB->getBasicBlock();
-  MachineBasicBlock *loop1MBB = MF->CreateMachineBasicBlock(LLVM_BB);
-  MachineBasicBlock *loop2MBB = MF->CreateMachineBasicBlock(LLVM_BB);
-  MachineBasicBlock *sinkMBB = MF->CreateMachineBasicBlock(LLVM_BB);
   MachineBasicBlock *exitMBB = MF->CreateMachineBasicBlock(LLVM_BB);
   MachineFunction::iterator It = ++BB->getIterator();
-  MF->insert(It, loop1MBB);
-  MF->insert(It, loop2MBB);
-  MF->insert(It, sinkMBB);
   MF->insert(It, exitMBB);
 
   // Transfer the remainder of BB and its successor edges to exitMBB.
   exitMBB->splice(exitMBB->begin(), BB,
                   std::next(MachineBasicBlock::iterator(MI)), BB->end());
   exitMBB->transferSuccessorsAndUpdatePHIs(BB);
 
-  BB->addSuccessor(loop1MBB);
-  loop1MBB->addSuccessor(sinkMBB);
-  loop1MBB->addSuccessor(loop2MBB);
-  loop2MBB->addSuccessor(loop1MBB);
-  loop2MBB->addSuccessor(sinkMBB);
-  sinkMBB->addSuccessor(exitMBB);
+  BB->addSuccessor(exitMBB, BranchProbability::getOne());
 
-  // FIXME: computation of newval2 can be moved to loop2MBB.
   //  thisMBB:
   //    addiu   masklsb2,$0,-4                # 0xfffffffc
   //    and     alignedaddr,ptr,masklsb2
   //    andi    ptrlsb2,ptr,3
   //    xori    ptrlsb2,ptrlsb2,3              # Only for BE
   //    sll     shiftamt,ptrlsb2,3
   //    ori     maskupper,$0,255               # 0xff
   //    sll     mask,maskupper,shiftamt
BuildMI(BB, DL, TII->get(Mips::ANDi), MaskedCmpVal)
     .addReg(CmpVal).addImm(MaskImm);
   BuildMI(BB, DL, TII->get(Mips::SLLV), ShiftedCmpVal)
     .addReg(MaskedCmpVal).addReg(ShiftAmt);
   BuildMI(BB, DL, TII->get(Mips::ANDi), MaskedNewVal)
     .addReg(NewVal).addImm(MaskImm);
   BuildMI(BB, DL, TII->get(Mips::SLLV), ShiftedNewVal)
     .addReg(MaskedNewVal).addReg(ShiftAmt);
 
-  //  loop1MBB:
-  //    ll      oldval,0(alginedaddr)
-  //    and     maskedoldval0,oldval,mask
-  //    bne     maskedoldval0,shiftedcmpval,sinkMBB
-  BB = loop1MBB;
-  BuildMI(BB, DL, TII->get(LL), OldVal).addReg(AlignedAddr).addImm(0);
-  BuildMI(BB, DL, TII->get(Mips::AND), MaskedOldVal0)
-    .addReg(OldVal).addReg(Mask);
-  BuildMI(BB, DL, TII->get(Mips::BNE))
-    .addReg(MaskedOldVal0).addReg(ShiftedCmpVal).addMBB(sinkMBB);
-
-  //  loop2MBB:
-  //    and     maskedoldval1,oldval,mask2
-  //    or      storeval,maskedoldval1,shiftednewval
-  //    sc      success,storeval,0(alignedaddr)
-  //    beq     success,$0,loop1MBB
-  BB = loop2MBB;
-  BuildMI(BB, DL, TII->get(Mips::AND), MaskedOldVal1)
-    .addReg(OldVal).addReg(Mask2);
-  BuildMI(BB, DL, TII->get(Mips::OR), StoreVal)
-    .addReg(MaskedOldVal1).addReg(ShiftedNewVal);
-  BuildMI(BB, DL, TII->get(SC), Success)
-      .addReg(StoreVal).addReg(AlignedAddr).addImm(0);
-  BuildMI(BB, DL, TII->get(Mips::BEQ))
-      .addReg(Success).addReg(Mips::ZERO).addMBB(loop1MBB);
-
-  //  sinkMBB:
-  //    srl     srlres,maskedoldval0,shiftamt
-  //    sign_extend dest,srlres
-  BB = sinkMBB;
-
-  BuildMI(BB, DL, TII->get(Mips::SRLV), SrlRes)
-      .addReg(MaskedOldVal0).addReg(ShiftAmt);
-  BB = emitSignExtendToI32InReg(MI, BB, Size, Dest, SrlRes);
+  // The purposes of the flags on the scratch registers are explained in
+  // emitAtomicBinary. In summary, we need a scratch register which is going to
+  // be undef, that is unique among the register chosen for the instruction.
+
+  BuildMI(BB, DL, TII->get(AtomicOp))
+      .addReg(Dest, RegState::Define | RegState::EarlyClobber)
+      .addReg(AlignedAddr)
+      .addReg(Mask)
+      .addReg(ShiftedCmpVal)
+      .addReg(Mask2)
+      .addReg(ShiftedNewVal)
+      .addReg(ShiftAmt)
+      .addReg(Scratch, RegState::EarlyClobber | RegState::Define |
+                           RegState::Dead | RegState::Implicit)
+      .addReg(Scratch2, RegState::EarlyClobber | RegState::Define |
+                            RegState::Dead | RegState::Implicit);
 
   MI.eraseFromParent(); // The instruction is gone now.
 
   return exitMBB;
 }
 
 SDValue MipsTargetLowering::lowerBRCOND(SDValue Op, SelectionDAG &DAG) const {
   // The first operand is the chain, the second is the condition, the third is

llvm/trunk/lib/Target/Mips/MipsInstrInfo.td

class InsBase<string opstr, RegisterOperand RO, Operand PosOpnd,
   let Constraints = "$src = $rt";
 }
 
 // Atomic instructions with 2 source operands (ATOMIC_SWAP & ATOMIC_LOAD_*).
 class Atomic2Ops<PatFrag Op, RegisterClass DRC> :
   PseudoSE<(outs DRC:$dst), (ins PtrRC:$ptr, DRC:$incr),
            [(set DRC:$dst, (Op iPTR:$ptr, DRC:$incr))]>;
 
+class Atomic2OpsPostRA<RegisterClass RC> :
+  PseudoSE<(outs RC:$dst), (ins PtrRC:$ptr, RC:$incr), []> {
+  let mayLoad = 1;
+  let mayStore = 1;
+}
+
+class Atomic2OpsSubwordPostRA<RegisterClass RC> :
+  PseudoSE<(outs RC:$dst), (ins PtrRC:$ptr, RC:$incr, RC:$mask, RC:$mask2,
+                                RC:$shiftamnt), []>;
+
 // Atomic Compare & Swap.
+// Atomic compare and swap is lowered into two stages. The first stage happens
+// during ISelLowering, which produces the PostRA version of this instruction.
 class AtomicCmpSwap<PatFrag Op, RegisterClass DRC> :
   PseudoSE<(outs DRC:$dst), (ins PtrRC:$ptr, DRC:$cmp, DRC:$swap),
            [(set DRC:$dst, (Op iPTR:$ptr, DRC:$cmp, DRC:$swap))]>;
 
+class AtomicCmpSwapPostRA<RegisterClass RC> :
+  PseudoSE<(outs RC:$dst), (ins PtrRC:$ptr, RC:$cmp, RC:$swap), []> {
+  let mayLoad = 1;
+  let mayStore = 1;
+}
+
+class AtomicCmpSwapSubwordPostRA<RegisterClass RC> :
+  PseudoSE<(outs RC:$dst), (ins PtrRC:$ptr, RC:$mask, RC:$ShiftCmpVal,
+                                RC:$mask2, RC:$ShiftNewVal, RC:$ShiftAmt), []> {
+  let mayLoad = 1;
+  let mayStore = 1;
+}
+
+
 class LLBase<string opstr, RegisterOperand RO, DAGOperand MO = mem> :
   InstSE<(outs RO:$rt), (ins MO:$addr), !strconcat(opstr, "\t$rt, $addr"),
          [], II_LL, FrmI, opstr> {
   let DecoderMethod = "DecodeMem";
   let mayLoad = 1;
 }
 
 class SCBase<string opstr, RegisterOperand RO> :
let usesCustomInserter = 1 in {
 
   def ATOMIC_SWAP_I8       : Atomic2Ops<atomic_swap_8, GPR32>;
   def ATOMIC_SWAP_I16      : Atomic2Ops<atomic_swap_16, GPR32>;
   def ATOMIC_SWAP_I32      : Atomic2Ops<atomic_swap_32, GPR32>;
 
   def ATOMIC_CMP_SWAP_I8   : AtomicCmpSwap<atomic_cmp_swap_8, GPR32>;
   def ATOMIC_CMP_SWAP_I16  : AtomicCmpSwap<atomic_cmp_swap_16, GPR32>;
   def ATOMIC_CMP_SWAP_I32  : AtomicCmpSwap<atomic_cmp_swap_32, GPR32>;
+
 }
 
+def ATOMIC_LOAD_ADD_I8_POSTRA   : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_LOAD_ADD_I16_POSTRA  : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_LOAD_ADD_I32_POSTRA  : Atomic2OpsPostRA<GPR32>;
+def ATOMIC_LOAD_SUB_I8_POSTRA   : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_LOAD_SUB_I16_POSTRA  : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_LOAD_SUB_I32_POSTRA  : Atomic2OpsPostRA<GPR32>;
+def ATOMIC_LOAD_AND_I8_POSTRA   : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_LOAD_AND_I16_POSTRA  : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_LOAD_AND_I32_POSTRA  : Atomic2OpsPostRA<GPR32>;
+def ATOMIC_LOAD_OR_I8_POSTRA    : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_LOAD_OR_I16_POSTRA   : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_LOAD_OR_I32_POSTRA   : Atomic2OpsPostRA<GPR32>;
+def ATOMIC_LOAD_XOR_I8_POSTRA   : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_LOAD_XOR_I16_POSTRA  : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_LOAD_XOR_I32_POSTRA  : Atomic2OpsPostRA<GPR32>;
+def ATOMIC_LOAD_NAND_I8_POSTRA  : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_LOAD_NAND_I16_POSTRA : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_LOAD_NAND_I32_POSTRA : Atomic2OpsPostRA<GPR32>;
+
+def ATOMIC_SWAP_I8_POSTRA  : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_SWAP_I16_POSTRA : Atomic2OpsSubwordPostRA<GPR32>;
+def ATOMIC_SWAP_I32_POSTRA : Atomic2OpsPostRA<GPR32>;
+
+def ATOMIC_CMP_SWAP_I8_POSTRA : AtomicCmpSwapSubwordPostRA<GPR32>;
+def ATOMIC_CMP_SWAP_I16_POSTRA : AtomicCmpSwapSubwordPostRA<GPR32>;
+def ATOMIC_CMP_SWAP_I32_POSTRA : AtomicCmpSwapPostRA<GPR32>;
+
 /// Pseudo instructions for loading and storing accumulator registers.
 let isPseudo = 1, isCodeGenOnly = 1, hasNoSchedulingInfo = 1 in {
   def LOAD_ACC64  : Load<"", ACC64>;
   def STORE_ACC64 : Store<"", ACC64>;
 }
 
 // We need these two pseudo instructions to avoid offset calculation for long
 // branches.  See the comment in file MipsLongBranch.cpp for detailed

llvm/trunk/lib/Target/Mips/MipsTargetMachine.cpp

public:
   const MipsSubtarget &getMipsSubtarget() const {
     return *getMipsTargetMachine().getSubtargetImpl();
   }
 
   void addIRPasses() override;
   bool addInstSelector() override;
   void addPreEmitPass() override;
   void addPreRegAlloc() override;
+  void addPreEmit2() ;
   bool addIRTranslator() override;
   bool addLegalizeMachineIR() override;
   bool addRegBankSelect() override;
   bool addGlobalInstructionSelect() override;
 };
 
 } // end anonymous namespace
 
if (Subtarget->allowMixed16_32()) {
     // FIXME: This is no longer necessary as the TTI returned is per-function.
     return TargetTransformInfo(F.getParent()->getDataLayout());
   }
 
   LLVM_DEBUG(errs() << "Target Transform Info Pass Added\n");
   return TargetTransformInfo(BasicTTIImpl(this, F));
 }
 
+void MipsPassConfig::addPreEmit2() {
+}
+
 // Implemented by targets that want to run passes immediately before
 // machine code is emitted. return true if -print-machineinstrs should
 // print out the code after the passes.
 void MipsPassConfig::addPreEmitPass() {
+  // Expand pseudo instructions that are sensitive to register allocation.
+  addPass(createMipsExpandPseudoPass());
+
+  // The microMIPS size reduction pass performs instruction reselection for
+  // instructions which can be remapped to a 16 bit instruction.
   addPass(createMicroMipsSizeReducePass());
 
   // The delay slot filler pass can potientially create forbidden slot hazards
   // for MIPSR6 and therefore it should go before MipsBranchExpansion pass.
   addPass(createMipsDelaySlotFillerPass());
 
   // This pass expands branches and takes care about the forbidden slot hazards.
   // Expanding branches may potentially create forbidden slot hazards for

llvm/trunk/test/CodeGen/Mips/atomic.ll

-; RUN: llc -march=mipsel --disable-machine-licm -mcpu=mips32   -relocation-model=pic < %s | \
-; RUN:   FileCheck %s -check-prefixes=ALL,MIPS32-ANY,NO-SEB-SEH,CHECK-EL,NOT-MICROMIPS
-; RUN: llc -march=mipsel --disable-machine-licm -mcpu=mips32r2 -relocation-model=pic -verify-machineinstrs < %s | \
-; RUN:   FileCheck %s -check-prefixes=ALL,MIPS32-ANY,HAS-SEB-SEH,CHECK-EL,NOT-MICROMIPS
-; RUN: llc -march=mipsel --disable-machine-licm -mcpu=mips32r6 -relocation-model=pic -verify-machineinstrs < %s | \
-; RUN:   FileCheck %s -check-prefixes=ALL,MIPS32-ANY,HAS-SEB-SEH,CHECK-EL,MIPSR6
-; RUN: llc -march=mips64el --disable-machine-licm -mcpu=mips4    -relocation-model=pic < %s | \
-; RUN:   FileCheck %s -check-prefixes=ALL,MIPS64-ANY,NO-SEB-SEH,CHECK-EL,NOT-MICROMIPS
-; RUN: llc -march=mips64el --disable-machine-licm -mcpu=mips64   -relocation-model=pic < %s | \
-; RUN:   FileCheck %s -check-prefixes=ALL,MIPS64-ANY,NO-SEB-SEH,CHECK-EL,NOT-MICROMIPS
-; RUN: llc -march=mips64el --disable-machine-licm -mcpu=mips64r2 -relocation-model=pic -verify-machineinstrs < %s | \
-; RUN:   FileCheck %s -check-prefixes=ALL,MIPS64-ANY,HAS-SEB-SEH,CHECK-EL,NOT-MICROMIPS
-; RUN: llc -march=mips64el --disable-machine-licm -mcpu=mips64r6 -relocation-model=pic < %s | \
-; RUN:   FileCheck %s -check-prefixes=ALL,MIPS64-ANY,HAS-SEB-SEH,CHECK-EL,MIPSR6
-; RUN: llc -march=mips64 -O0 -mcpu=mips64r6 -relocation-model=pic -verify-machineinstrs < %s | \
-; RUN:   FileCheck %s -check-prefixes=ALL-LABEL,MIPS64-ANY,O0
-; RUN: llc -march=mipsel --disable-machine-licm -mcpu=mips32r2 -mattr=micromips -relocation-model=pic < %s | \
-; RUN:   FileCheck %s -check-prefixes=ALL,MIPS32-ANY,HAS-SEB-SEH,CHECK-EL,MICROMIPS
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=mipsel-unknown-linux-gnu --disable-machine-licm -mcpu=mips32 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS32
+; RUN: llc -mtriple=mipsel-unknown-linux-gnu -O0 --disable-machine-licm -mcpu=mips32 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS32O0
+; RUN: llc -mtriple=mipsel-unknown-linux-gnu --disable-machine-licm -mcpu=mips32r2 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS32R2
+; RUN: llc -mtriple=mipsel-unknown-linux-gnu --disable-machine-licm -mcpu=mips32r6 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS32R6
+; RUN: llc -mtriple=mipsel-unknown-linux-gnu -O0 --disable-machine-licm -mcpu=mips32r6 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS32R6O0
+; RUN: llc -mtriple=mips64el-unknown-linux-gnu --disable-machine-licm -mcpu=mips4 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS4
+; RUN: llc -mtriple=mips64el-unknown-linux-gnu --disable-machine-licm -mcpu=mips64 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS64
+; RUN: llc -mtriple=mips64el-unknown-linux-gnu --disable-machine-licm -mcpu=mips64r2 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS64R2
+; RUN: llc -mtriple=mips64el-unknown-linux-gnu --disable-machine-licm -mcpu=mips64r6 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS64R6
+; RUN: llc -mtriple=mips64-unknown-linux-gnu -O0 -mcpu=mips64r6 -relocation-model=pic -verify-machineinstrs -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS64R6O0
+; RUN: llc -mtriple=mipsel-unknown-linux-gnu --disable-machine-licm -mcpu=mips32r2 -mattr=micromips -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MM32
+
+; We want to verify the produced code is well formed all optimization levels, the rest of the tests which ensure correctness.
+; RUN: llc -mtriple=mipsel-unknown-linux-gnu -O1 --disable-machine-licm -mcpu=mips32 -relocation-model=pic -verify-machineinstrs < %s | FileCheck %s --check-prefix=O1
+; RUN: llc -mtriple=mipsel-unknown-linux-gnu -O2 --disable-machine-licm -mcpu=mips32 -relocation-model=pic -verify-machineinstrs < %s | FileCheck %s --check-prefix=O2
+; RUN: llc -mtriple=mipsel-unknown-linux-gnu -O3 --disable-machine-licm -mcpu=mips32 -relocation-model=pic -verify-machineinstrs < %s | FileCheck %s --check-prefix=O3
 
 ; Keep one big-endian check so that we don't reduce testing, but don't add more
 ; since endianness doesn't affect the body of the atomic operations.
-; RUN: llc -march=mips   --disable-machine-licm -mcpu=mips32 -relocation-model=pic < %s | \
-; RUN:   FileCheck %s -check-prefixes=ALL,MIPS32-ANY,NO-SEB-SEH,CHECK-EB,NOT-MICROMIPS
+; RUN: llc -mtriple=mips-unknown-linux-gnu   --disable-machine-licm -mcpu=mips32 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS32EB
 
 @x = common global i32 0, align 4
 
 define i32 @AtomicLoadAdd32(i32 signext %incr) nounwind {
+; MIPS32-LABEL: AtomicLoadAdd32:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    lw $1, %got(x)($1)
+; MIPS32-NEXT:  $BB0_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $2, 0($1)
+; MIPS32-NEXT:    addu $3, $2, $4
+; MIPS32-NEXT:    sc $3, 0($1)
+; MIPS32-NEXT:    beqz $3, $BB0_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    nop
+;
+; MIPS32O0-LABEL: AtomicLoadAdd32:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32O0-NEXT:  $BB0_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $25, 0($2)
+; MIPS32O0-NEXT:    addu $1, $25, $4
+; MIPS32O0-NEXT:    sc $1, 0($2)
+; MIPS32O0-NEXT:    beqz $1, $BB0_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    move $2, $25
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicLoadAdd32:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    lw $1, %got(x)($1)
+; MIPS32R2-NEXT:  $BB0_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $2, 0($1)
+; MIPS32R2-NEXT:    addu $3, $2, $4
+; MIPS32R2-NEXT:    sc $3, 0($1)
+; MIPS32R2-NEXT:    beqz $3, $BB0_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    nop
+;
+; MIPS32R6-LABEL: AtomicLoadAdd32:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    lw $1, %got(x)($1)
+; MIPS32R6-NEXT:  $BB0_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $2, 0($1)
+; MIPS32R6-NEXT:    addu $3, $2, $4
+; MIPS32R6-NEXT:    sc $3, 0($1)
+; MIPS32R6-NEXT:    beqzc $3, $BB0_1
+; MIPS32R6-NEXT:    nop
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    jrc $ra
+;
+; MIPS32R6O0-LABEL: AtomicLoadAdd32:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32R6O0-NEXT:  $BB0_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $1, 0($2)
+; MIPS32R6O0-NEXT:    addu $3, $1, $4
+; MIPS32R6O0-NEXT:    sc $3, 0($2)
+; MIPS32R6O0-NEXT:    beqzc $3, $BB0_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    move $2, $1
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicLoadAdd32:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd32)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd32)))
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB0_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $2, 0($1)
+; MIPS4-NEXT:    addu $3, $2, $4
+; MIPS4-NEXT:    sc $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB0_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: AtomicLoadAdd32:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd32)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd32)))
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB0_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $2, 0($1)
+; MIPS64-NEXT:    addu $3, $2, $4
+; MIPS64-NEXT:    sc $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB0_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: AtomicLoadAdd32:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd32)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd32)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB0_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $2, 0($1)
+; MIPS64R2-NEXT:    addu $3, $2, $4
+; MIPS64R2-NEXT:    sc $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB0_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicLoadAdd32:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd32)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd32)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB0_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $2, 0($1)
+; MIPS64R6-NEXT:    addu $3, $2, $4
+; MIPS64R6-NEXT:    sc $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB0_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicLoadAdd32:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd32)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd32)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB0_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $3, 0($1)
+; MIPS64R6O0-NEXT:    addu $5, $3, $2
+; MIPS64R6O0-NEXT:    sc $5, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $5, .LBB0_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    move $2, $3
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicLoadAdd32:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    lw $1, %got(x)($2)
+; MM32-NEXT:  $BB0_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $2, 0($1)
+; MM32-NEXT:    addu16 $3, $2, $4
+; MM32-NEXT:    sc $3, 0($1)
+; MM32-NEXT:    beqzc $3, $BB0_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicLoadAdd32:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    lw $1, %got(x)($1)
+; O1-NEXT:  $BB0_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $2, 0($1)
+; O1-NEXT:    addu $3, $2, $4
+; O1-NEXT:    sc $3, 0($1)
+; O1-NEXT:    beqz $3, $BB0_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: AtomicLoadAdd32:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    lw $1, %got(x)($1)
+; O2-NEXT:  $BB0_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $2, 0($1)
+; O2-NEXT:    addu $3, $2, $4
+; O2-NEXT:    sc $3, 0($1)
+; O2-NEXT:    beqz $3, $BB0_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: AtomicLoadAdd32:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    lw $1, %got(x)($1)
+; O3-NEXT:  $BB0_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $2, 0($1)
+; O3-NEXT:    addu $3, $2, $4
+; O3-NEXT:    sc $3, 0($1)
+; O3-NEXT:    beqz $3, $BB0_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS32EB-LABEL: AtomicLoadAdd32:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    lw $1, %got(x)($1)
+; MIPS32EB-NEXT:  $BB0_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $2, 0($1)
+; MIPS32EB-NEXT:    addu $3, $2, $4
+; MIPS32EB-NEXT:    sc $3, 0($1)
+; MIPS32EB-NEXT:    beqz $3, $BB0_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    nop
 entry:
   %0 = atomicrmw add i32* @x, i32 %incr monotonic
   ret i32 %0
 
-; ALL-LABEL: AtomicLoadAdd32:
+}
 
-; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(x)
-; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(x)(
+define i32 @AtomicLoadSub32(i32 signext %incr) nounwind {
+; MIPS32-LABEL: AtomicLoadSub32:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    lw $1, %got(x)($1)
+; MIPS32-NEXT:  $BB1_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $2, 0($1)
+; MIPS32-NEXT:    subu $3, $2, $4
+; MIPS32-NEXT:    sc $3, 0($1)
+; MIPS32-NEXT:    beqz $3, $BB1_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    nop
+;
+; MIPS32O0-LABEL: AtomicLoadSub32:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32O0-NEXT:  $BB1_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $25, 0($2)
+; MIPS32O0-NEXT:    subu $1, $25, $4
+; MIPS32O0-NEXT:    sc $1, 0($2)
+; MIPS32O0-NEXT:    beqz $1, $BB1_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    move $2, $25
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicLoadSub32:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    lw $1, %got(x)($1)
+; MIPS32R2-NEXT:  $BB1_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $2, 0($1)
+; MIPS32R2-NEXT:    subu $3, $2, $4
+; MIPS32R2-NEXT:    sc $3, 0($1)
+; MIPS32R2-NEXT:    beqz $3, $BB1_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    nop
+;
+; MIPS32R6-LABEL: AtomicLoadSub32:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    lw $1, %got(x)($1)
+; MIPS32R6-NEXT:  $BB1_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $2, 0($1)
+; MIPS32R6-NEXT:    subu $3, $2, $4
+; MIPS32R6-NEXT:    sc $3, 0($1)
+; MIPS32R6-NEXT:    beqzc $3, $BB1_1
+; MIPS32R6-NEXT:    nop
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    jrc $ra
+;
+; MIPS32R6O0-LABEL: AtomicLoadSub32:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32R6O0-NEXT:  $BB1_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $1, 0($2)
+; MIPS32R6O0-NEXT:    subu $3, $1, $4
+; MIPS32R6O0-NEXT:    sc $3, 0($2)
+; MIPS32R6O0-NEXT:    beqzc $3, $BB1_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    move $2, $1
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicLoadSub32:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub32)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub32)))
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB1_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $2, 0($1)
+; MIPS4-NEXT:    subu $3, $2, $4
+; MIPS4-NEXT:    sc $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB1_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: AtomicLoadSub32:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub32)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub32)))
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB1_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $2, 0($1)
+; MIPS64-NEXT:    subu $3, $2, $4
+; MIPS64-NEXT:    sc $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB1_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: AtomicLoadSub32:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub32)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub32)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB1_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $2, 0($1)
+; MIPS64R2-NEXT:    subu $3, $2, $4
+; MIPS64R2-NEXT:    sc $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB1_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicLoadSub32:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub32)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub32)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB1_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $2, 0($1)
+; MIPS64R6-NEXT:    subu $3, $2, $4
+; MIPS64R6-NEXT:    sc $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB1_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicLoadSub32:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub32)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub32)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB1_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $3, 0($1)
+; MIPS64R6O0-NEXT:    subu $5, $3, $2
+; MIPS64R6O0-NEXT:    sc $5, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $5, .LBB1_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    move $2, $3
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicLoadSub32:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    lw $1, %got(x)($2)
+; MM32-NEXT:  $BB1_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $2, 0($1)
+; MM32-NEXT:    subu16 $3, $2, $4
+; MM32-NEXT:    sc $3, 0($1)
+; MM32-NEXT:    beqzc $3, $BB1_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicLoadSub32:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    lw $1, %got(x)($1)
+; O1-NEXT:  $BB1_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $2, 0($1)
+; O1-NEXT:    subu $3, $2, $4
+; O1-NEXT:    sc $3, 0($1)
+; O1-NEXT:    beqz $3, $BB1_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: AtomicLoadSub32:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    lw $1, %got(x)($1)
+; O2-NEXT:  $BB1_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $2, 0($1)
+; O2-NEXT:    subu $3, $2, $4
+; O2-NEXT:    sc $3, 0($1)
+; O2-NEXT:    beqz $3, $BB1_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: AtomicLoadSub32:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    lw $1, %got(x)($1)
+; O3-NEXT:  $BB1_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $2, 0($1)
+; O3-NEXT:    subu $3, $2, $4
+; O3-NEXT:    sc $3, 0($1)
+; O3-NEXT:    beqz $3, $BB1_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS32EB-LABEL: AtomicLoadSub32:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    lw $1, %got(x)($1)
+; MIPS32EB-NEXT:  $BB1_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $2, 0($1)
+; MIPS32EB-NEXT:    subu $3, $2, $4
+; MIPS32EB-NEXT:    sc $3, 0($1)
+; MIPS32EB-NEXT:    beqz $3, $BB1_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    nop
+entry:
+  %0 = atomicrmw sub i32* @x, i32 %incr monotonic
+  ret i32 %0
 
-; O0:        [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; O0:            ld      $[[R1:[0-9]+]]
-; O0-NEXT:       ll      $[[R2:[0-9]+]], 0($[[R1]])
-
-; ALL:       [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; ALL:           ll      $[[R3:[0-9]+]], 0($[[R0]])
-; ALL:           addu    $[[R4:[0-9]+]], $[[R3]], $4
-; ALL:           sc      $[[R4]], 0($[[R0]])
-; NOT-MICROMIPS: beqz    $[[R4]], [[BB0]]
-; MICROMIPS:     beqzc   $[[R4]], [[BB0]]
-; MIPSR6:        beqzc   $[[R4]], [[BB0]]
 }
 
-define i32 @AtomicLoadNand32(i32 signext %incr) nounwind {
+define i32 @AtomicLoadXor32(i32 signext %incr) nounwind {
+; MIPS32-LABEL: AtomicLoadXor32:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    lw $1, %got(x)($1)
+; MIPS32-NEXT:  $BB2_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $2, 0($1)
+; MIPS32-NEXT:    xor $3, $2, $4
+; MIPS32-NEXT:    sc $3, 0($1)
+; MIPS32-NEXT:    beqz $3, $BB2_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    nop
+;
+; MIPS32O0-LABEL: AtomicLoadXor32:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32O0-NEXT:  $BB2_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $25, 0($2)
+; MIPS32O0-NEXT:    xor $1, $25, $4
+; MIPS32O0-NEXT:    sc $1, 0($2)
+; MIPS32O0-NEXT:    beqz $1, $BB2_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    move $2, $25
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicLoadXor32:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    lw $1, %got(x)($1)
+; MIPS32R2-NEXT:  $BB2_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $2, 0($1)
+; MIPS32R2-NEXT:    xor $3, $2, $4
+; MIPS32R2-NEXT:    sc $3, 0($1)
+; MIPS32R2-NEXT:    beqz $3, $BB2_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    nop
+;
+; MIPS32R6-LABEL: AtomicLoadXor32:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    lw $1, %got(x)($1)
+; MIPS32R6-NEXT:  $BB2_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $2, 0($1)
+; MIPS32R6-NEXT:    xor $3, $2, $4
+; MIPS32R6-NEXT:    sc $3, 0($1)
+; MIPS32R6-NEXT:    beqzc $3, $BB2_1
+; MIPS32R6-NEXT:    nop
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    jrc $ra
+;
+; MIPS32R6O0-LABEL: AtomicLoadXor32:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32R6O0-NEXT:  $BB2_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $1, 0($2)
+; MIPS32R6O0-NEXT:    xor $3, $1, $4
+; MIPS32R6O0-NEXT:    sc $3, 0($2)
+; MIPS32R6O0-NEXT:    beqzc $3, $BB2_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    move $2, $1
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicLoadXor32:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor32)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor32)))
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB2_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $2, 0($1)
+; MIPS4-NEXT:    xor $3, $2, $4
+; MIPS4-NEXT:    sc $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB2_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: AtomicLoadXor32:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor32)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor32)))
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB2_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $2, 0($1)
+; MIPS64-NEXT:    xor $3, $2, $4
+; MIPS64-NEXT:    sc $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB2_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: AtomicLoadXor32:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor32)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor32)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB2_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $2, 0($1)
+; MIPS64R2-NEXT:    xor $3, $2, $4
+; MIPS64R2-NEXT:    sc $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB2_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicLoadXor32:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor32)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor32)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB2_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $2, 0($1)
+; MIPS64R6-NEXT:    xor $3, $2, $4
+; MIPS64R6-NEXT:    sc $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB2_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicLoadXor32:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor32)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor32)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB2_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $3, 0($1)
+; MIPS64R6O0-NEXT:    xor $5, $3, $2
+; MIPS64R6O0-NEXT:    sc $5, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $5, .LBB2_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    move $2, $3
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicLoadXor32:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    lw $1, %got(x)($2)
+; MM32-NEXT:  $BB2_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $2, 0($1)
+; MM32-NEXT:    xor $3, $2, $4
+; MM32-NEXT:    sc $3, 0($1)
+; MM32-NEXT:    beqzc $3, $BB2_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicLoadXor32:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    lw $1, %got(x)($1)
+; O1-NEXT:  $BB2_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $2, 0($1)
+; O1-NEXT:    xor $3, $2, $4
+; O1-NEXT:    sc $3, 0($1)
+; O1-NEXT:    beqz $3, $BB2_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: AtomicLoadXor32:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    lw $1, %got(x)($1)
+; O2-NEXT:  $BB2_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $2, 0($1)
+; O2-NEXT:    xor $3, $2, $4
+; O2-NEXT:    sc $3, 0($1)
+; O2-NEXT:    beqz $3, $BB2_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: AtomicLoadXor32:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    lw $1, %got(x)($1)
+; O3-NEXT:  $BB2_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $2, 0($1)
+; O3-NEXT:    xor $3, $2, $4
+; O3-NEXT:    sc $3, 0($1)
+; O3-NEXT:    beqz $3, $BB2_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS32EB-LABEL: AtomicLoadXor32:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    lw $1, %got(x)($1)
+; MIPS32EB-NEXT:  $BB2_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $2, 0($1)
+; MIPS32EB-NEXT:    xor $3, $2, $4
+; MIPS32EB-NEXT:    sc $3, 0($1)
+; MIPS32EB-NEXT:    beqz $3, $BB2_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    nop
 entry:
-  %0 = atomicrmw nand i32* @x, i32 %incr monotonic
+  %0 = atomicrmw xor i32* @x, i32 %incr monotonic
   ret i32 %0
+}
 
-; ALL-LABEL: AtomicLoadNand32:
-
-; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(x)
-; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(x)(
+define i32 @AtomicLoadOr32(i32 signext %incr) nounwind {
+; MIPS32-LABEL: AtomicLoadOr32:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    lw $1, %got(x)($1)
+; MIPS32-NEXT:  $BB3_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $2, 0($1)
+; MIPS32-NEXT:    or $3, $2, $4
+; MIPS32-NEXT:    sc $3, 0($1)
+; MIPS32-NEXT:    beqz $3, $BB3_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    nop
+;
+; MIPS32O0-LABEL: AtomicLoadOr32:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32O0-NEXT:  $BB3_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $25, 0($2)
+; MIPS32O0-NEXT:    or $1, $25, $4
+; MIPS32O0-NEXT:    sc $1, 0($2)
+; MIPS32O0-NEXT:    beqz $1, $BB3_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    move $2, $25
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicLoadOr32:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    lw $1, %got(x)($1)
+; MIPS32R2-NEXT:  $BB3_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $2, 0($1)
+; MIPS32R2-NEXT:    or $3, $2, $4
+; MIPS32R2-NEXT:    sc $3, 0($1)
+; MIPS32R2-NEXT:    beqz $3, $BB3_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    nop
+;
+; MIPS32R6-LABEL: AtomicLoadOr32:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    lw $1, %got(x)($1)
+; MIPS32R6-NEXT:  $BB3_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $2, 0($1)
+; MIPS32R6-NEXT:    or $3, $2, $4
+; MIPS32R6-NEXT:    sc $3, 0($1)
+; MIPS32R6-NEXT:    beqzc $3, $BB3_1
+; MIPS32R6-NEXT:    nop
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    jrc $ra
+;
+; MIPS32R6O0-LABEL: AtomicLoadOr32:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32R6O0-NEXT:  $BB3_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $1, 0($2)
+; MIPS32R6O0-NEXT:    or $3, $1, $4
+; MIPS32R6O0-NEXT:    sc $3, 0($2)
+; MIPS32R6O0-NEXT:    beqzc $3, $BB3_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    move $2, $1
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicLoadOr32:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr32)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr32)))
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB3_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $2, 0($1)
+; MIPS4-NEXT:    or $3, $2, $4
+; MIPS4-NEXT:    sc $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB3_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: AtomicLoadOr32:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr32)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr32)))
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB3_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $2, 0($1)
+; MIPS64-NEXT:    or $3, $2, $4
+; MIPS64-NEXT:    sc $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB3_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: AtomicLoadOr32:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr32)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr32)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB3_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $2, 0($1)
+; MIPS64R2-NEXT:    or $3, $2, $4
+; MIPS64R2-NEXT:    sc $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB3_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicLoadOr32:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr32)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr32)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB3_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $2, 0($1)
+; MIPS64R6-NEXT:    or $3, $2, $4
+; MIPS64R6-NEXT:    sc $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB3_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicLoadOr32:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr32)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr32)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB3_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $3, 0($1)
+; MIPS64R6O0-NEXT:    or $5, $3, $2
+; MIPS64R6O0-NEXT:    sc $5, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $5, .LBB3_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    move $2, $3
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicLoadOr32:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    lw $1, %got(x)($2)
+; MM32-NEXT:  $BB3_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $2, 0($1)
+; MM32-NEXT:    or $3, $2, $4
+; MM32-NEXT:    sc $3, 0($1)
+; MM32-NEXT:    beqzc $3, $BB3_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicLoadOr32:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    lw $1, %got(x)($1)
+; O1-NEXT:  $BB3_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $2, 0($1)
+; O1-NEXT:    or $3, $2, $4
+; O1-NEXT:    sc $3, 0($1)
+; O1-NEXT:    beqz $3, $BB3_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: AtomicLoadOr32:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    lw $1, %got(x)($1)
+; O2-NEXT:  $BB3_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $2, 0($1)
+; O2-NEXT:    or $3, $2, $4
+; O2-NEXT:    sc $3, 0($1)
+; O2-NEXT:    beqz $3, $BB3_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: AtomicLoadOr32:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    lw $1, %got(x)($1)
+; O3-NEXT:  $BB3_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $2, 0($1)
+; O3-NEXT:    or $3, $2, $4
+; O3-NEXT:    sc $3, 0($1)
+; O3-NEXT:    beqz $3, $BB3_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS32EB-LABEL: AtomicLoadOr32:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    lw $1, %got(x)($1)
+; MIPS32EB-NEXT:  $BB3_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $2, 0($1)
+; MIPS32EB-NEXT:    or $3, $2, $4
+; MIPS32EB-NEXT:    sc $3, 0($1)
+; MIPS32EB-NEXT:    beqz $3, $BB3_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    nop
+entry:
+  %0 = atomicrmw or i32* @x, i32 %incr monotonic
+  ret i32 %0
+}
 
+define i32 @AtomicLoadAnd32(i32 signext %incr) nounwind {
+; MIPS32-LABEL: AtomicLoadAnd32:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    lw $1, %got(x)($1)
+; MIPS32-NEXT:  $BB4_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $2, 0($1)
+; MIPS32-NEXT:    and $3, $2, $4
+; MIPS32-NEXT:    sc $3, 0($1)
+; MIPS32-NEXT:    beqz $3, $BB4_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    nop
+;
+; MIPS32O0-LABEL: AtomicLoadAnd32:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32O0-NEXT:  $BB4_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $25, 0($2)
+; MIPS32O0-NEXT:    and $1, $25, $4
+; MIPS32O0-NEXT:    sc $1, 0($2)
+; MIPS32O0-NEXT:    beqz $1, $BB4_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    move $2, $25
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicLoadAnd32:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    lw $1, %got(x)($1)
+; MIPS32R2-NEXT:  $BB4_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $2, 0($1)
+; MIPS32R2-NEXT:    and $3, $2, $4
+; MIPS32R2-NEXT:    sc $3, 0($1)
+; MIPS32R2-NEXT:    beqz $3, $BB4_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    nop
+;
+; MIPS32R6-LABEL: AtomicLoadAnd32:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    lw $1, %got(x)($1)
+; MIPS32R6-NEXT:  $BB4_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $2, 0($1)
+; MIPS32R6-NEXT:    and $3, $2, $4
+; MIPS32R6-NEXT:    sc $3, 0($1)
+; MIPS32R6-NEXT:    beqzc $3, $BB4_1
+; MIPS32R6-NEXT:    nop
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    jrc $ra
+;
+; MIPS32R6O0-LABEL: AtomicLoadAnd32:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32R6O0-NEXT:  $BB4_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $1, 0($2)
+; MIPS32R6O0-NEXT:    and $3, $1, $4
+; MIPS32R6O0-NEXT:    sc $3, 0($2)
+; MIPS32R6O0-NEXT:    beqzc $3, $BB4_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    move $2, $1
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicLoadAnd32:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd32)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd32)))
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB4_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $2, 0($1)
+; MIPS4-NEXT:    and $3, $2, $4
+; MIPS4-NEXT:    sc $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB4_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: AtomicLoadAnd32:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd32)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd32)))
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB4_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $2, 0($1)
+; MIPS64-NEXT:    and $3, $2, $4
+; MIPS64-NEXT:    sc $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB4_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: AtomicLoadAnd32:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd32)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd32)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB4_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $2, 0($1)
+; MIPS64R2-NEXT:    and $3, $2, $4
+; MIPS64R2-NEXT:    sc $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB4_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicLoadAnd32:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd32)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd32)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB4_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $2, 0($1)
+; MIPS64R6-NEXT:    and $3, $2, $4
+; MIPS64R6-NEXT:    sc $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB4_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicLoadAnd32:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd32)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd32)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB4_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $3, 0($1)
+; MIPS64R6O0-NEXT:    and $5, $3, $2
+; MIPS64R6O0-NEXT:    sc $5, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $5, .LBB4_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    move $2, $3
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicLoadAnd32:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    lw $1, %got(x)($2)
+; MM32-NEXT:  $BB4_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $2, 0($1)
+; MM32-NEXT:    and $3, $2, $4
+; MM32-NEXT:    sc $3, 0($1)
+; MM32-NEXT:    beqzc $3, $BB4_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicLoadAnd32:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    lw $1, %got(x)($1)
+; O1-NEXT:  $BB4_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $2, 0($1)
+; O1-NEXT:    and $3, $2, $4
+; O1-NEXT:    sc $3, 0($1)
+; O1-NEXT:    beqz $3, $BB4_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: AtomicLoadAnd32:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    lw $1, %got(x)($1)
+; O2-NEXT:  $BB4_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $2, 0($1)
+; O2-NEXT:    and $3, $2, $4
+; O2-NEXT:    sc $3, 0($1)
+; O2-NEXT:    beqz $3, $BB4_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: AtomicLoadAnd32:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    lw $1, %got(x)($1)
+; O3-NEXT:  $BB4_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $2, 0($1)
+; O3-NEXT:    and $3, $2, $4
+; O3-NEXT:    sc $3, 0($1)
+; O3-NEXT:    beqz $3, $BB4_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS32EB-LABEL: AtomicLoadAnd32:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    lw $1, %got(x)($1)
+; MIPS32EB-NEXT:  $BB4_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $2, 0($1)
+; MIPS32EB-NEXT:    and $3, $2, $4
+; MIPS32EB-NEXT:    sc $3, 0($1)
+; MIPS32EB-NEXT:    beqz $3, $BB4_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    nop
+entry:
+  %0 = atomicrmw and i32* @x, i32 %incr monotonic
+  ret i32 %0
+}
 
+define i32 @AtomicLoadNand32(i32 signext %incr) nounwind {
+; MIPS32-LABEL: AtomicLoadNand32:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    lw $1, %got(x)($1)
+; MIPS32-NEXT:  $BB5_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $2, 0($1)
+; MIPS32-NEXT:    and $3, $2, $4
+; MIPS32-NEXT:    nor $3, $zero, $3
+; MIPS32-NEXT:    sc $3, 0($1)
+; MIPS32-NEXT:    beqz $3, $BB5_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    nop
+;
+; MIPS32O0-LABEL: AtomicLoadNand32:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32O0-NEXT:  $BB5_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $25, 0($2)
+; MIPS32O0-NEXT:    and $1, $25, $4
+; MIPS32O0-NEXT:    nor $1, $zero, $1
+; MIPS32O0-NEXT:    sc $1, 0($2)
+; MIPS32O0-NEXT:    beqz $1, $BB5_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    move $2, $25
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicLoadNand32:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    lw $1, %got(x)($1)
+; MIPS32R2-NEXT:  $BB5_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $2, 0($1)
+; MIPS32R2-NEXT:    and $3, $2, $4
+; MIPS32R2-NEXT:    nor $3, $zero, $3
+; MIPS32R2-NEXT:    sc $3, 0($1)
+; MIPS32R2-NEXT:    beqz $3, $BB5_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    nop
+;
+; MIPS32R6-LABEL: AtomicLoadNand32:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    lw $1, %got(x)($1)
+; MIPS32R6-NEXT:  $BB5_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $2, 0($1)
+; MIPS32R6-NEXT:    and $3, $2, $4
+; MIPS32R6-NEXT:    nor $3, $zero, $3
+; MIPS32R6-NEXT:    sc $3, 0($1)
+; MIPS32R6-NEXT:    beqzc $3, $BB5_1
+; MIPS32R6-NEXT:    nop
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    jrc $ra
+;
+; MIPS32R6O0-LABEL: AtomicLoadNand32:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32R6O0-NEXT:  $BB5_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $1, 0($2)
+; MIPS32R6O0-NEXT:    and $3, $1, $4
+; MIPS32R6O0-NEXT:    nor $3, $zero, $3
+; MIPS32R6O0-NEXT:    sc $3, 0($2)
+; MIPS32R6O0-NEXT:    beqzc $3, $BB5_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    move $2, $1
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicLoadNand32:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand32)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand32)))
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB5_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $2, 0($1)
+; MIPS4-NEXT:    and $3, $2, $4
+; MIPS4-NEXT:    nor $3, $zero, $3
+; MIPS4-NEXT:    sc $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB5_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: AtomicLoadNand32:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand32)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand32)))
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB5_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $2, 0($1)
+; MIPS64-NEXT:    and $3, $2, $4
+; MIPS64-NEXT:    nor $3, $zero, $3
+; MIPS64-NEXT:    sc $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB5_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: AtomicLoadNand32:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand32)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand32)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB5_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $2, 0($1)
+; MIPS64R2-NEXT:    and $3, $2, $4
+; MIPS64R2-NEXT:    nor $3, $zero, $3
+; MIPS64R2-NEXT:    sc $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB5_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicLoadNand32:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand32)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand32)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB5_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $2, 0($1)
+; MIPS64R6-NEXT:    and $3, $2, $4
+; MIPS64R6-NEXT:    nor $3, $zero, $3
+; MIPS64R6-NEXT:    sc $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB5_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicLoadNand32:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand32)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand32)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB5_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $3, 0($1)
+; MIPS64R6O0-NEXT:    and $5, $3, $2
+; MIPS64R6O0-NEXT:    nor $5, $zero, $5
+; MIPS64R6O0-NEXT:    sc $5, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $5, .LBB5_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    move $2, $3
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicLoadNand32:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    lw $1, %got(x)($2)
+; MM32-NEXT:  $BB5_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $2, 0($1)
+; MM32-NEXT:    and $3, $2, $4
+; MM32-NEXT:    nor $3, $zero, $3
+; MM32-NEXT:    sc $3, 0($1)
+; MM32-NEXT:    beqzc $3, $BB5_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicLoadNand32:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    lw $1, %got(x)($1)
+; O1-NEXT:  $BB5_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $2, 0($1)
+; O1-NEXT:    and $3, $2, $4
+; O1-NEXT:    nor $3, $zero, $3
+; O1-NEXT:    sc $3, 0($1)
+; O1-NEXT:    beqz $3, $BB5_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: AtomicLoadNand32:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    lw $1, %got(x)($1)
+; O2-NEXT:  $BB5_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $2, 0($1)
+; O2-NEXT:    and $3, $2, $4
+; O2-NEXT:    nor $3, $zero, $3
+; O2-NEXT:    sc $3, 0($1)
+; O2-NEXT:    beqz $3, $BB5_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: AtomicLoadNand32:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    lw $1, %got(x)($1)
+; O3-NEXT:  $BB5_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $2, 0($1)
+; O3-NEXT:    and $3, $2, $4
+; O3-NEXT:    nor $3, $zero, $3
+; O3-NEXT:    sc $3, 0($1)
+; O3-NEXT:    beqz $3, $BB5_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS32EB-LABEL: AtomicLoadNand32:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    lw $1, %got(x)($1)
+; MIPS32EB-NEXT:  $BB5_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $2, 0($1)
+; MIPS32EB-NEXT:    and $3, $2, $4
+; MIPS32EB-NEXT:    nor $3, $zero, $3
+; MIPS32EB-NEXT:    sc $3, 0($1)
+; MIPS32EB-NEXT:    beqz $3, $BB5_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    nop
+entry:
+  %0 = atomicrmw nand i32* @x, i32 %incr monotonic
+  ret i32 %0
 
-; ALL:       [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; ALL:           ll      $[[R1:[0-9]+]], 0($[[R0]])
-; ALL:           and     $[[R3:[0-9]+]], $[[R1]], $4
-; ALL:           nor     $[[R2:[0-9]+]], $zero, $[[R3]]
-; ALL:           sc      $[[R2]], 0($[[R0]])
-; NOT-MICROMIPS: beqz    $[[R2]], [[BB0]]
-; MICROMIPS:     beqzc   $[[R2]], [[BB0]]
-; MIPSR6:        beqzc   $[[R2]], [[BB0]]
 }
 
 define i32 @AtomicSwap32(i32 signext %newval) nounwind {
+; MIPS32-LABEL: AtomicSwap32:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addiu $sp, $sp, -8
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    sw $4, 4($sp)
+; MIPS32-NEXT:    lw $1, %got(x)($1)
+; MIPS32-NEXT:  $BB6_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $2, 0($1)
+; MIPS32-NEXT:    move $3, $4
+; MIPS32-NEXT:    sc $3, 0($1)
+; MIPS32-NEXT:    beqz $3, $BB6_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    addiu $sp, $sp, 8
+;
+; MIPS32O0-LABEL: AtomicSwap32:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    sw $4, 4($sp)
+; MIPS32O0-NEXT:    lw $4, 4($sp)
+; MIPS32O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32O0-NEXT:  $BB6_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $25, 0($2)
+; MIPS32O0-NEXT:    move $1, $4
+; MIPS32O0-NEXT:    sc $1, 0($2)
+; MIPS32O0-NEXT:    beqz $1, $BB6_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    move $2, $25
+; MIPS32O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicSwap32:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addiu $sp, $sp, -8
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    sw $4, 4($sp)
+; MIPS32R2-NEXT:    lw $1, %got(x)($1)
+; MIPS32R2-NEXT:  $BB6_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $2, 0($1)
+; MIPS32R2-NEXT:    move $3, $4
+; MIPS32R2-NEXT:    sc $3, 0($1)
+; MIPS32R2-NEXT:    beqz $3, $BB6_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    addiu $sp, $sp, 8
+;
+; MIPS32R6-LABEL: AtomicSwap32:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    sw $4, 4($sp)
+; MIPS32R6-NEXT:    lw $1, %got(x)($1)
+; MIPS32R6-NEXT:  $BB6_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $2, 0($1)
+; MIPS32R6-NEXT:    move $3, $4
+; MIPS32R6-NEXT:    sc $3, 0($1)
+; MIPS32R6-NEXT:    beqzc $3, $BB6_1
+; MIPS32R6-NEXT:    nop
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    jr $ra
+; MIPS32R6-NEXT:    addiu $sp, $sp, 8
+;
+; MIPS32R6O0-LABEL: AtomicSwap32:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    sw $4, 4($sp)
+; MIPS32R6O0-NEXT:    lw $4, 4($sp)
+; MIPS32R6O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32R6O0-NEXT:  $BB6_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $1, 0($2)
+; MIPS32R6O0-NEXT:    move $3, $4
+; MIPS32R6O0-NEXT:    sc $3, 0($2)
+; MIPS32R6O0-NEXT:    beqzc $3, $BB6_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    move $2, $1
+; MIPS32R6O0-NEXT:    sw $25, 0($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicSwap32:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    daddiu $sp, $sp, -16
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap32)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap32)))
+; MIPS4-NEXT:    sw $4, 12($sp)
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB6_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $2, 0($1)
+; MIPS4-NEXT:    move $3, $4
+; MIPS4-NEXT:    sc $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB6_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64-LABEL: AtomicSwap32:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    daddiu $sp, $sp, -16
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap32)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap32)))
+; MIPS64-NEXT:    sw $4, 12($sp)
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB6_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $2, 0($1)
+; MIPS64-NEXT:    move $3, $4
+; MIPS64-NEXT:    sc $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB6_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64R2-LABEL: AtomicSwap32:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap32)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap32)))
+; MIPS64R2-NEXT:    sw $4, 12($sp)
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB6_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $2, 0($1)
+; MIPS64R2-NEXT:    move $3, $4
+; MIPS64R2-NEXT:    sc $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB6_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64R6-LABEL: AtomicSwap32:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap32)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap32)))
+; MIPS64R6-NEXT:    sw $4, 12($sp)
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB6_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $2, 0($1)
+; MIPS64R6-NEXT:    move $3, $4
+; MIPS64R6-NEXT:    sc $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB6_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jr $ra
+; MIPS64R6-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64R6O0-LABEL: AtomicSwap32:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap32)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap32)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    sw $2, 12($sp)
+; MIPS64R6O0-NEXT:    lw $2, 12($sp)
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB6_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $3, 0($1)
+; MIPS64R6O0-NEXT:    move $5, $2
+; MIPS64R6O0-NEXT:    sc $5, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $5, .LBB6_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    move $2, $3
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicSwap32:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addiu $sp, $sp, -8
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    sw $4, 4($sp)
+; MM32-NEXT:    lw $1, %got(x)($2)
+; MM32-NEXT:  $BB6_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $2, 0($1)
+; MM32-NEXT:    move $3, $4
+; MM32-NEXT:    sc $3, 0($1)
+; MM32-NEXT:    beqzc $3, $BB6_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    addiusp 8
+; MM32-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicSwap32:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addiu $sp, $sp, -8
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    sw $4, 4($sp)
+; O1-NEXT:    lw $1, %got(x)($1)
+; O1-NEXT:  $BB6_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $2, 0($1)
+; O1-NEXT:    move $3, $4
+; O1-NEXT:    sc $3, 0($1)
+; O1-NEXT:    beqz $3, $BB6_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    addiu $sp, $sp, 8
+;
+; O2-LABEL: AtomicSwap32:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addiu $sp, $sp, -8
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    sw $4, 4($sp)
+; O2-NEXT:    lw $1, %got(x)($1)
+; O2-NEXT:  $BB6_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $2, 0($1)
+; O2-NEXT:    move $3, $4
+; O2-NEXT:    sc $3, 0($1)
+; O2-NEXT:    beqz $3, $BB6_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    addiu $sp, $sp, 8
+;
+; O3-LABEL: AtomicSwap32:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addiu $sp, $sp, -8
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    sw $4, 4($sp)
+; O3-NEXT:    lw $1, %got(x)($1)
+; O3-NEXT:  $BB6_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $2, 0($1)
+; O3-NEXT:    move $3, $4
+; O3-NEXT:    sc $3, 0($1)
+; O3-NEXT:    beqz $3, $BB6_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    addiu $sp, $sp, 8
+;
+; MIPS32EB-LABEL: AtomicSwap32:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addiu $sp, $sp, -8
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    sw $4, 4($sp)
+; MIPS32EB-NEXT:    lw $1, %got(x)($1)
+; MIPS32EB-NEXT:  $BB6_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $2, 0($1)
+; MIPS32EB-NEXT:    move $3, $4
+; MIPS32EB-NEXT:    sc $3, 0($1)
+; MIPS32EB-NEXT:    beqz $3, $BB6_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    addiu $sp, $sp, 8
 entry:
   %newval.addr = alloca i32, align 4
   store i32 %newval, i32* %newval.addr, align 4
   %tmp = load i32, i32* %newval.addr, align 4
   %0 = atomicrmw xchg i32* @x, i32 %tmp monotonic
   ret i32 %0
 
-; ALL-LABEL: AtomicSwap32:
-
-; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(x)
-; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(x)
-
-; ALL:       [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; ALL:           ll      ${{[0-9]+}}, 0($[[R0]])
-; ALL:           sc      $[[R2:[0-9]+]], 0($[[R0]])
-; NOT-MICROMIPS: beqz    $[[R2]], [[BB0]]
-; MICROMIPS:     beqzc   $[[R2]], [[BB0]]
-; MIPSR6:        beqzc   $[[R2]], [[BB0]]
 }
 
 define i32 @AtomicCmpSwap32(i32 signext %oldval, i32 signext %newval) nounwind {
+; MIPS32-LABEL: AtomicCmpSwap32:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addiu $sp, $sp, -8
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    sw $5, 4($sp)
+; MIPS32-NEXT:    lw $1, %got(x)($1)
+; MIPS32-NEXT:  $BB7_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $2, 0($1)
+; MIPS32-NEXT:    bne $2, $4, $BB7_3
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS32-NEXT:    move $3, $5
+; MIPS32-NEXT:    sc $3, 0($1)
+; MIPS32-NEXT:    beqz $3, $BB7_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  $BB7_3: # %entry
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    addiu $sp, $sp, 8
+;
+; MIPS32O0-LABEL: AtomicCmpSwap32:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addiu $sp, $sp, -16
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    sw $5, 12($sp)
+; MIPS32O0-NEXT:    lw $5, 12($sp)
+; MIPS32O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32O0-NEXT:    lw $25, 8($sp) # 4-byte Folded Reload
+; MIPS32O0-NEXT:    move $1, $4
+; MIPS32O0-NEXT:  $BB7_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $3, 0($2)
+; MIPS32O0-NEXT:    bne $3, $1, $BB7_3
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS32O0-NEXT:    move $6, $5
+; MIPS32O0-NEXT:    sc $6, 0($2)
+; MIPS32O0-NEXT:    beqz $6, $BB7_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  $BB7_3: # %entry
+; MIPS32O0-NEXT:    xor $1, $3, $4
+; MIPS32O0-NEXT:    sltiu $1, $1, 1
+; MIPS32O0-NEXT:    move $2, $3
+; MIPS32O0-NEXT:    sw $3, 8($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:    sw $1, 0($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:    addiu $sp, $sp, 16
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicCmpSwap32:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addiu $sp, $sp, -8
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    sw $5, 4($sp)
+; MIPS32R2-NEXT:    lw $1, %got(x)($1)
+; MIPS32R2-NEXT:  $BB7_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $2, 0($1)
+; MIPS32R2-NEXT:    bne $2, $4, $BB7_3
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS32R2-NEXT:    move $3, $5
+; MIPS32R2-NEXT:    sc $3, 0($1)
+; MIPS32R2-NEXT:    beqz $3, $BB7_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  $BB7_3: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    addiu $sp, $sp, 8
+;
+; MIPS32R6-LABEL: AtomicCmpSwap32:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    sw $5, 4($sp)
+; MIPS32R6-NEXT:    lw $1, %got(x)($1)
+; MIPS32R6-NEXT:  $BB7_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $2, 0($1)
+; MIPS32R6-NEXT:    bnec $2, $4, $BB7_3
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS32R6-NEXT:    move $3, $5
+; MIPS32R6-NEXT:    sc $3, 0($1)
+; MIPS32R6-NEXT:    beqzc $3, $BB7_1
+; MIPS32R6-NEXT:    nop
+; MIPS32R6-NEXT:  $BB7_3: # %entry
+; MIPS32R6-NEXT:    jr $ra
+; MIPS32R6-NEXT:    addiu $sp, $sp, 8
+;
+; MIPS32R6O0-LABEL: AtomicCmpSwap32:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -24
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $5
+; MIPS32R6O0-NEXT:    move $1, $4
+; MIPS32R6O0-NEXT:    sw $5, 20($sp)
+; MIPS32R6O0-NEXT:    lw $5, 20($sp)
+; MIPS32R6O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32R6O0-NEXT:    lw $3, 16($sp) # 4-byte Folded Reload
+; MIPS32R6O0-NEXT:  $BB7_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $6, 0($2)
+; MIPS32R6O0-NEXT:    bnec $6, $4, $BB7_3
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS32R6O0-NEXT:    move $7, $5
+; MIPS32R6O0-NEXT:    sc $7, 0($2)
+; MIPS32R6O0-NEXT:    beqzc $7, $BB7_1
+; MIPS32R6O0-NEXT:  $BB7_3: # %entry
+; MIPS32R6O0-NEXT:    move $2, $6
+; MIPS32R6O0-NEXT:    sw $6, 16($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $1, 12($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $3, 8($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 24
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicCmpSwap32:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    daddiu $sp, $sp, -16
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap32)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap32)))
+; MIPS4-NEXT:    sw $5, 12($sp)
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB7_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $2, 0($1)
+; MIPS4-NEXT:    bne $2, $4, .LBB7_3
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS4-NEXT:    move $3, $5
+; MIPS4-NEXT:    sc $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB7_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  .LBB7_3: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64-LABEL: AtomicCmpSwap32:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    daddiu $sp, $sp, -16
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap32)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap32)))
+; MIPS64-NEXT:    sw $5, 12($sp)
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB7_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $2, 0($1)
+; MIPS64-NEXT:    bne $2, $4, .LBB7_3
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS64-NEXT:    move $3, $5
+; MIPS64-NEXT:    sc $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB7_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  .LBB7_3: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64R2-LABEL: AtomicCmpSwap32:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap32)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap32)))
+; MIPS64R2-NEXT:    sw $5, 12($sp)
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB7_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $2, 0($1)
+; MIPS64R2-NEXT:    bne $2, $4, .LBB7_3
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS64R2-NEXT:    move $3, $5
+; MIPS64R2-NEXT:    sc $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB7_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  .LBB7_3: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64R6-LABEL: AtomicCmpSwap32:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap32)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap32)))
+; MIPS64R6-NEXT:    sw $5, 12($sp)
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB7_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $2, 0($1)
+; MIPS64R6-NEXT:    bnec $2, $4, .LBB7_3
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS64R6-NEXT:    move $3, $5
+; MIPS64R6-NEXT:    sc $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB7_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  .LBB7_3: # %entry
+; MIPS64R6-NEXT:    jr $ra
+; MIPS64R6-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64R6O0-LABEL: AtomicCmpSwap32:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap32)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap32)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    move $3, $5
+; MIPS64R6O0-NEXT:    sw $3, 12($sp)
+; MIPS64R6O0-NEXT:    lw $3, 12($sp)
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:    lw $6, 8($sp) # 4-byte Folded Reload
+; MIPS64R6O0-NEXT:  .LBB7_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $7, 0($1)
+; MIPS64R6O0-NEXT:    bnec $7, $2, .LBB7_3
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS64R6O0-NEXT:    move $8, $3
+; MIPS64R6O0-NEXT:    sc $8, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $8, .LBB7_1
+; MIPS64R6O0-NEXT:  .LBB7_3: # %entry
+; MIPS64R6O0-NEXT:    move $2, $7
+; MIPS64R6O0-NEXT:    sw $7, 8($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:    sw $6, 4($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicCmpSwap32:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addiu $sp, $sp, -8
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    sw $5, 4($sp)
+; MM32-NEXT:    lw $1, %got(x)($2)
+; MM32-NEXT:  $BB7_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $2, 0($1)
+; MM32-NEXT:    bne $2, $4, $BB7_3
+; MM32-NEXT:    nop
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MM32-NEXT:    move $3, $5
+; MM32-NEXT:    sc $3, 0($1)
+; MM32-NEXT:    beqzc $3, $BB7_1
+; MM32-NEXT:  $BB7_3: # %entry
+; MM32-NEXT:    addiusp 8
+; MM32-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicCmpSwap32:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addiu $sp, $sp, -8
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    sw $5, 4($sp)
+; O1-NEXT:    lw $1, %got(x)($1)
+; O1-NEXT:  $BB7_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $2, 0($1)
+; O1-NEXT:    bne $2, $4, $BB7_3
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; O1-NEXT:    move $3, $5
+; O1-NEXT:    sc $3, 0($1)
+; O1-NEXT:    beqz $3, $BB7_1
+; O1-NEXT:    nop
+; O1-NEXT:  $BB7_3: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    addiu $sp, $sp, 8
+;
+; O2-LABEL: AtomicCmpSwap32:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addiu $sp, $sp, -8
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    sw $5, 4($sp)
+; O2-NEXT:    lw $1, %got(x)($1)
+; O2-NEXT:  $BB7_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $2, 0($1)
+; O2-NEXT:    bne $2, $4, $BB7_3
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; O2-NEXT:    move $3, $5
+; O2-NEXT:    sc $3, 0($1)
+; O2-NEXT:    beqz $3, $BB7_1
+; O2-NEXT:    nop
+; O2-NEXT:  $BB7_3: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    addiu $sp, $sp, 8
+;
+; O3-LABEL: AtomicCmpSwap32:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addiu $sp, $sp, -8
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    sw $5, 4($sp)
+; O3-NEXT:    lw $1, %got(x)($1)
+; O3-NEXT:  $BB7_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $2, 0($1)
+; O3-NEXT:    bne $2, $4, $BB7_3
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; O3-NEXT:    move $3, $5
+; O3-NEXT:    sc $3, 0($1)
+; O3-NEXT:    beqz $3, $BB7_1
+; O3-NEXT:    nop
+; O3-NEXT:  $BB7_3: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    addiu $sp, $sp, 8
+;
+; MIPS32EB-LABEL: AtomicCmpSwap32:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addiu $sp, $sp, -8
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    sw $5, 4($sp)
+; MIPS32EB-NEXT:    lw $1, %got(x)($1)
+; MIPS32EB-NEXT:  $BB7_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $2, 0($1)
+; MIPS32EB-NEXT:    bne $2, $4, $BB7_3
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS32EB-NEXT:    move $3, $5
+; MIPS32EB-NEXT:    sc $3, 0($1)
+; MIPS32EB-NEXT:    beqz $3, $BB7_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  $BB7_3: # %entry
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    addiu $sp, $sp, 8
 entry:
   %newval.addr = alloca i32, align 4
   store i32 %newval, i32* %newval.addr, align 4
   %tmp = load i32, i32* %newval.addr, align 4
   %0 = cmpxchg i32* @x, i32 %oldval, i32 %tmp monotonic monotonic
   %1 = extractvalue { i32, i1 } %0, 0
   ret i32 %1
 
-; ALL-LABEL: AtomicCmpSwap32:
-
-; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(x)
-; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(x)(
-
-; ALL:       [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; ALL:           ll      $2, 0($[[R0]])
-; NOT-MICROMIPS: bne     $2, $4, [[BB1:(\$|\.L)[A-Z_0-9]+]]
-; MICROMIPS:     bne     $2, $4, [[BB1:(\$|\.L)[A-Z_0-9]+]]
-; MIPSR6:        bnec    $2, $4, [[BB1:(\$|\.L)[A-Z_0-9]+]]
-; ALL:           sc      $[[R2:[0-9]+]], 0($[[R0]])
-; NOT-MICROMIPS: beqz    $[[R2]], [[BB0]]
-; MICROMIPS:     beqzc   $[[R2]], [[BB0]]
-; MIPSR6:        beqzc   $[[R2]], [[BB0]]
-; ALL:       [[BB1]]:
 }
 
-
-
 @y = common global i8 0, align 1
 
 define signext i8 @AtomicLoadAdd8(i8 signext %incr) nounwind {
+; MIPS32-LABEL: AtomicLoadAdd8:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    lw $1, %got(y)($1)
+; MIPS32-NEXT:    addiu $2, $zero, -4
+; MIPS32-NEXT:    and $2, $1, $2
+; MIPS32-NEXT:    andi $1, $1, 3
+; MIPS32-NEXT:    sll $3, $1, 3
+; MIPS32-NEXT:    ori $1, $zero, 255
+; MIPS32-NEXT:    sllv $5, $1, $3
+; MIPS32-NEXT:    nor $6, $zero, $5
+; MIPS32-NEXT:    sllv $4, $4, $3
+; MIPS32-NEXT:  $BB8_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $7, 0($2)
+; MIPS32-NEXT:    addu $8, $7, $4
+; MIPS32-NEXT:    and $8, $8, $5
+; MIPS32-NEXT:    and $9, $7, $6
+; MIPS32-NEXT:    or $9, $9, $8
+; MIPS32-NEXT:    sc $9, 0($2)
+; MIPS32-NEXT:    beqz $9, $BB8_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    and $1, $7, $5
+; MIPS32-NEXT:    srlv $1, $1, $3
+; MIPS32-NEXT:    sll $1, $1, 24
+; MIPS32-NEXT:    sra $1, $1, 24
+; MIPS32-NEXT:  # %bb.3: # %entry
+; MIPS32-NEXT:    sll $1, $1, 24
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    sra $2, $1, 24
+;
+; MIPS32O0-LABEL: AtomicLoadAdd8:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    lw $2, %got(y)($2)
+; MIPS32O0-NEXT:    addiu $25, $zero, -4
+; MIPS32O0-NEXT:    and $25, $2, $25
+; MIPS32O0-NEXT:    andi $2, $2, 3
+; MIPS32O0-NEXT:    sll $2, $2, 3
+; MIPS32O0-NEXT:    ori $1, $zero, 255
+; MIPS32O0-NEXT:    sllv $1, $1, $2
+; MIPS32O0-NEXT:    nor $3, $zero, $1
+; MIPS32O0-NEXT:    sllv $4, $4, $2
+; MIPS32O0-NEXT:  $BB8_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $6, 0($25)
+; MIPS32O0-NEXT:    addu $7, $6, $4
+; MIPS32O0-NEXT:    and $7, $7, $1
+; MIPS32O0-NEXT:    and $8, $6, $3
+; MIPS32O0-NEXT:    or $8, $8, $7
+; MIPS32O0-NEXT:    sc $8, 0($25)
+; MIPS32O0-NEXT:    beqz $8, $BB8_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    and $5, $6, $1
+; MIPS32O0-NEXT:    srlv $5, $5, $2
+; MIPS32O0-NEXT:    sll $5, $5, 24
+; MIPS32O0-NEXT:    sra $5, $5, 24
+; MIPS32O0-NEXT:  # %bb.3: # %entry
+; MIPS32O0-NEXT:    sw $5, 4($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:  # %bb.4: # %entry
+; MIPS32O0-NEXT:    lw $1, 4($sp) # 4-byte Folded Reload
+; MIPS32O0-NEXT:    sll $2, $1, 24
+; MIPS32O0-NEXT:    sra $2, $2, 24
+; MIPS32O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicLoadAdd8:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    lw $1, %got(y)($1)
+; MIPS32R2-NEXT:    addiu $2, $zero, -4
+; MIPS32R2-NEXT:    and $2, $1, $2
+; MIPS32R2-NEXT:    andi $1, $1, 3
+; MIPS32R2-NEXT:    sll $3, $1, 3
+; MIPS32R2-NEXT:    ori $1, $zero, 255
+; MIPS32R2-NEXT:    sllv $5, $1, $3
+; MIPS32R2-NEXT:    nor $6, $zero, $5
+; MIPS32R2-NEXT:    sllv $4, $4, $3
+; MIPS32R2-NEXT:  $BB8_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $7, 0($2)
+; MIPS32R2-NEXT:    addu $8, $7, $4
+; MIPS32R2-NEXT:    and $8, $8, $5
+; MIPS32R2-NEXT:    and $9, $7, $6
+; MIPS32R2-NEXT:    or $9, $9, $8
+; MIPS32R2-NEXT:    sc $9, 0($2)
+; MIPS32R2-NEXT:    beqz $9, $BB8_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    and $1, $7, $5
+; MIPS32R2-NEXT:    srlv $1, $1, $3
+; MIPS32R2-NEXT:    seb $1, $1
+; MIPS32R2-NEXT:  # %bb.3: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    seb $2, $1
+;
+; MIPS32R6-LABEL: AtomicLoadAdd8:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    lw $1, %got(y)($1)
+; MIPS32R6-NEXT:    addiu $2, $zero, -4
+; MIPS32R6-NEXT:    and $2, $1, $2
+; MIPS32R6-NEXT:    andi $1, $1, 3
+; MIPS32R6-NEXT:    sll $3, $1, 3
+; MIPS32R6-NEXT:    ori $1, $zero, 255
+; MIPS32R6-NEXT:    sllv $5, $1, $3
+; MIPS32R6-NEXT:    nor $6, $zero, $5
+; MIPS32R6-NEXT:    sllv $4, $4, $3
+; MIPS32R6-NEXT:  $BB8_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $7, 0($2)
+; MIPS32R6-NEXT:    addu $8, $7, $4
+; MIPS32R6-NEXT:    and $8, $8, $5
+; MIPS32R6-NEXT:    and $9, $7, $6
+; MIPS32R6-NEXT:    or $9, $9, $8
+; MIPS32R6-NEXT:    sc $9, 0($2)
+; MIPS32R6-NEXT:    beqzc $9, $BB8_1
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    and $1, $7, $5
+; MIPS32R6-NEXT:    srlv $1, $1, $3
+; MIPS32R6-NEXT:    seb $1, $1
+; MIPS32R6-NEXT:  # %bb.3: # %entry
+; MIPS32R6-NEXT:    jr $ra
+; MIPS32R6-NEXT:    seb $2, $1
+;
+; MIPS32R6O0-LABEL: AtomicLoadAdd8:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    lw $2, %got(y)($2)
+; MIPS32R6O0-NEXT:    addiu $1, $zero, -4
+; MIPS32R6O0-NEXT:    and $1, $2, $1
+; MIPS32R6O0-NEXT:    andi $2, $2, 3
+; MIPS32R6O0-NEXT:    sll $2, $2, 3
+; MIPS32R6O0-NEXT:    ori $3, $zero, 255
+; MIPS32R6O0-NEXT:    sllv $3, $3, $2
+; MIPS32R6O0-NEXT:    nor $5, $zero, $3
+; MIPS32R6O0-NEXT:    sllv $4, $4, $2
+; MIPS32R6O0-NEXT:  $BB8_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $7, 0($1)
+; MIPS32R6O0-NEXT:    addu $8, $7, $4
+; MIPS32R6O0-NEXT:    and $8, $8, $3
+; MIPS32R6O0-NEXT:    and $9, $7, $5
+; MIPS32R6O0-NEXT:    or $9, $9, $8
+; MIPS32R6O0-NEXT:    sc $9, 0($1)
+; MIPS32R6O0-NEXT:    beqzc $9, $BB8_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    and $6, $7, $3
+; MIPS32R6O0-NEXT:    srlv $6, $6, $2
+; MIPS32R6O0-NEXT:    seb $6, $6
+; MIPS32R6O0-NEXT:  # %bb.3: # %entry
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $6, 0($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:  # %bb.4: # %entry
+; MIPS32R6O0-NEXT:    lw $1, 0($sp) # 4-byte Folded Reload
+; MIPS32R6O0-NEXT:    seb $2, $1
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicLoadAdd8:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd8)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd8)))
+; MIPS4-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS4-NEXT:    daddiu $2, $zero, -4
+; MIPS4-NEXT:    and $2, $1, $2
+; MIPS4-NEXT:    andi $1, $1, 3
+; MIPS4-NEXT:    sll $3, $1, 3
+; MIPS4-NEXT:    ori $1, $zero, 255
+; MIPS4-NEXT:    sllv $5, $1, $3
+; MIPS4-NEXT:    nor $6, $zero, $5
+; MIPS4-NEXT:    sllv $4, $4, $3
+; MIPS4-NEXT:  .LBB8_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $7, 0($2)
+; MIPS4-NEXT:    addu $8, $7, $4
+; MIPS4-NEXT:    and $8, $8, $5
+; MIPS4-NEXT:    and $9, $7, $6
+; MIPS4-NEXT:    or $9, $9, $8
+; MIPS4-NEXT:    sc $9, 0($2)
+; MIPS4-NEXT:    beqz $9, .LBB8_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    and $1, $7, $5
+; MIPS4-NEXT:    srlv $1, $1, $3
+; MIPS4-NEXT:    sll $1, $1, 24
+; MIPS4-NEXT:    sra $1, $1, 24
+; MIPS4-NEXT:  # %bb.3: # %entry
+; MIPS4-NEXT:    sll $1, $1, 24
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    sra $2, $1, 24
+;
+; MIPS64-LABEL: AtomicLoadAdd8:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd8)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd8)))
+; MIPS64-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64-NEXT:    daddiu $2, $zero, -4
+; MIPS64-NEXT:    and $2, $1, $2
+; MIPS64-NEXT:    andi $1, $1, 3
+; MIPS64-NEXT:    sll $3, $1, 3
+; MIPS64-NEXT:    ori $1, $zero, 255
+; MIPS64-NEXT:    sllv $5, $1, $3
+; MIPS64-NEXT:    nor $6, $zero, $5
+; MIPS64-NEXT:    sllv $4, $4, $3
+; MIPS64-NEXT:  .LBB8_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $7, 0($2)
+; MIPS64-NEXT:    addu $8, $7, $4
+; MIPS64-NEXT:    and $8, $8, $5
+; MIPS64-NEXT:    and $9, $7, $6
+; MIPS64-NEXT:    or $9, $9, $8
+; MIPS64-NEXT:    sc $9, 0($2)
+; MIPS64-NEXT:    beqz $9, .LBB8_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    and $1, $7, $5
+; MIPS64-NEXT:    srlv $1, $1, $3
+; MIPS64-NEXT:    sll $1, $1, 24
+; MIPS64-NEXT:    sra $1, $1, 24
+; MIPS64-NEXT:  # %bb.3: # %entry
+; MIPS64-NEXT:    sll $1, $1, 24
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    sra $2, $1, 24
+;
+; MIPS64R2-LABEL: AtomicLoadAdd8:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd8)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd8)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R2-NEXT:    daddiu $2, $zero, -4
+; MIPS64R2-NEXT:    and $2, $1, $2
+; MIPS64R2-NEXT:    andi $1, $1, 3
+; MIPS64R2-NEXT:    sll $3, $1, 3
+; MIPS64R2-NEXT:    ori $1, $zero, 255
+; MIPS64R2-NEXT:    sllv $5, $1, $3
+; MIPS64R2-NEXT:    nor $6, $zero, $5
+; MIPS64R2-NEXT:    sllv $4, $4, $3
+; MIPS64R2-NEXT:  .LBB8_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $7, 0($2)
+; MIPS64R2-NEXT:    addu $8, $7, $4
+; MIPS64R2-NEXT:    and $8, $8, $5
+; MIPS64R2-NEXT:    and $9, $7, $6
+; MIPS64R2-NEXT:    or $9, $9, $8
+; MIPS64R2-NEXT:    sc $9, 0($2)
+; MIPS64R2-NEXT:    beqz $9, .LBB8_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    and $1, $7, $5
+; MIPS64R2-NEXT:    srlv $1, $1, $3
+; MIPS64R2-NEXT:    seb $1, $1
+; MIPS64R2-NEXT:  # %bb.3: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    seb $2, $1
+;
+; MIPS64R6-LABEL: AtomicLoadAdd8:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd8)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd8)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R6-NEXT:    daddiu $2, $zero, -4
+; MIPS64R6-NEXT:    and $2, $1, $2
+; MIPS64R6-NEXT:    andi $1, $1, 3
+; MIPS64R6-NEXT:    sll $3, $1, 3
+; MIPS64R6-NEXT:    ori $1, $zero, 255
+; MIPS64R6-NEXT:    sllv $5, $1, $3
+; MIPS64R6-NEXT:    nor $6, $zero, $5
+; MIPS64R6-NEXT:    sllv $4, $4, $3
+; MIPS64R6-NEXT:  .LBB8_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $7, 0($2)
+; MIPS64R6-NEXT:    addu $8, $7, $4
+; MIPS64R6-NEXT:    and $8, $8, $5
+; MIPS64R6-NEXT:    and $9, $7, $6
+; MIPS64R6-NEXT:    or $9, $9, $8
+; MIPS64R6-NEXT:    sc $9, 0($2)
+; MIPS64R6-NEXT:    beqzc $9, .LBB8_1
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    and $1, $7, $5
+; MIPS64R6-NEXT:    srlv $1, $1, $3
+; MIPS64R6-NEXT:    seb $1, $1
+; MIPS64R6-NEXT:  # %bb.3: # %entry
+; MIPS64R6-NEXT:    jr $ra
+; MIPS64R6-NEXT:    seb $2, $1
+;
+; MIPS64R6O0-LABEL: AtomicLoadAdd8:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd8)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd8)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R6O0-NEXT:    daddiu $4, $zero, -4
+; MIPS64R6O0-NEXT:    and $4, $1, $4
+; MIPS64R6O0-NEXT:    andi $3, $1, 3
+; MIPS64R6O0-NEXT:    xori $3, $3, 3
+; MIPS64R6O0-NEXT:    sll $3, $3, 3
+; MIPS64R6O0-NEXT:    ori $5, $zero, 255
+; MIPS64R6O0-NEXT:    sllv $5, $5, $3
+; MIPS64R6O0-NEXT:    nor $6, $zero, $5
+; MIPS64R6O0-NEXT:    sllv $2, $2, $3
+; MIPS64R6O0-NEXT:  .LBB8_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $8, 0($4)
+; MIPS64R6O0-NEXT:    addu $9, $8, $2
+; MIPS64R6O0-NEXT:    and $9, $9, $5
+; MIPS64R6O0-NEXT:    and $10, $8, $6
+; MIPS64R6O0-NEXT:    or $10, $10, $9
+; MIPS64R6O0-NEXT:    sc $10, 0($4)
+; MIPS64R6O0-NEXT:    beqzc $10, .LBB8_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    and $7, $8, $5
+; MIPS64R6O0-NEXT:    srlv $7, $7, $3
+; MIPS64R6O0-NEXT:    seb $7, $7
+; MIPS64R6O0-NEXT:  # %bb.3: # %entry
+; MIPS64R6O0-NEXT:    sw $7, 12($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:  # %bb.4: # %entry
+; MIPS64R6O0-NEXT:    lw $1, 12($sp) # 4-byte Folded Reload
+; MIPS64R6O0-NEXT:    seb $2, $1
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicLoadAdd8:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    lw $1, %got(y)($2)
+; MM32-NEXT:    addiu $2, $zero, -4
+; MM32-NEXT:    and $2, $1, $2
+; MM32-NEXT:    andi $1, $1, 3
+; MM32-NEXT:    sll $3, $1, 3
+; MM32-NEXT:    ori $1, $zero, 255
+; MM32-NEXT:    sllv $5, $1, $3
+; MM32-NEXT:    nor $6, $zero, $5
+; MM32-NEXT:    sllv $4, $4, $3
+; MM32-NEXT:  $BB8_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $7, 0($2)
+; MM32-NEXT:    addu $8, $7, $4
+; MM32-NEXT:    and $8, $8, $5
+; MM32-NEXT:    and $9, $7, $6
+; MM32-NEXT:    or $9, $9, $8
+; MM32-NEXT:    sc $9, 0($2)
+; MM32-NEXT:    beqzc $9, $BB8_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    and $1, $7, $5
+; MM32-NEXT:    srlv $1, $1, $3
+; MM32-NEXT:    seb $1, $1
+; MM32-NEXT:  # %bb.3: # %entry
+; MM32-NEXT:    jr $ra
+; MM32-NEXT:    seb $2, $1
+;
+; O1-LABEL: AtomicLoadAdd8:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    lw $1, %got(y)($1)
+; O1-NEXT:    addiu $2, $zero, -4
+; O1-NEXT:    and $2, $1, $2
+; O1-NEXT:    andi $1, $1, 3
+; O1-NEXT:    sll $3, $1, 3
+; O1-NEXT:    ori $1, $zero, 255
+; O1-NEXT:    sllv $5, $1, $3
+; O1-NEXT:    nor $6, $zero, $5
+; O1-NEXT:    sllv $4, $4, $3
+; O1-NEXT:  $BB8_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $7, 0($2)
+; O1-NEXT:    addu $8, $7, $4
+; O1-NEXT:    and $8, $8, $5
+; O1-NEXT:    and $9, $7, $6
+; O1-NEXT:    or $9, $9, $8
+; O1-NEXT:    sc $9, 0($2)
+; O1-NEXT:    beqz $9, $BB8_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    and $1, $7, $5
+; O1-NEXT:    srlv $1, $1, $3
+; O1-NEXT:    sll $1, $1, 24
+; O1-NEXT:    sra $1, $1, 24
+; O1-NEXT:  # %bb.3: # %entry
+; O1-NEXT:    sll $1, $1, 24
+; O1-NEXT:    jr $ra
+; O1-NEXT:    sra $2, $1, 24
+;
+; O2-LABEL: AtomicLoadAdd8:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    lw $1, %got(y)($1)
+; O2-NEXT:    addiu $2, $zero, -4
+; O2-NEXT:    and $2, $1, $2
+; O2-NEXT:    andi $1, $1, 3
+; O2-NEXT:    sll $3, $1, 3
+; O2-NEXT:    ori $1, $zero, 255
+; O2-NEXT:    sllv $5, $1, $3
+; O2-NEXT:    nor $6, $zero, $5
+; O2-NEXT:    sllv $4, $4, $3
+; O2-NEXT:  $BB8_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $7, 0($2)
+; O2-NEXT:    addu $8, $7, $4
+; O2-NEXT:    and $8, $8, $5
+; O2-NEXT:    and $9, $7, $6
+; O2-NEXT:    or $9, $9, $8
+; O2-NEXT:    sc $9, 0($2)
+; O2-NEXT:    beqz $9, $BB8_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    and $1, $7, $5
+; O2-NEXT:    srlv $1, $1, $3
+; O2-NEXT:    sll $1, $1, 24
+; O2-NEXT:    sra $1, $1, 24
+; O2-NEXT:  # %bb.3: # %entry
+; O2-NEXT:    sll $1, $1, 24
+; O2-NEXT:    jr $ra
+; O2-NEXT:    sra $2, $1, 24
+;
+; O3-LABEL: AtomicLoadAdd8:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    addiu $2, $zero, -4
+; O3-NEXT:    lw $1, %got(y)($1)
+; O3-NEXT:    and $2, $1, $2
+; O3-NEXT:    andi $1, $1, 3
+; O3-NEXT:    sll $3, $1, 3
+; O3-NEXT:    ori $1, $zero, 255
+; O3-NEXT:    sllv $5, $1, $3
+; O3-NEXT:    sllv $4, $4, $3
+; O3-NEXT:    nor $6, $zero, $5
+; O3-NEXT:  $BB8_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $7, 0($2)
+; O3-NEXT:    addu $8, $7, $4
+; O3-NEXT:    and $8, $8, $5
+; O3-NEXT:    and $9, $7, $6
+; O3-NEXT:    or $9, $9, $8
+; O3-NEXT:    sc $9, 0($2)
+; O3-NEXT:    beqz $9, $BB8_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    and $1, $7, $5
+; O3-NEXT:    srlv $1, $1, $3
+; O3-NEXT:    sll $1, $1, 24
+; O3-NEXT:    sra $1, $1, 24
+; O3-NEXT:  # %bb.3: # %entry
+; O3-NEXT:    sll $1, $1, 24
+; O3-NEXT:    jr $ra
+; O3-NEXT:    sra $2, $1, 24
+;
+; MIPS32EB-LABEL: AtomicLoadAdd8:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    lw $1, %got(y)($1)
+; MIPS32EB-NEXT:    addiu $2, $zero, -4
+; MIPS32EB-NEXT:    and $2, $1, $2
+; MIPS32EB-NEXT:    andi $1, $1, 3
+; MIPS32EB-NEXT:    xori $1, $1, 3
+; MIPS32EB-NEXT:    sll $3, $1, 3
+; MIPS32EB-NEXT:    ori $1, $zero, 255
+; MIPS32EB-NEXT:    sllv $5, $1, $3
+; MIPS32EB-NEXT:    nor $6, $zero, $5
+; MIPS32EB-NEXT:    sllv $4, $4, $3
+; MIPS32EB-NEXT:  $BB8_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $7, 0($2)
+; MIPS32EB-NEXT:    addu $8, $7, $4
+; MIPS32EB-NEXT:    and $8, $8, $5
+; MIPS32EB-NEXT:    and $9, $7, $6
+; MIPS32EB-NEXT:    or $9, $9, $8
+; MIPS32EB-NEXT:    sc $9, 0($2)
+; MIPS32EB-NEXT:    beqz $9, $BB8_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    and $1, $7, $5
+; MIPS32EB-NEXT:    srlv $1, $1, $3
+; MIPS32EB-NEXT:    sll $1, $1, 24
+; MIPS32EB-NEXT:    sra $1, $1, 24
+; MIPS32EB-NEXT:  # %bb.3: # %entry
+; MIPS32EB-NEXT:    sll $1, $1, 24
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    sra $2, $1, 24
 entry:
   %0 = atomicrmw add i8* @y, i8 %incr monotonic
   ret i8 %0
-
-; ALL-LABEL: AtomicLoadAdd8:
-
-; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(y)
-; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(y)(
-
-; ALL:           addiu   $[[R1:[0-9]+]], $zero, -4
-; ALL:           and     $[[R2:[0-9]+]], $[[R0]], $[[R1]]
-; ALL:           andi    $[[R3:[0-9]+]], $[[R0]], 3
-; CHECK-EB:      xori    $[[R4:[0-9]+]], $[[R3]], 3
-; CHECK-EB:      sll     $[[R5:[0-9]+]], $[[R4]], 3
-; CHECK-EL:      sll     $[[R5:[0-9]+]], $[[R3]], 3
-; ALL:           ori     $[[R6:[0-9]+]], $zero, 255
-; ALL:           sllv    $[[R7:[0-9]+]], $[[R6]], $[[R5]]
-; ALL:           nor     $[[R8:[0-9]+]], $zero, $[[R7]]
-; ALL:           sllv    $[[R9:[0-9]+]], $4, $[[R5]]
-
-; O0:        [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; O0:            ld      $[[R10:[0-9]+]]
-; O0-NEXT:       ll      $[[R11:[0-9]+]], 0($[[R10]])
-
-; ALL:       [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; ALL:           ll      $[[R12:[0-9]+]], 0($[[R2]])
-; ALL:           addu    $[[R13:[0-9]+]], $[[R12]], $[[R9]]
-; ALL:           and     $[[R14:[0-9]+]], $[[R13]], $[[R7]]
-; ALL:           and     $[[R15:[0-9]+]], $[[R12]], $[[R8]]
-; ALL:           or      $[[R16:[0-9]+]], $[[R15]], $[[R14]]
-; ALL:           sc      $[[R16]], 0($[[R2]])
-; NOT-MICROMIPS: beqz    $[[R16]], [[BB0]]
-; MICROMIPS:     beqzc   $[[R16]], [[BB0]]
-; MIPSR6:        beqzc   $[[R16]], [[BB0]]
-
-; ALL:           and     $[[R17:[0-9]+]], $[[R12]], $[[R7]]
-; ALL:           srlv    $[[R18:[0-9]+]], $[[R17]], $[[R5]]
-
-; NO-SEB-SEH:    sll     $[[R19:[0-9]+]], $[[R18]], 24
-; NO-SEB-SEH:    sra     $2, $[[R19]], 24
-
-; HAS-SEB-SEH:   seb     $2, $[[R18]]
 }
 
 define signext i8 @AtomicLoadSub8(i8 signext %incr) nounwind {
+; MIPS32-LABEL: AtomicLoadSub8:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    lw $1, %got(y)($1)
+; MIPS32-NEXT:    addiu $2, $zero, -4
+; MIPS32-NEXT:    and $2, $1, $2
+; MIPS32-NEXT:    andi $1, $1, 3
+; MIPS32-NEXT:    sll $3, $1, 3
+; MIPS32-NEXT:    ori $1, $zero, 255
+; MIPS32-NEXT:    sllv $5, $1, $3
+; MIPS32-NEXT:    nor $6, $zero, $5
+; MIPS32-NEXT:    sllv $4, $4, $3
+; MIPS32-NEXT:  $BB9_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $7, 0($2)
+; MIPS32-NEXT:    subu $8, $7, $4
+; MIPS32-NEXT:    and $8, $8, $5
+; MIPS32-NEXT:    and $9, $7, $6
+; MIPS32-NEXT:    or $9, $9, $8
+; MIPS32-NEXT:    sc $9, 0($2)
+; MIPS32-NEXT:    beqz $9, $BB9_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    and $1, $7, $5
+; MIPS32-NEXT:    srlv $1, $1, $3
+; MIPS32-NEXT:    sll $1, $1, 24
+; MIPS32-NEXT:    sra $1, $1, 24
+; MIPS32-NEXT:  # %bb.3: # %entry
+; MIPS32-NEXT:    sll $1, $1, 24
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    sra $2, $1, 24
+;
+; MIPS32O0-LABEL: AtomicLoadSub8:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    lw $2, %got(y)($2)
+; MIPS32O0-NEXT:    addiu $25, $zero, -4
+; MIPS32O0-NEXT:    and $25, $2, $25
+; MIPS32O0-NEXT:    andi $2, $2, 3
+; MIPS32O0-NEXT:    sll $2, $2, 3
+; MIPS32O0-NEXT:    ori $1, $zero, 255
+; MIPS32O0-NEXT:    sllv $1, $1, $2
+; MIPS32O0-NEXT:    nor $3, $zero, $1
+; MIPS32O0-NEXT:    sllv $4, $4, $2
+; MIPS32O0-NEXT:  $BB9_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $6, 0($25)
+; MIPS32O0-NEXT:    subu $7, $6, $4
+; MIPS32O0-NEXT:    and $7, $7, $1
+; MIPS32O0-NEXT:    and $8, $6, $3
+; MIPS32O0-NEXT:    or $8, $8, $7
+; MIPS32O0-NEXT:    sc $8, 0($25)
+; MIPS32O0-NEXT:    beqz $8, $BB9_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    and $5, $6, $1
+; MIPS32O0-NEXT:    srlv $5, $5, $2
+; MIPS32O0-NEXT:    sll $5, $5, 24
+; MIPS32O0-NEXT:    sra $5, $5, 24
+; MIPS32O0-NEXT:  # %bb.3: # %entry
+; MIPS32O0-NEXT:    sw $5, 4($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:  # %bb.4: # %entry
+; MIPS32O0-NEXT:    lw $1, 4($sp) # 4-byte Folded Reload
+; MIPS32O0-NEXT:    sll $2, $1, 24
+; MIPS32O0-NEXT:    sra $2, $2, 24
+; MIPS32O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicLoadSub8:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    lw $1, %got(y)($1)
+; MIPS32R2-NEXT:    addiu $2, $zero, -4
+; MIPS32R2-NEXT:    and $2, $1, $2
+; MIPS32R2-NEXT:    andi $1, $1, 3
+; MIPS32R2-NEXT:    sll $3, $1, 3
+; MIPS32R2-NEXT:    ori $1, $zero, 255
+; MIPS32R2-NEXT:    sllv $5, $1, $3
+; MIPS32R2-NEXT:    nor $6, $zero, $5
+; MIPS32R2-NEXT:    sllv $4, $4, $3
+; MIPS32R2-NEXT:  $BB9_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $7, 0($2)
+; MIPS32R2-NEXT:    subu $8, $7, $4
+; MIPS32R2-NEXT:    and $8, $8, $5
+; MIPS32R2-NEXT:    and $9, $7, $6
+; MIPS32R2-NEXT:    or $9, $9, $8
+; MIPS32R2-NEXT:    sc $9, 0($2)
+; MIPS32R2-NEXT:    beqz $9, $BB9_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    and $1, $7, $5
+; MIPS32R2-NEXT:    srlv $1, $1, $3
+; MIPS32R2-NEXT:    seb $1, $1
+; MIPS32R2-NEXT:  # %bb.3: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    seb $2, $1
+;
+; MIPS32R6-LABEL: AtomicLoadSub8:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    lw $1, %got(y)($1)
+; MIPS32R6-NEXT:    addiu $2, $zero, -4
+; MIPS32R6-NEXT:    and $2, $1, $2
+; MIPS32R6-NEXT:    andi $1, $1, 3
+; MIPS32R6-NEXT:    sll $3, $1, 3
+; MIPS32R6-NEXT:    ori $1, $zero, 255
+; MIPS32R6-NEXT:    sllv $5, $1, $3
+; MIPS32R6-NEXT:    nor $6, $zero, $5
+; MIPS32R6-NEXT:    sllv $4, $4, $3
+; MIPS32R6-NEXT:  $BB9_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $7, 0($2)
+; MIPS32R6-NEXT:    subu $8, $7, $4
+; MIPS32R6-NEXT:    and $8, $8, $5
+; MIPS32R6-NEXT:    and $9, $7, $6
+; MIPS32R6-NEXT:    or $9, $9, $8
+; MIPS32R6-NEXT:    sc $9, 0($2)
+; MIPS32R6-NEXT:    beqzc $9, $BB9_1
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    and $1, $7, $5
+; MIPS32R6-NEXT:    srlv $1, $1, $3
+; MIPS32R6-NEXT:    seb $1, $1
+; MIPS32R6-NEXT:  # %bb.3: # %entry
+; MIPS32R6-NEXT:    jr $ra
+; MIPS32R6-NEXT:    seb $2, $1
+;
+; MIPS32R6O0-LABEL: AtomicLoadSub8:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    lw $2, %got(y)($2)
+; MIPS32R6O0-NEXT:    addiu $1, $zero, -4
+; MIPS32R6O0-NEXT:    and $1, $2, $1
+; MIPS32R6O0-NEXT:    andi $2, $2, 3
+; MIPS32R6O0-NEXT:    sll $2, $2, 3
+; MIPS32R6O0-NEXT:    ori $3, $zero, 255
+; MIPS32R6O0-NEXT:    sllv $3, $3, $2
+; MIPS32R6O0-NEXT:    nor $5, $zero, $3
+; MIPS32R6O0-NEXT:    sllv $4, $4, $2
+; MIPS32R6O0-NEXT:  $BB9_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $7, 0($1)
+; MIPS32R6O0-NEXT:    subu $8, $7, $4
+; MIPS32R6O0-NEXT:    and $8, $8, $3
+; MIPS32R6O0-NEXT:    and $9, $7, $5
+; MIPS32R6O0-NEXT:    or $9, $9, $8
+; MIPS32R6O0-NEXT:    sc $9, 0($1)
+; MIPS32R6O0-NEXT:    beqzc $9, $BB9_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    and $6, $7, $3
+; MIPS32R6O0-NEXT:    srlv $6, $6, $2
+; MIPS32R6O0-NEXT:    seb $6, $6
+; MIPS32R6O0-NEXT:  # %bb.3: # %entry
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $6, 0($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:  # %bb.4: # %entry
+; MIPS32R6O0-NEXT:    lw $1, 0($sp) # 4-byte Folded Reload
+; MIPS32R6O0-NEXT:    seb $2, $1
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicLoadSub8:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub8)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub8)))
+; MIPS4-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS4-NEXT:    daddiu $2, $zero, -4
+; MIPS4-NEXT:    and $2, $1, $2
+; MIPS4-NEXT:    andi $1, $1, 3
+; MIPS4-NEXT:    sll $3, $1, 3
+; MIPS4-NEXT:    ori $1, $zero, 255
+; MIPS4-NEXT:    sllv $5, $1, $3
+; MIPS4-NEXT:    nor $6, $zero, $5
+; MIPS4-NEXT:    sllv $4, $4, $3
+; MIPS4-NEXT:  .LBB9_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $7, 0($2)
+; MIPS4-NEXT:    subu $8, $7, $4
+; MIPS4-NEXT:    and $8, $8, $5
+; MIPS4-NEXT:    and $9, $7, $6
+; MIPS4-NEXT:    or $9, $9, $8
+; MIPS4-NEXT:    sc $9, 0($2)
+; MIPS4-NEXT:    beqz $9, .LBB9_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    and $1, $7, $5
+; MIPS4-NEXT:    srlv $1, $1, $3
+; MIPS4-NEXT:    sll $1, $1, 24
+; MIPS4-NEXT:    sra $1, $1, 24
+; MIPS4-NEXT:  # %bb.3: # %entry
+; MIPS4-NEXT:    sll $1, $1, 24
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    sra $2, $1, 24
+;
+; MIPS64-LABEL: AtomicLoadSub8:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub8)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub8)))
+; MIPS64-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64-NEXT:    daddiu $2, $zero, -4
+; MIPS64-NEXT:    and $2, $1, $2
+; MIPS64-NEXT:    andi $1, $1, 3
+; MIPS64-NEXT:    sll $3, $1, 3
+; MIPS64-NEXT:    ori $1, $zero, 255
+; MIPS64-NEXT:    sllv $5, $1, $3
+; MIPS64-NEXT:    nor $6, $zero, $5
+; MIPS64-NEXT:    sllv $4, $4, $3
+; MIPS64-NEXT:  .LBB9_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $7, 0($2)
+; MIPS64-NEXT:    subu $8, $7, $4
+; MIPS64-NEXT:    and $8, $8, $5
+; MIPS64-NEXT:    and $9, $7, $6
+; MIPS64-NEXT:    or $9, $9, $8
+; MIPS64-NEXT:    sc $9, 0($2)
+; MIPS64-NEXT:    beqz $9, .LBB9_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    and $1, $7, $5
+; MIPS64-NEXT:    srlv $1, $1, $3
+; MIPS64-NEXT:    sll $1, $1, 24
+; MIPS64-NEXT:    sra $1, $1, 24
+; MIPS64-NEXT:  # %bb.3: # %entry
+; MIPS64-NEXT:    sll $1, $1, 24
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    sra $2, $1, 24
+;
+; MIPS64R2-LABEL: AtomicLoadSub8:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub8)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub8)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R2-NEXT:    daddiu $2, $zero, -4
+; MIPS64R2-NEXT:    and $2, $1, $2
+; MIPS64R2-NEXT:    andi $1, $1, 3
+; MIPS64R2-NEXT:    sll $3, $1, 3
+; MIPS64R2-NEXT:    ori $1, $zero, 255
+; MIPS64R2-NEXT:    sllv $5, $1, $3
+; MIPS64R2-NEXT:    nor $6, $zero, $5
+; MIPS64R2-NEXT:    sllv $4, $4, $3
+; MIPS64R2-NEXT:  .LBB9_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $7, 0($2)
+; MIPS64R2-NEXT:    subu $8, $7, $4
+; MIPS64R2-NEXT:    and $8, $8, $5
+; MIPS64R2-NEXT:    and $9, $7, $6
+; MIPS64R2-NEXT:    or $9, $9, $8
+; MIPS64R2-NEXT:    sc $9, 0($2)
+; MIPS64R2-NEXT:    beqz $9, .LBB9_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    and $1, $7, $5
+; MIPS64R2-NEXT:    srlv $1, $1, $3
+; MIPS64R2-NEXT:    seb $1, $1
+; MIPS64R2-NEXT:  # %bb.3: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    seb $2, $1
+;
+; MIPS64R6-LABEL: AtomicLoadSub8:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub8)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub8)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R6-NEXT:    daddiu $2, $zero, -4
+; MIPS64R6-NEXT:    and $2, $1, $2
+; MIPS64R6-NEXT:    andi $1, $1, 3
+; MIPS64R6-NEXT:    sll $3, $1, 3
+; MIPS64R6-NEXT:    ori $1, $zero, 255
+; MIPS64R6-NEXT:    sllv $5, $1, $3
+; MIPS64R6-NEXT:    nor $6, $zero, $5
+; MIPS64R6-NEXT:    sllv $4, $4, $3
+; MIPS64R6-NEXT:  .LBB9_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $7, 0($2)
+; MIPS64R6-NEXT:    subu $8, $7, $4
+; MIPS64R6-NEXT:    and $8, $8, $5
+; MIPS64R6-NEXT:    and $9, $7, $6
+; MIPS64R6-NEXT:    or $9, $9, $8
+; MIPS64R6-NEXT:    sc $9, 0($2)
+; MIPS64R6-NEXT:    beqzc $9, .LBB9_1
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    and $1, $7, $5
+; MIPS64R6-NEXT:    srlv $1, $1, $3
+; MIPS64R6-NEXT:    seb $1, $1
+; MIPS64R6-NEXT:  # %bb.3: # %entry
+; MIPS64R6-NEXT:    jr $ra
+; MIPS64R6-NEXT:    seb $2, $1
+;
+; MIPS64R6O0-LABEL: AtomicLoadSub8:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub8)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub8)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R6O0-NEXT:    daddiu $4, $zero, -4
+; MIPS64R6O0-NEXT:    and $4, $1, $4
+; MIPS64R6O0-NEXT:    andi $3, $1, 3
+; MIPS64R6O0-NEXT:    xori $3, $3, 3
+; MIPS64R6O0-NEXT:    sll $3, $3, 3
+; MIPS64R6O0-NEXT:    ori $5, $zero, 255
+; MIPS64R6O0-NEXT:    sllv $5, $5, $3
+; MIPS64R6O0-NEXT:    nor $6, $zero, $5
+; MIPS64R6O0-NEXT:    sllv $2, $2, $3
+; MIPS64R6O0-NEXT:  .LBB9_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $8, 0($4)
+; MIPS64R6O0-NEXT:    subu $9, $8, $2
+; MIPS64R6O0-NEXT:    and $9, $9, $5
+; MIPS64R6O0-NEXT:    and $10, $8, $6
+; MIPS64R6O0-NEXT:    or $10, $10, $9
+; MIPS64R6O0-NEXT:    sc $10, 0($4)
+; MIPS64R6O0-NEXT:    beqzc $10, .LBB9_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    and $7, $8, $5
+; MIPS64R6O0-NEXT:    srlv $7, $7, $3
+; MIPS64R6O0-NEXT:    seb $7, $7
+; MIPS64R6O0-NEXT:  # %bb.3: # %entry
+; MIPS64R6O0-NEXT:    sw $7, 12($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:  # %bb.4: # %entry
+; MIPS64R6O0-NEXT:    lw $1, 12($sp) # 4-byte Folded Reload
+; MIPS64R6O0-NEXT:    seb $2, $1
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicLoadSub8:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    lw $1, %got(y)($2)
+; MM32-NEXT:    addiu $2, $zero, -4
+; MM32-NEXT:    and $2, $1, $2
+; MM32-NEXT:    andi $1, $1, 3
+; MM32-NEXT:    sll $3, $1, 3
+; MM32-NEXT:    ori $1, $zero, 255
+; MM32-NEXT:    sllv $5, $1, $3
+; MM32-NEXT:    nor $6, $zero, $5
+; MM32-NEXT:    sllv $4, $4, $3
+; MM32-NEXT:  $BB9_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $7, 0($2)
+; MM32-NEXT:    subu $8, $7, $4
+; MM32-NEXT:    and $8, $8, $5
+; MM32-NEXT:    and $9, $7, $6
+; MM32-NEXT:    or $9, $9, $8
+; MM32-NEXT:    sc $9, 0($2)
+; MM32-NEXT:    beqzc $9, $BB9_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    and $1, $7, $5
+; MM32-NEXT:    srlv $1, $1, $3
+; MM32-NEXT:    seb $1, $1
+; MM32-NEXT:  # %bb.3: # %entry
+; MM32-NEXT:    jr $ra
+; MM32-NEXT:    seb $2, $1
+;
+; O1-LABEL: AtomicLoadSub8:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    lw $1, %got(y)($1)
+; O1-NEXT:    addiu $2, $zero, -4
+; O1-NEXT:    and $2, $1, $2
+; O1-NEXT:    andi $1, $1, 3
+; O1-NEXT:    sll $3, $1, 3
+; O1-NEXT:    ori $1, $zero, 255
+; O1-NEXT:    sllv $5, $1, $3
+; O1-NEXT:    nor $6, $zero, $5
+; O1-NEXT:    sllv $4, $4, $3
+; O1-NEXT:  $BB9_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $7, 0($2)
+; O1-NEXT:    subu $8, $7, $4
+; O1-NEXT:    and $8, $8, $5
+; O1-NEXT:    and $9, $7, $6
+; O1-NEXT:    or $9, $9, $8
+; O1-NEXT:    sc $9, 0($2)
+; O1-NEXT:    beqz $9, $BB9_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    and $1, $7, $5
+; O1-NEXT:    srlv $1, $1, $3
+; O1-NEXT:    sll $1, $1, 24
+; O1-NEXT:    sra $1, $1, 24
+; O1-NEXT:  # %bb.3: # %entry
+; O1-NEXT:    sll $1, $1, 24
+; O1-NEXT:    jr $ra
+; O1-NEXT:    sra $2, $1, 24
+;
+; O2-LABEL: AtomicLoadSub8:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    lw $1, %got(y)($1)
+; O2-NEXT:    addiu $2, $zero, -4
+; O2-NEXT:    and $2, $1, $2
+; O2-NEXT:    andi $1, $1, 3
+; O2-NEXT:    sll $3, $1, 3
+; O2-NEXT:    ori $1, $zero, 255
+; O2-NEXT:    sllv $5, $1, $3
+; O2-NEXT:    nor $6, $zero, $5
+; O2-NEXT:    sllv $4, $4, $3
+; O2-NEXT:  $BB9_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $7, 0($2)
+; O2-NEXT:    subu $8, $7, $4
+; O2-NEXT:    and $8, $8, $5
+; O2-NEXT:    and $9, $7, $6
+; O2-NEXT:    or $9, $9, $8
+; O2-NEXT:    sc $9, 0($2)
+; O2-NEXT:    beqz $9, $BB9_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    and $1, $7, $5
+; O2-NEXT:    srlv $1, $1, $3
+; O2-NEXT:    sll $1, $1, 24
+; O2-NEXT:    sra $1, $1, 24
+; O2-NEXT:  # %bb.3: # %entry
+; O2-NEXT:    sll $1, $1, 24
+; O2-NEXT:    jr $ra
+; O2-NEXT:    sra $2, $1, 24
+;
+; O3-LABEL: AtomicLoadSub8:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    addiu $2, $zero, -4
+; O3-NEXT:    lw $1, %got(y)($1)
+; O3-NEXT:    and $2, $1, $2
+; O3-NEXT:    andi $1, $1, 3
+; O3-NEXT:    sll $3, $1, 3
+; O3-NEXT:    ori $1, $zero, 255
+; O3-NEXT:    sllv $5, $1, $3
+; O3-NEXT:    sllv $4, $4, $3
+; O3-NEXT:    nor $6, $zero, $5
+; O3-NEXT:  $BB9_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $7, 0($2)
+; O3-NEXT:    subu $8, $7, $4
+; O3-NEXT:    and $8, $8, $5
+; O3-NEXT:    and $9, $7, $6
+; O3-NEXT:    or $9, $9, $8
+; O3-NEXT:    sc $9, 0($2)
+; O3-NEXT:    beqz $9, $BB9_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    and $1, $7, $5
+; O3-NEXT:    srlv $1, $1, $3
+; O3-NEXT:    sll $1, $1, 24
+; O3-NEXT:    sra $1, $1, 24
+; O3-NEXT:  # %bb.3: # %entry
+; O3-NEXT:    sll $1, $1, 24
+; O3-NEXT:    jr $ra
+; O3-NEXT:    sra $2, $1, 24
+;
+; MIPS32EB-LABEL: AtomicLoadSub8:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    lw $1, %got(y)($1)
+; MIPS32EB-NEXT:    addiu $2, $zero, -4
+; MIPS32EB-NEXT:    and $2, $1, $2
+; MIPS32EB-NEXT:    andi $1, $1, 3
+; MIPS32EB-NEXT:    xori $1, $1, 3
+; MIPS32EB-NEXT:    sll $3, $1, 3
+; MIPS32EB-NEXT:    ori $1, $zero, 255
+; MIPS32EB-NEXT:    sllv $5, $1, $3
+; MIPS32EB-NEXT:    nor $6, $zero, $5
+; MIPS32EB-NEXT:    sllv $4, $4, $3
+; MIPS32EB-NEXT:  $BB9_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $7, 0($2)
+; MIPS32EB-NEXT:    subu $8, $7, $4
+; MIPS32EB-NEXT:    and $8, $8, $5
+; MIPS32EB-NEXT:    and $9, $7, $6
+; MIPS32EB-NEXT:    or $9, $9, $8
+; MIPS32EB-NEXT:    sc $9, 0($2)
+; MIPS32EB-NEXT:    beqz $9, $BB9_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    and $1, $7, $5
+; MIPS32EB-NEXT:    srlv $1, $1, $3
+; MIPS32EB-NEXT:    sll $1, $1, 24
+; MIPS32EB-NEXT:    sra $1, $1, 24
+; MIPS32EB-NEXT:  # %bb.3: # %entry
+; MIPS32EB-NEXT:    sll $1, $1, 24
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    sra $2, $1, 24
 entry:
   %0 = atomicrmw sub i8* @y, i8 %incr monotonic
   ret i8 %0
 
-; ALL-LABEL: AtomicLoadSub8:
-
-; MIPS32-ANY: lw      $[[R0:[0-9]+]], %got(y)
-; MIPS64-ANY: ld      $[[R0:[0-9]+]], %got_disp(y)(
-
-; ALL:        addiu   $[[R1:[0-9]+]], $zero, -4
-; ALL:        and     $[[R2:[0-9]+]], $[[R0]], $[[R1]]
-; ALL:        andi    $[[R3:[0-9]+]], $[[R0]], 3
-; CHECK-EL:   sll     $[[R5:[0-9]+]], $[[R3]], 3
-; CHECK-EB:   xori    $[[R4:[0-9]+]], $[[R3]], 3
-; CHECK-EB:   sll     $[[R5:[0-9]+]], $[[R4]], 3
-; ALL:        ori     $[[R6:[0-9]+]], $zero, 255
-; ALL:        sllv    $[[R7:[0-9]+]], $[[R6]], $[[R5]]
-; ALL:        nor     $[[R8:[0-9]+]], $zero, $[[R7]]
-; ALL:        sllv    $[[R9:[0-9]+]], $4, $[[R5]]
-
-; O0:        [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; O0:            ld      $[[R10:[0-9]+]]
-; O0-NEXT:       ll      $[[R11:[0-9]+]], 0($[[R10]])
-
-; ALL:    [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; ALL:        ll      $[[R12:[0-9]+]], 0($[[R2]])
-; ALL:        subu    $[[R13:[0-9]+]], $[[R12]], $[[R9]]
-; ALL:        and     $[[R14:[0-9]+]], $[[R13]], $[[R7]]
-; ALL:        and     $[[R15:[0-9]+]], $[[R12]], $[[R8]]
-; ALL:        or      $[[R16:[0-9]+]], $[[R15]], $[[R14]]
-; ALL:        sc      $[[R16]], 0($[[R2]])
-; NOT-MICROMIPS: beqz    $[[R16]], [[BB0]]
-; MICROMIPS:  beqzc   $[[R16]], [[BB0]]
-; MIPSR6:     beqzc   $[[R16]], [[BB0]]
-
-; ALL:        and     $[[R17:[0-9]+]], $[[R12]], $[[R7]]
-; ALL:        srlv    $[[R18:[0-9]+]], $[[R17]], $[[R5]]
-
-; NO-SEB-SEH: sll     $[[R19:[0-9]+]], $[[R18]], 24
-; NO-SEB-SEH: sra     $2, $[[R19]], 24
-
-; HAS-SEB-SEH:seb     $2, $[[R18]]
 }
 
 define signext i8 @AtomicLoadNand8(i8 signext %incr) nounwind {
+; MIPS32-LABEL: AtomicLoadNand8:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    lw $1, %got(y)($1)
+; MIPS32-NEXT:    addiu $2, $zero, -4
+; MIPS32-NEXT:    and $2, $1, $2
+; MIPS32-NEXT:    andi $1, $1, 3
+; MIPS32-NEXT:    sll $3, $1, 3
+; MIPS32-NEXT:    ori $1, $zero, 255
+; MIPS32-NEXT:    sllv $5, $1, $3
+; MIPS32-NEXT:    nor $6, $zero, $5
+; MIPS32-NEXT:    sllv $4, $4, $3
+; MIPS32-NEXT:  $BB10_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $7, 0($2)
+; MIPS32-NEXT:    and $8, $7, $4
+; MIPS32-NEXT:    nor $8, $zero, $8
+; MIPS32-NEXT:    and $8, $8, $5
+; MIPS32-NEXT:    and $9, $7, $6
+; MIPS32-NEXT:    or $9, $9, $8
+; MIPS32-NEXT:    sc $9, 0($2)
+; MIPS32-NEXT:    beqz $9, $BB10_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    and $1, $7, $5
+; MIPS32-NEXT:    srlv $1, $1, $3
+; MIPS32-NEXT:    sll $1, $1, 24
+; MIPS32-NEXT:    sra $1, $1, 24
+; MIPS32-NEXT:  # %bb.3: # %entry
+; MIPS32-NEXT:    sll $1, $1, 24
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    sra $2, $1, 24
+;
+; MIPS32O0-LABEL: AtomicLoadNand8:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    lw $2, %got(y)($2)
+; MIPS32O0-NEXT:    addiu $25, $zero, -4
+; MIPS32O0-NEXT:    and $25, $2, $25
+; MIPS32O0-NEXT:    andi $2, $2, 3
+; MIPS32O0-NEXT:    sll $2, $2, 3
+; MIPS32O0-NEXT:    ori $1, $zero, 255
+; MIPS32O0-NEXT:    sllv $1, $1, $2
+; MIPS32O0-NEXT:    nor $3, $zero, $1
+; MIPS32O0-NEXT:    sllv $4, $4, $2
+; MIPS32O0-NEXT:  $BB10_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $6, 0($25)
+; MIPS32O0-NEXT:    and $7, $6, $4
+; MIPS32O0-NEXT:    nor $7, $zero, $7
+; MIPS32O0-NEXT:    and $7, $7, $1
+; MIPS32O0-NEXT:    and $8, $6, $3
+; MIPS32O0-NEXT:    or $8, $8, $7
+; MIPS32O0-NEXT:    sc $8, 0($25)
+; MIPS32O0-NEXT:    beqz $8, $BB10_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    and $5, $6, $1
+; MIPS32O0-NEXT:    srlv $5, $5, $2
+; MIPS32O0-NEXT:    sll $5, $5, 24
+; MIPS32O0-NEXT:    sra $5, $5, 24
+; MIPS32O0-NEXT:  # %bb.3: # %entry
+; MIPS32O0-NEXT:    sw $5, 4($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:  # %bb.4: # %entry
+; MIPS32O0-NEXT:    lw $1, 4($sp) # 4-byte Folded Reload
+; MIPS32O0-NEXT:    sll $2, $1, 24
+; MIPS32O0-NEXT:    sra $2, $2, 24
+; MIPS32O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicLoadNand8:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    lw $1, %got(y)($1)
+; MIPS32R2-NEXT:    addiu $2, $zero, -4
+; MIPS32R2-NEXT:    and $2, $1, $2
+; MIPS32R2-NEXT:    andi $1, $1, 3
+; MIPS32R2-NEXT:    sll $3, $1, 3
+; MIPS32R2-NEXT:    ori $1, $zero, 255
+; MIPS32R2-NEXT:    sllv $5, $1, $3
+; MIPS32R2-NEXT:    nor $6, $zero, $5
+; MIPS32R2-NEXT:    sllv $4, $4, $3
+; MIPS32R2-NEXT:  $BB10_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $7, 0($2)
+; MIPS32R2-NEXT:    and $8, $7, $4
+; MIPS32R2-NEXT:    nor $8, $zero, $8
+; MIPS32R2-NEXT:    and $8, $8, $5
+; MIPS32R2-NEXT:    and $9, $7, $6
+; MIPS32R2-NEXT:    or $9, $9, $8
+; MIPS32R2-NEXT:    sc $9, 0($2)
+; MIPS32R2-NEXT:    beqz $9, $BB10_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    and $1, $7, $5
+; MIPS32R2-NEXT:    srlv $1, $1, $3
+; MIPS32R2-NEXT:    seb $1, $1
+; MIPS32R2-NEXT:  # %bb.3: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    seb $2, $1
+;
+; MIPS32R6-LABEL: AtomicLoadNand8:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    lw $1, %got(y)($1)
+; MIPS32R6-NEXT:    addiu $2, $zero, -4
+; MIPS32R6-NEXT:    and $2, $1, $2
+; MIPS32R6-NEXT:    andi $1, $1, 3
+; MIPS32R6-NEXT:    sll $3, $1, 3
+; MIPS32R6-NEXT:    ori $1, $zero, 255
+; MIPS32R6-NEXT:    sllv $5, $1, $3
+; MIPS32R6-NEXT:    nor $6, $zero, $5
+; MIPS32R6-NEXT:    sllv $4, $4, $3
+; MIPS32R6-NEXT:  $BB10_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $7, 0($2)
+; MIPS32R6-NEXT:    and $8, $7, $4
+; MIPS32R6-NEXT:    nor $8, $zero, $8
+; MIPS32R6-NEXT:    and $8, $8, $5
+; MIPS32R6-NEXT:    and $9, $7, $6
+; MIPS32R6-NEXT:    or $9, $9, $8
+; MIPS32R6-NEXT:    sc $9, 0($2)
+; MIPS32R6-NEXT:    beqzc $9, $BB10_1
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    and $1, $7, $5
+; MIPS32R6-NEXT:    srlv $1, $1, $3
+; MIPS32R6-NEXT:    seb $1, $1
+; MIPS32R6-NEXT:  # %bb.3: # %entry
+; MIPS32R6-NEXT:    jr $ra
+; MIPS32R6-NEXT:    seb $2, $1
+;
+; MIPS32R6O0-LABEL: AtomicLoadNand8:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    lw $2, %got(y)($2)
+; MIPS32R6O0-NEXT:    addiu $1, $zero, -4
+; MIPS32R6O0-NEXT:    and $1, $2, $1
+; MIPS32R6O0-NEXT:    andi $2, $2, 3
+; MIPS32R6O0-NEXT:    sll $2, $2, 3
+; MIPS32R6O0-NEXT:    ori $3, $zero, 255
+; MIPS32R6O0-NEXT:    sllv $3, $3, $2
+; MIPS32R6O0-NEXT:    nor $5, $zero, $3
+; MIPS32R6O0-NEXT:    sllv $4, $4, $2
+; MIPS32R6O0-NEXT:  $BB10_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $7, 0($1)
+; MIPS32R6O0-NEXT:    and $8, $7, $4
+; MIPS32R6O0-NEXT:    nor $8, $zero, $8
+; MIPS32R6O0-NEXT:    and $8, $8, $3
+; MIPS32R6O0-NEXT:    and $9, $7, $5
+; MIPS32R6O0-NEXT:    or $9, $9, $8
+; MIPS32R6O0-NEXT:    sc $9, 0($1)
+; MIPS32R6O0-NEXT:    beqzc $9, $BB10_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    and $6, $7, $3
+; MIPS32R6O0-NEXT:    srlv $6, $6, $2
+; MIPS32R6O0-NEXT:    seb $6, $6
+; MIPS32R6O0-NEXT:  # %bb.3: # %entry
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $6, 0($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:  # %bb.4: # %entry
+; MIPS32R6O0-NEXT:    lw $1, 0($sp) # 4-byte Folded Reload
+; MIPS32R6O0-NEXT:    seb $2, $1
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicLoadNand8:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand8)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand8)))
+; MIPS4-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS4-NEXT:    daddiu $2, $zero, -4
+; MIPS4-NEXT:    and $2, $1, $2
+; MIPS4-NEXT:    andi $1, $1, 3
+; MIPS4-NEXT:    sll $3, $1, 3
+; MIPS4-NEXT:    ori $1, $zero, 255
+; MIPS4-NEXT:    sllv $5, $1, $3
+; MIPS4-NEXT:    nor $6, $zero, $5
+; MIPS4-NEXT:    sllv $4, $4, $3
+; MIPS4-NEXT:  .LBB10_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $7, 0($2)
+; MIPS4-NEXT:    and $8, $7, $4
+; MIPS4-NEXT:    nor $8, $zero, $8
+; MIPS4-NEXT:    and $8, $8, $5
+; MIPS4-NEXT:    and $9, $7, $6
+; MIPS4-NEXT:    or $9, $9, $8
+; MIPS4-NEXT:    sc $9, 0($2)
+; MIPS4-NEXT:    beqz $9, .LBB10_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    and $1, $7, $5
+; MIPS4-NEXT:    srlv $1, $1, $3
+; MIPS4-NEXT:    sll $1, $1, 24
+; MIPS4-NEXT:    sra $1, $1, 24
+; MIPS4-NEXT:  # %bb.3: # %entry
+; MIPS4-NEXT:    sll $1, $1, 24
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    sra $2, $1, 24
+;
+; MIPS64-LABEL: AtomicLoadNand8:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand8)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand8)))
+; MIPS64-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64-NEXT:    daddiu $2, $zero, -4
+; MIPS64-NEXT:    and $2, $1, $2
+; MIPS64-NEXT:    andi $1, $1, 3
+; MIPS64-NEXT:    sll $3, $1, 3
+; MIPS64-NEXT:    ori $1, $zero, 255
+; MIPS64-NEXT:    sllv $5, $1, $3
+; MIPS64-NEXT:    nor $6, $zero, $5
+; MIPS64-NEXT:    sllv $4, $4, $3
+; MIPS64-NEXT:  .LBB10_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $7, 0($2)
+; MIPS64-NEXT:    and $8, $7, $4
+; MIPS64-NEXT:    nor $8, $zero, $8
+; MIPS64-NEXT:    and $8, $8, $5
+; MIPS64-NEXT:    and $9, $7, $6
+; MIPS64-NEXT:    or $9, $9, $8
+; MIPS64-NEXT:    sc $9, 0($2)
+; MIPS64-NEXT:    beqz $9, .LBB10_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    and $1, $7, $5
+; MIPS64-NEXT:    srlv $1, $1, $3
+; MIPS64-NEXT:    sll $1, $1, 24
+; MIPS64-NEXT:    sra $1, $1, 24
+; MIPS64-NEXT:  # %bb.3: # %entry
+; MIPS64-NEXT:    sll $1, $1, 24
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    sra $2, $1, 24
+;
+; MIPS64R2-LABEL: AtomicLoadNand8:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand8)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand8)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R2-NEXT:    daddiu $2, $zero, -4
+; MIPS64R2-NEXT:    and $2, $1, $2
+; MIPS64R2-NEXT:    andi $1, $1, 3
+; MIPS64R2-NEXT:    sll $3, $1, 3
+; MIPS64R2-NEXT:    ori $1, $zero, 255
+; MIPS64R2-NEXT:    sllv $5, $1, $3
+; MIPS64R2-NEXT:    nor $6, $zero, $5
+; MIPS64R2-NEXT:    sllv $4, $4, $3
+; MIPS64R2-NEXT:  .LBB10_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $7, 0($2)
+; MIPS64R2-NEXT:    and $8, $7, $4
+; MIPS64R2-NEXT:    nor $8, $zero, $8
+; MIPS64R2-NEXT:    and $8, $8, $5
+; MIPS64R2-NEXT:    and $9, $7, $6
+; MIPS64R2-NEXT:    or $9, $9, $8
+; MIPS64R2-NEXT:    sc $9, 0($2)
+; MIPS64R2-NEXT:    beqz $9, .LBB10_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    and $1, $7, $5
+; MIPS64R2-NEXT:    srlv $1, $1, $3
+; MIPS64R2-NEXT:    seb $1, $1
+; MIPS64R2-NEXT:  # %bb.3: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    seb $2, $1
+;
+; MIPS64R6-LABEL: AtomicLoadNand8:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand8)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand8)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R6-NEXT:    daddiu $2, $zero, -4
+; MIPS64R6-NEXT:    and $2, $1, $2
+; MIPS64R6-NEXT:    andi $1, $1, 3
+; MIPS64R6-NEXT:    sll $3, $1, 3
+; MIPS64R6-NEXT:    ori $1, $zero, 255
+; MIPS64R6-NEXT:    sllv $5, $1, $3
+; MIPS64R6-NEXT:    nor $6, $zero, $5
+; MIPS64R6-NEXT:    sllv $4, $4, $3
+; MIPS64R6-NEXT:  .LBB10_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $7, 0($2)
+; MIPS64R6-NEXT:    and $8, $7, $4
+; MIPS64R6-NEXT:    nor $8, $zero, $8
+; MIPS64R6-NEXT:    and $8, $8, $5
+; MIPS64R6-NEXT:    and $9, $7, $6
+; MIPS64R6-NEXT:    or $9, $9, $8
+; MIPS64R6-NEXT:    sc $9, 0($2)
+; MIPS64R6-NEXT:    beqzc $9, .LBB10_1
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    and $1, $7, $5
+; MIPS64R6-NEXT:    srlv $1, $1, $3
+; MIPS64R6-NEXT:    seb $1, $1
+; MIPS64R6-NEXT:  # %bb.3: # %entry
+; MIPS64R6-NEXT:    jr $ra
+; MIPS64R6-NEXT:    seb $2, $1
+;
+; MIPS64R6O0-LABEL: AtomicLoadNand8:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand8)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand8)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R6O0-NEXT:    daddiu $4, $zero, -4
+; MIPS64R6O0-NEXT:    and $4, $1, $4
+; MIPS64R6O0-NEXT:    andi $3, $1, 3
+; MIPS64R6O0-NEXT:    xori $3, $3, 3
+; MIPS64R6O0-NEXT:    sll $3, $3, 3
+; MIPS64R6O0-NEXT:    ori $5, $zero, 255
+; MIPS64R6O0-NEXT:    sllv $5, $5, $3
+; MIPS64R6O0-NEXT:    nor $6, $zero, $5
+; MIPS64R6O0-NEXT:    sllv $2, $2, $3
+; MIPS64R6O0-NEXT:  .LBB10_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $8, 0($4)
+; MIPS64R6O0-NEXT:    and $9, $8, $2
+; MIPS64R6O0-NEXT:    nor $9, $zero, $9
+; MIPS64R6O0-NEXT:    and $9, $9, $5
+; MIPS64R6O0-NEXT:    and $10, $8, $6
+; MIPS64R6O0-NEXT:    or $10, $10, $9
+; MIPS64R6O0-NEXT:    sc $10, 0($4)
+; MIPS64R6O0-NEXT:    beqzc $10, .LBB10_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    and $7, $8, $5
+; MIPS64R6O0-NEXT:    srlv $7, $7, $3
+; MIPS64R6O0-NEXT:    seb $7, $7
+; MIPS64R6O0-NEXT:  # %bb.3: # %entry
+; MIPS64R6O0-NEXT:    sw $7, 12($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:  # %bb.4: # %entry
+; MIPS64R6O0-NEXT:    lw $1, 12($sp) # 4-byte Folded Reload
+; MIPS64R6O0-NEXT:    seb $2, $1
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicLoadNand8:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    lw $1, %got(y)($2)
+; MM32-NEXT:    addiu $2, $zero, -4
+; MM32-NEXT:    and $2, $1, $2
+; MM32-NEXT:    andi $1, $1, 3
+; MM32-NEXT:    sll $3, $1, 3
+; MM32-NEXT:    ori $1, $zero, 255
+; MM32-NEXT:    sllv $5, $1, $3
+; MM32-NEXT:    nor $6, $zero, $5
+; MM32-NEXT:    sllv $4, $4, $3
+; MM32-NEXT:  $BB10_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $7, 0($2)
+; MM32-NEXT:    and $8, $7, $4
+; MM32-NEXT:    nor $8, $zero, $8
+; MM32-NEXT:    and $8, $8, $5
+; MM32-NEXT:    and $9, $7, $6
+; MM32-NEXT:    or $9, $9, $8
+; MM32-NEXT:    sc $9, 0($2)
+; MM32-NEXT:    beqzc $9, $BB10_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    and $1, $7, $5
+; MM32-NEXT:    srlv $1, $1, $3
+; MM32-NEXT:    seb $1, $1
+; MM32-NEXT:  # %bb.3: # %entry
+; MM32-NEXT:    jr $ra
+; MM32-NEXT:    seb $2, $1
+;
+; O1-LABEL: AtomicLoadNand8:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    lw $1, %got(y)($1)
+; O1-NEXT:    addiu $2, $zero, -4
+; O1-NEXT:    and $2, $1, $2
+; O1-NEXT:    andi $1, $1, 3
+; O1-NEXT:    sll $3, $1, 3
+; O1-NEXT:    ori $1, $zero, 255
+; O1-NEXT:    sllv $5, $1, $3
+; O1-NEXT:    nor $6, $zero, $5
+; O1-NEXT:    sllv $4, $4, $3
+; O1-NEXT:  $BB10_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $7, 0($2)
+; O1-NEXT:    and $8, $7, $4
+; O1-NEXT:    nor $8, $zero, $8
+; O1-NEXT:    and $8, $8, $5
+; O1-NEXT:    and $9, $7, $6
+; O1-NEXT:    or $9, $9, $8
+; O1-NEXT:    sc $9, 0($2)
+; O1-NEXT:    beqz $9, $BB10_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    and $1, $7, $5
+; O1-NEXT:    srlv $1, $1, $3
+; O1-NEXT:    sll $1, $1, 24
+; O1-NEXT:    sra $1, $1, 24
+; O1-NEXT:  # %bb.3: # %entry
+; O1-NEXT:    sll $1, $1, 24
+; O1-NEXT:    jr $ra
+; O1-NEXT:    sra $2, $1, 24
+;
+; O2-LABEL: AtomicLoadNand8:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    lw $1, %got(y)($1)
+; O2-NEXT:    addiu $2, $zero, -4
+; O2-NEXT:    and $2, $1, $2
+; O2-NEXT:    andi $1, $1, 3
+; O2-NEXT:    sll $3, $1, 3
+; O2-NEXT:    ori $1, $zero, 255
+; O2-NEXT:    sllv $5, $1, $3
+; O2-NEXT:    nor $6, $zero, $5
+; O2-NEXT:    sllv $4, $4, $3
+; O2-NEXT:  $BB10_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $7, 0($2)
+; O2-NEXT:    and $8, $7, $4
+; O2-NEXT:    nor $8, $zero, $8
+; O2-NEXT:    and $8, $8, $5
+; O2-NEXT:    and $9, $7, $6
+; O2-NEXT:    or $9, $9, $8
+; O2-NEXT:    sc $9, 0($2)
+; O2-NEXT:    beqz $9, $BB10_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    and $1, $7, $5
+; O2-NEXT:    srlv $1, $1, $3
+; O2-NEXT:    sll $1, $1, 24
+; O2-NEXT:    sra $1, $1, 24
+; O2-NEXT:  # %bb.3: # %entry
+; O2-NEXT:    sll $1, $1, 24
+; O2-NEXT:    jr $ra
+; O2-NEXT:    sra $2, $1, 24
+;
+; O3-LABEL: AtomicLoadNand8:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    addiu $2, $zero, -4
+; O3-NEXT:    lw $1, %got(y)($1)
+; O3-NEXT:    and $2, $1, $2
+; O3-NEXT:    andi $1, $1, 3
+; O3-NEXT:    sll $3, $1, 3
+; O3-NEXT:    ori $1, $zero, 255
+; O3-NEXT:    sllv $5, $1, $3
+; O3-NEXT:    sllv $4, $4, $3
+; O3-NEXT:    nor $6, $zero, $5
+; O3-NEXT:  $BB10_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $7, 0($2)
+; O3-NEXT:    and $8, $7, $4
+; O3-NEXT:    nor $8, $zero, $8
+; O3-NEXT:    and $8, $8, $5
+; O3-NEXT:    and $9, $7, $6
+; O3-NEXT:    or $9, $9, $8
+; O3-NEXT:    sc $9, 0($2)
+; O3-NEXT:    beqz $9, $BB10_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    and $1, $7, $5
+; O3-NEXT:    srlv $1, $1, $3
+; O3-NEXT:    sll $1, $1, 24
+; O3-NEXT:    sra $1, $1, 24
+; O3-NEXT:  # %bb.3: # %entry
+; O3-NEXT:    sll $1, $1, 24
+; O3-NEXT:    jr $ra
+; O3-NEXT:    sra $2, $1, 24
+;
+; MIPS32EB-LABEL: AtomicLoadNand8:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    lw $1, %got(y)($1)
+; MIPS32EB-NEXT:    addiu $2, $zero, -4
+; MIPS32EB-NEXT:    and $2, $1, $2
+; MIPS32EB-NEXT:    andi $1, $1, 3
+; MIPS32EB-NEXT:    xori $1, $1, 3
+; MIPS32EB-NEXT:    sll $3, $1, 3
+; MIPS32EB-NEXT:    ori $1, $zero, 255
+; MIPS32EB-NEXT:    sllv $5, $1, $3
+; MIPS32EB-NEXT:    nor $6, $zero, $5
+; MIPS32EB-NEXT:    sllv $4, $4, $3
+; MIPS32EB-NEXT:  $BB10_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $7, 0($2)
+; MIPS32EB-NEXT:    and $8, $7, $4
+; MIPS32EB-NEXT:    nor $8, $zero, $8
+; MIPS32EB-NEXT:    and $8, $8, $5
+; MIPS32EB-NEXT:    and $9, $7, $6
+; MIPS32EB-NEXT:    or $9, $9, $8
+; MIPS32EB-NEXT:    sc $9, 0($2)
+; MIPS32EB-NEXT:    beqz $9, $BB10_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    and $1, $7, $5
+; MIPS32EB-NEXT:    srlv $1, $1, $3
+; MIPS32EB-NEXT:    sll $1, $1, 24
+; MIPS32EB-NEXT:    sra $1, $1, 24
+; MIPS32EB-NEXT:  # %bb.3: # %entry
+; MIPS32EB-NEXT:    sll $1, $1, 24
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    sra $2, $1, 24
 entry:
   %0 = atomicrmw nand i8* @y, i8 %incr monotonic
   ret i8 %0
 
-; ALL-LABEL: AtomicLoadNand8:
-
-; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(y)
-; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(y)(
-
-; ALL:           addiu   $[[R1:[0-9]+]], $zero, -4
-; ALL:           and     $[[R2:[0-9]+]], $[[R0]], $[[R1]]
-; ALL:           andi    $[[R3:[0-9]+]], $[[R0]], 3
-; CHECK-EL:      sll     $[[R5:[0-9]+]], $[[R3]], 3
-; CHECK-EB:      xori    $[[R4:[0-9]+]], $[[R3]], 3
-; CHECK-EB:      sll     $[[R5:[0-9]+]], $[[R4]], 3
-; ALL:           ori     $[[R6:[0-9]+]], $zero, 255
-; ALL:           sllv    $[[R7:[0-9]+]], $[[R6]], $[[R5]]
-; ALL:           nor     $[[R8:[0-9]+]], $zero, $[[R7]]
-; ALL:           sllv    $[[R9:[0-9]+]], $4, $[[R5]]
-
-; O0:        [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; O0:            ld      $[[R10:[0-9]+]]
-; O0-NEXT:       ll      $[[R11:[0-9]+]], 0($[[R10]])
-
-; ALL:       [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; ALL:           ll      $[[R12:[0-9]+]], 0($[[R2]])
-; ALL:           and     $[[R13:[0-9]+]], $[[R12]], $[[R9]]
-; ALL:           nor     $[[R14:[0-9]+]], $zero, $[[R13]]
-; ALL:           and     $[[R15:[0-9]+]], $[[R14]], $[[R7]]
-; ALL:           and     $[[R16:[0-9]+]], $[[R12]], $[[R8]]
-; ALL:           or      $[[R17:[0-9]+]], $[[R16]], $[[R15]]
-; ALL:           sc      $[[R17]], 0($[[R2]])
-; NOT-MICROMIPS: beqz    $[[R17]], [[BB0]]
-; MICROMIPS:     beqzc   $[[R17]], [[BB0]]
-; MIPSR6:        beqzc   $[[R17]], [[BB0]]
-
-; ALL:           and     $[[R18:[0-9]+]], $[[R12]], $[[R7]]
-; ALL:           srlv    $[[R19:[0-9]+]], $[[R18]], $[[R5]]
-
-; NO-SEB-SEH:    sll     $[[R20:[0-9]+]], $[[R19]], 24
-; NO-SEB-SEH:    sra     $2, $[[R20]], 24
-
-; HAS-SEB-SEH:   seb     $2, $[[R19]]
 }
 
 define signext i8 @AtomicSwap8(i8 signext %newval) nounwind {
+; MIPS32-LABEL: AtomicSwap8:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    lw $1, %got(y)($1)
+; MIPS32-NEXT:    addiu $2, $zero, -4
+; MIPS32-NEXT:    and $2, $1, $2
+; MIPS32-NEXT:    andi $1, $1, 3
+; MIPS32-NEXT:    sll $3, $1, 3
+; MIPS32-NEXT:    ori $1, $zero, 255
+; MIPS32-NEXT:    sllv $5, $1, $3
+; MIPS32-NEXT:    nor $6, $zero, $5
+; MIPS32-NEXT:    sllv $4, $4, $3
+; MIPS32-NEXT:  $BB11_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $7, 0($2)
+; MIPS32-NEXT:    and $8, $4, $5
+; MIPS32-NEXT:    and $9, $7, $6
+; MIPS32-NEXT:    or $9, $9, $8
+; MIPS32-NEXT:    sc $9, 0($2)
+; MIPS32-NEXT:    beqz $9, $BB11_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    and $1, $7, $5
+; MIPS32-NEXT:    srlv $1, $1, $3
+; MIPS32-NEXT:    sll $1, $1, 24
+; MIPS32-NEXT:    sra $1, $1, 24
+; MIPS32-NEXT:  # %bb.3: # %entry
+; MIPS32-NEXT:    sll $1, $1, 24
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    sra $2, $1, 24
+;
+; MIPS32O0-LABEL: AtomicSwap8:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    lw $2, %got(y)($2)
+; MIPS32O0-NEXT:    addiu $25, $zero, -4
+; MIPS32O0-NEXT:    and $25, $2, $25
+; MIPS32O0-NEXT:    andi $2, $2, 3
+; MIPS32O0-NEXT:    sll $2, $2, 3
+; MIPS32O0-NEXT:    ori $1, $zero, 255
+; MIPS32O0-NEXT:    sllv $1, $1, $2
+; MIPS32O0-NEXT:    nor $3, $zero, $1
+; MIPS32O0-NEXT:    sllv $4, $4, $2
+; MIPS32O0-NEXT:  $BB11_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $6, 0($25)
+; MIPS32O0-NEXT:    and $7, $4, $1
+; MIPS32O0-NEXT:    and $8, $6, $3
+; MIPS32O0-NEXT:    or $8, $8, $7
+; MIPS32O0-NEXT:    sc $8, 0($25)
+; MIPS32O0-NEXT:    beqz $8, $BB11_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    and $5, $6, $1
+; MIPS32O0-NEXT:    srlv $5, $5, $2
+; MIPS32O0-NEXT:    sll $5, $5, 24
+; MIPS32O0-NEXT:    sra $5, $5, 24
+; MIPS32O0-NEXT:  # %bb.3: # %entry
+; MIPS32O0-NEXT:    sw $5, 4($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:  # %bb.4: # %entry
+; MIPS32O0-NEXT:    lw $1, 4($sp) # 4-byte Folded Reload
+; MIPS32O0-NEXT:    sll $2, $1, 24
+; MIPS32O0-NEXT:    sra $2, $2, 24
+; MIPS32O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicSwap8:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    lw $1, %got(y)($1)
+; MIPS32R2-NEXT:    addiu $2, $zero, -4
+; MIPS32R2-NEXT:    and $2, $1, $2
+; MIPS32R2-NEXT:    andi $1, $1, 3
+; MIPS32R2-NEXT:    sll $3, $1, 3
+; MIPS32R2-NEXT:    ori $1, $zero, 255
+; MIPS32R2-NEXT:    sllv $5, $1, $3
+; MIPS32R2-NEXT:    nor $6, $zero, $5
+; MIPS32R2-NEXT:    sllv $4, $4, $3
+; MIPS32R2-NEXT:  $BB11_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $7, 0($2)
+; MIPS32R2-NEXT:    and $8, $4, $5
+; MIPS32R2-NEXT:    and $9, $7, $6
+; MIPS32R2-NEXT:    or $9, $9, $8
+; MIPS32R2-NEXT:    sc $9, 0($2)
+; MIPS32R2-NEXT:    beqz $9, $BB11_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    and $1, $7, $5
+; MIPS32R2-NEXT:    srlv $1, $1, $3
+; MIPS32R2-NEXT:    seb $1, $1
+; MIPS32R2-NEXT:  # %bb.3: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    seb $2, $1
+;
+; MIPS32R6-LABEL: AtomicSwap8:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    lw $1, %got(y)($1)
+; MIPS32R6-NEXT:    addiu $2, $zero, -4
+; MIPS32R6-NEXT:    and $2, $1, $2
+; MIPS32R6-NEXT:    andi $1, $1, 3
+; MIPS32R6-NEXT:    sll $3, $1, 3
+; MIPS32R6-NEXT:    ori $1, $zero, 255
+; MIPS32R6-NEXT:    sllv $5, $1, $3
+; MIPS32R6-NEXT:    nor $6, $zero, $5
+; MIPS32R6-NEXT:    sllv $4, $4, $3
+; MIPS32R6-NEXT:  $BB11_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $7, 0($2)
+; MIPS32R6-NEXT:    and $8, $4, $5
+; MIPS32R6-NEXT:    and $9, $7, $6
+; MIPS32R6-NEXT:    or $9, $9, $8
+; MIPS32R6-NEXT:    sc $9, 0($2)
+; MIPS32R6-NEXT:    beqzc $9, $BB11_1
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    and $1, $7, $5
+; MIPS32R6-NEXT:    srlv $1, $1, $3
+; MIPS32R6-NEXT:    seb $1, $1
+; MIPS32R6-NEXT:  # %bb.3: # %entry
+; MIPS32R6-NEXT:    jr $ra
+; MIPS32R6-NEXT:    seb $2, $1
+;
+; MIPS32R6O0-LABEL: AtomicSwap8:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    lw $2, %got(y)($2)
+; MIPS32R6O0-NEXT:    addiu $1, $zero, -4
+; MIPS32R6O0-NEXT:    and $1, $2, $1
+; MIPS32R6O0-NEXT:    andi $2, $2, 3
+; MIPS32R6O0-NEXT:    sll $2, $2, 3
+; MIPS32R6O0-NEXT:    ori $3, $zero, 255
+; MIPS32R6O0-NEXT:    sllv $3, $3, $2
+; MIPS32R6O0-NEXT:    nor $5, $zero, $3
+; MIPS32R6O0-NEXT:    sllv $4, $4, $2
+; MIPS32R6O0-NEXT:  $BB11_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $7, 0($1)
+; MIPS32R6O0-NEXT:    and $8, $4, $3
+; MIPS32R6O0-NEXT:    and $9, $7, $5
+; MIPS32R6O0-NEXT:    or $9, $9, $8
+; MIPS32R6O0-NEXT:    sc $9, 0($1)
+; MIPS32R6O0-NEXT:    beqzc $9, $BB11_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    and $6, $7, $3
+; MIPS32R6O0-NEXT:    srlv $6, $6, $2
+; MIPS32R6O0-NEXT:    seb $6, $6
+; MIPS32R6O0-NEXT:  # %bb.3: # %entry
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $6, 0($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:  # %bb.4: # %entry
+; MIPS32R6O0-NEXT:    lw $1, 0($sp) # 4-byte Folded Reload
+; MIPS32R6O0-NEXT:    seb $2, $1
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicSwap8:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap8)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap8)))
+; MIPS4-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS4-NEXT:    daddiu $2, $zero, -4
+; MIPS4-NEXT:    and $2, $1, $2
+; MIPS4-NEXT:    andi $1, $1, 3
+; MIPS4-NEXT:    sll $3, $1, 3
+; MIPS4-NEXT:    ori $1, $zero, 255
+; MIPS4-NEXT:    sllv $5, $1, $3
+; MIPS4-NEXT:    nor $6, $zero, $5
+; MIPS4-NEXT:    sllv $4, $4, $3
+; MIPS4-NEXT:  .LBB11_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $7, 0($2)
+; MIPS4-NEXT:    and $8, $4, $5
+; MIPS4-NEXT:    and $9, $7, $6
+; MIPS4-NEXT:    or $9, $9, $8
+; MIPS4-NEXT:    sc $9, 0($2)
+; MIPS4-NEXT:    beqz $9, .LBB11_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    and $1, $7, $5
+; MIPS4-NEXT:    srlv $1, $1, $3
+; MIPS4-NEXT:    sll $1, $1, 24
+; MIPS4-NEXT:    sra $1, $1, 24
+; MIPS4-NEXT:  # %bb.3: # %entry
+; MIPS4-NEXT:    sll $1, $1, 24
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    sra $2, $1, 24
+;
+; MIPS64-LABEL: AtomicSwap8:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap8)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap8)))
+; MIPS64-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64-NEXT:    daddiu $2, $zero, -4
+; MIPS64-NEXT:    and $2, $1, $2
+; MIPS64-NEXT:    andi $1, $1, 3
+; MIPS64-NEXT:    sll $3, $1, 3
+; MIPS64-NEXT:    ori $1, $zero, 255
+; MIPS64-NEXT:    sllv $5, $1, $3
+; MIPS64-NEXT:    nor $6, $zero, $5
+; MIPS64-NEXT:    sllv $4, $4, $3
+; MIPS64-NEXT:  .LBB11_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $7, 0($2)
+; MIPS64-NEXT:    and $8, $4, $5
+; MIPS64-NEXT:    and $9, $7, $6
+; MIPS64-NEXT:    or $9, $9, $8
+; MIPS64-NEXT:    sc $9, 0($2)
+; MIPS64-NEXT:    beqz $9, .LBB11_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    and $1, $7, $5
+; MIPS64-NEXT:    srlv $1, $1, $3
+; MIPS64-NEXT:    sll $1, $1, 24
+; MIPS64-NEXT:    sra $1, $1, 24
+; MIPS64-NEXT:  # %bb.3: # %entry
+; MIPS64-NEXT:    sll $1, $1, 24
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    sra $2, $1, 24
+;
+; MIPS64R2-LABEL: AtomicSwap8:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap8)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap8)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R2-NEXT:    daddiu $2, $zero, -4
+; MIPS64R2-NEXT:    and $2, $1, $2
+; MIPS64R2-NEXT:    andi $1, $1, 3
+; MIPS64R2-NEXT:    sll $3, $1, 3
+; MIPS64R2-NEXT:    ori $1, $zero, 255
+; MIPS64R2-NEXT:    sllv $5, $1, $3
+; MIPS64R2-NEXT:    nor $6, $zero, $5
+; MIPS64R2-NEXT:    sllv $4, $4, $3
+; MIPS64R2-NEXT:  .LBB11_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $7, 0($2)
+; MIPS64R2-NEXT:    and $8, $4, $5
+; MIPS64R2-NEXT:    and $9, $7, $6
+; MIPS64R2-NEXT:    or $9, $9, $8
+; MIPS64R2-NEXT:    sc $9, 0($2)
+; MIPS64R2-NEXT:    beqz $9, .LBB11_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    and $1, $7, $5
+; MIPS64R2-NEXT:    srlv $1, $1, $3
+; MIPS64R2-NEXT:    seb $1, $1
+; MIPS64R2-NEXT:  # %bb.3: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    seb $2, $1
+;
+; MIPS64R6-LABEL: AtomicSwap8:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap8)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap8)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R6-NEXT:    daddiu $2, $zero, -4
+; MIPS64R6-NEXT:    and $2, $1, $2
+; MIPS64R6-NEXT:    andi $1, $1, 3
+; MIPS64R6-NEXT:    sll $3, $1, 3
+; MIPS64R6-NEXT:    ori $1, $zero, 255
+; MIPS64R6-NEXT:    sllv $5, $1, $3
+; MIPS64R6-NEXT:    nor $6, $zero, $5
+; MIPS64R6-NEXT:    sllv $4, $4, $3
+; MIPS64R6-NEXT:  .LBB11_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $7, 0($2)
+; MIPS64R6-NEXT:    and $8, $4, $5
+; MIPS64R6-NEXT:    and $9, $7, $6
+; MIPS64R6-NEXT:    or $9, $9, $8
+; MIPS64R6-NEXT:    sc $9, 0($2)
+; MIPS64R6-NEXT:    beqzc $9, .LBB11_1
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    and $1, $7, $5
+; MIPS64R6-NEXT:    srlv $1, $1, $3
+; MIPS64R6-NEXT:    seb $1, $1
+; MIPS64R6-NEXT:  # %bb.3: # %entry
+; MIPS64R6-NEXT:    jr $ra
+; MIPS64R6-NEXT:    seb $2, $1
+;
+; MIPS64R6O0-LABEL: AtomicSwap8:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap8)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap8)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R6O0-NEXT:    daddiu $4, $zero, -4
+; MIPS64R6O0-NEXT:    and $4, $1, $4
+; MIPS64R6O0-NEXT:    andi $3, $1, 3
+; MIPS64R6O0-NEXT:    xori $3, $3, 3
+; MIPS64R6O0-NEXT:    sll $3, $3, 3
+; MIPS64R6O0-NEXT:    ori $5, $zero, 255
+; MIPS64R6O0-NEXT:    sllv $5, $5, $3
+; MIPS64R6O0-NEXT:    nor $6, $zero, $5
+; MIPS64R6O0-NEXT:    sllv $2, $2, $3
+; MIPS64R6O0-NEXT:  .LBB11_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $8, 0($4)
+; MIPS64R6O0-NEXT:    and $9, $2, $5
+; MIPS64R6O0-NEXT:    and $10, $8, $6
+; MIPS64R6O0-NEXT:    or $10, $10, $9
+; MIPS64R6O0-NEXT:    sc $10, 0($4)
+; MIPS64R6O0-NEXT:    beqzc $10, .LBB11_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    and $7, $8, $5
+; MIPS64R6O0-NEXT:    srlv $7, $7, $3
+; MIPS64R6O0-NEXT:    seb $7, $7
+; MIPS64R6O0-NEXT:  # %bb.3: # %entry
+; MIPS64R6O0-NEXT:    sw $7, 12($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:  # %bb.4: # %entry
+; MIPS64R6O0-NEXT:    lw $1, 12($sp) # 4-byte Folded Reload
+; MIPS64R6O0-NEXT:    seb $2, $1
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicSwap8:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    lw $1, %got(y)($2)
+; MM32-NEXT:    addiu $2, $zero, -4
+; MM32-NEXT:    and $2, $1, $2
+; MM32-NEXT:    andi $1, $1, 3
+; MM32-NEXT:    sll $3, $1, 3
+; MM32-NEXT:    ori $1, $zero, 255
+; MM32-NEXT:    sllv $5, $1, $3
+; MM32-NEXT:    nor $6, $zero, $5
+; MM32-NEXT:    sllv $4, $4, $3
+; MM32-NEXT:  $BB11_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $7, 0($2)
+; MM32-NEXT:    and $8, $4, $5
+; MM32-NEXT:    and $9, $7, $6
+; MM32-NEXT:    or $9, $9, $8
+; MM32-NEXT:    sc $9, 0($2)
+; MM32-NEXT:    beqzc $9, $BB11_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    and $1, $7, $5
+; MM32-NEXT:    srlv $1, $1, $3
+; MM32-NEXT:    seb $1, $1
+; MM32-NEXT:  # %bb.3: # %entry
+; MM32-NEXT:    jr $ra
+; MM32-NEXT:    seb $2, $1
+;
+; O1-LABEL: AtomicSwap8:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    lw $1, %got(y)($1)
+; O1-NEXT:    addiu $2, $zero, -4
+; O1-NEXT:    and $2, $1, $2
+; O1-NEXT:    andi $1, $1, 3
+; O1-NEXT:    sll $3, $1, 3
+; O1-NEXT:    ori $1, $zero, 255
+; O1-NEXT:    sllv $5, $1, $3
+; O1-NEXT:    nor $6, $zero, $5
+; O1-NEXT:    sllv $4, $4, $3
+; O1-NEXT:  $BB11_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $7, 0($2)
+; O1-NEXT:    and $8, $4, $5
+; O1-NEXT:    and $9, $7, $6
+; O1-NEXT:    or $9, $9, $8
+; O1-NEXT:    sc $9, 0($2)
+; O1-NEXT:    beqz $9, $BB11_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    and $1, $7, $5
+; O1-NEXT:    srlv $1, $1, $3
+; O1-NEXT:    sll $1, $1, 24
+; O1-NEXT:    sra $1, $1, 24
+; O1-NEXT:  # %bb.3: # %entry
+; O1-NEXT:    sll $1, $1, 24
+; O1-NEXT:    jr $ra
+; O1-NEXT:    sra $2, $1, 24
+;
+; O2-LABEL: AtomicSwap8:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    lw $1, %got(y)($1)
+; O2-NEXT:    addiu $2, $zero, -4
+; O2-NEXT:    and $2, $1, $2
+; O2-NEXT:    andi $1, $1, 3
+; O2-NEXT:    sll $3, $1, 3
+; O2-NEXT:    ori $1, $zero, 255
+; O2-NEXT:    sllv $5, $1, $3
+; O2-NEXT:    nor $6, $zero, $5
+; O2-NEXT:    sllv $4, $4, $3
+; O2-NEXT:  $BB11_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $7, 0($2)
+; O2-NEXT:    and $8, $4, $5
+; O2-NEXT:    and $9, $7, $6
+; O2-NEXT:    or $9, $9, $8
+; O2-NEXT:    sc $9, 0($2)
+; O2-NEXT:    beqz $9, $BB11_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    and $1, $7, $5
+; O2-NEXT:    srlv $1, $1, $3
+; O2-NEXT:    sll $1, $1, 24
+; O2-NEXT:    sra $1, $1, 24
+; O2-NEXT:  # %bb.3: # %entry
+; O2-NEXT:    sll $1, $1, 24
+; O2-NEXT:    jr $ra
+; O2-NEXT:    sra $2, $1, 24
+;
+; O3-LABEL: AtomicSwap8:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    addiu $2, $zero, -4
+; O3-NEXT:    lw $1, %got(y)($1)
+; O3-NEXT:    and $2, $1, $2
+; O3-NEXT:    andi $1, $1, 3
+; O3-NEXT:    sll $3, $1, 3
+; O3-NEXT:    ori $1, $zero, 255
+; O3-NEXT:    sllv $5, $1, $3
+; O3-NEXT:    sllv $4, $4, $3
+; O3-NEXT:    nor $6, $zero, $5
+; O3-NEXT:  $BB11_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $7, 0($2)
+; O3-NEXT:    and $8, $4, $5
+; O3-NEXT:    and $9, $7, $6
+; O3-NEXT:    or $9, $9, $8
+; O3-NEXT:    sc $9, 0($2)
+; O3-NEXT:    beqz $9, $BB11_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    and $1, $7, $5
+; O3-NEXT:    srlv $1, $1, $3
+; O3-NEXT:    sll $1, $1, 24
+; O3-NEXT:    sra $1, $1, 24
+; O3-NEXT:  # %bb.3: # %entry
+; O3-NEXT:    sll $1, $1, 24
+; O3-NEXT:    jr $ra
+; O3-NEXT:    sra $2, $1, 24
+;
+; MIPS32EB-LABEL: AtomicSwap8:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    lw $1, %got(y)($1)
+; MIPS32EB-NEXT:    addiu $2, $zero, -4
+; MIPS32EB-NEXT:    and $2, $1, $2
+; MIPS32EB-NEXT:    andi $1, $1, 3
+; MIPS32EB-NEXT:    xori $1, $1, 3
+; MIPS32EB-NEXT:    sll $3, $1, 3
+; MIPS32EB-NEXT:    ori $1, $zero, 255
+; MIPS32EB-NEXT:    sllv $5, $1, $3
+; MIPS32EB-NEXT:    nor $6, $zero, $5
+; MIPS32EB-NEXT:    sllv $4, $4, $3
+; MIPS32EB-NEXT:  $BB11_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $7, 0($2)
+; MIPS32EB-NEXT:    and $8, $4, $5
+; MIPS32EB-NEXT:    and $9, $7, $6
+; MIPS32EB-NEXT:    or $9, $9, $8
+; MIPS32EB-NEXT:    sc $9, 0($2)
+; MIPS32EB-NEXT:    beqz $9, $BB11_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    and $1, $7, $5
+; MIPS32EB-NEXT:    srlv $1, $1, $3
+; MIPS32EB-NEXT:    sll $1, $1, 24
+; MIPS32EB-NEXT:    sra $1, $1, 24
+; MIPS32EB-NEXT:  # %bb.3: # %entry
+; MIPS32EB-NEXT:    sll $1, $1, 24
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    sra $2, $1, 24
 entry:
   %0 = atomicrmw xchg i8* @y, i8 %newval monotonic
   ret i8 %0
-
-; ALL-LABEL: AtomicSwap8:
-
-; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(y)
-; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(y)(
-
-; ALL:           addiu   $[[R1:[0-9]+]], $zero, -4
-; ALL:           and     $[[R2:[0-9]+]], $[[R0]], $[[R1]]
-; ALL:           andi    $[[R3:[0-9]+]], $[[R0]], 3
-; CHECK-EL:      sll     $[[R5:[0-9]+]], $[[R3]], 3
-; CHECK-EB:      xori    $[[R4:[0-9]+]], $[[R3]], 3
-; CHECK-EB:      sll     $[[R5:[0-9]+]], $[[R4]], 3
-; ALL:           ori     $[[R6:[0-9]+]], $zero, 255
-; ALL:           sllv    $[[R7:[0-9]+]], $[[R6]], $[[R5]]
-; ALL:           nor     $[[R8:[0-9]+]], $zero, $[[R7]]
-; ALL:           sllv    $[[R9:[0-9]+]], $4, $[[R5]]
-
-; ALL:       [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; ALL:           ll      $[[R10:[0-9]+]], 0($[[R2]])
-; ALL:           and     $[[R18:[0-9]+]], $[[R9]], $[[R7]]
-; ALL:           and     $[[R13:[0-9]+]], $[[R10]], $[[R8]]
-; ALL:           or      $[[R14:[0-9]+]], $[[R13]], $[[R18]]
-; ALL:           sc      $[[R14]], 0($[[R2]])
-; NOT-MICROMIPS: beqz    $[[R14]], [[BB0]]
-; MICROMIPS:     beqzc   $[[R14]], [[BB0]]
-; MIPSR6:        beqzc   $[[R14]], [[BB0]]
-
-; ALL:           and     $[[R15:[0-9]+]], $[[R10]], $[[R7]]
-; ALL:           srlv    $[[R16:[0-9]+]], $[[R15]], $[[R5]]
-
-; NO-SEB-SEH:    sll     $[[R17:[0-9]+]], $[[R16]], 24
-; NO-SEB-SEH:    sra     $2, $[[R17]], 24
-
-; HAS-SEB-SEH:   seb     $2, $[[R16]]
-
 }
 
 define signext i8 @AtomicCmpSwap8(i8 signext %oldval, i8 signext %newval) nounwind {
+; MIPS32-LABEL: AtomicCmpSwap8:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    lw $1, %got(y)($1)
+; MIPS32-NEXT:    addiu $2, $zero, -4
+; MIPS32-NEXT:    and $2, $1, $2
+; MIPS32-NEXT:    andi $1, $1, 3
+; MIPS32-NEXT:    sll $3, $1, 3
+; MIPS32-NEXT:    ori $1, $zero, 255
+; MIPS32-NEXT:    sllv $6, $1, $3
+; MIPS32-NEXT:    nor $7, $zero, $6
+; MIPS32-NEXT:    andi $1, $4, 255
+; MIPS32-NEXT:    sllv $4, $1, $3
+; MIPS32-NEXT:    andi $1, $5, 255
+; MIPS32-NEXT:    sllv $5, $1, $3
+; MIPS32-NEXT:  $BB12_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $8, 0($2)
+; MIPS32-NEXT:    and $9, $8, $6
+; MIPS32-NEXT:    bne $9, $4, $BB12_3
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; MIPS32-NEXT:    and $8, $8, $7
+; MIPS32-NEXT:    or $8, $8, $5
+; MIPS32-NEXT:    sc $8, 0($2)
+; MIPS32-NEXT:    beqz $8, $BB12_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  $BB12_3: # %entry
+; MIPS32-NEXT:    srlv $1, $9, $3
+; MIPS32-NEXT:    sll $1, $1, 24
+; MIPS32-NEXT:    sra $1, $1, 24
+; MIPS32-NEXT:  # %bb.4: # %entry
+; MIPS32-NEXT:    sll $1, $1, 24
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    sra $2, $1, 24
+;
+; MIPS32O0-LABEL: AtomicCmpSwap8:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    lw $2, %got(y)($2)
+; MIPS32O0-NEXT:    addiu $25, $zero, -4
+; MIPS32O0-NEXT:    and $25, $2, $25
+; MIPS32O0-NEXT:    andi $2, $2, 3
+; MIPS32O0-NEXT:    sll $2, $2, 3
+; MIPS32O0-NEXT:    ori $1, $zero, 255
+; MIPS32O0-NEXT:    sllv $1, $1, $2
+; MIPS32O0-NEXT:    nor $3, $zero, $1
+; MIPS32O0-NEXT:    andi $4, $4, 255
+; MIPS32O0-NEXT:    sllv $4, $4, $2
+; MIPS32O0-NEXT:    andi $5, $5, 255
+; MIPS32O0-NEXT:    sllv $5, $5, $2
+; MIPS32O0-NEXT:  $BB12_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $7, 0($25)
+; MIPS32O0-NEXT:    and $8, $7, $1
+; MIPS32O0-NEXT:    bne $8, $4, $BB12_3
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; MIPS32O0-NEXT:    and $7, $7, $3
+; MIPS32O0-NEXT:    or $7, $7, $5
+; MIPS32O0-NEXT:    sc $7, 0($25)
+; MIPS32O0-NEXT:    beqz $7, $BB12_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  $BB12_3: # %entry
+; MIPS32O0-NEXT:    srlv $6, $8, $2
+; MIPS32O0-NEXT:    sll $6, $6, 24
+; MIPS32O0-NEXT:    sra $6, $6, 24
+; MIPS32O0-NEXT:  # %bb.4: # %entry
+; MIPS32O0-NEXT:    sw $6, 4($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:  # %bb.5: # %entry
+; MIPS32O0-NEXT:    lw $1, 4($sp) # 4-byte Folded Reload
+; MIPS32O0-NEXT:    sll $2, $1, 24
+; MIPS32O0-NEXT:    sra $2, $2, 24
+; MIPS32O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicCmpSwap8:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    lw $1, %got(y)($1)
+; MIPS32R2-NEXT:    addiu $2, $zero, -4
+; MIPS32R2-NEXT:    and $3, $1, $2
+; MIPS32R2-NEXT:    andi $1, $1, 3
+; MIPS32R2-NEXT:    sll $1, $1, 3
+; MIPS32R2-NEXT:    ori $2, $zero, 255
+; MIPS32R2-NEXT:    sllv $6, $2, $1
+; MIPS32R2-NEXT:    nor $7, $zero, $6
+; MIPS32R2-NEXT:    andi $2, $4, 255
+; MIPS32R2-NEXT:    sllv $4, $2, $1
+; MIPS32R2-NEXT:    andi $2, $5, 255
+; MIPS32R2-NEXT:    sllv $5, $2, $1
+; MIPS32R2-NEXT:  $BB12_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $8, 0($3)
+; MIPS32R2-NEXT:    and $9, $8, $6
+; MIPS32R2-NEXT:    bne $9, $4, $BB12_3
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; MIPS32R2-NEXT:    and $8, $8, $7
+; MIPS32R2-NEXT:    or $8, $8, $5
+; MIPS32R2-NEXT:    sc $8, 0($3)
+; MIPS32R2-NEXT:    beqz $8, $BB12_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  $BB12_3: # %entry
+; MIPS32R2-NEXT:    srlv $2, $9, $1
+; MIPS32R2-NEXT:    seb $2, $2
+; MIPS32R2-NEXT:  # %bb.4: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    nop
+;
+; MIPS32R6-LABEL: AtomicCmpSwap8:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    lw $1, %got(y)($1)
+; MIPS32R6-NEXT:    addiu $2, $zero, -4
+; MIPS32R6-NEXT:    and $3, $1, $2
+; MIPS32R6-NEXT:    andi $1, $1, 3
+; MIPS32R6-NEXT:    sll $1, $1, 3
+; MIPS32R6-NEXT:    ori $2, $zero, 255
+; MIPS32R6-NEXT:    sllv $6, $2, $1
+; MIPS32R6-NEXT:    nor $7, $zero, $6
+; MIPS32R6-NEXT:    andi $2, $4, 255
+; MIPS32R6-NEXT:    sllv $4, $2, $1
+; MIPS32R6-NEXT:    andi $2, $5, 255
+; MIPS32R6-NEXT:    sllv $5, $2, $1
+; MIPS32R6-NEXT:  $BB12_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $8, 0($3)
+; MIPS32R6-NEXT:    and $9, $8, $6
+; MIPS32R6-NEXT:    bnec $9, $4, $BB12_3
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; MIPS32R6-NEXT:    and $8, $8, $7
+; MIPS32R6-NEXT:    or $8, $8, $5
+; MIPS32R6-NEXT:    sc $8, 0($3)
+; MIPS32R6-NEXT:    beqzc $8, $BB12_1
+; MIPS32R6-NEXT:  $BB12_3: # %entry
+; MIPS32R6-NEXT:    srlv $2, $9, $1
+; MIPS32R6-NEXT:    seb $2, $2
+; MIPS32R6-NEXT:  # %bb.4: # %entry
+; MIPS32R6-NEXT:    jrc $ra
+;
+; MIPS32R6O0-LABEL: AtomicCmpSwap8:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -16
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $5
+; MIPS32R6O0-NEXT:    move $1, $4
+; MIPS32R6O0-NEXT:    lw $2, %got(y)($2)
+; MIPS32R6O0-NEXT:    addiu $3, $zero, -4
+; MIPS32R6O0-NEXT:    and $3, $2, $3
+; MIPS32R6O0-NEXT:    andi $2, $2, 3
+; MIPS32R6O0-NEXT:    sll $2, $2, 3
+; MIPS32R6O0-NEXT:    ori $6, $zero, 255
+; MIPS32R6O0-NEXT:    sllv $6, $6, $2
+; MIPS32R6O0-NEXT:    nor $7, $zero, $6
+; MIPS32R6O0-NEXT:    andi $4, $4, 255
+; MIPS32R6O0-NEXT:    sllv $4, $4, $2
+; MIPS32R6O0-NEXT:    andi $5, $5, 255
+; MIPS32R6O0-NEXT:    sllv $5, $5, $2
+; MIPS32R6O0-NEXT:  $BB12_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $9, 0($3)
+; MIPS32R6O0-NEXT:    and $10, $9, $6
+; MIPS32R6O0-NEXT:    bnec $10, $4, $BB12_3
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; MIPS32R6O0-NEXT:    and $9, $9, $7
+; MIPS32R6O0-NEXT:    or $9, $9, $5
+; MIPS32R6O0-NEXT:    sc $9, 0($3)
+; MIPS32R6O0-NEXT:    beqzc $9, $BB12_1
+; MIPS32R6O0-NEXT:  $BB12_3: # %entry
+; MIPS32R6O0-NEXT:    srlv $8, $10, $2
+; MIPS32R6O0-NEXT:    seb $8, $8
+; MIPS32R6O0-NEXT:  # %bb.4: # %entry
+; MIPS32R6O0-NEXT:    sw $1, 12($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $8, 8($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:  # %bb.5: # %entry
+; MIPS32R6O0-NEXT:    lw $2, 8($sp) # 4-byte Folded Reload
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 16
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicCmpSwap8:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap8)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap8)))
+; MIPS4-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS4-NEXT:    daddiu $2, $zero, -4
+; MIPS4-NEXT:    and $2, $1, $2
+; MIPS4-NEXT:    andi $1, $1, 3
+; MIPS4-NEXT:    sll $3, $1, 3
+; MIPS4-NEXT:    ori $1, $zero, 255
+; MIPS4-NEXT:    sllv $6, $1, $3
+; MIPS4-NEXT:    nor $7, $zero, $6
+; MIPS4-NEXT:    andi $1, $4, 255
+; MIPS4-NEXT:    sllv $4, $1, $3
+; MIPS4-NEXT:    andi $1, $5, 255
+; MIPS4-NEXT:    sllv $5, $1, $3
+; MIPS4-NEXT:  .LBB12_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $8, 0($2)
+; MIPS4-NEXT:    and $9, $8, $6
+; MIPS4-NEXT:    bne $9, $4, .LBB12_3
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; MIPS4-NEXT:    and $8, $8, $7
+; MIPS4-NEXT:    or $8, $8, $5
+; MIPS4-NEXT:    sc $8, 0($2)
+; MIPS4-NEXT:    beqz $8, .LBB12_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  .LBB12_3: # %entry
+; MIPS4-NEXT:    srlv $1, $9, $3
+; MIPS4-NEXT:    sll $1, $1, 24
+; MIPS4-NEXT:    sra $1, $1, 24
+; MIPS4-NEXT:  # %bb.4: # %entry
+; MIPS4-NEXT:    sll $1, $1, 24
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    sra $2, $1, 24
+;
+; MIPS64-LABEL: AtomicCmpSwap8:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap8)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap8)))
+; MIPS64-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64-NEXT:    daddiu $2, $zero, -4
+; MIPS64-NEXT:    and $2, $1, $2
+; MIPS64-NEXT:    andi $1, $1, 3
+; MIPS64-NEXT:    sll $3, $1, 3
+; MIPS64-NEXT:    ori $1, $zero, 255
+; MIPS64-NEXT:    sllv $6, $1, $3
+; MIPS64-NEXT:    nor $7, $zero, $6
+; MIPS64-NEXT:    andi $1, $4, 255
+; MIPS64-NEXT:    sllv $4, $1, $3
+; MIPS64-NEXT:    andi $1, $5, 255
+; MIPS64-NEXT:    sllv $5, $1, $3
+; MIPS64-NEXT:  .LBB12_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $8, 0($2)
+; MIPS64-NEXT:    and $9, $8, $6
+; MIPS64-NEXT:    bne $9, $4, .LBB12_3
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; MIPS64-NEXT:    and $8, $8, $7
+; MIPS64-NEXT:    or $8, $8, $5
+; MIPS64-NEXT:    sc $8, 0($2)
+; MIPS64-NEXT:    beqz $8, .LBB12_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  .LBB12_3: # %entry
+; MIPS64-NEXT:    srlv $1, $9, $3
+; MIPS64-NEXT:    sll $1, $1, 24
+; MIPS64-NEXT:    sra $1, $1, 24
+; MIPS64-NEXT:  # %bb.4: # %entry
+; MIPS64-NEXT:    sll $1, $1, 24
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    sra $2, $1, 24
+;
+; MIPS64R2-LABEL: AtomicCmpSwap8:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap8)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap8)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R2-NEXT:    daddiu $2, $zero, -4
+; MIPS64R2-NEXT:    and $3, $1, $2
+; MIPS64R2-NEXT:    andi $1, $1, 3
+; MIPS64R2-NEXT:    sll $1, $1, 3
+; MIPS64R2-NEXT:    ori $2, $zero, 255
+; MIPS64R2-NEXT:    sllv $6, $2, $1
+; MIPS64R2-NEXT:    nor $7, $zero, $6
+; MIPS64R2-NEXT:    andi $2, $4, 255
+; MIPS64R2-NEXT:    sllv $4, $2, $1
+; MIPS64R2-NEXT:    andi $2, $5, 255
+; MIPS64R2-NEXT:    sllv $5, $2, $1
+; MIPS64R2-NEXT:  .LBB12_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $8, 0($3)
+; MIPS64R2-NEXT:    and $9, $8, $6
+; MIPS64R2-NEXT:    bne $9, $4, .LBB12_3
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; MIPS64R2-NEXT:    and $8, $8, $7
+; MIPS64R2-NEXT:    or $8, $8, $5
+; MIPS64R2-NEXT:    sc $8, 0($3)
+; MIPS64R2-NEXT:    beqz $8, .LBB12_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  .LBB12_3: # %entry
+; MIPS64R2-NEXT:    srlv $2, $9, $1
+; MIPS64R2-NEXT:    seb $2, $2
+; MIPS64R2-NEXT:  # %bb.4: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicCmpSwap8:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap8)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap8)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R6-NEXT:    daddiu $2, $zero, -4
+; MIPS64R6-NEXT:    and $3, $1, $2
+; MIPS64R6-NEXT:    andi $1, $1, 3
+; MIPS64R6-NEXT:    sll $1, $1, 3
+; MIPS64R6-NEXT:    ori $2, $zero, 255
+; MIPS64R6-NEXT:    sllv $6, $2, $1
+; MIPS64R6-NEXT:    nor $7, $zero, $6
+; MIPS64R6-NEXT:    andi $2, $4, 255
+; MIPS64R6-NEXT:    sllv $4, $2, $1
+; MIPS64R6-NEXT:    andi $2, $5, 255
+; MIPS64R6-NEXT:    sllv $5, $2, $1
+; MIPS64R6-NEXT:  .LBB12_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $8, 0($3)
+; MIPS64R6-NEXT:    and $9, $8, $6
+; MIPS64R6-NEXT:    bnec $9, $4, .LBB12_3
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; MIPS64R6-NEXT:    and $8, $8, $7
+; MIPS64R6-NEXT:    or $8, $8, $5
+; MIPS64R6-NEXT:    sc $8, 0($3)
+; MIPS64R6-NEXT:    beqzc $8, .LBB12_1
+; MIPS64R6-NEXT:  .LBB12_3: # %entry
+; MIPS64R6-NEXT:    srlv $2, $9, $1
+; MIPS64R6-NEXT:    seb $2, $2
+; MIPS64R6-NEXT:  # %bb.4: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicCmpSwap8:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap8)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap8)))
+; MIPS64R6O0-NEXT:    move $2, $5
+; MIPS64R6O0-NEXT:    move $3, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(y)($1)
+; MIPS64R6O0-NEXT:    daddiu $4, $zero, -4
+; MIPS64R6O0-NEXT:    and $4, $1, $4
+; MIPS64R6O0-NEXT:    andi $6, $1, 3
+; MIPS64R6O0-NEXT:    xori $6, $6, 3
+; MIPS64R6O0-NEXT:    sll $6, $6, 3
+; MIPS64R6O0-NEXT:    ori $7, $zero, 255
+; MIPS64R6O0-NEXT:    sllv $7, $7, $6
+; MIPS64R6O0-NEXT:    nor $8, $zero, $7
+; MIPS64R6O0-NEXT:    andi $3, $3, 255
+; MIPS64R6O0-NEXT:    sllv $3, $3, $6
+; MIPS64R6O0-NEXT:    andi $2, $2, 255
+; MIPS64R6O0-NEXT:    sllv $2, $2, $6
+; MIPS64R6O0-NEXT:  .LBB12_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $10, 0($4)
+; MIPS64R6O0-NEXT:    and $11, $10, $7
+; MIPS64R6O0-NEXT:    bnec $11, $3, .LBB12_3
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; MIPS64R6O0-NEXT:    and $10, $10, $8
+; MIPS64R6O0-NEXT:    or $10, $10, $2
+; MIPS64R6O0-NEXT:    sc $10, 0($4)
+; MIPS64R6O0-NEXT:    beqzc $10, .LBB12_1
+; MIPS64R6O0-NEXT:  .LBB12_3: # %entry
+; MIPS64R6O0-NEXT:    srlv $9, $11, $6
+; MIPS64R6O0-NEXT:    seb $9, $9
+; MIPS64R6O0-NEXT:  # %bb.4: # %entry
+; MIPS64R6O0-NEXT:    sw $9, 12($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:  # %bb.5: # %entry
+; MIPS64R6O0-NEXT:    lw $2, 12($sp) # 4-byte Folded Reload
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicCmpSwap8:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    lw $1, %got(y)($2)
+; MM32-NEXT:    addiu $2, $zero, -4
+; MM32-NEXT:    and $3, $1, $2
+; MM32-NEXT:    andi $1, $1, 3
+; MM32-NEXT:    sll $1, $1, 3
+; MM32-NEXT:    ori $2, $zero, 255
+; MM32-NEXT:    sllv $6, $2, $1
+; MM32-NEXT:    nor $7, $zero, $6
+; MM32-NEXT:    andi $2, $4, 255
+; MM32-NEXT:    sllv $4, $2, $1
+; MM32-NEXT:    andi $2, $5, 255
+; MM32-NEXT:    sllv $5, $2, $1
+; MM32-NEXT:  $BB12_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $8, 0($3)
+; MM32-NEXT:    and $9, $8, $6
+; MM32-NEXT:    bne $9, $4, $BB12_3
+; MM32-NEXT:    nop
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; MM32-NEXT:    and $8, $8, $7
+; MM32-NEXT:    or $8, $8, $5
+; MM32-NEXT:    sc $8, 0($3)
+; MM32-NEXT:    beqzc $8, $BB12_1
+; MM32-NEXT:  $BB12_3: # %entry
+; MM32-NEXT:    srlv $2, $9, $1
+; MM32-NEXT:    seb $2, $2
+; MM32-NEXT:  # %bb.4: # %entry
+; MM32-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicCmpSwap8:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    lw $1, %got(y)($1)
+; O1-NEXT:    addiu $2, $zero, -4
+; O1-NEXT:    and $2, $1, $2
+; O1-NEXT:    andi $1, $1, 3
+; O1-NEXT:    sll $3, $1, 3
+; O1-NEXT:    ori $1, $zero, 255
+; O1-NEXT:    sllv $6, $1, $3
+; O1-NEXT:    nor $7, $zero, $6
+; O1-NEXT:    andi $1, $4, 255
+; O1-NEXT:    sllv $4, $1, $3
+; O1-NEXT:    andi $1, $5, 255
+; O1-NEXT:    sllv $5, $1, $3
+; O1-NEXT:  $BB12_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $8, 0($2)
+; O1-NEXT:    and $9, $8, $6
+; O1-NEXT:    bne $9, $4, $BB12_3
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; O1-NEXT:    and $8, $8, $7
+; O1-NEXT:    or $8, $8, $5
+; O1-NEXT:    sc $8, 0($2)
+; O1-NEXT:    beqz $8, $BB12_1
+; O1-NEXT:    nop
+; O1-NEXT:  $BB12_3: # %entry
+; O1-NEXT:    srlv $1, $9, $3
+; O1-NEXT:    sll $1, $1, 24
+; O1-NEXT:    sra $1, $1, 24
+; O1-NEXT:  # %bb.4: # %entry
+; O1-NEXT:    sll $1, $1, 24
+; O1-NEXT:    jr $ra
+; O1-NEXT:    sra $2, $1, 24
+;
+; O2-LABEL: AtomicCmpSwap8:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    lw $1, %got(y)($1)
+; O2-NEXT:    addiu $2, $zero, -4
+; O2-NEXT:    and $2, $1, $2
+; O2-NEXT:    andi $1, $1, 3
+; O2-NEXT:    sll $3, $1, 3
+; O2-NEXT:    ori $1, $zero, 255
+; O2-NEXT:    sllv $6, $1, $3
+; O2-NEXT:    nor $7, $zero, $6
+; O2-NEXT:    andi $1, $4, 255
+; O2-NEXT:    sllv $4, $1, $3
+; O2-NEXT:    andi $1, $5, 255
+; O2-NEXT:    sllv $5, $1, $3
+; O2-NEXT:  $BB12_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $8, 0($2)
+; O2-NEXT:    and $9, $8, $6
+; O2-NEXT:    bne $9, $4, $BB12_3
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; O2-NEXT:    and $8, $8, $7
+; O2-NEXT:    or $8, $8, $5
+; O2-NEXT:    sc $8, 0($2)
+; O2-NEXT:    beqz $8, $BB12_1
+; O2-NEXT:    nop
+; O2-NEXT:  $BB12_3: # %entry
+; O2-NEXT:    srlv $1, $9, $3
+; O2-NEXT:    sll $1, $1, 24
+; O2-NEXT:    sra $1, $1, 24
+; O2-NEXT:  # %bb.4: # %entry
+; O2-NEXT:    sll $1, $1, 24
+; O2-NEXT:    jr $ra
+; O2-NEXT:    sra $2, $1, 24
+;
+; O3-LABEL: AtomicCmpSwap8:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    addiu $2, $zero, -4
+; O3-NEXT:    lw $1, %got(y)($1)
+; O3-NEXT:    and $2, $1, $2
+; O3-NEXT:    andi $1, $1, 3
+; O3-NEXT:    sll $3, $1, 3
+; O3-NEXT:    ori $1, $zero, 255
+; O3-NEXT:    sllv $6, $1, $3
+; O3-NEXT:    andi $1, $4, 255
+; O3-NEXT:    sllv $4, $1, $3
+; O3-NEXT:    andi $1, $5, 255
+; O3-NEXT:    nor $7, $zero, $6
+; O3-NEXT:    sllv $5, $1, $3
+; O3-NEXT:  $BB12_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $8, 0($2)
+; O3-NEXT:    and $9, $8, $6
+; O3-NEXT:    bne $9, $4, $BB12_3
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; O3-NEXT:    and $8, $8, $7
+; O3-NEXT:    or $8, $8, $5
+; O3-NEXT:    sc $8, 0($2)
+; O3-NEXT:    beqz $8, $BB12_1
+; O3-NEXT:    nop
+; O3-NEXT:  $BB12_3: # %entry
+; O3-NEXT:    srlv $1, $9, $3
+; O3-NEXT:    sll $1, $1, 24
+; O3-NEXT:    sra $1, $1, 24
+; O3-NEXT:  # %bb.4: # %entry
+; O3-NEXT:    sll $1, $1, 24
+; O3-NEXT:    jr $ra
+; O3-NEXT:    sra $2, $1, 24
+;
+; MIPS32EB-LABEL: AtomicCmpSwap8:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    lw $1, %got(y)($1)
+; MIPS32EB-NEXT:    addiu $2, $zero, -4
+; MIPS32EB-NEXT:    and $2, $1, $2
+; MIPS32EB-NEXT:    andi $1, $1, 3
+; MIPS32EB-NEXT:    xori $1, $1, 3
+; MIPS32EB-NEXT:    sll $3, $1, 3
+; MIPS32EB-NEXT:    ori $1, $zero, 255
+; MIPS32EB-NEXT:    sllv $6, $1, $3
+; MIPS32EB-NEXT:    nor $7, $zero, $6
+; MIPS32EB-NEXT:    andi $1, $4, 255
+; MIPS32EB-NEXT:    sllv $4, $1, $3
+; MIPS32EB-NEXT:    andi $1, $5, 255
+; MIPS32EB-NEXT:    sllv $5, $1, $3
+; MIPS32EB-NEXT:  $BB12_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $8, 0($2)
+; MIPS32EB-NEXT:    and $9, $8, $6
+; MIPS32EB-NEXT:    bne $9, $4, $BB12_3
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    # in Loop: Header=BB12_1 Depth=1
+; MIPS32EB-NEXT:    and $8, $8, $7
+; MIPS32EB-NEXT:    or $8, $8, $5
+; MIPS32EB-NEXT:    sc $8, 0($2)
+; MIPS32EB-NEXT:    beqz $8, $BB12_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  $BB12_3: # %entry
+; MIPS32EB-NEXT:    srlv $1, $9, $3
+; MIPS32EB-NEXT:    sll $1, $1, 24
+; MIPS32EB-NEXT:    sra $1, $1, 24
+; MIPS32EB-NEXT:  # %bb.4: # %entry
+; MIPS32EB-NEXT:    sll $1, $1, 24
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    sra $2, $1, 24
 entry:
   %pair0 = cmpxchg i8* @y, i8 %oldval, i8 %newval monotonic monotonic
   %0 = extractvalue { i8, i1 } %pair0, 0
   ret i8 %0
-
-; ALL-LABEL: AtomicCmpSwap8:
-
-; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(y)
-; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(y)(
-
-; ALL:           addiu   $[[R1:[0-9]+]], $zero, -4
-; ALL:           and     $[[R2:[0-9]+]], $[[R0]], $[[R1]]
-; ALL:           andi    $[[R3:[0-9]+]], $[[R0]], 3
-; CHECK-EL:      sll     $[[R5:[0-9]+]], $[[R3]], 3
-; CHECK-EB:      xori    $[[R4:[0-9]+]], $[[R3]], 3
-; CHECK-EB:      sll     $[[R5:[0-9]+]], $[[R4]], 3
-; ALL:           ori     $[[R6:[0-9]+]], $zero, 255
-; ALL:           sllv    $[[R7:[0-9]+]], $[[R6]], $[[R5]]
-; ALL:           nor     $[[R8:[0-9]+]], $zero, $[[R7]]
-; ALL:           andi    $[[R9:[0-9]+]], $4, 255
-; ALL:           sllv    $[[R10:[0-9]+]], $[[R9]], $[[R5]]
-; ALL:           andi    $[[R11:[0-9]+]], $5, 255
-; ALL:           sllv    $[[R12:[0-9]+]], $[[R11]], $[[R5]]
-
-; ALL:       [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; ALL:           ll      $[[R13:[0-9]+]], 0($[[R2]])
-; ALL:           and     $[[R14:[0-9]+]], $[[R13]], $[[R7]]
-; NOT-MICROMIPS: bne     $[[R14]], $[[R10]], [[BB1:(\$|\.L)[A-Z_0-9]+]]
-; MICROMIPS:     bne     $[[R14]], $[[R10]], [[BB1:(\$|\.L)[A-Z_0-9]+]]
-; MIPSR6:        bnec    $[[R14]], $[[R10]], [[BB1:(\$|\.L)[A-Z_0-9]+]]
-
-; ALL:           and     $[[R15:[0-9]+]], $[[R13]], $[[R8]]
-; ALL:           or      $[[R16:[0-9]+]], $[[R15]], $[[R12]]
-; ALL:           sc      $[[R16]], 0($[[R2]])
-; NOT-MICROMIPS: beqz    $[[R16]], [[BB0]]
-; MICROMIPS:     beqzc   $[[R16]], [[BB0]]
-; MIPSR6:        beqzc   $[[R16]], [[BB0]]
-
-; ALL:       [[BB1]]:
-; ALL:           srlv    $[[R17:[0-9]+]], $[[R14]], $[[R5]]
-
-; NO-SEB-SEH:    sll     $[[R18:[0-9]+]], $[[R17]], 24
-; NO-SEB-SEH:    sra     $2, $[[R18]], 24
-
-; HAS-SEB-SEH:   seb     $2, $[[R17]]
 }
 
 define i1 @AtomicCmpSwapRes8(i8* %ptr, i8 signext %oldval, i8 signext %newval) nounwind {
+; MIPS32-LABEL: AtomicCmpSwapRes8:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    addiu $1, $zero, -4
+; MIPS32-NEXT:    and $2, $4, $1
+; MIPS32-NEXT:    andi $1, $4, 3
+; MIPS32-NEXT:    sll $3, $1, 3
+; MIPS32-NEXT:    ori $1, $zero, 255
+; MIPS32-NEXT:    sllv $4, $1, $3
+; MIPS32-NEXT:    nor $7, $zero, $4
+; MIPS32-NEXT:    andi $1, $5, 255
+; MIPS32-NEXT:    sllv $8, $1, $3
+; MIPS32-NEXT:    andi $1, $6, 255
+; MIPS32-NEXT:    sllv $6, $1, $3
+; MIPS32-NEXT:  $BB13_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $9, 0($2)
+; MIPS32-NEXT:    and $10, $9, $4
+; MIPS32-NEXT:    bne $10, $8, $BB13_3
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; MIPS32-NEXT:    and $9, $9, $7
+; MIPS32-NEXT:    or $9, $9, $6
+; MIPS32-NEXT:    sc $9, 0($2)
+; MIPS32-NEXT:    beqz $9, $BB13_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  $BB13_3: # %entry
+; MIPS32-NEXT:    srlv $1, $10, $3
+; MIPS32-NEXT:    sll $1, $1, 24
+; MIPS32-NEXT:    sra $1, $1, 24
+; MIPS32-NEXT:  # %bb.4: # %entry
+; MIPS32-NEXT:    sll $2, $5, 24
+; MIPS32-NEXT:    sra $2, $2, 24
+; MIPS32-NEXT:    xor $1, $1, $2
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    sltiu $2, $1, 1
+;
+; MIPS32O0-LABEL: AtomicCmpSwapRes8:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32O0-NEXT:    addiu $1, $zero, -4
+; MIPS32O0-NEXT:    and $1, $4, $1
+; MIPS32O0-NEXT:    andi $4, $4, 3
+; MIPS32O0-NEXT:    sll $4, $4, 3
+; MIPS32O0-NEXT:    ori $2, $zero, 255
+; MIPS32O0-NEXT:    sllv $2, $2, $4
+; MIPS32O0-NEXT:    nor $3, $zero, $2
+; MIPS32O0-NEXT:    andi $7, $5, 255
+; MIPS32O0-NEXT:    sllv $7, $7, $4
+; MIPS32O0-NEXT:    andi $6, $6, 255
+; MIPS32O0-NEXT:    sllv $6, $6, $4
+; MIPS32O0-NEXT:  $BB13_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $9, 0($1)
+; MIPS32O0-NEXT:    and $10, $9, $2
+; MIPS32O0-NEXT:    bne $10, $7, $BB13_3
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; MIPS32O0-NEXT:    and $9, $9, $3
+; MIPS32O0-NEXT:    or $9, $9, $6
+; MIPS32O0-NEXT:    sc $9, 0($1)
+; MIPS32O0-NEXT:    beqz $9, $BB13_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  $BB13_3: # %entry
+; MIPS32O0-NEXT:    srlv $8, $10, $4
+; MIPS32O0-NEXT:    sll $8, $8, 24
+; MIPS32O0-NEXT:    sra $8, $8, 24
+; MIPS32O0-NEXT:  # %bb.4: # %entry
+; MIPS32O0-NEXT:    sw $5, 4($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:    sw $8, 0($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:  # %bb.5: # %entry
+; MIPS32O0-NEXT:    lw $1, 4($sp) # 4-byte Folded Reload
+; MIPS32O0-NEXT:    sll $2, $1, 24
+; MIPS32O0-NEXT:    sra $2, $2, 24
+; MIPS32O0-NEXT:    lw $3, 0($sp) # 4-byte Folded Reload
+; MIPS32O0-NEXT:    xor $2, $3, $2
+; MIPS32O0-NEXT:    sltiu $2, $2, 1
+; MIPS32O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicCmpSwapRes8:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    addiu $1, $zero, -4
+; MIPS32R2-NEXT:    and $2, $4, $1
+; MIPS32R2-NEXT:    andi $1, $4, 3
+; MIPS32R2-NEXT:    sll $3, $1, 3
+; MIPS32R2-NEXT:    ori $1, $zero, 255
+; MIPS32R2-NEXT:    sllv $4, $1, $3
+; MIPS32R2-NEXT:    nor $7, $zero, $4
+; MIPS32R2-NEXT:    andi $1, $5, 255
+; MIPS32R2-NEXT:    sllv $8, $1, $3
+; MIPS32R2-NEXT:    andi $1, $6, 255
+; MIPS32R2-NEXT:    sllv $6, $1, $3
+; MIPS32R2-NEXT:  $BB13_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $9, 0($2)
+; MIPS32R2-NEXT:    and $10, $9, $4
+; MIPS32R2-NEXT:    bne $10, $8, $BB13_3
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; MIPS32R2-NEXT:    and $9, $9, $7
+; MIPS32R2-NEXT:    or $9, $9, $6
+; MIPS32R2-NEXT:    sc $9, 0($2)
+; MIPS32R2-NEXT:    beqz $9, $BB13_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  $BB13_3: # %entry
+; MIPS32R2-NEXT:    srlv $1, $10, $3
+; MIPS32R2-NEXT:    seb $1, $1
+; MIPS32R2-NEXT:  # %bb.4: # %entry
+; MIPS32R2-NEXT:    xor $1, $1, $5
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    sltiu $2, $1, 1
+;
+; MIPS32R6-LABEL: AtomicCmpSwapRes8:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    addiu $1, $zero, -4
+; MIPS32R6-NEXT:    and $2, $4, $1
+; MIPS32R6-NEXT:    andi $1, $4, 3
+; MIPS32R6-NEXT:    sll $3, $1, 3
+; MIPS32R6-NEXT:    ori $1, $zero, 255
+; MIPS32R6-NEXT:    sllv $4, $1, $3
+; MIPS32R6-NEXT:    nor $7, $zero, $4
+; MIPS32R6-NEXT:    andi $1, $5, 255
+; MIPS32R6-NEXT:    sllv $8, $1, $3
+; MIPS32R6-NEXT:    andi $1, $6, 255
+; MIPS32R6-NEXT:    sllv $6, $1, $3
+; MIPS32R6-NEXT:  $BB13_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $9, 0($2)
+; MIPS32R6-NEXT:    and $10, $9, $4
+; MIPS32R6-NEXT:    bnec $10, $8, $BB13_3
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; MIPS32R6-NEXT:    and $9, $9, $7
+; MIPS32R6-NEXT:    or $9, $9, $6
+; MIPS32R6-NEXT:    sc $9, 0($2)
+; MIPS32R6-NEXT:    beqzc $9, $BB13_1
+; MIPS32R6-NEXT:  $BB13_3: # %entry
+; MIPS32R6-NEXT:    srlv $1, $10, $3
+; MIPS32R6-NEXT:    seb $1, $1
+; MIPS32R6-NEXT:  # %bb.4: # %entry
+; MIPS32R6-NEXT:    xor $1, $1, $5
+; MIPS32R6-NEXT:    jr $ra
+; MIPS32R6-NEXT:    sltiu $2, $1, 1
+;
+; MIPS32R6O0-LABEL: AtomicCmpSwapRes8:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -24
+; MIPS32R6O0-NEXT:    move $1, $6
+; MIPS32R6O0-NEXT:    move $2, $5
+; MIPS32R6O0-NEXT:    move $3, $4
+; MIPS32R6O0-NEXT:    addiu $7, $zero, -4
+; MIPS32R6O0-NEXT:    and $7, $4, $7
+; MIPS32R6O0-NEXT:    andi $4, $4, 3
+; MIPS32R6O0-NEXT:    sll $4, $4, 3
+; MIPS32R6O0-NEXT:    ori $8, $zero, 255
+; MIPS32R6O0-NEXT:    sllv $8, $8, $4
+; MIPS32R6O0-NEXT:    nor $9, $zero, $8
+; MIPS32R6O0-NEXT:    andi $10, $5, 255
+; MIPS32R6O0-NEXT:    sllv $10, $10, $4
+; MIPS32R6O0-NEXT:    andi $6, $6, 255
+; MIPS32R6O0-NEXT:    sllv $6, $6, $4
+; MIPS32R6O0-NEXT:  $BB13_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $12, 0($7)
+; MIPS32R6O0-NEXT:    and $13, $12, $8
+; MIPS32R6O0-NEXT:    bnec $13, $10, $BB13_3
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; MIPS32R6O0-NEXT:    and $12, $12, $9
+; MIPS32R6O0-NEXT:    or $12, $12, $6
+; MIPS32R6O0-NEXT:    sc $12, 0($7)
+; MIPS32R6O0-NEXT:    beqzc $12, $BB13_1
+; MIPS32R6O0-NEXT:  $BB13_3: # %entry
+; MIPS32R6O0-NEXT:    srlv $11, $13, $4
+; MIPS32R6O0-NEXT:    seb $11, $11
+; MIPS32R6O0-NEXT:  # %bb.4: # %entry
+; MIPS32R6O0-NEXT:    sw $11, 20($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $5, 16($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $3, 12($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $1, 8($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $2, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:  # %bb.5: # %entry
+; MIPS32R6O0-NEXT:    lw $1, 20($sp) # 4-byte Folded Reload
+; MIPS32R6O0-NEXT:    lw $2, 16($sp) # 4-byte Folded Reload
+; MIPS32R6O0-NEXT:    xor $1, $1, $2
+; MIPS32R6O0-NEXT:    sltiu $2, $1, 1
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 24
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicCmpSwapRes8:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    daddiu $1, $zero, -4
+; MIPS4-NEXT:    and $2, $4, $1
+; MIPS4-NEXT:    andi $1, $4, 3
+; MIPS4-NEXT:    sll $3, $1, 3
+; MIPS4-NEXT:    ori $1, $zero, 255
+; MIPS4-NEXT:    sllv $4, $1, $3
+; MIPS4-NEXT:    nor $7, $zero, $4
+; MIPS4-NEXT:    andi $1, $5, 255
+; MIPS4-NEXT:    sllv $8, $1, $3
+; MIPS4-NEXT:    andi $1, $6, 255
+; MIPS4-NEXT:    sllv $6, $1, $3
+; MIPS4-NEXT:  .LBB13_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $9, 0($2)
+; MIPS4-NEXT:    and $10, $9, $4
+; MIPS4-NEXT:    bne $10, $8, .LBB13_3
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; MIPS4-NEXT:    and $9, $9, $7
+; MIPS4-NEXT:    or $9, $9, $6
+; MIPS4-NEXT:    sc $9, 0($2)
+; MIPS4-NEXT:    beqz $9, .LBB13_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  .LBB13_3: # %entry
+; MIPS4-NEXT:    srlv $1, $10, $3
+; MIPS4-NEXT:    sll $1, $1, 24
+; MIPS4-NEXT:    sra $1, $1, 24
+; MIPS4-NEXT:  # %bb.4: # %entry
+; MIPS4-NEXT:    sll $2, $5, 24
+; MIPS4-NEXT:    sra $2, $2, 24
+; MIPS4-NEXT:    xor $1, $1, $2
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    sltiu $2, $1, 1
+;
+; MIPS64-LABEL: AtomicCmpSwapRes8:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    daddiu $1, $zero, -4
+; MIPS64-NEXT:    and $2, $4, $1
+; MIPS64-NEXT:    andi $1, $4, 3
+; MIPS64-NEXT:    sll $3, $1, 3
+; MIPS64-NEXT:    ori $1, $zero, 255
+; MIPS64-NEXT:    sllv $4, $1, $3
+; MIPS64-NEXT:    nor $7, $zero, $4
+; MIPS64-NEXT:    andi $1, $5, 255
+; MIPS64-NEXT:    sllv $8, $1, $3
+; MIPS64-NEXT:    andi $1, $6, 255
+; MIPS64-NEXT:    sllv $6, $1, $3
+; MIPS64-NEXT:  .LBB13_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $9, 0($2)
+; MIPS64-NEXT:    and $10, $9, $4
+; MIPS64-NEXT:    bne $10, $8, .LBB13_3
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; MIPS64-NEXT:    and $9, $9, $7
+; MIPS64-NEXT:    or $9, $9, $6
+; MIPS64-NEXT:    sc $9, 0($2)
+; MIPS64-NEXT:    beqz $9, .LBB13_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  .LBB13_3: # %entry
+; MIPS64-NEXT:    srlv $1, $10, $3
+; MIPS64-NEXT:    sll $1, $1, 24
+; MIPS64-NEXT:    sra $1, $1, 24
+; MIPS64-NEXT:  # %bb.4: # %entry
+; MIPS64-NEXT:    sll $2, $5, 24
+; MIPS64-NEXT:    sra $2, $2, 24
+; MIPS64-NEXT:    xor $1, $1, $2
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    sltiu $2, $1, 1
+;
+; MIPS64R2-LABEL: AtomicCmpSwapRes8:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    daddiu $1, $zero, -4
+; MIPS64R2-NEXT:    and $2, $4, $1
+; MIPS64R2-NEXT:    andi $1, $4, 3
+; MIPS64R2-NEXT:    sll $3, $1, 3
+; MIPS64R2-NEXT:    ori $1, $zero, 255
+; MIPS64R2-NEXT:    sllv $4, $1, $3
+; MIPS64R2-NEXT:    nor $7, $zero, $4
+; MIPS64R2-NEXT:    andi $1, $5, 255
+; MIPS64R2-NEXT:    sllv $8, $1, $3
+; MIPS64R2-NEXT:    andi $1, $6, 255
+; MIPS64R2-NEXT:    sllv $6, $1, $3
+; MIPS64R2-NEXT:  .LBB13_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $9, 0($2)
+; MIPS64R2-NEXT:    and $10, $9, $4
+; MIPS64R2-NEXT:    bne $10, $8, .LBB13_3
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; MIPS64R2-NEXT:    and $9, $9, $7
+; MIPS64R2-NEXT:    or $9, $9, $6
+; MIPS64R2-NEXT:    sc $9, 0($2)
+; MIPS64R2-NEXT:    beqz $9, .LBB13_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  .LBB13_3: # %entry
+; MIPS64R2-NEXT:    srlv $1, $10, $3
+; MIPS64R2-NEXT:    seb $1, $1
+; MIPS64R2-NEXT:  # %bb.4: # %entry
+; MIPS64R2-NEXT:    xor $1, $1, $5
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    sltiu $2, $1, 1
+;
+; MIPS64R6-LABEL: AtomicCmpSwapRes8:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    daddiu $1, $zero, -4
+; MIPS64R6-NEXT:    and $2, $4, $1
+; MIPS64R6-NEXT:    andi $1, $4, 3
+; MIPS64R6-NEXT:    sll $3, $1, 3
+; MIPS64R6-NEXT:    ori $1, $zero, 255
+; MIPS64R6-NEXT:    sllv $4, $1, $3
+; MIPS64R6-NEXT:    nor $7, $zero, $4
+; MIPS64R6-NEXT:    andi $1, $5, 255
+; MIPS64R6-NEXT:    sllv $8, $1, $3
+; MIPS64R6-NEXT:    andi $1, $6, 255
+; MIPS64R6-NEXT:    sllv $6, $1, $3
+; MIPS64R6-NEXT:  .LBB13_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $9, 0($2)
+; MIPS64R6-NEXT:    and $10, $9, $4
+; MIPS64R6-NEXT:    bnec $10, $8, .LBB13_3
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; MIPS64R6-NEXT:    and $9, $9, $7
+; MIPS64R6-NEXT:    or $9, $9, $6
+; MIPS64R6-NEXT:    sc $9, 0($2)
+; MIPS64R6-NEXT:    beqzc $9, .LBB13_1
+; MIPS64R6-NEXT:  .LBB13_3: # %entry
+; MIPS64R6-NEXT:    srlv $1, $10, $3
+; MIPS64R6-NEXT:    seb $1, $1
+; MIPS64R6-NEXT:  # %bb.4: # %entry
+; MIPS64R6-NEXT:    xor $1, $1, $5
+; MIPS64R6-NEXT:    jr $ra
+; MIPS64R6-NEXT:    sltiu $2, $1, 1
+;
+; MIPS64R6O0-LABEL: AtomicCmpSwapRes8:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    move $1, $6
+; MIPS64R6O0-NEXT:    move $2, $5
+; MIPS64R6O0-NEXT:    move $5, $4
+; MIPS64R6O0-NEXT:    daddiu $6, $zero, -4
+; MIPS64R6O0-NEXT:    and $6, $4, $6
+; MIPS64R6O0-NEXT:    andi $3, $4, 3
+; MIPS64R6O0-NEXT:    xori $3, $3, 3
+; MIPS64R6O0-NEXT:    sll $3, $3, 3
+; MIPS64R6O0-NEXT:    ori $7, $zero, 255
+; MIPS64R6O0-NEXT:    sllv $7, $7, $3
+; MIPS64R6O0-NEXT:    nor $8, $zero, $7
+; MIPS64R6O0-NEXT:    andi $9, $2, 255
+; MIPS64R6O0-NEXT:    sllv $9, $9, $3
+; MIPS64R6O0-NEXT:    andi $1, $1, 255
+; MIPS64R6O0-NEXT:    sllv $1, $1, $3
+; MIPS64R6O0-NEXT:  .LBB13_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $11, 0($6)
+; MIPS64R6O0-NEXT:    and $12, $11, $7
+; MIPS64R6O0-NEXT:    bnec $12, $9, .LBB13_3
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; MIPS64R6O0-NEXT:    and $11, $11, $8
+; MIPS64R6O0-NEXT:    or $11, $11, $1
+; MIPS64R6O0-NEXT:    sc $11, 0($6)
+; MIPS64R6O0-NEXT:    beqzc $11, .LBB13_1
+; MIPS64R6O0-NEXT:  .LBB13_3: # %entry
+; MIPS64R6O0-NEXT:    srlv $10, $12, $3
+; MIPS64R6O0-NEXT:    seb $10, $10
+; MIPS64R6O0-NEXT:  # %bb.4: # %entry
+; MIPS64R6O0-NEXT:    sd $5, 8($sp) # 8-byte Folded Spill
+; MIPS64R6O0-NEXT:    sw $10, 4($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:    sw $2, 0($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:  # %bb.5: # %entry
+; MIPS64R6O0-NEXT:    lw $1, 4($sp) # 4-byte Folded Reload
+; MIPS64R6O0-NEXT:    lw $2, 0($sp) # 4-byte Folded Reload
+; MIPS64R6O0-NEXT:    xor $1, $1, $2
+; MIPS64R6O0-NEXT:    sltiu $2, $1, 1
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicCmpSwapRes8:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    addiu $1, $zero, -4
+; MM32-NEXT:    and $2, $4, $1
+; MM32-NEXT:    andi $1, $4, 3
+; MM32-NEXT:    sll $3, $1, 3
+; MM32-NEXT:    ori $1, $zero, 255
+; MM32-NEXT:    sllv $4, $1, $3
+; MM32-NEXT:    nor $7, $zero, $4
+; MM32-NEXT:    andi $1, $5, 255
+; MM32-NEXT:    sllv $8, $1, $3
+; MM32-NEXT:    andi $1, $6, 255
+; MM32-NEXT:    sllv $6, $1, $3
+; MM32-NEXT:  $BB13_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $9, 0($2)
+; MM32-NEXT:    and $10, $9, $4
+; MM32-NEXT:    bne $10, $8, $BB13_3
+; MM32-NEXT:    nop
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; MM32-NEXT:    and $9, $9, $7
+; MM32-NEXT:    or $9, $9, $6
+; MM32-NEXT:    sc $9, 0($2)
+; MM32-NEXT:    beqzc $9, $BB13_1
+; MM32-NEXT:  $BB13_3: # %entry
+; MM32-NEXT:    srlv $1, $10, $3
+; MM32-NEXT:    seb $1, $1
+; MM32-NEXT:  # %bb.4: # %entry
+; MM32-NEXT:    xor $1, $1, $5
+; MM32-NEXT:    jr $ra
+; MM32-NEXT:    sltiu $2, $1, 1
+;
+; O1-LABEL: AtomicCmpSwapRes8:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    addiu $1, $zero, -4
+; O1-NEXT:    and $2, $4, $1
+; O1-NEXT:    andi $1, $4, 3
+; O1-NEXT:    sll $3, $1, 3
+; O1-NEXT:    ori $1, $zero, 255
+; O1-NEXT:    sllv $4, $1, $3
+; O1-NEXT:    nor $7, $zero, $4
+; O1-NEXT:    andi $1, $5, 255
+; O1-NEXT:    sllv $8, $1, $3
+; O1-NEXT:    andi $1, $6, 255
+; O1-NEXT:    sllv $6, $1, $3
+; O1-NEXT:  $BB13_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $9, 0($2)
+; O1-NEXT:    and $10, $9, $4
+; O1-NEXT:    bne $10, $8, $BB13_3
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; O1-NEXT:    and $9, $9, $7
+; O1-NEXT:    or $9, $9, $6
+; O1-NEXT:    sc $9, 0($2)
+; O1-NEXT:    beqz $9, $BB13_1
+; O1-NEXT:    nop
+; O1-NEXT:  $BB13_3: # %entry
+; O1-NEXT:    srlv $1, $10, $3
+; O1-NEXT:    sll $1, $1, 24
+; O1-NEXT:    sra $1, $1, 24
+; O1-NEXT:  # %bb.4: # %entry
+; O1-NEXT:    sll $2, $5, 24
+; O1-NEXT:    sra $2, $2, 24
+; O1-NEXT:    xor $1, $1, $2
+; O1-NEXT:    jr $ra
+; O1-NEXT:    sltiu $2, $1, 1
+;
+; O2-LABEL: AtomicCmpSwapRes8:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    addiu $1, $zero, -4
+; O2-NEXT:    and $2, $4, $1
+; O2-NEXT:    andi $1, $4, 3
+; O2-NEXT:    sll $3, $1, 3
+; O2-NEXT:    ori $1, $zero, 255
+; O2-NEXT:    sllv $4, $1, $3
+; O2-NEXT:    nor $7, $zero, $4
+; O2-NEXT:    andi $1, $5, 255
+; O2-NEXT:    sllv $8, $1, $3
+; O2-NEXT:    andi $1, $6, 255
+; O2-NEXT:    sllv $6, $1, $3
+; O2-NEXT:  $BB13_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $9, 0($2)
+; O2-NEXT:    and $10, $9, $4
+; O2-NEXT:    bne $10, $8, $BB13_3
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; O2-NEXT:    and $9, $9, $7
+; O2-NEXT:    or $9, $9, $6
+; O2-NEXT:    sc $9, 0($2)
+; O2-NEXT:    beqz $9, $BB13_1
+; O2-NEXT:    nop
+; O2-NEXT:  $BB13_3: # %entry
+; O2-NEXT:    srlv $1, $10, $3
+; O2-NEXT:    sll $1, $1, 24
+; O2-NEXT:    sra $1, $1, 24
+; O2-NEXT:  # %bb.4: # %entry
+; O2-NEXT:    sll $2, $5, 24
+; O2-NEXT:    sra $2, $2, 24
+; O2-NEXT:    xor $1, $1, $2
+; O2-NEXT:    jr $ra
+; O2-NEXT:    sltiu $2, $1, 1
+;
+; O3-LABEL: AtomicCmpSwapRes8:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    addiu $1, $zero, -4
+; O3-NEXT:    and $2, $4, $1
+; O3-NEXT:    andi $1, $4, 3
+; O3-NEXT:    sll $3, $1, 3
+; O3-NEXT:    ori $1, $zero, 255
+; O3-NEXT:    sllv $4, $1, $3
+; O3-NEXT:    andi $1, $5, 255
+; O3-NEXT:    sllv $8, $1, $3
+; O3-NEXT:    andi $1, $6, 255
+; O3-NEXT:    nor $7, $zero, $4
+; O3-NEXT:    sllv $6, $1, $3
+; O3-NEXT:  $BB13_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $9, 0($2)
+; O3-NEXT:    and $10, $9, $4
+; O3-NEXT:    bne $10, $8, $BB13_3
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; O3-NEXT:    and $9, $9, $7
+; O3-NEXT:    or $9, $9, $6
+; O3-NEXT:    sc $9, 0($2)
+; O3-NEXT:    beqz $9, $BB13_1
+; O3-NEXT:    nop
+; O3-NEXT:  $BB13_3: # %entry
+; O3-NEXT:    srlv $1, $10, $3
+; O3-NEXT:    sll $1, $1, 24
+; O3-NEXT:    sra $1, $1, 24
+; O3-NEXT:  # %bb.4: # %entry
+; O3-NEXT:    sll $2, $5, 24
+; O3-NEXT:    sra $2, $2, 24
+; O3-NEXT:    xor $1, $1, $2
+; O3-NEXT:    jr $ra
+; O3-NEXT:    sltiu $2, $1, 1
+;
+; MIPS32EB-LABEL: AtomicCmpSwapRes8:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    addiu $1, $zero, -4
+; MIPS32EB-NEXT:    and $2, $4, $1
+; MIPS32EB-NEXT:    andi $1, $4, 3
+; MIPS32EB-NEXT:    xori $1, $1, 3
+; MIPS32EB-NEXT:    sll $3, $1, 3
+; MIPS32EB-NEXT:    ori $1, $zero, 255
+; MIPS32EB-NEXT:    sllv $4, $1, $3
+; MIPS32EB-NEXT:    nor $7, $zero, $4
+; MIPS32EB-NEXT:    andi $1, $5, 255
+; MIPS32EB-NEXT:    sllv $8, $1, $3
+; MIPS32EB-NEXT:    andi $1, $6, 255
+; MIPS32EB-NEXT:    sllv $6, $1, $3
+; MIPS32EB-NEXT:  $BB13_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $9, 0($2)
+; MIPS32EB-NEXT:    and $10, $9, $4
+; MIPS32EB-NEXT:    bne $10, $8, $BB13_3
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; MIPS32EB-NEXT:    and $9, $9, $7
+; MIPS32EB-NEXT:    or $9, $9, $6
+; MIPS32EB-NEXT:    sc $9, 0($2)
+; MIPS32EB-NEXT:    beqz $9, $BB13_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  $BB13_3: # %entry
+; MIPS32EB-NEXT:    srlv $1, $10, $3
+; MIPS32EB-NEXT:    sll $1, $1, 24
+; MIPS32EB-NEXT:    sra $1, $1, 24
+; MIPS32EB-NEXT:  # %bb.4: # %entry
+; MIPS32EB-NEXT:    sll $2, $5, 24
+; MIPS32EB-NEXT:    sra $2, $2, 24
+; MIPS32EB-NEXT:    xor $1, $1, $2
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    sltiu $2, $1, 1
 entry:
   %0 = cmpxchg i8* %ptr, i8 %oldval, i8 %newval monotonic monotonic
   %1 = extractvalue { i8, i1 } %0, 1
   ret i1 %1
-; ALL-LABEL: AtomicCmpSwapRes8
-
-; ALL:           addiu   $[[R1:[0-9]+]], $zero, -4
-; ALL:           and     $[[R2:[0-9]+]], $4, $[[R1]]
-; ALL:           andi    $[[R3:[0-9]+]], $4, 3
-; CHECK-EL:      sll     $[[R5:[0-9]+]], $[[R3]], 3
-; CHECK-EB:      xori    $[[R4:[0-9]+]], $[[R3]], 3
-; CHECK-EB:      sll     $[[R5:[0-9]+]], $[[R4]], 3
-; ALL:           ori     $[[R6:[0-9]+]], $zero, 255
-; ALL:           sllv    $[[R7:[0-9]+]], $[[R6]], $[[R5]]
-; ALL:           nor     $[[R8:[0-9]+]], $zero, $[[R7]]
-; ALL:           andi    $[[R9:[0-9]+]], $5, 255
-; ALL:           sllv    $[[R10:[0-9]+]], $[[R9]], $[[R5]]
-; ALL:           andi    $[[R11:[0-9]+]], $6, 255
-; ALL:           sllv    $[[R12:[0-9]+]], $[[R11]], $[[R5]]
-
-; ALL:       [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; ALL:           ll      $[[R13:[0-9]+]], 0($[[R2]])
-; ALL:           and     $[[R14:[0-9]+]], $[[R13]], $[[R7]]
-; NOT-MICROMIPS: bne     $[[R14]], $[[R10]], [[BB1:(\$|\.L)[A-Z_0-9]+]]
-; MICROMIPS:     bne     $[[R14]], $[[R10]], [[BB1:(\$|\.L)[A-Z_0-9]+]]
-; MIPSR6:        bnec    $[[R14]], $[[R10]], [[BB1:(\$|\.L)[A-Z_0-9]+]]
-
-; ALL:           and     $[[R15:[0-9]+]], $[[R13]], $[[R8]]
-; ALL:           or      $[[R16:[0-9]+]], $[[R15]], $[[R12]]
-; ALL:           sc      $[[R16]], 0($[[R2]])
-; NOT-MICROMIPS: beqz    $[[R16]], [[BB0]]
-; MICROMIPS:     beqzc   $[[R16]], [[BB0]]
-; MIPSR6:        beqzc   $[[R16]], [[BB0]]
-
-; ALL:       [[BB1]]:
-; ALL:           srlv    $[[R17:[0-9]+]], $[[R14]], $[[R5]]
-
-; NO-SEB-SEH:    sll     $[[R18:[0-9]+]], $[[R17]], 24
-; NO-SEB-SEH:    sra     $[[R19:[0-9]+]], $[[R18]], 24
-
 ; FIXME: -march=mips produces a redundant sign extension here...
-; NO-SEB-SEH:    sll     $[[R20:[0-9]+]], $5, 24
-; NO-SEB-SEH:    sra     $[[R20]], $[[R20]], 24
-
-; HAS-SEB-SEH:   seb     $[[R19:[0-9]+]], $[[R17]]
-
 ; FIXME: ...Leading to this split check.
-; NO-SEB-SEH:    xor     $[[R21:[0-9]+]], $[[R19]], $[[R20]]
-; HAS-SEB-SEH:   xor     $[[R21:[0-9]+]], $[[R19]], $5
 
-; ALL: sltiu   $2, $[[R21]], 1
 }
 
 ; Check one i16 so that we cover the seh sign extend
 @z = common global i16 0, align 1
 
 define signext i16 @AtomicLoadAdd16(i16 signext %incr) nounwind {
+; MIPS32-LABEL: AtomicLoadAdd16:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    lw $1, %got(z)($1)
+; MIPS32-NEXT:    addiu $2, $zero, -4
+; MIPS32-NEXT:    and $2, $1, $2
+; MIPS32-NEXT:    andi $1, $1, 3
+; MIPS32-NEXT:    sll $3, $1, 3
+; MIPS32-NEXT:    ori $1, $zero, 65535
+; MIPS32-NEXT:    sllv $5, $1, $3
+; MIPS32-NEXT:    nor $6, $zero, $5
+; MIPS32-NEXT:    sllv $4, $4, $3
+; MIPS32-NEXT:  $BB14_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $7, 0($2)
+; MIPS32-NEXT:    addu $8, $7, $4
+; MIPS32-NEXT:    and $8, $8, $5
+; MIPS32-NEXT:    and $9, $7, $6
+; MIPS32-NEXT:    or $9, $9, $8
+; MIPS32-NEXT:    sc $9, 0($2)
+; MIPS32-NEXT:    beqz $9, $BB14_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    and $1, $7, $5
+; MIPS32-NEXT:    srlv $1, $1, $3
+; MIPS32-NEXT:    sll $1, $1, 16
+; MIPS32-NEXT:    sra $1, $1, 16
+; MIPS32-NEXT:  # %bb.3: # %entry
+; MIPS32-NEXT:    sll $1, $1, 16
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    sra $2, $1, 16
+;
+; MIPS32O0-LABEL: AtomicLoadAdd16:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    lw $2, %got(z)($2)
+; MIPS32O0-NEXT:    addiu $25, $zero, -4
+; MIPS32O0-NEXT:    and $25, $2, $25
+; MIPS32O0-NEXT:    andi $2, $2, 3
+; MIPS32O0-NEXT:    sll $2, $2, 3
+; MIPS32O0-NEXT:    ori $1, $zero, 65535
+; MIPS32O0-NEXT:    sllv $1, $1, $2
+; MIPS32O0-NEXT:    nor $3, $zero, $1
+; MIPS32O0-NEXT:    sllv $4, $4, $2
+; MIPS32O0-NEXT:  $BB14_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $6, 0($25)
+; MIPS32O0-NEXT:    addu $7, $6, $4
+; MIPS32O0-NEXT:    and $7, $7, $1
+; MIPS32O0-NEXT:    and $8, $6, $3
+; MIPS32O0-NEXT:    or $8, $8, $7
+; MIPS32O0-NEXT:    sc $8, 0($25)
+; MIPS32O0-NEXT:    beqz $8, $BB14_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    and $5, $6, $1
+; MIPS32O0-NEXT:    srlv $5, $5, $2
+; MIPS32O0-NEXT:    sll $5, $5, 16
+; MIPS32O0-NEXT:    sra $5, $5, 16
+; MIPS32O0-NEXT:  # %bb.3: # %entry
+; MIPS32O0-NEXT:    sw $5, 4($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:  # %bb.4: # %entry
+; MIPS32O0-NEXT:    lw $1, 4($sp) # 4-byte Folded Reload
+; MIPS32O0-NEXT:    sll $2, $1, 16
+; MIPS32O0-NEXT:    sra $2, $2, 16
+; MIPS32O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicLoadAdd16:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    lw $1, %got(z)($1)
+; MIPS32R2-NEXT:    addiu $2, $zero, -4
+; MIPS32R2-NEXT:    and $2, $1, $2
+; MIPS32R2-NEXT:    andi $1, $1, 3
+; MIPS32R2-NEXT:    sll $3, $1, 3
+; MIPS32R2-NEXT:    ori $1, $zero, 65535
+; MIPS32R2-NEXT:    sllv $5, $1, $3
+; MIPS32R2-NEXT:    nor $6, $zero, $5
+; MIPS32R2-NEXT:    sllv $4, $4, $3
+; MIPS32R2-NEXT:  $BB14_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $7, 0($2)
+; MIPS32R2-NEXT:    addu $8, $7, $4
+; MIPS32R2-NEXT:    and $8, $8, $5
+; MIPS32R2-NEXT:    and $9, $7, $6
+; MIPS32R2-NEXT:    or $9, $9, $8
+; MIPS32R2-NEXT:    sc $9, 0($2)
+; MIPS32R2-NEXT:    beqz $9, $BB14_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    and $1, $7, $5
+; MIPS32R2-NEXT:    srlv $1, $1, $3
+; MIPS32R2-NEXT:    seh $1, $1
+; MIPS32R2-NEXT:  # %bb.3: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    seh $2, $1
+;
+; MIPS32R6-LABEL: AtomicLoadAdd16:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    lw $1, %got(z)($1)
+; MIPS32R6-NEXT:    addiu $2, $zero, -4
+; MIPS32R6-NEXT:    and $2, $1, $2
+; MIPS32R6-NEXT:    andi $1, $1, 3
+; MIPS32R6-NEXT:    sll $3, $1, 3
+; MIPS32R6-NEXT:    ori $1, $zero, 65535
+; MIPS32R6-NEXT:    sllv $5, $1, $3
+; MIPS32R6-NEXT:    nor $6, $zero, $5
+; MIPS32R6-NEXT:    sllv $4, $4, $3
+; MIPS32R6-NEXT:  $BB14_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $7, 0($2)
+; MIPS32R6-NEXT:    addu $8, $7, $4
+; MIPS32R6-NEXT:    and $8, $8, $5
+; MIPS32R6-NEXT:    and $9, $7, $6
+; MIPS32R6-NEXT:    or $9, $9, $8
+; MIPS32R6-NEXT:    sc $9, 0($2)
+; MIPS32R6-NEXT:    beqzc $9, $BB14_1
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    and $1, $7, $5
+; MIPS32R6-NEXT:    srlv $1, $1, $3
+; MIPS32R6-NEXT:    seh $1, $1
+; MIPS32R6-NEXT:  # %bb.3: # %entry
+; MIPS32R6-NEXT:    jr $ra
+; MIPS32R6-NEXT:    seh $2, $1
+;
+; MIPS32R6O0-LABEL: AtomicLoadAdd16:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    lw $2, %got(z)($2)
+; MIPS32R6O0-NEXT:    addiu $1, $zero, -4
+; MIPS32R6O0-NEXT:    and $1, $2, $1
+; MIPS32R6O0-NEXT:    andi $2, $2, 3
+; MIPS32R6O0-NEXT:    sll $2, $2, 3
+; MIPS32R6O0-NEXT:    ori $3, $zero, 65535
+; MIPS32R6O0-NEXT:    sllv $3, $3, $2
+; MIPS32R6O0-NEXT:    nor $5, $zero, $3
+; MIPS32R6O0-NEXT:    sllv $4, $4, $2
+; MIPS32R6O0-NEXT:  $BB14_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $7, 0($1)
+; MIPS32R6O0-NEXT:    addu $8, $7, $4
+; MIPS32R6O0-NEXT:    and $8, $8, $3
+; MIPS32R6O0-NEXT:    and $9, $7, $5
+; MIPS32R6O0-NEXT:    or $9, $9, $8
+; MIPS32R6O0-NEXT:    sc $9, 0($1)
+; MIPS32R6O0-NEXT:    beqzc $9, $BB14_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    and $6, $7, $3
+; MIPS32R6O0-NEXT:    srlv $6, $6, $2
+; MIPS32R6O0-NEXT:    seh $6, $6
+; MIPS32R6O0-NEXT:  # %bb.3: # %entry
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $6, 0($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:  # %bb.4: # %entry
+; MIPS32R6O0-NEXT:    lw $1, 0($sp) # 4-byte Folded Reload
+; MIPS32R6O0-NEXT:    seh $2, $1
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicLoadAdd16:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd16)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd16)))
+; MIPS4-NEXT:    ld $1, %got_disp(z)($1)
+; MIPS4-NEXT:    daddiu $2, $zero, -4
+; MIPS4-NEXT:    and $2, $1, $2
+; MIPS4-NEXT:    andi $1, $1, 3
+; MIPS4-NEXT:    sll $3, $1, 3
+; MIPS4-NEXT:    ori $1, $zero, 65535
+; MIPS4-NEXT:    sllv $5, $1, $3
+; MIPS4-NEXT:    nor $6, $zero, $5
+; MIPS4-NEXT:    sllv $4, $4, $3
+; MIPS4-NEXT:  .LBB14_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $7, 0($2)
+; MIPS4-NEXT:    addu $8, $7, $4
+; MIPS4-NEXT:    and $8, $8, $5
+; MIPS4-NEXT:    and $9, $7, $6
+; MIPS4-NEXT:    or $9, $9, $8
+; MIPS4-NEXT:    sc $9, 0($2)
+; MIPS4-NEXT:    beqz $9, .LBB14_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    and $1, $7, $5
+; MIPS4-NEXT:    srlv $1, $1, $3
+; MIPS4-NEXT:    sll $1, $1, 16
+; MIPS4-NEXT:    sra $1, $1, 16
+; MIPS4-NEXT:  # %bb.3: # %entry
+; MIPS4-NEXT:    sll $1, $1, 16
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    sra $2, $1, 16
+;
+; MIPS64-LABEL: AtomicLoadAdd16:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd16)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd16)))
+; MIPS64-NEXT:    ld $1, %got_disp(z)($1)
+; MIPS64-NEXT:    daddiu $2, $zero, -4
+; MIPS64-NEXT:    and $2, $1, $2
+; MIPS64-NEXT:    andi $1, $1, 3
+; MIPS64-NEXT:    sll $3, $1, 3
+; MIPS64-NEXT:    ori $1, $zero, 65535
+; MIPS64-NEXT:    sllv $5, $1, $3
+; MIPS64-NEXT:    nor $6, $zero, $5
+; MIPS64-NEXT:    sllv $4, $4, $3
+; MIPS64-NEXT:  .LBB14_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $7, 0($2)
+; MIPS64-NEXT:    addu $8, $7, $4
+; MIPS64-NEXT:    and $8, $8, $5
+; MIPS64-NEXT:    and $9, $7, $6
+; MIPS64-NEXT:    or $9, $9, $8
+; MIPS64-NEXT:    sc $9, 0($2)
+; MIPS64-NEXT:    beqz $9, .LBB14_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    and $1, $7, $5
+; MIPS64-NEXT:    srlv $1, $1, $3
+; MIPS64-NEXT:    sll $1, $1, 16
+; MIPS64-NEXT:    sra $1, $1, 16
+; MIPS64-NEXT:  # %bb.3: # %entry
+; MIPS64-NEXT:    sll $1, $1, 16
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    sra $2, $1, 16
+;
+; MIPS64R2-LABEL: AtomicLoadAdd16:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd16)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd16)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(z)($1)
+; MIPS64R2-NEXT:    daddiu $2, $zero, -4
+; MIPS64R2-NEXT:    and $2, $1, $2
+; MIPS64R2-NEXT:    andi $1, $1, 3
+; MIPS64R2-NEXT:    sll $3, $1, 3
+; MIPS64R2-NEXT:    ori $1, $zero, 65535
+; MIPS64R2-NEXT:    sllv $5, $1, $3
+; MIPS64R2-NEXT:    nor $6, $zero, $5
+; MIPS64R2-NEXT:    sllv $4, $4, $3
+; MIPS64R2-NEXT:  .LBB14_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $7, 0($2)
+; MIPS64R2-NEXT:    addu $8, $7, $4
+; MIPS64R2-NEXT:    and $8, $8, $5
+; MIPS64R2-NEXT:    and $9, $7, $6
+; MIPS64R2-NEXT:    or $9, $9, $8
+; MIPS64R2-NEXT:    sc $9, 0($2)
+; MIPS64R2-NEXT:    beqz $9, .LBB14_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    and $1, $7, $5
+; MIPS64R2-NEXT:    srlv $1, $1, $3
+; MIPS64R2-NEXT:    seh $1, $1
+; MIPS64R2-NEXT:  # %bb.3: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    seh $2, $1
+;
+; MIPS64R6-LABEL: AtomicLoadAdd16:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd16)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd16)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(z)($1)
+; MIPS64R6-NEXT:    daddiu $2, $zero, -4
+; MIPS64R6-NEXT:    and $2, $1, $2
+; MIPS64R6-NEXT:    andi $1, $1, 3
+; MIPS64R6-NEXT:    sll $3, $1, 3
+; MIPS64R6-NEXT:    ori $1, $zero, 65535
+; MIPS64R6-NEXT:    sllv $5, $1, $3
+; MIPS64R6-NEXT:    nor $6, $zero, $5
+; MIPS64R6-NEXT:    sllv $4, $4, $3
+; MIPS64R6-NEXT:  .LBB14_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $7, 0($2)
+; MIPS64R6-NEXT:    addu $8, $7, $4
+; MIPS64R6-NEXT:    and $8, $8, $5
+; MIPS64R6-NEXT:    and $9, $7, $6
+; MIPS64R6-NEXT:    or $9, $9, $8
+; MIPS64R6-NEXT:    sc $9, 0($2)
+; MIPS64R6-NEXT:    beqzc $9, .LBB14_1
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    and $1, $7, $5
+; MIPS64R6-NEXT:    srlv $1, $1, $3
+; MIPS64R6-NEXT:    seh $1, $1
+; MIPS64R6-NEXT:  # %bb.3: # %entry
+; MIPS64R6-NEXT:    jr $ra
+; MIPS64R6-NEXT:    seh $2, $1
+;
+; MIPS64R6O0-LABEL: AtomicLoadAdd16:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd16)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd16)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(z)($1)
+; MIPS64R6O0-NEXT:    daddiu $4, $zero, -4
+; MIPS64R6O0-NEXT:    and $4, $1, $4
+; MIPS64R6O0-NEXT:    andi $3, $1, 3
+; MIPS64R6O0-NEXT:    xori $3, $3, 2
+; MIPS64R6O0-NEXT:    sll $3, $3, 3
+; MIPS64R6O0-NEXT:    ori $5, $zero, 65535
+; MIPS64R6O0-NEXT:    sllv $5, $5, $3
+; MIPS64R6O0-NEXT:    nor $6, $zero, $5
+; MIPS64R6O0-NEXT:    sllv $2, $2, $3
+; MIPS64R6O0-NEXT:  .LBB14_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $8, 0($4)
+; MIPS64R6O0-NEXT:    addu $9, $8, $2
+; MIPS64R6O0-NEXT:    and $9, $9, $5
+; MIPS64R6O0-NEXT:    and $10, $8, $6
+; MIPS64R6O0-NEXT:    or $10, $10, $9
+; MIPS64R6O0-NEXT:    sc $10, 0($4)
+; MIPS64R6O0-NEXT:    beqzc $10, .LBB14_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    and $7, $8, $5
+; MIPS64R6O0-NEXT:    srlv $7, $7, $3
+; MIPS64R6O0-NEXT:    seh $7, $7
+; MIPS64R6O0-NEXT:  # %bb.3: # %entry
+; MIPS64R6O0-NEXT:    sw $7, 12($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:  # %bb.4: # %entry
+; MIPS64R6O0-NEXT:    lw $1, 12($sp) # 4-byte Folded Reload
+; MIPS64R6O0-NEXT:    seh $2, $1
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicLoadAdd16:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    lw $1, %got(z)($2)
+; MM32-NEXT:    addiu $2, $zero, -4
+; MM32-NEXT:    and $2, $1, $2
+; MM32-NEXT:    andi $1, $1, 3
+; MM32-NEXT:    sll $3, $1, 3
+; MM32-NEXT:    ori $1, $zero, 65535
+; MM32-NEXT:    sllv $5, $1, $3
+; MM32-NEXT:    nor $6, $zero, $5
+; MM32-NEXT:    sllv $4, $4, $3
+; MM32-NEXT:  $BB14_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $7, 0($2)
+; MM32-NEXT:    addu $8, $7, $4
+; MM32-NEXT:    and $8, $8, $5
+; MM32-NEXT:    and $9, $7, $6
+; MM32-NEXT:    or $9, $9, $8
+; MM32-NEXT:    sc $9, 0($2)
+; MM32-NEXT:    beqzc $9, $BB14_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    and $1, $7, $5
+; MM32-NEXT:    srlv $1, $1, $3
+; MM32-NEXT:    seh $1, $1
+; MM32-NEXT:  # %bb.3: # %entry
+; MM32-NEXT:    jr $ra
+; MM32-NEXT:    seh $2, $1
+;
+; O1-LABEL: AtomicLoadAdd16:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    lw $1, %got(z)($1)
+; O1-NEXT:    addiu $2, $zero, -4
+; O1-NEXT:    and $2, $1, $2
+; O1-NEXT:    andi $1, $1, 3
+; O1-NEXT:    sll $3, $1, 3
+; O1-NEXT:    ori $1, $zero, 65535
+; O1-NEXT:    sllv $5, $1, $3
+; O1-NEXT:    nor $6, $zero, $5
+; O1-NEXT:    sllv $4, $4, $3
+; O1-NEXT:  $BB14_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $7, 0($2)
+; O1-NEXT:    addu $8, $7, $4
+; O1-NEXT:    and $8, $8, $5
+; O1-NEXT:    and $9, $7, $6
+; O1-NEXT:    or $9, $9, $8
+; O1-NEXT:    sc $9, 0($2)
+; O1-NEXT:    beqz $9, $BB14_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    and $1, $7, $5
+; O1-NEXT:    srlv $1, $1, $3
+; O1-NEXT:    sll $1, $1, 16
+; O1-NEXT:    sra $1, $1, 16
+; O1-NEXT:  # %bb.3: # %entry
+; O1-NEXT:    sll $1, $1, 16
+; O1-NEXT:    jr $ra
+; O1-NEXT:    sra $2, $1, 16
+;
+; O2-LABEL: AtomicLoadAdd16:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    lw $1, %got(z)($1)
+; O2-NEXT:    addiu $2, $zero, -4
+; O2-NEXT:    and $2, $1, $2
+; O2-NEXT:    andi $1, $1, 3
+; O2-NEXT:    sll $3, $1, 3
+; O2-NEXT:    ori $1, $zero, 65535
+; O2-NEXT:    sllv $5, $1, $3
+; O2-NEXT:    nor $6, $zero, $5
+; O2-NEXT:    sllv $4, $4, $3
+; O2-NEXT:  $BB14_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $7, 0($2)
+; O2-NEXT:    addu $8, $7, $4
+; O2-NEXT:    and $8, $8, $5
+; O2-NEXT:    and $9, $7, $6
+; O2-NEXT:    or $9, $9, $8
+; O2-NEXT:    sc $9, 0($2)
+; O2-NEXT:    beqz $9, $BB14_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    and $1, $7, $5
+; O2-NEXT:    srlv $1, $1, $3
+; O2-NEXT:    sll $1, $1, 16
+; O2-NEXT:    sra $1, $1, 16
+; O2-NEXT:  # %bb.3: # %entry
+; O2-NEXT:    sll $1, $1, 16
+; O2-NEXT:    jr $ra
+; O2-NEXT:    sra $2, $1, 16
+;
+; O3-LABEL: AtomicLoadAdd16:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    addiu $2, $zero, -4
+; O3-NEXT:    lw $1, %got(z)($1)
+; O3-NEXT:    and $2, $1, $2
+; O3-NEXT:    andi $1, $1, 3
+; O3-NEXT:    sll $3, $1, 3
+; O3-NEXT:    ori $1, $zero, 65535
+; O3-NEXT:    sllv $5, $1, $3
+; O3-NEXT:    sllv $4, $4, $3
+; O3-NEXT:    nor $6, $zero, $5
+; O3-NEXT:  $BB14_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $7, 0($2)
+; O3-NEXT:    addu $8, $7, $4
+; O3-NEXT:    and $8, $8, $5
+; O3-NEXT:    and $9, $7, $6
+; O3-NEXT:    or $9, $9, $8
+; O3-NEXT:    sc $9, 0($2)
+; O3-NEXT:    beqz $9, $BB14_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    and $1, $7, $5
+; O3-NEXT:    srlv $1, $1, $3
+; O3-NEXT:    sll $1, $1, 16
+; O3-NEXT:    sra $1, $1, 16
+; O3-NEXT:  # %bb.3: # %entry
+; O3-NEXT:    sll $1, $1, 16
+; O3-NEXT:    jr $ra
+; O3-NEXT:    sra $2, $1, 16
+;
+; MIPS32EB-LABEL: AtomicLoadAdd16:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    lw $1, %got(z)($1)
+; MIPS32EB-NEXT:    addiu $2, $zero, -4
+; MIPS32EB-NEXT:    and $2, $1, $2
+; MIPS32EB-NEXT:    andi $1, $1, 3
+; MIPS32EB-NEXT:    xori $1, $1, 2
+; MIPS32EB-NEXT:    sll $3, $1, 3
+; MIPS32EB-NEXT:    ori $1, $zero, 65535
+; MIPS32EB-NEXT:    sllv $5, $1, $3
+; MIPS32EB-NEXT:    nor $6, $zero, $5
+; MIPS32EB-NEXT:    sllv $4, $4, $3
+; MIPS32EB-NEXT:  $BB14_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $7, 0($2)
+; MIPS32EB-NEXT:    addu $8, $7, $4
+; MIPS32EB-NEXT:    and $8, $8, $5
+; MIPS32EB-NEXT:    and $9, $7, $6
+; MIPS32EB-NEXT:    or $9, $9, $8
+; MIPS32EB-NEXT:    sc $9, 0($2)
+; MIPS32EB-NEXT:    beqz $9, $BB14_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    and $1, $7, $5
+; MIPS32EB-NEXT:    srlv $1, $1, $3
+; MIPS32EB-NEXT:    sll $1, $1, 16
+; MIPS32EB-NEXT:    sra $1, $1, 16
+; MIPS32EB-NEXT:  # %bb.3: # %entry
+; MIPS32EB-NEXT:    sll $1, $1, 16
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    sra $2, $1, 16
 entry:
   %0 = atomicrmw add i16* @z, i16 %incr monotonic
   ret i16 %0
 
-; ALL-LABEL: AtomicLoadAdd16:
-
-; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(z)
-; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(z)(
-
-; ALL:           addiu   $[[R1:[0-9]+]], $zero, -4
-; ALL:           and     $[[R2:[0-9]+]], $[[R0]], $[[R1]]
-; ALL:           andi    $[[R3:[0-9]+]], $[[R0]], 3
-; CHECK-EB:      xori    $[[R4:[0-9]+]], $[[R3]], 2
-; CHECK-EB:      sll     $[[R5:[0-9]+]], $[[R4]], 3
-; CHECK-EL:      sll     $[[R5:[0-9]+]], $[[R3]], 3
-; ALL:           ori     $[[R6:[0-9]+]], $zero, 65535
-; ALL:           sllv    $[[R7:[0-9]+]], $[[R6]], $[[R5]]
-; ALL:           nor     $[[R8:[0-9]+]], $zero, $[[R7]]
-; ALL:           sllv    $[[R9:[0-9]+]], $4, $[[R5]]
-
-; O0:        [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; O0:            ld      $[[R10:[0-9]+]]
-; O0-NEXT:       ll      $[[R11:[0-9]+]], 0($[[R10]])
-
-; ALL:       [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; ALL:           ll      $[[R12:[0-9]+]], 0($[[R2]])
-; ALL:           addu    $[[R13:[0-9]+]], $[[R12]], $[[R9]]
-; ALL:           and     $[[R14:[0-9]+]], $[[R13]], $[[R7]]
-; ALL:           and     $[[R15:[0-9]+]], $[[R12]], $[[R8]]
-; ALL:           or      $[[R16:[0-9]+]], $[[R15]], $[[R14]]
-; ALL:           sc      $[[R16]], 0($[[R2]])
-; NOT-MICROMIPS: beqz    $[[R16]], [[BB0]]
-; MICROMIPS:     beqzc   $[[R16]], [[BB0]]
-; MIPSR6:        beqzc   $[[R16]], [[BB0]]
-
-; ALL:           and     $[[R17:[0-9]+]], $[[R12]], $[[R7]]
-; ALL:           srlv    $[[R18:[0-9]+]], $[[R17]], $[[R5]]
-
-; NO-SEB-SEH:    sll     $[[R19:[0-9]+]], $[[R18]], 16
-; NO-SEB-SEH:    sra     $2, $[[R19]], 16
-
-; MIPS32R2:      seh     $2, $[[R18]]
 }
 
 ; Test that the i16 return value from cmpxchg is recognised as signed,
 ; so that setCC doesn't end up comparing an unsigned value to a signed
 ; value.
 ; The rest of the functions here are testing the atomic expansion, so
 ; we just match the end of the function.
 define {i16, i1} @foo(i16* %addr, i16 %l, i16 %r, i16 %new) {
+; MIPS32-LABEL: foo:
+; MIPS32:       # %bb.0:
+; MIPS32-NEXT:    addu $1, $5, $6
+; MIPS32-NEXT:    sync
+; MIPS32-NEXT:    addiu $2, $zero, -4
+; MIPS32-NEXT:    and $3, $4, $2
+; MIPS32-NEXT:    andi $2, $4, 3
+; MIPS32-NEXT:    sll $4, $2, 3
+; MIPS32-NEXT:    ori $2, $zero, 65535
+; MIPS32-NEXT:    sllv $5, $2, $4
+; MIPS32-NEXT:    nor $6, $zero, $5
+; MIPS32-NEXT:    andi $2, $1, 65535
+; MIPS32-NEXT:    sllv $8, $2, $4
+; MIPS32-NEXT:    andi $2, $7, 65535
+; MIPS32-NEXT:    sllv $7, $2, $4
+; MIPS32-NEXT:  $BB15_1: # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $9, 0($3)
+; MIPS32-NEXT:    and $10, $9, $5
+; MIPS32-NEXT:    bne $10, $8, $BB15_3
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; MIPS32-NEXT:    and $9, $9, $6
+; MIPS32-NEXT:    or $9, $9, $7
+; MIPS32-NEXT:    sc $9, 0($3)
+; MIPS32-NEXT:    beqz $9, $BB15_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  $BB15_3:
+; MIPS32-NEXT:    srlv $2, $10, $4
+; MIPS32-NEXT:    sll $2, $2, 16
+; MIPS32-NEXT:    sra $2, $2, 16
+; MIPS32-NEXT:  # %bb.4:
+; MIPS32-NEXT:    sll $1, $1, 16
+; MIPS32-NEXT:    sra $1, $1, 16
+; MIPS32-NEXT:    xor $1, $2, $1
+; MIPS32-NEXT:    sltiu $3, $1, 1
+; MIPS32-NEXT:    sync
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    nop
+;
+; MIPS32O0-LABEL: foo:
+; MIPS32O0:       # %bb.0:
+; MIPS32O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32O0-NEXT:    .cfi_def_cfa_offset 8
+; MIPS32O0-NEXT:    addu $5, $5, $6
+; MIPS32O0-NEXT:    sync
+; MIPS32O0-NEXT:    addiu $6, $zero, -4
+; MIPS32O0-NEXT:    and $6, $4, $6
+; MIPS32O0-NEXT:    andi $4, $4, 3
+; MIPS32O0-NEXT:    sll $4, $4, 3
+; MIPS32O0-NEXT:    ori $1, $zero, 65535
+; MIPS32O0-NEXT:    sllv $1, $1, $4
+; MIPS32O0-NEXT:    nor $2, $zero, $1
+; MIPS32O0-NEXT:    andi $3, $5, 65535
+; MIPS32O0-NEXT:    sllv $3, $3, $4
+; MIPS32O0-NEXT:    andi $7, $7, 65535
+; MIPS32O0-NEXT:    sllv $7, $7, $4
+; MIPS32O0-NEXT:  $BB15_1: # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $9, 0($6)
+; MIPS32O0-NEXT:    and $10, $9, $1
+; MIPS32O0-NEXT:    bne $10, $3, $BB15_3
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; MIPS32O0-NEXT:    and $9, $9, $2
+; MIPS32O0-NEXT:    or $9, $9, $7
+; MIPS32O0-NEXT:    sc $9, 0($6)
+; MIPS32O0-NEXT:    beqz $9, $BB15_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  $BB15_3:
+; MIPS32O0-NEXT:    srlv $8, $10, $4
+; MIPS32O0-NEXT:    sll $8, $8, 16
+; MIPS32O0-NEXT:    sra $8, $8, 16
+; MIPS32O0-NEXT:  # %bb.4:
+; MIPS32O0-NEXT:    sw $5, 4($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:    sw $8, 0($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:  # %bb.5:
+; MIPS32O0-NEXT:    lw $1, 4($sp) # 4-byte Folded Reload
+; MIPS32O0-NEXT:    sll $2, $1, 16
+; MIPS32O0-NEXT:    sra $2, $2, 16
+; MIPS32O0-NEXT:    lw $3, 0($sp) # 4-byte Folded Reload
+; MIPS32O0-NEXT:    xor $2, $3, $2
+; MIPS32O0-NEXT:    sltiu $3, $2, 1
+; MIPS32O0-NEXT:    sync
+; MIPS32O0-NEXT:    lw $2, 0($sp) # 4-byte Folded Reload
+; MIPS32O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: foo:
+; MIPS32R2:       # %bb.0:
+; MIPS32R2-NEXT:    addu $1, $5, $6
+; MIPS32R2-NEXT:    sync
+; MIPS32R2-NEXT:    addiu $2, $zero, -4
+; MIPS32R2-NEXT:    and $3, $4, $2
+; MIPS32R2-NEXT:    andi $2, $4, 3
+; MIPS32R2-NEXT:    sll $4, $2, 3
+; MIPS32R2-NEXT:    ori $2, $zero, 65535
+; MIPS32R2-NEXT:    sllv $5, $2, $4
+; MIPS32R2-NEXT:    nor $6, $zero, $5
+; MIPS32R2-NEXT:    andi $2, $1, 65535
+; MIPS32R2-NEXT:    sllv $8, $2, $4
+; MIPS32R2-NEXT:    andi $2, $7, 65535
+; MIPS32R2-NEXT:    sllv $7, $2, $4
+; MIPS32R2-NEXT:  $BB15_1: # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $9, 0($3)
+; MIPS32R2-NEXT:    and $10, $9, $5
+; MIPS32R2-NEXT:    bne $10, $8, $BB15_3
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; MIPS32R2-NEXT:    and $9, $9, $6
+; MIPS32R2-NEXT:    or $9, $9, $7
+; MIPS32R2-NEXT:    sc $9, 0($3)
+; MIPS32R2-NEXT:    beqz $9, $BB15_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  $BB15_3:
+; MIPS32R2-NEXT:    srlv $2, $10, $4
+; MIPS32R2-NEXT:    seh $2, $2
+; MIPS32R2-NEXT:  # %bb.4:
+; MIPS32R2-NEXT:    seh $1, $1
+; MIPS32R2-NEXT:    xor $1, $2, $1
+; MIPS32R2-NEXT:    sltiu $3, $1, 1
+; MIPS32R2-NEXT:    sync
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    nop
+;
+; MIPS32R6-LABEL: foo:
+; MIPS32R6:       # %bb.0:
+; MIPS32R6-NEXT:    addu $1, $5, $6
+; MIPS32R6-NEXT:    sync
+; MIPS32R6-NEXT:    addiu $2, $zero, -4
+; MIPS32R6-NEXT:    and $3, $4, $2
+; MIPS32R6-NEXT:    andi $2, $4, 3
+; MIPS32R6-NEXT:    sll $4, $2, 3
+; MIPS32R6-NEXT:    ori $2, $zero, 65535
+; MIPS32R6-NEXT:    sllv $5, $2, $4
+; MIPS32R6-NEXT:    nor $6, $zero, $5
+; MIPS32R6-NEXT:    andi $2, $1, 65535
+; MIPS32R6-NEXT:    sllv $8, $2, $4
+; MIPS32R6-NEXT:    andi $2, $7, 65535
+; MIPS32R6-NEXT:    sllv $7, $2, $4
+; MIPS32R6-NEXT:  $BB15_1: # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $9, 0($3)
+; MIPS32R6-NEXT:    and $10, $9, $5
+; MIPS32R6-NEXT:    bnec $10, $8, $BB15_3
+; MIPS32R6-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; MIPS32R6-NEXT:    and $9, $9, $6
+; MIPS32R6-NEXT:    or $9, $9, $7
+; MIPS32R6-NEXT:    sc $9, 0($3)
+; MIPS32R6-NEXT:    beqzc $9, $BB15_1
+; MIPS32R6-NEXT:  $BB15_3:
+; MIPS32R6-NEXT:    srlv $2, $10, $4
+; MIPS32R6-NEXT:    seh $2, $2
+; MIPS32R6-NEXT:  # %bb.4:
+; MIPS32R6-NEXT:    seh $1, $1
+; MIPS32R6-NEXT:    xor $1, $2, $1
+; MIPS32R6-NEXT:    sltiu $3, $1, 1
+; MIPS32R6-NEXT:    sync
+; MIPS32R6-NEXT:    jrc $ra
+;
+; MIPS32R6O0-LABEL: foo:
+; MIPS32R6O0:       # %bb.0:
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -24
+; MIPS32R6O0-NEXT:    .cfi_def_cfa_offset 24
+; MIPS32R6O0-NEXT:    move $1, $7
+; MIPS32R6O0-NEXT:    move $2, $6
+; MIPS32R6O0-NEXT:    move $3, $5
+; MIPS32R6O0-NEXT:    move $8, $4
+; MIPS32R6O0-NEXT:    addu $5, $5, $6
+; MIPS32R6O0-NEXT:    sync
+; MIPS32R6O0-NEXT:    addiu $6, $zero, -4
+; MIPS32R6O0-NEXT:    and $6, $4, $6
+; MIPS32R6O0-NEXT:    andi $4, $4, 3
+; MIPS32R6O0-NEXT:    sll $4, $4, 3
+; MIPS32R6O0-NEXT:    ori $9, $zero, 65535
+; MIPS32R6O0-NEXT:    sllv $9, $9, $4
+; MIPS32R6O0-NEXT:    nor $10, $zero, $9
+; MIPS32R6O0-NEXT:    andi $11, $5, 65535
+; MIPS32R6O0-NEXT:    sllv $11, $11, $4
+; MIPS32R6O0-NEXT:    andi $7, $7, 65535
+; MIPS32R6O0-NEXT:    sllv $7, $7, $4
+; MIPS32R6O0-NEXT:  $BB15_1: # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $13, 0($6)
+; MIPS32R6O0-NEXT:    and $14, $13, $9
+; MIPS32R6O0-NEXT:    bnec $14, $11, $BB15_3
+; MIPS32R6O0-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; MIPS32R6O0-NEXT:    and $13, $13, $10
+; MIPS32R6O0-NEXT:    or $13, $13, $7
+; MIPS32R6O0-NEXT:    sc $13, 0($6)
+; MIPS32R6O0-NEXT:    beqzc $13, $BB15_1
+; MIPS32R6O0-NEXT:  $BB15_3:
+; MIPS32R6O0-NEXT:    srlv $12, $14, $4
+; MIPS32R6O0-NEXT:    seh $12, $12
+; MIPS32R6O0-NEXT:  # %bb.4:
+; MIPS32R6O0-NEXT:    sw $12, 20($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $3, 16($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $8, 12($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $5, 8($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $2, 0($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:  # %bb.5:
+; MIPS32R6O0-NEXT:    lw $1, 8($sp) # 4-byte Folded Reload
+; MIPS32R6O0-NEXT:    seh $2, $1
+; MIPS32R6O0-NEXT:    lw $3, 20($sp) # 4-byte Folded Reload
+; MIPS32R6O0-NEXT:    xor $2, $3, $2
+; MIPS32R6O0-NEXT:    sltiu $3, $2, 1
+; MIPS32R6O0-NEXT:    sync
+; MIPS32R6O0-NEXT:    lw $2, 20($sp) # 4-byte Folded Reload
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 24
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: foo:
+; MIPS4:       # %bb.0:
+; MIPS4-NEXT:    sll $1, $6, 0
+; MIPS4-NEXT:    sll $2, $5, 0
+; MIPS4-NEXT:    addu $1, $2, $1
+; MIPS4-NEXT:    sync
+; MIPS4-NEXT:    sll $2, $7, 0
+; MIPS4-NEXT:    daddiu $3, $zero, -4
+; MIPS4-NEXT:    and $3, $4, $3
+; MIPS4-NEXT:    andi $4, $4, 3
+; MIPS4-NEXT:    sll $4, $4, 3
+; MIPS4-NEXT:    ori $5, $zero, 65535
+; MIPS4-NEXT:    sllv $5, $5, $4
+; MIPS4-NEXT:    nor $6, $zero, $5
+; MIPS4-NEXT:    andi $7, $1, 65535
+; MIPS4-NEXT:    sllv $7, $7, $4
+; MIPS4-NEXT:    andi $2, $2, 65535
+; MIPS4-NEXT:    sllv $8, $2, $4
+; MIPS4-NEXT:  .LBB15_1: # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $9, 0($3)
+; MIPS4-NEXT:    and $10, $9, $5
+; MIPS4-NEXT:    bne $10, $7, .LBB15_3
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; MIPS4-NEXT:    and $9, $9, $6
+; MIPS4-NEXT:    or $9, $9, $8
+; MIPS4-NEXT:    sc $9, 0($3)
+; MIPS4-NEXT:    beqz $9, .LBB15_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  .LBB15_3:
+; MIPS4-NEXT:    srlv $2, $10, $4
+; MIPS4-NEXT:    sll $2, $2, 16
+; MIPS4-NEXT:    sra $2, $2, 16
+; MIPS4-NEXT:  # %bb.4:
+; MIPS4-NEXT:    sll $1, $1, 16
+; MIPS4-NEXT:    sra $1, $1, 16
+; MIPS4-NEXT:    xor $1, $2, $1
+; MIPS4-NEXT:    sltiu $3, $1, 1
+; MIPS4-NEXT:    sync
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: foo:
+; MIPS64:       # %bb.0:
+; MIPS64-NEXT:    sll $1, $6, 0
+; MIPS64-NEXT:    sll $2, $5, 0
+; MIPS64-NEXT:    addu $1, $2, $1
+; MIPS64-NEXT:    sync
+; MIPS64-NEXT:    sll $2, $7, 0
+; MIPS64-NEXT:    daddiu $3, $zero, -4
+; MIPS64-NEXT:    and $3, $4, $3
+; MIPS64-NEXT:    andi $4, $4, 3
+; MIPS64-NEXT:    sll $4, $4, 3
+; MIPS64-NEXT:    ori $5, $zero, 65535
+; MIPS64-NEXT:    sllv $5, $5, $4
+; MIPS64-NEXT:    nor $6, $zero, $5
+; MIPS64-NEXT:    andi $7, $1, 65535
+; MIPS64-NEXT:    sllv $7, $7, $4
+; MIPS64-NEXT:    andi $2, $2, 65535
+; MIPS64-NEXT:    sllv $8, $2, $4
+; MIPS64-NEXT:  .LBB15_1: # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $9, 0($3)
+; MIPS64-NEXT:    and $10, $9, $5
+; MIPS64-NEXT:    bne $10, $7, .LBB15_3
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; MIPS64-NEXT:    and $9, $9, $6
+; MIPS64-NEXT:    or $9, $9, $8
+; MIPS64-NEXT:    sc $9, 0($3)
+; MIPS64-NEXT:    beqz $9, .LBB15_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  .LBB15_3:
+; MIPS64-NEXT:    srlv $2, $10, $4
+; MIPS64-NEXT:    sll $2, $2, 16
+; MIPS64-NEXT:    sra $2, $2, 16
+; MIPS64-NEXT:  # %bb.4:
+; MIPS64-NEXT:    sll $1, $1, 16
+; MIPS64-NEXT:    sra $1, $1, 16
+; MIPS64-NEXT:    xor $1, $2, $1
+; MIPS64-NEXT:    sltiu $3, $1, 1
+; MIPS64-NEXT:    sync
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: foo:
+; MIPS64R2:       # %bb.0:
+; MIPS64R2-NEXT:    sll $1, $6, 0
+; MIPS64R2-NEXT:    sll $2, $5, 0
+; MIPS64R2-NEXT:    addu $1, $2, $1
+; MIPS64R2-NEXT:    sync
+; MIPS64R2-NEXT:    sll $2, $7, 0
+; MIPS64R2-NEXT:    daddiu $3, $zero, -4
+; MIPS64R2-NEXT:    and $3, $4, $3
+; MIPS64R2-NEXT:    andi $4, $4, 3
+; MIPS64R2-NEXT:    sll $4, $4, 3
+; MIPS64R2-NEXT:    ori $5, $zero, 65535
+; MIPS64R2-NEXT:    sllv $5, $5, $4
+; MIPS64R2-NEXT:    nor $6, $zero, $5
+; MIPS64R2-NEXT:    andi $7, $1, 65535
+; MIPS64R2-NEXT:    sllv $7, $7, $4
+; MIPS64R2-NEXT:    andi $2, $2, 65535
+; MIPS64R2-NEXT:    sllv $8, $2, $4
+; MIPS64R2-NEXT:  .LBB15_1: # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $9, 0($3)
+; MIPS64R2-NEXT:    and $10, $9, $5
+; MIPS64R2-NEXT:    bne $10, $7, .LBB15_3
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; MIPS64R2-NEXT:    and $9, $9, $6
+; MIPS64R2-NEXT:    or $9, $9, $8
+; MIPS64R2-NEXT:    sc $9, 0($3)
+; MIPS64R2-NEXT:    beqz $9, .LBB15_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  .LBB15_3:
+; MIPS64R2-NEXT:    srlv $2, $10, $4
+; MIPS64R2-NEXT:    seh $2, $2
+; MIPS64R2-NEXT:  # %bb.4:
+; MIPS64R2-NEXT:    seh $1, $1
+; MIPS64R2-NEXT:    xor $1, $2, $1
+; MIPS64R2-NEXT:    sltiu $3, $1, 1
+; MIPS64R2-NEXT:    sync
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: foo:
+; MIPS64R6:       # %bb.0:
+; MIPS64R6-NEXT:    sll $1, $6, 0
+; MIPS64R6-NEXT:    sll $2, $5, 0
+; MIPS64R6-NEXT:    addu $1, $2, $1
+; MIPS64R6-NEXT:    sync
+; MIPS64R6-NEXT:    sll $2, $7, 0
+; MIPS64R6-NEXT:    daddiu $3, $zero, -4
+; MIPS64R6-NEXT:    and $3, $4, $3
+; MIPS64R6-NEXT:    andi $4, $4, 3
+; MIPS64R6-NEXT:    sll $4, $4, 3
+; MIPS64R6-NEXT:    ori $5, $zero, 65535
+; MIPS64R6-NEXT:    sllv $5, $5, $4
+; MIPS64R6-NEXT:    nor $6, $zero, $5
+; MIPS64R6-NEXT:    andi $7, $1, 65535
+; MIPS64R6-NEXT:    sllv $7, $7, $4
+; MIPS64R6-NEXT:    andi $2, $2, 65535
+; MIPS64R6-NEXT:    sllv $8, $2, $4
+; MIPS64R6-NEXT:  .LBB15_1: # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $9, 0($3)
+; MIPS64R6-NEXT:    and $10, $9, $5
+; MIPS64R6-NEXT:    bnec $10, $7, .LBB15_3
+; MIPS64R6-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; MIPS64R6-NEXT:    and $9, $9, $6
+; MIPS64R6-NEXT:    or $9, $9, $8
+; MIPS64R6-NEXT:    sc $9, 0($3)
+; MIPS64R6-NEXT:    beqzc $9, .LBB15_1
+; MIPS64R6-NEXT:  .LBB15_3:
+; MIPS64R6-NEXT:    srlv $2, $10, $4
+; MIPS64R6-NEXT:    seh $2, $2
+; MIPS64R6-NEXT:  # %bb.4:
+; MIPS64R6-NEXT:    seh $1, $1
+; MIPS64R6-NEXT:    xor $1, $2, $1
+; MIPS64R6-NEXT:    sltiu $3, $1, 1
+; MIPS64R6-NEXT:    sync
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: foo:
+; MIPS64R6O0:       # %bb.0:
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    .cfi_def_cfa_offset 16
+; MIPS64R6O0-NEXT:    move $1, $7
+; MIPS64R6O0-NEXT:    sll $1, $1, 0
+; MIPS64R6O0-NEXT:    move $2, $6
+; MIPS64R6O0-NEXT:    sll $2, $2, 0
+; MIPS64R6O0-NEXT:    move $3, $5
+; MIPS64R6O0-NEXT:    sll $3, $3, 0
+; MIPS64R6O0-NEXT:    move $5, $4
+; MIPS64R6O0-NEXT:    addu $2, $3, $2
+; MIPS64R6O0-NEXT:    sync
+; MIPS64R6O0-NEXT:    daddiu $6, $zero, -4
+; MIPS64R6O0-NEXT:    and $6, $4, $6
+; MIPS64R6O0-NEXT:    andi $3, $4, 3
+; MIPS64R6O0-NEXT:    xori $3, $3, 2
+; MIPS64R6O0-NEXT:    sll $3, $3, 3
+; MIPS64R6O0-NEXT:    ori $8, $zero, 65535
+; MIPS64R6O0-NEXT:    sllv $8, $8, $3
+; MIPS64R6O0-NEXT:    nor $9, $zero, $8
+; MIPS64R6O0-NEXT:    andi $10, $2, 65535
+; MIPS64R6O0-NEXT:    sllv $10, $10, $3
+; MIPS64R6O0-NEXT:    andi $1, $1, 65535
+; MIPS64R6O0-NEXT:    sllv $1, $1, $3
+; MIPS64R6O0-NEXT:  .LBB15_1: # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $12, 0($6)
+; MIPS64R6O0-NEXT:    and $13, $12, $8
+; MIPS64R6O0-NEXT:    bnec $13, $10, .LBB15_3
+; MIPS64R6O0-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; MIPS64R6O0-NEXT:    and $12, $12, $9
+; MIPS64R6O0-NEXT:    or $12, $12, $1
+; MIPS64R6O0-NEXT:    sc $12, 0($6)
+; MIPS64R6O0-NEXT:    beqzc $12, .LBB15_1
+; MIPS64R6O0-NEXT:  .LBB15_3:
+; MIPS64R6O0-NEXT:    srlv $11, $13, $3
+; MIPS64R6O0-NEXT:    seh $11, $11
+; MIPS64R6O0-NEXT:  # %bb.4:
+; MIPS64R6O0-NEXT:    sw $2, 12($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:    sw $11, 8($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:    sd $5, 0($sp) # 8-byte Folded Spill
+; MIPS64R6O0-NEXT:  # %bb.5:
+; MIPS64R6O0-NEXT:    lw $1, 12($sp) # 4-byte Folded Reload
+; MIPS64R6O0-NEXT:    seh $2, $1
+; MIPS64R6O0-NEXT:    lw $3, 8($sp) # 4-byte Folded Reload
+; MIPS64R6O0-NEXT:    xor $2, $3, $2
+; MIPS64R6O0-NEXT:    sltiu $3, $2, 1
+; MIPS64R6O0-NEXT:    sync
+; MIPS64R6O0-NEXT:    lw $2, 8($sp) # 4-byte Folded Reload
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: foo:
+; MM32:       # %bb.0:
+; MM32-NEXT:    addu16 $3, $5, $6
+; MM32-NEXT:    sync
+; MM32-NEXT:    addiu $1, $zero, -4
+; MM32-NEXT:    and $1, $4, $1
+; MM32-NEXT:    andi $2, $4, 3
+; MM32-NEXT:    sll $4, $2, 3
+; MM32-NEXT:    ori $2, $zero, 65535
+; MM32-NEXT:    sllv $5, $2, $4
+; MM32-NEXT:    nor $6, $zero, $5
+; MM32-NEXT:    andi $2, $3, 65535
+; MM32-NEXT:    sllv $8, $2, $4
+; MM32-NEXT:    andi $2, $7, 65535
+; MM32-NEXT:    sllv $7, $2, $4
+; MM32-NEXT:  $BB15_1: # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $9, 0($1)
+; MM32-NEXT:    and $10, $9, $5
+; MM32-NEXT:    bne $10, $8, $BB15_3
+; MM32-NEXT:    nop
+; MM32-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; MM32-NEXT:    and $9, $9, $6
+; MM32-NEXT:    or $9, $9, $7
+; MM32-NEXT:    sc $9, 0($1)
+; MM32-NEXT:    beqzc $9, $BB15_1
+; MM32-NEXT:  $BB15_3:
+; MM32-NEXT:    srlv $2, $10, $4
+; MM32-NEXT:    seh $2, $2
+; MM32-NEXT:  # %bb.4:
+; MM32-NEXT:    seh $1, $3
+; MM32-NEXT:    xor $1, $2, $1
+; MM32-NEXT:    sltiu $3, $1, 1
+; MM32-NEXT:    sync
+; MM32-NEXT:    jrc $ra
+;
+; O1-LABEL: foo:
+; O1:       # %bb.0:
+; O1-NEXT:    addu $1, $5, $6
+; O1-NEXT:    sync
+; O1-NEXT:    addiu $2, $zero, -4
+; O1-NEXT:    and $3, $4, $2
+; O1-NEXT:    andi $2, $4, 3
+; O1-NEXT:    sll $4, $2, 3
+; O1-NEXT:    ori $2, $zero, 65535
+; O1-NEXT:    sllv $5, $2, $4
+; O1-NEXT:    nor $6, $zero, $5
+; O1-NEXT:    andi $2, $1, 65535
+; O1-NEXT:    sllv $8, $2, $4
+; O1-NEXT:    andi $2, $7, 65535
+; O1-NEXT:    sllv $7, $2, $4
+; O1-NEXT:  $BB15_1: # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $9, 0($3)
+; O1-NEXT:    and $10, $9, $5
+; O1-NEXT:    bne $10, $8, $BB15_3
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; O1-NEXT:    and $9, $9, $6
+; O1-NEXT:    or $9, $9, $7
+; O1-NEXT:    sc $9, 0($3)
+; O1-NEXT:    beqz $9, $BB15_1
+; O1-NEXT:    nop
+; O1-NEXT:  $BB15_3:
+; O1-NEXT:    srlv $2, $10, $4
+; O1-NEXT:    sll $2, $2, 16
+; O1-NEXT:    sra $2, $2, 16
+; O1-NEXT:  # %bb.4:
+; O1-NEXT:    sll $1, $1, 16
+; O1-NEXT:    sra $1, $1, 16
+; O1-NEXT:    xor $1, $2, $1
+; O1-NEXT:    sltiu $3, $1, 1
+; O1-NEXT:    sync
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: foo:
+; O2:       # %bb.0:
+; O2-NEXT:    addu $1, $5, $6
+; O2-NEXT:    sync
+; O2-NEXT:    addiu $2, $zero, -4
+; O2-NEXT:    and $3, $4, $2
+; O2-NEXT:    andi $2, $4, 3
+; O2-NEXT:    sll $4, $2, 3
+; O2-NEXT:    ori $2, $zero, 65535
+; O2-NEXT:    sllv $5, $2, $4
+; O2-NEXT:    nor $6, $zero, $5
+; O2-NEXT:    andi $2, $1, 65535
+; O2-NEXT:    sllv $8, $2, $4
+; O2-NEXT:    andi $2, $7, 65535
+; O2-NEXT:    sllv $7, $2, $4
+; O2-NEXT:  $BB15_1: # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $9, 0($3)
+; O2-NEXT:    and $10, $9, $5
+; O2-NEXT:    bne $10, $8, $BB15_3
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; O2-NEXT:    and $9, $9, $6
+; O2-NEXT:    or $9, $9, $7
+; O2-NEXT:    sc $9, 0($3)
+; O2-NEXT:    beqz $9, $BB15_1
+; O2-NEXT:    nop
+; O2-NEXT:  $BB15_3:
+; O2-NEXT:    srlv $2, $10, $4
+; O2-NEXT:    sll $2, $2, 16
+; O2-NEXT:    sra $2, $2, 16
+; O2-NEXT:  # %bb.4:
+; O2-NEXT:    sll $1, $1, 16
+; O2-NEXT:    sra $1, $1, 16
+; O2-NEXT:    xor $1, $2, $1
+; O2-NEXT:    sltiu $3, $1, 1
+; O2-NEXT:    sync
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: foo:
+; O3:       # %bb.0:
+; O3-NEXT:    addiu $2, $zero, -4
+; O3-NEXT:    addu $1, $5, $6
+; O3-NEXT:    sync
+; O3-NEXT:    and $3, $4, $2
+; O3-NEXT:    andi $2, $4, 3
+; O3-NEXT:    sll $4, $2, 3
+; O3-NEXT:    ori $2, $zero, 65535
+; O3-NEXT:    sllv $5, $2, $4
+; O3-NEXT:    andi $2, $1, 65535
+; O3-NEXT:    sll $1, $1, 16
+; O3-NEXT:    sllv $8, $2, $4
+; O3-NEXT:    andi $2, $7, 65535
+; O3-NEXT:    nor $6, $zero, $5
+; O3-NEXT:    sra $1, $1, 16
+; O3-NEXT:    sllv $7, $2, $4
+; O3-NEXT:  $BB15_1: # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $9, 0($3)
+; O3-NEXT:    and $10, $9, $5
+; O3-NEXT:    bne $10, $8, $BB15_3
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; O3-NEXT:    and $9, $9, $6
+; O3-NEXT:    or $9, $9, $7
+; O3-NEXT:    sc $9, 0($3)
+; O3-NEXT:    beqz $9, $BB15_1
+; O3-NEXT:    nop
+; O3-NEXT:  $BB15_3:
+; O3-NEXT:    srlv $2, $10, $4
+; O3-NEXT:    sll $2, $2, 16
+; O3-NEXT:    sra $2, $2, 16
+; O3-NEXT:  # %bb.4:
+; O3-NEXT:    sync
+; O3-NEXT:    xor $1, $2, $1
+; O3-NEXT:    jr $ra
+; O3-NEXT:    sltiu $3, $1, 1
+;
+; MIPS32EB-LABEL: foo:
+; MIPS32EB:       # %bb.0:
+; MIPS32EB-NEXT:    addu $1, $5, $6
+; MIPS32EB-NEXT:    sync
+; MIPS32EB-NEXT:    addiu $2, $zero, -4
+; MIPS32EB-NEXT:    and $3, $4, $2
+; MIPS32EB-NEXT:    andi $2, $4, 3
+; MIPS32EB-NEXT:    xori $2, $2, 2
+; MIPS32EB-NEXT:    sll $4, $2, 3
+; MIPS32EB-NEXT:    ori $2, $zero, 65535
+; MIPS32EB-NEXT:    sllv $5, $2, $4
+; MIPS32EB-NEXT:    nor $6, $zero, $5
+; MIPS32EB-NEXT:    andi $2, $1, 65535
+; MIPS32EB-NEXT:    sllv $8, $2, $4
+; MIPS32EB-NEXT:    andi $2, $7, 65535
+; MIPS32EB-NEXT:    sllv $7, $2, $4
+; MIPS32EB-NEXT:  $BB15_1: # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $9, 0($3)
+; MIPS32EB-NEXT:    and $10, $9, $5
+; MIPS32EB-NEXT:    bne $10, $8, $BB15_3
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # in Loop: Header=BB15_1 Depth=1
+; MIPS32EB-NEXT:    and $9, $9, $6
+; MIPS32EB-NEXT:    or $9, $9, $7
+; MIPS32EB-NEXT:    sc $9, 0($3)
+; MIPS32EB-NEXT:    beqz $9, $BB15_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  $BB15_3:
+; MIPS32EB-NEXT:    srlv $2, $10, $4
+; MIPS32EB-NEXT:    sll $2, $2, 16
+; MIPS32EB-NEXT:    sra $2, $2, 16
+; MIPS32EB-NEXT:  # %bb.4:
+; MIPS32EB-NEXT:    sll $1, $1, 16
+; MIPS32EB-NEXT:    sra $1, $1, 16
+; MIPS32EB-NEXT:    xor $1, $2, $1
+; MIPS32EB-NEXT:    sltiu $3, $1, 1
+; MIPS32EB-NEXT:    sync
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    nop
   %desired = add i16 %l, %r
   %res = cmpxchg i16* %addr, i16 %desired, i16 %new seq_cst seq_cst
   ret {i16, i1} %res
-
-; ALL-LABEL: foo
-; MIPSR6:        addu    $[[R2:[0-9]+]], $[[R1:[0-9]+]], $[[R0:[0-9]+]]
-; NOT-MICROMIPS: addu    $[[R2:[0-9]+]], $[[R1:[0-9]+]], $[[R0:[0-9]+]]
-; MICROMIPS:     addu16  $[[R2:[0-9]+]], $[[R1:[0-9]+]], $[[R0:[0-9]+]]
-
-; ALL:           sync
-
-; ALL:           andi    $[[R3:[0-9]+]], $[[R2]], 65535
-; ALL:       [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; ALL:           ll      $[[R4:[0-9]+]], 0($[[R5:[0-9]+]])
-; ALL:           and     $[[R6:[0-9]+]], $[[R4]], $
-; ALL:           and     $[[R7:[0-9]+]], $[[R4]], $
-; ALL:           or      $[[R8:[0-9]+]], $[[R7]], $
-; ALL:           sc      $[[R8]], 0($[[R5]])
-; NOT-MICROMIPS: beqz    $[[R8]], [[BB0]]
-; MICROMIPS:     beqzc   $[[R8]], [[BB0]]
-; MIPSR6:        beqzc   $[[R8]], [[BB0]]
-
-; ALL:           srlv    $[[R9:[0-9]+]], $[[R6]], $
-
-; NO-SEB-SEH:    sll     $[[R10:[0-9]+]], $[[R9]], 16
-; NO-SEB-SEH:    sra     $[[R11:[0-9]+]], $[[R10]], 16
-
-; NO-SEB-SEH:    sll     $[[R12:[0-9]+]], $[[R2]], 16
-; NO-SEB-SEH:    sra     $[[R13:[0-9]+]], $[[R12]], 16
-
-; HAS-SEB-SEH:   seh     $[[R11:[0-9]+]], $[[R9]]
-; HAS-SEB-SEH:   seh     $[[R13:[0-9]+]], $[[R2]]
-
-; ALL:           xor     $[[R12:[0-9]+]], $[[R11]], $[[R13]]
-; ALL:           sltiu   $3, $[[R12]], 1
-; ALL:           sync
 }
 
 @countsint = common global i32 0, align 4
 
 define i32 @CheckSync(i32 signext %v) nounwind noinline {
+; MIPS32-LABEL: CheckSync:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    sync
+; MIPS32-NEXT:    lw $1, %got(countsint)($1)
+; MIPS32-NEXT:  $BB16_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $2, 0($1)
+; MIPS32-NEXT:    addu $3, $2, $4
+; MIPS32-NEXT:    sc $3, 0($1)
+; MIPS32-NEXT:    beqz $3, $BB16_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    sync
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    nop
+;
+; MIPS32O0-LABEL: CheckSync:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    sync
+; MIPS32O0-NEXT:    lw $2, %got(countsint)($2)
+; MIPS32O0-NEXT:  $BB16_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $25, 0($2)
+; MIPS32O0-NEXT:    addu $1, $25, $4
+; MIPS32O0-NEXT:    sc $1, 0($2)
+; MIPS32O0-NEXT:    beqz $1, $BB16_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    sync
+; MIPS32O0-NEXT:    move $2, $25
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: CheckSync:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    sync
+; MIPS32R2-NEXT:    lw $1, %got(countsint)($1)
+; MIPS32R2-NEXT:  $BB16_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $2, 0($1)
+; MIPS32R2-NEXT:    addu $3, $2, $4
+; MIPS32R2-NEXT:    sc $3, 0($1)
+; MIPS32R2-NEXT:    beqz $3, $BB16_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    sync
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    nop
+;
+; MIPS32R6-LABEL: CheckSync:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    sync
+; MIPS32R6-NEXT:    lw $1, %got(countsint)($1)
+; MIPS32R6-NEXT:  $BB16_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $2, 0($1)
+; MIPS32R6-NEXT:    addu $3, $2, $4
+; MIPS32R6-NEXT:    sc $3, 0($1)
+; MIPS32R6-NEXT:    beqzc $3, $BB16_1
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    sync
+; MIPS32R6-NEXT:    jrc $ra
+;
+; MIPS32R6O0-LABEL: CheckSync:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    sync
+; MIPS32R6O0-NEXT:    lw $2, %got(countsint)($2)
+; MIPS32R6O0-NEXT:  $BB16_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $1, 0($2)
+; MIPS32R6O0-NEXT:    addu $3, $1, $4
+; MIPS32R6O0-NEXT:    sc $3, 0($2)
+; MIPS32R6O0-NEXT:    beqzc $3, $BB16_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    sync
+; MIPS32R6O0-NEXT:    move $2, $1
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: CheckSync:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(CheckSync)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(CheckSync)))
+; MIPS4-NEXT:    sync
+; MIPS4-NEXT:    ld $1, %got_disp(countsint)($1)
+; MIPS4-NEXT:  .LBB16_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $2, 0($1)
+; MIPS4-NEXT:    addu $3, $2, $4
+; MIPS4-NEXT:    sc $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB16_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    sync
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: CheckSync:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(CheckSync)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(CheckSync)))
+; MIPS64-NEXT:    sync
+; MIPS64-NEXT:    ld $1, %got_disp(countsint)($1)
+; MIPS64-NEXT:  .LBB16_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $2, 0($1)
+; MIPS64-NEXT:    addu $3, $2, $4
+; MIPS64-NEXT:    sc $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB16_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    sync
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: CheckSync:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(CheckSync)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(CheckSync)))
+; MIPS64R2-NEXT:    sync
+; MIPS64R2-NEXT:    ld $1, %got_disp(countsint)($1)
+; MIPS64R2-NEXT:  .LBB16_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $2, 0($1)
+; MIPS64R2-NEXT:    addu $3, $2, $4
+; MIPS64R2-NEXT:    sc $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB16_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    sync
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: CheckSync:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(CheckSync)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(CheckSync)))
+; MIPS64R6-NEXT:    sync
+; MIPS64R6-NEXT:    ld $1, %got_disp(countsint)($1)
+; MIPS64R6-NEXT:  .LBB16_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $2, 0($1)
+; MIPS64R6-NEXT:    addu $3, $2, $4
+; MIPS64R6-NEXT:    sc $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB16_1
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    sync
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: CheckSync:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(CheckSync)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(CheckSync)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    sync
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(countsint)($1)
+; MIPS64R6O0-NEXT:  .LBB16_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $3, 0($1)
+; MIPS64R6O0-NEXT:    addu $5, $3, $2
+; MIPS64R6O0-NEXT:    sc $5, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $5, .LBB16_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    sync
+; MIPS64R6O0-NEXT:    move $2, $3
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: CheckSync:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    sync
+; MM32-NEXT:    lw $1, %got(countsint)($2)
+; MM32-NEXT:  $BB16_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $2, 0($1)
+; MM32-NEXT:    addu16 $3, $2, $4
+; MM32-NEXT:    sc $3, 0($1)
+; MM32-NEXT:    beqzc $3, $BB16_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    sync
+; MM32-NEXT:    jrc $ra
+;
+; O1-LABEL: CheckSync:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    sync
+; O1-NEXT:    lw $1, %got(countsint)($1)
+; O1-NEXT:  $BB16_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $2, 0($1)
+; O1-NEXT:    addu $3, $2, $4
+; O1-NEXT:    sc $3, 0($1)
+; O1-NEXT:    beqz $3, $BB16_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    sync
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: CheckSync:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    sync
+; O2-NEXT:    lw $1, %got(countsint)($1)
+; O2-NEXT:  $BB16_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $2, 0($1)
+; O2-NEXT:    addu $3, $2, $4
+; O2-NEXT:    sc $3, 0($1)
+; O2-NEXT:    beqz $3, $BB16_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    sync
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: CheckSync:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    sync
+; O3-NEXT:    lw $1, %got(countsint)($1)
+; O3-NEXT:  $BB16_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $2, 0($1)
+; O3-NEXT:    addu $3, $2, $4
+; O3-NEXT:    sc $3, 0($1)
+; O3-NEXT:    beqz $3, $BB16_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    sync
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS32EB-LABEL: CheckSync:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    sync
+; MIPS32EB-NEXT:    lw $1, %got(countsint)($1)
+; MIPS32EB-NEXT:  $BB16_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $2, 0($1)
+; MIPS32EB-NEXT:    addu $3, $2, $4
+; MIPS32EB-NEXT:    sc $3, 0($1)
+; MIPS32EB-NEXT:    beqz $3, $BB16_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    sync
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    nop
 entry:
   %0 = atomicrmw add i32* @countsint, i32 %v seq_cst
-  ret i32 %0 
-
-; ALL-LABEL: CheckSync:
-
-; ALL:           sync
-; ALL:           ll
-; ALL:           sc
-; ALL:           beq
-; ALL:           sync
+  ret i32 %0
 }
 
 ; make sure that this assertion in
 ; TwoAddressInstructionPass::TryInstructionTransform does not fail:
 ;
 ; line 1203: assert(TargetRegisterInfo::isVirtualRegister(regB) &&
 ;
 ; it failed when MipsDAGToDAGISel::ReplaceUsesWithZeroReg replaced an
-; operand of an atomic instruction with register $zero. 
+; operand of an atomic instruction with register $zero.
 @a = external global i32
 
 define i32 @zeroreg() nounwind {
+; MIPS32-LABEL: zeroreg:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    sync
+; MIPS32-NEXT:    addiu $2, $zero, 0
+; MIPS32-NEXT:    addiu $3, $zero, 1
+; MIPS32-NEXT:    lw $1, %got(a)($1)
+; MIPS32-NEXT:  $BB17_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $4, 0($1)
+; MIPS32-NEXT:    bne $4, $3, $BB17_3
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; MIPS32-NEXT:    move $5, $2
+; MIPS32-NEXT:    sc $5, 0($1)
+; MIPS32-NEXT:    beqz $5, $BB17_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  $BB17_3: # %entry
+; MIPS32-NEXT:    xor $1, $4, $3
+; MIPS32-NEXT:    sltiu $2, $1, 1
+; MIPS32-NEXT:    sync
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    nop
+;
+; MIPS32O0-LABEL: zeroreg:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addiu $sp, $sp, -16
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    sync
+; MIPS32O0-NEXT:    lw $2, %got(a)($2)
+; MIPS32O0-NEXT:    addiu $25, $zero, 0
+; MIPS32O0-NEXT:    addiu $1, $zero, 1
+; MIPS32O0-NEXT:    lw $3, 12($sp) # 4-byte Folded Reload
+; MIPS32O0-NEXT:    move $4, $1
+; MIPS32O0-NEXT:  $BB17_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $5, 0($2)
+; MIPS32O0-NEXT:    bne $5, $4, $BB17_3
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; MIPS32O0-NEXT:    move $6, $25
+; MIPS32O0-NEXT:    sc $6, 0($2)
+; MIPS32O0-NEXT:    beqz $6, $BB17_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  $BB17_3: # %entry
+; MIPS32O0-NEXT:    xor $1, $5, $1
+; MIPS32O0-NEXT:    sltiu $1, $1, 1
+; MIPS32O0-NEXT:    sync
+; MIPS32O0-NEXT:    addiu $2, $zero, 1
+; MIPS32O0-NEXT:    xor $2, $5, $2
+; MIPS32O0-NEXT:    sltiu $2, $2, 1
+; MIPS32O0-NEXT:    andi $2, $2, 1
+; MIPS32O0-NEXT:    sw $3, 8($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:    sw $5, 12($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
+; MIPS32O0-NEXT:    addiu $sp, $sp, 16
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: zeroreg:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    sync
+; MIPS32R2-NEXT:    addiu $2, $zero, 0
+; MIPS32R2-NEXT:    addiu $3, $zero, 1
+; MIPS32R2-NEXT:    lw $1, %got(a)($1)
+; MIPS32R2-NEXT:  $BB17_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $4, 0($1)
+; MIPS32R2-NEXT:    bne $4, $3, $BB17_3
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; MIPS32R2-NEXT:    move $5, $2
+; MIPS32R2-NEXT:    sc $5, 0($1)
+; MIPS32R2-NEXT:    beqz $5, $BB17_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  $BB17_3: # %entry
+; MIPS32R2-NEXT:    xor $1, $4, $3
+; MIPS32R2-NEXT:    sltiu $2, $1, 1
+; MIPS32R2-NEXT:    sync
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    nop
+;
+; MIPS32R6-LABEL: zeroreg:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    sync
+; MIPS32R6-NEXT:    addiu $2, $zero, 0
+; MIPS32R6-NEXT:    addiu $3, $zero, 1
+; MIPS32R6-NEXT:    lw $1, %got(a)($1)
+; MIPS32R6-NEXT:  $BB17_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $4, 0($1)
+; MIPS32R6-NEXT:    bnec $4, $3, $BB17_3
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; MIPS32R6-NEXT:    move $5, $2
+; MIPS32R6-NEXT:    sc $5, 0($1)
+; MIPS32R6-NEXT:    beqzc $5, $BB17_1
+; MIPS32R6-NEXT:  $BB17_3: # %entry
+; MIPS32R6-NEXT:    xor $1, $4, $3
+; MIPS32R6-NEXT:    sltiu $2, $1, 1
+; MIPS32R6-NEXT:    sync
+; MIPS32R6-NEXT:    jrc $ra
+;
+; MIPS32R6O0-LABEL: zeroreg:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    sync
+; MIPS32R6O0-NEXT:    lw $2, %got(a)($2)
+; MIPS32R6O0-NEXT:    addiu $25, $zero, 0
+; MIPS32R6O0-NEXT:    addiu $1, $zero, 1
+; MIPS32R6O0-NEXT:    lw $3, 4($sp) # 4-byte Folded Reload
+; MIPS32R6O0-NEXT:    move $4, $1
+; MIPS32R6O0-NEXT:  $BB17_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $5, 0($2)
+; MIPS32R6O0-NEXT:    bnec $5, $4, $BB17_3
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; MIPS32R6O0-NEXT:    move $6, $25
+; MIPS32R6O0-NEXT:    sc $6, 0($2)
+; MIPS32R6O0-NEXT:    beqzc $6, $BB17_1
+; MIPS32R6O0-NEXT:  $BB17_3: # %entry
+; MIPS32R6O0-NEXT:    xor $1, $5, $1
+; MIPS32R6O0-NEXT:    sltiu $2, $1, 1
+; MIPS32R6O0-NEXT:    sync
+; MIPS32R6O0-NEXT:    sw $3, 0($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    sw $5, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: zeroreg:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(zeroreg)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(zeroreg)))
+; MIPS4-NEXT:    sync
+; MIPS4-NEXT:    addiu $2, $zero, 0
+; MIPS4-NEXT:    addiu $3, $zero, 1
+; MIPS4-NEXT:    ld $1, %got_disp(a)($1)
+; MIPS4-NEXT:  .LBB17_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $4, 0($1)
+; MIPS4-NEXT:    bne $4, $3, .LBB17_3
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; MIPS4-NEXT:    move $5, $2
+; MIPS4-NEXT:    sc $5, 0($1)
+; MIPS4-NEXT:    beqz $5, .LBB17_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  .LBB17_3: # %entry
+; MIPS4-NEXT:    xor $1, $4, $3
+; MIPS4-NEXT:    sltiu $2, $1, 1
+; MIPS4-NEXT:    sync
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: zeroreg:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(zeroreg)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(zeroreg)))
+; MIPS64-NEXT:    sync
+; MIPS64-NEXT:    addiu $2, $zero, 0
+; MIPS64-NEXT:    addiu $3, $zero, 1
+; MIPS64-NEXT:    ld $1, %got_disp(a)($1)
+; MIPS64-NEXT:  .LBB17_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $4, 0($1)
+; MIPS64-NEXT:    bne $4, $3, .LBB17_3
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; MIPS64-NEXT:    move $5, $2
+; MIPS64-NEXT:    sc $5, 0($1)
+; MIPS64-NEXT:    beqz $5, .LBB17_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  .LBB17_3: # %entry
+; MIPS64-NEXT:    xor $1, $4, $3
+; MIPS64-NEXT:    sltiu $2, $1, 1
+; MIPS64-NEXT:    sync
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: zeroreg:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(zeroreg)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(zeroreg)))
+; MIPS64R2-NEXT:    sync
+; MIPS64R2-NEXT:    addiu $2, $zero, 0
+; MIPS64R2-NEXT:    addiu $3, $zero, 1
+; MIPS64R2-NEXT:    ld $1, %got_disp(a)($1)
+; MIPS64R2-NEXT:  .LBB17_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $4, 0($1)
+; MIPS64R2-NEXT:    bne $4, $3, .LBB17_3
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; MIPS64R2-NEXT:    move $5, $2
+; MIPS64R2-NEXT:    sc $5, 0($1)
+; MIPS64R2-NEXT:    beqz $5, .LBB17_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  .LBB17_3: # %entry
+; MIPS64R2-NEXT:    xor $1, $4, $3
+; MIPS64R2-NEXT:    sltiu $2, $1, 1
+; MIPS64R2-NEXT:    sync
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: zeroreg:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(zeroreg)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(zeroreg)))
+; MIPS64R6-NEXT:    sync
+; MIPS64R6-NEXT:    addiu $2, $zero, 0
+; MIPS64R6-NEXT:    addiu $3, $zero, 1
+; MIPS64R6-NEXT:    ld $1, %got_disp(a)($1)
+; MIPS64R6-NEXT:  .LBB17_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $4, 0($1)
+; MIPS64R6-NEXT:    bnec $4, $3, .LBB17_3
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; MIPS64R6-NEXT:    move $5, $2
+; MIPS64R6-NEXT:    sc $5, 0($1)
+; MIPS64R6-NEXT:    beqzc $5, .LBB17_1
+; MIPS64R6-NEXT:  .LBB17_3: # %entry
+; MIPS64R6-NEXT:    xor $1, $4, $3
+; MIPS64R6-NEXT:    sltiu $2, $1, 1
+; MIPS64R6-NEXT:    sync
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: zeroreg:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(zeroreg)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(zeroreg)))
+; MIPS64R6O0-NEXT:    sync
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(a)($1)
+; MIPS64R6O0-NEXT:    addiu $2, $zero, 0
+; MIPS64R6O0-NEXT:    addiu $3, $zero, 1
+; MIPS64R6O0-NEXT:    lw $4, 12($sp) # 4-byte Folded Reload
+; MIPS64R6O0-NEXT:    move $5, $3
+; MIPS64R6O0-NEXT:  .LBB17_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $6, 0($1)
+; MIPS64R6O0-NEXT:    bnec $6, $5, .LBB17_3
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; MIPS64R6O0-NEXT:    move $7, $2
+; MIPS64R6O0-NEXT:    sc $7, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $7, .LBB17_1
+; MIPS64R6O0-NEXT:  .LBB17_3: # %entry
+; MIPS64R6O0-NEXT:    xor $2, $6, $3
+; MIPS64R6O0-NEXT:    sltiu $2, $2, 1
+; MIPS64R6O0-NEXT:    sync
+; MIPS64R6O0-NEXT:    sw $4, 8($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:    sw $6, 12($sp) # 4-byte Folded Spill
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: zeroreg:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    sync
+; MM32-NEXT:    li16 $3, 0
+; MM32-NEXT:    li16 $4, 1
+; MM32-NEXT:    lw $1, %got(a)($2)
+; MM32-NEXT:  $BB17_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $2, 0($1)
+; MM32-NEXT:    bne $2, $4, $BB17_3
+; MM32-NEXT:    nop
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; MM32-NEXT:    move $5, $3
+; MM32-NEXT:    sc $5, 0($1)
+; MM32-NEXT:    beqzc $5, $BB17_1
+; MM32-NEXT:  $BB17_3: # %entry
+; MM32-NEXT:    xor $1, $2, $4
+; MM32-NEXT:    sltiu $2, $1, 1
+; MM32-NEXT:    sync
+; MM32-NEXT:    jrc $ra
+;
+; O1-LABEL: zeroreg:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    sync
+; O1-NEXT:    addiu $2, $zero, 0
+; O1-NEXT:    addiu $3, $zero, 1
+; O1-NEXT:    lw $1, %got(a)($1)
+; O1-NEXT:  $BB17_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $4, 0($1)
+; O1-NEXT:    bne $4, $3, $BB17_3
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; O1-NEXT:    move $5, $2
+; O1-NEXT:    sc $5, 0($1)
+; O1-NEXT:    beqz $5, $BB17_1
+; O1-NEXT:    nop
+; O1-NEXT:  $BB17_3: # %entry
+; O1-NEXT:    xor $1, $4, $3
+; O1-NEXT:    sltiu $2, $1, 1
+; O1-NEXT:    sync
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: zeroreg:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    sync
+; O2-NEXT:    addiu $2, $zero, 0
+; O2-NEXT:    addiu $3, $zero, 1
+; O2-NEXT:    lw $1, %got(a)($1)
+; O2-NEXT:  $BB17_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $4, 0($1)
+; O2-NEXT:    bne $4, $3, $BB17_3
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; O2-NEXT:    move $5, $2
+; O2-NEXT:    sc $5, 0($1)
+; O2-NEXT:    beqz $5, $BB17_1
+; O2-NEXT:    nop
+; O2-NEXT:  $BB17_3: # %entry
+; O2-NEXT:    xor $1, $4, $3
+; O2-NEXT:    sltiu $2, $1, 1
+; O2-NEXT:    sync
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: zeroreg:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    addiu $2, $zero, 0
+; O3-NEXT:    addiu $3, $zero, 1
+; O3-NEXT:    sync
+; O3-NEXT:    lw $1, %got(a)($1)
+; O3-NEXT:  $BB17_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $4, 0($1)
+; O3-NEXT:    bne $4, $3, $BB17_3
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; O3-NEXT:    move $5, $2
+; O3-NEXT:    sc $5, 0($1)
+; O3-NEXT:    beqz $5, $BB17_1
+; O3-NEXT:    nop
+; O3-NEXT:  $BB17_3: # %entry
+; O3-NEXT:    sync
+; O3-NEXT:    xor $1, $4, $3
+; O3-NEXT:    jr $ra
+; O3-NEXT:    sltiu $2, $1, 1
+;
+; MIPS32EB-LABEL: zeroreg:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    sync
+; MIPS32EB-NEXT:    addiu $2, $zero, 0
+; MIPS32EB-NEXT:    addiu $3, $zero, 1
+; MIPS32EB-NEXT:    lw $1, %got(a)($1)
+; MIPS32EB-NEXT:  $BB17_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $4, 0($1)
+; MIPS32EB-NEXT:    bne $4, $3, $BB17_3
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    # in Loop: Header=BB17_1 Depth=1
+; MIPS32EB-NEXT:    move $5, $2
+; MIPS32EB-NEXT:    sc $5, 0($1)
+; MIPS32EB-NEXT:    beqz $5, $BB17_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  $BB17_3: # %entry
+; MIPS32EB-NEXT:    xor $1, $4, $3
+; MIPS32EB-NEXT:    sltiu $2, $1, 1
+; MIPS32EB-NEXT:    sync
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    nop
 entry:
   %pair0 = cmpxchg i32* @a, i32 1, i32 0 seq_cst seq_cst
   %0 = extractvalue { i32, i1 } %pair0, 0
   %1 = icmp eq i32 %0, 1
   %conv = zext i1 %1 to i32
   ret i32 %conv
 }
 
 ; Check that MIPS32R6 has the correct offset range.
 ; FIXME: At the moment, we don't seem to do addr+offset for any atomic load/store.
 define i32 @AtomicLoadAdd32_OffGt9Bit(i32 signext %incr) nounwind {
+; MIPS32-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; MIPS32:       # %bb.0: # %entry
+; MIPS32-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32-NEXT:    addu $1, $2, $25
+; MIPS32-NEXT:    lw $1, %got(x)($1)
+; MIPS32-NEXT:    addiu $1, $1, 1024
+; MIPS32-NEXT:  $BB18_1: # %entry
+; MIPS32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32-NEXT:    ll $2, 0($1)
+; MIPS32-NEXT:    addu $3, $2, $4
+; MIPS32-NEXT:    sc $3, 0($1)
+; MIPS32-NEXT:    beqz $3, $BB18_1
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  # %bb.2: # %entry
+; MIPS32-NEXT:    jr $ra
+; MIPS32-NEXT:    nop
+;
+; MIPS32O0-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; MIPS32O0:       # %bb.0: # %entry
+; MIPS32O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32O0-NEXT:    addu $2, $2, $25
+; MIPS32O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32O0-NEXT:    addiu $2, $2, 1024
+; MIPS32O0-NEXT:  $BB18_1: # %entry
+; MIPS32O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32O0-NEXT:    ll $25, 0($2)
+; MIPS32O0-NEXT:    addu $1, $25, $4
+; MIPS32O0-NEXT:    sc $1, 0($2)
+; MIPS32O0-NEXT:    beqz $1, $BB18_1
+; MIPS32O0-NEXT:    nop
+; MIPS32O0-NEXT:  # %bb.2: # %entry
+; MIPS32O0-NEXT:    move $2, $25
+; MIPS32O0-NEXT:    jr $ra
+; MIPS32O0-NEXT:    nop
+;
+; MIPS32R2-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; MIPS32R2:       # %bb.0: # %entry
+; MIPS32R2-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R2-NEXT:    addu $1, $2, $25
+; MIPS32R2-NEXT:    lw $1, %got(x)($1)
+; MIPS32R2-NEXT:    addiu $1, $1, 1024
+; MIPS32R2-NEXT:  $BB18_1: # %entry
+; MIPS32R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R2-NEXT:    ll $2, 0($1)
+; MIPS32R2-NEXT:    addu $3, $2, $4
+; MIPS32R2-NEXT:    sc $3, 0($1)
+; MIPS32R2-NEXT:    beqz $3, $BB18_1
+; MIPS32R2-NEXT:    nop
+; MIPS32R2-NEXT:  # %bb.2: # %entry
+; MIPS32R2-NEXT:    jr $ra
+; MIPS32R2-NEXT:    nop
+;
+; MIPS32R6-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; MIPS32R6:       # %bb.0: # %entry
+; MIPS32R6-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6-NEXT:    addu $1, $2, $25
+; MIPS32R6-NEXT:    lw $1, %got(x)($1)
+; MIPS32R6-NEXT:    addiu $1, $1, 1024
+; MIPS32R6-NEXT:  $BB18_1: # %entry
+; MIPS32R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6-NEXT:    ll $2, 0($1)
+; MIPS32R6-NEXT:    addu $3, $2, $4
+; MIPS32R6-NEXT:    sc $3, 0($1)
+; MIPS32R6-NEXT:    beqzc $3, $BB18_1
+; MIPS32R6-NEXT:    nop
+; MIPS32R6-NEXT:  # %bb.2: # %entry
+; MIPS32R6-NEXT:    jrc $ra
+;
+; MIPS32R6O0-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; MIPS32R6O0:       # %bb.0: # %entry
+; MIPS32R6O0-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, -8
+; MIPS32R6O0-NEXT:    addu $2, $2, $25
+; MIPS32R6O0-NEXT:    move $25, $4
+; MIPS32R6O0-NEXT:    lw $2, %got(x)($2)
+; MIPS32R6O0-NEXT:    addiu $2, $2, 1024
+; MIPS32R6O0-NEXT:  $BB18_1: # %entry
+; MIPS32R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32R6O0-NEXT:    ll $1, 0($2)
+; MIPS32R6O0-NEXT:    addu $3, $1, $4
+; MIPS32R6O0-NEXT:    sc $3, 0($2)
+; MIPS32R6O0-NEXT:    beqzc $3, $BB18_1
+; MIPS32R6O0-NEXT:  # %bb.2: # %entry
+; MIPS32R6O0-NEXT:    move $2, $1
+; MIPS32R6O0-NEXT:    sw $25, 4($sp) # 4-byte Folded Spill
+; MIPS32R6O0-NEXT:    addiu $sp, $sp, 8
+; MIPS32R6O0-NEXT:    jrc $ra
+;
+; MIPS4-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd32_OffGt9Bit)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd32_OffGt9Bit)))
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:    daddiu $1, $1, 1024
+; MIPS4-NEXT:  .LBB18_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    ll $2, 0($1)
+; MIPS4-NEXT:    addu $3, $2, $4
+; MIPS4-NEXT:    sc $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB18_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd32_OffGt9Bit)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd32_OffGt9Bit)))
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:    daddiu $1, $1, 1024
+; MIPS64-NEXT:  .LBB18_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    ll $2, 0($1)
+; MIPS64-NEXT:    addu $3, $2, $4
+; MIPS64-NEXT:    sc $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB18_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd32_OffGt9Bit)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd32_OffGt9Bit)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:    daddiu $1, $1, 1024
+; MIPS64R2-NEXT:  .LBB18_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    ll $2, 0($1)
+; MIPS64R2-NEXT:    addu $3, $2, $4
+; MIPS64R2-NEXT:    sc $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB18_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd32_OffGt9Bit)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd32_OffGt9Bit)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:    daddiu $1, $1, 1024
+; MIPS64R6-NEXT:  .LBB18_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    ll $2, 0($1)
+; MIPS64R6-NEXT:    addu $3, $2, $4
+; MIPS64R6-NEXT:    sc $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB18_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd32_OffGt9Bit)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd32_OffGt9Bit)))
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:    daddiu $1, $1, 1024
+; MIPS64R6O0-NEXT:  .LBB18_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    ll $3, 0($1)
+; MIPS64R6O0-NEXT:    addu $5, $3, $2
+; MIPS64R6O0-NEXT:    sc $5, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $5, .LBB18_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    move $2, $3
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; MM32-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; MM32:       # %bb.0: # %entry
+; MM32-NEXT:    lui $2, %hi(_gp_disp)
+; MM32-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MM32-NEXT:    addu $2, $2, $25
+; MM32-NEXT:    lw $1, %got(x)($2)
+; MM32-NEXT:    addiu $1, $1, 1024
+; MM32-NEXT:  $BB18_1: # %entry
+; MM32-NEXT:    # =>This Inner Loop Header: Depth=1
+; MM32-NEXT:    ll $2, 0($1)
+; MM32-NEXT:    addu16 $3, $2, $4
+; MM32-NEXT:    sc $3, 0($1)
+; MM32-NEXT:    beqzc $3, $BB18_1
+; MM32-NEXT:  # %bb.2: # %entry
+; MM32-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $2, %hi(_gp_disp)
+; O1-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O1-NEXT:    addu $1, $2, $25
+; O1-NEXT:    lw $1, %got(x)($1)
+; O1-NEXT:    addiu $1, $1, 1024
+; O1-NEXT:  $BB18_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    ll $2, 0($1)
+; O1-NEXT:    addu $3, $2, $4
+; O1-NEXT:    sc $3, 0($1)
+; O1-NEXT:    beqz $3, $BB18_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $2, %hi(_gp_disp)
+; O2-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O2-NEXT:    addu $1, $2, $25
+; O2-NEXT:    lw $1, %got(x)($1)
+; O2-NEXT:    addiu $1, $1, 1024
+; O2-NEXT:  $BB18_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    ll $2, 0($1)
+; O2-NEXT:    addu $3, $2, $4
+; O2-NEXT:    sc $3, 0($1)
+; O2-NEXT:    beqz $3, $BB18_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $2, %hi(_gp_disp)
+; O3-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; O3-NEXT:    addu $1, $2, $25
+; O3-NEXT:    lw $1, %got(x)($1)
+; O3-NEXT:    addiu $1, $1, 1024
+; O3-NEXT:  $BB18_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    ll $2, 0($1)
+; O3-NEXT:    addu $3, $2, $4
+; O3-NEXT:    sc $3, 0($1)
+; O3-NEXT:    beqz $3, $BB18_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS32EB-LABEL: AtomicLoadAdd32_OffGt9Bit:
+; MIPS32EB:       # %bb.0: # %entry
+; MIPS32EB-NEXT:    lui $2, %hi(_gp_disp)
+; MIPS32EB-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; MIPS32EB-NEXT:    addu $1, $2, $25
+; MIPS32EB-NEXT:    lw $1, %got(x)($1)
+; MIPS32EB-NEXT:    addiu $1, $1, 1024
+; MIPS32EB-NEXT:  $BB18_1: # %entry
+; MIPS32EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS32EB-NEXT:    ll $2, 0($1)
+; MIPS32EB-NEXT:    addu $3, $2, $4
+; MIPS32EB-NEXT:    sc $3, 0($1)
+; MIPS32EB-NEXT:    beqz $3, $BB18_1
+; MIPS32EB-NEXT:    nop
+; MIPS32EB-NEXT:  # %bb.2: # %entry
+; MIPS32EB-NEXT:    jr $ra
+; MIPS32EB-NEXT:    nop
 entry:
   %0 = atomicrmw add i32* getelementptr(i32, i32* @x, i32 256), i32 %incr monotonic
   ret i32 %0
 
-; ALL-LABEL: AtomicLoadAdd32_OffGt9Bit:
-
-; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(x)
-; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(x)(
-
-; ALL:           addiu   $[[PTR:[0-9]+]], $[[R0]], 1024
-; ALL:       [[BB0:(\$|\.L)[A-Z_0-9]+]]:
-; ALL:           ll      $[[R1:[0-9]+]], 0($[[PTR]])
-; ALL:           addu    $[[R2:[0-9]+]], $[[R1]], $4
-; ALL:           sc      $[[R2]], 0($[[PTR]])
-; NOT-MICROMIPS: beqz    $[[R2]], [[BB0]]
-; MICROMIPS:     beqzc   $[[R2]], [[BB0]]
-; MIPSR6:        beqzc   $[[R2]], [[BB0]]
 }

llvm/trunk/test/CodeGen/Mips/atomic64.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=mips64el-unknown-linux-gnu --disable-machine-licm -mcpu=mips4 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS4
+; RUN: llc -mtriple=mips64el-unknown-linux-gnu --disable-machine-licm -mcpu=mips64 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS64
+; RUN: llc -mtriple=mips64el-unknown-linux-gnu --disable-machine-licm -mcpu=mips64r2 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS64R2
+; RUN: llc -mtriple=mips64el-unknown-linux-gnu --disable-machine-licm -mcpu=mips64r6 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS64R6
+; RUN: llc -mtriple=mips64-unknown-linux-gnu -O0 -mcpu=mips64r6 -relocation-model=pic -verify-machineinstrs -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS64R6O0
+
+; We want to verify the produced code is well formed all optimization levels, the rest of the test which ensure correctness.
+; RUN: llc -mtriple=mips64el-unknown-linux-gnu -O1 --disable-machine-licm -mcpu=mips64 -relocation-model=pic -verify-machineinstrs < %s | FileCheck %s --check-prefix=O1
+; RUN: llc -mtriple=mips64el-unknown-linux-gnu -O2 --disable-machine-licm -mcpu=mips64 -relocation-model=pic -verify-machineinstrs < %s | FileCheck %s --check-prefix=O2
+; RUN: llc -mtriple=mips64el-unknown-linux-gnu -O3 --disable-machine-licm -mcpu=mips64 -relocation-model=pic -verify-machineinstrs < %s | FileCheck %s --check-prefix=O3
+
+; Keep one big-endian check so that we don't reduce testing, but don't add more
+; since endianness doesn't affect the body of the atomic operations.
+; RUN: llc -mtriple=mips64-unknown-linux-gnu --disable-machine-licm -mcpu=mips64 -relocation-model=pic -verify-machineinstrs < %s | \
+; RUN:   FileCheck %s -check-prefix=MIPS64EB
+
+@x = common global i64 0, align 4
+
+define i64 @AtomicLoadAdd(i64 signext %incr) nounwind {
+; MIPS4-LABEL: AtomicLoadAdd:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd)))
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB0_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    lld $2, 0($1)
+; MIPS4-NEXT:    daddu $3, $2, $4
+; MIPS4-NEXT:    scd $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB0_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: AtomicLoadAdd:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd)))
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB0_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    lld $2, 0($1)
+; MIPS64-NEXT:    daddu $3, $2, $4
+; MIPS64-NEXT:    scd $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB0_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: AtomicLoadAdd:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB0_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    lld $2, 0($1)
+; MIPS64R2-NEXT:    daddu $3, $2, $4
+; MIPS64R2-NEXT:    scd $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB0_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicLoadAdd:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB0_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    lld $2, 0($1)
+; MIPS64R6-NEXT:    daddu $3, $2, $4
+; MIPS64R6-NEXT:    scd $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB0_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicLoadAdd:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd)))
+; MIPS64R6O0-NEXT:    move $25, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB0_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    lld $2, 0($1)
+; MIPS64R6O0-NEXT:    daddu $3, $2, $4
+; MIPS64R6O0-NEXT:    scd $3, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $3, .LBB0_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    sd $25, 8($sp) # 8-byte Folded Spill
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicLoadAdd:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd)))
+; O1-NEXT:    daddu $1, $1, $25
+; O1-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd)))
+; O1-NEXT:    ld $1, %got_disp(x)($1)
+; O1-NEXT:  .LBB0_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    lld $2, 0($1)
+; O1-NEXT:    daddu $3, $2, $4
+; O1-NEXT:    scd $3, 0($1)
+; O1-NEXT:    beqz $3, .LBB0_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: AtomicLoadAdd:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd)))
+; O2-NEXT:    daddu $1, $1, $25
+; O2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd)))
+; O2-NEXT:    ld $1, %got_disp(x)($1)
+; O2-NEXT:  .LBB0_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    lld $2, 0($1)
+; O2-NEXT:    daddu $3, $2, $4
+; O2-NEXT:    scd $3, 0($1)
+; O2-NEXT:    beqz $3, .LBB0_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: AtomicLoadAdd:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd)))
+; O3-NEXT:    daddu $1, $1, $25
+; O3-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd)))
+; O3-NEXT:    ld $1, %got_disp(x)($1)
+; O3-NEXT:  .LBB0_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    lld $2, 0($1)
+; O3-NEXT:    daddu $3, $2, $4
+; O3-NEXT:    scd $3, 0($1)
+; O3-NEXT:    beqz $3, .LBB0_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS64EB-LABEL: AtomicLoadAdd:
+; MIPS64EB:       # %bb.0: # %entry
+; MIPS64EB-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAdd)))
+; MIPS64EB-NEXT:    daddu $1, $1, $25
+; MIPS64EB-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAdd)))
+; MIPS64EB-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64EB-NEXT:  .LBB0_1: # %entry
+; MIPS64EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64EB-NEXT:    lld $2, 0($1)
+; MIPS64EB-NEXT:    daddu $3, $2, $4
+; MIPS64EB-NEXT:    scd $3, 0($1)
+; MIPS64EB-NEXT:    beqz $3, .LBB0_1
+; MIPS64EB-NEXT:    nop
+; MIPS64EB-NEXT:  # %bb.2: # %entry
+; MIPS64EB-NEXT:    jr $ra
+; MIPS64EB-NEXT:    nop
+entry:
+  %0 = atomicrmw add i64* @x, i64 %incr monotonic
+  ret i64 %0
+
+}
+
+define i64 @AtomicLoadSub(i64 signext %incr) nounwind {
+; MIPS4-LABEL: AtomicLoadSub:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub)))
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB1_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    lld $2, 0($1)
+; MIPS4-NEXT:    dsubu $3, $2, $4
+; MIPS4-NEXT:    scd $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB1_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: AtomicLoadSub:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub)))
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB1_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    lld $2, 0($1)
+; MIPS64-NEXT:    dsubu $3, $2, $4
+; MIPS64-NEXT:    scd $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB1_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: AtomicLoadSub:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB1_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    lld $2, 0($1)
+; MIPS64R2-NEXT:    dsubu $3, $2, $4
+; MIPS64R2-NEXT:    scd $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB1_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicLoadSub:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB1_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    lld $2, 0($1)
+; MIPS64R6-NEXT:    dsubu $3, $2, $4
+; MIPS64R6-NEXT:    scd $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB1_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicLoadSub:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub)))
+; MIPS64R6O0-NEXT:    move $25, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB1_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    lld $2, 0($1)
+; MIPS64R6O0-NEXT:    dsubu $3, $2, $4
+; MIPS64R6O0-NEXT:    scd $3, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $3, .LBB1_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    sd $25, 8($sp) # 8-byte Folded Spill
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicLoadSub:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub)))
+; O1-NEXT:    daddu $1, $1, $25
+; O1-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub)))
+; O1-NEXT:    ld $1, %got_disp(x)($1)
+; O1-NEXT:  .LBB1_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    lld $2, 0($1)
+; O1-NEXT:    dsubu $3, $2, $4
+; O1-NEXT:    scd $3, 0($1)
+; O1-NEXT:    beqz $3, .LBB1_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: AtomicLoadSub:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub)))
+; O2-NEXT:    daddu $1, $1, $25
+; O2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub)))
+; O2-NEXT:    ld $1, %got_disp(x)($1)
+; O2-NEXT:  .LBB1_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    lld $2, 0($1)
+; O2-NEXT:    dsubu $3, $2, $4
+; O2-NEXT:    scd $3, 0($1)
+; O2-NEXT:    beqz $3, .LBB1_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: AtomicLoadSub:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub)))
+; O3-NEXT:    daddu $1, $1, $25
+; O3-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub)))
+; O3-NEXT:    ld $1, %got_disp(x)($1)
+; O3-NEXT:  .LBB1_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    lld $2, 0($1)
+; O3-NEXT:    dsubu $3, $2, $4
+; O3-NEXT:    scd $3, 0($1)
+; O3-NEXT:    beqz $3, .LBB1_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS64EB-LABEL: AtomicLoadSub:
+; MIPS64EB:       # %bb.0: # %entry
+; MIPS64EB-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadSub)))
+; MIPS64EB-NEXT:    daddu $1, $1, $25
+; MIPS64EB-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadSub)))
+; MIPS64EB-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64EB-NEXT:  .LBB1_1: # %entry
+; MIPS64EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64EB-NEXT:    lld $2, 0($1)
+; MIPS64EB-NEXT:    dsubu $3, $2, $4
+; MIPS64EB-NEXT:    scd $3, 0($1)
+; MIPS64EB-NEXT:    beqz $3, .LBB1_1
+; MIPS64EB-NEXT:    nop
+; MIPS64EB-NEXT:  # %bb.2: # %entry
+; MIPS64EB-NEXT:    jr $ra
+; MIPS64EB-NEXT:    nop
+entry:
+  %0 = atomicrmw sub i64* @x, i64 %incr monotonic
+  ret i64 %0
+
+}
+
+define i64 @AtomicLoadAnd(i64 signext %incr) nounwind {
+; MIPS4-LABEL: AtomicLoadAnd:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd)))
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB2_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    lld $2, 0($1)
+; MIPS4-NEXT:    and $3, $2, $4
+; MIPS4-NEXT:    scd $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB2_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: AtomicLoadAnd:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd)))
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB2_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    lld $2, 0($1)
+; MIPS64-NEXT:    and $3, $2, $4
+; MIPS64-NEXT:    scd $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB2_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: AtomicLoadAnd:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB2_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    lld $2, 0($1)
+; MIPS64R2-NEXT:    and $3, $2, $4
+; MIPS64R2-NEXT:    scd $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB2_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicLoadAnd:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB2_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    lld $2, 0($1)
+; MIPS64R6-NEXT:    and $3, $2, $4
+; MIPS64R6-NEXT:    scd $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB2_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicLoadAnd:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd)))
+; MIPS64R6O0-NEXT:    move $25, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB2_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    lld $2, 0($1)
+; MIPS64R6O0-NEXT:    and $3, $2, $4
+; MIPS64R6O0-NEXT:    scd $3, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $3, .LBB2_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    sd $25, 8($sp) # 8-byte Folded Spill
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicLoadAnd:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd)))
+; O1-NEXT:    daddu $1, $1, $25
+; O1-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd)))
+; O1-NEXT:    ld $1, %got_disp(x)($1)
+; O1-NEXT:  .LBB2_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    lld $2, 0($1)
+; O1-NEXT:    and $3, $2, $4
+; O1-NEXT:    scd $3, 0($1)
+; O1-NEXT:    beqz $3, .LBB2_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: AtomicLoadAnd:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd)))
+; O2-NEXT:    daddu $1, $1, $25
+; O2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd)))
+; O2-NEXT:    ld $1, %got_disp(x)($1)
+; O2-NEXT:  .LBB2_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    lld $2, 0($1)
+; O2-NEXT:    and $3, $2, $4
+; O2-NEXT:    scd $3, 0($1)
+; O2-NEXT:    beqz $3, .LBB2_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: AtomicLoadAnd:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd)))
+; O3-NEXT:    daddu $1, $1, $25
+; O3-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd)))
+; O3-NEXT:    ld $1, %got_disp(x)($1)
+; O3-NEXT:  .LBB2_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    lld $2, 0($1)
+; O3-NEXT:    and $3, $2, $4
+; O3-NEXT:    scd $3, 0($1)
+; O3-NEXT:    beqz $3, .LBB2_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS64EB-LABEL: AtomicLoadAnd:
+; MIPS64EB:       # %bb.0: # %entry
+; MIPS64EB-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadAnd)))
+; MIPS64EB-NEXT:    daddu $1, $1, $25
+; MIPS64EB-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadAnd)))
+; MIPS64EB-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64EB-NEXT:  .LBB2_1: # %entry
+; MIPS64EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64EB-NEXT:    lld $2, 0($1)
+; MIPS64EB-NEXT:    and $3, $2, $4
+; MIPS64EB-NEXT:    scd $3, 0($1)
+; MIPS64EB-NEXT:    beqz $3, .LBB2_1
+; MIPS64EB-NEXT:    nop
+; MIPS64EB-NEXT:  # %bb.2: # %entry
+; MIPS64EB-NEXT:    jr $ra
+; MIPS64EB-NEXT:    nop
+entry:
+  %0 = atomicrmw and i64* @x, i64 %incr monotonic
+  ret i64 %0
+
+}
+
+define i64 @AtomicLoadOr(i64 signext %incr) nounwind {
+; MIPS4-LABEL: AtomicLoadOr:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr)))
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB3_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    lld $2, 0($1)
+; MIPS4-NEXT:    or $3, $2, $4
+; MIPS4-NEXT:    scd $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB3_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: AtomicLoadOr:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr)))
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB3_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    lld $2, 0($1)
+; MIPS64-NEXT:    or $3, $2, $4
+; MIPS64-NEXT:    scd $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB3_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: AtomicLoadOr:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB3_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    lld $2, 0($1)
+; MIPS64R2-NEXT:    or $3, $2, $4
+; MIPS64R2-NEXT:    scd $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB3_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicLoadOr:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB3_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    lld $2, 0($1)
+; MIPS64R6-NEXT:    or $3, $2, $4
+; MIPS64R6-NEXT:    scd $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB3_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicLoadOr:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr)))
+; MIPS64R6O0-NEXT:    move $25, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB3_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    lld $2, 0($1)
+; MIPS64R6O0-NEXT:    or $3, $2, $4
+; MIPS64R6O0-NEXT:    scd $3, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $3, .LBB3_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    sd $25, 8($sp) # 8-byte Folded Spill
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicLoadOr:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr)))
+; O1-NEXT:    daddu $1, $1, $25
+; O1-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr)))
+; O1-NEXT:    ld $1, %got_disp(x)($1)
+; O1-NEXT:  .LBB3_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    lld $2, 0($1)
+; O1-NEXT:    or $3, $2, $4
+; O1-NEXT:    scd $3, 0($1)
+; O1-NEXT:    beqz $3, .LBB3_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: AtomicLoadOr:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr)))
+; O2-NEXT:    daddu $1, $1, $25
+; O2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr)))
+; O2-NEXT:    ld $1, %got_disp(x)($1)
+; O2-NEXT:  .LBB3_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    lld $2, 0($1)
+; O2-NEXT:    or $3, $2, $4
+; O2-NEXT:    scd $3, 0($1)
+; O2-NEXT:    beqz $3, .LBB3_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: AtomicLoadOr:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr)))
+; O3-NEXT:    daddu $1, $1, $25
+; O3-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr)))
+; O3-NEXT:    ld $1, %got_disp(x)($1)
+; O3-NEXT:  .LBB3_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    lld $2, 0($1)
+; O3-NEXT:    or $3, $2, $4
+; O3-NEXT:    scd $3, 0($1)
+; O3-NEXT:    beqz $3, .LBB3_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS64EB-LABEL: AtomicLoadOr:
+; MIPS64EB:       # %bb.0: # %entry
+; MIPS64EB-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadOr)))
+; MIPS64EB-NEXT:    daddu $1, $1, $25
+; MIPS64EB-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadOr)))
+; MIPS64EB-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64EB-NEXT:  .LBB3_1: # %entry
+; MIPS64EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64EB-NEXT:    lld $2, 0($1)
+; MIPS64EB-NEXT:    or $3, $2, $4
+; MIPS64EB-NEXT:    scd $3, 0($1)
+; MIPS64EB-NEXT:    beqz $3, .LBB3_1
+; MIPS64EB-NEXT:    nop
+; MIPS64EB-NEXT:  # %bb.2: # %entry
+; MIPS64EB-NEXT:    jr $ra
+; MIPS64EB-NEXT:    nop
+entry:
+  %0 = atomicrmw or i64* @x, i64 %incr monotonic
+  ret i64 %0
+
+}
+
+define i64 @AtomicLoadXor(i64 signext %incr) nounwind {
+; MIPS4-LABEL: AtomicLoadXor:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor)))
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB4_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    lld $2, 0($1)
+; MIPS4-NEXT:    xor $3, $2, $4
+; MIPS4-NEXT:    scd $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB4_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: AtomicLoadXor:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor)))
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB4_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    lld $2, 0($1)
+; MIPS64-NEXT:    xor $3, $2, $4
+; MIPS64-NEXT:    scd $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB4_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: AtomicLoadXor:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB4_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    lld $2, 0($1)
+; MIPS64R2-NEXT:    xor $3, $2, $4
+; MIPS64R2-NEXT:    scd $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB4_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicLoadXor:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB4_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    lld $2, 0($1)
+; MIPS64R6-NEXT:    xor $3, $2, $4
+; MIPS64R6-NEXT:    scd $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB4_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicLoadXor:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor)))
+; MIPS64R6O0-NEXT:    move $25, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB4_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    lld $2, 0($1)
+; MIPS64R6O0-NEXT:    xor $3, $2, $4
+; MIPS64R6O0-NEXT:    scd $3, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $3, .LBB4_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    sd $25, 8($sp) # 8-byte Folded Spill
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicLoadXor:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor)))
+; O1-NEXT:    daddu $1, $1, $25
+; O1-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor)))
+; O1-NEXT:    ld $1, %got_disp(x)($1)
+; O1-NEXT:  .LBB4_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    lld $2, 0($1)
+; O1-NEXT:    xor $3, $2, $4
+; O1-NEXT:    scd $3, 0($1)
+; O1-NEXT:    beqz $3, .LBB4_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: AtomicLoadXor:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor)))
+; O2-NEXT:    daddu $1, $1, $25
+; O2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor)))
+; O2-NEXT:    ld $1, %got_disp(x)($1)
+; O2-NEXT:  .LBB4_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    lld $2, 0($1)
+; O2-NEXT:    xor $3, $2, $4
+; O2-NEXT:    scd $3, 0($1)
+; O2-NEXT:    beqz $3, .LBB4_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: AtomicLoadXor:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor)))
+; O3-NEXT:    daddu $1, $1, $25
+; O3-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor)))
+; O3-NEXT:    ld $1, %got_disp(x)($1)
+; O3-NEXT:  .LBB4_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    lld $2, 0($1)
+; O3-NEXT:    xor $3, $2, $4
+; O3-NEXT:    scd $3, 0($1)
+; O3-NEXT:    beqz $3, .LBB4_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS64EB-LABEL: AtomicLoadXor:
+; MIPS64EB:       # %bb.0: # %entry
+; MIPS64EB-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadXor)))
+; MIPS64EB-NEXT:    daddu $1, $1, $25
+; MIPS64EB-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadXor)))
+; MIPS64EB-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64EB-NEXT:  .LBB4_1: # %entry
+; MIPS64EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64EB-NEXT:    lld $2, 0($1)
+; MIPS64EB-NEXT:    xor $3, $2, $4
+; MIPS64EB-NEXT:    scd $3, 0($1)
+; MIPS64EB-NEXT:    beqz $3, .LBB4_1
+; MIPS64EB-NEXT:    nop
+; MIPS64EB-NEXT:  # %bb.2: # %entry
+; MIPS64EB-NEXT:    jr $ra
+; MIPS64EB-NEXT:    nop
+entry:
+  %0 = atomicrmw xor i64* @x, i64 %incr monotonic
+  ret i64 %0
+
+}
+
+define i64 @AtomicLoadNand(i64 signext %incr) nounwind {
+; MIPS4-LABEL: AtomicLoadNand:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand)))
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB5_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    lld $2, 0($1)
+; MIPS4-NEXT:    and $3, $2, $4
+; MIPS4-NEXT:    nor $3, $zero, $3
+; MIPS4-NEXT:    scd $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB5_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    nop
+;
+; MIPS64-LABEL: AtomicLoadNand:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand)))
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB5_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    lld $2, 0($1)
+; MIPS64-NEXT:    and $3, $2, $4
+; MIPS64-NEXT:    nor $3, $zero, $3
+; MIPS64-NEXT:    scd $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB5_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    nop
+;
+; MIPS64R2-LABEL: AtomicLoadNand:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand)))
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB5_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    lld $2, 0($1)
+; MIPS64R2-NEXT:    and $3, $2, $4
+; MIPS64R2-NEXT:    nor $3, $zero, $3
+; MIPS64R2-NEXT:    scd $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB5_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    nop
+;
+; MIPS64R6-LABEL: AtomicLoadNand:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand)))
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB5_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    lld $2, 0($1)
+; MIPS64R6-NEXT:    and $3, $2, $4
+; MIPS64R6-NEXT:    nor $3, $zero, $3
+; MIPS64R6-NEXT:    scd $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB5_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jrc $ra
+;
+; MIPS64R6O0-LABEL: AtomicLoadNand:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand)))
+; MIPS64R6O0-NEXT:    move $25, $4
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB5_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    lld $2, 0($1)
+; MIPS64R6O0-NEXT:    and $3, $2, $4
+; MIPS64R6O0-NEXT:    nor $3, $zero, $3
+; MIPS64R6O0-NEXT:    scd $3, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $3, .LBB5_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    sd $25, 8($sp) # 8-byte Folded Spill
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicLoadNand:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand)))
+; O1-NEXT:    daddu $1, $1, $25
+; O1-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand)))
+; O1-NEXT:    ld $1, %got_disp(x)($1)
+; O1-NEXT:  .LBB5_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    lld $2, 0($1)
+; O1-NEXT:    and $3, $2, $4
+; O1-NEXT:    nor $3, $zero, $3
+; O1-NEXT:    scd $3, 0($1)
+; O1-NEXT:    beqz $3, .LBB5_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    nop
+;
+; O2-LABEL: AtomicLoadNand:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand)))
+; O2-NEXT:    daddu $1, $1, $25
+; O2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand)))
+; O2-NEXT:    ld $1, %got_disp(x)($1)
+; O2-NEXT:  .LBB5_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    lld $2, 0($1)
+; O2-NEXT:    and $3, $2, $4
+; O2-NEXT:    nor $3, $zero, $3
+; O2-NEXT:    scd $3, 0($1)
+; O2-NEXT:    beqz $3, .LBB5_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    nop
+;
+; O3-LABEL: AtomicLoadNand:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand)))
+; O3-NEXT:    daddu $1, $1, $25
+; O3-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand)))
+; O3-NEXT:    ld $1, %got_disp(x)($1)
+; O3-NEXT:  .LBB5_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    lld $2, 0($1)
+; O3-NEXT:    and $3, $2, $4
+; O3-NEXT:    nor $3, $zero, $3
+; O3-NEXT:    scd $3, 0($1)
+; O3-NEXT:    beqz $3, .LBB5_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    nop
+;
+; MIPS64EB-LABEL: AtomicLoadNand:
+; MIPS64EB:       # %bb.0: # %entry
+; MIPS64EB-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicLoadNand)))
+; MIPS64EB-NEXT:    daddu $1, $1, $25
+; MIPS64EB-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicLoadNand)))
+; MIPS64EB-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64EB-NEXT:  .LBB5_1: # %entry
+; MIPS64EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64EB-NEXT:    lld $2, 0($1)
+; MIPS64EB-NEXT:    and $3, $2, $4
+; MIPS64EB-NEXT:    nor $3, $zero, $3
+; MIPS64EB-NEXT:    scd $3, 0($1)
+; MIPS64EB-NEXT:    beqz $3, .LBB5_1
+; MIPS64EB-NEXT:    nop
+; MIPS64EB-NEXT:  # %bb.2: # %entry
+; MIPS64EB-NEXT:    jr $ra
+; MIPS64EB-NEXT:    nop
+entry:
+  %0 = atomicrmw nand i64* @x, i64 %incr monotonic
+  ret i64 %0
+
+}
+
+define i64 @AtomicSwap64(i64 signext %newval) nounwind {
+; MIPS4-LABEL: AtomicSwap64:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    daddiu $sp, $sp, -16
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap64)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap64)))
+; MIPS4-NEXT:    sd $4, 8($sp)
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB6_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    lld $2, 0($1)
+; MIPS4-NEXT:    move $3, $4
+; MIPS4-NEXT:    scd $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB6_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64-LABEL: AtomicSwap64:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    daddiu $sp, $sp, -16
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap64)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap64)))
+; MIPS64-NEXT:    sd $4, 8($sp)
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB6_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    lld $2, 0($1)
+; MIPS64-NEXT:    move $3, $4
+; MIPS64-NEXT:    scd $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB6_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64R2-LABEL: AtomicSwap64:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap64)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap64)))
+; MIPS64R2-NEXT:    sd $4, 8($sp)
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB6_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    lld $2, 0($1)
+; MIPS64R2-NEXT:    move $3, $4
+; MIPS64R2-NEXT:    scd $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB6_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64R6-LABEL: AtomicSwap64:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap64)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap64)))
+; MIPS64R6-NEXT:    sd $4, 8($sp)
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB6_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    lld $2, 0($1)
+; MIPS64R6-NEXT:    move $3, $4
+; MIPS64R6-NEXT:    scd $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB6_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    jr $ra
+; MIPS64R6-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64R6O0-LABEL: AtomicSwap64:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap64)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap64)))
+; MIPS64R6O0-NEXT:    move $25, $4
+; MIPS64R6O0-NEXT:    sd $4, 8($sp)
+; MIPS64R6O0-NEXT:    ld $4, 8($sp)
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:  .LBB6_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    lld $2, 0($1)
+; MIPS64R6O0-NEXT:    move $3, $4
+; MIPS64R6O0-NEXT:    scd $3, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $3, .LBB6_1
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    sd $25, 0($sp) # 8-byte Folded Spill
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 16
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicSwap64:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    daddiu $sp, $sp, -16
+; O1-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap64)))
+; O1-NEXT:    daddu $1, $1, $25
+; O1-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap64)))
+; O1-NEXT:    sd $4, 8($sp)
+; O1-NEXT:    ld $1, %got_disp(x)($1)
+; O1-NEXT:  .LBB6_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    lld $2, 0($1)
+; O1-NEXT:    move $3, $4
+; O1-NEXT:    scd $3, 0($1)
+; O1-NEXT:    beqz $3, .LBB6_1
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    daddiu $sp, $sp, 16
+;
+; O2-LABEL: AtomicSwap64:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    daddiu $sp, $sp, -16
+; O2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap64)))
+; O2-NEXT:    daddu $1, $1, $25
+; O2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap64)))
+; O2-NEXT:    sd $4, 8($sp)
+; O2-NEXT:    ld $1, %got_disp(x)($1)
+; O2-NEXT:  .LBB6_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    lld $2, 0($1)
+; O2-NEXT:    move $3, $4
+; O2-NEXT:    scd $3, 0($1)
+; O2-NEXT:    beqz $3, .LBB6_1
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    daddiu $sp, $sp, 16
+;
+; O3-LABEL: AtomicSwap64:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    daddiu $sp, $sp, -16
+; O3-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap64)))
+; O3-NEXT:    sd $4, 8($sp)
+; O3-NEXT:    daddu $1, $1, $25
+; O3-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap64)))
+; O3-NEXT:    ld $1, %got_disp(x)($1)
+; O3-NEXT:  .LBB6_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    lld $2, 0($1)
+; O3-NEXT:    move $3, $4
+; O3-NEXT:    scd $3, 0($1)
+; O3-NEXT:    beqz $3, .LBB6_1
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64EB-LABEL: AtomicSwap64:
+; MIPS64EB:       # %bb.0: # %entry
+; MIPS64EB-NEXT:    daddiu $sp, $sp, -16
+; MIPS64EB-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicSwap64)))
+; MIPS64EB-NEXT:    daddu $1, $1, $25
+; MIPS64EB-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicSwap64)))
+; MIPS64EB-NEXT:    sd $4, 8($sp)
+; MIPS64EB-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64EB-NEXT:  .LBB6_1: # %entry
+; MIPS64EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64EB-NEXT:    lld $2, 0($1)
+; MIPS64EB-NEXT:    move $3, $4
+; MIPS64EB-NEXT:    scd $3, 0($1)
+; MIPS64EB-NEXT:    beqz $3, .LBB6_1
+; MIPS64EB-NEXT:    nop
+; MIPS64EB-NEXT:  # %bb.2: # %entry
+; MIPS64EB-NEXT:    jr $ra
+; MIPS64EB-NEXT:    daddiu $sp, $sp, 16
+entry:
+  %newval.addr = alloca i64, align 4
+  store i64 %newval, i64* %newval.addr, align 4
+  %tmp = load i64, i64* %newval.addr, align 4
+  %0 = atomicrmw xchg i64* @x, i64 %tmp monotonic
+  ret i64 %0
+
+}
+
+define i64 @AtomicCmpSwap64(i64 signext %oldval, i64 signext %newval) nounwind {
+; MIPS4-LABEL: AtomicCmpSwap64:
+; MIPS4:       # %bb.0: # %entry
+; MIPS4-NEXT:    daddiu $sp, $sp, -16
+; MIPS4-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap64)))
+; MIPS4-NEXT:    daddu $1, $1, $25
+; MIPS4-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap64)))
+; MIPS4-NEXT:    sd $5, 8($sp)
+; MIPS4-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS4-NEXT:  .LBB7_1: # %entry
+; MIPS4-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS4-NEXT:    lld $2, 0($1)
+; MIPS4-NEXT:    bne $2, $4, .LBB7_3
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  # %bb.2: # %entry
+; MIPS4-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS4-NEXT:    move $3, $5
+; MIPS4-NEXT:    scd $3, 0($1)
+; MIPS4-NEXT:    beqz $3, .LBB7_1
+; MIPS4-NEXT:    nop
+; MIPS4-NEXT:  .LBB7_3: # %entry
+; MIPS4-NEXT:    jr $ra
+; MIPS4-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64-LABEL: AtomicCmpSwap64:
+; MIPS64:       # %bb.0: # %entry
+; MIPS64-NEXT:    daddiu $sp, $sp, -16
+; MIPS64-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap64)))
+; MIPS64-NEXT:    daddu $1, $1, $25
+; MIPS64-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap64)))
+; MIPS64-NEXT:    sd $5, 8($sp)
+; MIPS64-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64-NEXT:  .LBB7_1: # %entry
+; MIPS64-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64-NEXT:    lld $2, 0($1)
+; MIPS64-NEXT:    bne $2, $4, .LBB7_3
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  # %bb.2: # %entry
+; MIPS64-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS64-NEXT:    move $3, $5
+; MIPS64-NEXT:    scd $3, 0($1)
+; MIPS64-NEXT:    beqz $3, .LBB7_1
+; MIPS64-NEXT:    nop
+; MIPS64-NEXT:  .LBB7_3: # %entry
+; MIPS64-NEXT:    jr $ra
+; MIPS64-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64R2-LABEL: AtomicCmpSwap64:
+; MIPS64R2:       # %bb.0: # %entry
+; MIPS64R2-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap64)))
+; MIPS64R2-NEXT:    daddu $1, $1, $25
+; MIPS64R2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap64)))
+; MIPS64R2-NEXT:    sd $5, 8($sp)
+; MIPS64R2-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R2-NEXT:  .LBB7_1: # %entry
+; MIPS64R2-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R2-NEXT:    lld $2, 0($1)
+; MIPS64R2-NEXT:    bne $2, $4, .LBB7_3
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  # %bb.2: # %entry
+; MIPS64R2-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS64R2-NEXT:    move $3, $5
+; MIPS64R2-NEXT:    scd $3, 0($1)
+; MIPS64R2-NEXT:    beqz $3, .LBB7_1
+; MIPS64R2-NEXT:    nop
+; MIPS64R2-NEXT:  .LBB7_3: # %entry
+; MIPS64R2-NEXT:    jr $ra
+; MIPS64R2-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64R6-LABEL: AtomicCmpSwap64:
+; MIPS64R6:       # %bb.0: # %entry
+; MIPS64R6-NEXT:    daddiu $sp, $sp, -16
+; MIPS64R6-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap64)))
+; MIPS64R6-NEXT:    daddu $1, $1, $25
+; MIPS64R6-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap64)))
+; MIPS64R6-NEXT:    sd $5, 8($sp)
+; MIPS64R6-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6-NEXT:  .LBB7_1: # %entry
+; MIPS64R6-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6-NEXT:    lld $2, 0($1)
+; MIPS64R6-NEXT:    bnec $2, $4, .LBB7_3
+; MIPS64R6-NEXT:  # %bb.2: # %entry
+; MIPS64R6-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS64R6-NEXT:    move $3, $5
+; MIPS64R6-NEXT:    scd $3, 0($1)
+; MIPS64R6-NEXT:    beqzc $3, .LBB7_1
+; MIPS64R6-NEXT:    nop
+; MIPS64R6-NEXT:  .LBB7_3: # %entry
+; MIPS64R6-NEXT:    jr $ra
+; MIPS64R6-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64R6O0-LABEL: AtomicCmpSwap64:
+; MIPS64R6O0:       # %bb.0: # %entry
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, -48
+; MIPS64R6O0-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap64)))
+; MIPS64R6O0-NEXT:    daddu $1, $1, $25
+; MIPS64R6O0-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap64)))
+; MIPS64R6O0-NEXT:    move $25, $5
+; MIPS64R6O0-NEXT:    move $2, $4
+; MIPS64R6O0-NEXT:    sd $5, 40($sp)
+; MIPS64R6O0-NEXT:    ld $5, 40($sp)
+; MIPS64R6O0-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64R6O0-NEXT:    ld $3, 32($sp) # 8-byte Folded Reload
+; MIPS64R6O0-NEXT:  .LBB7_1: # %entry
+; MIPS64R6O0-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64R6O0-NEXT:    lld $6, 0($1)
+; MIPS64R6O0-NEXT:    bnec $6, $4, .LBB7_3
+; MIPS64R6O0-NEXT:  # %bb.2: # %entry
+; MIPS64R6O0-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS64R6O0-NEXT:    move $7, $5
+; MIPS64R6O0-NEXT:    scd $7, 0($1)
+; MIPS64R6O0-NEXT:    beqzc $7, .LBB7_1
+; MIPS64R6O0-NEXT:  .LBB7_3: # %entry
+; MIPS64R6O0-NEXT:    sd $2, 24($sp) # 8-byte Folded Spill
+; MIPS64R6O0-NEXT:    move $2, $6
+; MIPS64R6O0-NEXT:    sd $6, 32($sp) # 8-byte Folded Spill
+; MIPS64R6O0-NEXT:    sd $25, 16($sp) # 8-byte Folded Spill
+; MIPS64R6O0-NEXT:    sd $3, 8($sp) # 8-byte Folded Spill
+; MIPS64R6O0-NEXT:    daddiu $sp, $sp, 48
+; MIPS64R6O0-NEXT:    jrc $ra
+;
+; O1-LABEL: AtomicCmpSwap64:
+; O1:       # %bb.0: # %entry
+; O1-NEXT:    daddiu $sp, $sp, -16
+; O1-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap64)))
+; O1-NEXT:    daddu $1, $1, $25
+; O1-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap64)))
+; O1-NEXT:    sd $5, 8($sp)
+; O1-NEXT:    ld $1, %got_disp(x)($1)
+; O1-NEXT:  .LBB7_1: # %entry
+; O1-NEXT:    # =>This Inner Loop Header: Depth=1
+; O1-NEXT:    lld $2, 0($1)
+; O1-NEXT:    bne $2, $4, .LBB7_3
+; O1-NEXT:    nop
+; O1-NEXT:  # %bb.2: # %entry
+; O1-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; O1-NEXT:    move $3, $5
+; O1-NEXT:    scd $3, 0($1)
+; O1-NEXT:    beqz $3, .LBB7_1
+; O1-NEXT:    nop
+; O1-NEXT:  .LBB7_3: # %entry
+; O1-NEXT:    jr $ra
+; O1-NEXT:    daddiu $sp, $sp, 16
+;
+; O2-LABEL: AtomicCmpSwap64:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    daddiu $sp, $sp, -16
+; O2-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap64)))
+; O2-NEXT:    daddu $1, $1, $25
+; O2-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap64)))
+; O2-NEXT:    sd $5, 8($sp)
+; O2-NEXT:    ld $1, %got_disp(x)($1)
+; O2-NEXT:  .LBB7_1: # %entry
+; O2-NEXT:    # =>This Inner Loop Header: Depth=1
+; O2-NEXT:    lld $2, 0($1)
+; O2-NEXT:    bne $2, $4, .LBB7_3
+; O2-NEXT:    nop
+; O2-NEXT:  # %bb.2: # %entry
+; O2-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; O2-NEXT:    move $3, $5
+; O2-NEXT:    scd $3, 0($1)
+; O2-NEXT:    beqz $3, .LBB7_1
+; O2-NEXT:    nop
+; O2-NEXT:  .LBB7_3: # %entry
+; O2-NEXT:    jr $ra
+; O2-NEXT:    daddiu $sp, $sp, 16
+;
+; O3-LABEL: AtomicCmpSwap64:
+; O3:       # %bb.0: # %entry
+; O3-NEXT:    daddiu $sp, $sp, -16
+; O3-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap64)))
+; O3-NEXT:    sd $5, 8($sp)
+; O3-NEXT:    daddu $1, $1, $25
+; O3-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap64)))
+; O3-NEXT:    ld $1, %got_disp(x)($1)
+; O3-NEXT:  .LBB7_1: # %entry
+; O3-NEXT:    # =>This Inner Loop Header: Depth=1
+; O3-NEXT:    lld $2, 0($1)
+; O3-NEXT:    bne $2, $4, .LBB7_3
+; O3-NEXT:    nop
+; O3-NEXT:  # %bb.2: # %entry
+; O3-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; O3-NEXT:    move $3, $5
+; O3-NEXT:    scd $3, 0($1)
+; O3-NEXT:    beqz $3, .LBB7_1
+; O3-NEXT:    nop
+; O3-NEXT:  .LBB7_3: # %entry
+; O3-NEXT:    jr $ra
+; O3-NEXT:    daddiu $sp, $sp, 16
+;
+; MIPS64EB-LABEL: AtomicCmpSwap64:
+; MIPS64EB:       # %bb.0: # %entry
+; MIPS64EB-NEXT:    daddiu $sp, $sp, -16
+; MIPS64EB-NEXT:    lui $1, %hi(%neg(%gp_rel(AtomicCmpSwap64)))
+; MIPS64EB-NEXT:    daddu $1, $1, $25
+; MIPS64EB-NEXT:    daddiu $1, $1, %lo(%neg(%gp_rel(AtomicCmpSwap64)))
+; MIPS64EB-NEXT:    sd $5, 8($sp)
+; MIPS64EB-NEXT:    ld $1, %got_disp(x)($1)
+; MIPS64EB-NEXT:  .LBB7_1: # %entry
+; MIPS64EB-NEXT:    # =>This Inner Loop Header: Depth=1
+; MIPS64EB-NEXT:    lld $2, 0($1)
+; MIPS64EB-NEXT:    bne $2, $4, .LBB7_3
+; MIPS64EB-NEXT:    nop
+; MIPS64EB-NEXT:  # %bb.2: # %entry
+; MIPS64EB-NEXT:    # in Loop: Header=BB7_1 Depth=1
+; MIPS64EB-NEXT:    move $3, $5
+; MIPS64EB-NEXT:    scd $3, 0($1)
+; MIPS64EB-NEXT:    beqz $3, .LBB7_1
+; MIPS64EB-NEXT:    nop
+; MIPS64EB-NEXT:  .LBB7_3: # %entry
+; MIPS64EB-NEXT:    jr $ra
+; MIPS64EB-NEXT:    daddiu $sp, $sp, 16
+entry:
+  %newval.addr = alloca i64, align 4
+  store i64 %newval, i64* %newval.addr, align 4
+  %tmp = load i64, i64* %newval.addr, align 4
+  %0 = cmpxchg i64* @x, i64 %oldval, i64 %tmp monotonic monotonic
+  %1 = extractvalue { i64, i1 } %0, 0
+  ret i64 %1
+
+}

llvm/trunk/test/CodeGen/Mips/atomicCmpSwapPW.ll

-; RUN: llc -O0 -march=mipsel -mcpu=mips32r2 -target-abi=o32 < %s -filetype=asm -o - \
-; RUN:   | FileCheck -check-prefixes=PTR32,ALL %s
-; RUN: llc -O0 -march=mips64el -mcpu=mips64r2 -target-abi=n32 < %s -filetype=asm -o - \
-; RUN:   | FileCheck  -check-prefixes=PTR32,ALL %s
-; RUN: llc -O0 -march=mips64el -mcpu=mips64r2 -target-abi=n64 < %s -filetype=asm -o - \
-; RUN:   | FileCheck -check-prefixes=PTR64,ALL %s
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -O0 -mtriple=mipsel-unknown-linux-gnu -mcpu=mips32r2 -target-abi=o32 < %s -filetype=asm -o - \
+; RUN:   | FileCheck -check-prefixes=O32 %s
+; RUN: llc -O0 -mtriple=mips64el-unknown-linux-gnu -mcpu=mips64r2 -target-abi=n32 < %s -filetype=asm -o - \
+; RUN:   | FileCheck  -check-prefixes=N32,ALL %s
+; RUN: llc -O0 -mtriple=mips64el-unknown-linux-gnu -mcpu=mips64r2 -target-abi=n64 < %s -filetype=asm -o - \
+; RUN:   | FileCheck -check-prefixes=N64 %s
 
-; PTR32: lw $[[R0:[0-9]+]]
-; PTR64: ld $[[R0:[0-9]+]]
+@sym = external global i32 *
 
-; ALL: ll ${{[0-9]+}}, 0($[[R0]])
-
-define {i16, i1} @foo(i16* %addr, i16 signext %r, i16 zeroext %new) {
-  %res = cmpxchg i16* %addr, i16 %r, i16 %new seq_cst seq_cst
-  ret {i16, i1} %res
+define void @foo(i32 %new, i32 %old) {
+; O32-LABEL: foo:
+; O32:       # %bb.0: # %entry
+; O32-NEXT:    addiu $sp, $sp, -16
+; O32-NEXT:    .cfi_def_cfa_offset 16
+; O32-NEXT:    move $1, $5
+; O32-NEXT:    move $2, $4
+; O32-NEXT:    lui $3, %hi(sym)
+; O32-NEXT:    lw $3, %lo(sym)($3)
+; O32-NEXT:    sync
+; O32-NEXT:    lw $6, 12($sp) # 4-byte Folded Reload
+; O32-NEXT:  $BB0_1: # %entry
+; O32-NEXT:    # =>This Inner Loop Header: Depth=1
+; O32-NEXT:    ll $7, 0($3)
+; O32-NEXT:    bne $7, $4, $BB0_3
+; O32-NEXT:    nop
+; O32-NEXT:  # %bb.2: # %entry
+; O32-NEXT:    # in Loop: Header=BB0_1 Depth=1
+; O32-NEXT:    move $8, $5
+; O32-NEXT:    sc $8, 0($3)
+; O32-NEXT:    beqz $8, $BB0_1
+; O32-NEXT:    nop
+; O32-NEXT:  $BB0_3: # %entry
+; O32-NEXT:    sync
+; O32-NEXT:    sw $7, 12($sp) # 4-byte Folded Spill
+; O32-NEXT:    sw $6, 8($sp) # 4-byte Folded Spill
+; O32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
+; O32-NEXT:    sw $2, 0($sp) # 4-byte Folded Spill
+; O32-NEXT:    addiu $sp, $sp, 16
+; O32-NEXT:    jr $ra
+; O32-NEXT:    nop
+;
+; N32-LABEL: foo:
+; N32:       # %bb.0: # %entry
+; N32-NEXT:    addiu $sp, $sp, -16
+; N32-NEXT:    .cfi_def_cfa_offset 16
+; N32-NEXT:    move $1, $5
+; N32-NEXT:    sll $1, $1, 0
+; N32-NEXT:    move $2, $4
+; N32-NEXT:    sll $2, $2, 0
+; N32-NEXT:    lui $3, %hi(sym)
+; N32-NEXT:    lw $3, %lo(sym)($3)
+; N32-NEXT:    sync
+; N32-NEXT:    lw $6, 12($sp) # 4-byte Folded Reload
+; N32-NEXT:  .LBB0_1: # %entry
+; N32-NEXT:    # =>This Inner Loop Header: Depth=1
+; N32-NEXT:    ll $7, 0($3)
+; N32-NEXT:    bne $7, $2, .LBB0_3
+; N32-NEXT:    nop
+; N32-NEXT:  # %bb.2: # %entry
+; N32-NEXT:    # in Loop: Header=BB0_1 Depth=1
+; N32-NEXT:    move $8, $1
+; N32-NEXT:    sc $8, 0($3)
+; N32-NEXT:    beqz $8, .LBB0_1
+; N32-NEXT:    nop
+; N32-NEXT:  .LBB0_3: # %entry
+; N32-NEXT:    sync
+; N32-NEXT:    sw $7, 12($sp) # 4-byte Folded Spill
+; N32-NEXT:    sw $6, 8($sp) # 4-byte Folded Spill
+; N32-NEXT:    addiu $sp, $sp, 16
+; N32-NEXT:    jr $ra
+; N32-NEXT:    nop
+;
+; N64-LABEL: foo:
+; N64:       # %bb.0: # %entry
+; N64-NEXT:    daddiu $sp, $sp, -16
+; N64-NEXT:    .cfi_def_cfa_offset 16
+; N64-NEXT:    move $1, $5
+; N64-NEXT:    sll $1, $1, 0
+; N64-NEXT:    move $2, $4
+; N64-NEXT:    sll $2, $2, 0
+; N64-NEXT:    lui $4, %highest(sym)
+; N64-NEXT:    daddiu $4, $4, %higher(sym)
+; N64-NEXT:    dsll $4, $4, 16
+; N64-NEXT:    daddiu $4, $4, %hi(sym)
+; N64-NEXT:    dsll $4, $4, 16
+; N64-NEXT:    ld $4, %lo(sym)($4)
+; N64-NEXT:    sync
+; N64-NEXT:    lw $3, 12($sp) # 4-byte Folded Reload
+; N64-NEXT:  .LBB0_1: # %entry
+; N64-NEXT:    # =>This Inner Loop Header: Depth=1
+; N64-NEXT:    ll $6, 0($4)
+; N64-NEXT:    bne $6, $2, .LBB0_3
+; N64-NEXT:    nop
+; N64-NEXT:  # %bb.2: # %entry
+; N64-NEXT:    # in Loop: Header=BB0_1 Depth=1
+; N64-NEXT:    move $7, $1
+; N64-NEXT:    sc $7, 0($4)
+; N64-NEXT:    beqz $7, .LBB0_1
+; N64-NEXT:    nop
+; N64-NEXT:  .LBB0_3: # %entry
+; N64-NEXT:    sync
+; N64-NEXT:    sw $6, 12($sp) # 4-byte Folded Spill
+; N64-NEXT:    sw $3, 8($sp) # 4-byte Folded Spill
+; N64-NEXT:    daddiu $sp, $sp, 16
+; N64-NEXT:    jr $ra
+; N64-NEXT:    nop
+entry:
+  %0 = load i32 *, i32 ** @sym
+  cmpxchg i32 * %0, i32 %new, i32 %old seq_cst seq_cst
+  ret void
 }
 

llvm/trunk/test/CodeGen/Mips/micromips-atomic.ll

-; RUN: llc %s -march=mipsel -mcpu=mips32r2 -mattr=micromips -filetype=asm \
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc %s -mtriple=mipsel-unknown-linux-gnu -mcpu=mips32r2 -mattr=micromips -filetype=asm \
 ; RUN: -relocation-model=pic -o - | FileCheck %s
 
 @x = common global i32 0, align 4
 
 define i32 @AtomicLoadAdd32(i32 %incr) nounwind {
+; CHECK-LABEL: AtomicLoadAdd32:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    lui $2, %hi(_gp_disp)
+; CHECK-NEXT:    addiu $2, $2, %lo(_gp_disp)
+; CHECK-NEXT:    addu $2, $2, $25
+; CHECK-NEXT:    lw $1, %got(x)($2)
+; CHECK-NEXT:  $BB0_1: # %entry
+; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    ll $2, 0($1)
+; CHECK-NEXT:    addu16 $3, $2, $4
+; CHECK-NEXT:    sc $3, 0($1)
+; CHECK-NEXT:    beqzc $3, $BB0_1
+; CHECK-NEXT:  # %bb.2: # %entry
+; CHECK-NEXT:    jrc $ra
 entry:
   %0 = atomicrmw add i32* @x, i32 %incr monotonic
   ret i32 %0
-
-; CHECK-LABEL:   AtomicLoadAdd32:
-; CHECK:   lw      $[[R0:[0-9]+]], %got(x)
-; CHECK:   $[[BB0:[A-Z_0-9]+]]:
-; CHECK:   ll      $[[R1:[0-9]+]], 0($[[R0]])
-; CHECK:   addu    $[[R2:[0-9]+]], $[[R1]], $4
-; CHECK:   sc      $[[R2]], 0($[[R0]])
-; CHECK:   beqzc   $[[R2]], $[[BB0]]
 }