[ARM] Prevent spilling between ldrex/strex pairs

[tmatheson]

Summary

Based on the same for AArch64: 4751cadcca45984d7671e594ce95aed8fe030bf1

At -O0, the fast register allocator may insert spills between the ldrex and
strex instructions inserted by AtomicExpandPass when expanding atomicrmw
instructions in LL/SC loops. To avoid this, expand to cmpxchg loops and
therefore expand the cmpxchg pseudos after register allocation.

Required a tweak to ARMExpandPseudo::ExpandCMP_SWAP to use the 4-byte encoding
of UXT, since the pseudo instruction can be allocated a high register (R8-R15)
which the 2-byte encoding doesn't support.

The previously committed attempt in D101164 had to be reverted due to runtime
failures in the test suites. Rather than spending time fixing that
implementation (adding another implementation of atomic operations and more
divergence between backends) I have chosen to follow the approach taken in
D101163.

Compared to D101164, this patch also fixes the problem for Thumb 2.

Depends on D101912

[lenary]

Please can you pre-commit atomicrmw_exclusive_monitor_all.ll, so we can see the changes this commit makes to those code sequences?

[LemonBoy]

llvm/lib/Target/ARM/ARMExpandPseudoInsts.cpp

1592

Nit, guard this line with if (UxtOp == t2UXTB) as already done for the ldrex/strex ops below.

[tmatheson]

• At more triples to the test (Thumb1/Thumb2)
• Remove tests for floating point atomics. They tend to be bitcast to ints by the frontend and the support for them is patchy across architecture versions.
• Rebase on top of pre-commit test in D101912

[tmatheson]

llvm/lib/Target/ARM/ARMExpandPseudoInsts.cpp

1592

All of the currently used variants (t2UXTB, UXTB, t2UXTH, UXTH) expect an immediate here, which is different from the strex/ldrex instructions of which only t2STREX and t2STREX accept an offset. So I don't think a guard is necessary.

[tmatheson]

Minor change to trigger rebuild

[tmatheson]

Minimize the CHECK lines in the test

[efriedma]

Really, using AtomicExpansionKind::LLSC at all on ARM/AArch64 is fragile; long-term, I think it still makes sense to pursue using pseudo-instructions. But this is clearly an improvement.

I'm a little surprised that ARM::CMP_SWAP_8/ARM::CMP_SWAP_16 generating broken instructions hasn't caused any issues before.

llvm/lib/Target/ARM/ARMExpandPseudoInsts.cpp

2800–2801

I don't think we can use ARM::t2UXTB here unconditionally. Specifically, I'm concerned about the armv8m.base target, which has ARM::t2LDREXBm/ARM::t2STREXB, but not ARM::t2UXTB.

Instead, can we add a separate ARM::tCMP_SWAP_8 instruction, with an appropriate register allocation restriction?

[tmatheson]

• Add tests for v8m.baseline
• Add two new pseudo instructions tCMP_SWAP_8 and tCMP_SWAP_16 to limit register for UXT instruction
• Add some asserts in ExpandCMP_SWAP

[efriedma]

llvm/lib/Target/ARM/ARMExpandPseudoInsts.cpp

1573

The text of this assertion seems seems wrong; the point of the assertion is that ARMv8-M.baseline doesn't have t2UXTB/t2UXTH

llvm/lib/Target/ARM/ARMISelDAGToDAG.cpp

3302

isThumb() includes both Thumb1 (here, v8m.base) and Thumb2. That's fine, but if you're going to do that, please remove the dead Thumb2 codepaths from ARMExpandPseudoInsts.

[tmatheson]

Remove dead code paths and fix assert string

[efriedma]

LGTM

[tmatheson]

Thanks for the feedback!

[efriedma]

llvm/test/CodeGen/ARM/atomicrmw_exclusive_monitor_ints.ll

380

Just realized I wasn't looking carefully enough when I reviewed this...

In the case where we're generating a call to the runtime library, we want to call __sync_fetch_and_sub_8, not __sync_val_compare_and_swap_8. Same for other similar atomicrmw operations. I mean, it's not semantically wrong the way it is, but using __sync_val_compare_and_swap_8 is worse for codesize.

I think the way to handle this is to switch around the logic to return AtomicExpansionKind::None instead of AtomicExpansionKind::CmpXChg in this case.

llvm/lib/Target/ARM/ARMExpandPseudoInsts.cpp

bool ARMExpandPseudo::ExpandCMP_SWAP(MachineBasicBlock &MBB,
   Register TempReg = MI.getOperand(1).getReg();
   // Duplicating undef operands into 2 instructions does not guarantee the same
   // value on both; However undef should be replaced by xzr anyway.
   assert(!MI.getOperand(2).isUndef() && "cannot handle undef");
   Register AddrReg = MI.getOperand(2).getReg();
   Register DesiredReg = MI.getOperand(3).getReg();
   Register NewReg = MI.getOperand(4).getReg();
 
+  if (IsThumb) {
+    assert(STI->hasV8MBaselineOps() &&
+           "CMP_SWAP not expected to be custom expanded for Thumb1");
+    assert((UxtOp == 0 || UxtOp == ARM::tUXTB || UxtOp == ARM::tUXTH) &&
+           "ARMv8-M.baseline does not have t2UXTB/t2UXTH");
+    assert(ARM::tGPRRegClass.contains(DesiredReg) &&
+           "DesiredReg used for UXT op must be tGPR");
+  }
+
   MachineFunction *MF = MBB.getParent();
   auto LoadCmpBB = MF->CreateMachineBasicBlock(MBB.getBasicBlock());
   auto StoreBB = MF->CreateMachineBasicBlock(MBB.getBasicBlock());
   auto DoneBB = MF->CreateMachineBasicBlock(MBB.getBasicBlock());
 
   MF->insert(++MBB.getIterator(), LoadCmpBB);
   MF->insert(++LoadCmpBB->getIterator(), StoreBB);
   MF->insert(++StoreBB->getIterator(), DoneBB);
 
   if (UxtOp) {
     MachineInstrBuilder MIB =
         BuildMI(MBB, MBBI, DL, TII->get(UxtOp), DesiredReg)
             .addReg(DesiredReg, RegState::Kill);
     if (!IsThumb)
       MIB.addImm(0);
     MIB.add(predOps(ARMCC::AL));
   }
 
   // .Lloadcmp:
   //     ldrex rDest, [rAddr]
   //     cmp rDest, rDesired
   //     bne .Ldone
 
case ARM::VST4LNq32Pseudo_UPD:
       ExpandLaneOp(MBBI);
       return true;
 
     case ARM::VTBL3Pseudo: ExpandVTBL(MBBI, ARM::VTBL3, false); return true;
     case ARM::VTBL4Pseudo: ExpandVTBL(MBBI, ARM::VTBL4, false); return true;
     case ARM::VTBX3Pseudo: ExpandVTBL(MBBI, ARM::VTBX3, true); return true;
     case ARM::VTBX4Pseudo: ExpandVTBL(MBBI, ARM::VTBX4, true); return true;
 
+    case ARM::tCMP_SWAP_8:
+      assert(STI->isThumb());
+      return ExpandCMP_SWAP(MBB, MBBI, ARM::t2LDREXB, ARM::t2STREXB, ARM::tUXTB,
+                            NextMBBI);
+    case ARM::tCMP_SWAP_16:
+      assert(STI->isThumb());
+      return ExpandCMP_SWAP(MBB, MBBI, ARM::t2LDREXH, ARM::t2STREXH, ARM::tUXTH,
+                            NextMBBI);
+
     case ARM::CMP_SWAP_8:
-      if (STI->isThumb())
-        return ExpandCMP_SWAP(MBB, MBBI, ARM::t2LDREXB, ARM::t2STREXB,
-                              ARM::tUXTB, NextMBBI);
-      else
-        return ExpandCMP_SWAP(MBB, MBBI, ARM::LDREXB, ARM::STREXB,
-                              ARM::UXTB, NextMBBI);
+      assert(!STI->isThumb());
+      return ExpandCMP_SWAP(MBB, MBBI, ARM::LDREXB, ARM::STREXB, ARM::UXTB,
+                            NextMBBI);
     case ARM::CMP_SWAP_16:
-      if (STI->isThumb())
-        return ExpandCMP_SWAP(MBB, MBBI, ARM::t2LDREXH, ARM::t2STREXH,
-                              ARM::tUXTH, NextMBBI);
-      else
-        return ExpandCMP_SWAP(MBB, MBBI, ARM::LDREXH, ARM::STREXH,
-                              ARM::UXTH, NextMBBI);
+      assert(!STI->isThumb());
+      return ExpandCMP_SWAP(MBB, MBBI, ARM::LDREXH, ARM::STREXH, ARM::UXTH,
+                            NextMBBI);
     case ARM::CMP_SWAP_32:
       if (STI->isThumb())
         return ExpandCMP_SWAP(MBB, MBBI, ARM::t2LDREX, ARM::t2STREX, 0,
                               NextMBBI);
       else
         return ExpandCMP_SWAP(MBB, MBBI, ARM::LDREX, ARM::STREX, 0, NextMBBI);
 
     case ARM::CMP_SWAP_64:

llvm/lib/Target/ARM/ARMISelDAGToDAG.cpp

bool ARMDAGToDAGISel::tryABSOp(SDNode *N){
   return false;
 }
 
 /// We've got special pseudo-instructions for these
 void ARMDAGToDAGISel::SelectCMP_SWAP(SDNode *N) {
   unsigned Opcode;
   EVT MemTy = cast<MemSDNode>(N)->getMemoryVT();
   if (MemTy == MVT::i8)
-    Opcode = ARM::CMP_SWAP_8;
+    Opcode = Subtarget->isThumb() ? ARM::tCMP_SWAP_8 : ARM::CMP_SWAP_8;
   else if (MemTy == MVT::i16)
-    Opcode = ARM::CMP_SWAP_16;
+    Opcode = Subtarget->isThumb() ? ARM::tCMP_SWAP_16 : ARM::CMP_SWAP_16;
   else if (MemTy == MVT::i32)
     Opcode = ARM::CMP_SWAP_32;
   else
     llvm_unreachable("Unknown AtomicCmpSwap type");
 
   SDValue Ops[] = {N->getOperand(1), N->getOperand(2), N->getOperand(3),
                    N->getOperand(0)};
   SDNode *CmpSwap = CurDAG->getMachineNode(

llvm/lib/Target/ARM/ARMISelLowering.cpp

 
 // For the real atomic operations, we have ldrex/strex up to 32 bits,
 // and up to 64 bits on the non-M profiles
 TargetLowering::AtomicExpansionKind
 ARMTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
   if (AI->isFloatingPointOperation())
     return AtomicExpansionKind::CmpXChg;
 
+  // At -O0, fast-regalloc cannot cope with the live vregs necessary to
+  // implement atomicrmw without spilling. If the target address is also on the
+  // stack and close enough to the spill slot, this can lead to a situation
+  // where the monitor always gets cleared and the atomic operation can never
+  // succeed. So at -O0 lower this operation to a CAS loop.
+  if (getTargetMachine().getOptLevel() == CodeGenOpt::None)
+    return AtomicExpansionKind::CmpXChg;
+
   unsigned Size = AI->getType()->getPrimitiveSizeInBits();
   bool hasAtomicRMW = !Subtarget->isThumb() || Subtarget->hasV8MBaselineOps();
   return (Size <= (Subtarget->isMClass() ? 32U : 64U) && hasAtomicRMW)
              ? AtomicExpansionKind::LLSC
              : AtomicExpansionKind::None;
 }
 
 // Similar to shouldExpandAtomicRMWInIR, ldrex/strex can be used  up to 32

llvm/lib/Target/ARM/ARMInstrThumb.td

def : tInstAlias<"lsr${s}${p} $Rdm, $imm",
              (tLSRri tGPR:$Rdm, cc_out:$s, tGPR:$Rdm, imm_sr:$imm, pred:$p)>;
 def : tInstAlias<"asr${s}${p} $Rdm, $imm",
              (tASRri tGPR:$Rdm, cc_out:$s, tGPR:$Rdm, imm_sr:$imm, pred:$p)>;
 
 // Pseudo instruction ldr Rt, =immediate
 def tLDRConstPool
   : tAsmPseudo<"ldr${p} $Rt, $immediate",
                (ins tGPR:$Rt, const_pool_asm_imm:$immediate, pred:$p)>;
+
+//===----------------------------------
+// Atomic cmpxchg for -O0
+//===----------------------------------
+
+// See ARMInstrInfo.td. These two thumb specific pseudos are required to
+// restrict the register class for the UXTB/UXTH ops used in the expansion.
+
+let Constraints = "@earlyclobber $Rd,@earlyclobber $temp",
+    mayLoad = 1, mayStore = 1 in {
+def tCMP_SWAP_8 : PseudoInst<(outs GPR:$Rd, GPR:$temp),
+                             (ins GPR:$addr, tGPR:$desired, GPR:$new),
+                             NoItinerary, []>, Sched<[]>;
+
+def tCMP_SWAP_16 : PseudoInst<(outs GPR:$Rd, GPR:$temp),
+                              (ins GPR:$addr, tGPR:$desired, GPR:$new),
+                              NoItinerary, []>, Sched<[]>;
+}

llvm/test/CodeGen/ARM/atomicrmw_exclusive_monitor_ints.ll

 @atomic_i8 = external global i8
 @atomic_i16 = external global i16
 @atomic_i32 = external global i32
 @atomic_i64 = external global i64
 
 define i8 @test_xchg_i8() {
 ; COMMON-LABEL: test_xchg_i8:
 ; EXPAND32: ldrexb
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexb
 ; THUMB1: bl __sync_lock_test_and_set_1
 entry:
   %0 = atomicrmw xchg i8* @atomic_i8, i8 1 monotonic
   ret i8 %0
 }
 define i8 @test_add_i8() {
 ; COMMON-LABEL: test_add_i8:
 ; EXPAND32: ldrexb
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexb
 ; THUMB1: bl __sync_fetch_and_add_1
 entry:
   %0 = atomicrmw add i8* @atomic_i8, i8 1 monotonic
   ret i8 %0
 }
 define i8 @test_sub_i8() {
 ; COMMON-LABEL: test_sub_i8:
 ; EXPAND32: ldrexb
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexb
 ; THUMB1: bl __sync_fetch_and_sub_1
 entry:
   %0 = atomicrmw sub i8* @atomic_i8, i8 1 monotonic
   ret i8 %0
 }
 define i8 @test_and_i8() {
 ; COMMON-LABEL: test_and_i8:
 ; EXPAND32: ldrexb
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexb
 ; THUMB1: bl __sync_fetch_and_and_1
 entry:
   %0 = atomicrmw and i8* @atomic_i8, i8 1 monotonic
   ret i8 %0
 }
 define i8 @test_nand_i8() {
 ; COMMON-LABEL: test_nand_i8:
 ; EXPAND32: ldrexb
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexb
 ; THUMB1: bl __sync_fetch_and_nand_1
 entry:
   %0 = atomicrmw nand i8* @atomic_i8, i8 1 monotonic
   ret i8 %0
 }
 define i8 @test_or_i8() {
 ; COMMON-LABEL: test_or_i8:
 ; EXPAND32: ldrexb
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexb
 ; THUMB1: bl __sync_fetch_and_or_1
 entry:
   %0 = atomicrmw or i8* @atomic_i8, i8 1 monotonic
   ret i8 %0
 }
 define i8 @test_xor_i8() {
 ; COMMON-LABEL: test_xor_i8:
 ; EXPAND32: ldrexb
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexb
 ; THUMB1: bl __sync_fetch_and_xor_1
 entry:
   %0 = atomicrmw xor i8* @atomic_i8, i8 1 monotonic
   ret i8 %0
 }
 define i8 @test_max_i8() {
 ; COMMON-LABEL: test_max_i8:
 ; EXPAND32: ldrexb
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexb
 ; THUMB1: bl __sync_fetch_and_max_1
 entry:
   %0 = atomicrmw max i8* @atomic_i8, i8 1 monotonic
   ret i8 %0
 }
 define i8 @test_min_i8() {
 ; COMMON-LABEL: test_min_i8:
 ; EXPAND32: ldrexb
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexb
 ; THUMB1: bl __sync_fetch_and_min_1
 entry:
   %0 = atomicrmw min i8* @atomic_i8, i8 1 monotonic
   ret i8 %0
 }
 define i8 @test_umax_i8() {
 ; COMMON-LABEL: test_umax_i8:
 ; EXPAND32: ldrexb
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexb
 ; THUMB1: bl __sync_fetch_and_umax_1
 entry:
   %0 = atomicrmw umax i8* @atomic_i8, i8 1 monotonic
   ret i8 %0
 }
 define i8 @test_umin_i8() {
 ; COMMON-LABEL: test_umin_i8:
 ; EXPAND32: ldrexb
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexb
 ; THUMB1: bl __sync_fetch_and_umin_1
 entry:
   %0 = atomicrmw umin i8* @atomic_i8, i8 1 monotonic
   ret i8 %0
 }
 
 
 define i16 @test_xchg_i16() {
 ; COMMON-LABEL: test_xchg_i16:
 ; EXPAND32: ldrexh
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexh
 ; THUMB1: bl __sync_lock_test_and_set_2
 entry:
   %0 = atomicrmw xchg i16* @atomic_i16, i16 1 monotonic
   ret i16 %0
 }
 define i16 @test_add_i16() {
 ; COMMON-LABEL: test_add_i16:
 ; EXPAND32: ldrexh
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexh
 ; THUMB1: bl __sync_fetch_and_add_2
 entry:
   %0 = atomicrmw add i16* @atomic_i16, i16 1 monotonic
   ret i16 %0
 }
 define i16 @test_sub_i16() {
 ; COMMON-LABEL: test_sub_i16:
 ; EXPAND32: ldrexh
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexh
 ; THUMB1: bl __sync_fetch_and_sub_2
 entry:
   %0 = atomicrmw sub i16* @atomic_i16, i16 1 monotonic
   ret i16 %0
 }
 define i16 @test_and_i16() {
 ; COMMON-LABEL: test_and_i16:
 ; EXPAND32: ldrexh
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexh
 ; THUMB1: bl __sync_fetch_and_and_2
 entry:
   %0 = atomicrmw and i16* @atomic_i16, i16 1 monotonic
   ret i16 %0
 }
 define i16 @test_nand_i16() {
 ; COMMON-LABEL: test_nand_i16:
 ; EXPAND32: ldrexh
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexh
 ; THUMB1: bl __sync_fetch_and_nand_2
 entry:
   %0 = atomicrmw nand i16* @atomic_i16, i16 1 monotonic
   ret i16 %0
 }
 define i16 @test_or_i16() {
 ; COMMON-LABEL: test_or_i16:
 ; EXPAND32: ldrexh
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexh
 ; THUMB1: bl __sync_fetch_and_or_2
 entry:
   %0 = atomicrmw or i16* @atomic_i16, i16 1 monotonic
   ret i16 %0
 }
 define i16 @test_xor_i16() {
 ; COMMON-LABEL: test_xor_i16:
 ; EXPAND32: ldrexh
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexh
 ; THUMB1: bl __sync_fetch_and_xor_2
 entry:
   %0 = atomicrmw xor i16* @atomic_i16, i16 1 monotonic
   ret i16 %0
 }
 define i16 @test_max_i16() {
 ; COMMON-LABEL: test_max_i16:
 ; EXPAND32: ldrexh
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexh
 ; THUMB1: bl __sync_fetch_and_max_2
 entry:
   %0 = atomicrmw max i16* @atomic_i16, i16 1 monotonic
   ret i16 %0
 }
 define i16 @test_min_i16() {
 ; COMMON-LABEL: test_min_i16:
 ; EXPAND32: ldrexh
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexh
 ; THUMB1: bl __sync_fetch_and_min_2
 entry:
   %0 = atomicrmw min i16* @atomic_i16, i16 1 monotonic
   ret i16 %0
 }
 define i16 @test_umax_i16() {
 ; COMMON-LABEL: test_umax_i16:
 ; EXPAND32: ldrexh
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexh
 ; THUMB1: bl __sync_fetch_and_umax_2
 entry:
   %0 = atomicrmw umax i16* @atomic_i16, i16 1 monotonic
   ret i16 %0
 }
 define i16 @test_umin_i16() {
 ; COMMON-LABEL: test_umin_i16:
 ; EXPAND32: ldrexh
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strexh
 ; THUMB1: bl __sync_fetch_and_umin_2
 entry:
   %0 = atomicrmw umin i16* @atomic_i16, i16 1 monotonic
   ret i16 %0
 }
 
 
 define i32 @test_xchg_i32() {
 ; COMMON-LABEL: test_xchg_i32:
 ; EXPAND32: ldrex
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strex
 ; THUMB1: bl __sync_lock_test_and_set_4
 entry:
   %0 = atomicrmw xchg i32* @atomic_i32, i32 1 monotonic
   ret i32 %0
 }
 define i32 @test_add_i32() {
 ; COMMON-LABEL: test_add_i32:
 ; EXPAND32: ldrex
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strex
 ; THUMB1: bl __sync_fetch_and_add_4
 entry:
   %0 = atomicrmw add i32* @atomic_i32, i32 1 monotonic
   ret i32 %0
 }
 define i32 @test_sub_i32() {
 ; COMMON-LABEL: test_sub_i32:
 ; EXPAND32: ldrex
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strex
 ; THUMB1: bl __sync_fetch_and_sub_4
 entry:
   %0 = atomicrmw sub i32* @atomic_i32, i32 1 monotonic
   ret i32 %0
 }
 define i32 @test_and_i32() {
 ; COMMON-LABEL: test_and_i32:
 ; EXPAND32: ldrex
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strex
 ; THUMB1: bl __sync_fetch_and_and_4
 entry:
   %0 = atomicrmw and i32* @atomic_i32, i32 1 monotonic
   ret i32 %0
 }
 define i32 @test_nand_i32() {
 ; COMMON-LABEL: test_nand_i32:
 ; EXPAND32: ldrex
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strex
 ; THUMB1: bl __sync_fetch_and_nand_4
 entry:
   %0 = atomicrmw nand i32* @atomic_i32, i32 1 monotonic
   ret i32 %0
 }
 define i32 @test_or_i32() {
 ; COMMON-LABEL: test_or_i32:
 ; EXPAND32: ldrex
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strex
 ; THUMB1: bl __sync_fetch_and_or_4
 entry:
   %0 = atomicrmw or i32* @atomic_i32, i32 1 monotonic
   ret i32 %0
 }
 define i32 @test_xor_i32() {
 ; COMMON-LABEL: test_xor_i32:
 ; EXPAND32: ldrex
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strex
 ; THUMB1: bl __sync_fetch_and_xor_4
 entry:
   %0 = atomicrmw xor i32* @atomic_i32, i32 1 monotonic
   ret i32 %0
 }
 define i32 @test_max_i32() {
 ; COMMON-LABEL: test_max_i32:
 ; EXPAND32: ldrex
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strex
 ; THUMB1: bl __sync_fetch_and_max_4
 entry:
   %0 = atomicrmw max i32* @atomic_i32, i32 1 monotonic
   ret i32 %0
 }
 define i32 @test_min_i32() {
 ; COMMON-LABEL: test_min_i32:
 ; EXPAND32: ldrex
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strex
 
 ; THUMB1: bl __sync_fetch_and_min_4
 entry:
   %0 = atomicrmw min i32* @atomic_i32, i32 1 monotonic
   ret i32 %0
 }
 define i32 @test_umax_i32() {
 ; COMMON-LABEL: test_umax_i32:
 ; EXPAND32: ldrex
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strex
 ; THUMB1: bl __sync_fetch_and_umax_4
 entry:
   %0 = atomicrmw umax i32* @atomic_i32, i32 1 monotonic
   ret i32 %0
 }
 define i32 @test_umin_i32() {
 ; COMMON-LABEL: test_umin_i32:
 ; EXPAND32: ldrex
-; EXPAND32: str
+; EXPAND32-NOT: str
 ; EXPAND32: strex
 ; THUMB1: bl __sync_fetch_and_umin_4
 entry:
   %0 = atomicrmw umin i32* @atomic_i32, i32 1 monotonic
   ret i32 %0
 }
 
 define i64 @test_xchg_i64() {
 ; COMMON-LABEL: test_xchg_i64:
 ; EXPAND64: ldrexd
-; EXPAND64: str
+; EXPAND64-NOT: str
 ; EXPAND64: strexd
 ; THUMB1: bl __sync_lock_test_and_set_8
-; BASELINE64: bl __sync_lock_test_and_set_8
+; BASELINE64: bl __sync_val_compare_and_swap_8
 entry:
   %0 = atomicrmw xchg i64* @atomic_i64, i64 1 monotonic
   ret i64 %0
 }
 define i64 @test_add_i64() {
 ; COMMON-LABEL: test_add_i64:
 ; EXPAND64: ldrexd
-; EXPAND64: str
+; EXPAND64-NOT: str
 ; EXPAND64: strexd
 ; THUMB1: bl __sync_fetch_and_add_8
-; BASELINE64: bl __sync_fetch_and_add_8
+; BASELINE64: bl __sync_val_compare_and_swap_8
 entry:
   %0 = atomicrmw add i64* @atomic_i64, i64 1 monotonic
   ret i64 %0
 }
 define i64 @test_sub_i64() {
 ; COMMON-LABEL: test_sub_i64:
 ; EXPAND64: ldrexd
-; EXPAND64: str
+; EXPAND64-NOT: str
 ; EXPAND64: strexd
 ; THUMB1: bl __sync_fetch_and_sub_8
-; BASELINE64: bl __sync_fetch_and_sub_8
+; BASELINE64: bl __sync_val_compare_and_swap_8
 entry:
   %0 = atomicrmw sub i64* @atomic_i64, i64 1 monotonic
   ret i64 %0
 }
 define i64 @test_and_i64() {
 ; COMMON-LABEL: test_and_i64:
 ; EXPAND64: ldrexd
-; EXPAND64: str
+; EXPAND64-NOT: str
 ; EXPAND64: strexd
 ; THUMB1: bl __sync_fetch_and_and_8
-; BASELINE64: bl __sync_fetch_and_and_8
+; BASELINE64: bl __sync_val_compare_and_swap_8
 entry:
   %0 = atomicrmw and i64* @atomic_i64, i64 1 monotonic
   ret i64 %0
 }
 define i64 @test_nand_i64() {
 ; COMMON-LABEL: test_nand_i64:
 ; EXPAND64: ldrexd
-; EXPAND64: str
+; EXPAND64-NOT: str
 ; EXPAND64: strexd
 ; THUMB1: bl __sync_fetch_and_nand_8
-; BASELINE64: bl __sync_fetch_and_nand_8
+; BASELINE64: bl __sync_val_compare_and_swap_8
 entry:
   %0 = atomicrmw nand i64* @atomic_i64, i64 1 monotonic
   ret i64 %0
 }
 define i64 @test_or_i64() {
 ; COMMON-LABEL: test_or_i64:
 ; EXPAND64: ldrexd
-; EXPAND64: str
+; EXPAND64-NOT: str
 ; EXPAND64: strexd
 ; THUMB1: bl __sync_fetch_and_or_8
-; BASELINE64: bl __sync_fetch_and_or_8
+; BASELINE64: bl __sync_val_compare_and_swap_8
 entry:
   %0 = atomicrmw or i64* @atomic_i64, i64 1 monotonic
   ret i64 %0
 }
 define i64 @test_xor_i64() {
 ; COMMON-LABEL: test_xor_i64:
 ; EXPAND64: ldrexd
-; EXPAND64: str
+; EXPAND64-NOT: str
 ; EXPAND64: strexd
 ; THUMB1: bl __sync_fetch_and_xor_8
-; BASELINE64: bl __sync_fetch_and_xor_8
+; BASELINE64: bl __sync_val_compare_and_swap_8
 entry:
   %0 = atomicrmw xor i64* @atomic_i64, i64 1 monotonic
   ret i64 %0
 }
 
 define i64 @test_max_i64() {
 ; COMMON-LABEL: test_max_i64:
 ; EXPAND64: ldrexd
-; EXPAND64: str
+; EXPAND64-NOT: str
 ; EXPAND64: strexd
 ; THUMB1: bl __sync_fetch_and_max_8
-; BASELINE64: bl __sync_fetch_and_max_8
+; BASELINE64: bl __sync_val_compare_and_swap_8
 entry:
   %0 = atomicrmw max i64* @atomic_i64, i64 1 monotonic
   ret i64 %0
 }
 define i64 @test_min_i64() {
 ; COMMON-LABEL: test_min_i64:
 ; EXPAND64: ldrexd
-; EXPAND64: str
+; EXPAND64-NOT: str
 ; EXPAND64: strexd
 ; THUMB1: bl __sync_fetch_and_min_8
-; BASELINE64: bl __sync_fetch_and_min_8
+; BASELINE64: bl __sync_val_compare_and_swap_8
 entry:
   %0 = atomicrmw min i64* @atomic_i64, i64 1 monotonic
   ret i64 %0
 }
 define i64 @test_umax_i64() {
 ; COMMON-LABEL: test_umax_i64:
 ; EXPAND64: ldrexd
-; EXPAND64: str
+; EXPAND64-NOT: str
 ; EXPAND64: strexd
 ; THUMB1: bl __sync_fetch_and_umax_8
-; BASELINE64: bl __sync_fetch_and_umax_8
+; BASELINE64: bl __sync_val_compare_and_swap_8
 entry:
   %0 = atomicrmw umax i64* @atomic_i64, i64 1 monotonic
   ret i64 %0
 }
 define i64 @test_umin_i64() {
 ; COMMON-LABEL: test_umin_i64:
 ; EXPAND64: ldrexd
-; EXPAND64: str
+; EXPAND64-NOT: str
 ; EXPAND64: strexd
 ; THUMB1: bl __sync_fetch_and_umin_8
-; BASELINE64: bl __sync_fetch_and_umin_8
+; BASELINE64: bl __sync_val_compare_and_swap_8
 entry:
   %0 = atomicrmw umin i64* @atomic_i64, i64 1 monotonic
   ret i64 %0
 }