[RISCV] Add target feature to force-enable atomics

[nikic]

Summary

This adds a +forced-atomics target feature with the same semantics as on ARM (D130480). For RISCV targets without the +a extension, this forces LLVM to assume that lock-free atomics (up to 32/64 bits for riscv32/64 respectively) are available.

This means that atomic load/store are lowered to a simple load/store (and fence as necessary), as these are guaranteed to be atomic (as long as they're aligned). Atomic RMW/CAS are lowered to __sync (rather than __atomic) libcalls. Responsibility for providing the __sync libcalls lies with the user (for privileged single-core code they can be implemented by disabling interrupts). Code using +forced-atomics and -forced-atomics are not ABI compatible if atomic variables cross the ABI boundary.

For context, the difference between __sync and __atomic is that the former are required to be lock-free, while the latter requires a shared global lock provided by a shared object library. See https://llvm.org/docs/Atomics.html#libcalls-atomic for a detailed discussion on the topic.

This target feature will be used by Rust's riscv32i target family to support the use of atomic load/store without atomic RMW/CAS.

[jrtc27]

Why atomics-32 and not something that generalises to include 64-bit atomics on RV64?

llvm/lib/Target/RISCV/RISCVInstrInfoA.td

111

Don’t duplicate patterns

[nikic]

llvm/lib/Target/RISCV/RISCVInstrInfoA.td

111

What's the right way to avoid the duplication here? Is there a way to use "or" inside Predicates?

[• pcwang-thead]

llvm/lib/Target/RISCV/RISCVInstrInfoA.td

111

You can define an or predicate in RISCV.td like HasStdExtMOrZmmul, HasStdExtZfhOrZfhmin, etc.

[nikic]

Convert +atomics-32 into +forced-atomics, which also works correctly on riscv64. Use a single predicate for +forced-atomics and +a when defining tablegen patterns.

[nikic]

In D130621#3681861, @jrtc27 wrote:

Why atomics-32 and not something that generalises to include 64-bit atomics on RV64?

Good point. I've changed this to a +force-atomics feature that works for both riscv32 and riscv64. (If this patch lands, I'll also rename the ARM feature to match.)

llvm/lib/Target/RISCV/RISCVInstrInfoA.td

111

Thanks, I did that now!

[reames]

Can you spell out in the review comment the difference between sync and atomic and why there's a strong preference for one over the other? Links to details preferred, but enough context to send a reader in the right direction is appreciated.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

414

What MinCmpXchgSizeInBits do you want for the forced atomics?

Once you answer that, this code can probably be rearranged to have a common if body for both A and forced atomics.

Also, use XLenVT

933

I believe that at least a subset of these goes away if you set setMinCmpXchgSizeInBits properly. Haven't definitely confirmed, but at glance at the code makes that look likely.

llvm/test/CodeGen/RISCV/forced-atomics32.ll

5 ↗

(On Diff #448017)

You need coverage here for non-seq_cst orderings. You don't need to repeat it for all sizes, but please make sure to at least add tests for all the orderings at the native width,

[nikic]

Use XLen, test atomic orderings.

[nikic]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

414

I think we don't want a minimum for this case. It would save the need to support two __sync_val_compare_and_swap libcalls at the expense of generating more complex code for i8/i16 cmpxchg. Here's what the diff would look like: https://gist.github.com/nikic/9025e91bde139fbb75225f4fbd0539e9

[asb]

Broadly speaking, this looks good to me (thanks!). I think in some previous discussions on this topic I hadn't appreciated the lock-free nature of the sync calls vs atomic.

My one request would be to flesh out the test coverage so the files cover all of the different atomicrmw operations - given they all had to be explicitly listed in the setOperationAction call to avoid their expansion, it would be worth verifying this (to provide some resilience against future refactorings).

[asb]

It might also be worth patching the __sync section in the atomics docs to mention that some targets support this forced-atomics feature and provide a bit more insight on the use case (" for various reasons, it is not practical to emit the instructions inline." is a little vague).

That can be a separate patch, but I've found the LLVM atomics guide to be a really helpful reference so it would be great to keep it as up to date and informative as possible.

[nikic]

Add tests for all rmw kinds, add note to atomics docs.

[nikic]

In D130621#3690619, @asb wrote:

Broadly speaking, this looks good to me (thanks!). I think in some previous discussions on this topic I hadn't appreciated the lock-free nature of the sync calls vs atomic.

My one request would be to flesh out the test coverage so the files cover all of the different atomicrmw operations - given they all had to be explicitly listed in the setOperationAction call to avoid their expansion, it would be worth verifying this (to provide some resilience against future refactorings).

Good call -- my implementation actually failed to handle the atomicrmw float operations -- those always need IR expansion.

[asb]

Just one question about the tests - typically we combine RV32 and RV64 tests in one file (this makes it easier if e.g. another operation was added or a new corner case found, as it's less likely we'd add a new case for RV32 but not RV64 or vice versa). Is there a reason you can't merge these into forced-atomics.ll?

Otherwise, looks good to me - thanks!

[nikic]

Combine tests for rv32 and rv64.

[asb]

LGTM, thanks!

llvm/lib/Target/RISCV/RISCV.td

 def TuneNoDefaultUnroll
     : SubtargetFeature<"no-default-unroll", "EnableDefaultUnroll", "false",
                        "Disable default unroll preference.">;
 
 def TuneSiFive7 : SubtargetFeature<"sifive7", "RISCVProcFamily", "SiFive7",
                                    "SiFive 7-Series processors",
                                    [TuneNoDefaultUnroll]>;
 
+// Assume that lock-free 32-bit atomics are available, even if the target
+// and operating system combination would not usually provide them. The user
+// is responsible for providing any necessary __sync implementations. Code
+// built with this feature is not ABI-compatible with code built without this
+// feature, if atomic variables are exposed across the ABI boundary.
+def FeatureAtomics32 : SubtargetFeature<
+    "atomics-32", "HasForced32BitAtomics", "true",
+    "Assume that lock-free 32-bit atomics are available">;
+def HasAtomicLdSt32 : Predicate<"Subtarget->hasForced32BitAtomics()">;
+
 //===----------------------------------------------------------------------===//
 // Named operands for CSR instructions.
 //===----------------------------------------------------------------------===//
 
 include "RISCVSystemOperands.td"
 
 //===----------------------------------------------------------------------===//
 // Registers, calling conventions, instruction descriptions.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

RISCVTargetLowering::RISCVTargetLowering(const TargetMachine &TM,
   setOperationAction({ISD::TRAP, ISD::DEBUGTRAP}, MVT::Other, Legal);
   setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::Other, Custom);
   if (Subtarget.is64Bit())
     setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::i32, Custom);
 
   if (Subtarget.hasStdExtA()) {
     setMaxAtomicSizeInBitsSupported(Subtarget.getXLen());
     setMinCmpXchgSizeInBits(32);
+  } else if (Subtarget.hasForced32BitAtomics()) {
+    setMaxAtomicSizeInBitsSupported(32);
   } else {
     setMaxAtomicSizeInBitsSupported(0);
   }
 
   setBooleanContents(ZeroOrOneBooleanContent);
 
   if (Subtarget.hasVInstructions()) {
     setBooleanVectorContents(ZeroOrOneBooleanContent);
if (Subtarget.useRVVForFixedLengthVectors()) {
         setOperationAction(ISD::BITCAST, MVT::f16, Custom);
       if (Subtarget.hasStdExtF())
         setOperationAction(ISD::BITCAST, MVT::f32, Custom);
       if (Subtarget.hasStdExtD())
         setOperationAction(ISD::BITCAST, MVT::f64, Custom);
     }
   }
 
+  if (Subtarget.hasForced32BitAtomics()) {
+    // Set atomic rmw/cas operations to expand to force __sync libcalls.
+    setOperationAction(ISD::ATOMIC_CMP_SWAP,  MVT::i32, Expand);
+    setOperationAction(ISD::ATOMIC_SWAP,      MVT::i32, Expand);
+    setOperationAction(ISD::ATOMIC_LOAD_ADD,  MVT::i32, Expand);
+    setOperationAction(ISD::ATOMIC_LOAD_SUB,  MVT::i32, Expand);
+    setOperationAction(ISD::ATOMIC_LOAD_AND,  MVT::i32, Expand);
+    setOperationAction(ISD::ATOMIC_LOAD_OR,   MVT::i32, Expand);
+    setOperationAction(ISD::ATOMIC_LOAD_XOR,  MVT::i32, Expand);
+    setOperationAction(ISD::ATOMIC_LOAD_NAND, MVT::i32, Expand);
+    setOperationAction(ISD::ATOMIC_LOAD_MIN,  MVT::i32, Expand);
+    setOperationAction(ISD::ATOMIC_LOAD_MAX,  MVT::i32, Expand);
+    setOperationAction(ISD::ATOMIC_LOAD_UMIN, MVT::i32, Expand);
+    setOperationAction(ISD::ATOMIC_LOAD_UMAX, MVT::i32, Expand);
+  }
+
   // Function alignments.
   const Align FunctionAlignment(Subtarget.hasStdExtC() ? 2 : 4);
   setMinFunctionAlignment(FunctionAlignment);
   setPrefFunctionAlignment(FunctionAlignment);
 
   setMinimumJumpTableEntries(5);
 
   // Jumps are expensive, compared to logic
Instruction *RISCVTargetLowering::emitTrailingFence(IRBuilderBase &Builder,
                                                     AtomicOrdering Ord) const {
   if (isa<LoadInst>(Inst) && isAcquireOrStronger(Ord))
     return Builder.CreateFence(AtomicOrdering::Acquire);
   return nullptr;
 }
 
 TargetLowering::AtomicExpansionKind
 RISCVTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
+  // Don't expand forced atomics, we want to have __sync libcalls instead.
+  if (Subtarget.hasForced32BitAtomics())
+    return AtomicExpansionKind::None;
+
   // atomicrmw {fadd,fsub} must be expanded to use compare-exchange, as floating
   // point operations can't be used in an lr/sc sequence without breaking the
   // forward-progress guarantee.
   if (AI->isFloatingPointOperation())
     return AtomicExpansionKind::CmpXChg;
 
   unsigned Size = AI->getType()->getPrimitiveSizeInBits();
   if (Size == 8 || Size == 16)
Value *RISCVTargetLowering::emitMaskedAtomicRMWIntrinsic(
   if (XLen == 64)
     Result = Builder.CreateTrunc(Result, Builder.getInt32Ty());
   return Result;
 }
 
 TargetLowering::AtomicExpansionKind
 RISCVTargetLowering::shouldExpandAtomicCmpXchgInIR(
     AtomicCmpXchgInst *CI) const {
+  // Don't expand forced atomics, we want to have __sync libcalls instead.
+  if (Subtarget.hasForced32BitAtomics())
+    return AtomicExpansionKind::None;
+
   unsigned Size = CI->getCompareOperand()->getType()->getPrimitiveSizeInBits();
   if (Size == 8 || Size == 16)
     return AtomicExpansionKind::MaskedIntrinsic;
   return AtomicExpansionKind::None;
 }
 
 Value *RISCVTargetLowering::emitMaskedAtomicCmpXchgIntrinsic(
     IRBuilderBase &Builder, AtomicCmpXchgInst *CI, Value *AlignedAddr,

llvm/lib/Target/RISCV/RISCVInstrInfoA.td

 defm AMOMAXU_D  : AMO_rr_aq_rl<0b11100, 0b011, "amomaxu.d">,
                   Sched<[WriteAtomicD, ReadAtomicDA, ReadAtomicDD]>;
 } // Predicates = [HasStdExtA, IsRV64]
 
 //===----------------------------------------------------------------------===//
 // Pseudo-instructions and codegen patterns
 //===----------------------------------------------------------------------===//
 
+// Make atomic load/store also available under the +atomics-32 feature.
+// Atomic rmw/cas operations are only available with +a.
+let Predicates = [HasAtomicLdSt32] in {
+  defm : LdPat<atomic_load_8,  LB>;
+  defm : LdPat<atomic_load_16, LH>;
+  defm : LdPat<atomic_load_32, LW>;
+
+  defm : AtomicStPat<atomic_store_8,  SB, GPR>;
+  defm : AtomicStPat<atomic_store_16, SH, GPR>;
+  defm : AtomicStPat<atomic_store_32, SW, GPR>;
+}
+
 let Predicates = [HasStdExtA] in {
 
 /// Atomic loads and stores
 
 // Fences will be inserted for atomic load/stores according to the logic in
 // RISCVTargetLowering::{emitLeadingFence,emitTrailingFence}.
 
 defm : LdPat<atomic_load_8,  LB>;

llvm/lib/Target/RISCV/RISCVSubtarget.h

private:
   bool HasRV64 = false;
   bool IsRV32E = false;
   bool EnableLinkerRelax = false;
   bool EnableRVCHintInstrs = true;
   bool EnableDefaultUnroll = true;
   bool EnableSaveRestore = false;
   bool EnableUnalignedScalarMem = false;
   bool HasLUIADDIFusion = false;
+  bool HasForced32BitAtomics = false;
   unsigned XLen = 32;
   unsigned ZvlLen = 0;
   MVT XLenVT = MVT::i32;
   uint8_t MaxInterleaveFactor = 2;
   RISCVABI::ABI TargetABI = RISCVABI::ABI_Unknown;
   BitVector UserReservedRegister;
   RISCVFrameLowering FrameLowering;
   RISCVInstrInfo InstrInfo;
public:
   bool is64Bit() const { return HasRV64; }
   bool isRV32E() const { return IsRV32E; }
   bool enableLinkerRelax() const { return EnableLinkerRelax; }
   bool enableRVCHintInstrs() const { return EnableRVCHintInstrs; }
   bool enableDefaultUnroll() const { return EnableDefaultUnroll; }
   bool enableSaveRestore() const { return EnableSaveRestore; }
   bool enableUnalignedScalarMem() const { return EnableUnalignedScalarMem; }
   bool hasLUIADDIFusion() const { return HasLUIADDIFusion; }
+  bool hasForced32BitAtomics() const { return HasForced32BitAtomics; }
   MVT getXLenVT() const { return XLenVT; }
   unsigned getXLen() const { return XLen; }
   unsigned getFLen() const {
     if (HasStdExtD)
       return 64;
 
     if (HasStdExtF)
       return 32;

llvm/test/CodeGen/RISCV/forced-atomic32.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 < %s | FileCheck %s --check-prefixes=CHECK,NO-ATOMIC32
+; RUN: llc -mtriple=riscv32 -mattr=+atomics-32 < %s | FileCheck %s --check-prefixes=CHECK,ATOMIC32
+
+define i8 @load8(ptr %p) nounwind {
+; NO-ATOMIC32-LABEL: load8:
+; NO-ATOMIC32:       # %bb.0:
+; NO-ATOMIC32-NEXT:    addi sp, sp, -16
+; NO-ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; NO-ATOMIC32-NEXT:    li a1, 5
+; NO-ATOMIC32-NEXT:    call __atomic_load_1@plt
+; NO-ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; NO-ATOMIC32-NEXT:    addi sp, sp, 16
+; NO-ATOMIC32-NEXT:    ret
+;
+; ATOMIC32-LABEL: load8:
+; ATOMIC32:       # %bb.0:
+; ATOMIC32-NEXT:    fence rw, rw
+; ATOMIC32-NEXT:    lb a0, 0(a0)
+; ATOMIC32-NEXT:    fence r, rw
+; ATOMIC32-NEXT:    ret
+  %v = load atomic i8, ptr %p seq_cst, align 1
+  ret i8 %v
+}
+
+define void @store8(ptr %p) nounwind {
+; NO-ATOMIC32-LABEL: store8:
+; NO-ATOMIC32:       # %bb.0:
+; NO-ATOMIC32-NEXT:    addi sp, sp, -16
+; NO-ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; NO-ATOMIC32-NEXT:    li a2, 5
+; NO-ATOMIC32-NEXT:    li a1, 0
+; NO-ATOMIC32-NEXT:    call __atomic_store_1@plt
+; NO-ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; NO-ATOMIC32-NEXT:    addi sp, sp, 16
+; NO-ATOMIC32-NEXT:    ret
+;
+; ATOMIC32-LABEL: store8:
+; ATOMIC32:       # %bb.0:
+; ATOMIC32-NEXT:    fence rw, w
+; ATOMIC32-NEXT:    sb zero, 0(a0)
+; ATOMIC32-NEXT:    ret
+  store atomic i8 0, ptr %p seq_cst, align 1
+  ret void
+}
+
+define i8 @rmw8(ptr %p) nounwind {
+; NO-ATOMIC32-LABEL: rmw8:
+; NO-ATOMIC32:       # %bb.0:
+; NO-ATOMIC32-NEXT:    addi sp, sp, -16
+; NO-ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; NO-ATOMIC32-NEXT:    li a1, 1
+; NO-ATOMIC32-NEXT:    li a2, 5
+; NO-ATOMIC32-NEXT:    call __atomic_fetch_add_1@plt
+; NO-ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; NO-ATOMIC32-NEXT:    addi sp, sp, 16
+; NO-ATOMIC32-NEXT:    ret
+;
+; ATOMIC32-LABEL: rmw8:
+; ATOMIC32:       # %bb.0:
+; ATOMIC32-NEXT:    addi sp, sp, -16
+; ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; ATOMIC32-NEXT:    li a1, 1
+; ATOMIC32-NEXT:    call __sync_fetch_and_add_1@plt
+; ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; ATOMIC32-NEXT:    addi sp, sp, 16
+; ATOMIC32-NEXT:    ret
+  %v = atomicrmw add ptr %p, i8 1 seq_cst, align 1
+  ret i8 %v
+}
+
+define i8 @cmpxchg8(ptr %p) nounwind {
+; NO-ATOMIC32-LABEL: cmpxchg8:
+; NO-ATOMIC32:       # %bb.0:
+; NO-ATOMIC32-NEXT:    addi sp, sp, -16
+; NO-ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; NO-ATOMIC32-NEXT:    sb zero, 11(sp)
+; NO-ATOMIC32-NEXT:    addi a1, sp, 11
+; NO-ATOMIC32-NEXT:    li a2, 1
+; NO-ATOMIC32-NEXT:    li a3, 5
+; NO-ATOMIC32-NEXT:    li a4, 5
+; NO-ATOMIC32-NEXT:    call __atomic_compare_exchange_1@plt
+; NO-ATOMIC32-NEXT:    lb a0, 11(sp)
+; NO-ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; NO-ATOMIC32-NEXT:    addi sp, sp, 16
+; NO-ATOMIC32-NEXT:    ret
+;
+; ATOMIC32-LABEL: cmpxchg8:
+; ATOMIC32:       # %bb.0:
+; ATOMIC32-NEXT:    addi sp, sp, -16
+; ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; ATOMIC32-NEXT:    li a2, 1
+; ATOMIC32-NEXT:    li a1, 0
+; ATOMIC32-NEXT:    call __sync_val_compare_and_swap_1@plt
+; ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; ATOMIC32-NEXT:    addi sp, sp, 16
+; ATOMIC32-NEXT:    ret
+  %res = cmpxchg ptr %p, i8 0, i8 1 seq_cst seq_cst
+  %res.0 = extractvalue { i8, i1 } %res, 0
+  ret i8 %res.0
+}
+
+define i16 @load16(ptr %p) nounwind {
+; NO-ATOMIC32-LABEL: load16:
+; NO-ATOMIC32:       # %bb.0:
+; NO-ATOMIC32-NEXT:    addi sp, sp, -16
+; NO-ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; NO-ATOMIC32-NEXT:    li a1, 5
+; NO-ATOMIC32-NEXT:    call __atomic_load_2@plt
+; NO-ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; NO-ATOMIC32-NEXT:    addi sp, sp, 16
+; NO-ATOMIC32-NEXT:    ret
+;
+; ATOMIC32-LABEL: load16:
+; ATOMIC32:       # %bb.0:
+; ATOMIC32-NEXT:    fence rw, rw
+; ATOMIC32-NEXT:    lh a0, 0(a0)
+; ATOMIC32-NEXT:    fence r, rw
+; ATOMIC32-NEXT:    ret
+  %v = load atomic i16, ptr %p seq_cst, align 2
+  ret i16 %v
+}
+
+define void @store16(ptr %p) nounwind {
+; NO-ATOMIC32-LABEL: store16:
+; NO-ATOMIC32:       # %bb.0:
+; NO-ATOMIC32-NEXT:    addi sp, sp, -16
+; NO-ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; NO-ATOMIC32-NEXT:    li a2, 5
+; NO-ATOMIC32-NEXT:    li a1, 0
+; NO-ATOMIC32-NEXT:    call __atomic_store_2@plt
+; NO-ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; NO-ATOMIC32-NEXT:    addi sp, sp, 16
+; NO-ATOMIC32-NEXT:    ret
+;
+; ATOMIC32-LABEL: store16:
+; ATOMIC32:       # %bb.0:
+; ATOMIC32-NEXT:    fence rw, w
+; ATOMIC32-NEXT:    sh zero, 0(a0)
+; ATOMIC32-NEXT:    ret
+  store atomic i16 0, ptr %p seq_cst, align 2
+  ret void
+}
+
+define i16 @rmw16(ptr %p) nounwind {
+; NO-ATOMIC32-LABEL: rmw16:
+; NO-ATOMIC32:       # %bb.0:
+; NO-ATOMIC32-NEXT:    addi sp, sp, -16
+; NO-ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; NO-ATOMIC32-NEXT:    li a1, 1
+; NO-ATOMIC32-NEXT:    li a2, 5
+; NO-ATOMIC32-NEXT:    call __atomic_fetch_add_2@plt
+; NO-ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; NO-ATOMIC32-NEXT:    addi sp, sp, 16
+; NO-ATOMIC32-NEXT:    ret
+;
+; ATOMIC32-LABEL: rmw16:
+; ATOMIC32:       # %bb.0:
+; ATOMIC32-NEXT:    addi sp, sp, -16
+; ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; ATOMIC32-NEXT:    li a1, 1
+; ATOMIC32-NEXT:    call __sync_fetch_and_add_2@plt
+; ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; ATOMIC32-NEXT:    addi sp, sp, 16
+; ATOMIC32-NEXT:    ret
+  %v = atomicrmw add ptr %p, i16 1 seq_cst, align 2
+  ret i16 %v
+}
+
+define i16 @cmpxchg16(ptr %p) nounwind {
+; NO-ATOMIC32-LABEL: cmpxchg16:
+; NO-ATOMIC32:       # %bb.0:
+; NO-ATOMIC32-NEXT:    addi sp, sp, -16
+; NO-ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; NO-ATOMIC32-NEXT:    sh zero, 10(sp)
+; NO-ATOMIC32-NEXT:    addi a1, sp, 10
+; NO-ATOMIC32-NEXT:    li a2, 1
+; NO-ATOMIC32-NEXT:    li a3, 5
+; NO-ATOMIC32-NEXT:    li a4, 5
+; NO-ATOMIC32-NEXT:    call __atomic_compare_exchange_2@plt
+; NO-ATOMIC32-NEXT:    lh a0, 10(sp)
+; NO-ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; NO-ATOMIC32-NEXT:    addi sp, sp, 16
+; NO-ATOMIC32-NEXT:    ret
+;
+; ATOMIC32-LABEL: cmpxchg16:
+; ATOMIC32:       # %bb.0:
+; ATOMIC32-NEXT:    addi sp, sp, -16
+; ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; ATOMIC32-NEXT:    li a2, 1
+; ATOMIC32-NEXT:    li a1, 0
+; ATOMIC32-NEXT:    call __sync_val_compare_and_swap_2@plt
+; ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; ATOMIC32-NEXT:    addi sp, sp, 16
+; ATOMIC32-NEXT:    ret
+  %res = cmpxchg ptr %p, i16 0, i16 1 seq_cst seq_cst
+  %res.0 = extractvalue { i16, i1 } %res, 0
+  ret i16 %res.0
+}
+
+define i32 @load32(ptr %p) nounwind {
+; NO-ATOMIC32-LABEL: load32:
+; NO-ATOMIC32:       # %bb.0:
+; NO-ATOMIC32-NEXT:    addi sp, sp, -16
+; NO-ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; NO-ATOMIC32-NEXT:    li a1, 5
+; NO-ATOMIC32-NEXT:    call __atomic_load_4@plt
+; NO-ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; NO-ATOMIC32-NEXT:    addi sp, sp, 16
+; NO-ATOMIC32-NEXT:    ret
+;
+; ATOMIC32-LABEL: load32:
+; ATOMIC32:       # %bb.0:
+; ATOMIC32-NEXT:    fence rw, rw
+; ATOMIC32-NEXT:    lw a0, 0(a0)
+; ATOMIC32-NEXT:    fence r, rw
+; ATOMIC32-NEXT:    ret
+  %v = load atomic i32, ptr %p seq_cst, align 4
+  ret i32 %v
+}
+
+define void @store32(ptr %p) nounwind {
+; NO-ATOMIC32-LABEL: store32:
+; NO-ATOMIC32:       # %bb.0:
+; NO-ATOMIC32-NEXT:    addi sp, sp, -16
+; NO-ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; NO-ATOMIC32-NEXT:    li a2, 5
+; NO-ATOMIC32-NEXT:    li a1, 0
+; NO-ATOMIC32-NEXT:    call __atomic_store_4@plt
+; NO-ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; NO-ATOMIC32-NEXT:    addi sp, sp, 16
+; NO-ATOMIC32-NEXT:    ret
+;
+; ATOMIC32-LABEL: store32:
+; ATOMIC32:       # %bb.0:
+; ATOMIC32-NEXT:    fence rw, w
+; ATOMIC32-NEXT:    sw zero, 0(a0)
+; ATOMIC32-NEXT:    ret
+  store atomic i32 0, ptr %p seq_cst, align 4
+  ret void
+}
+
+define i32 @rmw32(ptr %p) nounwind {
+; NO-ATOMIC32-LABEL: rmw32:
+; NO-ATOMIC32:       # %bb.0:
+; NO-ATOMIC32-NEXT:    addi sp, sp, -16
+; NO-ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; NO-ATOMIC32-NEXT:    li a1, 1
+; NO-ATOMIC32-NEXT:    li a2, 5
+; NO-ATOMIC32-NEXT:    call __atomic_fetch_add_4@plt
+; NO-ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; NO-ATOMIC32-NEXT:    addi sp, sp, 16
+; NO-ATOMIC32-NEXT:    ret
+;
+; ATOMIC32-LABEL: rmw32:
+; ATOMIC32:       # %bb.0:
+; ATOMIC32-NEXT:    addi sp, sp, -16
+; ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; ATOMIC32-NEXT:    li a1, 1
+; ATOMIC32-NEXT:    call __sync_fetch_and_add_4@plt
+; ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; ATOMIC32-NEXT:    addi sp, sp, 16
+; ATOMIC32-NEXT:    ret
+  %v = atomicrmw add ptr %p, i32 1 seq_cst, align 4
+  ret i32 %v
+}
+
+define i32 @cmpxchg32(ptr %p) nounwind {
+; NO-ATOMIC32-LABEL: cmpxchg32:
+; NO-ATOMIC32:       # %bb.0:
+; NO-ATOMIC32-NEXT:    addi sp, sp, -16
+; NO-ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; NO-ATOMIC32-NEXT:    sw zero, 8(sp)
+; NO-ATOMIC32-NEXT:    addi a1, sp, 8
+; NO-ATOMIC32-NEXT:    li a2, 1
+; NO-ATOMIC32-NEXT:    li a3, 5
+; NO-ATOMIC32-NEXT:    li a4, 5
+; NO-ATOMIC32-NEXT:    call __atomic_compare_exchange_4@plt
+; NO-ATOMIC32-NEXT:    lw a0, 8(sp)
+; NO-ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; NO-ATOMIC32-NEXT:    addi sp, sp, 16
+; NO-ATOMIC32-NEXT:    ret
+;
+; ATOMIC32-LABEL: cmpxchg32:
+; ATOMIC32:       # %bb.0:
+; ATOMIC32-NEXT:    addi sp, sp, -16
+; ATOMIC32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; ATOMIC32-NEXT:    li a2, 1
+; ATOMIC32-NEXT:    li a1, 0
+; ATOMIC32-NEXT:    call __sync_val_compare_and_swap_4@plt
+; ATOMIC32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; ATOMIC32-NEXT:    addi sp, sp, 16
+; ATOMIC32-NEXT:    ret
+  %res = cmpxchg ptr %p, i32 0, i32 1 seq_cst seq_cst
+  %res.0 = extractvalue { i32, i1 } %res, 0
+  ret i32 %res.0
+}
+
+define i64 @load64(ptr %p) nounwind {
+; CHECK-LABEL: load64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -16
+; CHECK-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; CHECK-NEXT:    li a1, 5
+; CHECK-NEXT:    call __atomic_load_8@plt
+; CHECK-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 16
+; CHECK-NEXT:    ret
+  %v = load atomic i64, ptr %p seq_cst, align 8
+  ret i64 %v
+}
+
+define void @store64(ptr %p) nounwind {
+; CHECK-LABEL: store64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -16
+; CHECK-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; CHECK-NEXT:    li a3, 5
+; CHECK-NEXT:    li a1, 0
+; CHECK-NEXT:    li a2, 0
+; CHECK-NEXT:    call __atomic_store_8@plt
+; CHECK-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 16
+; CHECK-NEXT:    ret
+  store atomic i64 0, ptr %p seq_cst, align 8
+  ret void
+}
+
+define i64 @rmw64(ptr %p) nounwind {
+; CHECK-LABEL: rmw64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -16
+; CHECK-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; CHECK-NEXT:    li a1, 1
+; CHECK-NEXT:    li a3, 5
+; CHECK-NEXT:    li a2, 0
+; CHECK-NEXT:    call __atomic_fetch_add_8@plt
+; CHECK-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 16
+; CHECK-NEXT:    ret
+  %v = atomicrmw add ptr %p, i64 1 seq_cst, align 8
+  ret i64 %v
+}
+
+define i64 @cmpxchg64(ptr %p) nounwind {
+; CHECK-LABEL: cmpxchg64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -16
+; CHECK-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; CHECK-NEXT:    sw zero, 4(sp)
+; CHECK-NEXT:    sw zero, 0(sp)
+; CHECK-NEXT:    mv a1, sp
+; CHECK-NEXT:    li a2, 1
+; CHECK-NEXT:    li a4, 5
+; CHECK-NEXT:    li a5, 5
+; CHECK-NEXT:    li a3, 0
+; CHECK-NEXT:    call __atomic_compare_exchange_8@plt
+; CHECK-NEXT:    lw a1, 4(sp)
+; CHECK-NEXT:    lw a0, 0(sp)
+; CHECK-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; CHECK-NEXT:    addi sp, sp, 16
+; CHECK-NEXT:    ret
+  %res = cmpxchg ptr %p, i64 0, i64 1 seq_cst seq_cst
+  %res.0 = extractvalue { i64, i1 } %res, 0
+  ret i64 %res.0
+}