[RISCV] Implement a proposed mapping for Ztso

[reames]

Summary

This change implements a proposed lowering from LLVM's memory model to the TSO memory model defined by the Ztso extension. Selecting the proposed mapping turns out to be an involved conversation that really didn't fit within a review description, so let me refer you to https://github.com/preames/public-notes/blob/master/riscv-tso-mappings.rst. This review implements the WMO compatible variant (the proposed one in that document).

Ztso is currently accepted as an experimental extension in LLVM. Despite the fact the extension was recently ratified, I think we need to leave it as experimental until we have wide agreement on the chosen mapping for ABI purposes.

I need to note that the current in-tree implementation defaults to generating WMO compatible fences. This is entirely compatible with the proposed mapping in this patch, but is unfortunately not compatible with the major alternative. The in tree implementation is explicitly experimental so the impact of this is limited, but it is worth calling out that if settle on the alternative we will have a minor ABI break. My apologies for not calling this out in the original patch; I had not realized at the time that one of our realistic choices for mappings wouldn't be WMO compatible.

This patch only contains the changes for load/store and fence. That is, it does not change the lowering for atomicrmw operations. This is a sound thing to do under the proposed mapping since the existing WMO mappings remain compatible. I do plan to change these; I'm just working incrementally.

[asb]

Thanks for writing up such a clear summary of the options for this mapping. Thoughts on the conceptual change:

• You're right to flag the ABI break, but I think we shouldn't worry about this - after all that's what the experimental flag is for.
• Compatibility with RVWMO seems like such a big win to me it would need a _really_ strong evidence-backed argument to prefer the alternate mapping.
  • Perhaps not overly relevant to the current day discussion, but wonder to what extent this issue was considered within the original memory model working group. X86 "compatibility" was definitely raised explicitly, but I didn't interpret this to necessarily mean naive porting of X86 assembly (more e.g. DBTs)
• There are enough mutually incompatible ABI variants for RISC-V already, I like the fact it seems like this lowering avoids introducing another one.

In terms of code review (all very minor points - it's a straightforward patch as all the work is clearly in the lowerings and the ecosystem tradeoffs mentioned above):

• This needs an edit to RISCVUsage.rst (which currently says ztso doesn't yet change code generation)
• I agree with the decision to just stick an if(ztso) at the beginning of the relevant methods vs trying to more tightly integrate the logic (which you could do in emitLeadingFence but I don't think it would be clearer and the LOC difference is negligible
• You could have WMO/TSO/BOTH (or WMO/TSO/CHECK) check prefixes on atomic-fence.ll to make it easier to see when the lowerings are the same. Perhaps the same could be done in atomic-load-store.ll too (RV64IA-TSO, change the existing to RV64IA-WMO, and add an RV64IA-BOTH or similar).

[reames]

Address review comments from @asb

[asb]

The code changes LGTM.

As this has been floated in the RISC-V LLVM contributor call, and it's behind an experimental flag anyway (i.e. we can still change tack if future discussion shows there's huge opposition to the proposed lowering), I've got no opposition to merging this sooner rather than later and don't know of other ztso stakeholders working upstream who might be inconvenienced in either case.

Is your preference to try to get this merged in the near future, or would you rather wait for more feedback on the lowerings? (I'm pretty agnostic, for the reasons above).

[reames]

In D143076#4113573, @asb wrote:

Is your preference to try to get this merged in the near future, or would you rather wait for more feedback on the lowerings? (I'm pretty agnostic, for the reasons above).

My preference is to merge, but it's not a hugely strong one.

[asb]

In D143076#4113617, @reames wrote:

In D143076#4113573, @asb wrote:

Is your preference to try to get this merged in the near future, or would you rather wait for more feedback on the lowerings? (I'm pretty agnostic, for the reasons above).

My preference is to merge, but it's not a hugely strong one.

I'm happy for this to merge. It's been up for a week and discussed in the call with no concerns raised, and the whole point (from my perspective at least) of accepting patches for accepting patches under the experimental extensions flag was to enable better upstream collaboration rather than having things sit downstream.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

3735

Was this whitespace change intentional?

3736

Drop curly braces

llvm/docs/RISCVUsage.rst

 
 ``experimental-zcb``
   LLVM implements the `0.70 draft specification <https://github.com/riscv/riscv-code-size-reduction/releases/tag/V0.70.1-TOOLCHAIN-DEV>`_.
 
 ``experimental-zihintntl``
   LLVM implements the `0.2 draft specification <https://github.com/riscv/riscv-isa-manual/releases/tag/draft-20220831-bf5a151>`_.
 
 ``experimental-ztso``
-  LLVM implements the `v0.1 proposed specification <https://github.com/riscv/riscv-isa-manual/releases/download/draft-20220723-10eea63/riscv-spec.pdf>`_ (see Chapter 25).  Using will set appropriate ELF flags and attributes, but does not yet change code generation.
+  LLVM implements the `v0.1 proposed specification <https://github.com/riscv/riscv-isa-manual/releases/download/draft-20220723-10eea63/riscv-spec.pdf>`_ (see Chapter 25).  The mapping from the C/C++ memory model to Ztso has not yet been ratified in any standards document.  There are multiple possible mappings, and they are *not* mutually ABI compatible.  The mapping LLVM implements is ABI compatible with the default WMO mapping.  This mapping may change and there is *explicitly* no ABI stability offered while the extension remains in experimental status.  User beware.
 
 ``experimental-zvfh``
   LLVM implements `this draft text <https://github.com/riscv/riscv-v-spec/pull/780>`_.
 
 To use an experimental extension from `clang`, you must add `-menable-experimental-extensions` to the command line, and specify the exact version of the experimental extension you are using.  To use an experimental extension with LLVM's internal developer tools (e.g. `llc`, `llvm-objdump`, `llvm-mc`), you must prefix the extension name with `experimental-`.  Note that you don't need to specify the version with internal tools, and shouldn't include the `experimental-` prefix with `clang`.
 
 Vendor Extensions
 =================

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

RISCVMatInt::InstSeq Seq =
       RISCVMatInt::generateInstSeq(Imm, Subtarget.getFeatureBits());
   if (Seq.size() <= Subtarget.getMaxBuildIntsCost())
     return Op;
 
   // Expand to a constant pool using the default expansion code.
   return SDValue();
 }
 
-static SDValue LowerATOMIC_FENCE(SDValue Op, SelectionDAG &DAG) {
+static SDValue LowerATOMIC_FENCE(SDValue Op, SelectionDAG &DAG,
+                                 const RISCVSubtarget &Subtarget) {
   SDLoc dl(Op);
+  AtomicOrdering FenceOrdering =
+      static_cast<AtomicOrdering>(Op.getConstantOperandVal(1));
   SyncScope::ID FenceSSID =
       static_cast<SyncScope::ID>(Op.getConstantOperandVal(2));
 
+  if (Subtarget.hasStdExtZtso()) {
+    // The only fence that needs an instruction is a sequentially-consistent
+    // cross-thread fence.
+    if (FenceOrdering == AtomicOrdering::SequentiallyConsistent &&
+        FenceSSID == SyncScope::System)
+      return Op;
+
+    // MEMBARRIER is a compiler barrier; it codegens to a no-op.
+    return DAG.getNode(ISD::MEMBARRIER, dl, MVT::Other, Op.getOperand(0));
+  }
+
   // singlethread fences only synchronize with signal handlers on the same
   // thread and thus only need to preserve instruction order, not actually
   // enforce memory ordering.
   if (FenceSSID == SyncScope::SingleThread)
     // MEMBARRIER is a compiler barrier; it codegens to a no-op.
     return DAG.getNode(ISD::MEMBARRIER, dl, MVT::Other, Op.getOperand(0));
 
   return Op;
 }
 
 SDValue RISCVTargetLowering::LowerOperation(SDValue Op,
                                             SelectionDAG &DAG) const {
   switch (Op.getOpcode()) {
   default:
     report_fatal_error("unimplemented operand");
   case ISD::ATOMIC_FENCE:
-    return LowerATOMIC_FENCE(Op, DAG);
+    return LowerATOMIC_FENCE(Op, DAG, Subtarget);
   case ISD::GlobalAddress:
     return lowerGlobalAddress(Op, DAG);
   case ISD::BlockAddress:
     return lowerBlockAddress(Op, DAG);
   case ISD::ConstantPool:
     return lowerConstantPool(Op, DAG);
   case ISD::JumpTable:
     return lowerJumpTable(Op, DAG);
if (Constraint.length() == 1) {
     }
   }
   TargetLowering::LowerAsmOperandForConstraint(Op, Constraint, Ops, DAG);
 }
 
 Instruction *RISCVTargetLowering::emitLeadingFence(IRBuilderBase &Builder,
                                                    Instruction *Inst,
                                                    AtomicOrdering Ord) const {
+  if (Subtarget.hasStdExtZtso()) {
+    if (isa<LoadInst>(Inst) && Ord == AtomicOrdering::SequentiallyConsistent)
+      return Builder.CreateFence(Ord);
+    return nullptr;
+  }
+
   if (isa<LoadInst>(Inst) && Ord == AtomicOrdering::SequentiallyConsistent)
     return Builder.CreateFence(Ord);
   if (isa<StoreInst>(Inst) && isReleaseOrStronger(Ord))
     return Builder.CreateFence(AtomicOrdering::Release);
   return nullptr;
 }
 
 Instruction *RISCVTargetLowering::emitTrailingFence(IRBuilderBase &Builder,
                                                     Instruction *Inst,
                                                     AtomicOrdering Ord) const {
+  if (Subtarget.hasStdExtZtso())
+    return nullptr;
+
   if (isa<LoadInst>(Inst) && isAcquireOrStronger(Ord))
     return Builder.CreateFence(AtomicOrdering::Acquire);
   return nullptr;
 }
 
 TargetLowering::AtomicExpansionKind
 RISCVTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
   // atomicrmw {fadd,fsub} must be expanded to use compare-exchange, as floating

llvm/test/CodeGen/RISCV/atomic-fence.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=riscv32 -verify-machineinstrs < %s \
-; RUN:   | FileCheck %s
+; RUN:   | FileCheck --check-prefixes=CHECK,WMO %s
 ; RUN: llc -mtriple=riscv32 -mattr=+a -verify-machineinstrs < %s \
-; RUN:   | FileCheck %s
+; RUN:   | FileCheck --check-prefixes=CHECK,WMO %s
 ; RUN: llc -mtriple=riscv32 -mattr=+a,+experimental-ztso -verify-machineinstrs < %s \
-; RUN:   | FileCheck %s
+; RUN:   | FileCheck --check-prefixes=CHECK,TSO %s
 ; RUN: llc -mtriple=riscv64 -verify-machineinstrs < %s \
-; RUN:   | FileCheck %s
+; RUN:   | FileCheck --check-prefixes=CHECK,WMO %s
 ; RUN: llc -mtriple=riscv64 -mattr=+a -verify-machineinstrs < %s \
-; RUN:   | FileCheck %s
+; RUN:   | FileCheck --check-prefixes=CHECK,WMO %s
 ; RUN: llc -mtriple=riscv64 -mattr=+a,+experimental-ztso -verify-machineinstrs < %s \
-; RUN:   | FileCheck %s
+; RUN:   | FileCheck --check-prefixes=CHECK,TSO %s
 
 define void @fence_acquire() nounwind {
-; CHECK-LABEL: fence_acquire:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    fence r, rw
-; CHECK-NEXT:    ret
+; WMO-LABEL: fence_acquire:
+; WMO:       # %bb.0:
+; WMO-NEXT:    fence r, rw
+; WMO-NEXT:    ret
+;
+; TSO-LABEL: fence_acquire:
+; TSO:       # %bb.0:
+; TSO-NEXT:    #MEMBARRIER
+; TSO-NEXT:    ret
   fence acquire
   ret void
 }
 
 define void @fence_release() nounwind {
-; CHECK-LABEL: fence_release:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    fence rw, w
-; CHECK-NEXT:    ret
+; WMO-LABEL: fence_release:
+; WMO:       # %bb.0:
+; WMO-NEXT:    fence rw, w
+; WMO-NEXT:    ret
+;
+; TSO-LABEL: fence_release:
+; TSO:       # %bb.0:
+; TSO-NEXT:    #MEMBARRIER
+; TSO-NEXT:    ret
   fence release
   ret void
 }
 
 define void @fence_acq_rel() nounwind {
-; CHECK-LABEL: fence_acq_rel:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    fence.tso
-; CHECK-NEXT:    ret
+; WMO-LABEL: fence_acq_rel:
+; WMO:       # %bb.0:
+; WMO-NEXT:    fence.tso
+; WMO-NEXT:    ret
+;
+; TSO-LABEL: fence_acq_rel:
+; TSO:       # %bb.0:
+; TSO-NEXT:    #MEMBARRIER
+; TSO-NEXT:    ret
   fence acq_rel
   ret void
 }
 
 define void @fence_seq_cst() nounwind {
 ; CHECK-LABEL: fence_seq_cst:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    fence rw, rw

llvm/test/CodeGen/RISCV/atomic-load-store.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=riscv32 -verify-machineinstrs < %s \
 ; RUN:   | FileCheck -check-prefix=RV32I %s
 ; RUN: llc -mtriple=riscv32 -mattr=+a -verify-machineinstrs < %s \
-; RUN:   | FileCheck -check-prefix=RV32IA %s
+; RUN:   | FileCheck -check-prefixes=RV32IA,RV32IA-WMO %s
 ; RUN: llc -mtriple=riscv32 -mattr=+a,+experimental-ztso -verify-machineinstrs < %s \
-; RUN:   | FileCheck -check-prefix=RV32IA %s
+; RUN:   | FileCheck -check-prefixes=RV32IA,RV32IA-TSO %s
 ; RUN: llc -mtriple=riscv64 -verify-machineinstrs < %s \
 ; RUN:   | FileCheck -check-prefix=RV64I %s
 ; RUN: llc -mtriple=riscv64 -mattr=+a -verify-machineinstrs < %s \
-; RUN:   | FileCheck -check-prefix=RV64IA %s
+; RUN:   | FileCheck -check-prefixes=RV64IA,RV64IA-WMO %s
 ; RUN: llc -mtriple=riscv64 -mattr=+a,+experimental-ztso -verify-machineinstrs < %s \
-; RUN:   | FileCheck -check-prefix=RV64IA %s
+; RUN:   | FileCheck -check-prefixes=RV64IA,RV64IA-TSO %s
 
 define i8 @atomic_load_i8_unordered(ptr %a) nounwind {
 ; RV32I-LABEL: atomic_load_i8_unordered:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    li a1, 0
 ; RV32I-NEXT:    call __atomic_load_1@plt

 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    li a1, 2
 ; RV32I-NEXT:    call __atomic_load_1@plt
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
 ;
-; RV32IA-LABEL: atomic_load_i8_acquire:
-; RV32IA:       # %bb.0:
-; RV32IA-NEXT:    lb a0, 0(a0)
-; RV32IA-NEXT:    fence r, rw
-; RV32IA-NEXT:    ret
+; RV32IA-WMO-LABEL: atomic_load_i8_acquire:
+; RV32IA-WMO:       # %bb.0:
+; RV32IA-WMO-NEXT:    lb a0, 0(a0)
+; RV32IA-WMO-NEXT:    fence r, rw
+; RV32IA-WMO-NEXT:    ret
+;
+; RV32IA-TSO-LABEL: atomic_load_i8_acquire:
+; RV32IA-TSO:       # %bb.0:
+; RV32IA-TSO-NEXT:    lb a0, 0(a0)
+; RV32IA-TSO-NEXT:    ret
 ;
 ; RV64I-LABEL: atomic_load_i8_acquire:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a1, 2
 ; RV64I-NEXT:    call __atomic_load_1@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_load_i8_acquire:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    lb a0, 0(a0)
-; RV64IA-NEXT:    fence r, rw
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_load_i8_acquire:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    lb a0, 0(a0)
+; RV64IA-WMO-NEXT:    fence r, rw
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_load_i8_acquire:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    lb a0, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   %1 = load atomic i8, ptr %a acquire, align 1
   ret i8 %1
 }
 
 define i8 @atomic_load_i8_seq_cst(ptr %a) nounwind {
 ; RV32I-LABEL: atomic_load_i8_seq_cst:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    li a1, 5
 ; RV32I-NEXT:    call __atomic_load_1@plt
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
 ;
-; RV32IA-LABEL: atomic_load_i8_seq_cst:
-; RV32IA:       # %bb.0:
-; RV32IA-NEXT:    fence rw, rw
-; RV32IA-NEXT:    lb a0, 0(a0)
-; RV32IA-NEXT:    fence r, rw
-; RV32IA-NEXT:    ret
+; RV32IA-WMO-LABEL: atomic_load_i8_seq_cst:
+; RV32IA-WMO:       # %bb.0:
+; RV32IA-WMO-NEXT:    fence rw, rw
+; RV32IA-WMO-NEXT:    lb a0, 0(a0)
+; RV32IA-WMO-NEXT:    fence r, rw
+; RV32IA-WMO-NEXT:    ret
+;
+; RV32IA-TSO-LABEL: atomic_load_i8_seq_cst:
+; RV32IA-TSO:       # %bb.0:
+; RV32IA-TSO-NEXT:    fence rw, rw
+; RV32IA-TSO-NEXT:    lb a0, 0(a0)
+; RV32IA-TSO-NEXT:    ret
 ;
 ; RV64I-LABEL: atomic_load_i8_seq_cst:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a1, 5
 ; RV64I-NEXT:    call __atomic_load_1@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_load_i8_seq_cst:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    fence rw, rw
-; RV64IA-NEXT:    lb a0, 0(a0)
-; RV64IA-NEXT:    fence r, rw
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_load_i8_seq_cst:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    fence rw, rw
+; RV64IA-WMO-NEXT:    lb a0, 0(a0)
+; RV64IA-WMO-NEXT:    fence r, rw
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_load_i8_seq_cst:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    fence rw, rw
+; RV64IA-TSO-NEXT:    lb a0, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   %1 = load atomic i8, ptr %a seq_cst, align 1
   ret i8 %1
 }
 
 define i16 @atomic_load_i16_unordered(ptr %a) nounwind {
 ; RV32I-LABEL: atomic_load_i16_unordered:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16

 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    li a1, 2
 ; RV32I-NEXT:    call __atomic_load_2@plt
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
 ;
-; RV32IA-LABEL: atomic_load_i16_acquire:
-; RV32IA:       # %bb.0:
-; RV32IA-NEXT:    lh a0, 0(a0)
-; RV32IA-NEXT:    fence r, rw
-; RV32IA-NEXT:    ret
+; RV32IA-WMO-LABEL: atomic_load_i16_acquire:
+; RV32IA-WMO:       # %bb.0:
+; RV32IA-WMO-NEXT:    lh a0, 0(a0)
+; RV32IA-WMO-NEXT:    fence r, rw
+; RV32IA-WMO-NEXT:    ret
+;
+; RV32IA-TSO-LABEL: atomic_load_i16_acquire:
+; RV32IA-TSO:       # %bb.0:
+; RV32IA-TSO-NEXT:    lh a0, 0(a0)
+; RV32IA-TSO-NEXT:    ret
 ;
 ; RV64I-LABEL: atomic_load_i16_acquire:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a1, 2
 ; RV64I-NEXT:    call __atomic_load_2@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_load_i16_acquire:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    lh a0, 0(a0)
-; RV64IA-NEXT:    fence r, rw
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_load_i16_acquire:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    lh a0, 0(a0)
+; RV64IA-WMO-NEXT:    fence r, rw
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_load_i16_acquire:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    lh a0, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   %1 = load atomic i16, ptr %a acquire, align 2
   ret i16 %1
 }
 
 define i16 @atomic_load_i16_seq_cst(ptr %a) nounwind {
 ; RV32I-LABEL: atomic_load_i16_seq_cst:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    li a1, 5
 ; RV32I-NEXT:    call __atomic_load_2@plt
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
 ;
-; RV32IA-LABEL: atomic_load_i16_seq_cst:
-; RV32IA:       # %bb.0:
-; RV32IA-NEXT:    fence rw, rw
-; RV32IA-NEXT:    lh a0, 0(a0)
-; RV32IA-NEXT:    fence r, rw
-; RV32IA-NEXT:    ret
+; RV32IA-WMO-LABEL: atomic_load_i16_seq_cst:
+; RV32IA-WMO:       # %bb.0:
+; RV32IA-WMO-NEXT:    fence rw, rw
+; RV32IA-WMO-NEXT:    lh a0, 0(a0)
+; RV32IA-WMO-NEXT:    fence r, rw
+; RV32IA-WMO-NEXT:    ret
+;
+; RV32IA-TSO-LABEL: atomic_load_i16_seq_cst:
+; RV32IA-TSO:       # %bb.0:
+; RV32IA-TSO-NEXT:    fence rw, rw
+; RV32IA-TSO-NEXT:    lh a0, 0(a0)
+; RV32IA-TSO-NEXT:    ret
 ;
 ; RV64I-LABEL: atomic_load_i16_seq_cst:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a1, 5
 ; RV64I-NEXT:    call __atomic_load_2@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_load_i16_seq_cst:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    fence rw, rw
-; RV64IA-NEXT:    lh a0, 0(a0)
-; RV64IA-NEXT:    fence r, rw
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_load_i16_seq_cst:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    fence rw, rw
+; RV64IA-WMO-NEXT:    lh a0, 0(a0)
+; RV64IA-WMO-NEXT:    fence r, rw
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_load_i16_seq_cst:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    fence rw, rw
+; RV64IA-TSO-NEXT:    lh a0, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   %1 = load atomic i16, ptr %a seq_cst, align 2
   ret i16 %1
 }
 
 define i32 @atomic_load_i32_unordered(ptr %a) nounwind {
 ; RV32I-LABEL: atomic_load_i32_unordered:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16

 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    li a1, 2
 ; RV32I-NEXT:    call __atomic_load_4@plt
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
 ;
-; RV32IA-LABEL: atomic_load_i32_acquire:
-; RV32IA:       # %bb.0:
-; RV32IA-NEXT:    lw a0, 0(a0)
-; RV32IA-NEXT:    fence r, rw
-; RV32IA-NEXT:    ret
+; RV32IA-WMO-LABEL: atomic_load_i32_acquire:
+; RV32IA-WMO:       # %bb.0:
+; RV32IA-WMO-NEXT:    lw a0, 0(a0)
+; RV32IA-WMO-NEXT:    fence r, rw
+; RV32IA-WMO-NEXT:    ret
+;
+; RV32IA-TSO-LABEL: atomic_load_i32_acquire:
+; RV32IA-TSO:       # %bb.0:
+; RV32IA-TSO-NEXT:    lw a0, 0(a0)
+; RV32IA-TSO-NEXT:    ret
 ;
 ; RV64I-LABEL: atomic_load_i32_acquire:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a1, 2
 ; RV64I-NEXT:    call __atomic_load_4@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_load_i32_acquire:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    lw a0, 0(a0)
-; RV64IA-NEXT:    fence r, rw
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_load_i32_acquire:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    lw a0, 0(a0)
+; RV64IA-WMO-NEXT:    fence r, rw
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_load_i32_acquire:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    lw a0, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   %1 = load atomic i32, ptr %a acquire, align 4
   ret i32 %1
 }
 
 define i32 @atomic_load_i32_seq_cst(ptr %a) nounwind {
 ; RV32I-LABEL: atomic_load_i32_seq_cst:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    li a1, 5
 ; RV32I-NEXT:    call __atomic_load_4@plt
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
 ;
-; RV32IA-LABEL: atomic_load_i32_seq_cst:
-; RV32IA:       # %bb.0:
-; RV32IA-NEXT:    fence rw, rw
-; RV32IA-NEXT:    lw a0, 0(a0)
-; RV32IA-NEXT:    fence r, rw
-; RV32IA-NEXT:    ret
+; RV32IA-WMO-LABEL: atomic_load_i32_seq_cst:
+; RV32IA-WMO:       # %bb.0:
+; RV32IA-WMO-NEXT:    fence rw, rw
+; RV32IA-WMO-NEXT:    lw a0, 0(a0)
+; RV32IA-WMO-NEXT:    fence r, rw
+; RV32IA-WMO-NEXT:    ret
+;
+; RV32IA-TSO-LABEL: atomic_load_i32_seq_cst:
+; RV32IA-TSO:       # %bb.0:
+; RV32IA-TSO-NEXT:    fence rw, rw
+; RV32IA-TSO-NEXT:    lw a0, 0(a0)
+; RV32IA-TSO-NEXT:    ret
 ;
 ; RV64I-LABEL: atomic_load_i32_seq_cst:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a1, 5
 ; RV64I-NEXT:    call __atomic_load_4@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_load_i32_seq_cst:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    fence rw, rw
-; RV64IA-NEXT:    lw a0, 0(a0)
-; RV64IA-NEXT:    fence r, rw
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_load_i32_seq_cst:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    fence rw, rw
+; RV64IA-WMO-NEXT:    lw a0, 0(a0)
+; RV64IA-WMO-NEXT:    fence r, rw
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_load_i32_seq_cst:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    fence rw, rw
+; RV64IA-TSO-NEXT:    lw a0, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   %1 = load atomic i32, ptr %a seq_cst, align 4
   ret i32 %1
 }
 
 define i64 @atomic_load_i64_unordered(ptr %a) nounwind {
 ; RV32I-LABEL: atomic_load_i64_unordered:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16

 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a1, 2
 ; RV64I-NEXT:    call __atomic_load_8@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_load_i64_acquire:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    ld a0, 0(a0)
-; RV64IA-NEXT:    fence r, rw
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_load_i64_acquire:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    ld a0, 0(a0)
+; RV64IA-WMO-NEXT:    fence r, rw
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_load_i64_acquire:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    ld a0, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   %1 = load atomic i64, ptr %a acquire, align 8
   ret i64 %1
 }
 
 define i64 @atomic_load_i64_seq_cst(ptr %a) nounwind {
 ; RV32I-LABEL: atomic_load_i64_seq_cst:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16

 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a1, 5
 ; RV64I-NEXT:    call __atomic_load_8@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_load_i64_seq_cst:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    fence rw, rw
-; RV64IA-NEXT:    ld a0, 0(a0)
-; RV64IA-NEXT:    fence r, rw
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_load_i64_seq_cst:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    fence rw, rw
+; RV64IA-WMO-NEXT:    ld a0, 0(a0)
+; RV64IA-WMO-NEXT:    fence r, rw
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_load_i64_seq_cst:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    fence rw, rw
+; RV64IA-TSO-NEXT:    ld a0, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   %1 = load atomic i64, ptr %a seq_cst, align 8
   ret i64 %1
 }
 
 define void @atomic_store_i8_unordered(ptr %a, i8 %b) nounwind {
 ; RV32I-LABEL: atomic_store_i8_unordered:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16

 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    li a2, 3
 ; RV32I-NEXT:    call __atomic_store_1@plt
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
 ;
-; RV32IA-LABEL: atomic_store_i8_release:
-; RV32IA:       # %bb.0:
-; RV32IA-NEXT:    fence rw, w
-; RV32IA-NEXT:    sb a1, 0(a0)
-; RV32IA-NEXT:    ret
+; RV32IA-WMO-LABEL: atomic_store_i8_release:
+; RV32IA-WMO:       # %bb.0:
+; RV32IA-WMO-NEXT:    fence rw, w
+; RV32IA-WMO-NEXT:    sb a1, 0(a0)
+; RV32IA-WMO-NEXT:    ret
+;
+; RV32IA-TSO-LABEL: atomic_store_i8_release:
+; RV32IA-TSO:       # %bb.0:
+; RV32IA-TSO-NEXT:    sb a1, 0(a0)
+; RV32IA-TSO-NEXT:    ret
 ;
 ; RV64I-LABEL: atomic_store_i8_release:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a2, 3
 ; RV64I-NEXT:    call __atomic_store_1@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_store_i8_release:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    fence rw, w
-; RV64IA-NEXT:    sb a1, 0(a0)
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_store_i8_release:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    fence rw, w
+; RV64IA-WMO-NEXT:    sb a1, 0(a0)
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_store_i8_release:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    sb a1, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   store atomic i8 %b, ptr %a release, align 1
   ret void
 }
 
 define void @atomic_store_i8_seq_cst(ptr %a, i8 %b) nounwind {
 ; RV32I-LABEL: atomic_store_i8_seq_cst:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    li a2, 5
 ; RV32I-NEXT:    call __atomic_store_1@plt
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
 ;
-; RV32IA-LABEL: atomic_store_i8_seq_cst:
-; RV32IA:       # %bb.0:
-; RV32IA-NEXT:    fence rw, w
-; RV32IA-NEXT:    sb a1, 0(a0)
-; RV32IA-NEXT:    ret
+; RV32IA-WMO-LABEL: atomic_store_i8_seq_cst:
+; RV32IA-WMO:       # %bb.0:
+; RV32IA-WMO-NEXT:    fence rw, w
+; RV32IA-WMO-NEXT:    sb a1, 0(a0)
+; RV32IA-WMO-NEXT:    ret
+;
+; RV32IA-TSO-LABEL: atomic_store_i8_seq_cst:
+; RV32IA-TSO:       # %bb.0:
+; RV32IA-TSO-NEXT:    sb a1, 0(a0)
+; RV32IA-TSO-NEXT:    ret
 ;
 ; RV64I-LABEL: atomic_store_i8_seq_cst:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a2, 5
 ; RV64I-NEXT:    call __atomic_store_1@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_store_i8_seq_cst:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    fence rw, w
-; RV64IA-NEXT:    sb a1, 0(a0)
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_store_i8_seq_cst:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    fence rw, w
+; RV64IA-WMO-NEXT:    sb a1, 0(a0)
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_store_i8_seq_cst:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    sb a1, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   store atomic i8 %b, ptr %a seq_cst, align 1
   ret void
 }
 
 define void @atomic_store_i16_unordered(ptr %a, i16 %b) nounwind {
 ; RV32I-LABEL: atomic_store_i16_unordered:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16

 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    li a2, 3
 ; RV32I-NEXT:    call __atomic_store_2@plt
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
 ;
-; RV32IA-LABEL: atomic_store_i16_release:
-; RV32IA:       # %bb.0:
-; RV32IA-NEXT:    fence rw, w
-; RV32IA-NEXT:    sh a1, 0(a0)
-; RV32IA-NEXT:    ret
+; RV32IA-WMO-LABEL: atomic_store_i16_release:
+; RV32IA-WMO:       # %bb.0:
+; RV32IA-WMO-NEXT:    fence rw, w
+; RV32IA-WMO-NEXT:    sh a1, 0(a0)
+; RV32IA-WMO-NEXT:    ret
+;
+; RV32IA-TSO-LABEL: atomic_store_i16_release:
+; RV32IA-TSO:       # %bb.0:
+; RV32IA-TSO-NEXT:    sh a1, 0(a0)
+; RV32IA-TSO-NEXT:    ret
 ;
 ; RV64I-LABEL: atomic_store_i16_release:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a2, 3
 ; RV64I-NEXT:    call __atomic_store_2@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_store_i16_release:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    fence rw, w
-; RV64IA-NEXT:    sh a1, 0(a0)
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_store_i16_release:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    fence rw, w
+; RV64IA-WMO-NEXT:    sh a1, 0(a0)
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_store_i16_release:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    sh a1, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   store atomic i16 %b, ptr %a release, align 2
   ret void
 }
 
 define void @atomic_store_i16_seq_cst(ptr %a, i16 %b) nounwind {
 ; RV32I-LABEL: atomic_store_i16_seq_cst:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    li a2, 5
 ; RV32I-NEXT:    call __atomic_store_2@plt
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
 ;
-; RV32IA-LABEL: atomic_store_i16_seq_cst:
-; RV32IA:       # %bb.0:
-; RV32IA-NEXT:    fence rw, w
-; RV32IA-NEXT:    sh a1, 0(a0)
-; RV32IA-NEXT:    ret
+; RV32IA-WMO-LABEL: atomic_store_i16_seq_cst:
+; RV32IA-WMO:       # %bb.0:
+; RV32IA-WMO-NEXT:    fence rw, w
+; RV32IA-WMO-NEXT:    sh a1, 0(a0)
+; RV32IA-WMO-NEXT:    ret
+;
+; RV32IA-TSO-LABEL: atomic_store_i16_seq_cst:
+; RV32IA-TSO:       # %bb.0:
+; RV32IA-TSO-NEXT:    sh a1, 0(a0)
+; RV32IA-TSO-NEXT:    ret
 ;
 ; RV64I-LABEL: atomic_store_i16_seq_cst:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a2, 5
 ; RV64I-NEXT:    call __atomic_store_2@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_store_i16_seq_cst:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    fence rw, w
-; RV64IA-NEXT:    sh a1, 0(a0)
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_store_i16_seq_cst:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    fence rw, w
+; RV64IA-WMO-NEXT:    sh a1, 0(a0)
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_store_i16_seq_cst:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    sh a1, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   store atomic i16 %b, ptr %a seq_cst, align 2
   ret void
 }
 
 define void @atomic_store_i32_unordered(ptr %a, i32 %b) nounwind {
 ; RV32I-LABEL: atomic_store_i32_unordered:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16

 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    li a2, 3
 ; RV32I-NEXT:    call __atomic_store_4@plt
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
 ;
-; RV32IA-LABEL: atomic_store_i32_release:
-; RV32IA:       # %bb.0:
-; RV32IA-NEXT:    fence rw, w
-; RV32IA-NEXT:    sw a1, 0(a0)
-; RV32IA-NEXT:    ret
+; RV32IA-WMO-LABEL: atomic_store_i32_release:
+; RV32IA-WMO:       # %bb.0:
+; RV32IA-WMO-NEXT:    fence rw, w
+; RV32IA-WMO-NEXT:    sw a1, 0(a0)
+; RV32IA-WMO-NEXT:    ret
+;
+; RV32IA-TSO-LABEL: atomic_store_i32_release:
+; RV32IA-TSO:       # %bb.0:
+; RV32IA-TSO-NEXT:    sw a1, 0(a0)
+; RV32IA-TSO-NEXT:    ret
 ;
 ; RV64I-LABEL: atomic_store_i32_release:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a2, 3
 ; RV64I-NEXT:    call __atomic_store_4@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_store_i32_release:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    fence rw, w
-; RV64IA-NEXT:    sw a1, 0(a0)
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_store_i32_release:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    fence rw, w
+; RV64IA-WMO-NEXT:    sw a1, 0(a0)
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_store_i32_release:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    sw a1, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   store atomic i32 %b, ptr %a release, align 4
   ret void
 }
 
 define void @atomic_store_i32_seq_cst(ptr %a, i32 %b) nounwind {
 ; RV32I-LABEL: atomic_store_i32_seq_cst:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    li a2, 5
 ; RV32I-NEXT:    call __atomic_store_4@plt
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
 ;
-; RV32IA-LABEL: atomic_store_i32_seq_cst:
-; RV32IA:       # %bb.0:
-; RV32IA-NEXT:    fence rw, w
-; RV32IA-NEXT:    sw a1, 0(a0)
-; RV32IA-NEXT:    ret
+; RV32IA-WMO-LABEL: atomic_store_i32_seq_cst:
+; RV32IA-WMO:       # %bb.0:
+; RV32IA-WMO-NEXT:    fence rw, w
+; RV32IA-WMO-NEXT:    sw a1, 0(a0)
+; RV32IA-WMO-NEXT:    ret
+;
+; RV32IA-TSO-LABEL: atomic_store_i32_seq_cst:
+; RV32IA-TSO:       # %bb.0:
+; RV32IA-TSO-NEXT:    sw a1, 0(a0)
+; RV32IA-TSO-NEXT:    ret
 ;
 ; RV64I-LABEL: atomic_store_i32_seq_cst:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a2, 5
 ; RV64I-NEXT:    call __atomic_store_4@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_store_i32_seq_cst:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    fence rw, w
-; RV64IA-NEXT:    sw a1, 0(a0)
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_store_i32_seq_cst:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    fence rw, w
+; RV64IA-WMO-NEXT:    sw a1, 0(a0)
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_store_i32_seq_cst:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    sw a1, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   store atomic i32 %b, ptr %a seq_cst, align 4
   ret void
 }
 
 define void @atomic_store_i64_unordered(ptr %a, i64 %b) nounwind {
 ; RV32I-LABEL: atomic_store_i64_unordered:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16

 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a2, 3
 ; RV64I-NEXT:    call __atomic_store_8@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_store_i64_release:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    fence rw, w
-; RV64IA-NEXT:    sd a1, 0(a0)
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_store_i64_release:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    fence rw, w
+; RV64IA-WMO-NEXT:    sd a1, 0(a0)
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_store_i64_release:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    sd a1, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   store atomic i64 %b, ptr %a release, align 8
   ret void
 }
 
 define void @atomic_store_i64_seq_cst(ptr %a, i64 %b) nounwind {
 ; RV32I-LABEL: atomic_store_i64_seq_cst:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    addi sp, sp, -16

 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    li a2, 5
 ; RV64I-NEXT:    call __atomic_store_8@plt
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
 ; RV64I-NEXT:    ret
 ;
-; RV64IA-LABEL: atomic_store_i64_seq_cst:
-; RV64IA:       # %bb.0:
-; RV64IA-NEXT:    fence rw, w
-; RV64IA-NEXT:    sd a1, 0(a0)
-; RV64IA-NEXT:    ret
+; RV64IA-WMO-LABEL: atomic_store_i64_seq_cst:
+; RV64IA-WMO:       # %bb.0:
+; RV64IA-WMO-NEXT:    fence rw, w
+; RV64IA-WMO-NEXT:    sd a1, 0(a0)
+; RV64IA-WMO-NEXT:    ret
+;
+; RV64IA-TSO-LABEL: atomic_store_i64_seq_cst:
+; RV64IA-TSO:       # %bb.0:
+; RV64IA-TSO-NEXT:    sd a1, 0(a0)
+; RV64IA-TSO-NEXT:    ret
   store atomic i64 %b, ptr %a seq_cst, align 8
   ret void
 }