[AArch64] Codegen for FEAT_LRCPC3

[tmatheson]

Summary

Implements support for the following 128-bit atomic operations with +rcpc3:

• store-release
• store-release volatile
• load-acquire
• load-acquire volatile
• load-acquire const
• load-acquire const volatile

D126250 and D137590 added support for emitting LDAPR (Load-Acquire RCPc) rather
than LDAP (Load-Acquire) when +rcpc is available. This patch allows emitting
the 128-bit RCPc instructions added in FEAT_LRCPC3 LDIAPP/STILP. The
implementation is different from LDAPR, because there are no non-RCPc
equivalents for these new instructions.

Support for the offset variants will be added in a later patch.

[lenary]

You need FEAT_LSE2 if you're going to generate STILP/LDIAPP from a 128-bit access, in order to get single-copy atomicity.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

5776

Outdated?

22232

Yes that sounds reasonable

llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp

1220

[tmatheson]

Require LSE2

[efriedma]

llvm/test/CodeGen/AArch64/Atomics/aarch64_be-atomic-load-rcpc3.ll

283

Sort of orthogonal to this change, but can someone at ARM verify if this is the sequence we actually want for sequentially consistent loads with lse2, as opposed to using caspal? (I'm a bit concerned given the issues we ran into with narrower widths on Windows; see D141748.)

[efriedma]

llvm/test/CodeGen/AArch64/Atomics/aarch64_be-atomic-load-rcpc3.ll

283

Any update here?

[tmatheson]

llvm/test/CodeGen/AArch64/Atomics/aarch64_be-atomic-load-rcpc3.ll

283

Not yet but I haven't forgotten about it.

[tmatheson]

llvm/test/CodeGen/AArch64/Atomics/aarch64_be-atomic-load-rcpc3.ll

283

Sorry this has taken a while, I've had to work on other things. The first thing to note is that a CASP implementation is slower, so as long as both are correct we should use the ldp+dmb.

@LukeGeeson has been working on adding CASP to herd7 so that we can compare ldp+dmb with casp. Using his Telechat tool, which can compare the C++ semantics with the machine code semantics and identify new behaviours introduced by the machine code, we aren't seeing any differences so far, although admittedly the number of cases we've checked is low because each CASP case has to be manually written.

We are still looking for at a way to run more extensive testing automatically, but if you have any concerns about specific scenarios we can check them.

[efriedma]

llvm/test/CodeGen/AArch64/Atomics/aarch64_be-atomic-load-rcpc3.ll

283

I guess my biggest concern here is that we're basically committing to never using sequentially consistent (non-RCPc) load/store instructions for i128, even if the instruction set adds them in the future. If you're fine with that, then I guess I don't have any other specific concerns?

llvm/lib/Target/AArch64/AArch64ISelLowering.h

enum NodeType : unsigned {
   ST4LANEpost,
 
   STG,
   STZG,
   ST2G,
   STZ2G,
 
   LDP,
+  LDIAPP,
   LDNP,
   STP,
+  STILP,
   STNP,
 
   // Memory Operations
   MOPS_MEMSET,
   MOPS_MEMSET_TAGGING,
   MOPS_MEMCOPY,
   MOPS_MEMMOVE,
 };
public:
   Value *emitLoadLinked(IRBuilderBase &Builder, Type *ValueTy, Value *Addr,
                         AtomicOrdering Ord) const override;
   Value *emitStoreConditional(IRBuilderBase &Builder, Value *Val, Value *Addr,
                               AtomicOrdering Ord) const override;
 
   void emitAtomicCmpXchgNoStoreLLBalance(IRBuilderBase &Builder) const override;
 
   bool isOpSuitableForLDPSTP(const Instruction *I) const;
+  bool isOpSuitableForRCPC3(const Instruction *I) const;
   bool shouldInsertFencesForAtomic(const Instruction *I) const override;
 
   TargetLoweringBase::AtomicExpansionKind
   shouldExpandAtomicLoadInIR(LoadInst *LI) const override;
   TargetLoweringBase::AtomicExpansionKind
   shouldExpandAtomicStoreInIR(StoreInst *SI) const override;
   TargetLoweringBase::AtomicExpansionKind
   shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

 #undef LCALLNAME5
   }
 
   // 128-bit loads and stores can be done without expanding
   setOperationAction(ISD::LOAD, MVT::i128, Custom);
   setOperationAction(ISD::STORE, MVT::i128, Custom);
 
   // Aligned 128-bit loads and stores are single-copy atomic according to the
-  // v8.4a spec.
+  // v8.4a spec. LRCPC3 introduces 128-bit STILP/LDIAPP but still requires LSE2.
   if (Subtarget->hasLSE2()) {
     setOperationAction(ISD::ATOMIC_LOAD, MVT::i128, Custom);
     setOperationAction(ISD::ATOMIC_STORE, MVT::i128, Custom);
   }
 
   // 256 bit non-temporal stores can be lowered to STNP. Do this as part of the
   // custom lowering, as there are no un-paired non-temporal stores and
   // legalization will break up 256 bit inputs.
case AArch64ISD::FIRST_NUMBER:
     MAKE_CASE(AArch64ISD::SST1_SXTW_PRED)
     MAKE_CASE(AArch64ISD::SST1_UXTW_PRED)
     MAKE_CASE(AArch64ISD::SST1_SXTW_SCALED_PRED)
     MAKE_CASE(AArch64ISD::SST1_UXTW_SCALED_PRED)
     MAKE_CASE(AArch64ISD::SST1_IMM_PRED)
     MAKE_CASE(AArch64ISD::SSTNT1_PRED)
     MAKE_CASE(AArch64ISD::SSTNT1_INDEX_PRED)
     MAKE_CASE(AArch64ISD::LDP)
+    MAKE_CASE(AArch64ISD::LDIAPP)
     MAKE_CASE(AArch64ISD::LDNP)
     MAKE_CASE(AArch64ISD::STP)
+    MAKE_CASE(AArch64ISD::STILP)
     MAKE_CASE(AArch64ISD::STNP)
     MAKE_CASE(AArch64ISD::BITREVERSE_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::BSWAP_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::REVH_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::REVW_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::REVD_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::CTLZ_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::CTPOP_MERGE_PASSTHRU)

 }
 
 /// Lower atomic or volatile 128-bit stores to a single STP instruction.
 SDValue AArch64TargetLowering::LowerStore128(SDValue Op,
                                              SelectionDAG &DAG) const {
   MemSDNode *StoreNode = cast<MemSDNode>(Op);
   assert(StoreNode->getMemoryVT() == MVT::i128);
   assert(StoreNode->isVolatile() || StoreNode->isAtomic());
-  assert(!StoreNode->isAtomic() ||
+
+  bool IsStoreRelease =
+      StoreNode->getMergedOrdering() == AtomicOrdering::Release;
+  if (StoreNode->isAtomic())
+    assert((Subtarget->hasFeature(AArch64::FeatureLSE2) &&
+            Subtarget->hasFeature(AArch64::FeatureRCPC3) && IsStoreRelease) ||
            StoreNode->getMergedOrdering() == AtomicOrdering::Unordered ||
            StoreNode->getMergedOrdering() == AtomicOrdering::Monotonic);
 
   SDValue Value = StoreNode->getOpcode() == ISD::STORE
                       ? StoreNode->getOperand(1)
                       : StoreNode->getOperand(2);
   SDLoc DL(Op);
   SDValue Lo = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i64, Value,
                            DAG.getConstant(0, DL, MVT::i64));
   SDValue Hi = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i64, Value,
                            DAG.getConstant(1, DL, MVT::i64));
+
+  unsigned Opcode = IsStoreRelease ? AArch64ISD::STILP : AArch64ISD::STP;
   SDValue Result = DAG.getMemIntrinsicNode(
-      AArch64ISD::STP, DL, DAG.getVTList(MVT::Other),
+      Opcode, DL, DAG.getVTList(MVT::Other),
       {StoreNode->getChain(), Lo, Hi, StoreNode->getBasePtr()},
       StoreNode->getMemoryVT(), StoreNode->getMemOperand());
   return Result;
 }
 
 SDValue AArch64TargetLowering::LowerLOAD(SDValue Op,
                                          SelectionDAG &DAG) const {
   SDLoc DL(Op);
SDValue AArch64TargetLowering::LowerOperation(SDValue Op,
   case ISD::INTRINSIC_W_CHAIN:
     return LowerINTRINSIC_W_CHAIN(Op, DAG);
   case ISD::INTRINSIC_WO_CHAIN:
     return LowerINTRINSIC_WO_CHAIN(Op, DAG);
   case ISD::INTRINSIC_VOID:
     return LowerINTRINSIC_VOID(Op, DAG);
   case ISD::ATOMIC_STORE:
     if (cast<MemSDNode>(Op)->getMemoryVT() == MVT::i128) {
-      assert(Subtarget->hasLSE2());
+      assert(Subtarget->hasLSE2() || Subtarget->hasRCPC3());
       return LowerStore128(Op, DAG);
     }
     return SDValue();
   case ISD::STORE:
     return LowerSTORE(Op, DAG);
   case ISD::MSTORE:
     return LowerFixedLengthVectorMStoreToSVE(Op, DAG);
   case ISD::MGATHER:
case ISD::LOAD: {
     if ((!LoadNode->isVolatile() && !LoadNode->isAtomic()) ||
         LoadNode->getMemoryVT() != MVT::i128) {
       // Non-volatile or atomic loads are optimized later in AArch64's load/store
       // optimizer.
       return;
     }
 
     if (SDValue(N, 0).getValueType() == MVT::i128) {
+      auto *AN = dyn_cast<AtomicSDNode>(LoadNode);
+      bool isLoadAcquire =
+          AN && AN->getSuccessOrdering() == AtomicOrdering::Acquire;
+      unsigned Opcode = isLoadAcquire ? AArch64ISD::LDIAPP : AArch64ISD::LDP;
+
+      if (isLoadAcquire)
+        assert(Subtarget->hasFeature(AArch64::FeatureRCPC3));
+
       SDValue Result = DAG.getMemIntrinsicNode(
-          AArch64ISD::LDP, SDLoc(N),
-          DAG.getVTList({MVT::i64, MVT::i64, MVT::Other}),
+          Opcode, SDLoc(N), DAG.getVTList({MVT::i64, MVT::i64, MVT::Other}),
           {LoadNode->getChain(), LoadNode->getBasePtr()},
           LoadNode->getMemoryVT(), LoadNode->getMemOperand());
 
       SDValue Pair = DAG.getNode(ISD::BUILD_PAIR, SDLoc(N), MVT::i128,
                                  Result.getValue(0), Result.getValue(1));
       Results.append({Pair, Result.getValue(2) /* Chain */});
     }
     return;
bool AArch64TargetLowering::isOpSuitableForLDPSTP(const Instruction *I) const {
 
   if (auto SI = dyn_cast<StoreInst>(I))
     return SI->getValueOperand()->getType()->getPrimitiveSizeInBits() == 128 &&
            SI->getAlign() >= Align(16);
 
   return false;
 }
 
+bool AArch64TargetLowering::isOpSuitableForRCPC3(const Instruction *I) const {
+  if (!Subtarget->hasLSE2() || !Subtarget->hasRCPC3())
+    return false;
+
+  if (auto LI = dyn_cast<LoadInst>(I))
+    return LI->getType()->getPrimitiveSizeInBits() == 128 &&
+           LI->getAlign() >= Align(16) &&
+           LI->getOrdering() == AtomicOrdering::Acquire;
+
+  if (auto SI = dyn_cast<StoreInst>(I))
+    return SI->getValueOperand()->getType()->getPrimitiveSizeInBits() == 128 &&
+           SI->getAlign() >= Align(16) &&
+           SI->getOrdering() == AtomicOrdering::Release;
+
+  return false;
+}
+
 bool AArch64TargetLowering::shouldInsertFencesForAtomic(
     const Instruction *I) const {
+  if (isOpSuitableForRCPC3(I))
+    return false;
   return isOpSuitableForLDPSTP(I);
 }
 
 // Loads and stores less than 128-bits are already atomic; ones above that
 // are doomed anyway, so defer to the default libcall and blame the OS when
 // things go wrong.
 TargetLoweringBase::AtomicExpansionKind
 AArch64TargetLowering::shouldExpandAtomicStoreInIR(StoreInst *SI) const {
   unsigned Size = SI->getValueOperand()->getType()->getPrimitiveSizeInBits();
-  if (Size != 128 || isOpSuitableForLDPSTP(SI))
+  if (Size != 128 || isOpSuitableForLDPSTP(SI) || isOpSuitableForRCPC3(SI))
     return AtomicExpansionKind::None;
   return AtomicExpansionKind::Expand;
 }
 
 // Loads and stores less than 128-bits are already atomic; ones above that
 // are doomed anyway, so defer to the default libcall and blame the OS when
 // things go wrong.
 TargetLowering::AtomicExpansionKind
 AArch64TargetLowering::shouldExpandAtomicLoadInIR(LoadInst *LI) const {
   unsigned Size = LI->getType()->getPrimitiveSizeInBits();
 
-  if (Size != 128 || isOpSuitableForLDPSTP(LI))
+  if (Size != 128 || isOpSuitableForLDPSTP(LI) || isOpSuitableForRCPC3(LI))
     return AtomicExpansionKind::None;
 
   // At -O0, fast-regalloc cannot cope with the live vregs necessary to
   // implement atomicrmw without spilling. If the target address is also on the
   // stack and close enough to the spill slot, this can lead to a situation
   // where the monitor always gets cleared and the atomic operation can never
   // succeed. So at -O0 lower this operation to a CAS loop.
   if (getTargetMachine().getOptLevel() == CodeGenOpt::None)

llvm/lib/Target/AArch64/AArch64InstrInfo.td

 def SDT_AArch64ITOF  : SDTypeProfile<1, 1, [SDTCisFP<0>, SDTCisSameAs<0,1>]>;
 
 def SDT_AArch64TLSDescCall : SDTypeProfile<0, -2, [SDTCisPtrTy<0>,
                                                  SDTCisPtrTy<1>]>;
 
 def SDT_AArch64uaddlp : SDTypeProfile<1, 1, [SDTCisVec<0>, SDTCisVec<1>]>;
 
 def SDT_AArch64ldp : SDTypeProfile<2, 1, [SDTCisVT<0, i64>, SDTCisSameAs<0, 1>, SDTCisPtrTy<2>]>;
+def SDT_AArch64ldiapp : SDTypeProfile<2, 1, [SDTCisVT<0, i64>, SDTCisSameAs<0, 1>, SDTCisPtrTy<2>]>;
 def SDT_AArch64ldnp : SDTypeProfile<2, 1, [SDTCisVT<0, v4i32>, SDTCisSameAs<0, 1>, SDTCisPtrTy<2>]>;
 def SDT_AArch64stp : SDTypeProfile<0, 3, [SDTCisVT<0, i64>, SDTCisSameAs<0, 1>, SDTCisPtrTy<2>]>;
+def SDT_AArch64stilp : SDTypeProfile<0, 3, [SDTCisVT<0, i64>, SDTCisSameAs<0, 1>, SDTCisPtrTy<2>]>;
 def SDT_AArch64stnp : SDTypeProfile<0, 3, [SDTCisVT<0, v4i32>, SDTCisSameAs<0, 1>, SDTCisPtrTy<2>]>;
 
 // Generates the general dynamic sequences, i.e.
 //  adrp  x0, :tlsdesc:var
 //  ldr   x1, [x0, #:tlsdesc_lo12:var]
 //  add   x0, x0, #:tlsdesc_lo12:var
 //  .tlsdesccall var
 //  blr   x1
def SDT_AArch64unpk : SDTypeProfile<1, 1, [
     SDTCisInt<0>, SDTCisInt<1>, SDTCisOpSmallerThanOp<1, 0>
 ]>;
 def AArch64sunpkhi : SDNode<"AArch64ISD::SUNPKHI", SDT_AArch64unpk>;
 def AArch64sunpklo : SDNode<"AArch64ISD::SUNPKLO", SDT_AArch64unpk>;
 def AArch64uunpkhi : SDNode<"AArch64ISD::UUNPKHI", SDT_AArch64unpk>;
 def AArch64uunpklo : SDNode<"AArch64ISD::UUNPKLO", SDT_AArch64unpk>;
 
 def AArch64ldp : SDNode<"AArch64ISD::LDP", SDT_AArch64ldp, [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;
+def AArch64ldiapp : SDNode<"AArch64ISD::LDIAPP", SDT_AArch64ldiapp, [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;
 def AArch64ldnp : SDNode<"AArch64ISD::LDNP", SDT_AArch64ldnp, [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;
 def AArch64stp : SDNode<"AArch64ISD::STP", SDT_AArch64stp, [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;
+def AArch64stilp : SDNode<"AArch64ISD::STILP", SDT_AArch64stilp, [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;
 def AArch64stnp : SDNode<"AArch64ISD::STNP", SDT_AArch64stnp, [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;
 
 def AArch64tbl : SDNode<"AArch64ISD::TBL", SDT_AArch64TBL>;
 def AArch64mrs : SDNode<"AArch64ISD::MRS",
                         SDTypeProfile<1, 1, [SDTCisVT<0, i64>, SDTCisVT<1, i32>]>,
                         [SDNPHasChain, SDNPOutGlue]>;
 
 // Match add node and also treat an 'or' node is as an 'add' if the or'ed operands

 
 // Pair (pre-indexed)
 def STPWpre : StorePairPreIdx<0b00, 0, GPR32z, simm7s4, "stp">;
 def STPXpre : StorePairPreIdx<0b10, 0, GPR64z, simm7s8, "stp">;
 def STPSpre : StorePairPreIdx<0b00, 1, FPR32Op, simm7s4, "stp">;
 def STPDpre : StorePairPreIdx<0b01, 1, FPR64Op, simm7s8, "stp">;
 def STPQpre : StorePairPreIdx<0b10, 1, FPR128Op, simm7s16, "stp">;
 
-// Pair (pre-indexed)
+// Pair (post-indexed)
 def STPWpost : StorePairPostIdx<0b00, 0, GPR32z, simm7s4, "stp">;
 def STPXpost : StorePairPostIdx<0b10, 0, GPR64z, simm7s8, "stp">;
 def STPSpost : StorePairPostIdx<0b00, 1, FPR32Op, simm7s4, "stp">;
 def STPDpost : StorePairPostIdx<0b01, 1, FPR64Op, simm7s8, "stp">;
 def STPQpost : StorePairPostIdx<0b10, 1, FPR128Op, simm7s16, "stp">;
 
 // Pair (no allocate)
 defm STNPW : StorePairNoAlloc<0b00, 0, GPR32z, simm7s4, "stnp">;
let Predicates = [HasRCPC3] in {
   def STILPXpre:  BaseLRCPC3IntegerLoadStorePair<0b11, 0b00, 0b0000, (outs GPR64sp:$wback), (ins GPR64:$Rt, GPR64:$Rt2, GPR64sp:$Rn), "stilp", "\t$Rt, $Rt2, [$Rn, #-16]!", "$Rn = $wback">;
   def STILPW:     BaseLRCPC3IntegerLoadStorePair<0b10, 0b00, 0b0001, (outs), (ins GPR32:$Rt, GPR32:$Rt2, GPR64sp:$Rn), "stilp", "\t$Rt, $Rt2, [$Rn]", "">;
   def STILPX:     BaseLRCPC3IntegerLoadStorePair<0b11, 0b00, 0b0001, (outs), (ins GPR64:$Rt, GPR64:$Rt2, GPR64sp:$Rn), "stilp", "\t$Rt, $Rt2, [$Rn]", "">;
   def LDIAPPWpre: BaseLRCPC3IntegerLoadStorePair<0b10, 0b01, 0b0000, (outs GPR64sp:$wback, GPR32:$Rt, GPR32:$Rt2), (ins GPR64sp:$Rn), "ldiapp", "\t$Rt, $Rt2, [$Rn], #8", "$Rn = $wback">;
   def LDIAPPXpre: BaseLRCPC3IntegerLoadStorePair<0b11, 0b01, 0b0000, (outs GPR64sp:$wback, GPR64:$Rt, GPR64:$Rt2), (ins GPR64sp:$Rn), "ldiapp", "\t$Rt, $Rt2, [$Rn], #16", "$Rn = $wback">;
   def LDIAPPW:    BaseLRCPC3IntegerLoadStorePair<0b10, 0b01, 0b0001, (outs GPR32:$Rt, GPR32:$Rt2), (ins GPR64sp0:$Rn), "ldiapp", "\t$Rt, $Rt2, [$Rn]", "">;
   def LDIAPPX:    BaseLRCPC3IntegerLoadStorePair<0b11, 0b01, 0b0001, (outs GPR64:$Rt, GPR64:$Rt2), (ins GPR64sp0:$Rn), "ldiapp", "\t$Rt, $Rt2, [$Rn]", "">;
 
+  def : Pat<(AArch64ldiapp GPR64sp:$Rn), (LDIAPPX GPR64sp:$Rn)>;
+  def : Pat<(AArch64stilp GPR64:$Rt, GPR64:$Rt2, GPR64sp:$Rn), (STILPX GPR64:$Rt, GPR64:$Rt2, GPR64sp:$Rn)>;
+
   // Aliases for when offset=0
   def : InstAlias<"stilp\t$Rt, $Rt2, [$Rn, #0]", (STILPW GPR32: $Rt, GPR32: $Rt2, GPR64sp:$Rn)>;
   def : InstAlias<"stilp\t$Rt, $Rt2, [$Rn, #0]", (STILPX GPR64: $Rt, GPR64: $Rt2, GPR64sp:$Rn)>;
 
   //                                        size   opc
   def STLRWpre:  BaseLRCPC3IntegerLoadStore<0b10, 0b10, (outs GPR64sp:$wback),            (ins GPR32:$Rt, GPR64sp:$Rn), "stlr",  "\t$Rt, [$Rn, #-4]!", "$Rn = $wback">;
   def STLRXpre:  BaseLRCPC3IntegerLoadStore<0b11, 0b10, (outs GPR64sp:$wback),            (ins GPR64:$Rt, GPR64sp:$Rn), "stlr",  "\t$Rt, [$Rn, #-8]!", "$Rn = $wback">;
   def LDAPRWpre: BaseLRCPC3IntegerLoadStore<0b10, 0b11, (outs GPR64sp:$wback, GPR32:$Rt), (ins GPR64sp:$Rn),            "ldapr", "\t$Rt, [$Rn], #4",   "$Rn = $wback">;

llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp

 //===- AArch64LegalizerInfo.cpp ----------------------------------*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
 /// This file implements the targeting of the Machinelegalizer class for
AArch64LegalizerInfo::AArch64LegalizerInfo(const AArch64Subtarget &ST)
   }
 
   // Some instructions only support s16 if the subtarget has full 16-bit FP
   // support.
   const bool HasFP16 = ST.hasFullFP16();
   const LLT &MinFPScalar = HasFP16 ? s16 : s32;
 
   const bool HasCSSC = ST.hasCSSC();
+  const bool HasRCPC3 = ST.hasRCPC3();
 
   getActionDefinitionsBuilder({G_IMPLICIT_DEF, G_FREEZE})
       .legalFor({p0, s8, s16, s32, s64})
       .legalFor(PackedVectorAllTypeList)
       .widenScalarToNextPow2(0)
       .clampScalar(0, s8, s64)
       .fewerElementsIf(
           [=](const LegalityQuery &Query) {
auto IsPtrVecPred = [=](const LegalityQuery &Query) {
     if (!ValTy.isVector())
       return false;
     const LLT EltTy = ValTy.getElementType();
     return EltTy.isPointer() && EltTy.getAddressSpace() == 0;
   };
 
   getActionDefinitionsBuilder(G_LOAD)
       .customIf([=](const LegalityQuery &Query) {
+        return HasRCPC3 && Query.Types[0] == s128 &&
+               Query.MMODescrs[0].Ordering == AtomicOrdering::Acquire;
+      })
+      .customIf([=](const LegalityQuery &Query) {
         return Query.Types[0] == s128 &&
                Query.MMODescrs[0].Ordering != AtomicOrdering::NotAtomic;
       })
       .legalForTypesWithMemDesc({{s8, p0, s8, 8},
                                  {s16, p0, s16, 8},
                                  {s32, p0, s32, 8},
                                  {s64, p0, s64, 8},
                                  {p0, p0, s64, 8},
getActionDefinitionsBuilder(G_LOAD)
       .clampMaxNumElements(0, s32, 4)
       .clampMaxNumElements(0, s64, 2)
       .clampMaxNumElements(0, p0, 2)
       .customIf(IsPtrVecPred)
       .scalarizeIf(typeIs(0, v2s16), 0);
 
   getActionDefinitionsBuilder(G_STORE)
       .customIf([=](const LegalityQuery &Query) {
+        return HasRCPC3 && Query.Types[0] == s128 &&
+               Query.MMODescrs[0].Ordering == AtomicOrdering::Release;
+      })
+      .customIf([=](const LegalityQuery &Query) {
         return Query.Types[0] == s128 &&
                Query.MMODescrs[0].Ordering != AtomicOrdering::NotAtomic;
       })
       .legalForTypesWithMemDesc({{s8, p0, s8, 8},
                                  {s16, p0, s8, 8}, // truncstorei8 from s16
                                  {s32, p0, s8, 8}, // truncstorei8 from s32
                                  {s64, p0, s8, 8}, // truncstorei8 from s64
                                  {s16, p0, s16, 8},
bool AArch64LegalizerInfo::legalizeLoadStore(
   // Custom legalization requires the instruction, if not deleted, must be fully
   // legalized. In order to allow further legalization of the inst, we create
   // a new instruction and erase the existing one.
 
   Register ValReg = MI.getOperand(0).getReg();
   const LLT ValTy = MRI.getType(ValReg);
 
   if (ValTy == LLT::scalar(128)) {
-    assert((*MI.memoperands_begin())->getSuccessOrdering() ==
-               AtomicOrdering::Monotonic ||
-           (*MI.memoperands_begin())->getSuccessOrdering() ==
-               AtomicOrdering::Unordered);
-    assert(ST->hasLSE2() && "ldp/stp not single copy atomic without +lse2");
+
+    AtomicOrdering Ordering = (*MI.memoperands_begin())->getSuccessOrdering();
+    bool IsLoad = MI.getOpcode() == TargetOpcode::G_LOAD;
+    bool IsLoadAcquire = IsLoad && Ordering == AtomicOrdering::Acquire;
+    bool IsStoreRelease = !IsLoad && Ordering == AtomicOrdering::Release;
+    bool IsRcpC3 =
+        ST->hasLSE2() && ST->hasRCPC3() && (IsLoadAcquire || IsStoreRelease);
+
     LLT s64 = LLT::scalar(64);
+
+    unsigned Opcode;
+    if (IsRcpC3) {
+      Opcode = IsLoad ? AArch64::LDIAPPX : AArch64::STILPX;
+    } else {
+      // For LSE2, loads/stores should have been converted to monotonic and had
+      // a fence inserted after them.
+      assert(Ordering == AtomicOrdering::Monotonic ||
+             Ordering == AtomicOrdering::Unordered);
+      assert(ST->hasLSE2() && "ldp/stp not single copy atomic without +lse2");
+
+      Opcode = IsLoad ? AArch64::LDPXi : AArch64::STPXi;
+    }
+
     MachineInstrBuilder NewI;
-    if (MI.getOpcode() == TargetOpcode::G_LOAD) {
-      NewI = MIRBuilder.buildInstr(AArch64::LDPXi, {s64, s64}, {});
+    if (IsLoad) {
+      NewI = MIRBuilder.buildInstr(Opcode, {s64, s64}, {});
       MIRBuilder.buildMerge(ValReg, {NewI->getOperand(0), NewI->getOperand(1)});
     } else {
       auto Split = MIRBuilder.buildUnmerge(s64, MI.getOperand(0));
       NewI = MIRBuilder.buildInstr(
-          AArch64::STPXi, {}, {Split->getOperand(0), Split->getOperand(1)});
+          Opcode, {}, {Split->getOperand(0), Split->getOperand(1)});
     }
+
+    if (IsRcpC3) {
+      NewI.addUse(MI.getOperand(1).getReg());
+    } else {
       Register Base;
       int Offset;
       matchLDPSTPAddrMode(MI.getOperand(1).getReg(), Base, Offset, MRI);
       NewI.addUse(Base);
       NewI.addImm(Offset / 8);
+    }
 
     NewI.cloneMemRefs(MI);
     constrainSelectedInstRegOperands(*NewI, *ST->getInstrInfo(),
                                      *MRI.getTargetRegisterInfo(),
                                      *ST->getRegBankInfo());
     MI.eraseFromParent();
     return true;
   }

llvm/test/CodeGen/AArch64/Atomics/aarch64-atomic-load-rcpc3.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --filter-out "\b(sp)\b" --filter "^\s*(ld|st[^r]|swp|cas|bl|add|and|eor|orn|orr|sub|mvn|sxt|cmp|ccmp|csel|dmb)"
 ; The base test file was generated by ./llvm/test/CodeGen/AArch64/Atomics/generate-tests.py
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse2,+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse2,+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 
 define dso_local i8 @load_atomic_i8_aligned_unordered(ptr %ptr) {
 ; CHECK-LABEL: load_atomic_i8_aligned_unordered:
 ; CHECK:    ldrb w0, [x0]
     %r = load atomic i8, ptr %ptr unordered, align 1
     ret i8 %r
 }
 

 define dso_local i64 @load_atomic_i64_aligned_seq_cst_const(ptr readonly %ptr) {
 ; CHECK-LABEL: load_atomic_i64_aligned_seq_cst_const:
 ; CHECK:    ldar x0, [x0]
     %r = load atomic i64, ptr %ptr seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_unordered(ptr %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_unordered:
-; -O0:    ldxp x0, x1, [x9]
-; -O0:    cmp x0, x10
-; -O0:    cmp x1, x10
-; -O0:    stxp w8, x10, x10, [x9]
-; -O0:    stxp w8, x0, x1, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_unordered:
-; -O1:    ldxp x0, x1, [x8]
-; -O1:    stxp w9, x0, x1, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_unordered:
+; CHECK:    ldp x0, x1, [x0]
     %r = load atomic i128, ptr %ptr unordered, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_unordered_const(ptr readonly %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_unordered_const:
-; -O0:    ldxp x0, x1, [x9]
-; -O0:    cmp x0, x10
-; -O0:    cmp x1, x10
-; -O0:    stxp w8, x10, x10, [x9]
-; -O0:    stxp w8, x0, x1, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_unordered_const:
-; -O1:    ldxp x0, x1, [x8]
-; -O1:    stxp w9, x0, x1, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_unordered_const:
+; CHECK:    ldp x0, x1, [x0]
     %r = load atomic i128, ptr %ptr unordered, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_monotonic(ptr %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_monotonic:
-; -O0:    ldxp x0, x1, [x9]
-; -O0:    cmp x0, x10
-; -O0:    cmp x1, x10
-; -O0:    stxp w8, x10, x10, [x9]
-; -O0:    stxp w8, x0, x1, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_monotonic:
-; -O1:    ldxp x0, x1, [x8]
-; -O1:    stxp w9, x0, x1, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_monotonic:
+; CHECK:    ldp x0, x1, [x0]
     %r = load atomic i128, ptr %ptr monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_monotonic_const(ptr readonly %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_monotonic_const:
-; -O0:    ldxp x0, x1, [x9]
-; -O0:    cmp x0, x10
-; -O0:    cmp x1, x10
-; -O0:    stxp w8, x10, x10, [x9]
-; -O0:    stxp w8, x0, x1, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_monotonic_const:
-; -O1:    ldxp x0, x1, [x8]
-; -O1:    stxp w9, x0, x1, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_monotonic_const:
+; CHECK:    ldp x0, x1, [x0]
     %r = load atomic i128, ptr %ptr monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_acquire(ptr %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_acquire:
-; -O0:    ldaxp x0, x1, [x9]
-; -O0:    cmp x0, x10
-; -O0:    cmp x1, x10
-; -O0:    stxp w8, x10, x10, [x9]
-; -O0:    stxp w8, x0, x1, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_acquire:
-; -O1:    ldaxp x0, x1, [x8]
-; -O1:    stxp w9, x0, x1, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_acquire:
+; CHECK:    ldiapp x0, x1, [x0]
     %r = load atomic i128, ptr %ptr acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_acquire_const(ptr readonly %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_acquire_const:
-; -O0:    ldaxp x0, x1, [x9]
-; -O0:    cmp x0, x10
-; -O0:    cmp x1, x10
-; -O0:    stxp w8, x10, x10, [x9]
-; -O0:    stxp w8, x0, x1, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_acquire_const:
-; -O1:    ldaxp x0, x1, [x8]
-; -O1:    stxp w9, x0, x1, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_acquire_const:
+; CHECK:    ldiapp x0, x1, [x0]
     %r = load atomic i128, ptr %ptr acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_seq_cst(ptr %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_seq_cst:
-; -O0:    ldaxp x0, x1, [x9]
-; -O0:    cmp x0, x10
-; -O0:    cmp x1, x10
-; -O0:    stlxp w8, x10, x10, [x9]
-; -O0:    stlxp w8, x0, x1, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_seq_cst:
-; -O1:    ldaxp x0, x1, [x8]
-; -O1:    stlxp w9, x0, x1, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_seq_cst:
+; CHECK:    ldp x0, x1, [x0]
+; CHECK:    dmb ish
     %r = load atomic i128, ptr %ptr seq_cst, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_seq_cst_const(ptr readonly %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_seq_cst_const:
-; -O0:    ldaxp x0, x1, [x9]
-; -O0:    cmp x0, x10
-; -O0:    cmp x1, x10
-; -O0:    stlxp w8, x10, x10, [x9]
-; -O0:    stlxp w8, x0, x1, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_seq_cst_const:
-; -O1:    ldaxp x0, x1, [x8]
-; -O1:    stlxp w9, x0, x1, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_seq_cst_const:
+; CHECK:    ldp x0, x1, [x0]
+; CHECK:    dmb ish
     %r = load atomic i128, ptr %ptr seq_cst, align 16
     ret i128 %r
 }
 
 define dso_local i8 @load_atomic_i8_unaligned_unordered(ptr %ptr) {
 ; CHECK-LABEL: load_atomic_i8_unaligned_unordered:
 ; CHECK:    ldrb w0, [x0]
     %r = load atomic i8, ptr %ptr unordered, align 1

 }
 
 define dso_local i128 @load_atomic_i128_unaligned_seq_cst_const(ptr readonly %ptr) {
 ; CHECK-LABEL: load_atomic_i128_unaligned_seq_cst_const:
 ; CHECK:    bl __atomic_load
     %r = load atomic i128, ptr %ptr seq_cst, align 1
     ret i128 %r
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; -O0: {{.*}}
+; -O1: {{.*}}

llvm/test/CodeGen/AArch64/Atomics/aarch64-atomic-store-rcpc3.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --filter-out "\b(sp)\b" --filter "^\s*(ld[^r]|st|swp|cas|bl|add|and|eor|orn|orr|sub|mvn|sxt|cmp|ccmp|csel|dmb)"
 ; The base test file was generated by ./llvm/test/CodeGen/AArch64/Atomics/generate-tests.py
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse2,+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse2,+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 
 define dso_local void @store_atomic_i8_aligned_unordered(i8 %value, ptr %ptr) {
 ; CHECK-LABEL: store_atomic_i8_aligned_unordered:
 ; CHECK:    strb w0, [x1]
     store atomic i8 %value, ptr %ptr unordered, align 1
     ret void
 }
 

 define dso_local void @store_atomic_i64_aligned_seq_cst(i64 %value, ptr %ptr) {
 ; CHECK-LABEL: store_atomic_i64_aligned_seq_cst:
 ; CHECK:    stlr x0, [x1]
     store atomic i64 %value, ptr %ptr seq_cst, align 8
     ret void
 }
 
 define dso_local void @store_atomic_i128_aligned_unordered(i128 %value, ptr %ptr) {
-; -O0-LABEL: store_atomic_i128_aligned_unordered:
-; -O0:    ldxp x10, x9, [x11]
-; -O0:    cmp x10, x12
-; -O0:    cmp x9, x13
-; -O0:    stxp w8, x14, x15, [x11]
-; -O0:    stxp w8, x10, x9, [x11]
-; -O0:    eor x8, x10, x8
-; -O0:    eor x11, x9, x11
-; -O0:    orr x8, x8, x11
-; -O0:    subs x8, x8, #0
-;
-; -O1-LABEL: store_atomic_i128_aligned_unordered:
-; -O1:    ldxp xzr, x8, [x2]
-; -O1:    stxp w8, x0, x1, [x2]
+; CHECK-LABEL: store_atomic_i128_aligned_unordered:
+; CHECK:    stp x0, x1, [x2]
     store atomic i128 %value, ptr %ptr unordered, align 16
     ret void
 }
 
 define dso_local void @store_atomic_i128_aligned_monotonic(i128 %value, ptr %ptr) {
-; -O0-LABEL: store_atomic_i128_aligned_monotonic:
-; -O0:    ldxp x10, x9, [x11]
-; -O0:    cmp x10, x12
-; -O0:    cmp x9, x13
-; -O0:    stxp w8, x14, x15, [x11]
-; -O0:    stxp w8, x10, x9, [x11]
-; -O0:    eor x8, x10, x8
-; -O0:    eor x11, x9, x11
-; -O0:    orr x8, x8, x11
-; -O0:    subs x8, x8, #0
-;
-; -O1-LABEL: store_atomic_i128_aligned_monotonic:
-; -O1:    ldxp xzr, x8, [x2]
-; -O1:    stxp w8, x0, x1, [x2]
+; CHECK-LABEL: store_atomic_i128_aligned_monotonic:
+; CHECK:    stp x0, x1, [x2]
     store atomic i128 %value, ptr %ptr monotonic, align 16
     ret void
 }
 
 define dso_local void @store_atomic_i128_aligned_release(i128 %value, ptr %ptr) {
-; -O0-LABEL: store_atomic_i128_aligned_release:
-; -O0:    ldxp x10, x9, [x11]
-; -O0:    cmp x10, x12
-; -O0:    cmp x9, x13
-; -O0:    stlxp w8, x14, x15, [x11]
-; -O0:    stlxp w8, x10, x9, [x11]
-; -O0:    eor x8, x10, x8
-; -O0:    eor x11, x9, x11
-; -O0:    orr x8, x8, x11
-; -O0:    subs x8, x8, #0
-;
-; -O1-LABEL: store_atomic_i128_aligned_release:
-; -O1:    ldxp xzr, x8, [x2]
-; -O1:    stlxp w8, x0, x1, [x2]
+; CHECK-LABEL: store_atomic_i128_aligned_release:
+; CHECK:    stilp x0, x1, [x2]
     store atomic i128 %value, ptr %ptr release, align 16
     ret void
 }
 
 define dso_local void @store_atomic_i128_aligned_seq_cst(i128 %value, ptr %ptr) {
-; -O0-LABEL: store_atomic_i128_aligned_seq_cst:
-; -O0:    ldaxp x10, x9, [x11]
-; -O0:    cmp x10, x12
-; -O0:    cmp x9, x13
-; -O0:    stlxp w8, x14, x15, [x11]
-; -O0:    stlxp w8, x10, x9, [x11]
-; -O0:    eor x8, x10, x8
-; -O0:    eor x11, x9, x11
-; -O0:    orr x8, x8, x11
-; -O0:    subs x8, x8, #0
-;
-; -O1-LABEL: store_atomic_i128_aligned_seq_cst:
-; -O1:    ldaxp xzr, x8, [x2]
-; -O1:    stlxp w8, x0, x1, [x2]
+; CHECK-LABEL: store_atomic_i128_aligned_seq_cst:
+; CHECK:    dmb ish
+; CHECK:    stp x0, x1, [x2]
+; CHECK:    dmb ish
     store atomic i128 %value, ptr %ptr seq_cst, align 16
     ret void
 }
 
 define dso_local void @store_atomic_i8_unaligned_unordered(i8 %value, ptr %ptr) {
 ; CHECK-LABEL: store_atomic_i8_unaligned_unordered:
 ; CHECK:    strb w0, [x1]
     store atomic i8 %value, ptr %ptr unordered, align 1

 }
 
 define dso_local void @store_atomic_i128_unaligned_seq_cst(i128 %value, ptr %ptr) {
 ; CHECK-LABEL: store_atomic_i128_unaligned_seq_cst:
 ; CHECK:    bl __atomic_store
     store atomic i128 %value, ptr %ptr seq_cst, align 1
     ret void
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; -O0: {{.*}}
+; -O1: {{.*}}

llvm/test/CodeGen/AArch64/Atomics/aarch64-atomicrmw-rcpc3.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --filter-out "\b(sp)\b" --filter "^\s*(ld[^r]|st[^r]|swp|cas|bl|add|and|eor|orn|orr|sub|mvn|sxt|cmp|ccmp|csel|dmb)"
 ; The base test file was generated by ./llvm/test/CodeGen/AArch64/Atomics/generate-tests.py
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse2,+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse2,+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 
 define dso_local i8 @atomicrmw_xchg_i8_aligned_monotonic(ptr %ptr, i8 %value) {
 ; -O0-LABEL: atomicrmw_xchg_i8_aligned_monotonic:
 ; -O0:    ldaxrb w9, [x11]
 ; -O0:    cmp w9, w10, uxtb
 ; -O0:    stlxrb w8, w12, [x11]
 ; -O0:    and w8, w9, #0xff
 ; -O0:    subs w8, w8, w10, uxtb

llvm/test/CodeGen/AArch64/Atomics/aarch64-cmpxchg-rcpc3.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --filter-out "\b(sp)\b" --filter "^\s*(ld[^r]|st[^r]|swp|cas|bl|add|and|eor|orn|orr|sub|mvn|sxt|cmp|ccmp|csel|dmb)"
 ; The base test file was generated by ./llvm/test/CodeGen/AArch64/Atomics/generate-tests.py
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse2,+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse2,+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 
 define dso_local i8 @cmpxchg_i8_aligned_monotonic_monotonic(i8 %expected, i8 %new, ptr %ptr) {
 ; -O0-LABEL: cmpxchg_i8_aligned_monotonic_monotonic:
 ; -O0:    ldaxrb w0, [x2]
 ; -O0:    cmp w0, w9, uxtb
 ; -O0:    stlxrb w8, w1, [x2]
 ;
 ; -O1-LABEL: cmpxchg_i8_aligned_monotonic_monotonic:

llvm/test/CodeGen/AArch64/Atomics/aarch64-fence.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --filter "^\s*(dmb)"
 ; The base test file was generated by ./llvm/test/CodeGen/AArch64/Atomics/generate-tests.py
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+v8a -O0 | FileCheck %s --check-prefixes=CHECK,-O0
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+v8a -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+v8.1a -O0 | FileCheck %s --check-prefixes=CHECK,-O0
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+v8.1a -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+rcpc -O0 | FileCheck %s --check-prefixes=CHECK,-O0
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+rcpc -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse2 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse2 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+outline-atomics -O0 | FileCheck %s --check-prefixes=CHECK,-O0
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+outline-atomics -O1 | FileCheck %s --check-prefixes=CHECK,-O1
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse2,+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse2,+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse128 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse128 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 
 define dso_local void @fence_acquire() {
 ; CHECK-LABEL: fence_acquire:
 ; CHECK:    dmb ishld
     fence acquire
     ret void

llvm/test/CodeGen/AArch64/Atomics/aarch64_be-atomic-load-rcpc3.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --filter-out "\b(sp)\b" --filter "^\s*(ld|st[^r]|swp|cas|bl|add|and|eor|orn|orr|sub|mvn|sxt|cmp|ccmp|csel|dmb)"
 ; The base test file was generated by ./llvm/test/CodeGen/AArch64/Atomics/generate-tests.py
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse2,+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse2,+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 
 define dso_local i8 @load_atomic_i8_aligned_unordered(ptr %ptr) {
 ; CHECK-LABEL: load_atomic_i8_aligned_unordered:
 ; CHECK:    ldrb w0, [x0]
     %r = load atomic i8, ptr %ptr unordered, align 1
     ret i8 %r
 }
 

 define dso_local i64 @load_atomic_i64_aligned_seq_cst_const(ptr readonly %ptr) {
 ; CHECK-LABEL: load_atomic_i64_aligned_seq_cst_const:
 ; CHECK:    ldar x0, [x0]
     %r = load atomic i64, ptr %ptr seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_unordered(ptr %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_unordered:
-; -O0:    ldxp x1, x0, [x9]
-; -O0:    cmp x1, x10
-; -O0:    cmp x0, x10
-; -O0:    stxp w8, x10, x10, [x9]
-; -O0:    stxp w8, x1, x0, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_unordered:
-; -O1:    ldxp x1, x0, [x8]
-; -O1:    stxp w9, x1, x0, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_unordered:
+; CHECK:    ldp x1, x0, [x0]
     %r = load atomic i128, ptr %ptr unordered, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_unordered_const(ptr readonly %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_unordered_const:
-; -O0:    ldxp x1, x0, [x9]
-; -O0:    cmp x1, x10
-; -O0:    cmp x0, x10
-; -O0:    stxp w8, x10, x10, [x9]
-; -O0:    stxp w8, x1, x0, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_unordered_const:
-; -O1:    ldxp x1, x0, [x8]
-; -O1:    stxp w9, x1, x0, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_unordered_const:
+; CHECK:    ldp x1, x0, [x0]
     %r = load atomic i128, ptr %ptr unordered, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_monotonic(ptr %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_monotonic:
-; -O0:    ldxp x1, x0, [x9]
-; -O0:    cmp x1, x10
-; -O0:    cmp x0, x10
-; -O0:    stxp w8, x10, x10, [x9]
-; -O0:    stxp w8, x1, x0, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_monotonic:
-; -O1:    ldxp x1, x0, [x8]
-; -O1:    stxp w9, x1, x0, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_monotonic:
+; CHECK:    ldp x1, x0, [x0]
     %r = load atomic i128, ptr %ptr monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_monotonic_const(ptr readonly %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_monotonic_const:
-; -O0:    ldxp x1, x0, [x9]
-; -O0:    cmp x1, x10
-; -O0:    cmp x0, x10
-; -O0:    stxp w8, x10, x10, [x9]
-; -O0:    stxp w8, x1, x0, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_monotonic_const:
-; -O1:    ldxp x1, x0, [x8]
-; -O1:    stxp w9, x1, x0, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_monotonic_const:
+; CHECK:    ldp x1, x0, [x0]
     %r = load atomic i128, ptr %ptr monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_acquire(ptr %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_acquire:
-; -O0:    ldaxp x1, x0, [x9]
-; -O0:    cmp x1, x10
-; -O0:    cmp x0, x10
-; -O0:    stxp w8, x10, x10, [x9]
-; -O0:    stxp w8, x1, x0, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_acquire:
-; -O1:    ldaxp x1, x0, [x8]
-; -O1:    stxp w9, x1, x0, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_acquire:
+; CHECK:    ldiapp x1, x0, [x0]
     %r = load atomic i128, ptr %ptr acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_acquire_const(ptr readonly %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_acquire_const:
-; -O0:    ldaxp x1, x0, [x9]
-; -O0:    cmp x1, x10
-; -O0:    cmp x0, x10
-; -O0:    stxp w8, x10, x10, [x9]
-; -O0:    stxp w8, x1, x0, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_acquire_const:
-; -O1:    ldaxp x1, x0, [x8]
-; -O1:    stxp w9, x1, x0, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_acquire_const:
+; CHECK:    ldiapp x1, x0, [x0]
     %r = load atomic i128, ptr %ptr acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_seq_cst(ptr %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_seq_cst:
-; -O0:    ldaxp x1, x0, [x9]
-; -O0:    cmp x1, x10
-; -O0:    cmp x0, x10
-; -O0:    stlxp w8, x10, x10, [x9]
-; -O0:    stlxp w8, x1, x0, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_seq_cst:
-; -O1:    ldaxp x1, x0, [x8]
-; -O1:    stlxp w9, x1, x0, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_seq_cst:
+; CHECK:    ldp x1, x0, [x0]
+; CHECK:    dmb ish
     %r = load atomic i128, ptr %ptr seq_cst, align 16
     ret i128 %r
 }
 
 define dso_local i128 @load_atomic_i128_aligned_seq_cst_const(ptr readonly %ptr) {
-; -O0-LABEL: load_atomic_i128_aligned_seq_cst_const:
-; -O0:    ldaxp x1, x0, [x9]
-; -O0:    cmp x1, x10
-; -O0:    cmp x0, x10
-; -O0:    stlxp w8, x10, x10, [x9]
-; -O0:    stlxp w8, x1, x0, [x9]
-;
-; -O1-LABEL: load_atomic_i128_aligned_seq_cst_const:
-; -O1:    ldaxp x1, x0, [x8]
-; -O1:    stlxp w9, x1, x0, [x8]
+; CHECK-LABEL: load_atomic_i128_aligned_seq_cst_const:
+; CHECK:    ldp x1, x0, [x0]
+; CHECK:    dmb ish
     %r = load atomic i128, ptr %ptr seq_cst, align 16
     ret i128 %r
 }
 
 define dso_local i8 @load_atomic_i8_unaligned_unordered(ptr %ptr) {
 ; CHECK-LABEL: load_atomic_i8_unaligned_unordered:
 ; CHECK:    ldrb w0, [x0]
     %r = load atomic i8, ptr %ptr unordered, align 1

 }
 
 define dso_local i128 @load_atomic_i128_unaligned_seq_cst_const(ptr readonly %ptr) {
 ; CHECK-LABEL: load_atomic_i128_unaligned_seq_cst_const:
 ; CHECK:    bl __atomic_load
     %r = load atomic i128, ptr %ptr seq_cst, align 1
     ret i128 %r
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; -O0: {{.*}}
+; -O1: {{.*}}

llvm/test/CodeGen/AArch64/Atomics/aarch64_be-atomic-store-rcpc3.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --filter-out "\b(sp)\b" --filter "^\s*(ld[^r]|st|swp|cas|bl|add|and|eor|orn|orr|sub|mvn|sxt|cmp|ccmp|csel|dmb)"
 ; The base test file was generated by ./llvm/test/CodeGen/AArch64/Atomics/generate-tests.py
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse2,+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse2,+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 
 define dso_local void @store_atomic_i8_aligned_unordered(i8 %value, ptr %ptr) {
 ; CHECK-LABEL: store_atomic_i8_aligned_unordered:
 ; CHECK:    strb w0, [x1]
     store atomic i8 %value, ptr %ptr unordered, align 1
     ret void
 }
 

 define dso_local void @store_atomic_i64_aligned_seq_cst(i64 %value, ptr %ptr) {
 ; CHECK-LABEL: store_atomic_i64_aligned_seq_cst:
 ; CHECK:    stlr x0, [x1]
     store atomic i64 %value, ptr %ptr seq_cst, align 8
     ret void
 }
 
 define dso_local void @store_atomic_i128_aligned_unordered(i128 %value, ptr %ptr) {
-; -O0-LABEL: store_atomic_i128_aligned_unordered:
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
-;
-; -O1-LABEL: store_atomic_i128_aligned_unordered:
-; -O1:    ldxp xzr, x8, [x2]
-; -O1:    stxp w8, x1, x0, [x2]
+; CHECK-LABEL: store_atomic_i128_aligned_unordered:
+; CHECK:    stp x1, x0, [x2]
     store atomic i128 %value, ptr %ptr unordered, align 16
     ret void
 }
 
 define dso_local void @store_atomic_i128_aligned_monotonic(i128 %value, ptr %ptr) {
-; -O0-LABEL: store_atomic_i128_aligned_monotonic:
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
-;
-; -O1-LABEL: store_atomic_i128_aligned_monotonic:
-; -O1:    ldxp xzr, x8, [x2]
-; -O1:    stxp w8, x1, x0, [x2]
+; CHECK-LABEL: store_atomic_i128_aligned_monotonic:
+; CHECK:    stp x1, x0, [x2]
     store atomic i128 %value, ptr %ptr monotonic, align 16
     ret void
 }
 
 define dso_local void @store_atomic_i128_aligned_release(i128 %value, ptr %ptr) {
-; -O0-LABEL: store_atomic_i128_aligned_release:
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
-;
-; -O1-LABEL: store_atomic_i128_aligned_release:
-; -O1:    ldxp xzr, x8, [x2]
-; -O1:    stlxp w8, x1, x0, [x2]
+; CHECK-LABEL: store_atomic_i128_aligned_release:
+; CHECK:    stilp x1, x0, [x2]
     store atomic i128 %value, ptr %ptr release, align 16
     ret void
 }
 
 define dso_local void @store_atomic_i128_aligned_seq_cst(i128 %value, ptr %ptr) {
-; -O0-LABEL: store_atomic_i128_aligned_seq_cst:
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
-;
-; -O1-LABEL: store_atomic_i128_aligned_seq_cst:
-; -O1:    ldaxp xzr, x8, [x2]
-; -O1:    stlxp w8, x1, x0, [x2]
+; CHECK-LABEL: store_atomic_i128_aligned_seq_cst:
+; CHECK:    dmb ish
+; CHECK:    stp x1, x0, [x2]
+; CHECK:    dmb ish
     store atomic i128 %value, ptr %ptr seq_cst, align 16
     ret void
 }
 
 define dso_local void @store_atomic_i8_unaligned_unordered(i8 %value, ptr %ptr) {
 ; CHECK-LABEL: store_atomic_i8_unaligned_unordered:
 ; CHECK:    strb w0, [x1]
     store atomic i8 %value, ptr %ptr unordered, align 1

 }
 
 define dso_local void @store_atomic_i128_unaligned_seq_cst(i128 %value, ptr %ptr) {
 ; CHECK-LABEL: store_atomic_i128_unaligned_seq_cst:
 ; CHECK:    bl __atomic_store
     store atomic i128 %value, ptr %ptr seq_cst, align 1
     ret void
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; -O0: {{.*}}
+; -O1: {{.*}}

llvm/test/CodeGen/AArch64/Atomics/aarch64_be-atomicrmw-rcpc3.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --filter-out "\b(sp)\b" --filter "^\s*(ld[^r]|st[^r]|swp|cas|bl|add|and|eor|orn|orr|sub|mvn|sxt|cmp|ccmp|csel|dmb)"
 ; The base test file was generated by ./llvm/test/CodeGen/AArch64/Atomics/generate-tests.py
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse2,+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse2,+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 
 define dso_local i8 @atomicrmw_xchg_i8_aligned_monotonic(ptr %ptr, i8 %value) {
 ; -O0-LABEL: atomicrmw_xchg_i8_aligned_monotonic:
 ; -O0:    ldaxrb w8, [x11]
 ; -O0:    cmp w8, w9, uxtb
 ; -O0:    stlxrb w10, w12, [x11]
 ; -O0:    subs w9, w8, w9, uxtb
 ; -O0:    subs w9, w9, #1

llvm/test/CodeGen/AArch64/Atomics/aarch64_be-cmpxchg-rcpc3.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --filter-out "\b(sp)\b" --filter "^\s*(ld[^r]|st[^r]|swp|cas|bl|add|and|eor|orn|orr|sub|mvn|sxt|cmp|ccmp|csel|dmb)"
 ; The base test file was generated by ./llvm/test/CodeGen/AArch64/Atomics/generate-tests.py
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse2,+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse2,+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 
 define dso_local i8 @cmpxchg_i8_aligned_monotonic_monotonic(i8 %expected, i8 %new, ptr %ptr) {
 ; -O0-LABEL: cmpxchg_i8_aligned_monotonic_monotonic:
 ; -O0:    ldaxrb w0, [x2]
 ; -O0:    cmp w0, w9, uxtb
 ; -O0:    stlxrb w8, w1, [x2]
 ;
 ; -O1-LABEL: cmpxchg_i8_aligned_monotonic_monotonic:

llvm/test/CodeGen/AArch64/Atomics/aarch64_be-fence.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --filter "^\s*(dmb)"
 ; The base test file was generated by ./llvm/test/CodeGen/AArch64/Atomics/generate-tests.py
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+v8a -O0 | FileCheck %s --check-prefixes=CHECK,-O0
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+v8a -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+v8.1a -O0 | FileCheck %s --check-prefixes=CHECK,-O0
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+v8.1a -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+rcpc -O0 | FileCheck %s --check-prefixes=CHECK,-O0
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+rcpc -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse2 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse2 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+outline-atomics -O0 | FileCheck %s --check-prefixes=CHECK,-O0
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+outline-atomics -O1 | FileCheck %s --check-prefixes=CHECK,-O1
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
-; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse2,+rcpc3 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse2,+rcpc3 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse128 -O0 | FileCheck %s --check-prefixes=CHECK,-O0
 ; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64_be -mattr=+lse128 -O1 | FileCheck %s --check-prefixes=CHECK,-O1
 
 define dso_local void @fence_acquire() {
 ; CHECK-LABEL: fence_acquire:
 ; CHECK:    dmb ishld
     fence acquire
     ret void

llvm/test/CodeGen/AArch64/Atomics/generate-tests.py

FENCE_ORDERS = [
     AtomicOrder.acquire,
     AtomicOrder.release,
     AtomicOrder.acq_rel,
     AtomicOrder.seq_cst,
 ]
 
 
 class Feature(enum.Flag):
+    # Feature names in filenames are determined by the spelling here:
     v8a = enum.auto()
     v8_1a = enum.auto()  # -mattr=+v8.1a, mandatory FEAT_LOR, FEAT_LSE
     rcpc = enum.auto()  # FEAT_LRCPC
     lse2 = enum.auto()  # FEAT_LSE2
     outline_atomics = enum.auto()  # -moutline-atomics
-    rcpc3 = enum.auto()  # FEAT_LRCPC3
+    rcpc3 = enum.auto()  # FEAT_LSE2 + FEAT_LRCPC3
     lse128 = enum.auto()  # FEAT_LSE128
 
     @property
     def mattr(self):
         if self == Feature.outline_atomics:
-            return 'outline-atomics'
+            return '+outline-atomics'
         if self == Feature.v8_1a:
-            return 'v8.1a'
-        return self.name
+            return '+v8.1a'
+        if self == Feature.rcpc3:
+            return '+lse2,+rcpc3'
+        return '+' + self.name
 
 
 ATOMICRMW_OPS = [
     'xchg',
     'add',
     'sub',
     'and',
     'nand',
f.write('; NOTE: Assertions have been autogenerated by '
             'utils/update_llc_test_checks.py UTC_ARGS: ')
     f.write(filter_args)
     f.write('\n')
     f.write(f'; The base test file was generated by {__file__}\n')
     for feat in features:
         for OptFlag in ['-O0', '-O1']:
             f.write(' '.join([
                 ';', 'RUN:', 'llc', '%s', '-o', '-', '-verify-machineinstrs',
-                f'-mtriple={triple}', f'-mattr=+{feat.mattr}', OptFlag, '|',
+                f'-mtriple={triple}', f'-mattr={feat.mattr}', OptFlag, '|',
                 'FileCheck', '%s', f'--check-prefixes=CHECK,{OptFlag}\n'
             ]))
 
 
 def write_lit_tests():
     os.chdir('llvm/test/CodeGen/AArch64/Atomics/')
     for triple in TRIPLES:
         # Feature has no effect on fence, so keep it to one file.