AArch64: support i128 cmpxchg in GlobalISel.

[t.p.northover]

Summary

Most operations get expanded to cmpxchg at i128 because it's the only primitive we have. So this is the bulk of the support needed for all i128 atomics.

There are three essentially different cases to handle:

• -O0, no LSE. The IR is expanded to ldxp/stxp and we need patterns & code to select them.
• -O1, no LSE. We get G_ATOMIC_CMPXCHG, and need to produce CMP_SWAP_N pseudos. The registers are all 64-bit so this is easy in the existing framework.
• LSE. We get G_ATOMIC_CMPXCHG and need to produce a CASP instruction with XSeqPair registers.

The last case is by far the hardest, and and adds 128-bit GPR support as a byproduct. The only way I could get it through all phases (and I tried many variants) was:

G_UNMERGE
REG_SEQUENCE
CAS
G_EXTRACT
G_MERGE

The merge/unmerge pair are needed to cancel out similar in surrounding instructions, but then the REG_SEQUENCE and G_EXTRACT are needed to put the values back into 128-bit vregs again.

Unfortunately the G_EXTRACT had to be handled with quite a nasty hack in RegisterBankInfo. Please tell me if anyone has any better ideas, but I couldn't make a RegBank stick from Legalizer to there, or think of any other way to check for an int op.

[paquette]

I think it would be good to add dedicated legalizer + regbankselect MIR tests?

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

2553

It seems like this assert can probably be replaced with something like

assert(SrcRB.getID() == DstRB.getID() && "Src and dst register banks must match!");

And then moved above the if (SrcRB.getID() == AArch64::GPRRegBankID...

llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp

1059

Some comments on the various cases would be nice here? (e.g. LSE, no LSE, etc.)

llvm/lib/Target/AArch64/GISel/AArch64RegisterBankInfo.cpp

880

I guess this is just for the s128 case, so maybe this can be refactored a little

...
if (SrcTy.getSizeInBits() != 128)
   break;
auto Idx = (MRI.getRegClassOrNull(Src) == &AArch64::XSeqPairsClassRegClass) ? PMI_FirstGPR : PMI_FirstFPR;
...

Can't think of a better way to decide this without checking the register class directly, unfortunately.

[t.p.northover]

Thanks for the improvements, I think I've implemented them.

[paquette]

LGTM

[paquette]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

2559

With the above assert, you don't have to check both SrcRB and DstRB here.

[efriedma]

This might be a bit crazy, but instead of emitting CASPX directly from globalisel, did you consider just emitting CMP_SWAP_128, then converting it to CASPX after isel? Not sure it's worth shifting around the complexity like this, but maybe worth considering.

• -O0, no LSE. The IR is expanded to ldxp/stxp and we need patterns & code to select them.
• -O1, no LSE. We get G_ATOMIC_CMPXCHG, and need to produce CMP_SWAP_N pseudos. The registers are all 64-bit so this is easy in the existing framework.
• LSE. We get G_ATOMIC_CMPXCHG and need to produce a CASP instruction with XSeqPair registers.

I think you accidentally switched -O0 and -O1.

[t.p.northover]

This might be a bit crazy, but instead of emitting CASPX directly from globalisel, did you consider just emitting CMP_SWAP_128, then converting it to CASPX after isel?

Unforuntately that won't work. The reason CMP_SWAP_128 works so easily is that its 64-bit register operands can be chosen independently (because LDXP/STXP allow that), CASP needs them to be consecutive (and the first one to be even).

I think you accidentally switched -O0 and -O1.

Yes. Thanks, I'll fix it.

[t.p.northover]

Thanks both. I've committed it as 4789fc75d350.

[efriedma]

In D101349#2759075, @t.p.northover wrote:

This might be a bit crazy, but instead of emitting CASPX directly from globalisel, did you consider just emitting CMP_SWAP_128, then converting it to CASPX after isel?

Unforuntately that won't work. The reason CMP_SWAP_128 works so easily is that its 64-bit register operands can be chosen independently (because LDXP/STXP allow that), CASP needs them to be consecutive (and the first one to be even).

I don't see any reason it can't work, as long as you make sure to do the conversion before regalloc. (After regalloc would be a problem for the reason you note.)

llvm/lib/Target/AArch64/AArch64GenRegisterBankInfo.def

RegisterBankInfo::PartialMapping AArch64GenRegisterBankInfo::PartMappings[]{
     // 4: FPR 256-bit value.
     {0, 256, AArch64::FPRRegBank},
     // 5: FPR 512-bit value.
     {0, 512, AArch64::FPRRegBank},
     // 6: GPR 32-bit value.
     {0, 32, AArch64::GPRRegBank},
     // 7: GPR 64-bit value.
     {0, 64, AArch64::GPRRegBank},
+    // 8: GPR 128-bit value.
+    {0, 128, AArch64::GPRRegBank},
 };
 
 // ValueMappings.
 RegisterBankInfo::ValueMapping AArch64GenRegisterBankInfo::ValMappings[]{
     /* BreakDown, NumBreakDowns */
     // 0: invalid
     {nullptr, 0},
     // 3-operands instructions (all binary operations should end up with one of
RegisterBankInfo::ValueMapping AArch64GenRegisterBankInfo::ValMappings[]{
     // 16: FPR 512-bit value.
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR512 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR512 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR512 - PMI_Min], 1},
     // 19: GPR 32-bit value.
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR32 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR32 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR32 - PMI_Min], 1},
-    // 22: GPR 64-bit value. <-- This must match Last3OpsIdx.
+    // 22: GPR 64-bit value.
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR64 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR64 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR64 - PMI_Min], 1},
+    // 25: GPR 128-bit value. <-- This must match Last3OpsIdx.
+    {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR128 - PMI_Min], 1},
+    {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR128 - PMI_Min], 1},
+    {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR128 - PMI_Min], 1},
     // Cross register bank copies.
-    // 25: FPR 16-bit value to GPR 16-bit. <-- This must match
+    // 28: FPR 16-bit value to GPR 16-bit. <-- This must match
     //                                         FirstCrossRegCpyIdx.
     // Note: This is the kind of copy we see with physical registers.
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR16 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR32 - PMI_Min], 1},
-    // 27: FPR 32-bit value to GPR 32-bit value.
+    // 30: FPR 32-bit value to GPR 32-bit value.
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR32 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR32 - PMI_Min], 1},
-    // 29: FPR 64-bit value to GPR 64-bit value.
+    // 32: FPR 64-bit value to GPR 64-bit value.
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR64 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR64 - PMI_Min], 1},
-    // 31: FPR 128-bit value to GPR 128-bit value (invalid)
+    // 34: FPR 128-bit value to GPR 128-bit value (invalid)
     {nullptr, 1},
     {nullptr, 1},
-    // 33: FPR 256-bit value to GPR 256-bit value (invalid)
+    // 36: FPR 256-bit value to GPR 256-bit value (invalid)
     {nullptr, 1},
     {nullptr, 1},
-    // 35: FPR 512-bit value to GPR 512-bit value (invalid)
+    // 38: FPR 512-bit value to GPR 512-bit value (invalid)
     {nullptr, 1},
     {nullptr, 1},
-    // 37: GPR 32-bit value to FPR 32-bit value.
+    // 40: GPR 32-bit value to FPR 32-bit value.
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR32 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR32 - PMI_Min], 1},
-    // 39: GPR 64-bit value to FPR 64-bit value. <-- This must match
+    // 42: GPR 64-bit value to FPR 64-bit value. <-- This must match
     //                                               LastCrossRegCpyIdx.
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR64 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR64 - PMI_Min], 1},
-    // 41: FPExt: 16 to 32. <-- This must match FPExt16To32Idx.
+    // 44: FPExt: 16 to 32. <-- This must match FPExt16To32Idx.
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR32 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR16 - PMI_Min], 1},
-    // 43: FPExt: 16 to 32. <-- This must match FPExt16To64Idx.
+    // 46: FPExt: 16 to 32. <-- This must match FPExt16To64Idx.
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR64 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR16 - PMI_Min], 1},
-    // 45: FPExt: 32 to 64. <-- This must match FPExt32To64Idx.
+    // 48: FPExt: 32 to 64. <-- This must match FPExt32To64Idx.
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR64 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR32 - PMI_Min], 1},
-    // 47: FPExt vector: 64 to 128. <-- This must match FPExt64To128Idx.
+    // 50: FPExt vector: 64 to 128. <-- This must match FPExt64To128Idx.
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR128 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_FPR64 - PMI_Min], 1},
-    // 49: Shift scalar with 64 bit shift imm
+    // 52: Shift scalar with 64 bit shift imm
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR32 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR32 - PMI_Min], 1},
     {&AArch64GenRegisterBankInfo::PartMappings[PMI_GPR64 - PMI_Min], 1},
 };
 
 bool AArch64GenRegisterBankInfo::checkPartialMap(unsigned Idx,
                                                  unsigned ValStartIdx,
                                                  unsigned ValLength,

 
 unsigned AArch64GenRegisterBankInfo::getRegBankBaseIdxOffset(unsigned RBIdx,
                                                              unsigned Size) {
   if (RBIdx == PMI_FirstGPR) {
     if (Size <= 32)
       return 0;
     if (Size <= 64)
       return 1;
+    if (Size <= 128)
+      return 2;
     return -1;
   }
   if (RBIdx == PMI_FirstFPR) {
     if (Size <= 16)
       return 0;
     if (Size <= 32)
       return 1;
     if (Size <= 64)

llvm/lib/Target/AArch64/AArch64InstrGISel.td

def : Pat<(atomic_cmp_swap_16 GPR64:$addr, GPR32:$desired, GPR32:$new),
           (CMP_SWAP_16 GPR64:$addr, GPR32:$desired, GPR32:$new)>;
 
 def : Pat<(atomic_cmp_swap_32 GPR64:$addr, GPR32:$desired, GPR32:$new),
           (CMP_SWAP_32 GPR64:$addr, GPR32:$desired, GPR32:$new)>;
 
 def : Pat<(atomic_cmp_swap_64 GPR64:$addr, GPR64:$desired, GPR64:$new),
           (CMP_SWAP_64 GPR64:$addr, GPR64:$desired, GPR64:$new)>;
 }
+
+def : Pat<(int_aarch64_stlxp GPR64:$lo, GPR64:$hi, GPR64:$addr),
+          (STLXPX GPR64:$lo, GPR64:$hi, GPR64:$addr)>;
+def : Pat<(int_aarch64_stxp GPR64:$lo, GPR64:$hi, GPR64:$addr),
+          (STXPX GPR64:$lo, GPR64:$hi, GPR64:$addr)>;

llvm/lib/Target/AArch64/AArch64RegisterBanks.td

 //=- AArch64RegisterBank.td - Describe the AArch64 Banks -----*- tablegen -*-=//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 //
 //===----------------------------------------------------------------------===//
 
 /// General Purpose Registers: W, X.
-def GPRRegBank : RegisterBank<"GPR", [GPR64all]>;
+def GPRRegBank : RegisterBank<"GPR", [XSeqPairsClass]>;
 
 /// Floating Point/Vector Registers: B, H, S, D, Q.
 def FPRRegBank : RegisterBank<"FPR", [QQQQ]>;
 
 /// Conditional register: NZCV.
 def CCRegBank : RegisterBank<"CC", [CCR]>;

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

getRegClassForTypeOnBank(LLT Ty, const RegisterBank &RB,
                          bool GetAllRegSet = false) {
   if (RB.getID() == AArch64::GPRRegBankID) {
     if (Ty.getSizeInBits() <= 32)
       return GetAllRegSet ? &AArch64::GPR32allRegClass
                           : &AArch64::GPR32RegClass;
     if (Ty.getSizeInBits() == 64)
       return GetAllRegSet ? &AArch64::GPR64allRegClass
                           : &AArch64::GPR64RegClass;
+    if (Ty.getSizeInBits() == 128)
+      return &AArch64::XSeqPairsClassRegClass;
     return nullptr;
   }
 
   if (RB.getID() == AArch64::FPRRegBankID) {
     if (Ty.getSizeInBits() <= 16)
       return &AArch64::FPR16RegClass;
     if (Ty.getSizeInBits() == 32)
       return &AArch64::FPR32RegClass;
getMinClassForRegBank(const RegisterBank &RB, unsigned SizeInBits,
 
   if (RegBankID == AArch64::GPRRegBankID) {
     if (SizeInBits <= 32)
       return GetAllRegSet ? &AArch64::GPR32allRegClass
                           : &AArch64::GPR32RegClass;
     if (SizeInBits == 64)
       return GetAllRegSet ? &AArch64::GPR64allRegClass
                           : &AArch64::GPR64RegClass;
+    if (SizeInBits == 128)
+      return &AArch64::XSeqPairsClassRegClass;
   }
 
   if (RegBankID == AArch64::FPRRegBankID) {
     switch (SizeInBits) {
     default:
       return nullptr;
     case 8:
       return &AArch64::FPR8RegClass;
case TargetOpcode::G_EXTRACT: {
     if (SrcTy.getSizeInBits() > 64) {
       // This should be an extract of an s128, which is like a vector extract.
       if (SrcTy.getSizeInBits() != 128)
         return false;
       // Only support extracting 64 bits from an s128 at the moment.
       if (DstTy.getSizeInBits() != 64)
         return false;
 
+      unsigned Offset = I.getOperand(2).getImm();
+      if (Offset % 64 != 0)
+        return false;
+
       const RegisterBank &SrcRB = *RBI.getRegBank(SrcReg, MRI, TRI);
       const RegisterBank &DstRB = *RBI.getRegBank(DstReg, MRI, TRI);
+      if (SrcRB.getID() == AArch64::GPRRegBankID &&
+          DstRB.getID() == AArch64::GPRRegBankID) {
+        MIB.buildInstr(TargetOpcode::COPY, {DstReg}, {})
+            .addUse(SrcReg, 0, Offset == 0 ? AArch64::sube64 : AArch64::subo64);
+        I.eraseFromParent();
+        return true;
+      }
+
       // Check we have the right regbank always.
       assert(SrcRB.getID() == AArch64::FPRRegBankID &&
              DstRB.getID() == AArch64::FPRRegBankID &&
              "Wrong extract regbank!");
       (void)SrcRB;
 
       // Emit the same code as a vector extract.
       // Offset must be a multiple of 64.
-      unsigned Offset = I.getOperand(2).getImm();
-      if (Offset % 64 != 0)
-        return false;
       unsigned LaneIdx = Offset / 64;
       MachineIRBuilder MIB(I);
       MachineInstr *Extract = emitExtractVectorElt(
           DstReg, DstRB, LLT::scalar(64), SrcReg, LaneIdx, MIB);
       if (!Extract)
         return false;
       I.eraseFromParent();
       return true;
bool AArch64InstructionSelector::selectIntrinsicWithSideEffects(
   if (!IntrinID)
     return false;
   MachineIRBuilder MIRBuilder(I);
 
   // Select the instruction.
   switch (IntrinID) {
   default:
     return false;
+  case Intrinsic::aarch64_ldxp:
+  case Intrinsic::aarch64_ldaxp: {
+    auto NewI = MIRBuilder.buildInstr(
+        IntrinID == Intrinsic::aarch64_ldxp ? AArch64::LDXPX : AArch64::LDAXPX,
+        {I.getOperand(0).getReg(), I.getOperand(1).getReg()},
+        {I.getOperand(3)});
+    NewI.cloneMemRefs(I);
+    break;
+  }
   case Intrinsic::trap:
     MIRBuilder.buildInstr(AArch64::BRK, {}, {}).addImm(1);
     break;
   case Intrinsic::debugtrap:
     MIRBuilder.buildInstr(AArch64::BRK, {}, {}).addImm(0xF000);
     break;
   case Intrinsic::ubsantrap:
     MIRBuilder.buildInstr(AArch64::BRK, {}, {})

llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.h

bool legalizeSmallCMGlobalValue(MachineInstr &MI, MachineRegisterInfo &MRI,
                                   GISelChangeObserver &Observer) const;
   bool legalizeVectorTrunc(MachineInstr &MI, LegalizerHelper &Helper) const;
   bool legalizeBitfieldExtract(MachineInstr &MI, MachineRegisterInfo &MRI,
                                LegalizerHelper &Helper) const;
   bool legalizeRotate(MachineInstr &MI, MachineRegisterInfo &MRI,
                       LegalizerHelper &Helper) const;
   bool legalizeCTPOP(MachineInstr &MI, MachineRegisterInfo &MRI,
                      LegalizerHelper &Helper) const;
+  bool legalizeAtomicCmpxchg128(MachineInstr &MI, MachineRegisterInfo &MRI,
+                                LegalizerHelper &Helper) const;
   const AArch64Subtarget *ST;
 };
 } // End llvm namespace.
 #endif

llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp

 //===- AArch64LegalizerInfo.cpp ----------------------------------*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
 /// This file implements the targeting of the Machinelegalizer class for
 /// AArch64.
 /// \todo This should be generated by TableGen.
 //===----------------------------------------------------------------------===//
 
 #include "AArch64LegalizerInfo.h"
+#include "AArch64RegisterBankInfo.h"
 #include "AArch64Subtarget.h"
 #include "llvm/CodeGen/GlobalISel/LegalizerHelper.h"
 #include "llvm/CodeGen/GlobalISel/LegalizerInfo.h"
 #include "llvm/CodeGen/GlobalISel/MachineIRBuilder.h"
 #include "llvm/CodeGen/GlobalISel/Utils.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/TargetOpcodes.h"
AArch64LegalizerInfo::AArch64LegalizerInfo(const AArch64Subtarget &ST)
   // va_list must be a pointer, but most sized types are pretty easy to handle
   // as the destination.
   getActionDefinitionsBuilder(G_VAARG)
       .customForCartesianProduct({s8, s16, s32, s64, p0}, {p0})
       .clampScalar(0, s8, s64)
       .widenScalarToNextPow2(0, /*Min*/ 8);
 
   getActionDefinitionsBuilder(G_ATOMIC_CMPXCHG_WITH_SUCCESS)
       .lowerIf(
-          all(typeInSet(0, {s8, s16, s32, s64}), typeIs(1, s1), typeIs(2, p0)));
+          all(typeInSet(0, {s8, s16, s32, s64, s128}), typeIs(1, s1), typeIs(2, p0)));
+
+  getActionDefinitionsBuilder(G_ATOMIC_CMPXCHG)
+      .legalIf(all(typeInSet(0, {s8, s16, s32, s64}), typeIs(1, p0)))
+      .customIf([](const LegalityQuery &Query) {
+        return Query.Types[0].getSizeInBits() == 128;
+      });
 
   getActionDefinitionsBuilder(
       {G_ATOMICRMW_XCHG, G_ATOMICRMW_ADD, G_ATOMICRMW_SUB, G_ATOMICRMW_AND,
        G_ATOMICRMW_OR, G_ATOMICRMW_XOR, G_ATOMICRMW_MIN, G_ATOMICRMW_MAX,
-       G_ATOMICRMW_UMIN, G_ATOMICRMW_UMAX, G_ATOMIC_CMPXCHG})
-      .legalIf(all(
-          typeInSet(0, {s8, s16, s32, s64}), typeIs(1, p0)));
+       G_ATOMICRMW_UMIN, G_ATOMICRMW_UMAX})
+      .legalIf(all(typeInSet(0, {s8, s16, s32, s64}), typeIs(1, p0)));
 
   getActionDefinitionsBuilder(G_BLOCK_ADDR).legalFor({p0});
 
   // Merge/Unmerge
   for (unsigned Op : {G_MERGE_VALUES, G_UNMERGE_VALUES}) {
     unsigned BigTyIdx = Op == G_MERGE_VALUES ? 0 : 1;
     unsigned LitTyIdx = Op == G_MERGE_VALUES ? 1 : 0;
 
case TargetOpcode::G_TRUNC:
     return legalizeVectorTrunc(MI, Helper);
   case TargetOpcode::G_SBFX:
   case TargetOpcode::G_UBFX:
     return legalizeBitfieldExtract(MI, MRI, Helper);
   case TargetOpcode::G_ROTR:
     return legalizeRotate(MI, MRI, Helper);
   case TargetOpcode::G_CTPOP:
     return legalizeCTPOP(MI, MRI, Helper);
+  case TargetOpcode::G_ATOMIC_CMPXCHG:
+    return legalizeAtomicCmpxchg128(MI, MRI, Helper);
   }
 
   llvm_unreachable("expected switch to return");
 }
 
 bool AArch64LegalizerInfo::legalizeRotate(MachineInstr &MI,
                                           MachineRegisterInfo &MRI,
                                           LegalizerHelper &Helper) const {
auto UADDLV =
           .addUse(CTPOP.getReg(0));
   if (Size == 64)
     MIRBuilder.buildZExt(Dst, UADDLV);
   else
     UADDLV->getOperand(0).setReg(Dst);
   MI.eraseFromParent();
   return true;
 }
+
+bool AArch64LegalizerInfo::legalizeAtomicCmpxchg128(MachineInstr &MI,
+                                         MachineRegisterInfo &MRI,
+                                         LegalizerHelper &Helper) const {
+  MachineIRBuilder &MIRBuilder = Helper.MIRBuilder;
+  LLT s64 = LLT::scalar(64);
+  auto Addr = MI.getOperand(1).getReg();
+  auto DesiredI = MIRBuilder.buildUnmerge({s64, s64}, MI.getOperand(2));
+  auto NewI = MIRBuilder.buildUnmerge({s64, s64}, MI.getOperand(3));
+  auto DstLo = MRI.createGenericVirtualRegister(s64);
+  auto DstHi = MRI.createGenericVirtualRegister(s64);
+
+  MachineInstrBuilder CAS;
+  if (ST->hasLSE()) {
+    auto Ordering = (*MI.memoperands_begin())->getOrdering();
+    unsigned Opcode;
+    switch(Ordering) {
+    case AtomicOrdering::Acquire:
+      Opcode = AArch64::CASPAX;
+      break;
+    case AtomicOrdering::Release:
+      Opcode = AArch64::CASPLX;
+      break;
+    case AtomicOrdering::AcquireRelease:
+    case AtomicOrdering::SequentiallyConsistent:
+      Opcode = AArch64::CASPALX;
+      break;
+    default:
+      Opcode = AArch64::CASPX;
+      break;
+    }
+
+    LLT s128 = LLT::scalar(128);
+    auto CASDst = MRI.createGenericVirtualRegister(s128);
+    auto CASDesired = MRI.createGenericVirtualRegister(s128);
+    auto CASNew = MRI.createGenericVirtualRegister(s128);
+    MIRBuilder.buildInstr(TargetOpcode::REG_SEQUENCE, {CASDesired}, {})
+        .addUse(DesiredI->getOperand(0).getReg())
+        .addImm(AArch64::sube64)
+        .addUse(DesiredI->getOperand(1).getReg())
+        .addImm(AArch64::subo64);
+    MIRBuilder.buildInstr(TargetOpcode::REG_SEQUENCE, {CASNew}, {})
+        .addUse(NewI->getOperand(0).getReg())
+        .addImm(AArch64::sube64)
+        .addUse(NewI->getOperand(1).getReg())
+        .addImm(AArch64::subo64);
+
+    CAS = MIRBuilder.buildInstr(Opcode, {CASDst}, {CASDesired, CASNew, Addr});
+
+    MIRBuilder.buildExtract({DstLo}, {CASDst}, 0);
+    MIRBuilder.buildExtract({DstHi}, {CASDst}, 64);
+  } else {
+    auto Scratch = MRI.createVirtualRegister(&AArch64::GPR64RegClass);
+    CAS = MIRBuilder.buildInstr(AArch64::CMP_SWAP_128, {DstLo, DstHi, Scratch},
+                                {Addr, DesiredI->getOperand(0),
+                                 DesiredI->getOperand(1), NewI->getOperand(0),
+                                 NewI->getOperand(1)});
+  }
+
+  CAS.cloneMemRefs(MI);
+  constrainSelectedInstRegOperands(*CAS, *ST->getInstrInfo(),
+                                   *MRI.getTargetRegisterInfo(),
+                                   *ST->getRegBankInfo());
+
+  MIRBuilder.buildMerge(MI.getOperand(0), {DstLo, DstHi});
+  MI.eraseFromParent();
+  return true;
+}

llvm/lib/Target/AArch64/GISel/AArch64RegisterBankInfo.h

enum PartialMappingIdx {
     PMI_FPR16 = 1,
     PMI_FPR32,
     PMI_FPR64,
     PMI_FPR128,
     PMI_FPR256,
     PMI_FPR512,
     PMI_GPR32,
     PMI_GPR64,
+    PMI_GPR128,
     PMI_FirstGPR = PMI_GPR32,
-    PMI_LastGPR = PMI_GPR64,
+    PMI_LastGPR = PMI_GPR128,
     PMI_FirstFPR = PMI_FPR16,
     PMI_LastFPR = PMI_FPR512,
     PMI_Min = PMI_FirstFPR,
   };
 
   static RegisterBankInfo::PartialMapping PartMappings[];
   static RegisterBankInfo::ValueMapping ValMappings[];
   static PartialMappingIdx BankIDToCopyMapIdx[];
 
   enum ValueMappingIdx {
     InvalidIdx = 0,
     First3OpsIdx = 1,
-    Last3OpsIdx = 22,
+    Last3OpsIdx = 25,
     DistanceBetweenRegBanks = 3,
-    FirstCrossRegCpyIdx = 25,
-    LastCrossRegCpyIdx = 39,
+    FirstCrossRegCpyIdx = 28,
+    LastCrossRegCpyIdx = 42,
     DistanceBetweenCrossRegCpy = 2,
-    FPExt16To32Idx = 41,
-    FPExt16To64Idx = 43,
-    FPExt32To64Idx = 45,
-    FPExt64To128Idx = 47,
-    Shift64Imm = 49
+    FPExt16To32Idx = 44,
+    FPExt16To64Idx = 46,
+    FPExt32To64Idx = 48,
+    FPExt64To128Idx = 50,
+    Shift64Imm = 52,
   };
 
   static bool checkPartialMap(unsigned Idx, unsigned ValStartIdx,
                               unsigned ValLength, const RegisterBank &RB);
   static bool checkValueMapImpl(unsigned Idx, unsigned FirstInBank,
                                 unsigned Size, unsigned Offset);
   static bool checkPartialMappingIdx(PartialMappingIdx FirstAlias,
                                      PartialMappingIdx LastAlias,

llvm/lib/Target/AArch64/GISel/AArch64RegisterBankInfo.cpp

static auto InitializeRegisterBankOnce = [&]() {
     (void)RBCCR;
     assert(&AArch64::CCRegBank == &RBCCR &&
            "The order in RegBanks is messed up");
 
     // The GPR register bank is fully defined by all the registers in
     // GR64all + its subclasses.
     assert(RBGPR.covers(*TRI.getRegClass(AArch64::GPR32RegClassID)) &&
            "Subclass not added?");
-    assert(RBGPR.getSize() == 64 && "GPRs should hold up to 64-bit");
+    assert(RBGPR.getSize() == 128 && "GPRs should hold up to 128-bit");
 
     // The FPR register bank is fully defined by all the registers in
     // GR64all + its subclasses.
     assert(RBFPR.covers(*TRI.getRegClass(AArch64::QQRegClassID)) &&
            "Subclass not added?");
     assert(RBFPR.covers(*TRI.getRegClass(AArch64::FPR64RegClassID)) &&
            "Subclass not added?");
     assert(RBFPR.getSize() == 512 &&
            "FPRs should hold up to 512-bit via QQQQ sequence");
 
     assert(RBCCR.covers(*TRI.getRegClass(AArch64::CCRRegClassID)) &&
            "Class not added?");
     assert(RBCCR.getSize() == 32 && "CCR should hold up to 32-bit");
 
     // Check that the TableGen'ed like file is in sync we our expectations.
     // First, the Idx.
     assert(checkPartialMappingIdx(PMI_FirstGPR, PMI_LastGPR,
-                                  {PMI_GPR32, PMI_GPR64}) &&
+                                  {PMI_GPR32, PMI_GPR64, PMI_GPR128}) &&
            "PartialMappingIdx's are incorrectly ordered");
     assert(checkPartialMappingIdx(PMI_FirstFPR, PMI_LastFPR,
                                   {PMI_FPR16, PMI_FPR32, PMI_FPR64, PMI_FPR128,
                                    PMI_FPR256, PMI_FPR512}) &&
            "PartialMappingIdx's are incorrectly ordered");
 // Now, the content.
 // Check partial mapping.
 #define CHECK_PARTIALMAP(Idx, ValStartIdx, ValLength, RB)                      \
   do {                                                                         \
     assert(                                                                    \
         checkPartialMap(PartialMappingIdx::Idx, ValStartIdx, ValLength, RB) && \
         #Idx " is incorrectly initialized");                                   \
   } while (false)
 
     CHECK_PARTIALMAP(PMI_GPR32, 0, 32, RBGPR);
     CHECK_PARTIALMAP(PMI_GPR64, 0, 64, RBGPR);
+    CHECK_PARTIALMAP(PMI_GPR128, 0, 128, RBGPR);
     CHECK_PARTIALMAP(PMI_FPR16, 0, 16, RBFPR);
     CHECK_PARTIALMAP(PMI_FPR32, 0, 32, RBFPR);
     CHECK_PARTIALMAP(PMI_FPR64, 0, 64, RBFPR);
     CHECK_PARTIALMAP(PMI_FPR128, 0, 128, RBFPR);
     CHECK_PARTIALMAP(PMI_FPR256, 0, 256, RBFPR);
     CHECK_PARTIALMAP(PMI_FPR512, 0, 512, RBFPR);
 
 // Check value mapping.
 #define CHECK_VALUEMAP_IMPL(RBName, Size, Offset)                              \
   do {                                                                         \
     assert(checkValueMapImpl(PartialMappingIdx::PMI_##RBName##Size,            \
                              PartialMappingIdx::PMI_First##RBName, Size,       \
                              Offset) &&                                        \
            #RBName #Size " " #Offset " is incorrectly initialized");           \
   } while (false)
 
 #define CHECK_VALUEMAP(RBName, Size) CHECK_VALUEMAP_IMPL(RBName, Size, 0)
 
     CHECK_VALUEMAP(GPR, 32);
     CHECK_VALUEMAP(GPR, 64);
+    CHECK_VALUEMAP(GPR, 128);
     CHECK_VALUEMAP(FPR, 16);
     CHECK_VALUEMAP(FPR, 32);
     CHECK_VALUEMAP(FPR, 64);
     CHECK_VALUEMAP(FPR, 128);
     CHECK_VALUEMAP(FPR, 256);
     CHECK_VALUEMAP(FPR, 512);
 
 // Check the value mapping for 3-operands instructions where all the operands
 // map to the same value mapping.
 #define CHECK_VALUEMAP_3OPS(RBName, Size)                                      \
   do {                                                                         \
     CHECK_VALUEMAP_IMPL(RBName, Size, 0);                                      \
     CHECK_VALUEMAP_IMPL(RBName, Size, 1);                                      \
     CHECK_VALUEMAP_IMPL(RBName, Size, 2);                                      \
   } while (false)
 
     CHECK_VALUEMAP_3OPS(GPR, 32);
     CHECK_VALUEMAP_3OPS(GPR, 64);
+    CHECK_VALUEMAP_3OPS(GPR, 128);
     CHECK_VALUEMAP_3OPS(FPR, 32);
     CHECK_VALUEMAP_3OPS(FPR, 64);
     CHECK_VALUEMAP_3OPS(FPR, 128);
     CHECK_VALUEMAP_3OPS(FPR, 256);
     CHECK_VALUEMAP_3OPS(FPR, 512);
 
 #define CHECK_VALUEMAP_CROSSREGCPY(RBNameDst, RBNameSrc, Size)                 \
   do {                                                                         \
if (getRegBank(MI.getOperand(2).getReg(), MRI, TRI) == &AArch64::FPRRegBank)
       OpRegBankIdx[2] = PMI_FirstFPR;
     else
       OpRegBankIdx[2] = PMI_FirstGPR;
 
     // Index needs to be a GPR.
     OpRegBankIdx[3] = PMI_FirstGPR;
     break;
   case TargetOpcode::G_EXTRACT: {
-    // For s128 sources we have to use fpr.
+    // For s128 sources we have to use fpr unless we know otherwise.
+    auto Src = MI.getOperand(1).getReg();
     LLT SrcTy = MRI.getType(MI.getOperand(1).getReg());
-    if (SrcTy.getSizeInBits() == 128) {
+    if (MRI.getRegClassOrNull(Src) == &AArch64::XSeqPairsClassRegClass) {
+      OpRegBankIdx[0] = PMI_FirstGPR;
+      OpRegBankIdx[1] = PMI_FirstGPR;
+    } else if (SrcTy.getSizeInBits() == 128) {
       OpRegBankIdx[0] = PMI_FirstFPR;
       OpRegBankIdx[1] = PMI_FirstFPR;
     }
     break;
   }
   case TargetOpcode::G_BUILD_VECTOR: {
     // If the first source operand belongs to a FPR register bank, then make
     // sure that we preserve that.

llvm/test/CodeGen/AArch64/GlobalISel/arm64-atomic-128.ll

+; RUN: llc < %s -mtriple=arm64-linux-gnu -verify-machineinstrs -global-isel -global-isel-abort=1 | FileCheck %s --check-prefix=CHECK-LLSC-O1
+; RUN: llc < %s -mtriple=arm64-linux-gnu -verify-machineinstrs -mcpu=apple-a13 -global-isel -global-isel-abort=1 | FileCheck %s --check-prefix=CHECK-CAS-O1
+; RUN: llc < %s -mtriple=arm64-linux-gnu -verify-machineinstrs -O0 -global-isel -global-isel-abort=1 | FileCheck %s --check-prefix=CHECK-LLSC-O0
+; RUN: llc < %s -mtriple=arm64-linux-gnu -verify-machineinstrs -O0 -mcpu=apple-a13 -global-isel -global-isel-abort=1 | FileCheck %s --check-prefix=CHECK-CAS-O0
+@var = global i128 0
+
+define void @val_compare_and_swap(i128* %p, i128 %oldval, i128 %newval) {
+; CHECK-LLSC-O1-LABEL: val_compare_and_swap:
+; CHECK-LLSC-O1:    ldaxp {{x[0-9]+}}, {{x[0-9]+}}, [x0]
+; [... LOTS of stuff that is generic IR unrelated to atomic operations ...]
+; CHECK-LLSC-O1:    stxp {{w[0-9]+}}, {{x[0-9]+}}, {{x[0-9]+}}, [x0]
+;
+; CHECK-CAS-O1-LABEL: val_compare_and_swap:
+; CHECK-CAS-O1:    caspa x2, x3, x4, x5, [x0]
+; CHECK-CAS-O1:    mov v[[OLD:[0-9]+]].d[0], x2
+; CHECK-CAS-O1:    mov v[[OLD]].d[1], x3
+; CHECK-CAS-O1:    str q[[OLD]], [x0]
+
+; CHECK-LLSC-O0-LABEL: val_compare_and_swap:
+; CHECK-LLSC-O0:  .LBB0_1:
+; CHECK-LLSC-O0:    ldaxp [[OLD_LO:x[0-9]+]], [[OLD_HI:x[0-9]+]], [x0]
+; CHECK-LLSC-O0:    cmp [[OLD_LO]], x2
+; CHECK-LLSC-O0:    cset [[EQUAL_TMP:w[0-9]+]], ne
+; CHECK-LLSC-O0:    cmp [[OLD_HI]], x3
+; CHECK-LLSC-O0:    cinc [[EQUAL:w[0-9]+]], [[EQUAL_TMP]], ne
+; CHECK-LLSC-O0:    cbnz [[EQUAL]], .LBB0_3
+; CHECK-LLSC-O0:    stlxp [[STATUS:w[0-9]+]], x4, x5, [x0]
+; CHECK-LLSC-O0:    cbnz [[STATUS]], .LBB0_1
+; CHECK-LLSC-O0:  .LBB0_3:
+; CHECK-LLSC-O0:    mov v[[OLD:[0-9]+]].d[0], [[OLD_LO]]
+; CHECK-LLSC-O0:    mov v[[OLD]].d[1], [[OLD_HI]]
+; CHECK-LLSC-O0:    str q[[OLD]], [x0]
+
+
+; CHECK-CAS-O0-LABEL: val_compare_and_swap:
+; CHECK-CAS-O0:    str x3, [sp, #[[SLOT:[0-9]+]]]
+; CHECK-CAS-O0:    mov [[NEW_HI_TMP:x[0-9]+]], x5
+; CHECK-CAS-O0:    ldr [[DESIRED_HI_TMP:x[0-9]+]], [sp, #[[SLOT]]]
+; CHECK-CAS-O0:    mov [[DESIRED_HI:x[0-9]+]], [[DESIRED_HI_TMP]]
+; CHECK-CAS-O0:    mov [[NEW_HI:x[0-9]+]], [[NEW_HI_TMP]]
+; CHECK-CAS-O0:    caspa x2, [[DESIRED_HI]], x4, [[NEW_HI]], [x0]
+; CHECK-CAS-O0:    mov [[OLD_LO:x[0-9]+]], x2
+; CHECK-CAS-O0:    mov [[OLD_HI:x[0-9]+]], x3
+; CHECK-CAS-O0:    mov v[[OLD:[0-9]+]].d[0], [[OLD_LO]]
+; CHECK-CAS-O0:    mov v[[OLD]].d[1], [[OLD_HI]]
+; CHECK-CAS-O0:    str q[[OLD]], [x0]
+
+%pair = cmpxchg i128* %p, i128 %oldval, i128 %newval acquire acquire
+  %val = extractvalue { i128, i1 } %pair, 0
+  store i128 %val, i128* %p
+  ret void
+}

llvm/test/CodeGen/AArch64/GlobalISel/legalizer-info-validation.mir

 # DEBUG-NEXT: .. imm index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: G_INDEXED_STORE (opcode {{[0-9]+}}): 3 type indices, 0 imm indices
 # DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: G_ATOMIC_CMPXCHG_WITH_SUCCESS (opcode {{[0-9]+}}): 3 type indices, 0 imm indices
 # DEBUG-NEXT: .. type index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: G_ATOMIC_CMPXCHG (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
-# DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. type index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: G_ATOMICRMW_XCHG (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. type index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: G_ATOMICRMW_ADD (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. type index coverage check SKIPPED: user-defined predicate detected