[PowerPC] Export 16 byte load-store instructions

[lkail]

Summary

Export lq, stq, lqarx and stqcx. in preparation for implementing 16-byte lock free atomic operations on AIX.
Add a new register class g8prc for these instructions, since these instructions require even-odd register pair.

[lkail]

Fixed failed pre-merge tests.

[nemanjai]

Although I am obviously not opposed to adding these instructions and the necessary register classes, we do have to devise a way to ensure RA doesn't allocate the same register to the base and result of LQ. An additional complication for this check is ensuring that if RA == 0, RTp != 0.

llvm/lib/Target/PowerPC/PPCRegisterInfo.cpp

118

This is likely problematic. What happens if we need to spill one of these at an offset from the stack pointer that doesn't fit in a DQ field? I imagine something breaks. These have to be handled differently in that we need to bump the base register rather than loading an immediate into a register and using it as an index.
See PPCRegisterInfo::eliminateFrameIndex()

llvm/lib/Target/PowerPC/PPCRegisterInfo.td

23

These names are odd, they seem to suggest that there are pairs that are subregisters, but it is actually the individual registers. Maybe:

sub_gp8_x0
sub_gp8_x1

[lkail]

we do have to devise a way to ensure RA doesn't allocate the same register to the base and result of LQ

One way I can figure out now is to have an additional pseudo instruction after LQ which uses the base register of LQ so that liveranges of base register and result register overlap.

[lkail]

we do have to devise a way to ensure RA doesn't allocate the same register to the base and result of LQ

Another approach for this might be adding an additional out $ea_result operand which can be considered pseudo for LQ. But there might be redundant copy. For example,

%0, %1 = LQ 128, %2(tied-def 1)
use %2

[lkail]

1. Solve LQ register constraint RTp != RA by adding early-clobber on RTp
2. Use doubleword load/store to perform quadword restore/spill

LQ and STQ are still needed for quadword atomic load/store.

[nemanjai]

llvm/lib/Target/PowerPC/PPCInstr64Bit.td

1253

nit: misaligned - move right by one more space.

llvm/lib/Target/PowerPC/PPCRegisterInfo.cpp

1155

Why use an X-Form for spilling/restoring rather than a D-Form the way all the other spilling code does? The FI eliminator should have what is needed to convert the D-Form to an X-Form (rather than relying on the pre-emit peephole which is what I assume is converting the X-Form to a D-Form in the attached test case).

[lkail]

1. Use DForm to keep consistency
2. Format fix

[lkail]

Add encoding tests.

[jsji]

llvm/lib/Target/PowerPC/AsmParser/PPCAsmParser.cpp

297

This is wrong and not needed , should be removed -- there is NO instruction using Regnum/2 as reg index.

445

isEvenRegNumber is better, this is not specific to G8p, just a plain even regnum check.

446

This is wrong.
We can have
lq 30, 128(4)

Should be something like:

isRegNumber() && ((getImm() & 1) == 0)

llvm/lib/Target/PowerPC/PPCInstr64Bit.td

286

isRecordForm ? lqarx is not a recordform, stdcx. is a record form.

llvm/lib/Target/PowerPC/PPCRegisterInfo.td

199

This looks wrong. The encoding should be using original regnum, as in getG8pReg.

eg: G8p1 is actually {X2,X3}, so the HW encoding should be 2/3, not 1.

201

Shouldn't define Index here, should be <[-1, -1]> -- G8p is reg number in llvm only, there is NO regnum defined visible to debugger.

Defining here will mess up mapping table.

485

Add comments about why the allocation order is defined like this? And why we need a AltOrders for ELF?

llvm/lib/Target/PowerPC/PPCSchedule.td

46

nit: these are in alphabetic order , so these two should be moved down to after IIC_LdStLMW

llvm/test/CodeGen/PowerPC/ldst-16-byte.mir

3

How about powerpc32?

[lkail]

llvm/test/CodeGen/PowerPC/ldst-16-byte.mir

3

I guess it's not supported in powerpc32 mode, though I can't find where it is documented(It's documented in AIX assembly manual that ldarx and stdcx are not supported in powerpc32 mode). I've tried following on AIX-7.2

main:
  lq 2,128(4)
  stq 2,128(4)
  lqarx 2,3,4
  lqarx 2,3,4,1
  stqcx. 2,3,4

# as above code
as -a64 -mpwr8 enc.s
objdump -D a.out

It gives correct dump result.

0000000000000000 <.text>:
   0:   e0 44 00 80     lq      r2,128(r4)
   4:   f8 44 00 82     stq     r2,128(r4)
   8:   7c 43 22 28     lqarx   r2,r3,r4
   c:   7c 43 22 29     lqarx   r2,r3,r4,1
  10:   7c 43 21 6d     stqcx.  r2,r3,r4

But for as -a32 -mpwr8 enc.s, the result is not correct.

00000000 <.text>:
   0:   e0 44 00 80     lfq     f2,128(r4)
   4:   f8 44 00 82     .long 0xf8440082
   8:   7c 43 22 28     .long 0x7c432228
   c:   7c 43 22 29     .long 0x7c432229
  10:   7c 43 21 6d     .long 0x7c43216d

[jsji]

llvm/test/CodeGen/PowerPC/ldst-16-byte.mir

3

Thanks. If so, there opcode should be guarded with isPPC64 pred.

[lkail]

llvm/lib/Target/PowerPC/PPCInstr64Bit.td

286

This should be a historical issue(IIUC, to avoid decoding conflict), I'll leave L(B|W|D|Q)ARXL as it is now and fix them in following patches.

[lkail]

Address comments.

[nemanjai]

llvm/lib/Target/PowerPC/PPCInstr64Bit.td

286

The L variant is a hinted variant and the hint sets the same bit as record-form instructions. So we just reuse that bit. There is no danger of PPCInstrInfo::optimizeCompareInstr() attempting to convert one to the other since this is not part of the RecFormRel relation.

So I don't really think there is anything here that needs fixing.

[nemanjai]

LGTM.

[jsji]

LGTM with some nits.

llvm/lib/Target/PowerPC/PPCInstr64Bit.td

286

Thanks for the background, yes, it is functionality correct here, but I think it is misleading and causing confusion about the semantic of isRecordForm.
We should be able to achieve same goal about the encoding/decoding here without messing up with isRecordForm.
But yeah, this can be done in a follow up patch.

1252

memrix16 ? We are restoring using two ld, do we still require DQ offset?

llvm/lib/Target/PowerPC/PPCRegisterInfo.td

125

nit: This doesn't enforce the relationship between SubReg0 and SubReg1 in Pair.

How about something like?

// GP8Pair - Paired GP8.
class GP8Pair<string n, bits<5> EvenIndex>: PPCReg<n>{
  assert !eq(EvenIndex{0},0), "Index should be even";
  let HWEncoding{4-0} = EvenIndex ;
  let SubRegs = [!cast<GP8>("X"#EvenIndex), !cast<GP8>("X"#!add(EvenIndex, 1))];
  let DwarfNumbers = [-1,-1] ;
  let SubRegIndices = [sub_gp8_x0, sub_gp8_x1] ;
}

then we can define pair safely and simply like:

// 16 paired even-odd consecutive GP8s.
foreach Index = { 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30 } in {
    def G8p#!srl(Index, 1) : GP8Pair<"r"#Index, Index>;
}

[lkail]

llvm/lib/Target/PowerPC/PPCRegisterInfo.td

125

Nice one.

llvm/lib/Target/PowerPC/AsmParser/PPCAsmParser.cpp

unsigned getACCReg() const {
     return (unsigned) Imm.Val;
   }
 
   unsigned getVSRpEvenReg() const {
     assert(isVSRpEvenRegNumber() && "Invalid access!");
     return (unsigned) Imm.Val >> 1;
   }
 
+  unsigned getG8pEvenReg() const {
+    assert(isG8pEvenRegNumber() && "Invalid access!");
+    return (unsigned)Imm.Val >> 1;
+  }
+
+  unsigned getG8pReg() const {
+    assert(isG8pEvenRegNumber() && "Invalid access!");
+    return (unsigned)Imm.Val;
+  }
+
   unsigned getCCReg() const {
     assert(isCCRegNumber() && "Invalid access!");
     return (unsigned) (Kind == Immediate ? Imm.Val : Expr.CRVal);
   }
 
   unsigned getCRBit() const {
     assert(isCRBitNumber() && "Invalid access!");
     return (unsigned) (Kind == Immediate ? Imm.Val : Expr.CRVal);
public:
   bool isCCRegNumber() const { return (Kind == Expression
                                        && isUInt<3>(getExprCRVal())) ||
                                       (Kind == Immediate
                                        && isUInt<3>(getImm())); }
   bool isCRBitNumber() const { return (Kind == Expression
                                        && isUInt<5>(getExprCRVal())) ||
                                       (Kind == Immediate
                                        && isUInt<5>(getImm())); }
+  bool isG8pEvenRegNumber() const {
+    return Kind == Immediate && isUInt<4>(getImm()) && ((getImm() & 1) == 0);
+  }
+
   bool isCRBitMask() const { return Kind == Immediate && isUInt<8>(getImm()) &&
                                     isPowerOf2_32(getImm()); }
   bool isATBitsAsHint() const { return false; }
   bool isMem() const override { return false; }
   bool isReg() const override { return false; }
 
   void addRegOperands(MCInst &Inst, unsigned N) const {
     llvm_unreachable("addRegOperands");
void addRegG8RCOperands(MCInst &Inst, unsigned N) const {
     Inst.addOperand(MCOperand::createReg(XRegs[getReg()]));
   }
 
   void addRegG8RCNoX0Operands(MCInst &Inst, unsigned N) const {
     assert(N == 1 && "Invalid number of operands!");
     Inst.addOperand(MCOperand::createReg(XRegsNoX0[getReg()]));
   }
 
+  void addRegG8pRCOperands(MCInst &Inst, unsigned N) const {
+    assert(N == 1 && "Invalid number of operands!");
+    Inst.addOperand(MCOperand::createReg(G8pRegs[getG8pEvenReg()]));
+  }
+
   void addRegGxRCOperands(MCInst &Inst, unsigned N) const {
     if (isPPC64())
       addRegG8RCOperands(Inst, N);
     else
       addRegGPRCOperands(Inst, N);
   }
 
   void addRegGxRCNoR0Operands(MCInst &Inst, unsigned N) const {

llvm/lib/Target/PowerPC/Disassembler/PPCDisassembler.cpp

 }
 
 static DecodeStatus DecodeG8RCRegisterClass(MCInst &Inst, uint64_t RegNo,
                                             uint64_t Address,
                                             const void *Decoder) {
   return decodeRegisterClass(Inst, RegNo, XRegs);
 }
 
+static DecodeStatus DecodeG8pRCRegisterClass(MCInst &Inst, uint64_t RegNo,
+                                             uint64_t Address,
+                                             const void *Decoder) {
+  return decodeRegisterClass(Inst, RegNo, G8pRegs);
+}
+
 static DecodeStatus DecodeG8RC_NOX0RegisterClass(MCInst &Inst, uint64_t RegNo,
                                             uint64_t Address,
                                             const void *Decoder) {
   return decodeRegisterClass(Inst, RegNo, XRegsNoX0);
 }
 
 #define DecodePointerLikeRegClass0 DecodeGPRCRegisterClass
 #define DecodePointerLikeRegClass1 DecodeGPRC_NOR0RegisterClass

llvm/lib/Target/PowerPC/MCTargetDesc/PPCMCTargetDesc.h

 #define GET_SUBTARGETINFO_ENUM
 #include "PPCGenSubtargetInfo.inc"
 
 #define PPC_REGS0_7(X)                                                         \
   {                                                                            \
     X##0, X##1, X##2, X##3, X##4, X##5, X##6, X##7                             \
   }
 
+#define PPC_REGS0_15(X)                                                        \
+  {                                                                            \
+    X##0, X##1, X##2, X##3, X##4, X##5, X##6, X##7, X##8, X##9, X##10, X##11,  \
+        X##12, X##13, X##14, X##15                                             \
+  }
+
 #define PPC_REGS0_31(X)                                                        \
   {                                                                            \
     X##0, X##1, X##2, X##3, X##4, X##5, X##6, X##7, X##8, X##9, X##10, X##11,  \
         X##12, X##13, X##14, X##15, X##16, X##17, X##18, X##19, X##20, X##21,  \
         X##22, X##23, X##24, X##25, X##26, X##27, X##28, X##29, X##30, X##31   \
   }
 
 #define PPC_REGS_NO0_31(Z, X)                                                  \
LO##0, LO##1, LO##2, LO##3, LO##4, LO##5, LO##6, LO##7, LO##8, LO##9,      \
         HI##3, HI##4, HI##5, HI##6, HI##7, HI##8, HI##9, HI##10, HI##11,       \
         HI##12, HI##13, HI##14, HI##15, HI##16, HI##17, HI##18, HI##19,        \
         HI##20, HI##21, HI##22, HI##23, HI##24, HI##25, HI##26, HI##27,        \
         HI##28, HI##29, HI##30, HI##31                                         \
   }
 
 using llvm::MCPhysReg;
 
 #define DEFINE_PPC_REGCLASSES \
   static const MCPhysReg RRegs[32] = PPC_REGS0_31(PPC::R); \
   static const MCPhysReg XRegs[32] = PPC_REGS0_31(PPC::X); \
+  static const MCPhysReg G8pRegs[16] = PPC_REGS0_15(PPC::G8p); \
   static const MCPhysReg FRegs[32] = PPC_REGS0_31(PPC::F); \
   static const MCPhysReg VSRpRegs[32] = PPC_REGS0_31(PPC::VSRp); \
   static const MCPhysReg SPERegs[32] = PPC_REGS0_31(PPC::S); \
   static const MCPhysReg VFRegs[32] = PPC_REGS0_31(PPC::VF); \
   static const MCPhysReg VRegs[32] = PPC_REGS0_31(PPC::V); \
   static const MCPhysReg RRegsNoR0[32] = \
     PPC_REGS_NO0_31(PPC::ZERO, PPC::R); \
   static const MCPhysReg XRegsNoX0[32] = \

llvm/lib/Target/PowerPC/PPCInstr64Bit.td

def ATOMIC_SWAP_I64 : PPCCustomInserterPseudo<
     (outs g8rc:$dst), (ins memrr:$ptr, g8rc:$new), "#ATOMIC_SWAP_I64",
     [(set i64:$dst, (atomic_swap_64 ForceXForm:$ptr, i64:$new))]>;
 }
 
 // Instructions to support atomic operations
 let mayLoad = 1, hasSideEffects = 0 in {
 def LDARX : XForm_1_memOp<31,  84, (outs g8rc:$rD), (ins memrr:$ptr),
                           "ldarx $rD, $ptr", IIC_LdStLDARX, []>;
+// TODO: Add scheduling info.
+let hasNoSchedulingInfo = 1 in
+def LQARX : XForm_1_memOp<31, 276, (outs g8prc:$RTp), (ins memrr:$ptr),
+                          "lqarx $RTp, $ptr", IIC_LdStLQARX, []>;
 
 // Instruction to support lock versions of atomics
 // (EH=1 - see Power ISA 2.07 Book II 4.4.2)
 def LDARXL : XForm_1<31,  84, (outs g8rc:$rD), (ins memrr:$ptr),
                      "ldarx $rD, $ptr, 1", IIC_LdStLDARX, []>, isRecordForm;
 
 let hasExtraDefRegAllocReq = 1 in
 def LDAT : X_RD5_RS5_IM5<31, 614, (outs g8rc:$rD), (ins g8rc:$rA, u5imm:$FC),
                          "ldat $rD, $rA, $FC", IIC_LdStLoad>, isPPC64,
            Requires<[IsISA3_0]>;
 }
 
-let Defs = [CR0], mayStore = 1, mayLoad = 0, hasSideEffects = 0 in
+let Defs = [CR0], mayStore = 1, mayLoad = 0, hasSideEffects = 0 in {
 def STDCX : XForm_1_memOp<31, 214, (outs), (ins g8rc:$rS, memrr:$dst),
                           "stdcx. $rS, $dst", IIC_LdStSTDCX, []>, isRecordForm;
+// TODO: Add scheduling info.
+let hasNoSchedulingInfo = 1 in
+def STQCX : XForm_1_memOp<31, 182, (outs), (ins g8prc:$RSp, memrr:$dst),
+                          "stqcx. $RSp, $dst", IIC_LdStSTQCX, []>, isRecordForm;
+}
 
 let mayStore = 1, mayLoad = 0, hasSideEffects = 0 in
 def STDAT : X_RD5_RS5_IM5<31, 742, (outs), (ins g8rc:$rS, g8rc:$rA, u5imm:$FC),
                           "stdat $rS, $rA, $FC", IIC_LdStStore>, isPPC64,
             Requires<[IsISA3_0]>;
 
 let Interpretation64Bit = 1, isCodeGenOnly = 1 in {
 let isCall = 1, isTerminator = 1, isReturn = 1, isBarrier = 1, Uses = [RM] in
def LDUX : XForm_1_memOp<31, 53, (outs g8rc:$rD, ptr_rc_nor0:$ea_result),
                         "ldux $rD, $addr", IIC_LdStLDUX,
                         []>, RegConstraint<"$addr.ptrreg = $ea_result">,
                         NoEncode<"$ea_result">, isPPC64;
 
 def LDMX : XForm_1<31, 309, (outs g8rc:$rD), (ins memrr:$src),
                    "ldmx $rD, $src", IIC_LdStLD, []>, isPPC64,
                    Requires<[IsISA3_0]>;
 }
+
+// Full 16-byte load.
+// TODO: Add scheduling info.
+let mayLoad = 1, hasNoSchedulingInfo = 1 in
+def LQ   : DQForm_RTp5_RA17_MEM<56, 0, (outs g8prc:$RTp), (ins memrix16:$src),
+                                "lq $RTp, $src", IIC_LdStLQ,
+                                [(set i128:$RTp, (load DQForm:$src))]>, isPPC64;
 }
 
 // Support for medium and large code model.
 let hasSideEffects = 0 in {
 let isReMaterializable = 1 in {
 def ADDIStocHA8: PPCEmitTimePseudo<(outs g8rc:$rD), (ins g8rc_nox0:$reg, tocentry:$disp),
                        "#ADDIStocHA8", []>, isPPC64;
 def ADDItocL: PPCEmitTimePseudo<(outs g8rc:$rD), (ins g8rc_nox0:$reg, tocentry:$disp),
                      "#ADDItocL", []>, isPPC64;
 }
 let mayLoad = 1 in
 def LDtocL: PPCEmitTimePseudo<(outs g8rc:$rD), (ins tocentry:$disp, g8rc_nox0:$reg),
                    "#LDtocL", []>, isPPC64;
 }
 

 def STDX  : XForm_8_memOp<31, 149, (outs), (ins g8rc:$rS, memrr:$dst),
                           "stdx $rS, $dst", IIC_LdStSTD,
                           [(store i64:$rS, XForm:$dst)]>, isPPC64,
                           PPC970_DGroup_Cracked;
 def STDBRX: XForm_8_memOp<31, 660, (outs), (ins g8rc:$rS, memrr:$dst),
                           "stdbrx $rS, $dst", IIC_LdStStore,
                           [(PPCstbrx i64:$rS, ForceXForm:$dst, i64)]>, isPPC64,
                           PPC970_DGroup_Cracked;
+// Normal 16-byte stores.
+// TODO: Add scheduling info.
+let mayStore = 1, hasNoSchedulingInfo = 1 in
+def STQ : DSForm_1<62, 2, (outs), (ins g8prc:$RSp, memrix:$dst),
+                   "stq $RSp, $dst", IIC_LdStSTQ,
+                   [(store i128:$RSp, DSForm:$dst)]>, isPPC64;
 }
 
 // Stores with Update (pre-inc).
 let PPC970_Unit = 2, mayStore = 1, mayLoad = 0 in {
 let Interpretation64Bit = 1, isCodeGenOnly = 1 in {
 def STBU8 : DForm_1<39, (outs ptr_rc_nor0:$ea_res), (ins g8rc:$rS, memri:$dst),
                    "stbu $rS, $dst", IIC_LdStSTU, []>,
                    RegConstraint<"$dst.reg = $ea_res">, NoEncode<"$ea_res">;

llvm/lib/Target/PowerPC/PPCInstrFormats.td

class DQ_RD6_RS5_DQ12<bits<6> opcode, bits<3> xo, dag OOL, dag IOL,
 
   let Inst{6-10}  = XT{4-0};
   let Inst{11-15} = DS_RA{16-12};  // Register #
   let Inst{16-27} = DS_RA{11-0};   // Displacement.
   let Inst{28}    = XT{5};
   let Inst{29-31} = xo;
 }
 
+class DQForm_RTp5_RA17_MEM<bits<6> opcode, bits<4> xo, dag OOL, dag IOL,
+                           string asmstr, InstrItinClass itin, list<dag> pattern>
+  : I<opcode, OOL, IOL, asmstr, itin> {
+  bits<5> RTp;
+  bits<17> DQ_RA;
+  let Pattern = pattern;
+
+  let Inst{6-10} =  RTp{4-0};
+  let Inst{11-15} = DQ_RA{16-12};  // Register #
+  let Inst{16-27} = DQ_RA{11-0};   // Displacement.
+  let Inst{28-31} = xo;
+}
+
 // 1.7.6 X-Form
 class XForm_base_r3xo<bits<6> opcode, bits<10> xo, dag OOL, dag IOL, string asmstr, 
                       InstrItinClass itin, list<dag> pattern>
   : I<opcode, OOL, IOL, asmstr, itin> {
   bits<5> RST;
   bits<5> A;
   bits<5> B;
 

llvm/lib/Target/PowerPC/PPCInstrInfo.h

enum SpillOpcodeKey {
   SOK_VSXVectorSpill,
   SOK_VectorFloat8Spill,
   SOK_VectorFloat4Spill,
   SOK_SpillToVSR,
   SOK_PairedVecSpill,
   SOK_AccumulatorSpill,
   SOK_UAccumulatorSpill,
   SOK_SPESpill,
+  SOK_PairedG8Spill,
   SOK_LastOpcodeSpill // This must be last on the enum.
 };
 
 // Define list of load and store spill opcodes.
 #define NoInstr PPC::INSTRUCTION_LIST_END
 #define Pwr8LoadOpcodes                                                        \
   {                                                                            \
     PPC::LWZ, PPC::LD, PPC::LFD, PPC::LFS, PPC::RESTORE_CR,                    \
         PPC::RESTORE_CRBIT, PPC::LVX, PPC::LXVD2X, PPC::LXSDX, PPC::LXSSPX,    \
-        PPC::SPILLTOVSR_LD, NoInstr, NoInstr, NoInstr, PPC::EVLDD              \
+        PPC::SPILLTOVSR_LD, NoInstr, NoInstr, NoInstr, PPC::EVLDD, PPC::LQ     \
   }
 
 #define Pwr9LoadOpcodes                                                        \
   {                                                                            \
     PPC::LWZ, PPC::LD, PPC::LFD, PPC::LFS, PPC::RESTORE_CR,                    \
         PPC::RESTORE_CRBIT, PPC::LVX, PPC::LXV, PPC::DFLOADf64,                \
-        PPC::DFLOADf32, PPC::SPILLTOVSR_LD, NoInstr, NoInstr, NoInstr, NoInstr \
+        PPC::DFLOADf32, PPC::SPILLTOVSR_LD, NoInstr, NoInstr, NoInstr,         \
+        NoInstr, PPC::LQ                                                       \
   }
 
 #define Pwr10LoadOpcodes                                                       \
   {                                                                            \
     PPC::LWZ, PPC::LD, PPC::LFD, PPC::LFS, PPC::RESTORE_CR,                    \
         PPC::RESTORE_CRBIT, PPC::LVX, PPC::LXV, PPC::DFLOADf64,                \
         PPC::DFLOADf32, PPC::SPILLTOVSR_LD, PPC::LXVP, PPC::RESTORE_ACC,       \
-        PPC::RESTORE_UACC, NoInstr                                             \
+        PPC::RESTORE_UACC, NoInstr, PPC::LQ                                    \
   }
 
 #define Pwr8StoreOpcodes                                                       \
   {                                                                            \
     PPC::STW, PPC::STD, PPC::STFD, PPC::STFS, PPC::SPILL_CR, PPC::SPILL_CRBIT, \
         PPC::STVX, PPC::STXVD2X, PPC::STXSDX, PPC::STXSSPX,                    \
-        PPC::SPILLTOVSR_ST, NoInstr, NoInstr, NoInstr, PPC::EVSTDD             \
+        PPC::SPILLTOVSR_ST, NoInstr, NoInstr, NoInstr, PPC::EVSTDD, PPC::STQ   \
   }
 
 #define Pwr9StoreOpcodes                                                       \
   {                                                                            \
     PPC::STW, PPC::STD, PPC::STFD, PPC::STFS, PPC::SPILL_CR, PPC::SPILL_CRBIT, \
         PPC::STVX, PPC::STXV, PPC::DFSTOREf64, PPC::DFSTOREf32,                \
-        PPC::SPILLTOVSR_ST, NoInstr, NoInstr, NoInstr, NoInstr                 \
+        PPC::SPILLTOVSR_ST, NoInstr, NoInstr, NoInstr, NoInstr, PPC::STQ       \
   }
 
 #define Pwr10StoreOpcodes                                                      \
   {                                                                            \
     PPC::STW, PPC::STD, PPC::STFD, PPC::STFS, PPC::SPILL_CR, PPC::SPILL_CRBIT, \
         PPC::STVX, PPC::STXV, PPC::DFSTOREf64, PPC::DFSTOREf32,                \
         PPC::SPILLTOVSR_ST, PPC::STXVP, PPC::SPILL_ACC, PPC::SPILL_UACC,       \
-        NoInstr                                                                \
+        NoInstr, PPC::STQ                                                      \
   }
 
 // Initialize arrays for load and store spill opcodes on supported subtargets.
 #define StoreOpcodesForSpill                                                   \
   { Pwr8StoreOpcodes, Pwr9StoreOpcodes, Pwr10StoreOpcodes }
 #define LoadOpcodesForSpill                                                    \
   { Pwr8LoadOpcodes, Pwr9LoadOpcodes, Pwr10LoadOpcodes }
 

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

for (unsigned Idx = 0; Idx < 4; Idx++)
       BuildMI(MBB, I, DL, get(PPC::XXLOR), VSLDestReg + Idx)
           .addReg(VSLSrcReg + Idx)
           .addReg(VSLSrcReg + Idx, getKillRegState(KillSrc));
     if (DestPrimed)
       BuildMI(MBB, I, DL, get(PPC::XXMTACC), DestReg).addReg(DestReg);
     if (SrcPrimed && !KillSrc)
       BuildMI(MBB, I, DL, get(PPC::XXMTACC), SrcReg).addReg(SrcReg);
     return;
+  } else if (PPC::G8pRCRegClass.contains(DestReg) &&
+             PPC::G8pRCRegClass.contains(SrcReg)) {
+    // TODO: Handle G8RC to G8pRC (and vice versa) copy.
+    unsigned DestRegIdx = DestReg - PPC::G8p0;
+    MCRegister DestRegSub0 = PPC::X0 + 2 * DestRegIdx;
+    MCRegister DestRegSub1 = PPC::X0 + 2 * DestRegIdx + 1;
+    unsigned SrcRegIdx = SrcReg - PPC::G8p0;
+    MCRegister SrcRegSub0 = PPC::X0 + 2 * SrcRegIdx;
+    MCRegister SrcRegSub1 = PPC::X0 + 2 * SrcRegIdx + 1;
+    BuildMI(MBB, I, DL, get(PPC::OR8), DestRegSub0)
+        .addReg(SrcRegSub0)
+        .addReg(SrcRegSub0, getKillRegState(KillSrc));
+    BuildMI(MBB, I, DL, get(PPC::OR8), DestRegSub1)
+        .addReg(SrcRegSub1)
+        .addReg(SrcRegSub1, getKillRegState(KillSrc));
+    return;
   } else
     llvm_unreachable("Impossible reg-to-reg copy");
 
   const MCInstrDesc &MCID = get(Opc);
   if (MCID.getNumOperands() == 3)
     BuildMI(MBB, I, DL, MCID, DestReg)
       .addReg(SrcReg).addReg(SrcReg, getKillRegState(KillSrc));
   else
unsigned PPCInstrInfo::getSpillIndex(const TargetRegisterClass *RC) const {
   } else if (PPC::UACCRCRegClass.hasSubClassEq(RC)) {
     assert(Subtarget.pairedVectorMemops() &&
            "Register unexpected when paired memops are disabled.");
     OpcodeIndex = SOK_UAccumulatorSpill;
   } else if (PPC::VSRpRCRegClass.hasSubClassEq(RC)) {
     assert(Subtarget.pairedVectorMemops() &&
            "Register unexpected when paired memops are disabled.");
     OpcodeIndex = SOK_PairedVecSpill;
+  } else if (PPC::G8pRCRegClass.hasSubClassEq(RC)) {
+    OpcodeIndex = SOK_PairedG8Spill;
   } else {
     llvm_unreachable("Unknown regclass!");
   }
   return OpcodeIndex;
 }
 
 unsigned
 PPCInstrInfo::getStoreOpcodeForSpill(const TargetRegisterClass *RC) const {

llvm/lib/Target/PowerPC/PPCInstrInfo.td

def gprc : RegisterOperand<GPRC> {
   let ParserMatchClass = PPCRegGPRCAsmOperand;
 }
 def PPCRegG8RCAsmOperand : AsmOperandClass {
   let Name = "RegG8RC"; let PredicateMethod = "isRegNumber";
 }
 def g8rc : RegisterOperand<G8RC> {
   let ParserMatchClass = PPCRegG8RCAsmOperand;
 }
+def PPCRegG8pRCAsmOperand : AsmOperandClass {
+  let Name = "RegG8pRC"; let PredicateMethod = "isG8pEvenRegNumber";
+}
+def g8prc : RegisterOperand<G8pRC> {
+  let ParserMatchClass = PPCRegG8pRCAsmOperand;
+}
 def PPCRegGPRCNoR0AsmOperand : AsmOperandClass {
   let Name = "RegGPRCNoR0"; let PredicateMethod = "isRegNumber";
 }
 def gprc_nor0 : RegisterOperand<GPRC_NOR0> {
   let ParserMatchClass = PPCRegGPRCNoR0AsmOperand;
 }
 def PPCRegG8RCNoX0AsmOperand : AsmOperandClass {
   let Name = "RegG8RCNoX0"; let PredicateMethod = "isRegNumber";

llvm/lib/Target/PowerPC/PPCRegisterInfo.cpp

PPCRegisterInfo::PPCRegisterInfo(const PPCTargetMachine &TM)
 
   // 64-bit
   ImmToIdxMap[PPC::LHA8] = PPC::LHAX8; ImmToIdxMap[PPC::LBZ8] = PPC::LBZX8;
   ImmToIdxMap[PPC::LHZ8] = PPC::LHZX8; ImmToIdxMap[PPC::LWZ8] = PPC::LWZX8;
   ImmToIdxMap[PPC::STB8] = PPC::STBX8; ImmToIdxMap[PPC::STH8] = PPC::STHX8;
   ImmToIdxMap[PPC::STW8] = PPC::STWX8; ImmToIdxMap[PPC::STDU] = PPC::STDUX;
   ImmToIdxMap[PPC::ADDI8] = PPC::ADD8;
 
+  // Quadword load and store.
+  ImmToIdxMap[PPC::LQ] = PPC::LQ;
+  ImmToIdxMap[PPC::STQ] = PPC::STQ;
+
   // VSX
   ImmToIdxMap[PPC::DFLOADf32] = PPC::LXSSPX;
   ImmToIdxMap[PPC::DFLOADf64] = PPC::LXSDX;
   ImmToIdxMap[PPC::SPILLTOVSR_LD] = PPC::SPILLTOVSR_LDX;
   ImmToIdxMap[PPC::SPILLTOVSR_ST] = PPC::SPILLTOVSR_STX;
   ImmToIdxMap[PPC::DFSTOREf32] = PPC::STXSSPX;
   ImmToIdxMap[PPC::DFSTOREf64] = PPC::STXSDX;
   ImmToIdxMap[PPC::LXV] = PPC::LXVX;
bool PPCRegisterInfo::hasReservedSpillSlot(const MachineFunction &MF,
   // For 32-bit ELF, we have previously created the stack slot if needed, so
   // return its FrameIdx.
   if (PPC::CR2 <= Reg && Reg <= PPC::CR4) {
     FrameIdx = MF.getInfo<PPCFunctionInfo>()->getCRSpillFrameIndex();
     return true;
   }
   return false;
 }
 
 // If the offset must be a multiple of some value, return what that value is.
 static unsigned offsetMinAlignForOpcode(unsigned OpC) {
   switch (OpC) {
   default:
     return 1;
   case PPC::LWA:
   case PPC::LWA_32:
   case PPC::LD:
static unsigned offsetMinAlignForOpcode(unsigned OpC) {
   case PPC::STXSD:
   case PPC::STXSSP:
     return 4;
   case PPC::EVLDD:
   case PPC::EVSTDD:
     return 8;
   case PPC::LXV:
   case PPC::STXV:
+  case PPC::STQ:
+  case PPC::LQ:
     return 16;
   }
 }
 
 // If the offset must be a multiple of some value, return what that value is.
 static unsigned offsetMinAlign(const MachineInstr &MI) {
   unsigned OpC = MI.getOpcode();
   return offsetMinAlignForOpcode(OpC);

llvm/lib/Target/PowerPC/PPCRegisterInfo.td

 def sub_eq : SubRegIndex<1, 2>;
 def sub_un : SubRegIndex<1, 3>;
 def sub_32 : SubRegIndex<32>;
 def sub_64 : SubRegIndex<64>;
 def sub_vsx0 : SubRegIndex<128>;
 def sub_vsx1 : SubRegIndex<128, 128>;
 def sub_pair0 : SubRegIndex<256>;
 def sub_pair1 : SubRegIndex<256, 256>;
+def sub_gp8_pair0 : SubRegIndex<64>;
+def sub_gp8_pair1 : SubRegIndex<64, 64>;
 }
 
 
 class PPCReg<string n> : Register<n> {
   let Namespace = "PPC";
 }
 
 // We identify all our registers with a 5-bit ID, for consistency's sake.

 }
 
 // VSR Pairs - One of the 32 paired even-odd consecutive VSRs.
 class VSRPair<bits<5> num, string n, list<Register> subregs> : PPCReg<n> {
   let HWEncoding{4-0} = num;
   let SubRegs = subregs;
 }
 
+// GP8Pair - One of the 16 paired even-odd consecutinve GP8s.
+class GP8Pair<bits<4> num, string n, list<Register> subregs> : PPCReg<n> {
+  let HWEncoding{3-0} = num;
+  let SubRegs = subregs;
+}
+
 // General-purpose registers
 foreach Index = 0-31 in {
   def R#Index : GPR<Index, "r"#Index>, DwarfRegNum<[-2, Index]>;
 }
 
 // 64-bit General-purpose registers
 foreach Index = 0-31 in {
   def X#Index : GP8<!cast<GPR>("R"#Index), "r"#Index>,
let SubRegIndices = [sub_vsx0, sub_vsx1] in {
   foreach Index = { 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30 } in {
     def VSRp#!add(!srl(Index, 1), 16) :
       VSRPair<!add(!srl(Index, 1), 16), "vsp"#!add(Index, 32),
               [!cast<VR>("V"#Index), !cast<VR>("V"#!add(Index, 1))]>,
       DwarfRegNum<[0, 0]>;
   }
 }
 
+let SubRegIndices = [sub_gp8_pair0, sub_gp8_pair1] in {
+  foreach Index = { 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30 } in {
+    def G8p#!srl(Index, 1) :
+      GP8Pair<!srl(Index, 1), "r"#Index,
+              [!cast<GP8>("X"#Index), !cast<GP8>("X"#!add(Index, 1))]>,
+      DwarfRegNum<[0, 0]>;
+  }
+}
+
 // The representation of r0 when treated as the constant 0.
 def ZERO  : GPR<0, "0">,    DwarfRegAlias<R0>;
 def ZERO8 : GP8<ZERO, "0">, DwarfRegAlias<X0>;
 
 // Representations of the frame pointer used by ISD::FRAMEADDR.
 def FP   : GPR<0 /* arbitrary */, "**FRAME POINTER**">;
 def FP8  : GP8<FP, "**FRAME POINTER**">;
 
def VSRpRC :
   RegisterClass<"PPC", [v256i1], 128,
                 (add (sequence "VSRp%u", 0, 6),
                      (sequence "VSRp%u", 15, 7), VSRp17, VSRp18,
                      VSRp16, VSRp19, VSRp20, VSRp21, VSRp22, VSRp23,
                      VSRp24, VSRp25, VSRp31, VSRp30, VSRp29, VSRp28,
                      VSRp27, VSRp26)> {
   let Size = 256;
 }
+
+def G8pRC :
+  RegisterClass<"PPC", [i128], 128,
+                (add (sequence "G8p%u", 1, 5),
+                     (sequence "G8p%u", 14, 7),
+                     G8p15, G8p6, G8p0)> {
+  let AltOrders = [(add (sub G8pRC, G8p1), G8p1)];
+  let AltOrderSelect = [{
+    return MF.getSubtarget<PPCSubtarget>().is64BitELFABI();
+  }];
+  let Size = 128;
+}

llvm/lib/Target/PowerPC/PPCSchedule.td

 def IIC_LdStDCBF     : InstrItinClass;
 def IIC_LdStDCBI     : InstrItinClass;
 def IIC_LdStLoad     : InstrItinClass;
 def IIC_LdStLoadUpd  : InstrItinClass;
 def IIC_LdStLoadUpdX : InstrItinClass;
 def IIC_LdStStore    : InstrItinClass;
 def IIC_LdStDSS      : InstrItinClass;
 def IIC_LdStICBI     : InstrItinClass;
+def IIC_LdStLQ       : InstrItinClass;
+def IIC_LdStLQARX    : InstrItinClass;
 def IIC_LdStLD       : InstrItinClass;
 def IIC_LdStLDU      : InstrItinClass;
 def IIC_LdStLDUX     : InstrItinClass;
 def IIC_LdStLDARX    : InstrItinClass;
 def IIC_LdStLFD      : InstrItinClass;
 def IIC_LdStLFDU     : InstrItinClass;
 def IIC_LdStLFDUX    : InstrItinClass;
 def IIC_LdStLHA      : InstrItinClass;
 def IIC_LdStLHAU     : InstrItinClass;
 def IIC_LdStLHAUX    : InstrItinClass;
 def IIC_LdStLMW      : InstrItinClass;
 def IIC_LdStLVecX    : InstrItinClass;
 def IIC_LdStLWA      : InstrItinClass;
 def IIC_LdStLWARX    : InstrItinClass;
 def IIC_LdStSLBIA    : InstrItinClass;
 def IIC_LdStSLBIE    : InstrItinClass;
+def IIC_LdStSTQ      : InstrItinClass;
+def IIC_LdStSTQCX    : InstrItinClass;
 def IIC_LdStSTD      : InstrItinClass;
 def IIC_LdStSTDCX    : InstrItinClass;
 def IIC_LdStSTU      : InstrItinClass;
 def IIC_LdStSTUX     : InstrItinClass;
 def IIC_LdStSTFD     : InstrItinClass;
 def IIC_LdStSTFDU    : InstrItinClass;
 def IIC_LdStSTVEBX   : InstrItinClass;
 def IIC_LdStSTWCX    : InstrItinClass;

llvm/test/CodeGen/PowerPC/ldst-16-byte.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -simplify-mir -verify-machineinstrs -mtriple=powerpc64-ibm-aix-xcoff \
+# RUN:   -stop-after=postrapseudos %s -o - | FileCheck %s
+
+---
+name: foobar
+alignment: 8
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+  liveins: $x3, $x4
+    ; CHECK-LABEL: name: foobar
+    ; CHECK: liveins: $x3, $x4
+    ; CHECK: renamable $g8p3 = LQARX $x3, $x4
+    ; CHECK: STQCX renamable $g8p3, $x3, $x4, implicit-def dead $cr0
+    ; CHECK: $x3 = OR8 $x7, $x7, implicit killed $g8p3
+    ; CHECK: BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+  %0:g8prc = LQARX $x3, $x4
+  STQCX %0:g8prc, $x3, $x4, implicit-def $cr0
+  $x3 = COPY %0.sub_gp8_pair1:g8prc
+  BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+...
+
+---
+name: bar
+alignment: 8
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+  liveins: $x3, $x4
+    ; CHECK-LABEL: name: bar
+    ; CHECK: liveins: $x3, $x4
+    ; CHECK: renamable $g8p2 = LQ 128, $x3
+    ; CHECK: STQ renamable $g8p2, 160, $x3
+    ; CHECK: $x3 = OR8 $x4, $x4, implicit killed $g8p2
+    ; CHECK: BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+  %0:g8prc = LQ 128, $x3
+  STQ %0:g8prc, 160, $x3
+  $x3 = COPY %0.sub_gp8_pair0:g8prc
+  BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+...
+
+---
+name: spill_g8prc
+alignment: 8
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+  liveins: $x3, $x4, $x5, $x6, $x7, $x8, $x9, $x10, $x11, $x12
+    ; CHECK-LABEL: name: spill_g8prc
+    ; CHECK: liveins: $x3, $x4, $x5, $x6, $x7, $x8, $x9, $x10, $x11, $x12, $x14, $x15, $x16, $x17, $x18, $x19, $x20, $x21, $x22, $x23, $x24, $x25, $x26, $x27, $x28, $x29, $x30, $x31, $x2
+    ; CHECK: STD killed $x14, -144, $x1 :: (store 8 into %fixed-stack.17, align 16)
+    ; CHECK: STD killed $x15, -136, $x1 :: (store 8 into %fixed-stack.16)
+    ; CHECK: STD killed $x16, -128, $x1 :: (store 8 into %fixed-stack.15, align 16)
+    ; CHECK: STD killed $x17, -120, $x1 :: (store 8 into %fixed-stack.14)
+    ; CHECK: STD killed $x18, -112, $x1 :: (store 8 into %fixed-stack.13, align 16)
+    ; CHECK: STD killed $x19, -104, $x1 :: (store 8 into %fixed-stack.12)
+    ; CHECK: STD killed $x20, -96, $x1 :: (store 8 into %fixed-stack.11, align 16)
+    ; CHECK: STD killed $x21, -88, $x1 :: (store 8 into %fixed-stack.10)
+    ; CHECK: STD killed $x22, -80, $x1 :: (store 8 into %fixed-stack.9, align 16)
+    ; CHECK: STD killed $x23, -72, $x1 :: (store 8 into %fixed-stack.8)
+    ; CHECK: STD killed $x24, -64, $x1 :: (store 8 into %fixed-stack.7, align 16)
+    ; CHECK: STD killed $x25, -56, $x1 :: (store 8 into %fixed-stack.6)
+    ; CHECK: STD killed $x26, -48, $x1 :: (store 8 into %fixed-stack.5, align 16)
+    ; CHECK: STD killed $x27, -40, $x1 :: (store 8 into %fixed-stack.4)
+    ; CHECK: STD killed $x28, -32, $x1 :: (store 8 into %fixed-stack.3, align 16)
+    ; CHECK: STD killed $x29, -24, $x1 :: (store 8 into %fixed-stack.2)
+    ; CHECK: STD killed $x30, -16, $x1 :: (store 8 into %fixed-stack.1, align 16)
+    ; CHECK: STD killed $x31, -8, $x1 :: (store 8 into %fixed-stack.0)
+    ; CHECK: STD killed $x2, -152, $x1 :: (store 8 into %stack.4)
+    ; CHECK: $x7 = OR8 $x3, $x3
+    ; CHECK: renamable $g8p4 = LQARX $x5, $x6
+    ; CHECK: STQ killed renamable $g8p4, -176, $x1 :: (store 16 into %stack.0)
+    ; CHECK: renamable $g8p1 = LQARX $x3, renamable $x4
+    ; CHECK: renamable $g8p4 = LQARX renamable $x7, renamable $x4
+    ; CHECK: STQ killed renamable $g8p4, -192, $x1 :: (store 16 into %stack.1)
+    ; CHECK: renamable $g8p4 = LQARX renamable $x7, renamable $x4
+    ; CHECK: STQ killed renamable $g8p4, -208, $x1 :: (store 16 into %stack.2)
+    ; CHECK: renamable $g8p4 = LQARX renamable $x7, renamable $x4
+    ; CHECK: STQ killed renamable $g8p4, -224, $x1 :: (store 16 into %stack.3)
+    ; CHECK: renamable $g8p12 = LQARX renamable $x7, renamable $x4
+    ; CHECK: renamable $g8p11 = LQARX renamable $x7, renamable $x4
+    ; CHECK: renamable $g8p10 = LQARX renamable $x7, renamable $x4
+    ; CHECK: renamable $g8p9 = LQARX renamable $x7, renamable $x4
+    ; CHECK: renamable $g8p8 = LQARX renamable $x7, renamable $x4
+    ; CHECK: renamable $g8p7 = LQARX renamable $x7, renamable $x4
+    ; CHECK: renamable $g8p15 = LQARX renamable $x7, renamable $x4
+    ; CHECK: renamable $g8p13 = LQARX renamable $x7, renamable $x4
+    ; CHECK: renamable $g8p14 = LQARX renamable $x7, renamable $x4
+    ; CHECK: renamable $g8p5 = LQARX renamable $x7, renamable $x4
+    ; CHECK: renamable $g8p4 = LQARX renamable $x7, renamable $x4
+    ; CHECK: STQCX killed renamable $g8p4, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p5, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p14, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p13, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p15, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p7, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p8, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p9, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p10, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p11, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p12, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: renamable $g8p4 = LQ -224, $x1 :: (load 16 from %stack.3)
+    ; CHECK: STQCX killed renamable $g8p4, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: renamable $g8p4 = LQ -208, $x1 :: (load 16 from %stack.2)
+    ; CHECK: STQCX killed renamable $g8p4, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: renamable $g8p4 = LQ -192, $x1 :: (load 16 from %stack.1)
+    ; CHECK: STQCX killed renamable $g8p4, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX renamable $g8p1, killed renamable $x7, killed renamable $x4, implicit-def dead $cr0
+    ; CHECK: renamable $g8p4 = LQ -176, $x1 :: (load 16 from %stack.0)
+    ; CHECK: STQCX killed renamable $g8p4, $x5, $x6, implicit-def dead $cr0
+    ; CHECK: $x2 = LD -152, $x1 :: (load 8 from %stack.4)
+    ; CHECK: $x31 = LD -8, $x1 :: (load 8 from %fixed-stack.0)
+    ; CHECK: $x30 = LD -16, $x1 :: (load 8 from %fixed-stack.1, align 16)
+    ; CHECK: $x29 = LD -24, $x1 :: (load 8 from %fixed-stack.2)
+    ; CHECK: $x28 = LD -32, $x1 :: (load 8 from %fixed-stack.3, align 16)
+    ; CHECK: $x27 = LD -40, $x1 :: (load 8 from %fixed-stack.4)
+    ; CHECK: $x26 = LD -48, $x1 :: (load 8 from %fixed-stack.5, align 16)
+    ; CHECK: $x25 = LD -56, $x1 :: (load 8 from %fixed-stack.6)
+    ; CHECK: $x24 = LD -64, $x1 :: (load 8 from %fixed-stack.7, align 16)
+    ; CHECK: $x23 = LD -72, $x1 :: (load 8 from %fixed-stack.8)
+    ; CHECK: $x22 = LD -80, $x1 :: (load 8 from %fixed-stack.9, align 16)
+    ; CHECK: $x21 = LD -88, $x1 :: (load 8 from %fixed-stack.10)
+    ; CHECK: $x20 = LD -96, $x1 :: (load 8 from %fixed-stack.11, align 16)
+    ; CHECK: $x19 = LD -104, $x1 :: (load 8 from %fixed-stack.12)
+    ; CHECK: $x18 = LD -112, $x1 :: (load 8 from %fixed-stack.13, align 16)
+    ; CHECK: $x17 = LD -120, $x1 :: (load 8 from %fixed-stack.14)
+    ; CHECK: $x16 = LD -128, $x1 :: (load 8 from %fixed-stack.15, align 16)
+    ; CHECK: $x15 = LD -136, $x1 :: (load 8 from %fixed-stack.16)
+    ; CHECK: $x14 = LD -144, $x1 :: (load 8 from %fixed-stack.17, align 16)
+    ; CHECK: BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+  %addr0:g8rc_nox0 = COPY $x3
+  %addr1:g8rc = COPY $x4
+  %0:g8prc = LQARX $x5, $x6
+  %1:g8prc = LQARX %addr0, %addr1
+  %2:g8prc = LQARX %addr0, %addr1
+  %3:g8prc = LQARX %addr0, %addr1
+  %4:g8prc = LQARX %addr0, %addr1
+  %5:g8prc = LQARX %addr0, %addr1
+  %6:g8prc = LQARX %addr0, %addr1
+  %7:g8prc = LQARX %addr0, %addr1
+  %8:g8prc = LQARX %addr0, %addr1
+  %9:g8prc = LQARX %addr0, %addr1
+  %10:g8prc = LQARX %addr0, %addr1
+  %11:g8prc = LQARX %addr0, %addr1
+  %12:g8prc = LQARX %addr0, %addr1
+  %13:g8prc = LQARX %addr0, %addr1
+  %14:g8prc = LQARX %addr0, %addr1
+  %15:g8prc = LQARX %addr0, %addr1
+  STQCX %15:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %14:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %13:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %12:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %11:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %10:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %9:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %8:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %7:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %6:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %5:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %4:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %3:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %2:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %1:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %0:g8prc, $x5, $x6, implicit-def $cr0
+  $x3 = COPY %1.sub_gp8_pair1:g8prc
+  BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+...
+
+---
+name: copy_g8prc
+alignment: 8
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+  liveins: $g8p8
+    ; CHECK-LABEL: name: copy_g8prc
+    ; CHECK: liveins: $g8p8, $x2
+    ; CHECK: STD killed $x2, -8, $x1 :: (store 8 into %stack.0)
+    ; CHECK: $x2 = OR8 $x16, $x16
+    ; CHECK: $x3 = OR8 $x17, $x17
+    ; CHECK: $x3 = KILL renamable $x3, implicit killed $g8p1
+    ; CHECK: $x2 = LD -8, $x1 :: (load 8 from %stack.0)
+    ; CHECK: BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+  %0:g8prc = COPY $g8p8
+  $x3 = COPY %0.sub_gp8_pair1:g8prc
+  BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+...