[PowerPC] Export 16 byte load-store instructions

[lkail]

Summary

Export lq, stq, lqarx and stqcx. in preparation for implementing 16-byte lock free atomic operations on AIX.
Add a new register class g8prc for these instructions, since these instructions require even-odd register pair.

[lkail]

Fixed failed pre-merge tests.

[nemanjai]

Although I am obviously not opposed to adding these instructions and the necessary register classes, we do have to devise a way to ensure RA doesn't allocate the same register to the base and result of LQ. An additional complication for this check is ensuring that if RA == 0, RTp != 0.

llvm/lib/Target/PowerPC/PPCRegisterInfo.cpp

118

This is likely problematic. What happens if we need to spill one of these at an offset from the stack pointer that doesn't fit in a DQ field? I imagine something breaks. These have to be handled differently in that we need to bump the base register rather than loading an immediate into a register and using it as an index.
See PPCRegisterInfo::eliminateFrameIndex()

llvm/lib/Target/PowerPC/PPCRegisterInfo.td

23

These names are odd, they seem to suggest that there are pairs that are subregisters, but it is actually the individual registers. Maybe:

sub_gp8_x0
sub_gp8_x1

[lkail]

we do have to devise a way to ensure RA doesn't allocate the same register to the base and result of LQ

One way I can figure out now is to have an additional pseudo instruction after LQ which uses the base register of LQ so that liveranges of base register and result register overlap.

[lkail]

we do have to devise a way to ensure RA doesn't allocate the same register to the base and result of LQ

Another approach for this might be adding an additional out $ea_result operand which can be considered pseudo for LQ. But there might be redundant copy. For example,

%0, %1 = LQ 128, %2(tied-def 1)
use %2

[lkail]

1. Solve LQ register constraint RTp != RA by adding early-clobber on RTp
2. Use doubleword load/store to perform quadword restore/spill

LQ and STQ are still needed for quadword atomic load/store.

[nemanjai]

llvm/lib/Target/PowerPC/PPCInstr64Bit.td

1257

nit: misaligned - move right by one more space.

llvm/lib/Target/PowerPC/PPCRegisterInfo.cpp

1151

Why use an X-Form for spilling/restoring rather than a D-Form the way all the other spilling code does? The FI eliminator should have what is needed to convert the D-Form to an X-Form (rather than relying on the pre-emit peephole which is what I assume is converting the X-Form to a D-Form in the attached test case).

[lkail]

1. Use DForm to keep consistency
2. Format fix

[lkail]

Add encoding tests.

[jsji]

llvm/lib/Target/PowerPC/AsmParser/PPCAsmParser.cpp

297

This is wrong and not needed , should be removed -- there is NO instruction using Regnum/2 as reg index.

445

isEvenRegNumber is better, this is not specific to G8p, just a plain even regnum check.

446

This is wrong.
We can have
lq 30, 128(4)

Should be something like:

isRegNumber() && ((getImm() & 1) == 0)

llvm/lib/Target/PowerPC/PPCInstr64Bit.td

286

isRecordForm ? lqarx is not a recordform, stdcx. is a record form.

llvm/lib/Target/PowerPC/PPCRegisterInfo.td

199

This looks wrong. The encoding should be using original regnum, as in getG8pReg.

eg: G8p1 is actually {X2,X3}, so the HW encoding should be 2/3, not 1.

201

Shouldn't define Index here, should be <[-1, -1]> -- G8p is reg number in llvm only, there is NO regnum defined visible to debugger.

Defining here will mess up mapping table.

485

Add comments about why the allocation order is defined like this? And why we need a AltOrders for ELF?

llvm/lib/Target/PowerPC/PPCSchedule.td

46

nit: these are in alphabetic order , so these two should be moved down to after IIC_LdStLMW

llvm/test/CodeGen/PowerPC/ldst-16-byte.mir

2

How about powerpc32?

[lkail]

llvm/test/CodeGen/PowerPC/ldst-16-byte.mir

2

I guess it's not supported in powerpc32 mode, though I can't find where it is documented(It's documented in AIX assembly manual that ldarx and stdcx are not supported in powerpc32 mode). I've tried following on AIX-7.2

main:
  lq 2,128(4)
  stq 2,128(4)
  lqarx 2,3,4
  lqarx 2,3,4,1
  stqcx. 2,3,4

# as above code
as -a64 -mpwr8 enc.s
objdump -D a.out

It gives correct dump result.

0000000000000000 <.text>:
   0:   e0 44 00 80     lq      r2,128(r4)
   4:   f8 44 00 82     stq     r2,128(r4)
   8:   7c 43 22 28     lqarx   r2,r3,r4
   c:   7c 43 22 29     lqarx   r2,r3,r4,1
  10:   7c 43 21 6d     stqcx.  r2,r3,r4

But for as -a32 -mpwr8 enc.s, the result is not correct.

00000000 <.text>:
   0:   e0 44 00 80     lfq     f2,128(r4)
   4:   f8 44 00 82     .long 0xf8440082
   8:   7c 43 22 28     .long 0x7c432228
   c:   7c 43 22 29     .long 0x7c432229
  10:   7c 43 21 6d     .long 0x7c43216d

[jsji]

llvm/test/CodeGen/PowerPC/ldst-16-byte.mir

2

Thanks. If so, there opcode should be guarded with isPPC64 pred.

[lkail]

llvm/lib/Target/PowerPC/PPCInstr64Bit.td

286

This should be a historical issue(IIUC, to avoid decoding conflict), I'll leave L(B|W|D|Q)ARXL as it is now and fix them in following patches.

[lkail]

Address comments.

[nemanjai]

llvm/lib/Target/PowerPC/PPCInstr64Bit.td

286

The L variant is a hinted variant and the hint sets the same bit as record-form instructions. So we just reuse that bit. There is no danger of PPCInstrInfo::optimizeCompareInstr() attempting to convert one to the other since this is not part of the RecFormRel relation.

So I don't really think there is anything here that needs fixing.

[nemanjai]

LGTM.

[jsji]

LGTM with some nits.

llvm/lib/Target/PowerPC/PPCInstr64Bit.td

286

Thanks for the background, yes, it is functionality correct here, but I think it is misleading and causing confusion about the semantic of isRecordForm.
We should be able to achieve same goal about the encoding/decoding here without messing up with isRecordForm.
But yeah, this can be done in a follow up patch.

1256

memrix16 ? We are restoring using two ld, do we still require DQ offset?

llvm/lib/Target/PowerPC/PPCRegisterInfo.td

125

nit: This doesn't enforce the relationship between SubReg0 and SubReg1 in Pair.

How about something like?

// GP8Pair - Paired GP8.
class GP8Pair<string n, bits<5> EvenIndex>: PPCReg<n>{
  assert !eq(EvenIndex{0},0), "Index should be even";
  let HWEncoding{4-0} = EvenIndex ;
  let SubRegs = [!cast<GP8>("X"#EvenIndex), !cast<GP8>("X"#!add(EvenIndex, 1))];
  let DwarfNumbers = [-1,-1] ;
  let SubRegIndices = [sub_gp8_x0, sub_gp8_x1] ;
}

then we can define pair safely and simply like:

// 16 paired even-odd consecutive GP8s.
foreach Index = { 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30 } in {
    def G8p#!srl(Index, 1) : GP8Pair<"r"#Index, Index>;
}

[lkail]

llvm/lib/Target/PowerPC/PPCRegisterInfo.td

125

Nice one.

llvm/lib/Target/PowerPC/AsmParser/PPCAsmParser.cpp

unsigned getACCReg() const {
     return (unsigned) Imm.Val;
   }
 
   unsigned getVSRpEvenReg() const {
     assert(isVSRpEvenRegNumber() && "Invalid access!");
     return (unsigned) Imm.Val >> 1;
   }
 
+  unsigned getG8pEvenReg() const {
+    assert(isG8pEvenRegNumber() && "Invalid access!");
+    return (unsigned)Imm.Val >> 1;
+  }
+
+  unsigned getG8pReg() const {
+    assert(isG8pEvenRegNumber() && "Invalid access!");
+    return (unsigned)Imm.Val;
+  }
+
   unsigned getCCReg() const {
     assert(isCCRegNumber() && "Invalid access!");
     return (unsigned) (Kind == Immediate ? Imm.Val : Expr.CRVal);
   }
 
   unsigned getCRBit() const {
     assert(isCRBitNumber() && "Invalid access!");
     return (unsigned) (Kind == Immediate ? Imm.Val : Expr.CRVal);
public:
   bool isCCRegNumber() const { return (Kind == Expression
                                        && isUInt<3>(getExprCRVal())) ||
                                       (Kind == Immediate
                                        && isUInt<3>(getImm())); }
   bool isCRBitNumber() const { return (Kind == Expression
                                        && isUInt<5>(getExprCRVal())) ||
                                       (Kind == Immediate
                                        && isUInt<5>(getImm())); }
+  bool isG8pEvenRegNumber() const {
+    return Kind == Immediate && isUInt<4>(getImm()) && ((getImm() & 1) == 0);
+  }
+
   bool isCRBitMask() const { return Kind == Immediate && isUInt<8>(getImm()) &&
                                     isPowerOf2_32(getImm()); }
   bool isATBitsAsHint() const { return false; }
   bool isMem() const override { return false; }
   bool isReg() const override { return false; }
 
   void addRegOperands(MCInst &Inst, unsigned N) const {
     llvm_unreachable("addRegOperands");
void addRegG8RCOperands(MCInst &Inst, unsigned N) const {
     Inst.addOperand(MCOperand::createReg(XRegs[getReg()]));
   }
 
   void addRegG8RCNoX0Operands(MCInst &Inst, unsigned N) const {
     assert(N == 1 && "Invalid number of operands!");
     Inst.addOperand(MCOperand::createReg(XRegsNoX0[getReg()]));
   }
 
+  void addRegG8pRCOperands(MCInst &Inst, unsigned N) const {
+    assert(N == 1 && "Invalid number of operands!");
+    Inst.addOperand(MCOperand::createReg(XRegs[getG8pReg()]));
+  }
+
   void addRegGxRCOperands(MCInst &Inst, unsigned N) const {
     if (isPPC64())
       addRegG8RCOperands(Inst, N);
     else
       addRegGPRCOperands(Inst, N);
   }
 
   void addRegGxRCNoR0Operands(MCInst &Inst, unsigned N) const {

llvm/lib/Target/PowerPC/Disassembler/PPCDisassembler.cpp

 }
 
 static DecodeStatus DecodeG8RCRegisterClass(MCInst &Inst, uint64_t RegNo,
                                             uint64_t Address,
                                             const void *Decoder) {
   return decodeRegisterClass(Inst, RegNo, XRegs);
 }
 
+static DecodeStatus DecodeG8pRCRegisterClass(MCInst &Inst, uint64_t RegNo,
+                                             uint64_t Address,
+                                             const void *Decoder) {
+  return decodeRegisterClass(Inst, RegNo, XRegs);
+}
+
 static DecodeStatus DecodeG8RC_NOX0RegisterClass(MCInst &Inst, uint64_t RegNo,
                                             uint64_t Address,
                                             const void *Decoder) {
   return decodeRegisterClass(Inst, RegNo, XRegsNoX0);
 }
 
 #define DecodePointerLikeRegClass0 DecodeGPRCRegisterClass
 #define DecodePointerLikeRegClass1 DecodeGPRC_NOR0RegisterClass

llvm/lib/Target/PowerPC/PPCInstr64Bit.td

def ATOMIC_SWAP_I64 : PPCCustomInserterPseudo<
     (outs g8rc:$dst), (ins memrr:$ptr, g8rc:$new), "#ATOMIC_SWAP_I64",
     [(set i64:$dst, (atomic_swap_64 ForceXForm:$ptr, i64:$new))]>;
 }
 
 // Instructions to support atomic operations
 let mayLoad = 1, hasSideEffects = 0 in {
 def LDARX : XForm_1_memOp<31,  84, (outs g8rc:$rD), (ins memrr:$ptr),
                           "ldarx $rD, $ptr", IIC_LdStLDARX, []>;
+// TODO: Add scheduling info.
+let hasNoSchedulingInfo = 1 in
+def LQARX : XForm_1_memOp<31, 276, (outs g8prc:$RTp), (ins memrr:$ptr),
+                          "lqarx $RTp, $ptr", IIC_LdStLQARX, []>;
 
 // Instruction to support lock versions of atomics
 // (EH=1 - see Power ISA 2.07 Book II 4.4.2)
 def LDARXL : XForm_1<31,  84, (outs g8rc:$rD), (ins memrr:$ptr),
                      "ldarx $rD, $ptr, 1", IIC_LdStLDARX, []>, isRecordForm;
+// TODO: Add scheduling info.
+let hasNoSchedulingInfo = 1 in
+def LQARXL : XForm_1<31, 276, (outs g8prc:$RTp), (ins memrr:$ptr),
+                     "lqarx $RTp, $ptr, 1", IIC_LdStLQARX, []>, isRecordForm;
 
 let hasExtraDefRegAllocReq = 1 in
 def LDAT : X_RD5_RS5_IM5<31, 614, (outs g8rc:$rD), (ins g8rc:$rA, u5imm:$FC),
                          "ldat $rD, $rA, $FC", IIC_LdStLoad>, isPPC64,
            Requires<[IsISA3_0]>;
 }
 
-let Defs = [CR0], mayStore = 1, mayLoad = 0, hasSideEffects = 0 in
+let Defs = [CR0], mayStore = 1, mayLoad = 0, hasSideEffects = 0 in {
 def STDCX : XForm_1_memOp<31, 214, (outs), (ins g8rc:$rS, memrr:$dst),
                           "stdcx. $rS, $dst", IIC_LdStSTDCX, []>, isRecordForm;
+// TODO: Add scheduling info.
+let hasNoSchedulingInfo = 1 in
+def STQCX : XForm_1_memOp<31, 182, (outs), (ins g8prc:$RSp, memrr:$dst),
+                          "stqcx. $RSp, $dst", IIC_LdStSTQCX, []>, isRecordForm;
+}
 
 let mayStore = 1, mayLoad = 0, hasSideEffects = 0 in
 def STDAT : X_RD5_RS5_IM5<31, 742, (outs), (ins g8rc:$rS, g8rc:$rA, u5imm:$FC),
                           "stdat $rS, $rA, $FC", IIC_LdStStore>, isPPC64,
             Requires<[IsISA3_0]>;
 
 let Interpretation64Bit = 1, isCodeGenOnly = 1 in {
 let isCall = 1, isTerminator = 1, isReturn = 1, isBarrier = 1, Uses = [RM] in
def LDUX : XForm_1_memOp<31, 53, (outs g8rc:$rD, ptr_rc_nor0:$ea_result),
                         "ldux $rD, $addr", IIC_LdStLDUX,
                         []>, RegConstraint<"$addr.ptrreg = $ea_result">,
                         NoEncode<"$ea_result">, isPPC64;
 
 def LDMX : XForm_1<31, 309, (outs g8rc:$rD), (ins memrr:$src),
                    "ldmx $rD, $src", IIC_LdStLD, []>, isPPC64,
                    Requires<[IsISA3_0]>;
 }
+
+let mayLoad = 1, hasNoSchedulingInfo = 1 in {
+// Full 16-byte load.
+// Early clobber $RTp to avoid assigned to the same register as RA.
+// TODO: Add scheduling info.
+def LQ   : DQForm_RTp5_RA17_MEM<56, 0,
+                                (outs g8prc:$RTp),
+                                (ins memrix16:$src),
+                                "lq $RTp, $src", IIC_LdStLQ,
+                                []>,
+                                RegConstraint<"@earlyclobber $RTp">,
+                                isPPC64;
+def RESTORE_QUADWORD : PPCEmitTimePseudo<(outs g8prc:$RTp), (ins memrix16:$src),
+                                         "#RESTORE_QUADWORD", []>;
+}
+
 }
 
 // Support for medium and large code model.
 let hasSideEffects = 0 in {
 let isReMaterializable = 1 in {
 def ADDIStocHA8: PPCEmitTimePseudo<(outs g8rc:$rD), (ins g8rc_nox0:$reg, tocentry:$disp),
                        "#ADDIStocHA8", []>, isPPC64;
 def ADDItocL: PPCEmitTimePseudo<(outs g8rc:$rD), (ins g8rc_nox0:$reg, tocentry:$disp),

 def STDX  : XForm_8_memOp<31, 149, (outs), (ins g8rc:$rS, memrr:$dst),
                           "stdx $rS, $dst", IIC_LdStSTD,
                           [(store i64:$rS, XForm:$dst)]>, isPPC64,
                           PPC970_DGroup_Cracked;
 def STDBRX: XForm_8_memOp<31, 660, (outs), (ins g8rc:$rS, memrr:$dst),
                           "stdbrx $rS, $dst", IIC_LdStStore,
                           [(PPCstbrx i64:$rS, ForceXForm:$dst, i64)]>, isPPC64,
                           PPC970_DGroup_Cracked;
+
+let mayStore = 1, hasNoSchedulingInfo = 1 in {
+// Normal 16-byte stores.
+// TODO: Add scheduling info.
+def STQ : DSForm_1<62, 2, (outs), (ins g8prc:$RSp, memrix:$dst),
+                   "stq $RSp, $dst", IIC_LdStSTQ,
+                   []>, isPPC64;
+def SPILL_QUADWORD : PPCEmitTimePseudo<(outs), (ins g8prc:$RSp, memrix:$dst),
+                                       "#SPILL_QUADWORD", []>;
+}
+
 }
 
 // Stores with Update (pre-inc).
 let PPC970_Unit = 2, mayStore = 1, mayLoad = 0 in {
 let Interpretation64Bit = 1, isCodeGenOnly = 1 in {
 def STBU8 : DForm_1<39, (outs ptr_rc_nor0:$ea_res), (ins g8rc:$rS, memri:$dst),
                    "stbu $rS, $dst", IIC_LdStSTU, []>,
                    RegConstraint<"$dst.reg = $ea_res">, NoEncode<"$ea_res">;

llvm/lib/Target/PowerPC/PPCInstrFormats.td

class DQ_RD6_RS5_DQ12<bits<6> opcode, bits<3> xo, dag OOL, dag IOL,
 
   let Inst{6-10}  = XT{4-0};
   let Inst{11-15} = DS_RA{16-12};  // Register #
   let Inst{16-27} = DS_RA{11-0};   // Displacement.
   let Inst{28}    = XT{5};
   let Inst{29-31} = xo;
 }
 
+class DQForm_RTp5_RA17_MEM<bits<6> opcode, bits<4> xo, dag OOL, dag IOL,
+                           string asmstr, InstrItinClass itin,
+                           list<dag> pattern>
+  : I<opcode, OOL, IOL, asmstr, itin> {
+  bits<5> RTp;
+  bits<17> DQ_RA;
+  let Pattern = pattern;
+
+  let Inst{6-10} =  RTp{4-0};
+  let Inst{11-15} = DQ_RA{16-12};  // Register #
+  let Inst{16-27} = DQ_RA{11-0};   // Displacement.
+  let Inst{28-31} = xo;
+}
+
 // 1.7.6 X-Form
 class XForm_base_r3xo<bits<6> opcode, bits<10> xo, dag OOL, dag IOL, string asmstr, 
                       InstrItinClass itin, list<dag> pattern>
   : I<opcode, OOL, IOL, asmstr, itin> {
   bits<5> RST;
   bits<5> A;
   bits<5> B;
 

llvm/lib/Target/PowerPC/PPCInstrInfo.h

enum SpillOpcodeKey {
   SOK_VSXVectorSpill,
   SOK_VectorFloat8Spill,
   SOK_VectorFloat4Spill,
   SOK_SpillToVSR,
   SOK_PairedVecSpill,
   SOK_AccumulatorSpill,
   SOK_UAccumulatorSpill,
   SOK_SPESpill,
+  SOK_PairedG8Spill,
   SOK_LastOpcodeSpill // This must be last on the enum.
 };
 
 // Define list of load and store spill opcodes.
 #define NoInstr PPC::INSTRUCTION_LIST_END
 #define Pwr8LoadOpcodes                                                        \
   {                                                                            \
     PPC::LWZ, PPC::LD, PPC::LFD, PPC::LFS, PPC::RESTORE_CR,                    \
         PPC::RESTORE_CRBIT, PPC::LVX, PPC::LXVD2X, PPC::LXSDX, PPC::LXSSPX,    \
-        PPC::SPILLTOVSR_LD, NoInstr, NoInstr, NoInstr, PPC::EVLDD              \
+        PPC::SPILLTOVSR_LD, NoInstr, NoInstr, NoInstr, PPC::EVLDD,             \
+        PPC::RESTORE_QUADWORD                                                  \
   }
 
 #define Pwr9LoadOpcodes                                                        \
   {                                                                            \
     PPC::LWZ, PPC::LD, PPC::LFD, PPC::LFS, PPC::RESTORE_CR,                    \
         PPC::RESTORE_CRBIT, PPC::LVX, PPC::LXV, PPC::DFLOADf64,                \
-        PPC::DFLOADf32, PPC::SPILLTOVSR_LD, NoInstr, NoInstr, NoInstr, NoInstr \
+        PPC::DFLOADf32, PPC::SPILLTOVSR_LD, NoInstr, NoInstr, NoInstr,         \
+        NoInstr, PPC::RESTORE_QUADWORD                                         \
   }
 
 #define Pwr10LoadOpcodes                                                       \
   {                                                                            \
     PPC::LWZ, PPC::LD, PPC::LFD, PPC::LFS, PPC::RESTORE_CR,                    \
         PPC::RESTORE_CRBIT, PPC::LVX, PPC::LXV, PPC::DFLOADf64,                \
         PPC::DFLOADf32, PPC::SPILLTOVSR_LD, PPC::LXVP, PPC::RESTORE_ACC,       \
-        PPC::RESTORE_UACC, NoInstr                                             \
+        PPC::RESTORE_UACC, NoInstr, PPC::RESTORE_QUADWORD                      \
   }
 
 #define Pwr8StoreOpcodes                                                       \
   {                                                                            \
     PPC::STW, PPC::STD, PPC::STFD, PPC::STFS, PPC::SPILL_CR, PPC::SPILL_CRBIT, \
         PPC::STVX, PPC::STXVD2X, PPC::STXSDX, PPC::STXSSPX,                    \
-        PPC::SPILLTOVSR_ST, NoInstr, NoInstr, NoInstr, PPC::EVSTDD             \
+        PPC::SPILLTOVSR_ST, NoInstr, NoInstr, NoInstr, PPC::EVSTDD,            \
+        PPC::SPILL_QUADWORD                                                    \
   }
 
 #define Pwr9StoreOpcodes                                                       \
   {                                                                            \
     PPC::STW, PPC::STD, PPC::STFD, PPC::STFS, PPC::SPILL_CR, PPC::SPILL_CRBIT, \
         PPC::STVX, PPC::STXV, PPC::DFSTOREf64, PPC::DFSTOREf32,                \
-        PPC::SPILLTOVSR_ST, NoInstr, NoInstr, NoInstr, NoInstr                 \
+        PPC::SPILLTOVSR_ST, NoInstr, NoInstr, NoInstr, NoInstr,                \
+        PPC::SPILL_QUADWORD                                                    \
   }
 
 #define Pwr10StoreOpcodes                                                      \
   {                                                                            \
     PPC::STW, PPC::STD, PPC::STFD, PPC::STFS, PPC::SPILL_CR, PPC::SPILL_CRBIT, \
         PPC::STVX, PPC::STXV, PPC::DFSTOREf64, PPC::DFSTOREf32,                \
         PPC::SPILLTOVSR_ST, PPC::STXVP, PPC::SPILL_ACC, PPC::SPILL_UACC,       \
-        NoInstr                                                                \
+        NoInstr, PPC::SPILL_QUADWORD                                           \
   }
 
 // Initialize arrays for load and store spill opcodes on supported subtargets.
 #define StoreOpcodesForSpill                                                   \
   { Pwr8StoreOpcodes, Pwr9StoreOpcodes, Pwr10StoreOpcodes }
 #define LoadOpcodesForSpill                                                    \
   { Pwr8LoadOpcodes, Pwr9LoadOpcodes, Pwr10LoadOpcodes }
 

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

for (unsigned Idx = 0; Idx < 4; Idx++)
       BuildMI(MBB, I, DL, get(PPC::XXLOR), VSLDestReg + Idx)
           .addReg(VSLSrcReg + Idx)
           .addReg(VSLSrcReg + Idx, getKillRegState(KillSrc));
     if (DestPrimed)
       BuildMI(MBB, I, DL, get(PPC::XXMTACC), DestReg).addReg(DestReg);
     if (SrcPrimed && !KillSrc)
       BuildMI(MBB, I, DL, get(PPC::XXMTACC), SrcReg).addReg(SrcReg);
     return;
+  } else if (PPC::G8pRCRegClass.contains(DestReg) &&
+             PPC::G8pRCRegClass.contains(SrcReg)) {
+    // TODO: Handle G8RC to G8pRC (and vice versa) copy.
+    unsigned DestRegIdx = DestReg - PPC::G8p0;
+    MCRegister DestRegSub0 = PPC::X0 + 2 * DestRegIdx;
+    MCRegister DestRegSub1 = PPC::X0 + 2 * DestRegIdx + 1;
+    unsigned SrcRegIdx = SrcReg - PPC::G8p0;
+    MCRegister SrcRegSub0 = PPC::X0 + 2 * SrcRegIdx;
+    MCRegister SrcRegSub1 = PPC::X0 + 2 * SrcRegIdx + 1;
+    BuildMI(MBB, I, DL, get(PPC::OR8), DestRegSub0)
+        .addReg(SrcRegSub0)
+        .addReg(SrcRegSub0, getKillRegState(KillSrc));
+    BuildMI(MBB, I, DL, get(PPC::OR8), DestRegSub1)
+        .addReg(SrcRegSub1)
+        .addReg(SrcRegSub1, getKillRegState(KillSrc));
+    return;
   } else
     llvm_unreachable("Impossible reg-to-reg copy");
 
   const MCInstrDesc &MCID = get(Opc);
   if (MCID.getNumOperands() == 3)
     BuildMI(MBB, I, DL, MCID, DestReg)
       .addReg(SrcReg).addReg(SrcReg, getKillRegState(KillSrc));
   else
unsigned PPCInstrInfo::getSpillIndex(const TargetRegisterClass *RC) const {
   } else if (PPC::UACCRCRegClass.hasSubClassEq(RC)) {
     assert(Subtarget.pairedVectorMemops() &&
            "Register unexpected when paired memops are disabled.");
     OpcodeIndex = SOK_UAccumulatorSpill;
   } else if (PPC::VSRpRCRegClass.hasSubClassEq(RC)) {
     assert(Subtarget.pairedVectorMemops() &&
            "Register unexpected when paired memops are disabled.");
     OpcodeIndex = SOK_PairedVecSpill;
+  } else if (PPC::G8pRCRegClass.hasSubClassEq(RC)) {
+    OpcodeIndex = SOK_PairedG8Spill;
   } else {
     llvm_unreachable("Unknown regclass!");
   }
   return OpcodeIndex;
 }
 
 unsigned
 PPCInstrInfo::getStoreOpcodeForSpill(const TargetRegisterClass *RC) const {

llvm/lib/Target/PowerPC/PPCInstrInfo.td

def gprc : RegisterOperand<GPRC> {
   let ParserMatchClass = PPCRegGPRCAsmOperand;
 }
 def PPCRegG8RCAsmOperand : AsmOperandClass {
   let Name = "RegG8RC"; let PredicateMethod = "isRegNumber";
 }
 def g8rc : RegisterOperand<G8RC> {
   let ParserMatchClass = PPCRegG8RCAsmOperand;
 }
+def PPCRegG8pRCAsmOperand : AsmOperandClass {
+  let Name = "RegG8pRC"; let PredicateMethod = "isG8pEvenRegNumber";
+}
+def g8prc : RegisterOperand<G8pRC> {
+  let ParserMatchClass = PPCRegG8pRCAsmOperand;
+}
 def PPCRegGPRCNoR0AsmOperand : AsmOperandClass {
   let Name = "RegGPRCNoR0"; let PredicateMethod = "isRegNumber";
 }
 def gprc_nor0 : RegisterOperand<GPRC_NOR0> {
   let ParserMatchClass = PPCRegGPRCNoR0AsmOperand;
 }
 def PPCRegG8RCNoX0AsmOperand : AsmOperandClass {
   let Name = "RegG8RCNoX0"; let PredicateMethod = "isRegNumber";

llvm/lib/Target/PowerPC/PPCRegisterInfo.h

public:
   void lowerCRBitRestore(MachineBasicBlock::iterator II,
                          unsigned FrameIndex) const;
 
   void lowerACCSpilling(MachineBasicBlock::iterator II,
                         unsigned FrameIndex) const;
   void lowerACCRestore(MachineBasicBlock::iterator II,
                        unsigned FrameIndex) const;
 
+  void lowerQuadwordSpilling(MachineBasicBlock::iterator II,
+                             unsigned FrameIndex) const;
+  void lowerQuadwordRestore(MachineBasicBlock::iterator II,
+                            unsigned FrameIndex) const;
+
   static void emitAccCopyInfo(MachineBasicBlock &MBB, MCRegister DestReg,
                               MCRegister SrcReg);
 
   bool hasReservedSpillSlot(const MachineFunction &MF, Register Reg,
                             int &FrameIdx) const override;
   void eliminateFrameIndex(MachineBasicBlock::iterator II, int SPAdj,
                            unsigned FIOperandNum,
                            RegScavenger *RS = nullptr) const override;

llvm/lib/Target/PowerPC/PPCRegisterInfo.cpp

PPCRegisterInfo::PPCRegisterInfo(const PPCTargetMachine &TM)
   // 64-bit
   ImmToIdxMap[PPC::LHA8] = PPC::LHAX8; ImmToIdxMap[PPC::LBZ8] = PPC::LBZX8;
   ImmToIdxMap[PPC::LHZ8] = PPC::LHZX8; ImmToIdxMap[PPC::LWZ8] = PPC::LWZX8;
   ImmToIdxMap[PPC::STB8] = PPC::STBX8; ImmToIdxMap[PPC::STH8] = PPC::STHX8;
   ImmToIdxMap[PPC::STW8] = PPC::STWX8; ImmToIdxMap[PPC::STDU] = PPC::STDUX;
   ImmToIdxMap[PPC::ADDI8] = PPC::ADD8;
 
   // VSX
   ImmToIdxMap[PPC::DFLOADf32] = PPC::LXSSPX;
   ImmToIdxMap[PPC::DFLOADf64] = PPC::LXSDX;
   ImmToIdxMap[PPC::SPILLTOVSR_LD] = PPC::SPILLTOVSR_LDX;
   ImmToIdxMap[PPC::SPILLTOVSR_ST] = PPC::SPILLTOVSR_STX;
   ImmToIdxMap[PPC::DFSTOREf32] = PPC::STXSSPX;
   ImmToIdxMap[PPC::DFSTOREf64] = PPC::STXSDX;
   ImmToIdxMap[PPC::LXV] = PPC::LXVX;
   ImmToIdxMap[PPC::LXSD] = PPC::LXSDX;
   ImmToIdxMap[PPC::LXSSP] = PPC::LXSSPX;
addFrameReference(BuildMI(MBB, II, DL, TII.get(PPC::LXVP), Reg + 1),
                     FrameIndex, IsLittleEndian ? 0 : 32);
   if (IsPrimed)
     BuildMI(MBB, II, DL, TII.get(PPC::XXMTACC), DestReg).addReg(DestReg);
 
   // Discard the pseudo instruction.
   MBB.erase(II);
 }
 
+/// lowerQuadwordSpilling - Generate code to spill paired general register.
+void PPCRegisterInfo::lowerQuadwordSpilling(MachineBasicBlock::iterator II,
+                                            unsigned FrameIndex) const {
+  MachineInstr &MI = *II;
+  MachineBasicBlock &MBB = *MI.getParent();
+  MachineFunction &MF = *MBB.getParent();
+  const PPCSubtarget &Subtarget = MF.getSubtarget<PPCSubtarget>();
+  const TargetInstrInfo &TII = *Subtarget.getInstrInfo();
+  DebugLoc DL = MI.getDebugLoc();
+
+  Register SrcReg = MI.getOperand(0).getReg();
+  bool IsKilled = MI.getOperand(0).isKill();
+
+  Register Reg = PPC::X0 + (SrcReg - PPC::G8p0) * 2;
+  bool IsLittleEndian = Subtarget.isLittleEndian();
+
+  addFrameReference(BuildMI(MBB, II, DL, TII.get(PPC::STD))
+                        .addReg(Reg, getKillRegState(IsKilled)),
+                    FrameIndex, IsLittleEndian ? 8 : 0);
+  addFrameReference(BuildMI(MBB, II, DL, TII.get(PPC::STD))
+                        .addReg(Reg + 1, getKillRegState(IsKilled)),
+                    FrameIndex, IsLittleEndian ? 0 : 8);
+
+  // Discard the pseudo instruction.
+  MBB.erase(II);
+}
+
+/// lowerQuadwordRestore - Generate code to restore paired general register.
+void PPCRegisterInfo::lowerQuadwordRestore(MachineBasicBlock::iterator II,
+                                           unsigned FrameIndex) const {
+  MachineInstr &MI = *II;
+  MachineBasicBlock &MBB = *MI.getParent();
+  MachineFunction &MF = *MBB.getParent();
+  const PPCSubtarget &Subtarget = MF.getSubtarget<PPCSubtarget>();
+  const TargetInstrInfo &TII = *Subtarget.getInstrInfo();
+  DebugLoc DL = MI.getDebugLoc();
+
+  Register DestReg = MI.getOperand(0).getReg();
+  assert(MI.definesRegister(DestReg) &&
+         "RESTORE_QUADWORD does not define its destination");
+
+  Register Reg = PPC::X0 + (DestReg - PPC::G8p0) * 2;
+  bool IsLittleEndian = Subtarget.isLittleEndian();
+
+  addFrameReference(BuildMI(MBB, II, DL, TII.get(PPC::LD), Reg), FrameIndex,
+                    IsLittleEndian ? 8 : 0);
+  addFrameReference(BuildMI(MBB, II, DL, TII.get(PPC::LD), Reg + 1), FrameIndex,
+                    IsLittleEndian ? 0 : 8);
+
+  // Discard the pseudo instruction.
+  MBB.erase(II);
+}
+
 bool PPCRegisterInfo::hasReservedSpillSlot(const MachineFunction &MF,
                                            Register Reg, int &FrameIdx) const {
   // For the nonvolatile condition registers (CR2, CR3, CR4) return true to
   // prevent allocating an additional frame slot.
   // For 64-bit ELF and AIX, the CR save area is in the linkage area at SP+8,
   // for 32-bit AIX the CR save area is in the linkage area at SP+4.
   // We have created a FrameIndex to that spill slot to keep the CalleSaveInfos
   // valid.
static unsigned offsetMinAlignForOpcode(unsigned OpC) {
   case PPC::DFLOADf32:
   case PPC::DFLOADf64:
   case PPC::DFSTOREf32:
   case PPC::DFSTOREf64:
   case PPC::LXSD:
   case PPC::LXSSP:
   case PPC::STXSD:
   case PPC::STXSSP:
+  case PPC::STQ:
     return 4;
   case PPC::EVLDD:
   case PPC::EVSTDD:
     return 8;
   case PPC::LXV:
   case PPC::STXV:
+  case PPC::LQ:
     return 16;
   }
 }
 
 // If the offset must be a multiple of some value, return what that value is.
 static unsigned offsetMinAlign(const MachineInstr &MI) {
   unsigned OpC = MI.getOpcode();
   return offsetMinAlignForOpcode(OpC);
if (OpC == PPC::SPILL_CR) {
     lowerCRBitRestore(II, FrameIndex);
     return;
   } else if (OpC == PPC::SPILL_ACC || OpC == PPC::SPILL_UACC) {
     lowerACCSpilling(II, FrameIndex);
     return;
   } else if (OpC == PPC::RESTORE_ACC || OpC == PPC::RESTORE_UACC) {
     lowerACCRestore(II, FrameIndex);
     return;
+  } else if (OpC == PPC::SPILL_QUADWORD) {
+    lowerQuadwordSpilling(II, FrameIndex);
+    return;
+  } else if (OpC == PPC::RESTORE_QUADWORD) {
+    lowerQuadwordRestore(II, FrameIndex);
+    return;
   }
 
   // Replace the FrameIndex with base register with GPR1 (SP) or GPR31 (FP).
   MI.getOperand(FIOperandNum).ChangeToRegister(
     FrameIndex < 0 ? getBaseRegister(MF) : getFrameRegister(MF), false);
 
   // If the instruction is not present in ImmToIdxMap, then it has no immediate
   // form (and must be r+r).

llvm/lib/Target/PowerPC/PPCRegisterInfo.td

 def sub_eq : SubRegIndex<1, 2>;
 def sub_un : SubRegIndex<1, 3>;
 def sub_32 : SubRegIndex<32>;
 def sub_64 : SubRegIndex<64>;
 def sub_vsx0 : SubRegIndex<128>;
 def sub_vsx1 : SubRegIndex<128, 128>;
 def sub_pair0 : SubRegIndex<256>;
 def sub_pair1 : SubRegIndex<256, 256>;
+def sub_gp8_x0 : SubRegIndex<64>;
+def sub_gp8_x1 : SubRegIndex<64, 64>;
 }
 
 
 class PPCReg<string n> : Register<n> {
   let Namespace = "PPC";
 }
 
 // We identify all our registers with a 5-bit ID, for consistency's sake.

 }
 
 // VSR Pairs - One of the 32 paired even-odd consecutive VSRs.
 class VSRPair<bits<5> num, string n, list<Register> subregs> : PPCReg<n> {
   let HWEncoding{4-0} = num;
   let SubRegs = subregs;
 }
 
+// GP8Pair - One of the 16 paired even-odd consecutinve GP8s.
+class GP8Pair<bits<4> num, string n, list<Register> subregs> : PPCReg<n> {
+  let HWEncoding{3-0} = num;
+  let SubRegs = subregs;
+}
+
 // General-purpose registers
 foreach Index = 0-31 in {
   def R#Index : GPR<Index, "r"#Index>, DwarfRegNum<[-2, Index]>;
 }
 
 // 64-bit General-purpose registers
 foreach Index = 0-31 in {
   def X#Index : GP8<!cast<GPR>("R"#Index), "r"#Index>,
let SubRegIndices = [sub_vsx0, sub_vsx1] in {
   foreach Index = { 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30 } in {
     def VSRp#!add(!srl(Index, 1), 16) :
       VSRPair<!add(!srl(Index, 1), 16), "vsp"#!add(Index, 32),
               [!cast<VR>("V"#Index), !cast<VR>("V"#!add(Index, 1))]>,
       DwarfRegNum<[0, 0]>;
   }
 }
 
+let SubRegIndices = [sub_gp8_x0, sub_gp8_x1] in {
+  foreach Index = { 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30 } in {
+    def G8p#!srl(Index, 1) :
+      GP8Pair<!srl(Index, 1), "r"#Index,
+              [!cast<GP8>("X"#Index), !cast<GP8>("X"#!add(Index, 1))]>,
+      DwarfRegNum<[Index, -2]>;
+  }
+}
+
 // The representation of r0 when treated as the constant 0.
 def ZERO  : GPR<0, "0">,    DwarfRegAlias<R0>;
 def ZERO8 : GP8<ZERO, "0">, DwarfRegAlias<X0>;
 
 // Representations of the frame pointer used by ISD::FRAMEADDR.
 def FP   : GPR<0 /* arbitrary */, "**FRAME POINTER**">;
 def FP8  : GP8<FP, "**FRAME POINTER**">;
 
def VSRpRC :
   RegisterClass<"PPC", [v256i1], 128,
                 (add (sequence "VSRp%u", 0, 6),
                      (sequence "VSRp%u", 15, 7), VSRp17, VSRp18,
                      VSRp16, VSRp19, VSRp20, VSRp21, VSRp22, VSRp23,
                      VSRp24, VSRp25, VSRp31, VSRp30, VSRp29, VSRp28,
                      VSRp27, VSRp26)> {
   let Size = 256;
 }
+
+def G8pRC :
+  RegisterClass<"PPC", [i128], 128,
+                (add (sequence "G8p%u", 1, 5),
+                     (sequence "G8p%u", 14, 7),
+                     G8p15, G8p6, G8p0)> {
+  let AltOrders = [(add (sub G8pRC, G8p1), G8p1)];
+  let AltOrderSelect = [{
+    return MF.getSubtarget<PPCSubtarget>().is64BitELFABI();
+  }];
+  let Size = 128;
+}

llvm/lib/Target/PowerPC/PPCSchedule.td

 def IIC_LdStDCBF     : InstrItinClass;
 def IIC_LdStDCBI     : InstrItinClass;
 def IIC_LdStLoad     : InstrItinClass;
 def IIC_LdStLoadUpd  : InstrItinClass;
 def IIC_LdStLoadUpdX : InstrItinClass;
 def IIC_LdStStore    : InstrItinClass;
 def IIC_LdStDSS      : InstrItinClass;
 def IIC_LdStICBI     : InstrItinClass;
+def IIC_LdStLQ       : InstrItinClass;
+def IIC_LdStLQARX    : InstrItinClass;
 def IIC_LdStLD       : InstrItinClass;
 def IIC_LdStLDU      : InstrItinClass;
 def IIC_LdStLDUX     : InstrItinClass;
 def IIC_LdStLDARX    : InstrItinClass;
 def IIC_LdStLFD      : InstrItinClass;
 def IIC_LdStLFDU     : InstrItinClass;
 def IIC_LdStLFDUX    : InstrItinClass;
 def IIC_LdStLHA      : InstrItinClass;
 def IIC_LdStLHAU     : InstrItinClass;
 def IIC_LdStLHAUX    : InstrItinClass;
 def IIC_LdStLMW      : InstrItinClass;
 def IIC_LdStLVecX    : InstrItinClass;
 def IIC_LdStLWA      : InstrItinClass;
 def IIC_LdStLWARX    : InstrItinClass;
 def IIC_LdStSLBIA    : InstrItinClass;
 def IIC_LdStSLBIE    : InstrItinClass;
+def IIC_LdStSTQ      : InstrItinClass;
+def IIC_LdStSTQCX    : InstrItinClass;
 def IIC_LdStSTD      : InstrItinClass;
 def IIC_LdStSTDCX    : InstrItinClass;
 def IIC_LdStSTU      : InstrItinClass;
 def IIC_LdStSTUX     : InstrItinClass;
 def IIC_LdStSTFD     : InstrItinClass;
 def IIC_LdStSTFDU    : InstrItinClass;
 def IIC_LdStSTVEBX   : InstrItinClass;
 def IIC_LdStSTWCX    : InstrItinClass;

llvm/test/CodeGen/PowerPC/ldst-16-byte.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -simplify-mir -verify-machineinstrs -mtriple=powerpc64-ibm-aix-xcoff \
+# RUN:   -stop-after=postrapseudos %s -o - | FileCheck %s
+
+---
+name: foo
+alignment: 8
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+  liveins: $x3, $x4
+    ; CHECK-LABEL: name: foo
+    ; CHECK: liveins: $x3, $x4
+    ; CHECK: early-clobber renamable $g8p3 = LQ 128, $x4
+    ; CHECK: $x3 = OR8 $x7, $x7
+    ; CHECK: STQ killed renamable $g8p3, 160, $x3
+    ; CHECK: BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+  %0:g8prc = LQ 128, $x4
+  $x3 = COPY %0.sub_gp8_x1:g8prc
+  STQ %0, 160, $x3
+  BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+...
+
+---
+name: foobar
+alignment: 8
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+  liveins: $x3, $x4
+    ; CHECK-LABEL: name: foobar
+    ; CHECK: liveins: $x3, $x4
+    ; CHECK: renamable $g8p3 = LQARX $x3, $x4
+    ; CHECK: STQCX renamable $g8p3, $x3, $x4, implicit-def dead $cr0
+    ; CHECK: $x3 = OR8 $x7, $x7, implicit killed $g8p3
+    ; CHECK: BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+  %0:g8prc = LQARX $x3, $x4
+  STQCX %0:g8prc, $x3, $x4, implicit-def $cr0
+  $x3 = COPY %0.sub_gp8_x1:g8prc
+  BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+...
+
+---
+name: bar
+alignment: 8
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+  liveins: $x3, $x4
+    ; CHECK-LABEL: name: bar
+    ; CHECK: liveins: $x3, $x4
+    ; CHECK: early-clobber renamable $g8p2 = LQ 128, renamable $x3
+    ; CHECK: STQ renamable $g8p2, 160, $x3
+    ; CHECK: $x3 = OR8 $x4, $x4, implicit killed $g8p2
+    ; CHECK: BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+  %0:g8rc_nox0 = COPY $x3
+  %1:g8prc = LQ 128, %0
+  STQ %1, 160, $x3
+  $x3 = COPY %1.sub_gp8_x0:g8prc
+  BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+...
+
+# `spill_g8prc`'s code contains undefined behaviors. This code is only to
+# demonstrate correctness of spiller.
+---
+name: spill_g8prc
+alignment: 8
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+  liveins: $x3, $x4, $x5, $x6, $x7, $x8, $x9, $x10, $x11, $x12
+    ; CHECK-LABEL: name: spill_g8prc
+    ; CHECK: liveins: $x3, $x4, $x5, $x6, $x7, $x8, $x9, $x10, $x11, $x12, $x14, $x15, $x16, $x17, $x18, $x19, $x20, $x21, $x22, $x23, $x24, $x25, $x26, $x27, $x28, $x29, $x30, $x31
+    ; CHECK: STD killed $x14, -144, $x1 :: (store 8 into %fixed-stack.17, align 16)
+    ; CHECK: STD killed $x15, -136, $x1 :: (store 8 into %fixed-stack.16)
+    ; CHECK: STD killed $x16, -128, $x1 :: (store 8 into %fixed-stack.15, align 16)
+    ; CHECK: STD killed $x17, -120, $x1 :: (store 8 into %fixed-stack.14)
+    ; CHECK: STD killed $x18, -112, $x1 :: (store 8 into %fixed-stack.13, align 16)
+    ; CHECK: STD killed $x19, -104, $x1 :: (store 8 into %fixed-stack.12)
+    ; CHECK: STD killed $x20, -96, $x1 :: (store 8 into %fixed-stack.11, align 16)
+    ; CHECK: STD killed $x21, -88, $x1 :: (store 8 into %fixed-stack.10)
+    ; CHECK: STD killed $x22, -80, $x1 :: (store 8 into %fixed-stack.9, align 16)
+    ; CHECK: STD killed $x23, -72, $x1 :: (store 8 into %fixed-stack.8)
+    ; CHECK: STD killed $x24, -64, $x1 :: (store 8 into %fixed-stack.7, align 16)
+    ; CHECK: STD killed $x25, -56, $x1 :: (store 8 into %fixed-stack.6)
+    ; CHECK: STD killed $x26, -48, $x1 :: (store 8 into %fixed-stack.5, align 16)
+    ; CHECK: STD killed $x27, -40, $x1 :: (store 8 into %fixed-stack.4)
+    ; CHECK: STD killed $x28, -32, $x1 :: (store 8 into %fixed-stack.3, align 16)
+    ; CHECK: STD killed $x29, -24, $x1 :: (store 8 into %fixed-stack.2)
+    ; CHECK: STD killed $x30, -16, $x1 :: (store 8 into %fixed-stack.1, align 16)
+    ; CHECK: STD killed $x31, -8, $x1 :: (store 8 into %fixed-stack.0)
+    ; CHECK: $x7 = OR8 $x3, $x3
+    ; CHECK: renamable $g8p4 = LQARX $x5, $x6
+    ; CHECK: STD killed $x8, -160, $x1
+    ; CHECK: STD killed $x9, -152, $x1
+    ; CHECK: renamable $g8p5 = LQARX $x3, renamable $x4
+    ; CHECK: renamable $g8p4 = LQARX $x3, renamable $x4
+    ; CHECK: STD killed $x8, -176, $x1
+    ; CHECK: STD killed $x9, -168, $x1
+    ; CHECK: renamable $g8p4 = LQARX $x3, renamable $x4
+    ; CHECK: STD killed $x8, -192, $x1
+    ; CHECK: STD killed $x9, -184, $x1
+    ; CHECK: renamable $g8p4 = LQARX $x3, renamable $x4
+    ; CHECK: STD killed $x8, -208, $x1
+    ; CHECK: STD killed $x9, -200, $x1
+    ; CHECK: renamable $g8p4 = LQARX $x3, renamable $x4
+    ; CHECK: STD killed $x8, -224, $x1
+    ; CHECK: STD killed $x9, -216, $x1
+    ; CHECK: renamable $g8p10 = LQARX $x3, renamable $x4
+    ; CHECK: renamable $g8p9 = LQARX $x3, renamable $x4
+    ; CHECK: renamable $g8p8 = LQARX $x3, renamable $x4
+    ; CHECK: renamable $g8p7 = LQARX $x3, renamable $x4
+    ; CHECK: renamable $g8p15 = LQARX $x3, renamable $x4
+    ; CHECK: renamable $g8p11 = LQARX $x3, renamable $x4
+    ; CHECK: renamable $g8p12 = LQARX $x3, renamable $x4
+    ; CHECK: renamable $g8p13 = LQARX $x3, renamable $x4
+    ; CHECK: renamable $g8p14 = LQARX $x3, renamable $x4
+    ; CHECK: renamable $g8p4 = LQARX $x3, renamable $x4
+    ; CHECK: $x3 = OR8 $x11, $x11
+    ; CHECK: STQCX killed renamable $g8p4, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p14, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p13, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p12, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p11, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p15, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p7, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p8, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p9, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p10, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: $x8 = LD -224, $x1
+    ; CHECK: $x9 = LD -216, $x1
+    ; CHECK: STQCX killed renamable $g8p4, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: $x8 = LD -208, $x1
+    ; CHECK: $x9 = LD -200, $x1
+    ; CHECK: STQCX killed renamable $g8p4, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: $x8 = LD -192, $x1
+    ; CHECK: $x9 = LD -184, $x1
+    ; CHECK: STQCX killed renamable $g8p4, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: $x8 = LD -176, $x1
+    ; CHECK: $x9 = LD -168, $x1
+    ; CHECK: STQCX killed renamable $g8p4, renamable $x7, renamable $x4, implicit-def dead $cr0
+    ; CHECK: STQCX killed renamable $g8p5, killed renamable $x7, killed renamable $x4, implicit-def dead $cr0
+    ; CHECK: $x8 = LD -160, $x1
+    ; CHECK: $x9 = LD -152, $x1
+    ; CHECK: STQCX killed renamable $g8p4, $x5, $x6, implicit-def dead $cr0
+    ; CHECK: $x31 = LD -8, $x1 :: (load 8 from %fixed-stack.0)
+    ; CHECK: $x30 = LD -16, $x1 :: (load 8 from %fixed-stack.1, align 16)
+    ; CHECK: $x29 = LD -24, $x1 :: (load 8 from %fixed-stack.2)
+    ; CHECK: $x28 = LD -32, $x1 :: (load 8 from %fixed-stack.3, align 16)
+    ; CHECK: $x27 = LD -40, $x1 :: (load 8 from %fixed-stack.4)
+    ; CHECK: $x26 = LD -48, $x1 :: (load 8 from %fixed-stack.5, align 16)
+    ; CHECK: $x25 = LD -56, $x1 :: (load 8 from %fixed-stack.6)
+    ; CHECK: $x24 = LD -64, $x1 :: (load 8 from %fixed-stack.7, align 16)
+    ; CHECK: $x23 = LD -72, $x1 :: (load 8 from %fixed-stack.8)
+    ; CHECK: $x22 = LD -80, $x1 :: (load 8 from %fixed-stack.9, align 16)
+    ; CHECK: $x21 = LD -88, $x1 :: (load 8 from %fixed-stack.10)
+    ; CHECK: $x20 = LD -96, $x1 :: (load 8 from %fixed-stack.11, align 16)
+    ; CHECK: $x19 = LD -104, $x1 :: (load 8 from %fixed-stack.12)
+    ; CHECK: $x18 = LD -112, $x1 :: (load 8 from %fixed-stack.13, align 16)
+    ; CHECK: $x17 = LD -120, $x1 :: (load 8 from %fixed-stack.14)
+    ; CHECK: $x16 = LD -128, $x1 :: (load 8 from %fixed-stack.15, align 16)
+    ; CHECK: $x15 = LD -136, $x1 :: (load 8 from %fixed-stack.16)
+    ; CHECK: $x14 = LD -144, $x1 :: (load 8 from %fixed-stack.17, align 16)
+    ; CHECK: BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+  %addr0:g8rc_nox0 = COPY $x3
+  %addr1:g8rc = COPY $x4
+  %0:g8prc = LQARX $x5, $x6
+  %1:g8prc = LQARX %addr0, %addr1
+  %2:g8prc = LQARX %addr0, %addr1
+  %3:g8prc = LQARX %addr0, %addr1
+  %4:g8prc = LQARX %addr0, %addr1
+  %5:g8prc = LQARX %addr0, %addr1
+  %6:g8prc = LQARX %addr0, %addr1
+  %7:g8prc = LQARX %addr0, %addr1
+  %8:g8prc = LQARX %addr0, %addr1
+  %9:g8prc = LQARX %addr0, %addr1
+  %10:g8prc = LQARX %addr0, %addr1
+  %11:g8prc = LQARX %addr0, %addr1
+  %12:g8prc = LQARX %addr0, %addr1
+  %13:g8prc = LQARX %addr0, %addr1
+  %14:g8prc = LQARX %addr0, %addr1
+  %15:g8prc = LQARX %addr0, %addr1
+  STQCX %15:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %14:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %13:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %12:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %11:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %10:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %9:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %8:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %7:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %6:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %5:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %4:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %3:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %2:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %1:g8prc, %addr0, %addr1, implicit-def $cr0
+  STQCX %0:g8prc, $x5, $x6, implicit-def $cr0
+  $x3 = COPY %1.sub_gp8_x1:g8prc
+  BLR8 implicit $lr8, implicit undef $rm, implicit $x3
+...
+
+---
+name: copy_g8prc
+alignment: 8
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+  liveins: $g8p8
+    ; CHECK-LABEL: name: copy_g8prc
+    ; CHECK: liveins: $g8p8
+    ; CHECK: $x4 = OR8 $x16, $x16
+    ; CHECK: $x5 = OR8 $x17, $x17
+    ; CHECK: $x3 = OR8 $x5, $x5
+    ; CHECK: $x4 = KILL renamable $x4, implicit killed $g8p2
+    ; CHECK: BLR8 implicit $lr8, implicit undef $rm, implicit killed $x3, implicit $x4
+  %0:g8prc = COPY $g8p8
+  $x3 = COPY %0.sub_gp8_x1:g8prc
+  $x4 = COPY %0.sub_gp8_x0:g8prc
+  BLR8 implicit $lr8, implicit undef $rm, implicit $x3, implicit $x4
+...

llvm/test/MC/Disassembler/PowerPC/ppc64-encoding-bookII.txt

 0x4c 0x00 0x01 0x2c
 
 # CHECK: stwcx. 2, 3, 4                  
 0x7c 0x43 0x21 0x2d
 
 # CHECK: stdcx. 2, 3, 4                  
 0x7c 0x43 0x21 0xad
 
+# CHECK: stqcx. 2, 3, 4
+0x7c 0x43 0x21 0x6d
+
 # CHECK: stwat 2, 3, 28
 0x7c 0x43 0xe5 0x8c
 
 # CHECK: stdat 2, 3, 28
 0x7c 0x43 0xe5 0xcc
 
 # CHECK: ptesync
 0x7c 0x40 0x04 0xac

 0x7c 0x43 0x20 0xe8
 
 # CHECK: lwarx 2, 3, 4
 0x7c 0x43 0x20 0x28
 
 # CHECK: ldarx 2, 3, 4
 0x7c 0x43 0x20 0xa8
 
+# CHECK: lqarx 2, 3, 4
+0x7c 0x43 0x22 0x28
+
 # CHECK: lbarx 2, 3, 4, 1
 0x7c 0x43 0x20 0x69
 
 # CHECK: lharx 2, 3, 4, 1
 0x7c 0x43 0x20 0xe9
 
 # CHECK: lwarx 2, 3, 4, 1
 0x7c 0x43 0x20 0x29
 
 # CHECK: ldarx 2, 3, 4, 1
 0x7c 0x43 0x20 0xa9
 
+# CHECK: lqarx 2, 3, 4, 1
+0x7c 0x43 0x22 0x29
+
 # CHECK: lwat 2, 3, 28
 0x7c 0x43 0xe4 0x8c
 
 # CHECK: ldat 2, 3, 28
 0x7c 0x43 0xe4 0xcc
 
 # CHECK: sync
 0x7c 0x00 0x04 0xac

llvm/test/MC/Disassembler/PowerPC/ppc64-encoding.txt

 0xe8 0x44 0x00 0x81
 
 # CHECK: ldux 2, 3, 4                    
 0x7c 0x43 0x20 0x6a
 
 # CHECK: ldmx 2, 3, 4
 0x7c 0x43 0x22 0x6a
 
+# CHECK: lq 2, 128(4)
+0xe0 0x44 0x00 0x80
+
 # CHECK: stb 2, 128(4)                   
 0x98 0x44 0x00 0x80
 
 # CHECK: stbx 2, 3, 4                    
 0x7c 0x43 0x21 0xae
 
 # CHECK: stbu 2, 128(4)                  
 0x9c 0x44 0x00 0x80

 0x7c 0x43 0x21 0x2a
 
 # CHECK: stdu 2, 128(4)                  
 0xf8 0x44 0x00 0x81
 
 # CHECK: stdux 2, 3, 4                   
 0x7c 0x43 0x21 0x6a
 
+# CHECK: stq 2, 128(4)
+0xf8 0x44 0x00 0x82
+
 # CHECK: lhbrx 2, 3, 4                   
 0x7c 0x43 0x26 0x2c
 
 # CHECK: sthbrx 2, 3, 4                  
 0x7c 0x43 0x27 0x2c
 
 # CHECK: lwbrx 2, 3, 4                   
 0x7c 0x43 0x24 0x2c

llvm/test/MC/PowerPC/ppc64-encoding-bookII.s

 # CHECK-BE: stwcx. 2, 3, 4                  # encoding: [0x7c,0x43,0x21,0x2d]
 # CHECK-LE: stwcx. 2, 3, 4                  # encoding: [0x2d,0x21,0x43,0x7c]
             stwcx. 2, 3, 4
 
 # CHECK-BE: stdcx. 2, 3, 4                  # encoding: [0x7c,0x43,0x21,0xad]
 # CHECK-LE: stdcx. 2, 3, 4                  # encoding: [0xad,0x21,0x43,0x7c]
             stdcx. 2, 3, 4
 
+# CHECK-BE: stqcx. 2, 3, 4                  # encoding: [0x7c,0x43,0x21,0x6d]
+# CHECK-LE: stqcx. 2, 3, 4                  # encoding: [0x6d,0x21,0x43,0x7c]
+            stqcx. 2, 3, 4
+
 # CHECK-BE: stwat 2, 3, 28                  # encoding: [0x7c,0x43,0xe5,0x8c]
 # CHECK-LE: stwat 2, 3, 28                  # encoding: [0x8c,0xe5,0x43,0x7c]
             stwat 2, 3, 28
 
 # CHECK-BE: stdat 2, 3, 28                  # encoding: [0x7c,0x43,0xe5,0xcc]
 # CHECK-LE: stdat 2, 3, 28                  # encoding: [0xcc,0xe5,0x43,0x7c]
             stdat 2, 3, 28
 

 # CHECK-BE: lwarx 2, 3, 4                   # encoding: [0x7c,0x43,0x20,0x28]
 # CHECK-LE: lwarx 2, 3, 4                   # encoding: [0x28,0x20,0x43,0x7c]
             lwarx 2, 3, 4
 
 # CHECK-BE: ldarx 2, 3, 4                   # encoding: [0x7c,0x43,0x20,0xa8]
 # CHECK-LE: ldarx 2, 3, 4                   # encoding: [0xa8,0x20,0x43,0x7c]
             ldarx 2, 3, 4
 
+# CHECK-BE: lqarx 2, 3, 4                   # encoding: [0x7c,0x43,0x22,0x28]
+# CHECK-LE: lqarx 2, 3, 4                   # encoding: [0x28,0x22,0x43,0x7c]
+            lqarx 2, 3, 4
+
 # CHECK-BE: lbarx 2, 3, 4, 1                # encoding: [0x7c,0x43,0x20,0x69]
 # CHECK-LE: lbarx 2, 3, 4, 1                # encoding: [0x69,0x20,0x43,0x7c]
             lbarx 2, 3, 4, 1
 
 # CHECK-BE: lharx 2, 3, 4, 1                # encoding: [0x7c,0x43,0x20,0xe9]
 # CHECK-LE: lharx 2, 3, 4, 1                # encoding: [0xe9,0x20,0x43,0x7c]
             lharx 2, 3, 4, 1
 
 # CHECK-BE: lwarx 2, 3, 4, 1                # encoding: [0x7c,0x43,0x20,0x29]
 # CHECK-LE: lwarx 2, 3, 4, 1                # encoding: [0x29,0x20,0x43,0x7c]
             lwarx 2, 3, 4, 1
 
 # CHECK-BE: ldarx 2, 3, 4, 1                # encoding: [0x7c,0x43,0x20,0xa9]
 # CHECK-LE: ldarx 2, 3, 4, 1                # encoding: [0xa9,0x20,0x43,0x7c]
             ldarx 2, 3, 4, 1
 
+# CHECK-BE: lqarx 2, 3, 4, 1                # encoding: [0x7c,0x43,0x22,0x29]
+# CHECK-LE: lqarx 2, 3, 4, 1                # encoding: [0x29,0x22,0x43,0x7c]
+            lqarx 2, 3, 4, 1
+
 # CHECK-BE: lwat 2, 3, 28                   # encoding: [0x7c,0x43,0xe4,0x8c]
 # CHECK-LE: lwat 2, 3, 28                   # encoding: [0x8c,0xe4,0x43,0x7c]
             lwat 2, 3, 28
 
 # CHECK-BE: ldat 2, 3, 28                   # encoding: [0x7c,0x43,0xe4,0xcc]
 # CHECK-LE: ldat 2, 3, 28                   # encoding: [0xcc,0xe4,0x43,0x7c]
             ldat 2, 3, 28
 

llvm/test/MC/PowerPC/ppc64-encoding.s

 # CHECK-LE: ldu 2, 128(4)                   # encoding: [0x81,0x00,0x44,0xe8]
             ldu 2, 128(4)
 # CHECK-BE: ldux 2, 3, 4                    # encoding: [0x7c,0x43,0x20,0x6a]
 # CHECK-LE: ldux 2, 3, 4                    # encoding: [0x6a,0x20,0x43,0x7c]
             ldux 2, 3, 4
 # CHECK-BE: ldmx 2, 3, 4                    # encoding: [0x7c,0x43,0x22,0x6a]
 # CHECK-LE: ldmx 2, 3, 4                    # encoding: [0x6a,0x22,0x43,0x7c]
             ldmx 2, 3, 4
+# CHECK-BE: lq 2, 128(4)                    # encoding: [0xe0,0x44,0x00,0x80]
+# CHECK-LE: lq 2, 128(4)                    # encoding: [0x80,0x00,0x44,0xe0]
+            lq 2, 128(4)
 
 # Fixed-point store instructions
 
 # CHECK-BE: stb 2, 128(4)                   # encoding: [0x98,0x44,0x00,0x80]
 # CHECK-LE: stb 2, 128(4)                   # encoding: [0x80,0x00,0x44,0x98]
             stb 2, 128(4)
 # CHECK-BE: stbx 2, 3, 4                    # encoding: [0x7c,0x43,0x21,0xae]
 # CHECK-LE: stbx 2, 3, 4                    # encoding: [0xae,0x21,0x43,0x7c]

 # CHECK-LE: stdx 2, 3, 4                    # encoding: [0x2a,0x21,0x43,0x7c]
             stdx 2, 3, 4
 # CHECK-BE: stdu 2, 128(4)                  # encoding: [0xf8,0x44,0x00,0x81]
 # CHECK-LE: stdu 2, 128(4)                  # encoding: [0x81,0x00,0x44,0xf8]
             stdu 2, 128(4)
 # CHECK-BE: stdux 2, 3, 4                   # encoding: [0x7c,0x43,0x21,0x6a]
 # CHECK-LE: stdux 2, 3, 4                   # encoding: [0x6a,0x21,0x43,0x7c]
             stdux 2, 3, 4
+# CHECK-BE: stq 2, 128(4)                   # encoding: [0xf8,0x44,0x00,0x82]
+# CHECK-LE: stq 2, 128(4)                   # encoding: [0x82,0x00,0x44,0xf8]
+            stq 2, 128(4)
 
 # Fixed-point load and store with byte reversal instructions
 
 # CHECK-BE: lhbrx 2, 3, 4                   # encoding: [0x7c,0x43,0x26,0x2c]
 # CHECK-LE: lhbrx 2, 3, 4                   # encoding: [0x2c,0x26,0x43,0x7c]
             lhbrx 2, 3, 4
 # CHECK-BE: sthbrx 2, 3, 4                  # encoding: [0x7c,0x43,0x27,0x2c]
 # CHECK-LE: sthbrx 2, 3, 4                  # encoding: [0x2c,0x27,0x43,0x7c]