Power9 - Add exploitation of vector load and store that do not require swaps

[nemanjai]

Summary

The new lxvx/stxvx instructions do not require the swaps to line the elements up correctly. In order to select them over the lxvd2x/lxvw4x instructions which require swaps, the corresponding patterns are given a higher AddedComplexity.

[nemanjai]

lib/Target/PowerPC/InstPrinter/PPCInstPrinter.cpp

36

If there are objections to adding this, I'm happy to remove it. The reason I added it is that I find it frustrating that I can't specify FileCheck patterns that ensure that a VSX instruction's result is the correct operand of a VMX instruction and vice-versa.

I could also wrap this in an

#ifndef _NDEBUG

block, but then we would have to ensure that test cases that use it somehow require LLVM Asserts.

[wschmidt]

This looks fine to me. A couple of inline comments. I'll let someone else give final approval.

lib/Target/PowerPC/InstPrinter/PPCInstPrinter.cpp

36

I'm agnostic about that; however, you should remove the extra "l" from "respectively." :)

lib/Target/PowerPC/PPCInstrInfo.cpp

1012

Could use ternary operator here.

1137

Same comment.

[nemanjai]

lib/Target/PowerPC/InstPrinter/PPCInstPrinter.cpp

36

:) I think I need the little red squiggly line in Vim.

lib/Target/PowerPC/PPCInstrInfo.cpp

1012

Will do. I noticed that the if is quite extraneous here after I posted the patch.

[nemanjai]

lib/Target/PowerPC/PPCInstrVSX.td

2159

I'll add to the readme that patterns can be added to emit lxvd2x and friends in cases where we happen to want the elements in the reverse order (i.e. something like the vec_xl use as well as if the load is followed by a vector_shuffle that will reverse the elements).

[echristo]

Few inline comments.

Thanks!

-eric

lib/Target/PowerPC/PPCISelLowering.cpp

10527

This comment and the isISA3_0 don't quite match up. At least I'm assuming that there may be more isa3.0 processors other than power9? If not, then why the feature :)

It also seems like this could be factored in a different way since you replicate it a bunch of times.

lib/Target/PowerPC/PPCInstrVSX.td

2159

Eh? What's with the AddedComplexity here?

[nemanjai]

lib/Target/PowerPC/PPCISelLowering.cpp

10527

I'll change the comment to:
Not needed on CPUs that implement ISA 3.0 since we have a load that lines things up correctly.

I'll define a local var as follows and use it in conditions:
bool NeedsSwapsForVSXMemOps = Subtarget.hasVSX() && Subtarget.isLittleEndian() && !Subtarget.isISA3_0();

lib/Target/PowerPC/PPCInstrVSX.td

2159

I need these patterns to be preferred over the ones that use LXVD2X when we are on a CPU that implements ISA 3.0. This is the hack that we use in order to favour VSX instructions over other choices, so I just took it a step further to favour specific VSX instructions.

[amehsan]

test/CodeGen/PowerPC/swaps-le-1.ll

4

What is grep here (toward the end of the line)?

[nemanjai]

test/CodeGen/PowerPC/swaps-le-1.ll

4

Nice catch. I have no idea how the stray grep made it in there nor do I understand why it doesn't make the test case fail. In any case, I'll remove it.

[amehsan]

test/CodeGen/PowerPC/swaps-le-1.ll

4

Did you try this test case individually and it did not fail or you run all tests? I just came across this PR that might explain it if you run all tests and did not see an error:

https://llvm.org/bugs/show_bug.cgi?id=27652

[nemanjai]

Addressed the comments:

• Removed the odd "grep" from the test case (it appears I had added that to the patch file accidentally - it does cause failures)
• Used ternary operator where it makes sense
• Updated comments to refer to ISA 3.0 rather than P9 for selecting whether swaps are needed

[kbarton]

lib/Target/PowerPC/InstPrinter/PPCInstPrinter.cpp

445

Can't you use RegNum instead of Op.getReg() here?

[nemanjai]

lib/Target/PowerPC/InstPrinter/PPCInstPrinter.cpp

445

Yeah, that's a good point. When I defined RegNum, I guess I forgot to change this one. I'll change it on the commit if no further changes are requested.

[amehsan]

test/CodeGen/PowerPC/vsx-p9.ll

2

mtriple here?

[nemanjai]

test/CodeGen/PowerPC/vsx-p9.ll

2

Yeah, absolutely. Thanks. One of those things I meant to do and forgot - since I really wanted to add the BE testing as well (i.e. to confirm that the same load/store code is emitted).

[cycheng]

lib/Target/PowerPC/InstPrinter/PPCInstPrinter.cpp

36

It's good to have this option : )

test/CodeGen/PowerPC/swaps-le-1.ll

105

; CHECK-P9-NOT: xxswapd

Looks like it has been covered by "-implicit-check-not xxswapd"?
Or this is intentional?

test/CodeGen/PowerPC/vsx-p9.ll

104

I'm curious about that we don't add pattern match of i128 type for lxvs in PPCInstrVSX.td, but why it finally gets matched to lxvx?

test/CodeGen/PowerPC/vsx_shuffle_le.ll

81

So looks like use 'lxvd2x' can be more benefit?
If it is true, then, of course we don't have to handle it in current milestone, just a todo in future.

[nemanjai]

test/CodeGen/PowerPC/swaps-le-1.ll

105

Good point. I initially started by adding a bunch of CHECK-NOT's and then realized that adding the implicit one is easier.

test/CodeGen/PowerPC/vsx-p9.ll

104

The legalization code takes care of this as far as I remember.

test/CodeGen/PowerPC/vsx_shuffle_le.ll

81

Yes, I looked into this. I think this can actually be handled rather cleanly in the swap optimization pass. For situations where we actually want the value loaded to be permuted in the register, we can use the lxvd2x instruction.
However, such situations should be reasonably rare that we can defer this work as a "nice-to-have".

[nemanjai]

Addressed the minor comments that were posted on the last review.

[nemanjai]

This patch was functionally tested on the P9 simulator and the behaviour with these loads and stores is the same as the behaviour on a Power8 machine with the swapping loads and stores.

[echristo]

lib/Target/PowerPC/PPCISelLowering.cpp

10531

Would it make sense for this to be a subtarget feature given all of the load/store instruction changes for vectors? Though I guess you couldn't just use Subtarget.isISA3_0 anymore as the canonical "am I targeting something P9 or later".

[nemanjai]

lib/Target/PowerPC/PPCISelLowering.cpp

10531

I can certainly define something like HasNonPermutingMemOps in PPC.td and make it part of the Power9 processor definition. The only reason it hasn't been defined is that we kind of decided not to have too much granularity with Power9 since there are no optional features. But if you feel that it might be useful to be able to turn this off and turn on the rest of the Power9 instructions, I can certainly do that.

[echristo]

Couple more inline comments... I'm just coming up to speed on the new vector stuff so excuse any obvious questions :)

lib/Target/PowerPC/PPCISelLowering.cpp

10531

Actually, you should probably just make it a PPCSubtarget routine, don't worry about the feature aspect - as you say, we don't need to worry about separating it out.

That way you can use the simple query all over the place.

lib/Target/PowerPC/PPCInstrInfo.cpp

1009–1012

... including here I imagine.

lib/Target/PowerPC/PPCInstrVSX.td

2159

Can we just exclude the other patterns? We don't ever want them to show up do we?

[kbarton]

As Eric suggested, remove the old patterns to ensure LXVX/STXVX instructions are generated, instead of using AddedComplexity.

lib/Target/PowerPC/PPCInstrVSX.td

2159

I agree, it would be better to just use the LXVX and STXVX instructions when they are available.
So, we would need to remove the previous patterns for ISA3_0, but keep them for older ISAs.

[nemanjai]

lib/Target/PowerPC/PPCInstrVSX.td

2159

@kbarton @echristo
Can you just elaborate a bit on what you mean by removing the other patterns?
If I remove all the patterns that generate LXV[DW][24]X/STXV[WD][24]X, we have will likely have issues on Power8 (assuming that these patterns were needed). So perhaps what I can do is predicate all the patterns for the old instructions with IsPreISA3_0 (which would be !Subtarget.isISA3_0()). Maybe even needSwapsForVSXMemoryOps().Then I wouldn't need the AddedComplexity.

[nemanjai]

Removed the additional "AddedComplexity" clause to favour new loads/stores over the old ones.
Predicated the old loads and stores on previous ISA levels (prior to 3.0).
Provided a subtarget method for querying whether the target needs swaps for VSX memory operations.

[nemanjai]

lib/Target/PowerPC/PPCInstrInfo.cpp

1009–1012

I prefer to leave this query as-is because I don't think we would gain anything by emitting PPC::STXVD2X on a big endian Power9 either.

[kbarton]

OK, I think this is better.
However, I still find this confusing because there are several different checks being used here in various places: ISA3_0, P9Vector and hasVSX. If they can be used interchangeably, we should pick one and use it, instead of mixing them up. I'll add specific comments inline as examples.

[kbarton]

Regarding the testing, if I understand the patch correctly, we should also be able to do: -mcpu=pwr9 -mattr=-P9Vector, and get the same codegen as we would expect with -mcpu=pwr8 (i.e., generate the additional swap instructions). If that's true, can you please add these runsteps as well to ensure we get the right codegen?

lib/Target/PowerPC/PPCISelLowering.cpp

10527

Here the comments refer to ISA3_0. Does this imply P9Vector? What about hasVSX()?

lib/Target/PowerPC/PPCInstrInfo.cpp

1009–1012

Here we check for ISA3_0. Again, does this imply P9Vector()? Are the LXVX and STXVX instructions guarded by ISA3_0 or P9Vector?

lib/Target/PowerPC/PPCInstrVSX.td

89

Here we're checking for P9Vector only, no ISA3_0.

lib/Target/PowerPC/PPCSubtarget.h

279 ↗

(On Diff #63797)

Here we're checking hasVSX() and ISA3_0, but no mention of P9Vector.

[kbarton]

Sorry for another revision, but I think we need to cleanup the way different pieces of this are guarded. Plus, we should add some additional testing to cover cases with P9 but no LXVX/STXVX instructions.

[nemanjai]

In D19825#483165, @kbarton wrote:

Sorry for another revision, but I think we need to cleanup the way different pieces of this are guarded. Plus, we should add some additional testing to cover cases with P9 but no LXVX/STXVX instructions.

I'm happy to publish another review, that's fine. Particularly because you've definitely pointed out a bug - as it is now, we'd try to use LXVX/STXVX to reload/spill even with -mcpu=pwr9 -mattr=-power9-vector which would probably result in a crash.

lib/Target/PowerPC/PPCISelLowering.cpp

10527

If you don't mind, I'd like to keep the text in the comment as "ISA 3.0" because that is when the actual instructions were introduced. I prefer to keep general comments like this descriptive and the use of P9Vector is a kind of an implementation detail that isn't necessarily relevant.

Of course, if you feel strongly about it, I'll certainly change it.

lib/Target/PowerPC/PPCInstrInfo.cpp

1009–1012

This is a good point. You're absolutely right - we need to be checking P9Vector because that is the Predicate in the .td file. I'll change this.

1134–1137

Same here. I'll change this to P9Vector.

lib/Target/PowerPC/PPCInstrVSX.td

89

Yup. This is the right thing to do - it's the other uses that are wrong. We need to do it this way because the new instructions are guarded by P9Vector.

lib/Target/PowerPC/PPCSubtarget.h

279 ↗

(On Diff #63797)

Yup. This will change as well.

[nemanjai]

Fixed the way we query for whether the new instructions are available (to match the predicate in the .td file).
Added run lines for combination -mcpu=pwr9 -mattr=-power9-vector to the test cases.

[kbarton]

lib/Target/PowerPC/PPCInstrVSX.td

2157–2176

Yes, I don't think you can remove them altogether, but predicate them so they only appear on preISA3.
Ideally you would want an inequality, but I'm not sure if that is possible. If you use !Subtarget.isISA3_0, then you need to make sure to include isISA3_0 in all future subtargets as well.
As you said, that should eliminate the need for the AddedComplexity, which would be good.

[kbarton]

LGTM

[nemanjai]

Committed revision 282143.

lib/Target/PowerPC/InstPrinter/PPCInstPrinter.cpp

 #define DEBUG_TYPE "asm-printer"
 
 // FIXME: Once the integrated assembler supports full register names, tie this
 // to the verbose-asm setting.
 static cl::opt<bool>
 FullRegNames("ppc-asm-full-reg-names", cl::Hidden, cl::init(false),
              cl::desc("Use full register names when printing assembly"));
 
+// Useful for testing purposes. Prints vs{31-63} as v{0-31} respectivelly.
+static cl::opt<bool>
+ShowVSRNumsAsVR("ppc-vsr-nums-as-vr", cl::Hidden, cl::init(false),
+             cl::desc("Prints full register names with vs{31-63} as v{0-31}"));
+
 #define PRINT_ALIAS_INSTR
 #include "PPCGenAsmWriter.inc"
 
 void PPCInstPrinter::printRegName(raw_ostream &OS, unsigned RegNo) const {
   const char *RegName = getRegisterName(RegNo);
   if (RegName[0] == 'q' /* QPX */) {
     // The system toolchain on the BG/Q does not understand QPX register names
     // in .cfi_* directives, so print the name of the floating-point
static const char *stripRegisterPrefix(const char *RegName) {
   return RegName;
 }
 
 void PPCInstPrinter::printOperand(const MCInst *MI, unsigned OpNo,
                                   raw_ostream &O) {
   const MCOperand &Op = MI->getOperand(OpNo);
   if (Op.isReg()) {
     const char *RegName = getRegisterName(Op.getReg());
+    if (ShowVSRNumsAsVR) {
+      unsigned RegNum = Op.getReg();
+      if (RegNum >= PPC::VSH0 && RegNum <= PPC::VSH31)
+        O << 'v' << Op.getReg() - PPC::VSH0;
+      else
+        O << RegName;
+      return;
+    }
     // The linux and AIX assembler does not take register prefixes.
     if (!isDarwinSyntax())
       RegName = stripRegisterPrefix(RegName);
     
     O << RegName;
     return;
   }
   

lib/Target/PowerPC/PPCISelLowering.cpp

if (cast<StoreSDNode>(N)->isUnindexed() &&
       };
       return
         DAG.getMemIntrinsicNode(PPCISD::STBRX, dl, DAG.getVTList(MVT::Other),
                                 Ops, cast<StoreSDNode>(N)->getMemoryVT(),
                                 cast<StoreSDNode>(N)->getMemOperand());
     }
 
     // For little endian, VSX stores require generating xxswapd/lxvd2x.
+    // Not needed on P9 since we have a load that lines things up correctly.
     EVT VT = N->getOperand(1).getValueType();
     if (VT.isSimple()) {
       MVT StoreVT = VT.getSimpleVT();
       if (Subtarget.hasVSX() && Subtarget.isLittleEndian() &&
           (StoreVT == MVT::v2f64 || StoreVT == MVT::v2i64 ||
-           StoreVT == MVT::v4f32 || StoreVT == MVT::v4i32))
+           StoreVT == MVT::v4f32 || StoreVT == MVT::v4i32) &&
+          !Subtarget.isISA3_0())
         return expandVSXStoreForLE(N, DCI);
     }
     break;
   }
   case ISD::LOAD: {
     LoadSDNode *LD = cast<LoadSDNode>(N);
     EVT VT = LD->getValueType(0);
 
     // For little endian, VSX loads require generating lxvd2x/xxswapd.
+    // Not needed on P9 since we have a load that lines things up correctly.
     if (VT.isSimple()) {
       MVT LoadVT = VT.getSimpleVT();
       if (Subtarget.hasVSX() && Subtarget.isLittleEndian() &&
           (LoadVT == MVT::v2f64 || LoadVT == MVT::v2i64 ||
-           LoadVT == MVT::v4f32 || LoadVT == MVT::v4i32))
+           LoadVT == MVT::v4f32 || LoadVT == MVT::v4i32) &&
+          !Subtarget.isISA3_0())
         return expandVSXLoadForLE(N, DCI);
     }
 
     // We sometimes end up with a 64-bit integer load, from which we extract
     // two single-precision floating-point numbers. This happens with
     // std::complex<float>, and other similar structures, because of the way we
     // canonicalize structure copies. However, if we lack direct moves,
     // then the final bitcasts from the extracted integer values to the
SDValue PPCTargetLowering::PerformDAGCombine(SDNode *N,
   case ISD::INTRINSIC_W_CHAIN: {
     // For little endian, VSX loads require generating lxvd2x/xxswapd.
     if (Subtarget.hasVSX() && Subtarget.isLittleEndian()) {
       switch (cast<ConstantSDNode>(N->getOperand(1))->getZExtValue()) {
       default:
         break;
       case Intrinsic::ppc_vsx_lxvw4x:
       case Intrinsic::ppc_vsx_lxvd2x:
+        if (!Subtarget.isISA3_0())
           return expandVSXLoadForLE(N, DCI);
       }
     }
     break;
   }
   case ISD::INTRINSIC_VOID: {
     // For little endian, VSX stores require generating xxswapd/stxvd2x.
     if (Subtarget.hasVSX() && Subtarget.isLittleEndian()) {
       switch (cast<ConstantSDNode>(N->getOperand(1))->getZExtValue()) {
       default:
         break;
       case Intrinsic::ppc_vsx_stxvw4x:
       case Intrinsic::ppc_vsx_stxvd2x:
+        if (!Subtarget.isISA3_0())
           return expandVSXStoreForLE(N, DCI);
       }
     }
     break;
   }
   case ISD::BSWAP:
     // Turn BSWAP (LOAD) -> lhbrx/lwbrx.
     if (ISD::isNON_EXTLoad(N->getOperand(0).getNode()) &&
         N->getOperand(0).hasOneUse() &&

lib/Target/PowerPC/PPCInstrInfo.cpp

unsigned PPCInstrInfo::isLoadFromStackSlot(const MachineInstr *MI,
   case PPC::LD:
   case PPC::LWZ:
   case PPC::LFS:
   case PPC::LFD:
   case PPC::RESTORE_CR:
   case PPC::RESTORE_CRBIT:
   case PPC::LVX:
   case PPC::LXVD2X:
+  case PPC::LXVX:
   case PPC::QVLFDX:
   case PPC::QVLFSXs:
   case PPC::QVLFDXb:
   case PPC::RESTORE_VRSAVE:
     // Check for the operands added by addFrameReference (the immediate is the
     // offset which defaults to 0).
     if (MI->getOperand(1).isImm() && !MI->getOperand(1).getImm() &&
         MI->getOperand(2).isFI()) {
unsigned PPCInstrInfo::isStoreToStackSlot(const MachineInstr *MI,
   case PPC::STD:
   case PPC::STW:
   case PPC::STFS:
   case PPC::STFD:
   case PPC::SPILL_CR:
   case PPC::SPILL_CRBIT:
   case PPC::STVX:
   case PPC::STXVD2X:
+  case PPC::STXVX:
   case PPC::QVSTFDX:
   case PPC::QVSTFSXs:
   case PPC::QVSTFDXb:
   case PPC::SPILL_VRSAVE:
     // Check for the operands added by addFrameReference (the immediate is the
     // offset which defaults to 0).
     if (MI->getOperand(1).isImm() && !MI->getOperand(1).getImm() &&
         MI->getOperand(2).isFI()) {
if (PPC::GPRCRegClass.hasSubClassEq(RC) ||
     return true;
   } else if (PPC::VRRCRegClass.hasSubClassEq(RC)) {
     NewMIs.push_back(addFrameReference(BuildMI(MF, DL, get(PPC::STVX))
                                        .addReg(SrcReg,
                                                getKillRegState(isKill)),
                                        FrameIdx));
     NonRI = true;
   } else if (PPC::VSRCRegClass.hasSubClassEq(RC)) {
-    NewMIs.push_back(addFrameReference(BuildMI(MF, DL, get(PPC::STXVD2X))
+    unsigned Op = PPC::STXVD2X;
+    if (Subtarget.isISA3_0())
+      Op = PPC::STXVX;
+    NewMIs.push_back(addFrameReference(BuildMI(MF, DL, get(Op))
                                        .addReg(SrcReg,
                                                getKillRegState(isKill)),
                                        FrameIdx));
     NonRI = true;
   } else if (PPC::VSFRCRegClass.hasSubClassEq(RC)) {
     NewMIs.push_back(addFrameReference(BuildMI(MF, DL, get(PPC::STXSDX))
                                        .addReg(SrcReg,
                                                getKillRegState(isKill)),
NewMIs.push_back(addFrameReference(BuildMI(MF, DL,
                                                get(PPC::RESTORE_CRBIT), DestReg),
                                        FrameIdx));
     return true;
   } else if (PPC::VRRCRegClass.hasSubClassEq(RC)) {
     NewMIs.push_back(addFrameReference(BuildMI(MF, DL, get(PPC::LVX), DestReg),
                                        FrameIdx));
     NonRI = true;
   } else if (PPC::VSRCRegClass.hasSubClassEq(RC)) {
-    NewMIs.push_back(addFrameReference(BuildMI(MF, DL, get(PPC::LXVD2X), DestReg),
+    unsigned Op = PPC::LXVD2X;
+    if (Subtarget.isISA3_0())
+      Op = PPC::LXVX;
+    NewMIs.push_back(addFrameReference(BuildMI(MF, DL, get(Op), DestReg),
                                        FrameIdx));
     NonRI = true;
   } else if (PPC::VSFRCRegClass.hasSubClassEq(RC)) {
     NewMIs.push_back(addFrameReference(BuildMI(MF, DL, get(PPC::LXSDX), DestReg),
                                        FrameIdx));
     NonRI = true;
   } else if (PPC::VSSRCRegClass.hasSubClassEq(RC)) {
     NewMIs.push_back(addFrameReference(BuildMI(MF, DL, get(PPC::LXSSPX), DestReg),

lib/Target/PowerPC/PPCInstrVSX.td

def o    : XX3Form_Rc<opcode, xo, (outs vsrc:$XT), (ins vsrc:$XA, vsrc:$XB),
                                 (InTy (PPCvcmp_o InTy:$XA, InTy:$XB, xo)))]>,
                        isDOT;
   }
 }
 
 def HasVSX : Predicate<"PPCSubTarget->hasVSX()">;
 def IsLittleEndian : Predicate<"PPCSubTarget->isLittleEndian()">;
 def IsBigEndian : Predicate<"!PPCSubTarget->isLittleEndian()">;
 
 let Predicates = [HasVSX] in {
 let AddedComplexity = 400 in { // Prefer VSX patterns over non-VSX patterns.
 let hasSideEffects = 0 in { // VSX instructions don't have side effects.
 let Uses = [RM] in {
 
   // Load indexed instructions
   let mayLoad = 1 in {
     def LXSDX : XX1Form<31, 588,
let Predicates = [HasP9Vector] in {
   def LXSIBZX : X_XT6_RA5_RB5<31, 781, "lxsibzx", vsfrc, []>;
   def LXSIHZX : X_XT6_RA5_RB5<31, 813, "lxsihzx", vsfrc, []>;
 
   // Load Vector Halfword*8/Byte*16 Indexed
   def LXVH8X  : X_XT6_RA5_RB5<31, 812, "lxvh8x" , vsrc, []>;
   def LXVB16X : X_XT6_RA5_RB5<31, 876, "lxvb16x", vsrc, []>;
 
   // Load Vector Indexed
-  def LXVX    : X_XT6_RA5_RB5<31, 268, "lxvx"   , vsrc, []>;
+  def LXVX    : X_XT6_RA5_RB5<31, 268, "lxvx"   , vsrc,
+                [(set v2f64:$XT, (load xoaddr:$src))]>;
 
   // Load Vector (Left-justified) with Length
   def LXVL    : X_XT6_RA5_RB5<31, 269, "lxvl"   , vsrc, []>;
   def LXVLL   : X_XT6_RA5_RB5<31, 301, "lxvll"  , vsrc, []>;
 
   // Load Vector Word & Splat Indexed
   def LXVWSX  : X_XT6_RA5_RB5<31, 364, "lxvwsx" , vsrc, []>;
   } // end mayLoad
let Predicates = [HasP9Vector] in {
   def STXSIBX  : X_XS6_RA5_RB5<31,  909, "stxsibx" , vsfrc, []>;
   def STXSIHX  : X_XS6_RA5_RB5<31,  941, "stxsihx" , vsfrc, []>;
 
   // Store Vector Halfword*8/Byte*16 Indexed
   def STXVH8X  : X_XS6_RA5_RB5<31,  940, "stxvh8x" , vsrc, []>;
   def STXVB16X : X_XS6_RA5_RB5<31, 1004, "stxvb16x", vsrc, []>;
 
   // Store Vector Indexed
-  def STXVX    : X_XS6_RA5_RB5<31,  396, "stxvx"   , vsrc, []>;
+  def STXVX    : X_XS6_RA5_RB5<31,  396, "stxvx"   , vsrc,
+                 [(store v2f64:$XT, xoaddr:$dst)]>;
 
   // Store Vector (Left-justified) with Length
   def STXVL    : X_XS6_RA5_RB5<31,  397, "stxvl"   , vsrc, []>;
   def STXVLL   : X_XS6_RA5_RB5<31,  429, "stxvll"  , vsrc, []>;
   } // end mayStore
+
+  let AddedComplexity = 500 in {
+    def : Pat<(v2f64 (load xoaddr:$src)), (LXVX xoaddr:$src)>;
+    def : Pat<(v2i64 (load xoaddr:$src)), (LXVX xoaddr:$src)>;
+    def : Pat<(v4f32 (load xoaddr:$src)), (LXVX xoaddr:$src)>;
+    def : Pat<(v4i32 (load xoaddr:$src)), (LXVX xoaddr:$src)>;
+    def : Pat<(v4i32 (int_ppc_vsx_lxvw4x xoaddr:$src)), (LXVX xoaddr:$src)>;
+    def : Pat<(v2f64 (int_ppc_vsx_lxvd2x xoaddr:$src)), (LXVX xoaddr:$src)>;
+    def : Pat<(store v2f64:$rS, xoaddr:$dst), (STXVX $rS, xoaddr:$dst)>;
+    def : Pat<(store v2i64:$rS, xoaddr:$dst), (STXVX $rS, xoaddr:$dst)>;
+    def : Pat<(store v4f32:$rS, xoaddr:$dst), (STXVX $rS, xoaddr:$dst)>;
+    def : Pat<(store v4i32:$rS, xoaddr:$dst), (STXVX $rS, xoaddr:$dst)>;
+    def : Pat<(int_ppc_vsx_stxvw4x v4i32:$rS, xoaddr:$dst),
+              (STXVX $rS, xoaddr:$dst)>;
+    def : Pat<(int_ppc_vsx_stxvd2x v2f64:$rS, xoaddr:$dst),
+              (STXVX $rS, xoaddr:$dst)>;
+  }
+
 } // end HasP9Vector

test/CodeGen/PowerPC/lxvw4x-bug.ll

 ; RUN: llc -O0 -mcpu=pwr8 -mtriple=powerpc64le-unknown-unknown < %s | FileCheck %s
+; RUN: llc -O0 -mcpu=pwr9 -mtriple=powerpc64le-unknown-unknown < %s \
+; RUN:   | FileCheck %s --check-prefix=CHECK-P9 --implicit-check-not xxswapd
 ; Function Attrs: nounwind
 define void @test() {
 entry:
   %__a.addr.i = alloca i32, align 4
   %__b.addr.i = alloca <4 x i32>*, align 8
   %i = alloca <4 x i32>, align 16
   %j = alloca <4 x i32>, align 16
   store <4 x i32> <i32 1, i32 2, i32 3, i32 4>, <4 x i32>* %i, align 16
   store i32 0, i32* %__a.addr.i, align 4
   store <4 x i32>* %i, <4 x i32>** %__b.addr.i, align 8
   %0 = load i32, i32* %__a.addr.i, align 4
   %1 = load <4 x i32>*, <4 x i32>** %__b.addr.i, align 8
   %2 = bitcast <4 x i32>* %1 to i8*
   %3 = getelementptr i8, i8* %2, i32 %0
   %4 = call <4 x i32> @llvm.ppc.vsx.lxvw4x(i8* %3)
 ; CHECK: lwa [[REG0:[0-9]+]],
 ; CHECK: lxvd2x [[REG1:[0-9]+]], {{[0-9]+}}, [[REG0]]
 ; CHECK: xxswapd [[REG1]], [[REG1]]
+; CHECK-P9: lwa [[REG0:[0-9]+]],
+; CHECK-P9: lxvx [[REG1:[0-9]+]], {{[0-9]+}}, [[REG0]]
   store <4 x i32> %4, <4 x i32>* %j, align 16
   ret void
 }
 
 ; Function Attrs: nounwind readonly
 declare <4 x i32> @llvm.ppc.vsx.lxvw4x(i8*)

test/CodeGen/PowerPC/ppc64-i128-abi.ll

 ; RUN: llc -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr8 < %s | FileCheck %s -check-prefix=CHECK-LE
 ; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr8 < %s | FileCheck %s -check-prefix=CHECK-BE
 ; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr8 -mattr=-vsx < %s | FileCheck %s -check-prefix=CHECK-NOVSX
 ; RUN: llc -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr8 -mattr=-vsx < %s | FileCheck %s -check-prefix=CHECK-NOVSX
 ; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr8 -mattr=-vsx < %s | FileCheck %s -check-prefix=CHECK-BE-NOVSX
 ; RUN: llc -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr8 -mattr=-vsx < %s | FileCheck %s -check-prefix=CHECK-LE-NOVSX
+; RUN: llc -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr9 < %s \
+; RUN:   | FileCheck %s -check-prefix=CHECK-P9 --implicit-check-not xxswapd
+; RUN: llc -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr9 -mattr=-vsx < %s \
+; RUN:   | FileCheck %s -check-prefix=CHECK-NOVSX --implicit-check-not xxswapd
 
 @x = common global <1 x i128> zeroinitializer, align 16
 @y = common global <1 x i128> zeroinitializer, align 16
 @a = common global i128 zeroinitializer, align 16
 @b = common global i128 zeroinitializer, align 16
 
 ; VSX:
 ;   %a is passed in register 34

 ;        should be a better way of doing this.
 
 ; CHECK-LE-LABEL: @v1i128_increment_by_one
 ; CHECK-LE: lxvd2x [[VAL:[0-9]+]], {{[0-9]+}}, {{[0-9]+}}
 ; CHECK-LE: xxswapd 35, [[VAL]]
 ; CHECK-LE: vadduqm 2, 2, 3
 ; CHECK-LE: blr
 
+; CHECK-P9-LABEL: @v1i128_increment_by_one
+; CHECK-P9: lxvx
+; CHECK-P9: vadduqm 2, 2, 3
+; CHECK-P9: blr
+
 ; CHECK-BE-LABEL: @v1i128_increment_by_one
 ; CHECK-BE: lxvd2x 35, {{[0-9]+}}, {{[0-9]+}}
 ; CHECK-BE-NOT: xxswapd 
 ; CHECK-BE: vadduqm 2, 2, 3 
 ; CHECK-BE-NOT: xxswapd 34, {{[0-9]+}}
 ; CHECK-BE: blr
 
 ; CHECK-NOVSX-LABEL: @v1i128_increment_by_one
define <1 x i128> @call_v1i128_increment_by_one() nounwind {
        ret <1 x i128> %ret
 
 ; CHECK-LE-LABEL: @call_v1i128_increment_by_one
 ; CHECK-LE: lxvd2x [[PARAM:[0-9]+]], {{[0-9]+}}, {{[0-9]+}}
 ; CHECK-LE: xxswapd 34, [[PARAM]]
 ; CHECK-LE: bl v1i128_increment_by_one
 ; CHECK-LE: blr
 
+; CHECK-P9-LABEL: @call_v1i128_increment_by_one
+; CHECK-P9: lxvx
+; CHECK-P9: bl v1i128_increment_by_one
+; CHECK-P9: blr
+
 ; CHECK-BE-LABEL: @call_v1i128_increment_by_one
 ; CHECK-BE: lxvw4x 34, {{[0-9]+}}, {{[0-9]+}}
 ; CHECK-BE-NOT: xxswapd 34, {{[0-9]+}}
 ; CHECK-BE: bl v1i128_increment_by_one
 ; CHECK-BE: blr
 
 ; CHECK-NOVSX-LABEL: @call_v1i128_increment_by_one
 ; CHECK-NOVSX: lvx 2, {{[0-9]+}}, {{[0-9]+}}

 ; CHECK-LE-LABEL: @call_v1i128_increment_by_val
 ; CHECK-LE: lxvd2x [[PARAM1:[0-9]+]], {{[0-9]+}}, {{[0-9]+}}
 ; CHECK-LE: lxvd2x [[PARAM2:[0-9]+]], {{[0-9]+}}, {{[0-9]+}}
 ; CHECK-LE-DAG: xxswapd 34, [[PARAM1]]
 ; CHECK-LE-DAG: xxswapd 35, [[PARAM2]]
 ; CHECK-LE: bl v1i128_increment_by_val
 ; CHECK-LE: blr
 
+; CHECK-P9-LABEL: @call_v1i128_increment_by_val
+; CHECK-P9-DAG: lxvx 34
+; CHECK-P9-DAG: lxvx 35
+; CHECK-P9: bl v1i128_increment_by_val
+; CHECK-P9: blr
+
 ; CHECK-BE-LABEL: @call_v1i128_increment_by_val
 
 
 ; CHECK-BE-DAG: lxvw4x 35, {{[0-9]+}}, {{[0-9]+}}
 ; CHECK-BE-NOT: xxswapd 34, {{[0-9]+}}
 ; CHECK-BE-NOT: xxswapd 35, {{[0-9]+}}
 ; CHECK-BE: bl v1i128_increment_by_val
 ; CHECK-BE: blr

test/CodeGen/PowerPC/swaps-le-1.ll

 ; RUN: llc -O3 -mcpu=pwr8 -mtriple=powerpc64le-unknown-linux-gnu < %s | FileCheck %s
 ; RUN: llc -O3 -mcpu=pwr8 -disable-ppc-vsx-swap-removal -mtriple=powerpc64le-unknown-linux-gnu < %s | FileCheck -check-prefix=NOOPTSWAP %s
+; RUN: llc -O3 -mcpu=pwr9 -mtriple=powerpc64le-unknown-linux-gnu \
+; RUN:  -ppc-vsr-nums-as-vr < %s | FileCheck -check-prefix=CHECK-P9 grep \
+; RUN:  --implicit-check-not xxswapd %s
 
 ; This test was generated from the following source:
 ;
 ; #define N 4096
 ; int ca[N] __attribute__((aligned(16)));
 ; int cb[N] __attribute__((aligned(16)));
 ; int cc[N] __attribute__((aligned(16)));
 ; int cd[N] __attribute__((aligned(16)));

 
 for.end:
   ret void
 }
 
 ; CHECK-LABEL: @foo
 ; CHECK-NOT: xxpermdi
 ; CHECK-NOT: xxswapd
+; CHECK-P9-NOT: xxpermdi
+; CHECK-P9-NOT: xxswapd
 
 ; CHECK: lxvd2x
 ; CHECK: lxvd2x
 ; CHECK-DAG: lxvd2x
 ; CHECK-DAG: vadduwm
 ; CHECK: vmuluwm
 ; CHECK: stxvd2x
 

 
 ; CHECK: lxvd2x
 ; CHECK: lxvd2x
 ; CHECK-DAG: lxvd2x
 ; CHECK-DAG: vadduwm
 ; CHECK: vmuluwm
 ; CHECK: stxvd2x
 
-
 ; NOOPTSWAP-LABEL: @foo
 
 ; NOOPTSWAP: lxvd2x
 ; NOOPTSWAP-DAG: lxvd2x
 ; NOOPTSWAP-DAG: lxvd2x
 ; NOOPTSWAP-DAG: xxswapd
 ; NOOPTSWAP-DAG: xxswapd
 ; NOOPTSWAP-DAG: xxswapd
 ; NOOPTSWAP-DAG: vadduwm
 ; NOOPTSWAP: vmuluwm
 ; NOOPTSWAP: xxswapd
 ; NOOPTSWAP-DAG: xxswapd
 ; NOOPTSWAP-DAG: xxswapd
 ; NOOPTSWAP-DAG: stxvd2x
 ; NOOPTSWAP-DAG: stxvd2x
 ; NOOPTSWAP: stxvd2x
 
+; CHECK-P9-LABEL: @foo
+; CHECK-P9: lxvx [[R1:v[0-9]+]],
+; CHECK-P9: lxvx [[R2:v[0-9]+]],
+; CHECK-P9: lxvx [[R3:v[0-9]+]],
+; CHECK-P9: lxvx [[R4:v[0-9]+]],
+; CHECK-P9: lxvx [[R5:v[0-9]+]],
+; CHECK-P9: lxvx [[R6:v[0-9]+]],
+; CHECK-P9: lxvx [[R7:v[0-9]+]],
+; CHECK-P9: lxvx [[R8:v[0-9]+]],
+; CHECK-P9-DAG: vadduwm v{{[0-9]+}}, [[R2]], [[R1]]
+; CHECK-P9: lxvx [[R9:v[0-9]+]],
+; CHECK-P9: lxvx [[R10:v[0-9]+]],
+; CHECK-P9-DAG: vadduwm v{{[0-9]+}}, [[R4]], [[R3]]
+; CHECK-P9: lxvx [[R11:v[0-9]+]],
+; CHECK-P9: lxvx [[R12:v[0-9]+]],
+; CHECK-P9-DAG: vadduwm v{{[0-9]+}}, [[R6]], [[R5]]
+; CHECK-P9-DAG: vadduwm v{{[0-9]+}}, [[R8]], [[R7]]
+; CHECK-P9-DAG: vmuluwm v{{[0-9]+}}, v{{[0-9]+}}, [[R9]]
+; CHECK-P9-DAG: vmuluwm v{{[0-9]+}}, v{{[0-9]+}}, [[R10]]
+; CHECK-P9-DAG: vmuluwm v{{[0-9]+}}, v{{[0-9]+}}, [[R11]]
+; CHECK-P9-DAG: vmuluwm v{{[0-9]+}}, v{{[0-9]+}}, [[R12]]
+; CHECK-P9-DAG: stxvx
+; CHECK-P9: stxvx
+; CHECK-P9: stxvx
+; CHECK-P9: stxvx
+

test/CodeGen/PowerPC/swaps-le-6.ll

 ; RUN: llc -mcpu=pwr8 -mtriple=powerpc64le-unknown-linux-gnu -O3 < %s | FileCheck %s
+; RUN: llc -mcpu=pwr9 -mtriple=powerpc64le-unknown-linux-gnu -O3 < %s \
+; RUN:   | FileCheck %s --check-prefix=CHECK-P9 --implicit-check-not xxswapd
 
 ; These tests verify that VSX swap optimization works when loading a scalar
 ; into a vector register.
 
 
 @x = global <2 x double> <double 9.970000e+01, double -1.032220e+02>, align 16
 @z = global <2 x double> <double 2.332000e+01, double 3.111111e+01>, align 16
 @y = global double 1.780000e+00, align 8

 
 ; CHECK-LABEL: @bar0
 ; CHECK-DAG: lxvd2x [[REG1:[0-9]+]]
 ; CHECK-DAG: lxsdx [[REG2:[0-9]+]]
 ; CHECK: xxspltd [[REG4:[0-9]+]], [[REG2]], 0
 ; CHECK: xxpermdi [[REG5:[0-9]+]], [[REG4]], [[REG1]], 1
 ; CHECK: stxvd2x [[REG5]]
 
+; CHECK-P9-LABEL: @bar0
+; CHECK-P9-DAG: lxvx [[REG1:[0-9]+]]
+; CHECK-P9-DAG: lxsdx [[REG2:[0-9]+]]
+; CHECK-P9: xxspltd [[REG4:[0-9]+]], [[REG2]], 0
+; CHECK-P9: xxpermdi [[REG5:[0-9]+]], [[REG4]], [[REG1]], 1
+; CHECK-P9: stxvx [[REG5]]
+
 define void @bar1() {
 entry:
   %0 = load <2 x double>, <2 x double>* @x, align 16
   %1 = load double, double* @y, align 8
   %vecins = insertelement <2 x double> %0, double %1, i32 1
   store <2 x double> %vecins, <2 x double>* @z, align 16
   ret void
 }
 
 ; CHECK-LABEL: @bar1
 ; CHECK-DAG: lxvd2x [[REG1:[0-9]+]]
 ; CHECK-DAG: lxsdx [[REG2:[0-9]+]]
 ; CHECK: xxspltd [[REG4:[0-9]+]], [[REG2]], 0
 ; CHECK: xxmrghd [[REG5:[0-9]+]], [[REG1]], [[REG4]]
 ; CHECK: stxvd2x [[REG5]]
 
+; CHECK-P9-LABEL: @bar1
+; CHECK-P9-DAG: lxvx [[REG1:[0-9]+]]
+; CHECK-P9-DAG: lxsdx [[REG2:[0-9]+]]
+; CHECK-P9: xxspltd [[REG4:[0-9]+]], [[REG2]], 0
+; CHECK-P9: xxmrghd [[REG5:[0-9]+]], [[REG1]], [[REG4]]
+; CHECK-P9: stxvx [[REG5]]
+

test/CodeGen/PowerPC/vsx-ldst-builtin-le.ll

 ; RUN: llc -mcpu=pwr8 -mattr=+vsx -O2 -mtriple=powerpc64le-unknown-linux-gnu < %s > %t
 ; RUN: grep lxvd2x < %t | count 18
 ; RUN: grep stxvd2x < %t | count 18
 
+; RUN: llc -mcpu=pwr9 -O2 -mtriple=powerpc64le-unknown-linux-gnu < %s > %t
+; RUN: grep lxvx < %t | count 18
+; RUN: grep stxvx < %t | count 18
+
 @vf = global <4 x float> <float -1.500000e+00, float 2.500000e+00, float -3.500000e+00, float 4.500000e+00>, align 16
 @vd = global <2 x double> <double 3.500000e+00, double -7.500000e+00>, align 16
 @vsi = global <4 x i32> <i32 -1, i32 2, i32 -3, i32 4>, align 16
 @vui = global <4 x i32> <i32 0, i32 1, i32 2, i32 3>, align 16
 @vsll = global <2 x i64> <i64 255, i64 -937>, align 16
 @vull = global <2 x i64> <i64 1447, i64 2894>, align 16
 @res_vsi = common global <4 x i32> zeroinitializer, align 16
 @res_vui = common global <4 x i32> zeroinitializer, align 16

test/CodeGen/PowerPC/vsx-ldst.ll

 ; RUN: llc -mcpu=pwr8 -mattr=+vsx -O2 -mtriple=powerpc64-unknown-linux-gnu < %s > %t
 ; RUN: grep lxvw4x < %t | count 3
 ; RUN: grep lxvd2x < %t | count 3
 ; RUN: grep stxvw4x < %t | count 3
 ; RUN: grep stxvd2x < %t | count 3
 ; RUN: llc -mcpu=pwr8 -mattr=+vsx -O0 -fast-isel=1 -mtriple=powerpc64-unknown-linux-gnu < %s > %t
 ; RUN: grep lxvw4x < %t | count 3
 ; RUN: grep lxvd2x < %t | count 3
 ; RUN: grep stxvw4x < %t | count 3
 ; RUN: grep stxvd2x < %t | count 3
 
 ; RUN: llc -mcpu=pwr8 -mattr=+vsx -O2 -mtriple=powerpc64le-unknown-linux-gnu < %s > %t
 ; RUN: grep lxvd2x < %t | count 6
 ; RUN: grep stxvd2x < %t | count 6
 
+; RUN: llc -mcpu=pwr9 -O2 -mtriple=powerpc64le-unknown-linux-gnu < %s > %t
+; RUN: grep lxvx < %t | count 6
+; RUN: grep stxvx < %t | count 6
+
+
 @vsi = global <4 x i32> <i32 -1, i32 2, i32 -3, i32 4>, align 16
 @vui = global <4 x i32> <i32 0, i32 1, i32 2, i32 3>, align 16
 @vf = global <4 x float> <float -1.500000e+00, float 2.500000e+00, float -3.500000e+00, float 4.500000e+00>, align 16
 @vsll = global <2 x i64> <i64 255, i64 -937>, align 16
 @vull = global <2 x i64> <i64 1447, i64 2894>, align 16
 @vd = global <2 x double> <double 3.500000e+00, double -7.500000e+00>, align 16
 @res_vsi = common global <4 x i32> zeroinitializer, align 16
 @res_vui = common global <4 x i32> zeroinitializer, align 16

test/CodeGen/PowerPC/vsx-p9.ll

+; RUN: llc -mcpu=pwr9 < %s | FileCheck %s
+
+@uca = global <16 x i8> zeroinitializer, align 16
+@ucb = global <16 x i8> zeroinitializer, align 16
+@sca = global <16 x i8> zeroinitializer, align 16
+@scb = global <16 x i8> zeroinitializer, align 16
+@usa = global <8 x i16> zeroinitializer, align 16
+@usb = global <8 x i16> zeroinitializer, align 16
+@ssa = global <8 x i16> zeroinitializer, align 16
+@ssb = global <8 x i16> zeroinitializer, align 16
+@uia = global <4 x i32> zeroinitializer, align 16
+@uib = global <4 x i32> zeroinitializer, align 16
+@sia = global <4 x i32> zeroinitializer, align 16
+@sib = global <4 x i32> zeroinitializer, align 16
+@ulla = global <2 x i64> zeroinitializer, align 16
+@ullb = global <2 x i64> zeroinitializer, align 16
+@slla = global <2 x i64> zeroinitializer, align 16
+@sllb = global <2 x i64> zeroinitializer, align 16
+@uxa = global <1 x i128> zeroinitializer, align 16
+@uxb = global <1 x i128> zeroinitializer, align 16
+@sxa = global <1 x i128> zeroinitializer, align 16
+@sxb = global <1 x i128> zeroinitializer, align 16
+@vfa = global <4 x float> zeroinitializer, align 16
+@vfb = global <4 x float> zeroinitializer, align 16
+@vda = global <2 x double> zeroinitializer, align 16
+@vdb = global <2 x double> zeroinitializer, align 16
+
+define void @_Z4testv() {
+entry:
+; CHECK-LABEL: @_Z4testv
+  %0 = load <16 x i8>, <16 x i8>* @uca, align 16
+  %1 = load <16 x i8>, <16 x i8>* @ucb, align 16
+  %add.i = add <16 x i8> %1, %0
+  tail call void (...) @sink(<16 x i8> %add.i)
+; CHECK: lxvx 34, 0, 3
+; CHECK: lxvx 35, 0, 4
+; CHECK: vaddubm 2, 3, 2
+; CHECK: stxvx 34,
+; CHECK: bl sink
+  %2 = load <16 x i8>, <16 x i8>* @sca, align 16
+  %3 = load <16 x i8>, <16 x i8>* @scb, align 16
+  %add.i22 = add <16 x i8> %3, %2
+  tail call void (...) @sink(<16 x i8> %add.i22)
+; CHECK: lxvx 34, 0, 3
+; CHECK: lxvx 35, 0, 4
+; CHECK: vaddubm 2, 3, 2
+; CHECK: stxvx 34,
+; CHECK: bl sink
+  %4 = load <8 x i16>, <8 x i16>* @usa, align 16
+  %5 = load <8 x i16>, <8 x i16>* @usb, align 16
+  %add.i21 = add <8 x i16> %5, %4
+  tail call void (...) @sink(<8 x i16> %add.i21)
+; CHECK: lxvx 34, 0, 3
+; CHECK: lxvx 35, 0, 4
+; CHECK: vadduhm 2, 3, 2
+; CHECK: stxvx 34,
+; CHECK: bl sink
+  %6 = load <8 x i16>, <8 x i16>* @ssa, align 16
+  %7 = load <8 x i16>, <8 x i16>* @ssb, align 16
+  %add.i20 = add <8 x i16> %7, %6
+  tail call void (...) @sink(<8 x i16> %add.i20)
+; CHECK: lxvx 34, 0, 3
+; CHECK: lxvx 35, 0, 4
+; CHECK: vadduhm 2, 3, 2
+; CHECK: stxvx 34,
+; CHECK: bl sink
+  %8 = load <4 x i32>, <4 x i32>* @uia, align 16
+  %9 = load <4 x i32>, <4 x i32>* @uib, align 16
+  %add.i19 = add <4 x i32> %9, %8
+  tail call void (...) @sink(<4 x i32> %add.i19)
+; CHECK: lxvx 34, 0, 3
+; CHECK: lxvx 35, 0, 4
+; CHECK: vadduwm 2, 3, 2
+; CHECK: stxvx 34,
+; CHECK: bl sink
+  %10 = load <4 x i32>, <4 x i32>* @sia, align 16
+  %11 = load <4 x i32>, <4 x i32>* @sib, align 16
+  %add.i18 = add <4 x i32> %11, %10
+  tail call void (...) @sink(<4 x i32> %add.i18)
+; CHECK: lxvx 34, 0, 3
+; CHECK: lxvx 35, 0, 4
+; CHECK: vadduwm 2, 3, 2
+; CHECK: stxvx 34,
+; CHECK: bl sink
+  %12 = load <2 x i64>, <2 x i64>* @ulla, align 16
+  %13 = load <2 x i64>, <2 x i64>* @ullb, align 16
+  %add.i17 = add <2 x i64> %13, %12
+  tail call void (...) @sink(<2 x i64> %add.i17)
+; CHECK: lxvx 34, 0, 3
+; CHECK: lxvx 35, 0, 4
+; CHECK: vaddudm 2, 3, 2
+; CHECK: stxvx 34,
+; CHECK: bl sink
+  %14 = load <2 x i64>, <2 x i64>* @slla, align 16
+  %15 = load <2 x i64>, <2 x i64>* @sllb, align 16
+  %add.i16 = add <2 x i64> %15, %14
+  tail call void (...) @sink(<2 x i64> %add.i16)
+; CHECK: lxvx 34, 0, 3
+; CHECK: lxvx 35, 0, 4
+; CHECK: vaddudm 2, 3, 2
+; CHECK: stxvx 34,
+; CHECK: bl sink
+  %16 = load <1 x i128>, <1 x i128>* @uxa, align 16
+  %17 = load <1 x i128>, <1 x i128>* @uxb, align 16
+  %add.i15 = add <1 x i128> %17, %16
+  tail call void (...) @sink(<1 x i128> %add.i15)
+; CHECK: lxvx 34, 0, 3
+; CHECK: lxvx 35, 0, 4
+; CHECK: vadduqm 2, 3, 2
+; CHECK: stxvx 34,
+; CHECK: bl sink
+  %18 = load <1 x i128>, <1 x i128>* @sxa, align 16
+  %19 = load <1 x i128>, <1 x i128>* @sxb, align 16
+  %add.i14 = add <1 x i128> %19, %18
+  tail call void (...) @sink(<1 x i128> %add.i14)
+; CHECK: lxvx 34, 0, 3
+; CHECK: lxvx 35, 0, 4
+; CHECK: vadduqm 2, 3, 2
+; CHECK: stxvx 34,
+; CHECK: bl sink
+  %20 = load <4 x float>, <4 x float>* @vfa, align 16
+  %21 = load <4 x float>, <4 x float>* @vfb, align 16
+  %add.i13 = fadd <4 x float> %20, %21
+  tail call void (...) @sink(<4 x float> %add.i13)
+; CHECK: lxvx 0, 0, 3
+; CHECK: lxvx 1, 0, 4
+; CHECK: xvaddsp 34, 0, 1
+; CHECK: stxvx 34,
+; CHECK: bl sink
+  %22 = load <2 x double>, <2 x double>* @vda, align 16
+  %23 = load <2 x double>, <2 x double>* @vdb, align 16
+  %add.i12 = fadd <2 x double> %22, %23
+  tail call void (...) @sink(<2 x double> %add.i12)
+; CHECK: lxvx 0, 0, 3
+; CHECK: lxvx 1, 0, 4
+; CHECK: xvadddp 0, 0, 1
+; CHECK: stxvx 0,
+; CHECK: bl sink
+  ret void
+}
+
+declare void @sink(...)

test/CodeGen/PowerPC/vsx_insert_extract_le.ll

 ; RUN: llc -mcpu=pwr8 -mattr=+vsx -mtriple=powerpc64le-unknown-linux-gnu < %s | FileCheck %s
+; RUN: llc -mcpu=pwr9 -mtriple=powerpc64le-unknown-linux-gnu < %s \
+; RUN:   | FileCheck %s --check-prefix=CHECK-P9 --implicit-check-not xxswapd
 
 define <2 x double> @testi0(<2 x double>* %p1, double* %p2) {
   %v = load <2 x double>, <2 x double>* %p1
   %s = load double, double* %p2
   %r = insertelement <2 x double> %v, double %s, i32 0
   ret <2 x double> %r
 
 ; CHECK-LABEL: testi0
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: lxsdx 1, 0, 4
 ; CHECK: xxswapd 0, 0
 ; CHECK: xxspltd 1, 1, 0
 ; CHECK: xxpermdi 34, 0, 1, 1
+
+; CHECK-P9-LABEL: testi0
+; CHECK-P9: lxsdx 0, 0, 4
+; CHECK-P9: lxvx 1, 0, 3
+; CHECK-P9: xxspltd 0, 0, 0
+; CHECK-P9: xxpermdi 34, 1, 0, 1
 }
 
 define <2 x double> @testi1(<2 x double>* %p1, double* %p2) {
   %v = load <2 x double>, <2 x double>* %p1
   %s = load double, double* %p2
   %r = insertelement <2 x double> %v, double %s, i32 1
   ret <2 x double> %r
 
 ; CHECK-LABEL: testi1
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: lxsdx 1, 0, 4
 ; CHECK: xxswapd 0, 0
 ; CHECK: xxspltd 1, 1, 0
 ; CHECK: xxmrgld 34, 1, 0
+
+; CHECK-P9-LABEL: testi1
+; CHECK-P9: lxsdx 0, 0, 4
+; CHECK-P9: lxvx 1, 0, 3
+; CHECK-P9: xxspltd 0, 0, 0
+; CHECK-P9: xxmrgld 34, 0, 1
 }
 
 define double @teste0(<2 x double>* %p1) {
   %v = load <2 x double>, <2 x double>* %p1
   %r = extractelement <2 x double> %v, i32 0
   ret double %r
 
 ; CHECK-LABEL: teste0
 ; CHECK: lxvd2x 1, 0, 3
+
+; CHECK-P9-LABEL: teste0
+; CHECK-P9: lxsdx 1, 0, 3
 }
 
 define double @teste1(<2 x double>* %p1) {
   %v = load <2 x double>, <2 x double>* %p1
   %r = extractelement <2 x double> %v, i32 1
   ret double %r
 
 ; CHECK-LABEL: teste1
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: xxswapd 1, 0
+
+; CHECK-P9-LABEL: teste1
+; CHECK-P9: li 4, 8
+; CHECK-P9: lxsdx 1, 3, 4
 }

test/CodeGen/PowerPC/vsx_shuffle_le.ll

 ; RUN: llc -mcpu=pwr8 -mattr=+vsx -mtriple=powerpc64le-unknown-linux-gnu < %s | FileCheck %s
+; RUN: llc -mcpu=pwr9 -mattr=+vsx -mtriple=powerpc64le-unknown-linux-gnu < %s \
+; RUN:   | FileCheck %s --check-prefix=CHECK-P9 --implicit-check-not xxswapd
 
 define <2 x double> @test00(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 0, i32 0>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: test00
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: xxspltd 34, 0, 0
+
+; CHECK-P9-LABEL: test00
+; CHECK-P9: lxvx 0, 0, 3
+; CHECK-P9: xxspltd 34, 0, 1
 }
 
 define <2 x double> @test01(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 0, i32 1>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: test01
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: xxswapd 34, 0
+
+; CHECK-P9-LABEL: test01
+; CHECK-P9: lxvx 34, 0, 3
 }
 
 define <2 x double> @test02(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 0, i32 2>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test02
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: lxvd2x 1, 0, 4
 ; CHECK: xxswapd 0, 0
 ; CHECK: xxswapd 1, 1
 ; CHECK: xxmrgld 34, 1, 0
+
+; CHECK-P9-LABEL: @test02
+; CHECK-P9: lxvx 0, 0, 3
+; CHECK-P9: lxvx 1, 0, 4
+; CHECK-P9: xxmrgld 34, 1, 0
 }
 
 define <2 x double> @test03(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 0, i32 3>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test03
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: lxvd2x 1, 0, 4
 ; CHECK: xxswapd 0, 0
 ; CHECK: xxswapd 1, 1
 ; CHECK: xxpermdi 34, 1, 0, 1
+
+; CHECK-P9-LABEL: @test03
+; CHECK-P9: lxvx 0, 0, 3
+; CHECK-P9: lxvx 1, 0, 4
+; CHECK-P9: xxpermdi 34, 1, 0, 1
 }
 
 define <2 x double> @test10(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 1, i32 0>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test10
 ; CHECK: lxvd2x 34, 0, 3
+
+; CHECK-P9-LABEL: @test10
+; CHECK-P9: lxvx 0, 0, 3
+; CHECK-P9: xxswapd 34, 0
 }
 
 define <2 x double> @test11(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 1, i32 1>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test11
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: xxspltd 34, 0, 1
+
+; CHECK-P9-LABEL: @test11
+; CHECK-P9: lxvx 0, 0, 3
+; CHECK-P9: xxspltd 34, 0, 0
 }
 
 define <2 x double> @test12(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 1, i32 2>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test12
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: lxvd2x 1, 0, 4
 ; CHECK: xxswapd 0, 0
 ; CHECK: xxswapd 1, 1
 ; CHECK: xxpermdi 34, 1, 0, 2
+
+; CHECK-P9-LABEL: @test12
+; CHECK-P9: lxvx 0, 0, 3
+; CHECK-P9: lxvx 1, 0, 4
+; CHECK-P9: xxpermdi 34, 1, 0, 2
 }
 
 define <2 x double> @test13(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 1, i32 3>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test13
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: lxvd2x 1, 0, 4
 ; CHECK: xxswapd 0, 0
 ; CHECK: xxswapd 1, 1
 ; CHECK: xxmrghd 34, 1, 0
+
+; CHECK-P9-LABEL: @test13
+; CHECK-P9: lxvx 0, 0, 3
+; CHECK-P9: lxvx 1, 0, 4
+; CHECK-P9: xxmrghd 34, 1, 0
 }
 
 define <2 x double> @test20(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 2, i32 0>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test20
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: lxvd2x 1, 0, 4
 ; CHECK: xxswapd 0, 0
 ; CHECK: xxswapd 1, 1
 ; CHECK: xxmrgld 34, 0, 1
+
+; CHECK-P9-LABEL: @test20
+; CHECK-P9: lxvx 0, 0, 3
+; CHECK-P9: lxvx 1, 0, 4
+; CHECK-P9: xxmrgld 34, 0, 1
 }
 
 define <2 x double> @test21(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 2, i32 1>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test21
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: lxvd2x 1, 0, 4
 ; CHECK: xxswapd 0, 0
 ; CHECK: xxswapd 1, 1
 ; CHECK: xxpermdi 34, 0, 1, 1
+
+; CHECK-P9-LABEL: @test21
+; CHECK-P9: lxvx 0, 0, 3
+; CHECK-P9: lxvx 1, 0, 4
+; CHECK-P9: xxpermdi 34, 0, 1, 1
 }
 
 define <2 x double> @test22(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 2, i32 2>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test22
 ; CHECK: lxvd2x 0, 0, 4
 ; CHECK: xxspltd 34, 0, 0
+
+; CHECK-P9-LABEL: @test22
+; CHECK-P9: lxvx 0, 0, 4
+; CHECK-P9: xxspltd 34, 0, 1
 }
 
 define <2 x double> @test23(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 2, i32 3>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test23
 ; CHECK: lxvd2x 0, 0, 4
 ; CHECK: xxswapd 34, 0
+
+; CHECK-P9-LABEL: @test23
+; CHECK-P9: lxvx 34, 0, 4
 }
 
 define <2 x double> @test30(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 3, i32 0>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test30
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: lxvd2x 1, 0, 4
 ; CHECK: xxswapd 0, 0
 ; CHECK: xxswapd 1, 1
 ; CHECK: xxpermdi 34, 0, 1, 2
+
+; CHECK-P9-LABEL: @test30
+; CHECK-P9: lxvx 0, 0, 3
+; CHECK-P9: lxvx 1, 0, 4
+; CHECK-P9: xxpermdi 34, 0, 1, 2
 }
 
 define <2 x double> @test31(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 3, i32 1>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test31
 ; CHECK: lxvd2x 0, 0, 3
 ; CHECK: lxvd2x 1, 0, 4
 ; CHECK: xxswapd 0, 0
 ; CHECK: xxswapd 1, 1
 ; CHECK: xxmrghd 34, 0, 1
+
+; CHECK-P9-LABEL: @test31
+; CHECK-P9: lxvx 0, 0, 3
+; CHECK-P9: lxvx 1, 0, 4
+; CHECK-P9: xxmrghd 34, 0, 1
 }
 
 define <2 x double> @test32(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 3, i32 2>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test32
 ; CHECK: lxvd2x 34, 0, 4
+
+; CHECK-P9-LABEL: @test32
+; CHECK-P9: lxvx 0, 0, 4
+; CHECK-P9: xxswapd 34, 0
 }
 
 define <2 x double> @test33(<2 x double>* %p1, <2 x double>* %p2) {
   %v1 = load <2 x double>, <2 x double>* %p1
   %v2 = load <2 x double>, <2 x double>* %p2
   %v3 = shufflevector <2 x double> %v1, <2 x double> %v2, <2 x i32> < i32 3, i32 3>
   ret <2 x double> %v3
 
 ; CHECK-LABEL: @test33
 ; CHECK: lxvd2x 0, 0, 4
 ; CHECK: xxspltd 34, 0, 1
+
+; CHECK-P9-LABEL: @test33
+; CHECK-P9: lxvx 0, 0, 4
+; CHECK-P9: xxspltd 34, 0, 0
 }