[PowerPC] Fix vector_shuffle combines when inputs are scalar_to_vector of differing types.

[amyk]

Summary

This patch fixes the combines for vector_shuffles when either or both of its
left and right hand side inputs are scalar_to_vector nodes.

Previously, when both left and right side inputs are scalar_to_vector nodes,
the current combine could not handle this situation, as the shuffle mask was
updated incorrectly. https://reviews.llvm.org/D127818 was a temporary solution
to this issue. Now, not only does this patch aim to resolve the previous issue the
of incorrect shuffle mask adjustments respectively, it also updates any test cases
that are affected by this change.

[amyk]

Rebase patch based on new tests added within D130485.

[amyk]

Ping.

[amyk]

Ping.

[amyk]

Ping.

[amyk]

Ping.

[amyk]

Ping.

[amyk]

Ping.

[nemanjai]

Can you also comment on whether this was thoroughly tested on both little endian and big endian systems (bootstrap, test-suite, SPEC, additional internal tests).

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14917–14918

I don't follow why we need these here. They both seem to only be needed in the respective conditions (i.e. depending on whether the LHS/RHS are scalar_to_vector nodes). And within those conditional blocks, they are reset before they're used.

So why do we need to define them here and initialize them to the width of a vector?

llvm/test/CodeGen/PowerPC/v16i8_scalar_to_vector_shuffle.ll

267

The code for this one gets worse on big endian. Do we know why?

347

The code for this one gets worse on big endian. Do we know why?

578

The code for this one gets worse on big endian. Do we know why?

659

The code for this one gets worse on big endian. Do we know why?

1431

The code for this one gets worse on big endian. Do we know why?

1658

The code for this one gets worse on big endian. Do we know why?

There are probably a bunch of other places. Can you please review what is happening there? I'll stop adding further similar comments.

llvm/test/CodeGen/PowerPC/v4i32_scalar_to_vector_shuffle.ll

123 ↗

(On Diff #452354)

The code generated for this one gets worse on all subtargets. Do we know why?

[amyk]

Discussed this patch with Nemanja outside of the review.

Many of the cases where we get worse codegen on BE (and some on LE as well) within the test cases occur either when the shuffle mask is explicitly asking for undefined vector elements from the original scalar_to_vector, or when the values in the vector are partially defined in the cases where the scalar_to_vector element size is smaller than the vector_shuffle element size. These cases primarily seem to appear within these contrived test cases, rather than real code. As a result, I've rebased this patch and addressed some review comments to move forward with this patch.

[amyk]

@nemanjai Yeah, at the time of posting the patch, I did do little endian and big endian system tests as you mentioned. Since it's been awhile and I need update the patch, I'd like to retest these particular runs with this patch.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14917–14918

Ya, I think I had that in the beginning and meant to remove it prior to putting up the patch but didn't realize that it was still left in there. I don't believe I need it either, so I've removed it.

[amyk]

Ping.

[amyk]

Ping.

[amyk]

Ping.

[amyk]

Ping.

[amyk]

Ping.

[amyk]

Ping.

[amyk]

Ping.

[nemanjai]

Although I have a fair number of comments, they're mostly stylistic comments that probably don't really require another revision. So LGTM and please address the comments prior to committing.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14847

I don't think we modify ShuffV so it should be a const reference.

14848–14849

These two values are actually the last *element* rather than *byte* aren't they? If so, please rename accordingly.

14855

Why do we only check here if LHSLastByteDefined >= 0 and similarly for the RHS below? Do we really want to pretend that a shuffle mask is in range if the last byte/element defined is undefined (i.e. presumably no bytes are defined)?

14903–14904

The name LaneWidth is misleading here. I kept thinking the width (i.e. the number of bits) of the lane that contains a defined value. But it is actually the number of valid elements in the vector.
For a node:

(shuff (v4i32 s_to_v i32), arbitrary_v4i32), mask)
LHSValidLaneWidth = 1
RHSValidLaneWidth = 4

And for a node:

(shuff (v4i32 s_to_v i32), (bitcast (s_to_v i64), v4i32), mask)
LHSValidLaneWidth = 1
RHSValidLaneWidth = 2

If I'm interpreting it correctly, please rename them to something like NumValidElts.

14932

I think we should have an early exit here if the valid lane width is zero:

if (LHSValidLaneWidth == 0)
  return false;

Since it is not really reasonable to do this transform if we are pulling in more bits than the original scalar_to_vector actually defined. Similarly with the RHS below.

14933

Nit: maybe a comment to make this clearer:

// The last element that comes from the LHS. For example:
// (shuff (s_to_v i32), (bitcast (s_to_v i64), v4i32), ...)
// The last element that comes from the LHS is actually 0, not 3
// because elements 1 and higher of a scalar_to_vector are undefined.

14945

Similar nit as above. A comment along the lines of:

// The last element that comes from the RHS. For example:
// (shuff (s_to_v i32), (bitcast (s_to_v i64), v4i32), ...)
// The last element that comes from the RHS is actually 5, not 7
// because elements 1 and higher of a scalar_to_vector are undefined.
// It is also not 4 because the original scalar_to_vector is wider and
// actually contains two i32 elements.

[kamaub]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14932

move LHSScalarSize up and use it in this block

14944

Same here, please raise RHSScalarSize and reuse it.

[kamaub]

Request changes because of the bug in the isShuffleMaskInRange() conditions

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14850

please use a range based loop

14855–14856

14856

Please change this to > (and for below).

14860–14861

14907–14908

please expand this comment to point out that -1 means all elements are defined

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

 // Fix up the shuffle mask to account for the fact that the result of
 // scalar_to_vector is not in lane zero. This just takes all values in
 // the ranges specified by the min/max indices and adds the number of
 // elements required to ensure each element comes from the respective
 // position in the valid lane.
 // On little endian, that's just the corresponding element in the other
 // half of the vector. On big endian, it is in the same half but right
 // justified rather than left justified in that half.
-static void fixupShuffleMaskForPermutedSToV(SmallVectorImpl<int> &ShuffV,
-                                            int LHSMaxIdx, int RHSMinIdx,
-                                            int RHSMaxIdx, int HalfVec,
-                                            unsigned ValidLaneWidth,
-                                            const PPCSubtarget &Subtarget) {
+static void fixupShuffleMaskForPermutedSToV(
+    SmallVectorImpl<int> &ShuffV, int LHSFirstElt, int LHSLastElt,
+    int RHSFirstElt, int RHSLastElt, int HalfVec, unsigned LHSValidLaneWidth,
+    unsigned RHSValidLaneWidth, const PPCSubtarget &Subtarget) {
   for (int i = 0, e = ShuffV.size(); i < e; i++) {
     int Idx = ShuffV[i];
-    if ((Idx >= 0 && Idx < LHSMaxIdx) || (Idx >= RHSMinIdx && Idx < RHSMaxIdx))
+    if (Idx >= LHSFirstElt && Idx <= LHSLastElt)
+      ShuffV[i] +=
+          Subtarget.isLittleEndian() ? HalfVec : HalfVec - LHSValidLaneWidth;
+    if (Idx >= RHSFirstElt && Idx <= RHSLastElt)
       ShuffV[i] +=
-          Subtarget.isLittleEndian() ? HalfVec : HalfVec - ValidLaneWidth;
+          Subtarget.isLittleEndian() ? HalfVec : HalfVec - RHSValidLaneWidth;
   }
 }
 
 // Replace a SCALAR_TO_VECTOR with a SCALAR_TO_VECTOR_PERMUTED except if
 // the original is:
 // (<n x Ty> (scalar_to_vector (Ty (extract_elt <n x Ty> %a, C))))
 // In such a case, just change the shuffle mask to extract the element
 // from the permuted index.
if (Idx && VT == OrigVector.getValueType()) {
       NewMask[ResultInElt] = Idx->getZExtValue();
       return DAG.getVectorShuffle(VT, dl, OrigVector, OrigVector, NewMask);
     }
   }
   return DAG.getNode(PPCISD::SCALAR_TO_VECTOR_PERMUTED, dl, VT,
                      OrigSToV.getOperand(0));
 }
 
 // On little endian subtargets, combine shuffles such as:
 // vector_shuffle<16,1,17,3,18,5,19,7,20,9,21,11,22,13,23,15>, <zero>, %b
 // into:
 // vector_shuffle<16,0,17,1,18,2,19,3,20,4,21,5,22,6,23,7>, <zero>, %b
 // because the latter can be matched to a single instruction merge.
 // Furthermore, SCALAR_TO_VECTOR on little endian always involves a permute
 // to put the value into element zero. Adjust the shuffle mask so that the
 // vector can remain in permuted form (to prevent a swap prior to a shuffle).
 // On big endian targets, this is still useful for SCALAR_TO_VECTOR
 // nodes with elements smaller than doubleword because all the ways
 // of getting scalar data into a vector register put the value in the
 // rightmost element of the left half of the vector.
 SDValue PPCTargetLowering::combineVectorShuffle(ShuffleVectorSDNode *SVN,
                                                 SelectionDAG &DAG) const {
   SDValue LHS = SVN->getOperand(0);
   SDValue RHS = SVN->getOperand(1);
   auto Mask = SVN->getMask();
   int NumElts = LHS.getValueType().getVectorNumElements();
   SDValue Res(SVN, 0);
   SDLoc dl(SVN);
   bool IsLittleEndian = Subtarget.isLittleEndian();
 
   // On big endian targets this is only useful for subtargets with direct moves.
SDValue PPCTargetLowering::combineVectorShuffle(ShuffleVectorSDNode *SVN,
 
   // Adjust the shuffle mask if either input vector comes from a
   // SCALAR_TO_VECTOR and keep the respective input vector in permuted
   // form (to prevent the need for a swap).
   SmallVector<int, 16> ShuffV(Mask.begin(), Mask.end());
   SDValue SToVLHS = isScalarToVec(LHS);
   SDValue SToVRHS = isScalarToVec(RHS);
   if (SToVLHS || SToVRHS) {
-    // FIXME: If both LHS and RHS are SCALAR_TO_VECTOR, but are not the
-    // same type and have differing element sizes, then do not perform
-    // the following transformation. The current transformation for
-    // SCALAR_TO_VECTOR assumes that both input vectors have the same
-    // element size. This will be updated in the future to account for
-    // differing sizes of the LHS and RHS.
-    if (SToVLHS && SToVRHS &&
-        (SToVLHS.getValueType().getScalarSizeInBits() !=
-         SToVRHS.getValueType().getScalarSizeInBits()))
-      return Res;
-
-    int NumEltsIn = SToVLHS ? SToVLHS.getValueType().getVectorNumElements()
-                            : SToVRHS.getValueType().getVectorNumElements();
-    int NumEltsOut = ShuffV.size();
+    int ShuffleEltWidth =
+        SVN->getValueType(0).getVectorElementType().getSizeInBits();
+    int ShuffleNumElts = ShuffV.size();
+    int HalfVec = ShuffleNumElts / 2;
     // The width of the "valid lane" (i.e. the lane that contains the value that
     // is vectorized) needs to be expressed in terms of the number of elements
     // of the shuffle. It is thereby the ratio of the values before and after
-    // any bitcast.
-    unsigned ValidLaneWidth =
-        SToVLHS ? SToVLHS.getValueType().getScalarSizeInBits() /
-                      LHS.getValueType().getScalarSizeInBits()
-                : SToVRHS.getValueType().getScalarSizeInBits() /
-                      RHS.getValueType().getScalarSizeInBits();
+    // any bitcast, which will be set later on if the LHS or RHS are
+    // SCALAR_TO_VECTOR nodes.
+    unsigned LHSValidLaneWidth = HalfVec;
+    unsigned RHSValidLaneWidth = HalfVec;
 
     // Initially assume that neither input is permuted. These will be adjusted
     // accordingly if either input is.
-    int LHSMaxIdx = -1;
-    int RHSMinIdx = -1;
-    int RHSMaxIdx = -1;
-    int HalfVec = LHS.getValueType().getVectorNumElements() / 2;
+    int LHSFirstElt = 0;
+    int RHSFirstElt = ShuffleNumElts;
+    int LHSLastElt = -1;
+    int RHSLastElt = -1;
+
+    // The scalar size of the LHS and RHS is initially assumed to be 128, as
+    // 128 is the default scalar size for any vector.
+    // If the LHS or RHS are SCALAR_TO_VECTOR nodes, we will adjust these values
+    // to the size of the scalar input to the SCALAR_TO_VECTOR later on.
+    int LHSScalarSize = 128;
+    int RHSScalarSize = 128;
 
     // Get the permuted scalar to vector nodes for the source(s) that come from
     // ISD::SCALAR_TO_VECTOR.
     // On big endian systems, this only makes sense for element sizes smaller
     // than 64 bits since for 64-bit elements, all instructions already put
     // the value into element zero. Since scalar size of LHS and RHS may differ
     // after isScalarToVec, this should be checked using their own sizes.
     if (SToVLHS) {
       if (!IsLittleEndian && SToVLHS.getValueType().getScalarSizeInBits() >= 64)
         return Res;
       // Set up the values for the shuffle vector fixup.
-      LHSMaxIdx = NumEltsOut / NumEltsIn;
+      LHSValidLaneWidth = SToVLHS.getValueType().getScalarSizeInBits() /
+                          LHS.getValueType().getScalarSizeInBits();
+      LHSScalarSize = SToVLHS.getValueType().getScalarSizeInBits();
+      LHSLastElt = LHSScalarSize / (ShuffleEltWidth + 1);
       SToVLHS = getSToVPermuted(SToVLHS, DAG, Subtarget);
       if (SToVLHS.getValueType() != LHS.getValueType())
         SToVLHS = DAG.getBitcast(LHS.getValueType(), SToVLHS);
       LHS = SToVLHS;
     }
     if (SToVRHS) {
       if (!IsLittleEndian && SToVRHS.getValueType().getScalarSizeInBits() >= 64)
         return Res;
-      RHSMinIdx = NumEltsOut;
-      RHSMaxIdx = NumEltsOut / NumEltsIn + RHSMinIdx;
+      RHSValidLaneWidth = SToVRHS.getValueType().getScalarSizeInBits() /
+                          RHS.getValueType().getScalarSizeInBits();
+      RHSScalarSize = SToVRHS.getValueType().getScalarSizeInBits();
+      RHSLastElt = RHSScalarSize / (ShuffleEltWidth + 1) + RHSFirstElt;
       SToVRHS = getSToVPermuted(SToVRHS, DAG, Subtarget);
       if (SToVRHS.getValueType() != RHS.getValueType())
         SToVRHS = DAG.getBitcast(RHS.getValueType(), SToVRHS);
       RHS = SToVRHS;
     }
 
     // Fix up the shuffle mask to reflect where the desired element actually is.
     // The minimum and maximum indices that correspond to element zero for both
     // the LHS and RHS are computed and will control which shuffle mask entries
     // are to be changed. For example, if the RHS is permuted, any shuffle mask
-    // entries in the range [RHSMinIdx,RHSMaxIdx) will be adjusted.
-    fixupShuffleMaskForPermutedSToV(ShuffV, LHSMaxIdx, RHSMinIdx, RHSMaxIdx,
-                                    HalfVec, ValidLaneWidth, Subtarget);
+    // entries in the range [RHSFirstElt,RHSLastElt] will be adjusted.
+    fixupShuffleMaskForPermutedSToV(
+        ShuffV, LHSFirstElt, LHSLastElt, RHSFirstElt, RHSLastElt, HalfVec,
+        LHSValidLaneWidth, RHSValidLaneWidth, Subtarget);
     Res = DAG.getVectorShuffle(SVN->getValueType(0), dl, LHS, RHS, ShuffV);
 
     // We may have simplified away the shuffle. We won't be able to do anything
     // further with it here.
     if (!isa<ShuffleVectorSDNode>(Res))
       return Res;
     Mask = cast<ShuffleVectorSDNode>(Res)->getMask();
   }

llvm/test/CodeGen/PowerPC/p8-scalar_vector_conversions.ll

 ; CHECK-NEXT:    sldi r4, r4, 32
 ; CHECK-NEXT:    mtfprd f1, r3
 ; CHECK-NEXT:    mtfprd f0, r4
 ; CHECK-NEXT:    xxmrghd v2, vs0, vs1
 ; CHECK-NEXT:    blr
 ;
 ; CHECK-LE-LABEL: buildi2:
 ; CHECK-LE:       # %bb.0: # %entry
-; CHECK-LE-NEXT:    mtfprd f0, r4
+; CHECK-LE-NEXT:    mtfprwz f0, r4
 ; CHECK-LE-NEXT:    mtfprd f1, r3
-; CHECK-LE-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-NEXT:    xxswapd v2, vs1
-; CHECK-LE-NEXT:    xxmrgld v2, v2, vs0
+; CHECK-LE-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-NEXT:    blr
 ;
 ; CHECK-AIX-LABEL: buildi2:
 ; CHECK-AIX:       # %bb.0: # %entry
 ; CHECK-AIX-NEXT:    sldi 4, 4, 32
 ; CHECK-AIX-NEXT:    mtfprd 1, 3
 ; CHECK-AIX-NEXT:    mtfprd 0, 4
 ; CHECK-AIX-NEXT:    xxmrghd 34, 0, 1
 ; CHECK-AIX-NEXT:    blr
 entry:
   %lhs.i32 = insertelement <4 x i32> undef, i32 %arg1, i32 0
   %rhs = insertelement <2 x i64> undef, i64 %arg, i32 0
   %lhs = bitcast <4 x i32> %lhs.i32 to <2 x i64>
   %shuffle = shufflevector <2 x i64> %lhs, <2 x i64> %rhs, <2 x i32> <i32 0, i32 2>
   ret <2 x i64> %shuffle
 }

llvm/test/CodeGen/PowerPC/v16i8_scalar_to_vector_shuffle.ll

 ; CHECK-AIX-32-P9-NEXT:    blr
 entry:
   %lhs = load <16 x i8>, ptr %b, align 4
   %rhs = insertelement <16 x i8> undef, i8 %arg, i32 0
   %shuffle = shufflevector <16 x i8> %lhs, <16 x i8> %rhs, <16 x i32> <i32 0, i32 1, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   ret <16 x i8> %shuffle
 }
 
 define <16 x i8> @test_v16i8_v8i16(i16 %arg, i8 %arg1) {
 ; CHECK-LE-P8-LABEL: test_v16i8_v8i16:
 ; CHECK-LE-P8:       # %bb.0: # %entry
-; CHECK-LE-P8-NEXT:    mtfprd f0, r4
-; CHECK-LE-P8-NEXT:    mtfprd f1, r3
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v3, vs1
-; CHECK-LE-P8-NEXT:    vmrglh v2, v3, v2
+; CHECK-LE-P8-NEXT:    mtvsrd v2, r4
+; CHECK-LE-P8-NEXT:    mtvsrd v3, r3
+; CHECK-LE-P8-NEXT:    vmrghh v2, v3, v2
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v16i8_v8i16:
 ; CHECK-LE-P9:       # %bb.0: # %entry
-; CHECK-LE-P9-NEXT:    mtfprd f0, r4
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    xxswapd v3, vs0
-; CHECK-LE-P9-NEXT:    vmrglh v2, v3, v2
+; CHECK-LE-P9-NEXT:    mtvsrd v2, r4
+; CHECK-LE-P9-NEXT:    mtvsrd v3, r3
+; CHECK-LE-P9-NEXT:    vmrghh v2, v3, v2
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v16i8_v8i16:
 ; CHECK-BE-P8:       # %bb.0: # %entry
-; CHECK-BE-P8-NEXT:    sldi r4, r4, 56
-; CHECK-BE-P8-NEXT:    sldi r3, r3, 48
-; CHECK-BE-P8-NEXT:    mtvsrd v2, r4
-; CHECK-BE-P8-NEXT:    mtvsrd v3, r3
-; CHECK-BE-P8-NEXT:    vmrghh v2, v2, v3
+; CHECK-BE-P8-NEXT:    addis r5, r2, .LCPI3_0@toc@ha
+; CHECK-BE-P8-NEXT:    mtvsrwz v2, r4
+; CHECK-BE-P8-NEXT:    addi r5, r5, .LCPI3_0@toc@l
+; CHECK-BE-P8-NEXT:    mtvsrwz v4, r3
+; CHECK-BE-P8-NEXT:    lxvw4x v3, 0, r5
+; CHECK-BE-P8-NEXT:    vperm v2, v2, v4, v3
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v16i8_v8i16:
 ; CHECK-BE-P9:       # %bb.0: # %entry
-; CHECK-BE-P9-NEXT:    sldi r4, r4, 56
-; CHECK-BE-P9-NEXT:    sldi r3, r3, 48
-; CHECK-BE-P9-NEXT:    mtvsrd v2, r4
-; CHECK-BE-P9-NEXT:    mtvsrd v3, r3
-; CHECK-BE-P9-NEXT:    vmrghh v2, v2, v3
+; CHECK-BE-P9-NEXT:    mtvsrwz v3, r3
+; CHECK-BE-P9-NEXT:    addis r3, r2, .LCPI3_0@toc@ha
+; CHECK-BE-P9-NEXT:    mtvsrwz v2, r4
+; CHECK-BE-P9-NEXT:    addi r3, r3, .LCPI3_0@toc@l
+; CHECK-BE-P9-NEXT:    lxv v4, 0(r3)
+; CHECK-BE-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v16i8_v8i16:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    sldi r4, r4, 56
-; CHECK-AIX-64-P8-NEXT:    sldi r3, r3, 48
-; CHECK-AIX-64-P8-NEXT:    mtvsrd v2, r4
-; CHECK-AIX-64-P8-NEXT:    mtvsrd v3, r3
-; CHECK-AIX-64-P8-NEXT:    vmrghh v2, v2, v3
+; CHECK-AIX-64-P8-NEXT:    ld r5, L..C3(r2) # %const.0
+; CHECK-AIX-64-P8-NEXT:    mtvsrwz v2, r4
+; CHECK-AIX-64-P8-NEXT:    mtvsrwz v4, r3
+; CHECK-AIX-64-P8-NEXT:    lxvw4x v3, 0, r5
+; CHECK-AIX-64-P8-NEXT:    vperm v2, v2, v4, v3
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v16i8_v8i16:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
-; CHECK-AIX-64-P9-NEXT:    sldi r4, r4, 56
-; CHECK-AIX-64-P9-NEXT:    sldi r3, r3, 48
-; CHECK-AIX-64-P9-NEXT:    mtvsrd v2, r4
-; CHECK-AIX-64-P9-NEXT:    mtvsrd v3, r3
-; CHECK-AIX-64-P9-NEXT:    vmrghh v2, v2, v3
+; CHECK-AIX-64-P9-NEXT:    mtvsrwz v3, r3
+; CHECK-AIX-64-P9-NEXT:    ld r3, L..C2(r2) # %const.0
+; CHECK-AIX-64-P9-NEXT:    mtvsrwz v2, r4
+; CHECK-AIX-64-P9-NEXT:    lxv v4, 0(r3)
+; CHECK-AIX-64-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v16i8_v8i16:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    stb r4, -32(r1)
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x v2, 0, r5
 ; CHECK-AIX-32-P8-NEXT:    sth r3, -16(r1)

 entry:
   %lhs = insertelement <16 x i8> undef, i8 %arg1, i32 0
   %rhs.tmp = insertelement <8 x i16> undef, i16 %arg, i32 0
   %rhs = bitcast <8 x i16> %rhs.tmp to <16 x i8>
   %shuffle = shufflevector <16 x i8> %lhs, <16 x i8> %rhs, <16 x i32> <i32 0, i32 1, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   ret <16 x i8> %shuffle
 }
 
 define <16 x i8> @test_v8i16_v16i8(i16 %arg, i8 %arg1) {
 ; CHECK-LE-P8-LABEL: test_v8i16_v16i8:
 ; CHECK-LE-P8:       # %bb.0: # %entry
-; CHECK-LE-P8-NEXT:    mtfprd f0, r4
-; CHECK-LE-P8-NEXT:    mtfprd f1, r3
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v3, vs1
-; CHECK-LE-P8-NEXT:    vmrglh v2, v2, v3
+; CHECK-LE-P8-NEXT:    mtvsrd v2, r4
+; CHECK-LE-P8-NEXT:    mtvsrd v3, r3
+; CHECK-LE-P8-NEXT:    vmrghh v2, v2, v3
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v8i16_v16i8:
 ; CHECK-LE-P9:       # %bb.0: # %entry
-; CHECK-LE-P9-NEXT:    mtfprd f0, r4
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    xxswapd v3, vs0
-; CHECK-LE-P9-NEXT:    vmrglh v2, v2, v3
+; CHECK-LE-P9-NEXT:    mtvsrd v2, r4
+; CHECK-LE-P9-NEXT:    mtvsrd v3, r3
+; CHECK-LE-P9-NEXT:    vmrghh v2, v2, v3
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v8i16_v16i8:
 ; CHECK-BE-P8:       # %bb.0: # %entry
-; CHECK-BE-P8-NEXT:    sldi r4, r4, 56
-; CHECK-BE-P8-NEXT:    sldi r3, r3, 48
-; CHECK-BE-P8-NEXT:    mtvsrd v2, r4
-; CHECK-BE-P8-NEXT:    mtvsrd v3, r3
-; CHECK-BE-P8-NEXT:    vmrghh v2, v3, v2
+; CHECK-BE-P8-NEXT:    addis r5, r2, .LCPI4_0@toc@ha
+; CHECK-BE-P8-NEXT:    mtvsrwz v2, r4
+; CHECK-BE-P8-NEXT:    addi r5, r5, .LCPI4_0@toc@l
+; CHECK-BE-P8-NEXT:    mtvsrwz v4, r3
+; CHECK-BE-P8-NEXT:    lxvw4x v3, 0, r5
+; CHECK-BE-P8-NEXT:    vperm v2, v4, v2, v3
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v8i16_v16i8:
 ; CHECK-BE-P9:       # %bb.0: # %entry
-; CHECK-BE-P9-NEXT:    sldi r4, r4, 56
-; CHECK-BE-P9-NEXT:    sldi r3, r3, 48
-; CHECK-BE-P9-NEXT:    mtvsrd v2, r4
-; CHECK-BE-P9-NEXT:    mtvsrd v3, r3
-; CHECK-BE-P9-NEXT:    vmrghh v2, v3, v2
+; CHECK-BE-P9-NEXT:    mtvsrwz v3, r3
+; CHECK-BE-P9-NEXT:    addis r3, r2, .LCPI4_0@toc@ha
+; CHECK-BE-P9-NEXT:    mtvsrwz v2, r4
+; CHECK-BE-P9-NEXT:    addi r3, r3, .LCPI4_0@toc@l
+; CHECK-BE-P9-NEXT:    lxv v4, 0(r3)
+; CHECK-BE-P9-NEXT:    vperm v2, v3, v2, v4
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v8i16_v16i8:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    sldi r4, r4, 56
-; CHECK-AIX-64-P8-NEXT:    sldi r3, r3, 48
-; CHECK-AIX-64-P8-NEXT:    mtvsrd v2, r4
-; CHECK-AIX-64-P8-NEXT:    mtvsrd v3, r3
-; CHECK-AIX-64-P8-NEXT:    vmrghh v2, v3, v2
+; CHECK-AIX-64-P8-NEXT:    ld r5, L..C4(r2) # %const.0
+; CHECK-AIX-64-P8-NEXT:    mtvsrwz v2, r4
+; CHECK-AIX-64-P8-NEXT:    mtvsrwz v4, r3
+; CHECK-AIX-64-P8-NEXT:    lxvw4x v3, 0, r5
+; CHECK-AIX-64-P8-NEXT:    vperm v2, v4, v2, v3
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v8i16_v16i8:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
-; CHECK-AIX-64-P9-NEXT:    sldi r4, r4, 56
-; CHECK-AIX-64-P9-NEXT:    sldi r3, r3, 48
-; CHECK-AIX-64-P9-NEXT:    mtvsrd v2, r4
-; CHECK-AIX-64-P9-NEXT:    mtvsrd v3, r3
-; CHECK-AIX-64-P9-NEXT:    vmrghh v2, v3, v2
+; CHECK-AIX-64-P9-NEXT:    mtvsrwz v3, r3
+; CHECK-AIX-64-P9-NEXT:    ld r3, L..C3(r2) # %const.0
+; CHECK-AIX-64-P9-NEXT:    mtvsrwz v2, r4
+; CHECK-AIX-64-P9-NEXT:    lxv v4, 0(r3)
+; CHECK-AIX-64-P9-NEXT:    vperm v2, v3, v2, v4
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v8i16_v16i8:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    stb r4, -32(r1)
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x v2, 0, r5
 ; CHECK-AIX-32-P8-NEXT:    sth r3, -16(r1)

 ; CHECK-BE-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-BE-P9-NEXT:    addi r3, r3, .LCPI5_0@toc@l
 ; CHECK-BE-P9-NEXT:    lxv v4, 0(r3)
 ; CHECK-BE-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_none_v8i16:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    ld r5, L..C3(r2) # %const.0
+; CHECK-AIX-64-P8-NEXT:    ld r5, L..C5(r2) # %const.0
 ; CHECK-AIX-64-P8-NEXT:    lxvw4x v2, 0, r4
 ; CHECK-AIX-64-P8-NEXT:    mtvsrwz v4, r3
 ; CHECK-AIX-64-P8-NEXT:    lxvw4x v3, 0, r5
 ; CHECK-AIX-64-P8-NEXT:    vperm v2, v2, v4, v3
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_none_v8i16:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    mtvsrwz v3, r3
-; CHECK-AIX-64-P9-NEXT:    ld r3, L..C2(r2) # %const.0
+; CHECK-AIX-64-P9-NEXT:    ld r3, L..C4(r2) # %const.0
 ; CHECK-AIX-64-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-AIX-64-P9-NEXT:    lxv v4, 0(r3)
 ; CHECK-AIX-64-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_none_v8i16:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16

 ; CHECK-BE-P9-LABEL: test_v8i16_none:
 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    mtvsrwz v3, r5
 ; CHECK-BE-P9-NEXT:    vinserth v2, v3, 0
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v8i16_none:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    ld r4, L..C4(r2) # %const.0
+; CHECK-AIX-64-P8-NEXT:    ld r4, L..C6(r2) # %const.0
 ; CHECK-AIX-64-P8-NEXT:    mtvsrwz v4, r3
 ; CHECK-AIX-64-P8-NEXT:    lxvw4x v3, 0, r4
 ; CHECK-AIX-64-P8-NEXT:    vperm v2, v4, v2, v3
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v8i16_none:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    mtvsrwz v3, r3

 ; CHECK-AIX-32-P9-NEXT:    mtvsrwz v3, r3
 ; CHECK-AIX-32-P9-NEXT:    vinserth v2, v3, 0
 ; CHECK-AIX-32-P9-NEXT:    blr
 entry:
   %vecins = insertelement <8 x i16> %a, i16 %b, i32 0
   ret <8 x i16> %vecins
 }
 
 define <16 x i8> @test_v16i8_v4i32(i8 %arg, i32 %arg1, <16 x i8> %a, <4 x i32> %b) {
 ; CHECK-LE-P8-LABEL: test_v16i8_v4i32:
 ; CHECK-LE-P8:       # %bb.0: # %entry
-; CHECK-LE-P8-NEXT:    mtfprd f0, r3
-; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v3, vs1
-; CHECK-LE-P8-NEXT:    vmrglb v2, v3, v2
+; CHECK-LE-P8-NEXT:    mtvsrd v2, r3
+; CHECK-LE-P8-NEXT:    mtvsrwz v3, r4
+; CHECK-LE-P8-NEXT:    vmrghb v2, v3, v2
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v16i8_v4i32:
 ; CHECK-LE-P9:       # %bb.0: # %entry
-; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    mtvsrws v3, r4
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P9-NEXT:    vmrglb v2, v3, v2
+; CHECK-LE-P9-NEXT:    mtvsrd v2, r3
+; CHECK-LE-P9-NEXT:    mtvsrwz v3, r4
+; CHECK-LE-P9-NEXT:    vmrghb v2, v3, v2
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v16i8_v4i32:
 ; CHECK-BE-P8:       # %bb.0: # %entry
-; CHECK-BE-P8-NEXT:    sldi r3, r3, 56
-; CHECK-BE-P8-NEXT:    sldi r4, r4, 32
-; CHECK-BE-P8-NEXT:    mtvsrd v2, r3
-; CHECK-BE-P8-NEXT:    mtvsrd v3, r4
-; CHECK-BE-P8-NEXT:    vmrghb v2, v2, v3
+; CHECK-BE-P8-NEXT:    addis r5, r2, .LCPI7_0@toc@ha
+; CHECK-BE-P8-NEXT:    mtvsrwz v2, r3
+; CHECK-BE-P8-NEXT:    addi r5, r5, .LCPI7_0@toc@l
+; CHECK-BE-P8-NEXT:    mtvsrwz v4, r4
+; CHECK-BE-P8-NEXT:    lxvw4x v3, 0, r5
+; CHECK-BE-P8-NEXT:    vperm v2, v2, v4, v3
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v16i8_v4i32:
 ; CHECK-BE-P9:       # %bb.0: # %entry
-; CHECK-BE-P9-NEXT:    sldi r3, r3, 56
-; CHECK-BE-P9-NEXT:    mtvsrws v3, r4
-; CHECK-BE-P9-NEXT:    mtvsrd v2, r3
-; CHECK-BE-P9-NEXT:    vmrghb v2, v2, v3
+; CHECK-BE-P9-NEXT:    mtvsrwz v2, r3
+; CHECK-BE-P9-NEXT:    addis r3, r2, .LCPI7_0@toc@ha
+; CHECK-BE-P9-NEXT:    mtvsrwz v3, r4
+; CHECK-BE-P9-NEXT:    addi r3, r3, .LCPI7_0@toc@l
+; CHECK-BE-P9-NEXT:    lxv v4, 0(r3)
+; CHECK-BE-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v16i8_v4i32:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    sldi r3, r3, 56
-; CHECK-AIX-64-P8-NEXT:    sldi r4, r4, 32
-; CHECK-AIX-64-P8-NEXT:    mtvsrd v2, r3
-; CHECK-AIX-64-P8-NEXT:    mtvsrd v3, r4
-; CHECK-AIX-64-P8-NEXT:    vmrghb v2, v2, v3
+; CHECK-AIX-64-P8-NEXT:    ld r5, L..C7(r2) # %const.0
+; CHECK-AIX-64-P8-NEXT:    mtvsrwz v2, r3
+; CHECK-AIX-64-P8-NEXT:    mtvsrwz v4, r4
+; CHECK-AIX-64-P8-NEXT:    lxvw4x v3, 0, r5
+; CHECK-AIX-64-P8-NEXT:    vperm v2, v2, v4, v3
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v16i8_v4i32:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
-; CHECK-AIX-64-P9-NEXT:    sldi r3, r3, 56
-; CHECK-AIX-64-P9-NEXT:    mtvsrws v3, r4
-; CHECK-AIX-64-P9-NEXT:    mtvsrd v2, r3
-; CHECK-AIX-64-P9-NEXT:    vmrghb v2, v2, v3
+; CHECK-AIX-64-P9-NEXT:    mtvsrwz v2, r3
+; CHECK-AIX-64-P9-NEXT:    ld r3, L..C5(r2) # %const.0
+; CHECK-AIX-64-P9-NEXT:    mtvsrwz v3, r4
+; CHECK-AIX-64-P9-NEXT:    lxv v4, 0(r3)
+; CHECK-AIX-64-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v16i8_v4i32:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stb r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    addi r3, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x v2, 0, r5
entry:
   %lhs.tmp = insertelement <16 x i8> %a, i8 %arg, i32 0
   %lhs = bitcast <16 x i8> %lhs.tmp to <16 x i8>
   %rhs.tmp = insertelement <4 x i32> %b, i32 %arg1, i32 0
   %rhs = bitcast <4 x i32> %rhs.tmp to <16 x i8>
   %shuffle = shufflevector <16 x i8> %lhs, <16 x i8> %rhs, <16 x i32> <i32 0, i32 16, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   ret <16 x i8> %shuffle
 }
 
 define <16 x i8> @test_v4i32_v16i8(i32 %arg, i8 %arg1) {
 ; CHECK-LE-P8-LABEL: test_v4i32_v16i8:
 ; CHECK-LE-P8:       # %bb.0: # %entry
-; CHECK-LE-P8-NEXT:    mtfprd f0, r4
-; CHECK-LE-P8-NEXT:    mtfprd f1, r3
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v3, vs1
-; CHECK-LE-P8-NEXT:    vmrglh v2, v2, v3
+; CHECK-LE-P8-NEXT:    mtvsrd v2, r4
+; CHECK-LE-P8-NEXT:    mtvsrwz v3, r3
+; CHECK-LE-P8-NEXT:    vmrghh v2, v2, v3
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v4i32_v16i8:
 ; CHECK-LE-P9:       # %bb.0: # %entry
-; CHECK-LE-P9-NEXT:    mtfprd f0, r4
-; CHECK-LE-P9-NEXT:    mtvsrws v3, r3
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P9-NEXT:    vmrglh v2, v2, v3
+; CHECK-LE-P9-NEXT:    mtvsrd v2, r4
+; CHECK-LE-P9-NEXT:    mtvsrwz v3, r3
+; CHECK-LE-P9-NEXT:    vmrghh v2, v2, v3
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v4i32_v16i8:
 ; CHECK-BE-P8:       # %bb.0: # %entry
-; CHECK-BE-P8-NEXT:    sldi r4, r4, 56
-; CHECK-BE-P8-NEXT:    sldi r3, r3, 32
-; CHECK-BE-P8-NEXT:    mtvsrd v2, r4
-; CHECK-BE-P8-NEXT:    mtvsrd v3, r3
-; CHECK-BE-P8-NEXT:    vmrghh v2, v3, v2
+; CHECK-BE-P8-NEXT:    addis r5, r2, .LCPI8_0@toc@ha
+; CHECK-BE-P8-NEXT:    mtvsrwz v2, r4
+; CHECK-BE-P8-NEXT:    addi r5, r5, .LCPI8_0@toc@l
+; CHECK-BE-P8-NEXT:    mtvsrwz v4, r3
+; CHECK-BE-P8-NEXT:    lxvw4x v3, 0, r5
+; CHECK-BE-P8-NEXT:    vperm v2, v4, v2, v3
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v4i32_v16i8:
 ; CHECK-BE-P9:       # %bb.0: # %entry
-; CHECK-BE-P9-NEXT:    sldi r4, r4, 56
-; CHECK-BE-P9-NEXT:    mtvsrws v3, r3
-; CHECK-BE-P9-NEXT:    mtvsrd v2, r4
-; CHECK-BE-P9-NEXT:    vmrghh v2, v3, v2
+; CHECK-BE-P9-NEXT:    mtvsrwz v3, r3
+; CHECK-BE-P9-NEXT:    addis r3, r2, .LCPI8_0@toc@ha
+; CHECK-BE-P9-NEXT:    mtvsrwz v2, r4
+; CHECK-BE-P9-NEXT:    addi r3, r3, .LCPI8_0@toc@l
+; CHECK-BE-P9-NEXT:    lxv v4, 0(r3)
+; CHECK-BE-P9-NEXT:    vperm v2, v3, v2, v4
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v4i32_v16i8:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    sldi r4, r4, 56
-; CHECK-AIX-64-P8-NEXT:    sldi r3, r3, 32
-; CHECK-AIX-64-P8-NEXT:    mtvsrd v2, r4
-; CHECK-AIX-64-P8-NEXT:    mtvsrd v3, r3
-; CHECK-AIX-64-P8-NEXT:    vmrghh v2, v3, v2
+; CHECK-AIX-64-P8-NEXT:    ld r5, L..C8(r2) # %const.0
+; CHECK-AIX-64-P8-NEXT:    mtvsrwz v2, r4
+; CHECK-AIX-64-P8-NEXT:    mtvsrwz v4, r3
+; CHECK-AIX-64-P8-NEXT:    lxvw4x v3, 0, r5
+; CHECK-AIX-64-P8-NEXT:    vperm v2, v4, v2, v3
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v4i32_v16i8:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
-; CHECK-AIX-64-P9-NEXT:    sldi r4, r4, 56
-; CHECK-AIX-64-P9-NEXT:    mtvsrws v3, r3
-; CHECK-AIX-64-P9-NEXT:    mtvsrd v2, r4
-; CHECK-AIX-64-P9-NEXT:    vmrghh v2, v3, v2
+; CHECK-AIX-64-P9-NEXT:    mtvsrwz v3, r3
+; CHECK-AIX-64-P9-NEXT:    ld r3, L..C6(r2) # %const.0
+; CHECK-AIX-64-P9-NEXT:    mtvsrwz v2, r4
+; CHECK-AIX-64-P9-NEXT:    lxv v4, 0(r3)
+; CHECK-AIX-64-P9-NEXT:    vperm v2, v3, v2, v4
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v4i32_v16i8:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    stb r4, -32(r1)
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x v2, 0, r5
 ; CHECK-AIX-32-P8-NEXT:    stw r3, -16(r1)

 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    mtfprwz f0, r5
 ; CHECK-BE-P9-NEXT:    xxinsertw v2, vs0, 4
 ; CHECK-BE-P9-NEXT:    xxinsertw v2, vs0, 12
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_none_v4i32:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    ld r4, L..C5(r2) # %const.0
+; CHECK-AIX-64-P8-NEXT:    ld r4, L..C9(r2) # %const.0
 ; CHECK-AIX-64-P8-NEXT:    mtvsrwz v4, r3
-; CHECK-AIX-64-P8-NEXT:    ld r3, L..C6(r2) # %const.1
+; CHECK-AIX-64-P8-NEXT:    ld r3, L..C10(r2) # %const.1
 ; CHECK-AIX-64-P8-NEXT:    lxvw4x v3, 0, r4
 ; CHECK-AIX-64-P8-NEXT:    vperm v2, v2, v4, v3
 ; CHECK-AIX-64-P8-NEXT:    lxvw4x v3, 0, r3
 ; CHECK-AIX-64-P8-NEXT:    vperm v2, v2, v4, v3
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_none_v4i32:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry

 ; CHECK-BE-P9-NEXT:    lxv v4, 0(r3)
 ; CHECK-BE-P9-NEXT:    vspltb v3, v3, 7
 ; CHECK-BE-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v4i32_none:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
 ; CHECK-AIX-64-P8-NEXT:    lbzx r4, 0, r4
-; CHECK-AIX-64-P8-NEXT:    ld r5, L..C7(r2) # %const.0
+; CHECK-AIX-64-P8-NEXT:    ld r5, L..C11(r2) # %const.0
 ; CHECK-AIX-64-P8-NEXT:    lxsiwzx v3, 0, r3
 ; CHECK-AIX-64-P8-NEXT:    mtvsrwz v2, r4
 ; CHECK-AIX-64-P8-NEXT:    lxvw4x v4, 0, r5
 ; CHECK-AIX-64-P8-NEXT:    vspltb v2, v2, 7
 ; CHECK-AIX-64-P8-NEXT:    vperm v2, v3, v2, v4
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v4i32_none:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    lxsiwzx v2, 0, r3
-; CHECK-AIX-64-P9-NEXT:    ld r3, L..C3(r2) # %const.0
+; CHECK-AIX-64-P9-NEXT:    ld r3, L..C7(r2) # %const.0
 ; CHECK-AIX-64-P9-NEXT:    lxsibzx v3, 0, r4
 ; CHECK-AIX-64-P9-NEXT:    lxv v4, 0(r3)
 ; CHECK-AIX-64-P9-NEXT:    vspltb v3, v3, 7
 ; CHECK-AIX-64-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v4i32_none:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
entry:
   %3 = bitcast <16 x i8> %vecinit6 to <16 x i8>
   %shuffle = shufflevector <16 x i8> %2, <16 x i8> %3, <16 x i32> <i32 0, i32 1, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   ret <16 x i8> %shuffle
 }
 
 define <16 x i8> @test_v16i8_v2i64(i8 %arg, i64 %arg1, <16 x i8> %a, <2 x i64> %b) {
 ; CHECK-LE-P8-LABEL: test_v16i8_v2i64:
 ; CHECK-LE-P8:       # %bb.0: # %entry
-; CHECK-LE-P8-NEXT:    mtfprd f0, r3
-; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v3, vs1
-; CHECK-LE-P8-NEXT:    vmrglb v2, v3, v2
+; CHECK-LE-P8-NEXT:    mtvsrd v2, r3
+; CHECK-LE-P8-NEXT:    mtvsrd v3, r4
+; CHECK-LE-P8-NEXT:    vmrghb v2, v3, v2
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v16i8_v2i64:
 ; CHECK-LE-P9:       # %bb.0: # %entry
-; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P9-NEXT:    mtfprd f0, r4
-; CHECK-LE-P9-NEXT:    xxswapd v3, vs0
-; CHECK-LE-P9-NEXT:    vmrglb v2, v3, v2
+; CHECK-LE-P9-NEXT:    mtvsrd v2, r3
+; CHECK-LE-P9-NEXT:    mtvsrd v3, r4
+; CHECK-LE-P9-NEXT:    vmrghb v2, v3, v2
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v16i8_v2i64:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    sldi r3, r3, 56
 ; CHECK-BE-P8-NEXT:    mtvsrd v3, r4
 ; CHECK-BE-P8-NEXT:    mtvsrd v2, r3
 ; CHECK-BE-P8-NEXT:    vmrghb v2, v2, v3
entry:
   %rhs = bitcast <2 x i64> %rhs.tmp to <16 x i8>
   %shuffle = shufflevector <16 x i8> %lhs, <16 x i8> %rhs, <16 x i32> <i32 0, i32 16, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   ret <16 x i8> %shuffle
 }
 
 define <16 x i8> @test_v2i64_v16i8(i64 %arg, i8 %arg1) {
 ; CHECK-LE-P8-LABEL: test_v2i64_v16i8:
 ; CHECK-LE-P8:       # %bb.0: # %entry
-; CHECK-LE-P8-NEXT:    mtfprd f0, r4
-; CHECK-LE-P8-NEXT:    mtfprd f1, r3
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v3, vs1
-; CHECK-LE-P8-NEXT:    vmrglh v2, v2, v3
+; CHECK-LE-P8-NEXT:    mtvsrd v2, r4
+; CHECK-LE-P8-NEXT:    mtvsrd v3, r3
+; CHECK-LE-P8-NEXT:    vmrghh v2, v2, v3
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v2i64_v16i8:
 ; CHECK-LE-P9:       # %bb.0: # %entry
-; CHECK-LE-P9-NEXT:    mtfprd f0, r4
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    xxswapd v3, vs0
-; CHECK-LE-P9-NEXT:    vmrglh v2, v2, v3
+; CHECK-LE-P9-NEXT:    mtvsrd v2, r4
+; CHECK-LE-P9-NEXT:    mtvsrd v3, r3
+; CHECK-LE-P9-NEXT:    vmrghh v2, v2, v3
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v2i64_v16i8:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    sldi r4, r4, 56
 ; CHECK-BE-P8-NEXT:    mtvsrd v3, r3
 ; CHECK-BE-P8-NEXT:    mtvsrd v2, r4
 ; CHECK-BE-P8-NEXT:    vmrghh v2, v3, v2

 ; CHECK-BE-P9-NEXT:    mtvsrwz v4, r3
 ; CHECK-BE-P9-NEXT:    addi r5, r5, .LCPI16_0@toc@l
 ; CHECK-BE-P9-NEXT:    lxv v2, 0(r5)
 ; CHECK-BE-P9-NEXT:    vperm v2, v4, v3, v2
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v8i16_v8i16rhs:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    ld r5, L..C8(r2) # %const.0
+; CHECK-AIX-64-P8-NEXT:    ld r5, L..C12(r2) # %const.0
 ; CHECK-AIX-64-P8-NEXT:    mtvsrwz v2, r4
 ; CHECK-AIX-64-P8-NEXT:    mtvsrwz v4, r3
 ; CHECK-AIX-64-P8-NEXT:    lxvw4x v3, 0, r5
 ; CHECK-AIX-64-P8-NEXT:    vperm v2, v4, v2, v3
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v8i16_v8i16rhs:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
-; CHECK-AIX-64-P9-NEXT:    ld r5, L..C4(r2) # %const.0
+; CHECK-AIX-64-P9-NEXT:    ld r5, L..C8(r2) # %const.0
 ; CHECK-AIX-64-P9-NEXT:    mtvsrwz v3, r4
 ; CHECK-AIX-64-P9-NEXT:    mtvsrwz v4, r3
 ; CHECK-AIX-64-P9-NEXT:    lxv v2, 0(r5)
 ; CHECK-AIX-64-P9-NEXT:    vperm v2, v4, v3, v2
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v8i16_v8i16rhs:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
entry:
   %rhs.tmp = insertelement <8 x i16> undef, i16 %arg1, i32 0
   %rhs = bitcast <8 x i16> %rhs.tmp to <16 x i8>
   %lhs.tmp = insertelement <8 x i16> undef, i16 %arg, i32 0
   %lhs = bitcast <8 x i16> %lhs.tmp to <16 x i8>
   %shuffle = shufflevector <16 x i8> %lhs, <16 x i8> %rhs, <16 x i32> <i32 0, i32 1, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   ret <16 x i8> %shuffle
 }
 
 define <16 x i8> @test_v8i16_v4i32(<8 x i16> %a, <4 x i32> %b, i16 %arg, i32 %arg1) {
 ; CHECK-LE-P8-LABEL: test_v8i16_v4i32:
 ; CHECK-LE-P8:       # %bb.0: # %entry
-; CHECK-LE-P8-NEXT:    mtfprd f0, r7
-; CHECK-LE-P8-NEXT:    mtfprd f1, r8
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v3, vs1
-; CHECK-LE-P8-NEXT:    vmrglb v2, v3, v2
+; CHECK-LE-P8-NEXT:    mtvsrd v2, r7
+; CHECK-LE-P8-NEXT:    mtvsrwz v3, r8
+; CHECK-LE-P8-NEXT:    vmrghb v2, v3, v2
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v8i16_v4i32:
 ; CHECK-LE-P9:       # %bb.0: # %entry
-; CHECK-LE-P9-NEXT:    mtfprd f0, r7
-; CHECK-LE-P9-NEXT:    mtvsrws v3, r8
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P9-NEXT:    vmrglb v2, v3, v2
+; CHECK-LE-P9-NEXT:    mtvsrd v2, r7
+; CHECK-LE-P9-NEXT:    mtvsrwz v3, r8
+; CHECK-LE-P9-NEXT:    vmrghb v2, v3, v2
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v8i16_v4i32:
 ; CHECK-BE-P8:       # %bb.0: # %entry
-; CHECK-BE-P8-NEXT:    sldi r3, r7, 48
-; CHECK-BE-P8-NEXT:    sldi r4, r8, 32
-; CHECK-BE-P8-NEXT:    mtvsrd v2, r3
-; CHECK-BE-P8-NEXT:    mtvsrd v3, r4
-; CHECK-BE-P8-NEXT:    vmrghb v2, v2, v3
+; CHECK-BE-P8-NEXT:    addis r3, r2, .LCPI17_0@toc@ha
+; CHECK-BE-P8-NEXT:    mtvsrwz v2, r7
+; CHECK-BE-P8-NEXT:    addi r3, r3, .LCPI17_0@toc@l
+; CHECK-BE-P8-NEXT:    mtvsrwz v4, r8
+; CHECK-BE-P8-NEXT:    lxvw4x v3, 0, r3
+; CHECK-BE-P8-NEXT:    vperm v2, v2, v4, v3
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v8i16_v4i32:
 ; CHECK-BE-P9:       # %bb.0: # %entry
-; CHECK-BE-P9-NEXT:    sldi r3, r7, 48
-; CHECK-BE-P9-NEXT:    mtvsrws v3, r8
-; CHECK-BE-P9-NEXT:    mtvsrd v2, r3
-; CHECK-BE-P9-NEXT:    vmrghb v2, v2, v3
+; CHECK-BE-P9-NEXT:    addis r3, r2, .LCPI17_0@toc@ha
+; CHECK-BE-P9-NEXT:    mtvsrwz v2, r7
+; CHECK-BE-P9-NEXT:    mtvsrwz v3, r8
+; CHECK-BE-P9-NEXT:    addi r3, r3, .LCPI17_0@toc@l
+; CHECK-BE-P9-NEXT:    lxv v4, 0(r3)
+; CHECK-BE-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v8i16_v4i32:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    sldi r3, r3, 48
-; CHECK-AIX-64-P8-NEXT:    sldi r4, r4, 32
-; CHECK-AIX-64-P8-NEXT:    mtvsrd v2, r3
-; CHECK-AIX-64-P8-NEXT:    mtvsrd v3, r4
-; CHECK-AIX-64-P8-NEXT:    vmrghb v2, v2, v3
+; CHECK-AIX-64-P8-NEXT:    ld r5, L..C13(r2) # %const.0
+; CHECK-AIX-64-P8-NEXT:    mtvsrwz v2, r3
+; CHECK-AIX-64-P8-NEXT:    mtvsrwz v4, r4
+; CHECK-AIX-64-P8-NEXT:    lxvw4x v3, 0, r5
+; CHECK-AIX-64-P8-NEXT:    vperm v2, v2, v4, v3
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v8i16_v4i32:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
-; CHECK-AIX-64-P9-NEXT:    sldi r3, r3, 48
-; CHECK-AIX-64-P9-NEXT:    mtvsrws v3, r4
-; CHECK-AIX-64-P9-NEXT:    mtvsrd v2, r3
-; CHECK-AIX-64-P9-NEXT:    vmrghb v2, v2, v3
+; CHECK-AIX-64-P9-NEXT:    mtvsrwz v2, r3
+; CHECK-AIX-64-P9-NEXT:    ld r3, L..C9(r2) # %const.0
+; CHECK-AIX-64-P9-NEXT:    mtvsrwz v3, r4
+; CHECK-AIX-64-P9-NEXT:    lxv v4, 0(r3)
+; CHECK-AIX-64-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v8i16_v4i32:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    sth r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    addi r3, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x v2, 0, r5
entry:
   %rhs = bitcast <4 x i32> %rhs.tmp to <16 x i8>
   %shuffle = shufflevector <16 x i8> %lhs, <16 x i8> %rhs, <16 x i32> <i32 0, i32 16, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   ret <16 x i8> %shuffle
 }
 
 define <16 x i8> @test_v8i16_v2i64(<8 x i16> %a, <2 x i64> %b, i16 %arg, i64 %arg1) {
 ; CHECK-LE-P8-LABEL: test_v8i16_v2i64:
 ; CHECK-LE-P8:       # %bb.0: # %entry
-; CHECK-LE-P8-NEXT:    mtfprd f0, r7
-; CHECK-LE-P8-NEXT:    mtfprd f1, r8
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v3, vs1
-; CHECK-LE-P8-NEXT:    vmrglb v2, v3, v2
+; CHECK-LE-P8-NEXT:    mtvsrd v2, r7
+; CHECK-LE-P8-NEXT:    mtvsrd v3, r8
+; CHECK-LE-P8-NEXT:    vmrghb v2, v3, v2
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v8i16_v2i64:
 ; CHECK-LE-P9:       # %bb.0: # %entry
-; CHECK-LE-P9-NEXT:    mtfprd f0, r7
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P9-NEXT:    mtfprd f0, r8
-; CHECK-LE-P9-NEXT:    xxswapd v3, vs0
-; CHECK-LE-P9-NEXT:    vmrglb v2, v3, v2
+; CHECK-LE-P9-NEXT:    mtvsrd v2, r7
+; CHECK-LE-P9-NEXT:    mtvsrd v3, r8
+; CHECK-LE-P9-NEXT:    vmrghb v2, v3, v2
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v8i16_v2i64:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    sldi r3, r7, 48
 ; CHECK-BE-P8-NEXT:    mtvsrd v3, r8
 ; CHECK-BE-P8-NEXT:    mtvsrd v2, r3
 ; CHECK-BE-P8-NEXT:    vmrghb v2, v2, v3
entry:
   %lhs.tmp = insertelement <4 x i32> %a, i32 %arg, i32 0
   %lhs = bitcast <4 x i32> %lhs.tmp to <16 x i8>
   %rhs.tmp = insertelement <4 x i32> %b, i32 %arg1, i32 0
   %rhs = bitcast <4 x i32> %rhs.tmp to <16 x i8>
   %shuffle = shufflevector <16 x i8> %lhs, <16 x i8> %rhs, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 16, i32 17, i32 18, i32 19, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   ret <16 x i8> %shuffle
 }
 
 define <16 x i8> @test_v4i32_v8i16(i32 %arg, i16 %arg1) {
 ; CHECK-LE-P8-LABEL: test_v4i32_v8i16:
 ; CHECK-LE-P8:       # %bb.0: # %entry
-; CHECK-LE-P8-NEXT:    mtfprd f0, r3
-; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v3, vs1
-; CHECK-LE-P8-NEXT:    vmrglh v2, v3, v2
+; CHECK-LE-P8-NEXT:    mtvsrwz v2, r3
+; CHECK-LE-P8-NEXT:    mtvsrd v3, r4
+; CHECK-LE-P8-NEXT:    vmrghh v2, v3, v2
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v4i32_v8i16:
 ; CHECK-LE-P9:       # %bb.0: # %entry
-; CHECK-LE-P9-NEXT:    mtfprd f0, r4
-; CHECK-LE-P9-NEXT:    mtvsrws v2, r3
-; CHECK-LE-P9-NEXT:    xxswapd v3, vs0
-; CHECK-LE-P9-NEXT:    vmrglh v2, v3, v2
+; CHECK-LE-P9-NEXT:    mtvsrwz v2, r3
+; CHECK-LE-P9-NEXT:    mtvsrd v3, r4
+; CHECK-LE-P9-NEXT:    vmrghh v2, v3, v2
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v4i32_v8i16:
 ; CHECK-BE-P8:       # %bb.0: # %entry
-; CHECK-BE-P8-NEXT:    sldi r3, r3, 32
-; CHECK-BE-P8-NEXT:    sldi r4, r4, 48
-; CHECK-BE-P8-NEXT:    mtvsrd v2, r3
-; CHECK-BE-P8-NEXT:    mtvsrd v3, r4
-; CHECK-BE-P8-NEXT:    vmrghh v2, v2, v3
+; CHECK-BE-P8-NEXT:    addis r5, r2, .LCPI20_0@toc@ha
+; CHECK-BE-P8-NEXT:    mtvsrwz v2, r3
+; CHECK-BE-P8-NEXT:    addi r5, r5, .LCPI20_0@toc@l
+; CHECK-BE-P8-NEXT:    mtvsrwz v4, r4
+; CHECK-BE-P8-NEXT:    lxvw4x v3, 0, r5
+; CHECK-BE-P8-NEXT:    vperm v2, v2, v4, v3
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v4i32_v8i16:
 ; CHECK-BE-P9:       # %bb.0: # %entry
-; CHECK-BE-P9-NEXT:    mtvsrws v2, r3
-; CHECK-BE-P9-NEXT:    sldi r3, r4, 48
-; CHECK-BE-P9-NEXT:    mtvsrd v3, r3
-; CHECK-BE-P9-NEXT:    vmrghh v2, v2, v3
+; CHECK-BE-P9-NEXT:    mtvsrwz v2, r3
+; CHECK-BE-P9-NEXT:    addis r3, r2, .LCPI20_0@toc@ha
+; CHECK-BE-P9-NEXT:    mtvsrwz v3, r4
+; CHECK-BE-P9-NEXT:    addi r3, r3, .LCPI20_0@toc@l
+; CHECK-BE-P9-NEXT:    lxv v4, 0(r3)
+; CHECK-BE-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v4i32_v8i16:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    sldi r3, r3, 32
-; CHECK-AIX-64-P8-NEXT:    sldi r4, r4, 48
-; CHECK-AIX-64-P8-NEXT:    mtvsrd v2, r3
-; CHECK-AIX-64-P8-NEXT:    mtvsrd v3, r4
-; CHECK-AIX-64-P8-NEXT:    vmrghh v2, v2, v3
+; CHECK-AIX-64-P8-NEXT:    ld r5, L..C14(r2) # %const.0
+; CHECK-AIX-64-P8-NEXT:    mtvsrwz v2, r3
+; CHECK-AIX-64-P8-NEXT:    mtvsrwz v4, r4
+; CHECK-AIX-64-P8-NEXT:    lxvw4x v3, 0, r5
+; CHECK-AIX-64-P8-NEXT:    vperm v2, v2, v4, v3
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v4i32_v8i16:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
-; CHECK-AIX-64-P9-NEXT:    mtvsrws v2, r3
-; CHECK-AIX-64-P9-NEXT:    sldi r3, r4, 48
-; CHECK-AIX-64-P9-NEXT:    mtvsrd v3, r3
-; CHECK-AIX-64-P9-NEXT:    vmrghh v2, v2, v3
+; CHECK-AIX-64-P9-NEXT:    mtvsrwz v2, r3
+; CHECK-AIX-64-P9-NEXT:    ld r3, L..C10(r2) # %const.0
+; CHECK-AIX-64-P9-NEXT:    mtvsrwz v3, r4
+; CHECK-AIX-64-P9-NEXT:    lxv v4, 0(r3)
+; CHECK-AIX-64-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v4i32_v8i16:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stw r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    addi r3, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x v2, 0, r5
entry:
   %shuffle = shufflevector <16 x i8> %2, <16 x i8> %3, <16 x i32> <i32 0, i32 1, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   ret <16 x i8> %shuffle
 }
 
 define <16 x i8> @test_v2i64_v4i32(i64 %arg, i32 %arg1, <2 x i64> %a, <4 x i32> %b) {
 ; CHECK-LE-P8-LABEL: test_v2i64_v4i32:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
-; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P8-NEXT:    xxswapd vs1, vs1
-; CHECK-LE-P8-NEXT:    xxmrglw v2, vs1, vs0
+; CHECK-LE-P8-NEXT:    mtfprwz f1, r4
+; CHECK-LE-P8-NEXT:    xxmrghw v2, vs1, vs0
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v2i64_v4i32:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    mtvsrws vs1, r4
-; CHECK-LE-P9-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P9-NEXT:    xxmrglw v2, vs1, vs0
+; CHECK-LE-P9-NEXT:    mtfprwz f1, r4
+; CHECK-LE-P9-NEXT:    xxmrghw v2, vs1, vs0
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v2i64_v4i32:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-BE-P8-NEXT:    sldi r3, r4, 32
 ; CHECK-BE-P8-NEXT:    mtfprd f1, r3
 ; CHECK-BE-P8-NEXT:    xxmrghw v2, vs0, vs1
entry:
   %rhs = bitcast <4 x i32> %rhs.tmp to <16 x i8>
   %shuffle = shufflevector <16 x i8> %lhs, <16 x i8> %rhs, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 16, i32 17, i32 18, i32 19, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   ret <16 x i8> %shuffle
 }
 
 define <16 x i8> @test_v2i64_v8i16(i64 %arg, i16 %arg1) {
 ; CHECK-LE-P8-LABEL: test_v2i64_v8i16:
 ; CHECK-LE-P8:       # %bb.0: # %entry
-; CHECK-LE-P8-NEXT:    mtfprd f0, r3
-; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v3, vs1
-; CHECK-LE-P8-NEXT:    vmrglh v2, v3, v2
+; CHECK-LE-P8-NEXT:    mtvsrd v2, r3
+; CHECK-LE-P8-NEXT:    mtvsrd v3, r4
+; CHECK-LE-P8-NEXT:    vmrghh v2, v3, v2
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v2i64_v8i16:
 ; CHECK-LE-P9:       # %bb.0: # %entry
-; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P9-NEXT:    mtfprd f0, r4
-; CHECK-LE-P9-NEXT:    xxswapd v3, vs0
-; CHECK-LE-P9-NEXT:    vmrglh v2, v3, v2
+; CHECK-LE-P9-NEXT:    mtvsrd v2, r3
+; CHECK-LE-P9-NEXT:    mtvsrd v3, r4
+; CHECK-LE-P9-NEXT:    vmrghh v2, v3, v2
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v2i64_v8i16:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    mtvsrd v2, r3
 ; CHECK-BE-P8-NEXT:    sldi r3, r4, 48
 ; CHECK-BE-P8-NEXT:    mtvsrd v3, r3
 ; CHECK-BE-P8-NEXT:    vmrghh v2, v2, v3
entry:
   %shuffle = shufflevector <16 x i8> %lhs, <16 x i8> %rhs, <16 x i32> <i32 0, i32 1, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   ret <16 x i8> %shuffle
 }
 
 define <16 x i8> @test_v4i32_v2i64(ptr nocapture noundef readonly %a, ptr nocapture noundef readonly %b) {
 ; CHECK-LE-P8-LABEL: test_v4i32_v2i64:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    addis r5, r2, .LCPI24_0@toc@ha
-; CHECK-LE-P8-NEXT:    lfiwzx f0, 0, r3
-; CHECK-LE-P8-NEXT:    lfdx f1, 0, r4
-; CHECK-LE-P8-NEXT:    addi r3, r5, .LCPI24_0@toc@l
-; CHECK-LE-P8-NEXT:    lxvd2x vs2, 0, r3
-; CHECK-LE-P8-NEXT:    xxswapd v2, f0
-; CHECK-LE-P8-NEXT:    xxswapd v3, f1
-; CHECK-LE-P8-NEXT:    xxswapd v4, vs2
+; CHECK-LE-P8-NEXT:    lxsiwzx v2, 0, r3
+; CHECK-LE-P8-NEXT:    lxsdx v3, 0, r4
+; CHECK-LE-P8-NEXT:    addi r5, r5, .LCPI24_0@toc@l
+; CHECK-LE-P8-NEXT:    lxvd2x vs0, 0, r5
+; CHECK-LE-P8-NEXT:    xxswapd v4, vs0
 ; CHECK-LE-P8-NEXT:    vperm v2, v3, v2, v4
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v4i32_v2i64:
 ; CHECK-LE-P9:       # %bb.0: # %entry
-; CHECK-LE-P9-NEXT:    lfiwzx f0, 0, r3
+; CHECK-LE-P9-NEXT:    lxsiwzx v2, 0, r3
 ; CHECK-LE-P9-NEXT:    addis r3, r2, .LCPI24_0@toc@ha
+; CHECK-LE-P9-NEXT:    lxsd v3, 0(r4)
 ; CHECK-LE-P9-NEXT:    addi r3, r3, .LCPI24_0@toc@l
-; CHECK-LE-P9-NEXT:    xxswapd v2, f0
-; CHECK-LE-P9-NEXT:    lfd f0, 0(r4)
 ; CHECK-LE-P9-NEXT:    lxv v4, 0(r3)
-; CHECK-LE-P9-NEXT:    xxswapd v3, f0
 ; CHECK-LE-P9-NEXT:    vperm v2, v3, v2, v4
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v4i32_v2i64:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    lfiwzx f0, 0, r3
 ; CHECK-BE-P8-NEXT:    addis r5, r2, .LCPI24_0@toc@ha
 ; CHECK-BE-P8-NEXT:    lxsdx v3, 0, r4

 ; CHECK-BE-P9-NEXT:    addi r3, r3, .LCPI24_0@toc@l
 ; CHECK-BE-P9-NEXT:    lxv v4, 0(r3)
 ; CHECK-BE-P9-NEXT:    xxsldwi v2, f0, f0, 1
 ; CHECK-BE-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v4i32_v2i64:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    ld r5, L..C9(r2) # %const.0
+; CHECK-AIX-64-P8-NEXT:    ld r5, L..C15(r2) # %const.0
 ; CHECK-AIX-64-P8-NEXT:    lfiwzx f0, 0, r3
 ; CHECK-AIX-64-P8-NEXT:    lxsdx v3, 0, r4
 ; CHECK-AIX-64-P8-NEXT:    xxsldwi v2, f0, f0, 1
 ; CHECK-AIX-64-P8-NEXT:    lxvw4x v4, 0, r5
 ; CHECK-AIX-64-P8-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v4i32_v2i64:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    lfiwzx f0, 0, r3
-; CHECK-AIX-64-P9-NEXT:    ld r3, L..C5(r2) # %const.0
+; CHECK-AIX-64-P9-NEXT:    ld r3, L..C11(r2) # %const.0
 ; CHECK-AIX-64-P9-NEXT:    lxsd v3, 0(r4)
 ; CHECK-AIX-64-P9-NEXT:    xxsldwi v2, f0, f0, 1
 ; CHECK-AIX-64-P9-NEXT:    lxv v4, 0(r3)
 ; CHECK-AIX-64-P9-NEXT:    vperm v2, v2, v3, v4
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v4i32_v2i64:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry

llvm/test/CodeGen/PowerPC/v2i64_scalar_to_vector_shuffle.ll

 ; RUN:   -ppc-vsr-nums-as-vr -mtriple=powerpc-ibm-aix < %s | \
 ; RUN: FileCheck %s --check-prefix=CHECK-AIX-32-P9
 
 define <2 x i64> @test_v16i8_v16i8(i8 %arg1, i8 %arg) {
 ; CHECK-LE-P8-LABEL: test_v16i8_v16i8:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v16i8_v16i8:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P9-NEXT:    mtfprd f1, r4
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v16i8_v16i8:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
 ; CHECK-BE-P8-NEXT:    mtfprwz f1, r4
-; CHECK-BE-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v16i8_v16i8:
 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
 ; CHECK-BE-P9-NEXT:    mtfprwz f1, r4
-; CHECK-BE-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v16i8_v16i8:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f1, r4
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v16i8_v16i8:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f1, r4
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v16i8_v16i8:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stb r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    addi r3, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5

 }
 
 define <2 x i64> @test_none_v16i8(i8 %arg1, ptr nocapture noundef readonly %b) {
 ; CHECK-LE-P8-LABEL: test_none_v16i8:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    lxvd2x vs0, 0, r4
 ; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs0, v2
+; CHECK-LE-P8-NEXT:    xxpermdi v2, vs0, v2, 1
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_none_v16i8:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs0, v2
+; CHECK-LE-P9-NEXT:    xxpermdi v2, vs0, v2, 1
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_none_v16i8:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    lxvd2x v2, 0, r4
 ; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
-; CHECK-BE-P8-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-BE-P8-NEXT:    xxpermdi v2, v2, vs0, 1
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_none_v16i8:
 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-BE-P9-NEXT:    xxpermdi v2, v2, vs0, 1
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_none_v16i8:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
 ; CHECK-AIX-64-P8-NEXT:    lxvd2x v2, 0, r4
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-AIX-64-P8-NEXT:    xxpermdi v2, v2, vs0, 1
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_none_v16i8:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-AIX-64-P9-NEXT:    xxpermdi v2, v2, vs0, 1
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_none_v16i8:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stb r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5
 ; CHECK-AIX-32-P8-NEXT:    lxvd2x v2, 0, r4

 }
 
 define <2 x i64> @test_v16i8_none(i8 %arg1, ptr nocapture noundef readonly %b) {
 ; CHECK-LE-P8-LABEL: test_v16i8_none:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    lxvd2x vs0, 0, r4
 ; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
-; CHECK-LE-P8-NEXT:    xxmrgld v2, v2, vs0
+; CHECK-LE-P8-NEXT:    xxpermdi v2, v2, vs0, 2
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v16i8_none:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    xxmrgld v2, v2, vs0
+; CHECK-LE-P9-NEXT:    xxpermdi v2, v2, vs0, 2
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v16i8_none:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    lxvd2x v2, 0, r4
 ; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
-; CHECK-BE-P8-NEXT:    xxmrghd v2, vs0, v2
+; CHECK-BE-P8-NEXT:    xxpermdi v2, vs0, v2, 2
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v16i8_none:
 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, vs0, v2
+; CHECK-BE-P9-NEXT:    xxpermdi v2, vs0, v2, 2
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v16i8_none:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
 ; CHECK-AIX-64-P8-NEXT:    lxvd2x v2, 0, r4
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs0, v2
+; CHECK-AIX-64-P8-NEXT:    xxpermdi v2, vs0, v2, 2
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v16i8_none:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, vs0, v2
+; CHECK-AIX-64-P9-NEXT:    xxpermdi v2, vs0, v2, 2
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v16i8_none:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stb r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5
 ; CHECK-AIX-32-P8-NEXT:    lxvd2x v2, 0, r4
entry:
   ret <2 x i64> %shuffle
 }
 
 define <2 x i64> @test_v16i8_v8i16(i8 %arg1, i16 %arg) {
 ; CHECK-LE-P8-LABEL: test_v16i8_v8i16:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P8-NEXT:    xxswapd vs1, vs1
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v16i8_v8i16:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P9-NEXT:    mtfprd f1, r4
-; CHECK-LE-P9-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P9-NEXT:    xxswapd vs1, vs1
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v16i8_v8i16:
 ; CHECK-BE-P8:       # %bb.0: # %entry
-; CHECK-BE-P8-NEXT:    sldi r3, r3, 56
-; CHECK-BE-P8-NEXT:    sldi r4, r4, 48
-; CHECK-BE-P8-NEXT:    mtfprd f0, r3
-; CHECK-BE-P8-NEXT:    mtfprd f1, r4
-; CHECK-BE-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
+; CHECK-BE-P8-NEXT:    mtfprwz f1, r4
+; CHECK-BE-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v16i8_v8i16:
 ; CHECK-BE-P9:       # %bb.0: # %entry
-; CHECK-BE-P9-NEXT:    sldi r3, r3, 56
-; CHECK-BE-P9-NEXT:    mtfprd f0, r3
-; CHECK-BE-P9-NEXT:    sldi r3, r4, 48
-; CHECK-BE-P9-NEXT:    mtfprd f1, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
+; CHECK-BE-P9-NEXT:    mtfprwz f1, r4
+; CHECK-BE-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v16i8_v8i16:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    sldi r3, r3, 56
-; CHECK-AIX-64-P8-NEXT:    sldi r4, r4, 48
-; CHECK-AIX-64-P8-NEXT:    mtfprd f0, r3
-; CHECK-AIX-64-P8-NEXT:    mtfprd f1, r4
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
+; CHECK-AIX-64-P8-NEXT:    mtfprwz f1, r4
+; CHECK-AIX-64-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v16i8_v8i16:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
-; CHECK-AIX-64-P9-NEXT:    sldi r3, r3, 56
-; CHECK-AIX-64-P9-NEXT:    mtfprd f0, r3
-; CHECK-AIX-64-P9-NEXT:    sldi r3, r4, 48
-; CHECK-AIX-64-P9-NEXT:    mtfprd f1, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
+; CHECK-AIX-64-P9-NEXT:    mtfprwz f1, r4
+; CHECK-AIX-64-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v16i8_v8i16:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stb r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    addi r3, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5
entry:
   ret <2 x i64> %shuffle
 }
 
 define <2 x i64> @test_v8i16_v16i8(i8 %arg1, i16 %arg) {
 ; CHECK-LE-P8-LABEL: test_v8i16_v16i8:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P8-NEXT:    xxswapd vs1, vs1
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs0, vs1
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs0, vs1
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v8i16_v16i8:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P9-NEXT:    mtfprd f1, r4
-; CHECK-LE-P9-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P9-NEXT:    xxswapd vs1, vs1
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs0, vs1
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs0, vs1
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v8i16_v16i8:
 ; CHECK-BE-P8:       # %bb.0: # %entry
-; CHECK-BE-P8-NEXT:    sldi r3, r3, 56
-; CHECK-BE-P8-NEXT:    sldi r4, r4, 48
-; CHECK-BE-P8-NEXT:    mtfprd f0, r3
-; CHECK-BE-P8-NEXT:    mtfprd f1, r4
-; CHECK-BE-P8-NEXT:    xxmrghd v2, vs1, vs0
+; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
+; CHECK-BE-P8-NEXT:    mtfprwz f1, r4
+; CHECK-BE-P8-NEXT:    xxmrgld v2, vs1, vs0
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v8i16_v16i8:
 ; CHECK-BE-P9:       # %bb.0: # %entry
-; CHECK-BE-P9-NEXT:    sldi r3, r3, 56
-; CHECK-BE-P9-NEXT:    mtfprd f0, r3
-; CHECK-BE-P9-NEXT:    sldi r3, r4, 48
-; CHECK-BE-P9-NEXT:    mtfprd f1, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, vs1, vs0
+; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
+; CHECK-BE-P9-NEXT:    mtfprwz f1, r4
+; CHECK-BE-P9-NEXT:    xxmrgld v2, vs1, vs0
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v8i16_v16i8:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    sldi r3, r3, 56
-; CHECK-AIX-64-P8-NEXT:    sldi r4, r4, 48
-; CHECK-AIX-64-P8-NEXT:    mtfprd f0, r3
-; CHECK-AIX-64-P8-NEXT:    mtfprd f1, r4
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs1, vs0
+; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
+; CHECK-AIX-64-P8-NEXT:    mtfprwz f1, r4
+; CHECK-AIX-64-P8-NEXT:    xxmrgld v2, vs1, vs0
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v8i16_v16i8:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
-; CHECK-AIX-64-P9-NEXT:    sldi r3, r3, 56
-; CHECK-AIX-64-P9-NEXT:    mtfprd f0, r3
-; CHECK-AIX-64-P9-NEXT:    sldi r3, r4, 48
-; CHECK-AIX-64-P9-NEXT:    mtfprd f1, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, vs1, vs0
+; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
+; CHECK-AIX-64-P9-NEXT:    mtfprwz f1, r4
+; CHECK-AIX-64-P9-NEXT:    xxmrgld v2, vs1, vs0
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v8i16_v16i8:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stb r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    addi r3, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5

 }
 
 define <2 x i64> @test_v8i16_none(i16 %arg1, ptr nocapture noundef readonly %b) {
 ; CHECK-LE-P8-LABEL: test_v8i16_none:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    lxvd2x vs0, 0, r4
 ; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
-; CHECK-LE-P8-NEXT:    xxmrgld v2, v2, vs0
+; CHECK-LE-P8-NEXT:    xxpermdi v2, v2, vs0, 2
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v8i16_none:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    xxmrgld v2, v2, vs0
+; CHECK-LE-P9-NEXT:    xxpermdi v2, v2, vs0, 2
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v8i16_none:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    lxvd2x v2, 0, r4
 ; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
-; CHECK-BE-P8-NEXT:    xxmrghd v2, vs0, v2
+; CHECK-BE-P8-NEXT:    xxpermdi v2, vs0, v2, 2
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v8i16_none:
 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, vs0, v2
+; CHECK-BE-P9-NEXT:    xxpermdi v2, vs0, v2, 2
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v8i16_none:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
 ; CHECK-AIX-64-P8-NEXT:    lxvd2x v2, 0, r4
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs0, v2
+; CHECK-AIX-64-P8-NEXT:    xxpermdi v2, vs0, v2, 2
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v8i16_none:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, vs0, v2
+; CHECK-AIX-64-P9-NEXT:    xxpermdi v2, vs0, v2, 2
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v8i16_none:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    sth r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5
 ; CHECK-AIX-32-P8-NEXT:    lxvd2x v2, 0, r4

 }
 
 define <2 x i64> @test_none_v8i16(i16 %arg1, ptr nocapture noundef readonly %b) {
 ; CHECK-LE-P8-LABEL: test_none_v8i16:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    lxvd2x vs0, 0, r4
 ; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs0, v2
+; CHECK-LE-P8-NEXT:    xxpermdi v2, vs0, v2, 1
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_none_v8i16:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs0, v2
+; CHECK-LE-P9-NEXT:    xxpermdi v2, vs0, v2, 1
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_none_v8i16:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    lxvd2x v2, 0, r4
 ; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
-; CHECK-BE-P8-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-BE-P8-NEXT:    xxpermdi v2, v2, vs0, 1
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_none_v8i16:
 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-BE-P9-NEXT:    xxpermdi v2, v2, vs0, 1
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_none_v8i16:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
 ; CHECK-AIX-64-P8-NEXT:    lxvd2x v2, 0, r4
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-AIX-64-P8-NEXT:    xxpermdi v2, v2, vs0, 1
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_none_v8i16:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-AIX-64-P9-NEXT:    xxpermdi v2, v2, vs0, 1
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_none_v8i16:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    sth r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5
 ; CHECK-AIX-32-P8-NEXT:    lxvd2x v2, 0, r4
entry:
   %shuffle = shufflevector <2 x i64> %rhs, <2 x i64> %lhs, <2 x i32> <i32 0, i32 2>
   ret <2 x i64> %shuffle
 }
 
 define <2 x i64> @test_v16i8_v4i32(i8 %arg1, i32 %arg) {
 ; CHECK-LE-P8-LABEL: test_v16i8_v4i32:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
-; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P8-NEXT:    xxswapd vs1, vs1
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P8-NEXT:    mtfprwz f1, r4
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v16i8_v4i32:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    mtvsrws vs1, r4
-; CHECK-LE-P9-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P9-NEXT:    mtfprwz f1, r4
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v16i8_v4i32:
 ; CHECK-BE-P8:       # %bb.0: # %entry
-; CHECK-BE-P8-NEXT:    sldi r3, r3, 56
-; CHECK-BE-P8-NEXT:    sldi r4, r4, 32
-; CHECK-BE-P8-NEXT:    mtfprd f0, r3
-; CHECK-BE-P8-NEXT:    mtfprd f1, r4
-; CHECK-BE-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
+; CHECK-BE-P8-NEXT:    mtfprwz f1, r4
+; CHECK-BE-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v16i8_v4i32:
 ; CHECK-BE-P9:       # %bb.0: # %entry
-; CHECK-BE-P9-NEXT:    sldi r3, r3, 56
-; CHECK-BE-P9-NEXT:    mtvsrws vs1, r4
-; CHECK-BE-P9-NEXT:    mtfprd f0, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
+; CHECK-BE-P9-NEXT:    mtfprwz f1, r4
+; CHECK-BE-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v16i8_v4i32:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    sldi r3, r3, 56
-; CHECK-AIX-64-P8-NEXT:    sldi r4, r4, 32
-; CHECK-AIX-64-P8-NEXT:    mtfprd f0, r3
-; CHECK-AIX-64-P8-NEXT:    mtfprd f1, r4
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
+; CHECK-AIX-64-P8-NEXT:    mtfprwz f1, r4
+; CHECK-AIX-64-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v16i8_v4i32:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
-; CHECK-AIX-64-P9-NEXT:    sldi r3, r3, 56
-; CHECK-AIX-64-P9-NEXT:    mtvsrws vs1, r4
-; CHECK-AIX-64-P9-NEXT:    mtfprd f0, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
+; CHECK-AIX-64-P9-NEXT:    mtfprwz f1, r4
+; CHECK-AIX-64-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v16i8_v4i32:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stb r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    addi r3, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5
entry:
   %shuffle = shufflevector <2 x i64> %lhs, <2 x i64> %rhs, <2 x i32> <i32 0, i32 2>
   ret <2 x i64> %shuffle
 }
 
 define <2 x i64> @test_v4i32_v16i8(i8 %arg1, i32 %arg) {
 ; CHECK-LE-P8-LABEL: test_v4i32_v16i8:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
-; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P8-NEXT:    xxswapd vs1, vs1
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs0, vs1
+; CHECK-LE-P8-NEXT:    mtfprwz f1, r4
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs0, vs1
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v4i32_v16i8:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    mtvsrws vs1, r4
-; CHECK-LE-P9-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs0, vs1
+; CHECK-LE-P9-NEXT:    mtfprwz f1, r4
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs0, vs1
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v4i32_v16i8:
 ; CHECK-BE-P8:       # %bb.0: # %entry
-; CHECK-BE-P8-NEXT:    sldi r3, r3, 56
-; CHECK-BE-P8-NEXT:    sldi r4, r4, 32
-; CHECK-BE-P8-NEXT:    mtfprd f0, r3
-; CHECK-BE-P8-NEXT:    mtfprd f1, r4
-; CHECK-BE-P8-NEXT:    xxmrghd v2, vs1, vs0
+; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
+; CHECK-BE-P8-NEXT:    mtfprwz f1, r4
+; CHECK-BE-P8-NEXT:    xxmrgld v2, vs1, vs0
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v4i32_v16i8:
 ; CHECK-BE-P9:       # %bb.0: # %entry
-; CHECK-BE-P9-NEXT:    sldi r3, r3, 56
-; CHECK-BE-P9-NEXT:    mtvsrws vs1, r4
-; CHECK-BE-P9-NEXT:    mtfprd f0, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, vs1, vs0
+; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
+; CHECK-BE-P9-NEXT:    mtfprwz f1, r4
+; CHECK-BE-P9-NEXT:    xxmrgld v2, vs1, vs0
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v4i32_v16i8:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    sldi r3, r3, 56
-; CHECK-AIX-64-P8-NEXT:    sldi r4, r4, 32
-; CHECK-AIX-64-P8-NEXT:    mtfprd f0, r3
-; CHECK-AIX-64-P8-NEXT:    mtfprd f1, r4
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs1, vs0
+; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
+; CHECK-AIX-64-P8-NEXT:    mtfprwz f1, r4
+; CHECK-AIX-64-P8-NEXT:    xxmrgld v2, vs1, vs0
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v4i32_v16i8:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
-; CHECK-AIX-64-P9-NEXT:    sldi r3, r3, 56
-; CHECK-AIX-64-P9-NEXT:    mtvsrws vs1, r4
-; CHECK-AIX-64-P9-NEXT:    mtfprd f0, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, vs1, vs0
+; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
+; CHECK-AIX-64-P9-NEXT:    mtfprwz f1, r4
+; CHECK-AIX-64-P9-NEXT:    xxmrgld v2, vs1, vs0
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v4i32_v16i8:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stb r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    addi r3, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5

 }
 
 define <2 x i64> @test_none_v4i32(i32 %arg1, ptr nocapture noundef readonly %b) {
 ; CHECK-LE-P8-LABEL: test_none_v4i32:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    lxvd2x vs0, 0, r4
 ; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
 ; CHECK-LE-P8-NEXT:    mtfprwz f0, r3
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs0, v2
+; CHECK-LE-P8-NEXT:    xxpermdi v2, vs0, v2, 1
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_none_v4i32:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-LE-P9-NEXT:    mtfprwz f0, r3
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs0, v2
+; CHECK-LE-P9-NEXT:    xxpermdi v2, vs0, v2, 1
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_none_v4i32:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    lxvd2x v2, 0, r4
 ; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
-; CHECK-BE-P8-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-BE-P8-NEXT:    xxpermdi v2, v2, vs0, 1
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_none_v4i32:
 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-BE-P9-NEXT:    xxpermdi v2, v2, vs0, 1
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_none_v4i32:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
 ; CHECK-AIX-64-P8-NEXT:    lxvd2x v2, 0, r4
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-AIX-64-P8-NEXT:    xxpermdi v2, v2, vs0, 1
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_none_v4i32:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-AIX-64-P9-NEXT:    xxpermdi v2, v2, vs0, 1
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_none_v4i32:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stw r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5
 ; CHECK-AIX-32-P8-NEXT:    lxvd2x v2, 0, r4

 }
 
 define <2 x i64> @test_v4i32_none(i32 %arg1, ptr nocapture noundef readonly %b) {
 ; CHECK-LE-P8-LABEL: test_v4i32_none:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    lxvd2x vs0, 0, r4
 ; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
 ; CHECK-LE-P8-NEXT:    mtfprwz f0, r3
-; CHECK-LE-P8-NEXT:    xxmrgld v2, v2, vs0
+; CHECK-LE-P8-NEXT:    xxpermdi v2, v2, vs0, 2
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v4i32_none:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-LE-P9-NEXT:    mtfprwz f0, r3
-; CHECK-LE-P9-NEXT:    xxmrgld v2, v2, vs0
+; CHECK-LE-P9-NEXT:    xxpermdi v2, v2, vs0, 2
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v4i32_none:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    lxvd2x v2, 0, r4
 ; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
-; CHECK-BE-P8-NEXT:    xxmrghd v2, vs0, v2
+; CHECK-BE-P8-NEXT:    xxpermdi v2, vs0, v2, 2
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v4i32_none:
 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, vs0, v2
+; CHECK-BE-P9-NEXT:    xxpermdi v2, vs0, v2, 2
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v4i32_none:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
 ; CHECK-AIX-64-P8-NEXT:    lxvd2x v2, 0, r4
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs0, v2
+; CHECK-AIX-64-P8-NEXT:    xxpermdi v2, vs0, v2, 2
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v4i32_none:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    lxv v2, 0(r4)
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, vs0, v2
+; CHECK-AIX-64-P9-NEXT:    xxpermdi v2, vs0, v2, 2
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v4i32_none:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stw r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5
 ; CHECK-AIX-32-P8-NEXT:    lxvd2x v2, 0, r4
entry:
   ret <2 x i64> %shuffle
 }
 
 define <2 x i64> @test_v16i8_v2i64(i8 %arg1, i64 %arg) {
 ; CHECK-LE-P8-LABEL: test_v16i8_v2i64:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs1
-; CHECK-LE-P8-NEXT:    xxmrgld v2, v2, vs0
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v16i8_v2i64:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P9-NEXT:    mtfprd f1, r4
-; CHECK-LE-P9-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs1
-; CHECK-LE-P9-NEXT:    xxmrgld v2, v2, vs0
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v16i8_v2i64:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    sldi r3, r3, 56
 ; CHECK-BE-P8-NEXT:    mtfprd f1, r4
 ; CHECK-BE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-BE-P8-NEXT:    xxmrghd v2, vs0, vs1
entry:
   ret <2 x i64> %shuffle
 }
 
 define <2 x i64> @test_v2i64_v16i8(i8 %arg1, i64 %arg) {
 ; CHECK-LE-P8-LABEL: test_v2i64_v16i8:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs1
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs0, v2
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs0, vs1
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v2i64_v16i8:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P9-NEXT:    mtfprd f1, r4
-; CHECK-LE-P9-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs1
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs0, v2
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs0, vs1
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v2i64_v16i8:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    mtfprd f0, r4
 ; CHECK-BE-P8-NEXT:    xxspltd v2, vs0, 0
 ; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
-; CHECK-BE-P8-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-BE-P8-NEXT:    xxmrgld v2, v2, vs0
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v2i64_v16i8:
 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
 ; CHECK-BE-P9-NEXT:    mtvsrdd v2, r4, r4
-; CHECK-BE-P9-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-BE-P9-NEXT:    xxmrgld v2, v2, vs0
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v2i64_v16i8:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
 ; CHECK-AIX-64-P8-NEXT:    mtfprd f0, r4
 ; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs0, vs0
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-AIX-64-P8-NEXT:    xxmrgld v2, v2, vs0
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v2i64_v16i8:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
 ; CHECK-AIX-64-P9-NEXT:    mtvsrdd v2, r4, r4
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-AIX-64-P9-NEXT:    xxmrgld v2, v2, vs0
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v2i64_v16i8:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r6, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stb r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    addi r3, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r6
entry:
   ret <2 x i64> %shuffle
 }
 
 define <2 x i64> @test_v8i16_v8i16(i16 %arg1, i16 %arg) {
 ; CHECK-LE-P8-LABEL: test_v8i16_v8i16:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v8i16_v8i16:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P9-NEXT:    mtfprd f1, r4
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v8i16_v8i16:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
 ; CHECK-BE-P8-NEXT:    mtfprwz f1, r4
-; CHECK-BE-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v8i16_v8i16:
 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
 ; CHECK-BE-P9-NEXT:    mtfprwz f1, r4
-; CHECK-BE-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v8i16_v8i16:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f1, r4
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v8i16_v8i16:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f1, r4
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v8i16_v8i16:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    sth r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    addi r3, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5
entry:
   %shuffle = shufflevector <2 x i64> %lhs, <2 x i64> %rhs, <2 x i32> <i32 0, i32 2>
   ret <2 x i64> %shuffle
 }
 
 define <2 x i64> @test_v8i16_v4i32(i16 %arg1, i32 %arg) {
 ; CHECK-LE-P8-LABEL: test_v8i16_v4i32:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
-; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P8-NEXT:    xxswapd vs1, vs1
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P8-NEXT:    mtfprwz f1, r4
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v8i16_v4i32:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    mtvsrws vs1, r4
-; CHECK-LE-P9-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P9-NEXT:    mtfprwz f1, r4
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v8i16_v4i32:
 ; CHECK-BE-P8:       # %bb.0: # %entry
-; CHECK-BE-P8-NEXT:    sldi r3, r3, 48
-; CHECK-BE-P8-NEXT:    sldi r4, r4, 32
-; CHECK-BE-P8-NEXT:    mtfprd f0, r3
-; CHECK-BE-P8-NEXT:    mtfprd f1, r4
-; CHECK-BE-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
+; CHECK-BE-P8-NEXT:    mtfprwz f1, r4
+; CHECK-BE-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v8i16_v4i32:
 ; CHECK-BE-P9:       # %bb.0: # %entry
-; CHECK-BE-P9-NEXT:    sldi r3, r3, 48
-; CHECK-BE-P9-NEXT:    mtvsrws vs1, r4
-; CHECK-BE-P9-NEXT:    mtfprd f0, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
+; CHECK-BE-P9-NEXT:    mtfprwz f1, r4
+; CHECK-BE-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v8i16_v4i32:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    sldi r3, r3, 48
-; CHECK-AIX-64-P8-NEXT:    sldi r4, r4, 32
-; CHECK-AIX-64-P8-NEXT:    mtfprd f0, r3
-; CHECK-AIX-64-P8-NEXT:    mtfprd f1, r4
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
+; CHECK-AIX-64-P8-NEXT:    mtfprwz f1, r4
+; CHECK-AIX-64-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v8i16_v4i32:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
-; CHECK-AIX-64-P9-NEXT:    sldi r3, r3, 48
-; CHECK-AIX-64-P9-NEXT:    mtvsrws vs1, r4
-; CHECK-AIX-64-P9-NEXT:    mtfprd f0, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
+; CHECK-AIX-64-P9-NEXT:    mtfprwz f1, r4
+; CHECK-AIX-64-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v8i16_v4i32:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    sth r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    addi r3, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5
entry:
   ret <2 x i64> %shuffle
 }
 
 define <2 x i64> @test_v8i16_v2i64(i16 %arg1, i64 %arg) {
 ; CHECK-LE-P8-LABEL: test_v8i16_v2i64:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs1
-; CHECK-LE-P8-NEXT:    xxmrgld v2, v2, vs0
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v8i16_v2i64:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P9-NEXT:    mtfprd f1, r4
-; CHECK-LE-P9-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs1
-; CHECK-LE-P9-NEXT:    xxmrgld v2, v2, vs0
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v8i16_v2i64:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    sldi r3, r3, 48
 ; CHECK-BE-P8-NEXT:    mtfprd f1, r4
 ; CHECK-BE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-BE-P8-NEXT:    xxmrghd v2, vs0, vs1
entry:
   ret <2 x i64> %shuffle
 }
 
 define <2 x i64> @test_v4i32_v4i32(i32 %arg1, i32 %arg) {
 ; CHECK-LE-P8-LABEL: test_v4i32_v4i32:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprwz f0, r3
 ; CHECK-LE-P8-NEXT:    mtfprwz f1, r4
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v4i32_v4i32:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprwz f0, r3
 ; CHECK-LE-P9-NEXT:    mtfprwz f1, r4
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v4i32_v4i32:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
 ; CHECK-BE-P8-NEXT:    mtfprwz f1, r4
-; CHECK-BE-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v4i32_v4i32:
 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
 ; CHECK-BE-P9-NEXT:    mtfprwz f1, r4
-; CHECK-BE-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v4i32_v4i32:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f1, r4
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v4i32_v4i32:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f1, r4
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v4i32_v4i32:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stw r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    addi r3, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5
entry:
   %rhs = bitcast <4 x i32> %rhs.tmp to <2 x i64>
   %shuffle = shufflevector <2 x i64> %lhs, <2 x i64> %rhs, <2 x i32> <i32 0, i32 2>
   ret <2 x i64> %shuffle
 }
 
 define <2 x i64> @test_v4i32_v8i16(i32 %arg1, i16 %arg) {
 ; CHECK-LE-P8-LABEL: test_v4i32_v8i16:
 ; CHECK-LE-P8:       # %bb.0: # %entry
-; CHECK-LE-P8-NEXT:    mtfprd f0, r3
+; CHECK-LE-P8-NEXT:    mtfprwz f0, r3
 ; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P8-NEXT:    xxswapd vs1, vs1
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v4i32_v8i16:
 ; CHECK-LE-P9:       # %bb.0: # %entry
+; CHECK-LE-P9-NEXT:    mtfprwz f0, r3
 ; CHECK-LE-P9-NEXT:    mtfprd f1, r4
-; CHECK-LE-P9-NEXT:    mtvsrws vs0, r3
-; CHECK-LE-P9-NEXT:    xxswapd vs1, vs1
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs1, vs0
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v4i32_v8i16:
 ; CHECK-BE-P8:       # %bb.0: # %entry
-; CHECK-BE-P8-NEXT:    sldi r3, r3, 32
-; CHECK-BE-P8-NEXT:    sldi r4, r4, 48
-; CHECK-BE-P8-NEXT:    mtfprd f0, r3
-; CHECK-BE-P8-NEXT:    mtfprd f1, r4
-; CHECK-BE-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P8-NEXT:    mtfprwz f0, r3
+; CHECK-BE-P8-NEXT:    mtfprwz f1, r4
+; CHECK-BE-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v4i32_v8i16:
 ; CHECK-BE-P9:       # %bb.0: # %entry
-; CHECK-BE-P9-NEXT:    mtvsrws vs0, r3
-; CHECK-BE-P9-NEXT:    sldi r3, r4, 48
-; CHECK-BE-P9-NEXT:    mtfprd f1, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-BE-P9-NEXT:    mtfprwz f0, r3
+; CHECK-BE-P9-NEXT:    mtfprwz f1, r4
+; CHECK-BE-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v4i32_v8i16:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
-; CHECK-AIX-64-P8-NEXT:    sldi r3, r3, 32
-; CHECK-AIX-64-P8-NEXT:    sldi r4, r4, 48
-; CHECK-AIX-64-P8-NEXT:    mtfprd f0, r3
-; CHECK-AIX-64-P8-NEXT:    mtfprd f1, r4
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r3
+; CHECK-AIX-64-P8-NEXT:    mtfprwz f1, r4
+; CHECK-AIX-64-P8-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v4i32_v8i16:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
-; CHECK-AIX-64-P9-NEXT:    mtvsrws vs0, r3
-; CHECK-AIX-64-P9-NEXT:    sldi r3, r4, 48
-; CHECK-AIX-64-P9-NEXT:    mtfprd f1, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, vs0, vs1
+; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r3
+; CHECK-AIX-64-P9-NEXT:    mtfprwz f1, r4
+; CHECK-AIX-64-P9-NEXT:    xxmrgld v2, vs0, vs1
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v4i32_v8i16:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r5, r1, -16
 ; CHECK-AIX-32-P8-NEXT:    stw r3, -16(r1)
 ; CHECK-AIX-32-P8-NEXT:    addi r3, r1, -32
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r5
entry:
   %rhs = bitcast <8 x i16> %rhs.tmp to <2 x i64>
   %shuffle = shufflevector <2 x i64> %lhs, <2 x i64> %rhs, <2 x i32> <i32 0, i32 2>
   ret <2 x i64> %shuffle
 }
 
 define <2 x i64> @test_v4i32_v2i64(i32 %arg1, i64 %arg) {
 ; CHECK-LE-P8-LABEL: test_v4i32_v2i64:
 ; CHECK-LE-P8:       # %bb.0: # %entry
-; CHECK-LE-P8-NEXT:    mtfprd f0, r3
+; CHECK-LE-P8-NEXT:    mtfprwz f0, r3
 ; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs1
-; CHECK-LE-P8-NEXT:    xxmrgld v2, v2, vs0
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v4i32_v2i64:
 ; CHECK-LE-P9:       # %bb.0: # %entry
+; CHECK-LE-P9-NEXT:    mtfprwz f0, r3
 ; CHECK-LE-P9-NEXT:    mtfprd f1, r4
-; CHECK-LE-P9-NEXT:    mtvsrws vs0, r3
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs1
-; CHECK-LE-P9-NEXT:    xxmrgld v2, v2, vs0
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v4i32_v2i64:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    sldi r3, r3, 32
 ; CHECK-BE-P8-NEXT:    mtfprd f1, r4
 ; CHECK-BE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-BE-P8-NEXT:    xxmrghd v2, vs0, vs1
entry:
   %4 = add <2 x i64> %shuffle, %2
   ret <2 x i64> %4
 }
 
 define <2 x i64> @test_v2i64_v4i32(i64 %arg1, i32 %arg) {
 ; CHECK-LE-P8-LABEL: test_v2i64_v4i32:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
-; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P8-NEXT:    xxswapd vs0, vs1
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs0, v2
+; CHECK-LE-P8-NEXT:    mtfprwz f1, r4
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v2i64_v4i32:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P9-NEXT:    mtvsrws vs0, r4
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs0, v2
+; CHECK-LE-P9-NEXT:    mtfprwz f1, r4
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v2i64_v4i32:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-BE-P8-NEXT:    xxspltd v2, vs0, 0
 ; CHECK-BE-P8-NEXT:    mtfprwz f0, r4
-; CHECK-BE-P8-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-BE-P8-NEXT:    xxmrgld v2, v2, vs0
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v2i64_v4i32:
 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    mtfprwz f0, r4
 ; CHECK-BE-P9-NEXT:    mtvsrdd v2, r3, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-BE-P9-NEXT:    xxmrgld v2, v2, vs0
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v2i64_v4i32:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
 ; CHECK-AIX-64-P8-NEXT:    mtfprd f0, r3
 ; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs0, vs0
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r4
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-AIX-64-P8-NEXT:    xxmrgld v2, v2, vs0
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v2i64_v4i32:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r4
 ; CHECK-AIX-64-P9-NEXT:    mtvsrdd v2, r3, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-AIX-64-P9-NEXT:    xxmrgld v2, v2, vs0
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v2i64_v4i32:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r6, r1, -48
 ; CHECK-AIX-32-P8-NEXT:    stw r5, -48(r1)
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r6
 ; CHECK-AIX-32-P8-NEXT:    stw r4, -16(r1)
entry:
   ret <2 x i64> %shuffle
 }
 
 define <2 x i64> @test_v2i64_v8i16(i64 %arg1, i16 %arg) {
 ; CHECK-LE-P8-LABEL: test_v2i64_v8i16:
 ; CHECK-LE-P8:       # %bb.0: # %entry
 ; CHECK-LE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-LE-P8-NEXT:    mtfprd f1, r4
-; CHECK-LE-P8-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P8-NEXT:    xxswapd vs0, vs1
-; CHECK-LE-P8-NEXT:    xxmrgld v2, vs0, v2
+; CHECK-LE-P8-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P8-NEXT:    blr
 ;
 ; CHECK-LE-P9-LABEL: test_v2i64_v8i16:
 ; CHECK-LE-P9:       # %bb.0: # %entry
 ; CHECK-LE-P9-NEXT:    mtfprd f0, r3
-; CHECK-LE-P9-NEXT:    xxswapd v2, vs0
-; CHECK-LE-P9-NEXT:    mtfprd f0, r4
-; CHECK-LE-P9-NEXT:    xxswapd vs0, vs0
-; CHECK-LE-P9-NEXT:    xxmrgld v2, vs0, v2
+; CHECK-LE-P9-NEXT:    mtfprd f1, r4
+; CHECK-LE-P9-NEXT:    xxmrghd v2, vs1, vs0
 ; CHECK-LE-P9-NEXT:    blr
 ;
 ; CHECK-BE-P8-LABEL: test_v2i64_v8i16:
 ; CHECK-BE-P8:       # %bb.0: # %entry
 ; CHECK-BE-P8-NEXT:    mtfprd f0, r3
 ; CHECK-BE-P8-NEXT:    xxspltd v2, vs0, 0
 ; CHECK-BE-P8-NEXT:    mtfprwz f0, r4
-; CHECK-BE-P8-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-BE-P8-NEXT:    xxmrgld v2, v2, vs0
 ; CHECK-BE-P8-NEXT:    blr
 ;
 ; CHECK-BE-P9-LABEL: test_v2i64_v8i16:
 ; CHECK-BE-P9:       # %bb.0: # %entry
 ; CHECK-BE-P9-NEXT:    mtfprwz f0, r4
 ; CHECK-BE-P9-NEXT:    mtvsrdd v2, r3, r3
-; CHECK-BE-P9-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-BE-P9-NEXT:    xxmrgld v2, v2, vs0
 ; CHECK-BE-P9-NEXT:    blr
 ;
 ; CHECK-AIX-64-P8-LABEL: test_v2i64_v8i16:
 ; CHECK-AIX-64-P8:       # %bb.0: # %entry
 ; CHECK-AIX-64-P8-NEXT:    mtfprd f0, r3
 ; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, vs0, vs0
 ; CHECK-AIX-64-P8-NEXT:    mtfprwz f0, r4
-; CHECK-AIX-64-P8-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-AIX-64-P8-NEXT:    xxmrgld v2, v2, vs0
 ; CHECK-AIX-64-P8-NEXT:    blr
 ;
 ; CHECK-AIX-64-P9-LABEL: test_v2i64_v8i16:
 ; CHECK-AIX-64-P9:       # %bb.0: # %entry
 ; CHECK-AIX-64-P9-NEXT:    mtfprwz f0, r4
 ; CHECK-AIX-64-P9-NEXT:    mtvsrdd v2, r3, r3
-; CHECK-AIX-64-P9-NEXT:    xxmrghd v2, v2, vs0
+; CHECK-AIX-64-P9-NEXT:    xxmrgld v2, v2, vs0
 ; CHECK-AIX-64-P9-NEXT:    blr
 ;
 ; CHECK-AIX-32-P8-LABEL: test_v2i64_v8i16:
 ; CHECK-AIX-32-P8:       # %bb.0: # %entry
 ; CHECK-AIX-32-P8-NEXT:    addi r6, r1, -48
 ; CHECK-AIX-32-P8-NEXT:    sth r5, -48(r1)
 ; CHECK-AIX-32-P8-NEXT:    lxvw4x vs0, 0, r6
 ; CHECK-AIX-32-P8-NEXT:    stw r4, -16(r1)