[PowerPC] Canonicalize shuffles to match more single-instruction masks on LE

[nemanjai]

Summary

We currently miss a number of opportunities to emit single-instruction VMRG[LH][BHW] instructions for shuffles on little endian subtargets. Although this in itself is not a huge performance opportunity since loading the permute vector for a VPERM can always be pulled out of loops, producing such merge instructions is useful to downstream optimizations.
Since VPERM is essentially opaque to all subsequent optimizations, we want to avoid it as much as possible. Other permute instructions have semantics that can be reasoned about much more easily in later optimizations.

This patch does the following:

• Canonicalize shuffles so that the first element comes from the first vector (since that's what most of the mask matching functions want)
• Switch the elements that come from splat vectors so that they match the corresponding elements from the other vector (to allow for merges)
• Adds debugging messages for when a shuffle is matched to a VPERM so that anyone interested in improving this further can get the info for their code

[nemanjai]

Updated to reflect changes in the pre-committed new test case.

[amyk]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13850

I see that our other combine functions have an assert in the beginning checking the opcode. It might make sense to have one here checking SVN->getOpcode() == ISD::VECTOR_SHUFFLE?

13868

Is it possible to add a comment regarding the splats here? Or is the comment above supposed to explain this bit, too?

13884

s/condusive/conducive

[nemanjai]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13850

The reason for the asserts in other combines is that they do not take a specialized node. This takes a ShuffleVectorSDNode pointer. It is not possible for a shuffle vector node to have an opcode other than vector shuffle.

[nemanjai]

Add handling for shuffles that are fed by SCALAR_TO_VECTOR to avoid the swap before the shuffle.

[lei]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13840

Can this and the for-loop below be a range based for-loop?

for (auto FirstOp : Op->op_values())
  if (!FirstOp.isUndef())
    break;

llvm/lib/Target/PowerPC/PPCISelLowering.h

226

Maybe consider using existing naming used for scalar and vector ISD nodes: SCALAR_TO_VEC_PERMUTED

[RolandF]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

9426

I don't think it is safe to use a load and splat for a smaller than word sized load. The larger load might now cross a page boundary.

13880

This code is only used on the early, conditional returns, and not on the final return. The function could be refactored such that the final return case is moved first, or the early returns could return the result of a function that generates the instruction, to prevent unused code.

[RolandF]

llvm/test/CodeGen/PowerPC/build-vector-tests.ll

1723 ↗

(On Diff #260001)

Looks like the code got worse here?

3243 ↗

(On Diff #260001)

This code is also worse.

4918 ↗

(On Diff #260001)

Okay it looks like there are a number of similar cases where the code is slightly worse. I will stop flagging them individually.

[RolandF]

llvm/lib/Target/PowerPC/PPCISelLowering.h

226

Can we rename this opcode? S and V are too short to have meaning and permute is too general. Maybe SCALAR_TO_VECTOR_BE or _RIGHT or _UPPER or something?

[NeHuang]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13871

nit: Is it possible to combine line 14068 and 14066 into one if check since the two operations are same?

13909

nit: Little -> little

[steven.zhang]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13840

Can we use llvm::any_of to make the code more compact ?

[nemanjai]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

9426

I think this code is a bit confusing because of the <=. I actually don't really know why I used <= rather than == since the memory value type can never be narrower than the splat width since we are checking for normal (unindexed, non-extending) loads.

13840

How? I need to find the first operand that is *not* undef.

13840

But then FirstOp is out of scope after the loop and we need it.

13880

I am not sure this would be an improvement. The early exit conditions are the common case. The final return is only for the pattern:
(<n x Ty> (scalar_to_vector (Ty (extract_elt <n x Ty> %a, C)))).

I'll refactor it as per your suggestion and if it is more readable we'll go with that.
Unused nodes aren't really a problem - the SDAG will just get rid of them, but I agree that it is less than ideal to create nodes that will just be discarded.

llvm/lib/Target/PowerPC/PPCISelLowering.h

226

SCALAR_TO_VECTOR_BE is close but the problem is that SCALAR_TO_VECTOR assumes the scalar is placed into vector element zero. However, (almost) none of the instructions we have actually put the scalar into vector element zero for either endianness. The value always goes to the least significant portion of the most significant doubleword. It is very hard to encapsulate that level of weirdness in a name.

I think I will opt for either SCALAR_TO_VECTOR_PERMUTED or SCALAR_TO_VECTOR_PPC with a more detailed comments:

/// PowerPC instructions that have SCALAR_TO_VECTOR semantics tend to
/// place the value into the least significant element of the most significant
/// doubleword in the vector. This is not element zero for anything smaller
/// than a doubleword on either endianness. This node has the same semantics
/// as SCALAR_TO_VECTOR except that the value remains in the
/// aforementioned location in the vector register.

If you have a preference for either of those, please let me know.

[anil9]

llvm/lib/Target/PowerPC/PPCInstrVSX.td

3325 ↗

(On Diff #260001)

Indentation discrepancy.

3553 ↗

(On Diff #260001)

Indentation discrepancy.

4471 ↗

(On Diff #260001)

Same as before.

[nemanjai]

• Fix some cases where code got worse
• Refactor some code for mode readability
• Fix some naming and comments

[steven.zhang]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13840

Ah, sorry. Maybe, llvm::find_if_not is the right one. But it seems not help too much for this case. So, it depends on you.

[RolandF]

LGTM

[nathanchance]

Hi! This patch causes a crash when compiling the Linux kernel with certain distribution configs:

$ git bisect log
# bad: [b885b1b92d3d9bfcc8229476f4ad4660305d6160] [mlir] Fix gcc build break due to previous commit
# good: [158e734af19d6be206f80c213a028b569c441b24] [ARM] Adjust AND/OR combines to not call isConstantSplat on i1 vectors. NFC.
git bisect start 'b885b1b92d3d9bfcc8229476f4ad4660305d6160' '158e734af19d6be206f80c213a028b569c441b24'
# good: [fcd67665a8de61223313e1e1582faf17d9ee76b8] [StackSafety] Add "Must Live" logic
git bisect good fcd67665a8de61223313e1e1582faf17d9ee76b8
# bad: [d3b752845df0331348dad48000fc8b82afb3de5b] [clang][test][NFC] Also test for serialization in AST dump tests, part 1/n.
git bisect bad d3b752845df0331348dad48000fc8b82afb3de5b
# bad: [bb480056602daab86fbcd6aac5c6bc92ce350bb3] [NFC] Make AST_BLOCK_HASH test more robust with downstream changes
git bisect bad bb480056602daab86fbcd6aac5c6bc92ce350bb3
# bad: [d938ec4509c47d461377527fc2877ae14b91275c] [AArch64] Avoid incompatibility between SLSBLR mitigation and BTI codegen.
git bisect bad d938ec4509c47d461377527fc2877ae14b91275c
# good: [9ca50e887db7f903c04a90593d2beed8a96794f1] [libTooling] Add parser for string representation of `RangeSelector`.
git bisect good 9ca50e887db7f903c04a90593d2beed8a96794f1
# bad: [1fed131660b2c5d3ea7007e273a7a5da80699445] [PowerPC] Canonicalize shuffles to match more single-instruction masks on LE
git bisect bad 1fed131660b2c5d3ea7007e273a7a5da80699445
# good: [9c9b71a2908d47ebd65cb7c2e1e499484aaa547e] [gn build] Port 9ca50e887db
git bisect good 9c9b71a2908d47ebd65cb7c2e1e499484aaa547e
# good: [8f3b2c8aa3175628128a32a6bcaecc67efd03514] AMDGPU/GlobalISel: Remove selection of MAD/MAC when not available
git bisect good 8f3b2c8aa3175628128a32a6bcaecc67efd03514
# first bad commit: [1fed131660b2c5d3ea7007e273a7a5da80699445] [PowerPC] Canonicalize shuffles to match more single-instruction masks on LE

To reproduce with the Linux kernel source tree:

$ mkdir -p out/ppc64le

$ curl -LSso out/ppc64le/.config 'https://git.kernel.org/pub/scm/linux/kernel/git/jwboyer/fedora.git/plain/fedora/configs/kernel-5.6.17-ppc64le.config?h=kernel-5.6.17-300.fc32'

$ make -skj"$(nproc)" ARCH=powerpc CROSS_COMPILE=powerpc64le-linux-gnu- LD=powerpc64le-linux-gnu-ld LLVM=1 O=out/ppc64le OBJDUMP=powerpc64le-linux-gnu-objdump olddefconfig drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn20/display_mode_vba_20.o
...
fatal error: error in backend: Cannot select: t146: v16i8 = PPCISD::SCALAR_TO_VECTOR_PERMUTED t110, drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn20/display_mode_vba_20.c:2855:3 @[ drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn20/display_mode_vba_20.c:241:2 ]
  t110: i32 = any_extend t32, drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn20/display_mode_vba_20.c:2855:3 @[ drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn20/display_mode_vba_20.c:241:2 ]
    t32: i1,ch = CopyFromReg t0, Register:i1 %31, drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn20/display_mode_vba_20.c:2855:3 @[ drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn20/display_mode_vba_20.c:241:2 ]
      t31: i1 = Register %31
In function: dml20_recalculate
clang-11: error: clang frontend command failed with exit code 70 (use -v to see invocation)
ClangBuiltLinux clang version 11.0.0 (https://github.com/llvm/llvm-project b885b1b92d3d9bfcc8229476f4ad4660305d6160)
...

cvise spits out (not cleaned up or anything):

enum { a, b } c;
d;
e() {
  _Bool f = a;
  for (; d; ++d)
    if (c)
      f = b;
  if (f)
    g();
}

Interestingness test and original preprocessed file available here: https://github.com/nathanchance/creduce-files/tree/ac42e9b24f969b3bd8a72d0a8be0bccb88e570d2/D77448

[nathanchance]

Gentle ping. This is still an issue at 4c6548222b3c41d024581d28f42b3f02510bcfe3 and I have not heard anything about a fix. I know I sent that message on Friday night before the weekend so I am hoping that it did not get lost. If I need to file a bug report, I am more than happy to do so.

[nemanjai]

In D77448#2116050, @nathanchance wrote:

Gentle ping. This is still an issue at 4c6548222b3c41d024581d28f42b3f02510bcfe3 and I have not heard anything about a fix. I know I sent that message on Friday night before the weekend so I am hoping that it did not get lost. If I need to file a bug report, I am more than happy to do so.

I am really sorry I missed this message. Should be fixed in https://reviews.llvm.org/rG57ad8f4.

[nathanchance]

@nemanjai thanks for the fix, I can confirm all of my PowerPC builds are successful with LLVM at 4772b99dffec4f87bb7bc9273495066058ac0186!

[MaskRay]

Our internal testing finds a correctness bug related to this patch (after the last known fix eafe7c14ea38946e8c1fb64d548effaee7614718)

The code is at https://github.com/google/dimsum/blob/master/dimsum_fuzz.cc#L402

With the following patch it can build with -mcpu=pwr9 -O0. When I feed the executable dimsum_fuzz itself to dimsum_fuzz, it triggers __builtin_trap because hmin (horizontal min) on typedef signed char simd_char __attribute__((vector_size__(8))); computed result is different from a naive algorithm. I'll inform you when I can get a minimized reproduce or in the meantime if you can find problem that'd also be nice:)

diff --git i/dimsum_fuzz.cc w/dimsum_fuzz.cc
index fafd1fa..b7188fd 100644
--- i/dimsum_fuzz.cc
+++ w/dimsum_fuzz.cc
@@ -532 +532,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
 }
+
+int main() {
+  const long data[] = {0x00010102464c457f, 0x0000000000000000};
+  LLVMFuzzerTestOneInput((const uint8_t*)data, 16);
+}
diff --git i/simd_vsx.h w/simd_vsx.h
index 5c59e2a..15a32fe 100644
--- i/simd_vsx.h
+++ w/simd_vsx.h
@@ -23,2 +23,6 @@
 
+inline __vector long vec_abs(__vector long a) {
+  return (__vector long)vec_abs((__vector long long)a);
+}
+
 namespace dimsum {

cgdb session

2004x typename std::enable_if<(_SimdType::size() > 1 && is_simd<_SimdType>::value &&
2005x                          __floor_pow_of_2(_SimdType::size()) ==
2006x                              _SimdType::size()),
2007x                         typename _SimdType::value_type>::type
2008x __hmin(const _SimdType& __v) {
2009x   auto __arr = split_by<2>(__v);
2010x   return __hmin(min(__arr[0], __arr[1]));                                                                                                       
2011x }
2012x
2013x template <class _SimdType>
2014x typename std::enable_if<(_SimdType::size() > 1 &&
2015x                          !(is_simd<_SimdType>::value &&
2016x                            __floor_pow_of_2(_SimdType::size()) ==
/home/maskray/test/src/third_party/dimsum/simd.h                                                                                                      
t third_party/dimsum/simd.h:2079
0x0000000010083c40 in std::experimental::__hmin<std::experimental::simd<signed char, std::experimental::__simd_abi<(std::experimental::_StorageKind)2,
 16> > > (__v=...) at third_party/dimsum/simd.h:2010
Value returned is $2 = <incomplete type>
(gdb) s
std::experimental::__hmin<std::experimental::simd<signed char, std::experimental::__simd_abi<(std::experimental::_StorageKind)2, 8> > > (__v=...) at t
hird_party/dimsum/simd.h:2009
(gdb) n
(gdb) x/2xg &__v
0x7fffffffed88: 0x0000000000000000      0x00010102464c457f
(gdb) x/2xg &__arr
0x7fffffffecf8: 0x00007fff00000000      0x00007fffffffed88
(gdb) p sizeof(__arr)            # under the hood this is a signed char vector_size(8) type
$3 = 8
(gdb) 

------

142x template <typename T>
143x void TestHMin(const uint8_t* data) {
144x   NativeSimd<T> simd;
145x   LoadFromRaw(data, &simd);
146x   if (!IsNormal(simd)) return;
147x   if (dimsum::simulated::hmin(simd) != dimsum::hmin(simd))     ######   dimsum::hmin(simd) is wrong                                                                                
148x     __builtin_trap();
149x }

[MaskRay]

@nemanjai See https://gist.github.com/MaskRay/ba7547f6a587176666ff77527cf12c4d for a reproduce.

[nemanjai]

In D77448#2165900, @MaskRay wrote:

@nemanjai See https://gist.github.com/MaskRay/ba7547f6a587176666ff77527cf12c4d for a reproduce.

Fixed in https://reviews.llvm.org/rG7d076e19e31a
I'll wait for it to go through the bots and then I'll open a PR to ask for a merge into 11.0. Thanks for reporting this and providing the repro.

llvm/lib/Target/PowerPC/PPCISelLowering.h

enum NodeType : unsigned {
     SINT_VEC_TO_FP,
 
     /// Extract a subvector from unsigned integer vector and convert to FP.
     /// As with SINT_VEC_TO_FP, used for converting illegal types.
     UINT_VEC_TO_FP,
 
     // FIXME: Remove these once the ANDI glue bug is fixed:
     /// i1 = ANDI_rec_1_[EQ|GT]_BIT(i32 or i64 x) - Represents the result of the
     /// eq or gt bit of CR0 after executing andi. x, 1. This is used to
     /// implement truncation of i32 or i64 to i1.
     ANDI_rec_1_EQ_BIT,
     ANDI_rec_1_GT_BIT,
 
     // READ_TIME_BASE - A read of the 64-bit time-base register on a 32-bit
     // target (returns (Lo, Hi)). It takes a chain operand.
     READ_TIME_BASE,
 
private:
     SDValue combineSRA(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineSRL(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineMUL(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineADD(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineTRUNCATE(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineSetCC(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineABS(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineVSelect(SDNode *N, DAGCombinerInfo &DCI) const;
+    SDValue combineVectorShuffle(ShuffleVectorSDNode *SVN,
+                                 SelectionDAG &DAG) const;
     SDValue combineVReverseMemOP(ShuffleVectorSDNode *SVN, LSBaseSDNode *LSBase,
                                  DAGCombinerInfo &DCI) const;
 
     /// ConvertSETCCToSubtract - looks at SETCC that compares ints. It replaces
     /// SETCC with integer subtraction when (1) there is a legal way of doing it
     /// (2) keeping the result of comparison in GPR has performance benefit.
     SDValue ConvertSETCCToSubtract(SDNode *N, DAGCombinerInfo &DCI) const;
 

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

 static cl::opt<bool> EnableQuadPrecision("enable-ppc-quad-precision",
 cl::desc("enable quad precision float support on ppc"), cl::Hidden);
 
 static cl::opt<bool> UseAbsoluteJumpTables("ppc-use-absolute-jumptables",
 cl::desc("use absolute jump tables on ppc"), cl::Hidden);
 
 STATISTIC(NumTailCalls, "Number of tail calls");
 STATISTIC(NumSiblingCalls, "Number of sibling calls");
+STATISTIC(ShufflesHandledWithVPERM, "Number of shuffles lowered to a VPERM");
 
 static bool isNByteElemShuffleMask(ShuffleVectorSDNode *, unsigned, int);
 
 static SDValue widenVec(SelectionDAG &DAG, SDValue Vec, const SDLoc &dl);
 
 // FIXME: Remove this once the bug has been fixed!
 extern cl::opt<bool> ANDIGlueBug;
 

 /// return the code it can be lowered into.  Worst case, it can always be
 /// lowered into a vperm.
 SDValue PPCTargetLowering::LowerVECTOR_SHUFFLE(SDValue Op,
                                                SelectionDAG &DAG) const {
   SDLoc dl(Op);
   SDValue V1 = Op.getOperand(0);
   SDValue V2 = Op.getOperand(1);
   ShuffleVectorSDNode *SVOp = cast<ShuffleVectorSDNode>(Op);
+
+  // Any nodes that were combined in the target-independent combiner prior
+  // to vector legalization will not be sent to the target combine. Try to
+  // combine it here.
+  if (SDValue NewShuffle = combineVectorShuffle(SVOp, DAG)) {
+    DAG.ReplaceAllUsesOfValueWith(Op, NewShuffle);
+    Op = NewShuffle;
+    SVOp = cast<ShuffleVectorSDNode>(Op);
+  }
   EVT VT = Op.getValueType();
   bool isLittleEndian = Subtarget.isLittleEndian();
 
   unsigned ShiftElts, InsertAtByte;
   bool Swap = false;
 
   // If this is a load-and-splat, we can do that with a single instruction
   // in some cases. However if the load has multiple uses, we don't want to
if ((IsFourByte && Subtarget.hasP9Vector()) || !IsFourByte) {
       uint64_t Offset = 0;
       if (IsFourByte)
         Offset = isLittleEndian ? (3 - SplatIdx) * 4 : SplatIdx * 4;
       else
         Offset = isLittleEndian ? (1 - SplatIdx) * 8 : SplatIdx * 8;
       SDValue BasePtr = LD->getBasePtr();
       if (Offset != 0)
         BasePtr = DAG.getNode(ISD::ADD, dl, getPointerTy(DAG.getDataLayout()),
                               BasePtr, DAG.getIntPtrConstant(Offset, dl));
       SDValue Ops[] = {
         LD->getChain(),    // Chain
         BasePtr,           // BasePtr
         DAG.getValueType(Op.getValueType()) // VT
       };
       SDVTList VTL =
         DAG.getVTList(IsFourByte ? MVT::v4i32 : MVT::v2i64, MVT::Other);
       SDValue LdSplt =
for (unsigned j = 0; j != BytesPerElement; ++j)
       if (isLittleEndian)
         ResultMask.push_back(DAG.getConstant(31 - (SrcElt*BytesPerElement + j),
                                              dl, MVT::i32));
       else
         ResultMask.push_back(DAG.getConstant(SrcElt*BytesPerElement + j, dl,
                                              MVT::i32));
   }
 
+  ShufflesHandledWithVPERM++;
   SDValue VPermMask = DAG.getBuildVector(MVT::v16i8, dl, ResultMask);
+  LLVM_DEBUG(dbgs() << "Emitting a VPERM for the following shuffle:\n");
+  LLVM_DEBUG(SVOp->dump());
+  LLVM_DEBUG(dbgs() << "With the following permute control vector:\n");
+  LLVM_DEBUG(VPermMask.dump());
+
   if (isLittleEndian)
     return DAG.getNode(PPCISD::VPERM, dl, V1.getValueType(),
                        V2, V1, VPermMask);
   else
     return DAG.getNode(PPCISD::VPERM, dl, V1.getValueType(),
                        V1, V2, VPermMask);
 }
 
Val = DAG.getMemIntrinsicNode(PPCISD::ST_VSR_SCAL_INT, dl,
           DAG.getVTList(MVT::Other), Ops,
           cast<StoreSDNode>(N)->getMemoryVT(),
           cast<StoreSDNode>(N)->getMemOperand());
 
   DCI.AddToWorklist(Val.getNode());
   return Val;
 }
 
+static bool isAlternatingShuffMask(const ArrayRef<int> &Mask, int NumElts) {
+  // Check that the source of the element keeps flipping
+  // (i.e. Mask[i] < NumElts -> Mask[i+i] >= NumElts).
+  bool PrevElemFromFirstVec = Mask[0] < NumElts;
+  for (int i = 1, e = Mask.size(); i < e; i++) {
+    if (PrevElemFromFirstVec && Mask[i] < NumElts)
+      return false;
+    if (!PrevElemFromFirstVec && Mask[i] >= NumElts)
+      return false;
+    PrevElemFromFirstVec = !PrevElemFromFirstVec;
+  }
+  return true;
+}
+
+static bool isSplatBV(SDValue Op) {
+  if (Op.getOpcode() != ISD::BUILD_VECTOR)
+    return false;
+  SDValue FirstOp = Op.getOperand(0);
+  for (int i = 1, e = Op.getNumOperands(); i < e; i++)
+    if (Op.getOperand(i) != FirstOp)
+      return false;
+  return true;
+}
+
+// On little endian subtargets, combine shuffles such as:
+// vector_shuffle<16,1,17,3,18,5,19,7,20,9,21,11,22,13,23,15>, <zero>, %b
+// into:
+// vector_shuffle<16,0,17,1,18,2,19,3,20,4,21,5,22,6,23,7>, <zero>, %b
+// because the latter can be matched to a single instruction merge.
+SDValue PPCTargetLowering::combineVectorShuffle(ShuffleVectorSDNode *SVN,
+                                                SelectionDAG &DAG) const {
+  SDValue LHS = SVN->getOperand(0);
+  SDValue RHS = SVN->getOperand(1);
+  auto Mask = SVN->getMask();
+  int NumElts = LHS.getValueType().getVectorNumElements();
+  SDValue Res;
+  SDLoc dl(SVN);
+
+  // If this is not a shuffle of a shuffle and the first element comes from
+  // the second vector, canonicalize to the commuted form. This will make it
+  // more likely to match one of the single instruction patterns.
+  if (Subtarget.isLittleEndian() && Mask[0] >= NumElts &&
+      LHS.getOpcode() != ISD::VECTOR_SHUFFLE &&
+      RHS.getOpcode() != ISD::VECTOR_SHUFFLE) {
+    std::swap(LHS, RHS);
+    Res = DAG.getCommutedVectorShuffle(*SVN);
+    Mask = cast<ShuffleVectorSDNode>(Res)->getMask();
+  }
+  bool IsLHSSplat = isSplatBV(LHS);
+  bool IsRHSSplat = isSplatBV(RHS);
+  if (!IsLHSSplat && !IsRHSSplat)
+    return Res;
+
+  // We are looking for a mask such that all even elements are from
+  // one vector and all odd elements from the other.
+  if (!isAlternatingShuffMask(Mask, NumElts))
+    return Res;
+
+  SmallVector<int, 16> ShuffV(Mask.size());
+  for (int i = 0, e = Mask.size(); i < e; i++)
+    ShuffV[i] = Mask[i];
+
+  // The common case after we commuted the shuffle is that the RHS is a splat
+  // and we have elements coming in from the splat at indices that are not
+  // condusive to using a merge.
+  // Example:
+  // vector_shuffle<0,17,1,19,2,21,3,23,4,25,5,27,6,29,7,31> t1, <zero>
+  if (!IsRHSSplat || Mask[0] >= NumElts)
+    return Res;
+
+  // Adjust the mask so we are pulling in the same index from the splat
+  // as the index from the interesting vector in consecutive elements.
+  // Example:
+  // vector_shuffle<0,16,1,17,2,18,3,19,4,20,5,21,6,22,7,23> t1, <zero>
+  for (int i = 1, e = Mask.size(); i < e; i += 2)
+    ShuffV[i] = (ShuffV[i - 1] + NumElts);
+
+  Res = DAG.getVectorShuffle(SVN->getValueType(0), dl, LHS, RHS, ShuffV);
+  return Res;
+}
+
 SDValue PPCTargetLowering::combineVReverseMemOP(ShuffleVectorSDNode *SVN,
                                                 LSBaseSDNode *LSBase,
                                                 DAGCombinerInfo &DCI) const {
   assert((ISD::isNormalLoad(LSBase) || ISD::isNormalStore(LSBase)) &&
         "Not a reverse memop pattern!");
 
   auto IsElementReverse = [](const ShuffleVectorSDNode *SVN) -> bool {
     auto Mask = SVN->getMask();
     int i = 0;
     auto I = Mask.rbegin();
     auto E = Mask.rend();
 
     for (; I != E; ++I) {
       if (*I != i)
         return false;
       i++;
     }
SDValue PPCTargetLowering::PerformDAGCombine(SDNode *N,
   case ISD::SINT_TO_FP:
   case ISD::UINT_TO_FP:
     return combineFPToIntToFP(N, DCI);
   case ISD::VECTOR_SHUFFLE:
     if (ISD::isNormalLoad(N->getOperand(0).getNode())) {
       LSBaseSDNode* LSBase = cast<LSBaseSDNode>(N->getOperand(0));
       return combineVReverseMemOP(cast<ShuffleVectorSDNode>(N), LSBase, DCI);
     }
-    break;
+    return combineVectorShuffle(cast<ShuffleVectorSDNode>(N), DCI.DAG);
   case ISD::STORE: {
 
     EVT Op1VT = N->getOperand(1).getValueType();
     unsigned Opcode = N->getOperand(1).getOpcode();
 
     if (Opcode == ISD::FP_TO_SINT || Opcode == ISD::FP_TO_UINT) {
       SDValue Val= combineStoreFPToInt(N, DCI);
       if (Val)

llvm/test/CodeGen/PowerPC/canonical-merge-shuffles.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu \
+; RUN:     -mcpu=pwr8 -ppc-asm-full-reg-names -ppc-vsr-nums-as-vr < %s | \
+; RUN: FileCheck %s --check-prefix=CHECK-P8
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu \
+; RUN:     -mcpu=pwr9 -ppc-asm-full-reg-names -ppc-vsr-nums-as-vr < %s | \
+; RUN: FileCheck %s --check-prefix=CHECK-P9
+
+define dso_local <16 x i8> @testmrghb(<16 x i8> %a, <16 x i8> %b) local_unnamed_addr #0 {
+; CHECK-P8-LABEL: testmrghb:
+; CHECK-P8:       # %bb.0: # %entry
+; CHECK-P8-NEXT:    vmrghb v2, v3, v2
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-P9-LABEL: testmrghb:
+; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    vmrghb v2, v3, v2
+; CHECK-P9-NEXT:    blr
+entry:
+  %shuffle = shufflevector <16 x i8> %a, <16 x i8> %b, <16 x i32> <i32 8, i32 24, i32 9, i32 25, i32 10, i32 26, i32 11, i32 27, i32 12, i32 28, i32 13, i32 29, i32 14, i32 30, i32 15, i32 31>
+  ret <16 x i8> %shuffle
+}
+define dso_local <16 x i8> @testmrghb2(<16 x i8> %a, <16 x i8> %b) local_unnamed_addr #0 {
+; CHECK-P8-LABEL: testmrghb2:
+; CHECK-P8:       # %bb.0: # %entry
+; CHECK-P8-NEXT:    vmrghb v2, v2, v3
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-P9-LABEL: testmrghb2:
+; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    vmrghb v2, v2, v3
+; CHECK-P9-NEXT:    blr
+entry:
+  %shuffle = shufflevector <16 x i8> %a, <16 x i8> %b, <16 x i32> <i32 24, i32 8, i32 25, i32 9, i32 26, i32 10, i32 27, i32 11, i32 28, i32 12, i32 29, i32 13, i32 30, i32 14, i32 31, i32 15>
+  ret <16 x i8> %shuffle
+}
+define dso_local <16 x i8> @testmrghh(<16 x i8> %a, <16 x i8> %b) local_unnamed_addr #0 {
+; CHECK-P8-LABEL: testmrghh:
+; CHECK-P8:       # %bb.0: # %entry
+; CHECK-P8-NEXT:    vmrghh v2, v3, v2
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-P9-LABEL: testmrghh:
+; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    vmrghh v2, v3, v2
+; CHECK-P9-NEXT:    blr
+entry:
+  %shuffle = shufflevector <16 x i8> %a, <16 x i8> %b, <16 x i32> <i32 8, i32 9, i32 24, i32 25, i32 10, i32 11, i32 26, i32 27, i32 12, i32 13, i32 28, i32 29, i32 14, i32 15, i32 30, i32 31>
+  ret <16 x i8> %shuffle
+}
+define dso_local <16 x i8> @testmrghh2(<16 x i8> %a, <16 x i8> %b) local_unnamed_addr #0 {
+; CHECK-P8-LABEL: testmrghh2:
+; CHECK-P8:       # %bb.0: # %entry
+; CHECK-P8-NEXT:    vmrghh v2, v2, v3
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-P9-LABEL: testmrghh2:
+; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    vmrghh v2, v2, v3
+; CHECK-P9-NEXT:    blr
+entry:
+  %shuffle = shufflevector <16 x i8> %a, <16 x i8> %b, <16 x i32> <i32 24, i32 25, i32 8, i32 9, i32 26, i32 27, i32 10, i32 11, i32 28, i32 29, i32 12, i32 13, i32 30, i32 31, i32 14, i32 15>
+  ret <16 x i8> %shuffle
+}
+define dso_local <16 x i8> @testmrglb(<16 x i8> %a, <16 x i8> %b) local_unnamed_addr #0 {
+; CHECK-P8-LABEL: testmrglb:
+; CHECK-P8:       # %bb.0: # %entry
+; CHECK-P8-NEXT:    vmrglb v2, v3, v2
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-P9-LABEL: testmrglb:
+; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    vmrglb v2, v3, v2
+; CHECK-P9-NEXT:    blr
+entry:
+  %shuffle = shufflevector <16 x i8> %a, <16 x i8> %b, <16 x i32> <i32 0, i32 16, i32 1, i32 17, i32 2, i32 18, i32 3, i32 19, i32 4, i32 20, i32 5, i32 21, i32 6, i32 22, i32 7, i32 23>
+  ret <16 x i8> %shuffle
+}
+define dso_local <16 x i8> @testmrglb2(<16 x i8> %a, <16 x i8> %b) local_unnamed_addr #0 {
+; CHECK-P8-LABEL: testmrglb2:
+; CHECK-P8:       # %bb.0: # %entry
+; CHECK-P8-NEXT:    vmrglb v2, v2, v3
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-P9-LABEL: testmrglb2:
+; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    vmrglb v2, v2, v3
+; CHECK-P9-NEXT:    blr
+entry:
+  %shuffle = shufflevector <16 x i8> %a, <16 x i8> %b, <16 x i32> <i32 16, i32 0, i32 17, i32 1, i32 18, i32 2, i32 19, i32 3, i32 20, i32 4, i32 21, i32 5, i32 22, i32 6, i32 23, i32 7>
+  ret <16 x i8> %shuffle
+}
+define dso_local <16 x i8> @testmrglh(<16 x i8> %a, <16 x i8> %b) local_unnamed_addr #0 {
+; CHECK-P8-LABEL: testmrglh:
+; CHECK-P8:       # %bb.0: # %entry
+; CHECK-P8-NEXT:    vmrglh v2, v3, v2
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-P9-LABEL: testmrglh:
+; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    vmrglh v2, v3, v2
+; CHECK-P9-NEXT:    blr
+entry:
+  %shuffle = shufflevector <16 x i8> %a, <16 x i8> %b, <16 x i32> <i32 0, i32 1, i32 16, i32 17, i32 2, i32 3, i32 18, i32 19, i32 4, i32 5, i32 20, i32 21, i32 6, i32 7, i32 22, i32 23>
+  ret <16 x i8> %shuffle
+}
+define dso_local <16 x i8> @testmrglh2(<16 x i8> %a, <16 x i8> %b) local_unnamed_addr #0 {
+; CHECK-P8-LABEL: testmrglh2:
+; CHECK-P8:       # %bb.0: # %entry
+; CHECK-P8-NEXT:    vmrglh v2, v2, v3
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-P9-LABEL: testmrglh2:
+; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    vmrglh v2, v2, v3
+; CHECK-P9-NEXT:    blr
+entry:
+  %shuffle = shufflevector <16 x i8> %a, <16 x i8> %b, <16 x i32> <i32 16, i32 17, i32 0, i32 1, i32 18, i32 19, i32 2, i32 3, i32 20, i32 21, i32 4, i32 5, i32 22, i32 23, i32 6, i32 7>
+  ret <16 x i8> %shuffle
+}
+define dso_local <16 x i8> @testmrghw(<16 x i8> %a, <16 x i8> %b) local_unnamed_addr #0 {
+; CHECK-P8-LABEL: testmrghw:
+; CHECK-P8:       # %bb.0: # %entry
+; CHECK-P8-NEXT:    vmrghw v2, v3, v2
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-P9-LABEL: testmrghw:
+; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    vmrghw v2, v3, v2
+; CHECK-P9-NEXT:    blr
+entry:
+  %shuffle = shufflevector <16 x i8> %a, <16 x i8> %b, <16 x i32> <i32 8, i32 9, i32 10, i32 11, i32 24, i32 25, i32 26, i32 27, i32 12, i32 13, i32 14, i32 15, i32 28, i32 29, i32 30, i32 31>
+  ret <16 x i8> %shuffle
+}
+define dso_local <16 x i8> @testmrghw2(<16 x i8> %a, <16 x i8> %b) local_unnamed_addr #0 {
+; CHECK-P8-LABEL: testmrghw2:
+; CHECK-P8:       # %bb.0: # %entry
+; CHECK-P8-NEXT:    vmrghw v2, v2, v3
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-P9-LABEL: testmrghw2:
+; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    vmrghw v2, v2, v3
+; CHECK-P9-NEXT:    blr
+entry:
+  %shuffle = shufflevector <16 x i8> %a, <16 x i8> %b, <16 x i32> <i32 24, i32 25, i32 26, i32 27, i32 8, i32 9, i32 10, i32 11, i32 28, i32 29, i32 30, i32 31, i32 12, i32 13, i32 14, i32 15>
+  ret <16 x i8> %shuffle
+}
+define dso_local <16 x i8> @testmrglw(<16 x i8> %a, <16 x i8> %b) local_unnamed_addr #0 {
+; CHECK-P8-LABEL: testmrglw:
+; CHECK-P8:       # %bb.0: # %entry
+; CHECK-P8-NEXT:    vmrglw v2, v3, v2
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-P9-LABEL: testmrglw:
+; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    vmrglw v2, v3, v2
+; CHECK-P9-NEXT:    blr
+entry:
+  %shuffle = shufflevector <16 x i8> %a, <16 x i8> %b, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 16, i32 17, i32 18, i32 19, i32 4, i32 5, i32 6, i32 7, i32 20, i32 21, i32 22, i32 23>
+  ret <16 x i8> %shuffle
+}
+define dso_local <16 x i8> @testmrglw2(<16 x i8> %a, <16 x i8> %b) local_unnamed_addr #0 {
+; CHECK-P8-LABEL: testmrglw2:
+; CHECK-P8:       # %bb.0: # %entry
+; CHECK-P8-NEXT:    vmrglw v2, v2, v3
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-P9-LABEL: testmrglw2:
+; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    vmrglw v2, v2, v3
+; CHECK-P9-NEXT:    blr
+entry:
+  %shuffle = shufflevector <16 x i8> %a, <16 x i8> %b, <16 x i32> <i32 16, i32 17, i32 18, i32 19, i32 0, i32 1, i32 2, i32 3, i32 20, i32 21, i32 22, i32 23, i32 4, i32 5, i32 6, i32 7>
+  ret <16 x i8> %shuffle
+}
+
+define dso_local <8 x i16> @testmrglb3(<8 x i8>* nocapture readonly %a) local_unnamed_addr #0 {
+; CHECK-P8-LABEL: testmrglb3:
+; CHECK-P8:       # %bb.0: # %entry
+; CHECK-P8-NEXT:    ld r3, 0(r3)
+; CHECK-P8-NEXT:    xxlxor v3, v3, v3
+; CHECK-P8-NEXT:    mtvsrd f0, r3
+; CHECK-P8-NEXT:    xxswapd v2, vs0
+; CHECK-P8-NEXT:    vmrglb v2, v3, v2
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-P9-LABEL: testmrglb3:
+; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    lfd f0, 0(r3)
+; CHECK-P9-NEXT:    xxpermdi v2, f0, f0, 2
+; CHECK-P9-NEXT:    xxlxor v3, v3, v3
+; CHECK-P9-NEXT:    vmrglb v2, v3, v2
+; CHECK-P9-NEXT:    blr
+entry:
+  %0 = load <8 x i8>, <8 x i8>* %a, align 8
+  %1 = zext <8 x i8> %0 to <8 x i16>
+  ret <8 x i16> %1
+}
+
+attributes #0 = { nounwind }

llvm/test/CodeGen/PowerPC/pre-inc-disable.ll

 ; CHECK-NEXT:    addi r5, r5, .LCPI0_1@toc@l
 ; CHECK-NEXT:    lxvx v4, 0, r5
 ; CHECK-NEXT:    xxpermdi v5, f0, f0, 2
 ; CHECK-NEXT:    xxlxor v3, v3, v3
 ; CHECK-NEXT:    li r5, 4
 ; CHECK-NEXT:    vperm v0, v3, v5, v2
 ; CHECK-NEXT:    mtctr r5
 ; CHECK-NEXT:    li r5, 0
-; CHECK-NEXT:    vperm v1, v5, v3, v4
+; CHECK-NEXT:    vperm v1, v3, v5, v4
 ; CHECK-NEXT:    li r6, 0
 ; CHECK-NEXT:    xvnegsp v5, v0
 ; CHECK-NEXT:    xvnegsp v0, v1
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  .LBB0_1: # %for.cond1.preheader
 ; CHECK-NEXT:    #
 ; CHECK-NEXT:    lfd f0, 0(r3)
 ; CHECK-NEXT:    xxpermdi v1, f0, f0, 2
 ; CHECK-NEXT:    lfdx f0, r3, r4
-; CHECK-NEXT:    vperm v6, v1, v3, v4
+; CHECK-NEXT:    vperm v6, v3, v1, v4
 ; CHECK-NEXT:    vperm v1, v3, v1, v2
 ; CHECK-NEXT:    xvnegsp v1, v1
 ; CHECK-NEXT:    add r7, r3, r4
 ; CHECK-NEXT:    xvnegsp v6, v6
 ; CHECK-NEXT:    vabsduw v1, v1, v5
 ; CHECK-NEXT:    vabsduw v6, v6, v0
 ; CHECK-NEXT:    vadduwm v1, v6, v1
 ; CHECK-NEXT:    xxswapd v6, v1
 ; CHECK-NEXT:    vadduwm v1, v1, v6
 ; CHECK-NEXT:    xxspltw v6, v1, 2
 ; CHECK-NEXT:    vadduwm v1, v1, v6
 ; CHECK-NEXT:    xxpermdi v6, f0, f0, 2
 ; CHECK-NEXT:    vextuwrx r3, r5, v1
-; CHECK-NEXT:    vperm v7, v6, v3, v4
+; CHECK-NEXT:    vperm v7, v3, v6, v4
 ; CHECK-NEXT:    vperm v6, v3, v6, v2
 ; CHECK-NEXT:    add r6, r3, r6
 ; CHECK-NEXT:    add r3, r7, r4
 ; CHECK-NEXT:    xvnegsp v6, v6
 ; CHECK-NEXT:    xvnegsp v1, v7
 ; CHECK-NEXT:    vabsduw v6, v6, v5
 ; CHECK-NEXT:    vabsduw v1, v1, v0
 ; CHECK-NEXT:    vadduwm v1, v1, v6

 ; CHECK-NEXT:    lxvx v4, 0, r3
 ; CHECK-NEXT:    addis r3, r2, .LCPI1_1@toc@ha
 ; CHECK-NEXT:    xxpermdi v2, f0, f0, 2
 ; CHECK-NEXT:    lfd f0, 0(r4)
 ; CHECK-NEXT:    addi r3, r3, .LCPI1_1@toc@l
 ; CHECK-NEXT:    xxlxor v3, v3, v3
 ; CHECK-NEXT:    lxvx v0, 0, r3
 ; CHECK-NEXT:    xxpermdi v1, f0, f0, 2
-; CHECK-NEXT:    vperm v5, v2, v3, v4
+; CHECK-NEXT:    vperm v5, v3, v2, v4
 ; CHECK-NEXT:    vperm v2, v3, v2, v0
 ; CHECK-NEXT:    vperm v0, v3, v1, v0
-; CHECK-NEXT:    vperm v3, v1, v3, v4
+; CHECK-NEXT:    vperm v3, v3, v1, v4
 ; CHECK-NEXT:    vabsduw v2, v2, v0
 ; CHECK-NEXT:    vabsduw v3, v5, v3
 ; CHECK-NEXT:    vadduwm v2, v3, v2
 ; CHECK-NEXT:    xxswapd v3, v2
 ; CHECK-NEXT:    li r3, 0
 ; CHECK-NEXT:    vadduwm v2, v2, v3
 ; CHECK-NEXT:    xxspltw v3, v2, 2
 ; CHECK-NEXT:    vadduwm v2, v2, v3

llvm/test/CodeGen/PowerPC/scalar_vector_test_4.ll

 define <4 x i32> @s2v_test1(i32* nocapture readonly %int32, <4 x i32> %vec)  {
 ; P8LE-LABEL: s2v_test1:
 ; P8LE:       # %bb.0: # %entry
 ; P8LE-NEXT:    lfiwzx f0, 0, r3
 ; P8LE-NEXT:    addis r4, r2, .LCPI0_0@toc@ha
 ; P8LE-NEXT:    addi r3, r4, .LCPI0_0@toc@l
 ; P8LE-NEXT:    lvx v4, 0, r3
 ; P8LE-NEXT:    xxpermdi v3, f0, f0, 2
-; P8LE-NEXT:    vperm v2, v3, v2, v4
+; P8LE-NEXT:    vperm v2, v2, v3, v4
 ; P8LE-NEXT:    blr
 
 ; P8BE-LABEL: s2v_test1:
 ; P8BE:       # %bb.0: # %entry
 ; P8BE:         lfiwzx f0, 0, r3
 ; P8BE-NEXT:    xxsldwi vs0, f0, f0, 1
 ; P8BE:         xxsldwi vs0, v2, vs0, 1
 ; P8BE:         xxsldwi v2, vs0, vs0, 3

 ; P8LE-LABEL: s2v_test2:
 ; P8LE:       # %bb.0: # %entry
 ; P8LE-NEXT:    addi r3, r3, 4
 ; P8LE-NEXT:    addis r4, r2, .LCPI1_0@toc@ha
 ; P8LE-NEXT:    lfiwzx f0, 0, r3
 ; P8LE-NEXT:    addi r3, r4, .LCPI1_0@toc@l
 ; P8LE-NEXT:    lvx v4, 0, r3
 ; P8LE-NEXT:    xxpermdi v3, f0, f0, 2
-; P8LE-NEXT:    vperm v2, v3, v2, v4
+; P8LE-NEXT:    vperm v2, v2, v3, v4
 ; P8LE-NEXT:    blr
 
 ; P8BE-LABEL: s2v_test2:
 ; P8BE:       # %bb.0: # %entry
 ; P8BE:         addi r3, r3, 4
 ; P8BE:         lfiwzx f0, 0, r3
 ; P8BE-NEXT:    xxsldwi vs0, f0, f0, 1
 ; P8BE:         xxsldwi vs0, v2, vs0, 1

 ; P8LE-LABEL: s2v_test3:
 ; P8LE:       # %bb.0: # %entry
 ; P8LE-NEXT:    sldi r5, r7, 2
 ; P8LE-NEXT:    addis r4, r2, .LCPI2_0@toc@ha
 ; P8LE-NEXT:    lfiwzx f0, r3, r5
 ; P8LE-NEXT:    addi r3, r4, .LCPI2_0@toc@l
 ; P8LE-NEXT:    lvx v4, 0, r3
 ; P8LE-NEXT:    xxpermdi v3, f0, f0, 2
-; P8LE-NEXT:    vperm v2, v3, v2, v4
+; P8LE-NEXT:    vperm v2, v2, v3, v4
 ; P8LE-NEXT:    blr
 
 ; P8BE-LABEL: s2v_test3:
 ; P8BE:       # %bb.0: # %entry
 ; P8BE:         sldi r4, r7, 2
 ; P8BE:         lfiwzx f0, r3, r4
 ; P8BE-NEXT:    xxsldwi vs0, f0, f0, 1
 ; P8BE:         xxsldwi vs0, v2, vs0, 1

 ; P8LE-LABEL: s2v_test4:
 ; P8LE:       # %bb.0: # %entry
 ; P8LE-NEXT:    addi r3, r3, 4
 ; P8LE-NEXT:    addis r4, r2, .LCPI3_0@toc@ha
 ; P8LE-NEXT:    lfiwzx f0, 0, r3
 ; P8LE-NEXT:    addi r3, r4, .LCPI3_0@toc@l
 ; P8LE-NEXT:    lvx v4, 0, r3
 ; P8LE-NEXT:    xxpermdi v3, f0, f0, 2
-; P8LE-NEXT:    vperm v2, v3, v2, v4
+; P8LE-NEXT:    vperm v2, v2, v3, v4
 ; P8LE-NEXT:    blr
 
 ; P8BE-LABEL: s2v_test4:
 ; P8BE:       # %bb.0: # %entry
 ; P8BE:         addi r3, r3, 4
 ; P8BE:         lfiwzx f0, 0, r3
 ; P8BE-NEXT:    xxsldwi vs0, f0, f0, 1
 ; P8BE:         xxsldwi vs0, v2, vs0, 1

 define <4 x i32> @s2v_test5(<4 x i32> %vec, i32* nocapture readonly %ptr1)  {
 ; P8LE-LABEL: s2v_test5:
 ; P8LE:       # %bb.0: # %entry
 ; P8LE-NEXT:    lfiwzx f0, 0, r5
 ; P8LE-NEXT:    addis r3, r2, .LCPI4_0@toc@ha
 ; P8LE-NEXT:    addi r3, r3, .LCPI4_0@toc@l
 ; P8LE-NEXT:    lvx v4, 0, r3
 ; P8LE-NEXT:    xxpermdi v3, f0, f0, 2
-; P8LE-NEXT:    vperm v2, v3, v2, v4
+; P8LE-NEXT:    vperm v2, v2, v3, v4
 ; P8LE-NEXT:    blr
 
 ; P8BE-LABEL: s2v_test5:
 ; P8BE:       # %bb.0: # %entry
 ; P8BE:         lfiwzx f0, 0, r5
 ; P8BE-NEXT:    xxsldwi vs0, f0, f0, 1
 ; P8BE:         xxsldwi vs0, v2, vs0, 1
 ; P8BE:         xxsldwi v2, vs0, vs0, 3
 ; P8BE-NEXT:    blr
 entry:
   %0 = load i32, i32* %ptr1, align 4
   %vecins = insertelement <4 x i32> %vec, i32 %0, i32 0
   ret <4 x i32> %vecins
 }
 
 ; Function Attrs: norecurse nounwind readonly
 define <4 x float> @s2v_test_f1(float* nocapture readonly %f64, <4 x float> %vec)  {
 ; P8LE-LABEL: s2v_test_f1:
 ; P8LE:       # %bb.0: # %entry
 ; P8LE-NEXT:    lfiwzx f0, 0, r3
 ; P8LE-NEXT:    addis r4, r2, .LCPI5_0@toc@ha
 ; P8LE-NEXT:    addi r3, r4, .LCPI5_0@toc@l
 ; P8LE-NEXT:    lvx v4, 0, r3
 ; P8LE-NEXT:    xxpermdi v3, f0, f0, 2
-; P8LE-NEXT:    vperm v2, v3, v2, v4
+; P8LE-NEXT:    vperm v2, v2, v3, v4
 ; P8LE-NEXT:    blr
 
 ; P8BE-LABEL: s2v_test_f1:
 ; P8BE:       # %bb.0: # %entry
 ; P8BE:         lfiwzx f0, 0, r3
 ; P8BE-NEXT:    xxsldwi vs0, f0, f0, 1
 ; P8BE:         xxsldwi vs0, v2, vs0, 1
 ; P8BE:         xxsldwi v2, vs0, vs0, 3

llvm/test/CodeGen/PowerPC/vec_conv_i16_to_fp32_elts.ll

entry:
   %1 = uitofp <2 x i16> %0 to <2 x float>
   %2 = bitcast <2 x float> %1 to i64
   ret i64 %2
 }
 
 define <4 x float> @test4elt(i64 %a.coerce) local_unnamed_addr #1 {
 ; CHECK-P8-LABEL: test4elt:
 ; CHECK-P8:       # %bb.0: # %entry
-; CHECK-P8-NEXT:    addis r4, r2, .LCPI1_0@toc@ha
 ; CHECK-P8-NEXT:    mtvsrd f0, r3
-; CHECK-P8-NEXT:    addi r3, r4, .LCPI1_0@toc@l
-; CHECK-P8-NEXT:    xxlxor v4, v4, v4
+; CHECK-P8-NEXT:    xxlxor v3, v3, v3
 ; CHECK-P8-NEXT:    xxswapd v2, vs0
-; CHECK-P8-NEXT:    lvx v3, 0, r3
-; CHECK-P8-NEXT:    vperm v2, v4, v2, v3
+; CHECK-P8-NEXT:    vmrglh v2, v3, v2
 ; CHECK-P8-NEXT:    xvcvuxwsp v2, v2
 ; CHECK-P8-NEXT:    blr
 ;
 ; CHECK-P9-LABEL: test4elt:
 ; CHECK-P9:       # %bb.0: # %entry
 ; CHECK-P9-NEXT:    mtvsrd f0, r3
-; CHECK-P9-NEXT:    addis r3, r2, .LCPI1_0@toc@ha
-; CHECK-P9-NEXT:    addi r3, r3, .LCPI1_0@toc@l
-; CHECK-P9-NEXT:    lxvx v3, 0, r3
 ; CHECK-P9-NEXT:    xxswapd v2, vs0
-; CHECK-P9-NEXT:    xxlxor v4, v4, v4
-; CHECK-P9-NEXT:    vperm v2, v4, v2, v3
+; CHECK-P9-NEXT:    xxlxor v3, v3, v3
+; CHECK-P9-NEXT:    vmrglh v2, v3, v2
 ; CHECK-P9-NEXT:    xvcvuxwsp v2, v2
 ; CHECK-P9-NEXT:    blr
 ;
 ; CHECK-BE-LABEL: test4elt:
 ; CHECK-BE:       # %bb.0: # %entry
 ; CHECK-BE-NEXT:    mtvsrd v2, r3
 ; CHECK-BE-NEXT:    addis r3, r2, .LCPI1_0@toc@ha
 ; CHECK-BE-NEXT:    addi r3, r3, .LCPI1_0@toc@l
 ; CHECK-BE-NEXT:    lxvx v3, 0, r3
 ; CHECK-BE-NEXT:    xxlxor v4, v4, v4
 ; CHECK-BE-NEXT:    vperm v2, v2, v4, v3
 ; CHECK-BE-NEXT:    xvcvuxwsp v2, v2
 ; CHECK-BE-NEXT:    blr
 entry:
   %0 = bitcast i64 %a.coerce to <4 x i16>
   %1 = uitofp <4 x i16> %0 to <4 x float>
   ret <4 x float> %1
 }
 
 define void @test8elt(<8 x float>* noalias nocapture sret %agg.result, <8 x i16> %a) local_unnamed_addr #2 {
 ; CHECK-P8-LABEL: test8elt:
 ; CHECK-P8:       # %bb.0: # %entry
-; CHECK-P8-NEXT:    addis r4, r2, .LCPI2_0@toc@ha
-; CHECK-P8-NEXT:    addis r5, r2, .LCPI2_1@toc@ha
-; CHECK-P8-NEXT:    xxlxor v4, v4, v4
-; CHECK-P8-NEXT:    addi r4, r4, .LCPI2_0@toc@l
-; CHECK-P8-NEXT:    lvx v3, 0, r4
-; CHECK-P8-NEXT:    addi r4, r5, .LCPI2_1@toc@l
-; CHECK-P8-NEXT:    lvx v5, 0, r4
+; CHECK-P8-NEXT:    xxlxor v3, v3, v3
 ; CHECK-P8-NEXT:    li r4, 16
-; CHECK-P8-NEXT:    vperm v3, v4, v2, v3
-; CHECK-P8-NEXT:    vperm v2, v4, v2, v5
-; CHECK-P8-NEXT:    xvcvuxwsp v3, v3
+; CHECK-P8-NEXT:    vmrglh v4, v3, v2
+; CHECK-P8-NEXT:    vmrghh v2, v3, v2
+; CHECK-P8-NEXT:    xvcvuxwsp v3, v4
 ; CHECK-P8-NEXT:    xvcvuxwsp v2, v2
 ; CHECK-P8-NEXT:    stvx v3, 0, r3
 ; CHECK-P8-NEXT:    stvx v2, r3, r4
 ; CHECK-P8-NEXT:    blr
 ;
 ; CHECK-P9-LABEL: test8elt:
 ; CHECK-P9:       # %bb.0: # %entry
-; CHECK-P9-NEXT:    addis r4, r2, .LCPI2_0@toc@ha
-; CHECK-P9-NEXT:    addi r4, r4, .LCPI2_0@toc@l
-; CHECK-P9-NEXT:    lxvx v3, 0, r4
-; CHECK-P9-NEXT:    xxlxor v4, v4, v4
-; CHECK-P9-NEXT:    addis r4, r2, .LCPI2_1@toc@ha
-; CHECK-P9-NEXT:    addi r4, r4, .LCPI2_1@toc@l
-; CHECK-P9-NEXT:    vperm v3, v4, v2, v3
-; CHECK-P9-NEXT:    xvcvuxwsp vs0, v3
-; CHECK-P9-NEXT:    lxvx v3, 0, r4
-; CHECK-P9-NEXT:    vperm v2, v4, v2, v3
-; CHECK-P9-NEXT:    stxv vs0, 0(r3)
+; CHECK-P9-NEXT:    xxlxor v3, v3, v3
+; CHECK-P9-NEXT:    vmrglh v4, v3, v2
+; CHECK-P9-NEXT:    vmrghh v2, v3, v2
+; CHECK-P9-NEXT:    xvcvuxwsp vs0, v4
 ; CHECK-P9-NEXT:    xvcvuxwsp vs1, v2
 ; CHECK-P9-NEXT:    stxv vs1, 16(r3)
+; CHECK-P9-NEXT:    stxv vs0, 0(r3)
 ; CHECK-P9-NEXT:    blr
 ;
 ; CHECK-BE-LABEL: test8elt:
 ; CHECK-BE:       # %bb.0: # %entry
 ; CHECK-BE-NEXT:    addis r4, r2, .LCPI2_0@toc@ha
 ; CHECK-BE-NEXT:    addi r4, r4, .LCPI2_0@toc@l
 ; CHECK-BE-NEXT:    lxvx v3, 0, r4
 ; CHECK-BE-NEXT:    xxlxor v4, v4, v4