[PowerPC] Handle Vector Sum Reduction

[stefanp]

Summary

Better codegen is possible for Vector Sum Reduction on Power 9 and up.
This patch adds handling for several vector sum reductions depending
on the type of the vector.

[nemanjai]

llvm/lib/Target/PowerPC/PPCMIPeephole.cpp

1067

This is a more general and orthogonal peephole. Can you please separate this out to another patch and make it more general? This patch can then depend on that one.

1072

Please make this more general by implementing a separate function called something like isVectorSymmetrical() that will just check whether the two halves of the vector are the same. Then this can be flipped to an early exit condition.

P.S. Keep in mind that it is possible that DefVecReg == nullptr which would cause a crash when you call DefVecReg->getOpcode().

[stefanp]

Split out the peephole optimization that would replace MFVSRLD with MFVSRD
when possible.

That part of the patch will be a new patch soon.

[amyk]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

10757

It looks like this portion:

if (VecInput.getOpcode() != ISD::SIGN_EXTEND &&
    VecInput.getOpcode() != ISD::ZERO_EXTEND)
  return SDValue();

// Check that we are extending from v16i8 to v16i32.
if (VecInput.getOperand(0).getSimpleValueType() != MVT::v16i8)
  return SDValue();

is common in both cases. Does it make sense to pull this part out of the case statements so we don't duplicate it?

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

93

I think it would be good to save the opcode first, and then use it in this condition.

[amyk]

Also, I noticed that I think you may have meant to put [PowerPC] Handle Vector Sum Reduction instead of [PowerPC] Handle Vector Sum Reducation for the title. :-)

[stefanp]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

93

Sounds good. I will do that!

[stefanp]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

10757

There is actually more common code between the two case statements. I think I'm going to merge to two cases but I don't want to pull the if statements out of the switch. The way that I look at this is that anything that is outside the switch should apply to all of the possible types in the switch. Since I cannot guarantee that for those two early exists I will just leave them in the switch.

[stefanp]

Merged some common code to make things easier to read.

llvm/lib/Target/PowerPC/PPCISelLowering.h

private:
     SDValue LowerMUL(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerFP_EXTEND(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerFP_ROUND(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerROTL(SDValue Op, SelectionDAG &DAG) const;
 
     SDValue LowerVectorLoad(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerVectorStore(SDValue Op, SelectionDAG &DAG) const;
 
+    SDValue LowerVECREDUCE_ADD(SDValue Op, SelectionDAG &DAG) const;
+
     SDValue LowerCallResult(SDValue Chain, SDValue InFlag,
                             CallingConv::ID CallConv, bool isVarArg,
                             const SmallVectorImpl<ISD::InputArg> &Ins,
                             const SDLoc &dl, SelectionDAG &DAG,
                             SmallVectorImpl<SDValue> &InVals) const;
 
     SDValue FinishCall(CallFlags CFlags, const SDLoc &dl, SelectionDAG &DAG,
                        SmallVector<std::pair<unsigned, SDValue>, 8> &RegsToPass,

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

if (Subtarget.hasVSX()) {
       setOperationAction(ISD::INSERT_VECTOR_ELT, MVT::v2f64, Custom);
     }
 
     if (Subtarget.hasP8Altivec())
       setOperationAction(ISD::MUL, MVT::v4i32, Legal);
     else
       setOperationAction(ISD::MUL, MVT::v4i32, Custom);
 
+    if (Subtarget.isISA3_0()) {
+      setOperationAction(ISD::VECREDUCE_ADD, MVT::v16i8, Legal);
+      // This has to be custom because we want to lower it before type
+      // legalization gets to the type and expands everything out.
+      setOperationAction(ISD::VECREDUCE_ADD, MVT::v16i16, Custom);
+      setOperationAction(ISD::VECREDUCE_ADD, MVT::v16i32, Custom);
+    }
+
     if (Subtarget.isISA3_1()) {
       setOperationAction(ISD::MUL, MVT::v2i64, Legal);
       setOperationAction(ISD::MULHS, MVT::v2i64, Legal);
       setOperationAction(ISD::MULHU, MVT::v2i64, Legal);
       setOperationAction(ISD::MULHS, MVT::v4i32, Legal);
       setOperationAction(ISD::MULHU, MVT::v4i32, Legal);
       setOperationAction(ISD::UDIV, MVT::v2i64, Legal);
       setOperationAction(ISD::SDIV, MVT::v2i64, Legal);
case ISD::ATOMIC_STORE: {
     return DAG.getMemIntrinsicNode(ISD::INTRINSIC_VOID, dl, Tys, Ops, MemVT,
                                    N->getMemOperand());
   }
   default:
     llvm_unreachable("Unexpected atomic opcode");
   }
 }
 
+SDValue PPCTargetLowering::LowerVECREDUCE_ADD(SDValue Op,
+                                              SelectionDAG &DAG) const {
+  SDValue VecInput = Op.getOperand(0);
+  MVT InputType = VecInput.getSimpleValueType();
+  SDLoc dl(Op);
+
+  // A vector with all zeros for the form: {0, 0, 0, 0}
+  SDValue ZeroVector = DAG.getTargetConstant(0, dl, MVT::v4i32);
+  // A vector where all bytes are one of the form:
+  // {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}
+  SDNode *AllOnes = DAG.getMachineNode(PPC::XXSPLTIB, dl, MVT::v16i8,
+                                       DAG.getTargetConstant(1, dl, MVT::i32));
+
+  switch (InputType.SimpleTy) {
+  default:
+    return SDValue();
+  case MVT::v16i32: {
+    if (VecInput.getOpcode() != ISD::SIGN_EXTEND &&
+        VecInput.getOpcode() != ISD::ZERO_EXTEND)
+      return SDValue();
+
+    // Check that we are extending from v16i8 to v16i32.
+    if (VecInput.getOperand(0).getSimpleValueType() != MVT::v16i8)
+      return SDValue();
+
+    // Sum Bytes and place the result in each word.
+    // {a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p} turns into
+    // {a+b+c+d, e+f+g+h, i+j+k+l, m+n+o+p}
+    SDNode *SumBytes = DAG.getMachineNode(PPC::VMSUMUBM, dl, MVT::v4i32,
+                                          VecInput.getOperand(0),
+                                          SDValue(AllOnes, 0), ZeroVector);
+    // Sum words and place result in single word.
+    /// {a, b, c, d} turns into {0, 0, 0, a+b+c+d}
+    SDNode *SumAll = DAG.getMachineNode(PPC::VSUMSWS, dl, MVT::v2i64,
+                                        SDValue(SumBytes, 0), ZeroVector);
+    SDValue Bitcast =
+        DAG.getNode(ISD::BITCAST, dl, MVT::v4i32, SDValue(SumAll, 0));
+    return DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, MVT::i32, Bitcast,
+                       DAG.getTargetConstant(3, dl, MVT::i32));
+    break;
+  }
+  case MVT::v16i16: {
+    if (VecInput.getOpcode() != ISD::SIGN_EXTEND &&
+        VecInput.getOpcode() != ISD::ZERO_EXTEND)
+      return SDValue();
+
+    // Check that we are extending from v16i8 to v16i32.
+    if (VecInput.getOperand(0).getSimpleValueType() != MVT::v16i8)
+      return SDValue();
+
+    // Sum Bytes and place the result in each word.
+    // {a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p} turns into
+    // {a+b+c+d, e+f+g+h, i+j+k+l, m+n+o+p}
+    SDNode *SumBytes = DAG.getMachineNode(PPC::VMSUMUBM, dl, MVT::v4i32,
+                                          VecInput.getOperand(0),
+                                          SDValue(AllOnes, 0), ZeroVector);
+    // Sum words and place result in single word.
+    /// {a, b, c, d} turns into {0, 0, 0, a+b+c+d}
+    SDNode *SumAll = DAG.getMachineNode(PPC::VSUMSWS, dl, MVT::v2i64,
+                                        SDValue(SumBytes, 0), ZeroVector);
+    SDValue Bitcast =
+        DAG.getNode(ISD::BITCAST, dl, MVT::v8i16, SDValue(SumAll, 0));
+    return DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, MVT::i16, Bitcast,
+                       DAG.getTargetConstant(7, dl, MVT::i32));
+  }
+  }
+
+  return SDValue();
+}
+
 SDValue PPCTargetLowering::LowerSCALAR_TO_VECTOR(SDValue Op,
                                                  SelectionDAG &DAG) const {
   SDLoc dl(Op);
   // Create a stack slot that is 16-byte aligned.
   MachineFrameInfo &MFI = DAG.getMachineFunction().getFrameInfo();
   int FrameIdx = MFI.CreateStackObject(16, Align(16), false);
   EVT PtrVT = getPointerTy(DAG.getDataLayout());
   SDValue FIdx = DAG.getFrameIndex(FrameIdx, PtrVT);
SDValue PPCTargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   case ISD::INTRINSIC_VOID:
     return LowerINTRINSIC_VOID(Op, DAG);
   case ISD::BSWAP:
     return LowerBSWAP(Op, DAG);
   case ISD::ATOMIC_CMP_SWAP:
     return LowerATOMIC_CMP_SWAP(Op, DAG);
   case ISD::ATOMIC_STORE:
     return LowerATOMIC_LOAD_STORE(Op, DAG);
+  case ISD::VECREDUCE_ADD:
+    return LowerVECREDUCE_ADD(Op, DAG);
   }
 }
 
 void PPCTargetLowering::ReplaceNodeResults(SDNode *N,
                                            SmallVectorImpl<SDValue>&Results,
                                            SelectionDAG &DAG) const {
   SDLoc dl(N);
   switch (N->getOpcode()) {

llvm/lib/Target/PowerPC/PPCInstrVSX.td

 def : Pat<(v4f32 (PPCldsplat ForceXForm:$A)),
           (v4f32 (LXVWSX ForceXForm:$A))>;
 def : Pat<(v4i32 (PPCldsplat ForceXForm:$A)),
           (v4i32 (LXVWSX ForceXForm:$A))>;
 def : Pat<(v8i16 (PPCldsplat ForceXForm:$A)),
           (v8i16 (VSPLTHs 3, (LXSIHZX ForceXForm:$A)))>;
 def : Pat<(v16i8 (PPCldsplat ForceXForm:$A)),
           (v16i8 (VSPLTBs 7, (LXSIBZX ForceXForm:$A)))>;
+
+// Vector Sum Reductions
+def : Pat<(i64 (anyext (i32 (vecreduce_add v16i8:$vA)))),
+          (i64 (MFVSRLD (VSUMSWS
+                         (VMSUMUBM v16i8:$vA, (v16i8 (COPY_TO_REGCLASS (XXSPLTIB 1), VSRC)), (v4i32(V_SET0))),
+                         (v4i32(V_SET0)))))>;
+def : Pat<(i32 (vecreduce_add v16i8:$vA)),
+          (i32 (EXTRACT_SUBREG (MFVSRLD (VSUMSWS
+                         (VMSUMUBM v16i8:$vA, (v16i8 (COPY_TO_REGCLASS (XXSPLTIB 1), VSRC)), (v4i32(V_SET0))),
+                         (v4i32(V_SET0)))), sub_32))>;
+
 } // HasVSX, HasP9Vector
 
 // Any Power9 VSX subtarget with equivalent length but better Power10 VSX
 // patterns.
 // Two identical blocks are required due to the slightly different predicates:
 // One without P10 instructions, the other is BigEndian only with P10 instructions.
 let Predicates = [HasVSX, HasP9Vector, NoP10Vector] in {
 // Little endian Power10 subtargets produce a shorter pattern but require a

llvm/lib/Target/PowerPC/PPCMIPeephole.cpp

for (MachineInstr &MI : MBB) {
           BuildMI(MBB, &MI, MI.getDebugLoc(), TII->get(PPC::TRAP));
           TrapOpt = true;
         }
         // We will delete the MI if it will never trap.
         ToErase = &MI;
         Simplified = true;
         break;
       }
+      case PPC::MFVSRLD:
+        // It is more efficient to use MFVSRD instead of MFVSRLD in cases where
+        // it is known that the two doublewords of the vector are identical.
+        MachineOperand &VecInput = MI.getOperand(1);
+        MachineInstr *DefVecReg = getVRegDefOrNull(&VecInput, MRI);
+        if (DefVecReg->getOpcode() == PPC::VADDUDM) {
+          MachineOperand &AddInput1 = DefVecReg->getOperand(1);
+          MachineOperand &AddInput2 = DefVecReg->getOperand(2);
+          MachineInstr *DefAdd1 = getVRegDefOrNull(&AddInput1, MRI);
+          MachineInstr *DefAdd2 = getVRegDefOrNull(&AddInput2, MRI);
+
+          if (!DefAdd1 || !DefAdd2)
+            break;
+
+          // Check if one of the inputs to the vector add is a permute.
+          MachineInstr *PermInstr = nullptr;
+          Register NonPermReg;
+          if (DefAdd1->getOpcode() == PPC::XXPERMDI) {
+            PermInstr = DefAdd1;
+            NonPermReg = AddInput2.getReg();
+          } else if (DefAdd2->getOpcode() == PPC::XXPERMDI) {
+            PermInstr = DefAdd2;
+            NonPermReg = AddInput1.getReg();
+          }
+
+          // Abort if there is no permute instruction.
+          if (!PermInstr)
+            break;
+
+          MachineOperand &PermInput1 = PermInstr->getOperand(1);
+          MachineOperand &PermInput2 = PermInstr->getOperand(2);
+          unsigned PermType = PermInstr->getOperand(3).getImm();
+          if (PermInput1.getReg() == PermInput2.getReg() && PermType == 2 &&
+              PermInput1.getReg() == NonPermReg) {
+            Register VecInputReg = VecInput.getReg();
+            BuildMI(MBB, &MI, MI.getDebugLoc(), TII->get(PPC::MFVSRD),
+                    MI.getOperand(0).getReg())
+                .addReg(VecInputReg, 0, PPC::sub_64);
+            ToErase = &MI;
+            Simplified = true;
+          }
+        }
+        break;
       }
     }
 
     // If the last instruction was marked for elimination,
     // remove it now.
     if (ToErase) {
       ToErase->eraseFromParent();
       ToErase = nullptr;

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h

public:
   InstructionCost getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx,
                                       const APInt &Imm, Type *Ty,
                                       TTI::TargetCostKind CostKind);
 
   InstructionCost getUserCost(const User *U, ArrayRef<const Value *> Operands,
                               TTI::TargetCostKind CostKind);
 
   TTI::PopcntSupportKind getPopcntSupport(unsigned TyWidth);
+  bool shouldExpandReduction(const IntrinsicInst *II) const;
   bool isHardwareLoopProfitable(Loop *L, ScalarEvolution &SE,
                                 AssumptionCache &AC,
                                 TargetLibraryInfo *LibInfo,
                                 HardwareLoopInfo &HWLoopInfo);
   bool canSaveCmp(Loop *L, BranchInst **BI, ScalarEvolution *SE, LoopInfo *LI,
                   DominatorTree *DT, AssumptionCache *AC,
                   TargetLibraryInfo *LibInfo);
   bool getTgtMemIntrinsic(IntrinsicInst *Inst, MemIntrinsicInfo &Info);

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

 PPCTTIImpl::getPopcntSupport(unsigned TyWidth) {
   assert(isPowerOf2_32(TyWidth) && "Ty width must be power of 2");
   if (ST->hasPOPCNTD() != PPCSubtarget::POPCNTD_Unavailable && TyWidth <= 64)
     return ST->hasPOPCNTD() == PPCSubtarget::POPCNTD_Slow ?
              TTI::PSK_SlowHardware : TTI::PSK_FastHardware;
   return TTI::PSK_Software;
 }
 
+bool PPCTTIImpl::shouldExpandReduction(const IntrinsicInst *II) const {
+  const unsigned Directive = ST->getCPUDirective();
+
+  // P9 and up only.
+  if (Directive != PPC::DIR_PWR9 && Directive != PPC::DIR_PWR10 &&
+      Directive != PPC::DIR_PWR_FUTURE)
+    return true;
+
+  // Check if we are doing a vector reduce add.
+  if (II->getIntrinsicID() != Intrinsic::vector_reduce_add)
+    return true;
+
+  Value *Operand = II->getArgOperand(0);
+  Type *OpType = Operand->getType();
+  Type *V16x8Ty =
+      FixedVectorType::get(Type::getInt8Ty(Operand->getContext()), 16);
+  Type *V16x16Ty =
+      FixedVectorType::get(Type::getInt16Ty(Operand->getContext()), 16);
+  Type *V16x32Ty =
+      FixedVectorType::get(Type::getInt32Ty(Operand->getContext()), 16);
+
+  Instruction *OperandInstr = dyn_cast<Instruction>(Operand);
+  if (!OperandInstr)
+    return true;
+
+  // Only expand types that are fed by zero extends or sign extends.
+  if (OperandInstr->getOpcode() != Instruction::SExt &&
+      OperandInstr->getOpcode() != Instruction::ZExt)
+    return true;
+
+  if (OpType == V16x8Ty || OpType == V16x16Ty || OpType == V16x32Ty)
+    return false;
+
+  // By default we want to expand everything we cannot cover.
+  return true;
+}
+
 Optional<Instruction *>
 PPCTTIImpl::instCombineIntrinsic(InstCombiner &IC, IntrinsicInst &II) const {
   Intrinsic::ID IID = II.getIntrinsicID();
   switch (IID) {
   default:
     break;
   case Intrinsic::ppc_altivec_lvx:
   case Intrinsic::ppc_altivec_lvxl:

llvm/test/CodeGen/PowerPC/vector-reduce-add.ll

 entry:
   %0 = call i16 @llvm.vector.reduce.add.v16i16(<16 x i16> %a)
   ret i16 %0
 }
 
 define dso_local signext i16 @v16i8tov16i16_sign(<16 x i8> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v16i8tov16i16_sign:
 ; PWR9LE:       # %bb.0: # %entry
-; PWR9LE-NEXT:    vmrghb v3, v2, v2
-; PWR9LE-NEXT:    vspltish v4, 8
-; PWR9LE-NEXT:    li r3, 0
-; PWR9LE-NEXT:    vmrglb v2, v2, v2
-; PWR9LE-NEXT:    vslh v3, v3, v4
-; PWR9LE-NEXT:    vslh v2, v2, v4
-; PWR9LE-NEXT:    vsrah v3, v3, v4
-; PWR9LE-NEXT:    vsrah v2, v2, v4
-; PWR9LE-NEXT:    vadduhm v2, v2, v3
-; PWR9LE-NEXT:    xxswapd v3, v2
-; PWR9LE-NEXT:    vadduhm v2, v2, v3
-; PWR9LE-NEXT:    xxspltw v3, v2, 2
-; PWR9LE-NEXT:    vadduhm v2, v2, v3
-; PWR9LE-NEXT:    vsplth v3, v2, 6
-; PWR9LE-NEXT:    vadduhm v2, v2, v3
+; PWR9LE-NEXT:    xxspltib v4, 1
+; PWR9LE-NEXT:    xxlxor v3, v3, v3
+; PWR9LE-NEXT:    li r3, 14
+; PWR9LE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR9LE-NEXT:    vsumsws v2, v2, v3
 ; PWR9LE-NEXT:    vextuhrx r3, r3, v2
 ; PWR9LE-NEXT:    extsh r3, r3
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i8tov16i16_sign:
 ; PWR9BE:       # %bb.0: # %entry
-; PWR9BE-NEXT:    vmrglb v3, v2, v2
-; PWR9BE-NEXT:    vspltish v4, 8
-; PWR9BE-NEXT:    li r3, 0
-; PWR9BE-NEXT:    vmrghb v2, v2, v2
-; PWR9BE-NEXT:    vslh v3, v3, v4
-; PWR9BE-NEXT:    vslh v2, v2, v4
-; PWR9BE-NEXT:    vsrah v3, v3, v4
-; PWR9BE-NEXT:    vsrah v2, v2, v4
-; PWR9BE-NEXT:    vadduhm v2, v2, v3
-; PWR9BE-NEXT:    xxswapd v3, v2
-; PWR9BE-NEXT:    vadduhm v2, v2, v3
-; PWR9BE-NEXT:    xxspltw v3, v2, 1
-; PWR9BE-NEXT:    vadduhm v2, v2, v3
-; PWR9BE-NEXT:    vsplth v3, v2, 1
-; PWR9BE-NEXT:    vadduhm v2, v2, v3
+; PWR9BE-NEXT:    xxspltib v4, 1
+; PWR9BE-NEXT:    xxlxor v3, v3, v3
+; PWR9BE-NEXT:    li r3, 14
+; PWR9BE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR9BE-NEXT:    vsumsws v2, v2, v3
 ; PWR9BE-NEXT:    vextuhlx r3, r3, v2
 ; PWR9BE-NEXT:    extsh r3, r3
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v16i8tov16i16_sign:
 ; PWR10LE:       # %bb.0: # %entry
-; PWR10LE-NEXT:    vmrghb v3, v2, v2
-; PWR10LE-NEXT:    xxspltiw v4, 524296
-; PWR10LE-NEXT:    vmrglb v2, v2, v2
-; PWR10LE-NEXT:    li r3, 0
-; PWR10LE-NEXT:    vslh v3, v3, v4
-; PWR10LE-NEXT:    vslh v2, v2, v4
-; PWR10LE-NEXT:    vsrah v3, v3, v4
-; PWR10LE-NEXT:    vsrah v2, v2, v4
-; PWR10LE-NEXT:    vadduhm v2, v2, v3
-; PWR10LE-NEXT:    xxswapd v3, v2
-; PWR10LE-NEXT:    vadduhm v2, v2, v3
-; PWR10LE-NEXT:    xxspltw v3, v2, 2
-; PWR10LE-NEXT:    vadduhm v2, v2, v3
-; PWR10LE-NEXT:    vsplth v3, v2, 6
-; PWR10LE-NEXT:    vadduhm v2, v2, v3
+; PWR10LE-NEXT:    xxspltib v4, 1
+; PWR10LE-NEXT:    xxlxor v3, v3, v3
+; PWR10LE-NEXT:    li r3, 14
+; PWR10LE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR10LE-NEXT:    vsumsws v2, v2, v3
 ; PWR10LE-NEXT:    vextuhrx r3, r3, v2
 ; PWR10LE-NEXT:    extsh r3, r3
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i8tov16i16_sign:
 ; PWR10BE:       # %bb.0: # %entry
-; PWR10BE-NEXT:    vmrglb v3, v2, v2
-; PWR10BE-NEXT:    xxspltiw v4, 524296
-; PWR10BE-NEXT:    vmrghb v2, v2, v2
-; PWR10BE-NEXT:    li r3, 0
-; PWR10BE-NEXT:    vslh v3, v3, v4
-; PWR10BE-NEXT:    vslh v2, v2, v4
-; PWR10BE-NEXT:    vsrah v3, v3, v4
-; PWR10BE-NEXT:    vsrah v2, v2, v4
-; PWR10BE-NEXT:    vadduhm v2, v2, v3
-; PWR10BE-NEXT:    xxswapd v3, v2
-; PWR10BE-NEXT:    vadduhm v2, v2, v3
-; PWR10BE-NEXT:    xxspltw v3, v2, 1
-; PWR10BE-NEXT:    vadduhm v2, v2, v3
-; PWR10BE-NEXT:    vsplth v3, v2, 1
-; PWR10BE-NEXT:    vadduhm v2, v2, v3
+; PWR10BE-NEXT:    xxspltib v4, 1
+; PWR10BE-NEXT:    xxlxor v3, v3, v3
+; PWR10BE-NEXT:    li r3, 14
+; PWR10BE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR10BE-NEXT:    vsumsws v2, v2, v3
 ; PWR10BE-NEXT:    vextuhlx r3, r3, v2
 ; PWR10BE-NEXT:    extsh r3, r3
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = sext <16 x i8> %a to <16 x i16>
   %1 = call i16 @llvm.vector.reduce.add.v16i16(<16 x i16> %0)
   ret i16 %1
 }
 
 define dso_local zeroext i16 @v16i8tov16i16_zero(<16 x i8> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v16i8tov16i16_zero:
 ; PWR9LE:       # %bb.0: # %entry
+; PWR9LE-NEXT:    xxspltib v4, 1
 ; PWR9LE-NEXT:    xxlxor v3, v3, v3
-; PWR9LE-NEXT:    li r3, 0
-; PWR9LE-NEXT:    vmrghb v4, v3, v2
-; PWR9LE-NEXT:    vmrglb v2, v3, v2
-; PWR9LE-NEXT:    vadduhm v2, v2, v4
-; PWR9LE-NEXT:    xxswapd v3, v2
-; PWR9LE-NEXT:    vadduhm v2, v2, v3
-; PWR9LE-NEXT:    xxspltw v3, v2, 2
-; PWR9LE-NEXT:    vadduhm v2, v2, v3
-; PWR9LE-NEXT:    vsplth v3, v2, 6
-; PWR9LE-NEXT:    vadduhm v2, v2, v3
+; PWR9LE-NEXT:    li r3, 14
+; PWR9LE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR9LE-NEXT:    vsumsws v2, v2, v3
 ; PWR9LE-NEXT:    vextuhrx r3, r3, v2
 ; PWR9LE-NEXT:    clrldi r3, r3, 48
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i8tov16i16_zero:
 ; PWR9BE:       # %bb.0: # %entry
+; PWR9BE-NEXT:    xxspltib v4, 1
 ; PWR9BE-NEXT:    xxlxor v3, v3, v3
-; PWR9BE-NEXT:    li r3, 0
-; PWR9BE-NEXT:    vmrglb v4, v3, v2
-; PWR9BE-NEXT:    vmrghb v2, v3, v2
-; PWR9BE-NEXT:    vadduhm v2, v2, v4
-; PWR9BE-NEXT:    xxswapd v3, v2
-; PWR9BE-NEXT:    vadduhm v2, v2, v3
-; PWR9BE-NEXT:    xxspltw v3, v2, 1
-; PWR9BE-NEXT:    vadduhm v2, v2, v3
-; PWR9BE-NEXT:    vsplth v3, v2, 1
-; PWR9BE-NEXT:    vadduhm v2, v2, v3
+; PWR9BE-NEXT:    li r3, 14
+; PWR9BE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR9BE-NEXT:    vsumsws v2, v2, v3
 ; PWR9BE-NEXT:    vextuhlx r3, r3, v2
 ; PWR9BE-NEXT:    clrldi r3, r3, 48
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v16i8tov16i16_zero:
 ; PWR10LE:       # %bb.0: # %entry
+; PWR10LE-NEXT:    xxspltib v4, 1
 ; PWR10LE-NEXT:    xxlxor v3, v3, v3
-; PWR10LE-NEXT:    li r3, 0
-; PWR10LE-NEXT:    vmrghb v4, v3, v2
-; PWR10LE-NEXT:    vmrglb v2, v3, v2
-; PWR10LE-NEXT:    vadduhm v2, v2, v4
-; PWR10LE-NEXT:    xxswapd v3, v2
-; PWR10LE-NEXT:    vadduhm v2, v2, v3
-; PWR10LE-NEXT:    xxspltw v3, v2, 2
-; PWR10LE-NEXT:    vadduhm v2, v2, v3
-; PWR10LE-NEXT:    vsplth v3, v2, 6
-; PWR10LE-NEXT:    vadduhm v2, v2, v3
+; PWR10LE-NEXT:    li r3, 14
+; PWR10LE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR10LE-NEXT:    vsumsws v2, v2, v3
 ; PWR10LE-NEXT:    vextuhrx r3, r3, v2
 ; PWR10LE-NEXT:    clrldi r3, r3, 48
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i8tov16i16_zero:
 ; PWR10BE:       # %bb.0: # %entry
+; PWR10BE-NEXT:    xxspltib v4, 1
 ; PWR10BE-NEXT:    xxlxor v3, v3, v3
-; PWR10BE-NEXT:    li r3, 0
-; PWR10BE-NEXT:    vmrglb v4, v3, v2
-; PWR10BE-NEXT:    vmrghb v2, v3, v2
-; PWR10BE-NEXT:    vadduhm v2, v2, v4
-; PWR10BE-NEXT:    xxswapd v3, v2
-; PWR10BE-NEXT:    vadduhm v2, v2, v3
-; PWR10BE-NEXT:    xxspltw v3, v2, 1
-; PWR10BE-NEXT:    vadduhm v2, v2, v3
-; PWR10BE-NEXT:    vsplth v3, v2, 1
-; PWR10BE-NEXT:    vadduhm v2, v2, v3
+; PWR10BE-NEXT:    li r3, 14
+; PWR10BE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR10BE-NEXT:    vsumsws v2, v2, v3
 ; PWR10BE-NEXT:    vextuhlx r3, r3, v2
 ; PWR10BE-NEXT:    clrldi r3, r3, 48
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = zext <16 x i8> %a to <16 x i16>
   %1 = call i16 @llvm.vector.reduce.add.v16i16(<16 x i16> %0)
   ret i16 %1
 }

 entry:
   %0 = call i32 @llvm.vector.reduce.add.v32i32(<32 x i32> %a)
   ret i32 %0
 }
 
 define dso_local signext i32 @v16i8tov16i32_sign(<16 x i8> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v16i8tov16i32_sign:
 ; PWR9LE:       # %bb.0: # %entry
-; PWR9LE-NEXT:    addis r3, r2, .LCPI17_0@toc@ha
-; PWR9LE-NEXT:    addi r3, r3, .LCPI17_0@toc@l
-; PWR9LE-NEXT:    lxv v3, 0(r3)
-; PWR9LE-NEXT:    addis r3, r2, .LCPI17_1@toc@ha
-; PWR9LE-NEXT:    addi r3, r3, .LCPI17_1@toc@l
-; PWR9LE-NEXT:    lxv v4, 0(r3)
-; PWR9LE-NEXT:    addis r3, r2, .LCPI17_2@toc@ha
-; PWR9LE-NEXT:    vperm v3, v2, v2, v3
-; PWR9LE-NEXT:    addi r3, r3, .LCPI17_2@toc@l
-; PWR9LE-NEXT:    lxv v5, 0(r3)
-; PWR9LE-NEXT:    addis r3, r2, .LCPI17_3@toc@ha
-; PWR9LE-NEXT:    vextsb2w v3, v3
-; PWR9LE-NEXT:    vperm v4, v2, v2, v4
-; PWR9LE-NEXT:    addi r3, r3, .LCPI17_3@toc@l
-; PWR9LE-NEXT:    lxv v0, 0(r3)
-; PWR9LE-NEXT:    vextsb2w v4, v4
-; PWR9LE-NEXT:    li r3, 0
-; PWR9LE-NEXT:    vperm v5, v2, v2, v5
-; PWR9LE-NEXT:    vadduwm v3, v4, v3
-; PWR9LE-NEXT:    vextsb2w v5, v5
-; PWR9LE-NEXT:    vperm v2, v2, v2, v0
-; PWR9LE-NEXT:    vextsb2w v2, v2
-; PWR9LE-NEXT:    vadduwm v2, v2, v5
-; PWR9LE-NEXT:    vadduwm v2, v3, v2
-; PWR9LE-NEXT:    xxswapd v3, v2
-; PWR9LE-NEXT:    vadduwm v2, v2, v3
-; PWR9LE-NEXT:    xxspltw v3, v2, 2
-; PWR9LE-NEXT:    vadduwm v2, v2, v3
+; PWR9LE-NEXT:    xxspltib v4, 1
+; PWR9LE-NEXT:    xxlxor v3, v3, v3
+; PWR9LE-NEXT:    li r3, 12
+; PWR9LE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR9LE-NEXT:    vsumsws v2, v2, v3
 ; PWR9LE-NEXT:    vextuwrx r3, r3, v2
 ; PWR9LE-NEXT:    extsw r3, r3
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i8tov16i32_sign:
 ; PWR9BE:       # %bb.0: # %entry
-; PWR9BE-NEXT:    addis r3, r2, .LCPI17_0@toc@ha
-; PWR9BE-NEXT:    addi r3, r3, .LCPI17_0@toc@l
-; PWR9BE-NEXT:    lxv v3, 0(r3)
-; PWR9BE-NEXT:    addis r3, r2, .LCPI17_1@toc@ha
-; PWR9BE-NEXT:    addi r3, r3, .LCPI17_1@toc@l
-; PWR9BE-NEXT:    lxv v4, 0(r3)
-; PWR9BE-NEXT:    addis r3, r2, .LCPI17_2@toc@ha
-; PWR9BE-NEXT:    vperm v3, v2, v2, v3
-; PWR9BE-NEXT:    addi r3, r3, .LCPI17_2@toc@l
-; PWR9BE-NEXT:    lxv v5, 0(r3)
-; PWR9BE-NEXT:    addis r3, r2, .LCPI17_3@toc@ha
-; PWR9BE-NEXT:    vextsb2w v3, v3
-; PWR9BE-NEXT:    vperm v4, v2, v2, v4
-; PWR9BE-NEXT:    addi r3, r3, .LCPI17_3@toc@l
-; PWR9BE-NEXT:    lxv v0, 0(r3)
-; PWR9BE-NEXT:    vextsb2w v4, v4
-; PWR9BE-NEXT:    li r3, 0
-; PWR9BE-NEXT:    vperm v5, v2, v2, v5
-; PWR9BE-NEXT:    vadduwm v3, v4, v3
-; PWR9BE-NEXT:    vextsb2w v5, v5
-; PWR9BE-NEXT:    vperm v2, v2, v2, v0
-; PWR9BE-NEXT:    vextsb2w v2, v2
-; PWR9BE-NEXT:    vadduwm v2, v2, v5
-; PWR9BE-NEXT:    vadduwm v2, v3, v2
-; PWR9BE-NEXT:    xxswapd v3, v2
-; PWR9BE-NEXT:    vadduwm v2, v2, v3
-; PWR9BE-NEXT:    xxspltw v3, v2, 1
-; PWR9BE-NEXT:    vadduwm v2, v2, v3
+; PWR9BE-NEXT:    xxspltib v4, 1
+; PWR9BE-NEXT:    xxlxor v3, v3, v3
+; PWR9BE-NEXT:    li r3, 12
+; PWR9BE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR9BE-NEXT:    vsumsws v2, v2, v3
 ; PWR9BE-NEXT:    vextuwlx r3, r3, v2
 ; PWR9BE-NEXT:    extsw r3, r3
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v16i8tov16i32_sign:
 ; PWR10LE:       # %bb.0: # %entry
-; PWR10LE-NEXT:    plxv v3, .LCPI17_0@PCREL(0), 1
-; PWR10LE-NEXT:    plxv v4, .LCPI17_1@PCREL(0), 1
-; PWR10LE-NEXT:    li r3, 0
-; PWR10LE-NEXT:    vperm v3, v2, v2, v3
-; PWR10LE-NEXT:    plxv v5, .LCPI17_2@PCREL(0), 1
-; PWR10LE-NEXT:    plxv v0, .LCPI17_3@PCREL(0), 1
-; PWR10LE-NEXT:    vperm v4, v2, v2, v4
-; PWR10LE-NEXT:    vperm v5, v2, v2, v5
-; PWR10LE-NEXT:    vperm v2, v2, v2, v0
-; PWR10LE-NEXT:    vextsb2w v3, v3
-; PWR10LE-NEXT:    vextsb2w v4, v4
-; PWR10LE-NEXT:    vextsb2w v5, v5
-; PWR10LE-NEXT:    vextsb2w v2, v2
-; PWR10LE-NEXT:    vadduwm v2, v2, v5
-; PWR10LE-NEXT:    vadduwm v3, v4, v3
-; PWR10LE-NEXT:    vadduwm v2, v3, v2
-; PWR10LE-NEXT:    xxswapd v3, v2
-; PWR10LE-NEXT:    vadduwm v2, v2, v3
-; PWR10LE-NEXT:    xxspltw v3, v2, 2
-; PWR10LE-NEXT:    vadduwm v2, v2, v3
+; PWR10LE-NEXT:    xxspltib v4, 1
+; PWR10LE-NEXT:    xxlxor v3, v3, v3
+; PWR10LE-NEXT:    li r3, 12
+; PWR10LE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR10LE-NEXT:    vsumsws v2, v2, v3
 ; PWR10LE-NEXT:    vextuwrx r3, r3, v2
 ; PWR10LE-NEXT:    extsw r3, r3
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i8tov16i32_sign:
 ; PWR10BE:       # %bb.0: # %entry
-; PWR10BE-NEXT:    addis r3, r2, .LCPI17_0@toc@ha
-; PWR10BE-NEXT:    addi r3, r3, .LCPI17_0@toc@l
-; PWR10BE-NEXT:    lxv v3, 0(r3)
-; PWR10BE-NEXT:    addis r3, r2, .LCPI17_1@toc@ha
-; PWR10BE-NEXT:    addi r3, r3, .LCPI17_1@toc@l
-; PWR10BE-NEXT:    lxv v4, 0(r3)
-; PWR10BE-NEXT:    addis r3, r2, .LCPI17_2@toc@ha
-; PWR10BE-NEXT:    addi r3, r3, .LCPI17_2@toc@l
-; PWR10BE-NEXT:    vperm v3, v2, v2, v3
-; PWR10BE-NEXT:    lxv v5, 0(r3)
-; PWR10BE-NEXT:    addis r3, r2, .LCPI17_3@toc@ha
-; PWR10BE-NEXT:    vextsb2w v3, v3
-; PWR10BE-NEXT:    addi r3, r3, .LCPI17_3@toc@l
-; PWR10BE-NEXT:    vperm v4, v2, v2, v4
-; PWR10BE-NEXT:    lxv v0, 0(r3)
-; PWR10BE-NEXT:    li r3, 0
-; PWR10BE-NEXT:    vextsb2w v4, v4
-; PWR10BE-NEXT:    vperm v5, v2, v2, v5
-; PWR10BE-NEXT:    vadduwm v3, v4, v3
-; PWR10BE-NEXT:    vextsb2w v5, v5
-; PWR10BE-NEXT:    vperm v2, v2, v2, v0
-; PWR10BE-NEXT:    vextsb2w v2, v2
-; PWR10BE-NEXT:    vadduwm v2, v2, v5
-; PWR10BE-NEXT:    vadduwm v2, v3, v2
-; PWR10BE-NEXT:    xxswapd v3, v2
-; PWR10BE-NEXT:    vadduwm v2, v2, v3
-; PWR10BE-NEXT:    xxspltw v3, v2, 1
-; PWR10BE-NEXT:    vadduwm v2, v2, v3
+; PWR10BE-NEXT:    xxspltib v4, 1
+; PWR10BE-NEXT:    xxlxor v3, v3, v3
+; PWR10BE-NEXT:    li r3, 12
+; PWR10BE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR10BE-NEXT:    vsumsws v2, v2, v3
 ; PWR10BE-NEXT:    vextuwlx r3, r3, v2
 ; PWR10BE-NEXT:    extsw r3, r3
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = sext <16 x i8> %a to <16 x i32>
   %1 = call i32 @llvm.vector.reduce.add.v16i32(<16 x i32> %0)
   ret i32 %1
 }
 
 define dso_local zeroext i32 @v16i8tov16i32_zero(<16 x i8> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v16i8tov16i32_zero:
 ; PWR9LE:       # %bb.0: # %entry
-; PWR9LE-NEXT:    addis r3, r2, .LCPI18_0@toc@ha
-; PWR9LE-NEXT:    xxlxor v4, v4, v4
-; PWR9LE-NEXT:    addi r3, r3, .LCPI18_0@toc@l
-; PWR9LE-NEXT:    lxv v3, 0(r3)
-; PWR9LE-NEXT:    addis r3, r2, .LCPI18_1@toc@ha
-; PWR9LE-NEXT:    addi r3, r3, .LCPI18_1@toc@l
-; PWR9LE-NEXT:    lxv v5, 0(r3)
-; PWR9LE-NEXT:    addis r3, r2, .LCPI18_2@toc@ha
-; PWR9LE-NEXT:    vperm v3, v4, v2, v3
-; PWR9LE-NEXT:    addi r3, r3, .LCPI18_2@toc@l
-; PWR9LE-NEXT:    lxv v0, 0(r3)
-; PWR9LE-NEXT:    addis r3, r2, .LCPI18_3@toc@ha
-; PWR9LE-NEXT:    vperm v5, v4, v2, v5
-; PWR9LE-NEXT:    addi r3, r3, .LCPI18_3@toc@l
-; PWR9LE-NEXT:    lxv v1, 0(r3)
-; PWR9LE-NEXT:    vadduwm v3, v5, v3
-; PWR9LE-NEXT:    li r3, 0
-; PWR9LE-NEXT:    vperm v0, v4, v2, v0
-; PWR9LE-NEXT:    vperm v2, v4, v2, v1
-; PWR9LE-NEXT:    vadduwm v2, v2, v0
-; PWR9LE-NEXT:    vadduwm v2, v3, v2
-; PWR9LE-NEXT:    xxswapd v3, v2
-; PWR9LE-NEXT:    vadduwm v2, v2, v3
-; PWR9LE-NEXT:    xxspltw v3, v2, 2
-; PWR9LE-NEXT:    vadduwm v2, v2, v3
+; PWR9LE-NEXT:    xxspltib v4, 1
+; PWR9LE-NEXT:    xxlxor v3, v3, v3
+; PWR9LE-NEXT:    li r3, 12
+; PWR9LE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR9LE-NEXT:    vsumsws v2, v2, v3
 ; PWR9LE-NEXT:    vextuwrx r3, r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i8tov16i32_zero:
 ; PWR9BE:       # %bb.0: # %entry
-; PWR9BE-NEXT:    addis r3, r2, .LCPI18_0@toc@ha
-; PWR9BE-NEXT:    xxlxor v4, v4, v4
-; PWR9BE-NEXT:    addi r3, r3, .LCPI18_0@toc@l
-; PWR9BE-NEXT:    lxv v3, 0(r3)
-; PWR9BE-NEXT:    addis r3, r2, .LCPI18_1@toc@ha
-; PWR9BE-NEXT:    addi r3, r3, .LCPI18_1@toc@l
-; PWR9BE-NEXT:    lxv v5, 0(r3)
-; PWR9BE-NEXT:    addis r3, r2, .LCPI18_2@toc@ha
-; PWR9BE-NEXT:    vperm v3, v4, v2, v3
-; PWR9BE-NEXT:    addi r3, r3, .LCPI18_2@toc@l
-; PWR9BE-NEXT:    lxv v0, 0(r3)
-; PWR9BE-NEXT:    addis r3, r2, .LCPI18_3@toc@ha
-; PWR9BE-NEXT:    vperm v5, v4, v2, v5
-; PWR9BE-NEXT:    addi r3, r3, .LCPI18_3@toc@l
-; PWR9BE-NEXT:    lxv v1, 0(r3)
-; PWR9BE-NEXT:    vadduwm v3, v5, v3
-; PWR9BE-NEXT:    li r3, 0
-; PWR9BE-NEXT:    vperm v0, v4, v2, v0
-; PWR9BE-NEXT:    vperm v2, v4, v2, v1
-; PWR9BE-NEXT:    vadduwm v2, v2, v0
-; PWR9BE-NEXT:    vadduwm v2, v3, v2
-; PWR9BE-NEXT:    xxswapd v3, v2
-; PWR9BE-NEXT:    vadduwm v2, v2, v3
-; PWR9BE-NEXT:    xxspltw v3, v2, 1
-; PWR9BE-NEXT:    vadduwm v2, v2, v3
+; PWR9BE-NEXT:    xxspltib v4, 1
+; PWR9BE-NEXT:    xxlxor v3, v3, v3
+; PWR9BE-NEXT:    li r3, 12
+; PWR9BE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR9BE-NEXT:    vsumsws v2, v2, v3
 ; PWR9BE-NEXT:    vextuwlx r3, r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v16i8tov16i32_zero:
 ; PWR10LE:       # %bb.0: # %entry
-; PWR10LE-NEXT:    plxv v3, .LCPI18_0@PCREL(0), 1
-; PWR10LE-NEXT:    plxv v5, .LCPI18_1@PCREL(0), 1
-; PWR10LE-NEXT:    xxlxor v4, v4, v4
-; PWR10LE-NEXT:    li r3, 0
-; PWR10LE-NEXT:    vperm v3, v4, v2, v3
-; PWR10LE-NEXT:    plxv v0, .LCPI18_2@PCREL(0), 1
-; PWR10LE-NEXT:    plxv v1, .LCPI18_3@PCREL(0), 1
-; PWR10LE-NEXT:    vperm v5, v4, v2, v5
-; PWR10LE-NEXT:    vperm v0, v4, v2, v0
-; PWR10LE-NEXT:    vperm v2, v4, v2, v1
-; PWR10LE-NEXT:    vadduwm v2, v2, v0
-; PWR10LE-NEXT:    vadduwm v3, v5, v3
-; PWR10LE-NEXT:    vadduwm v2, v3, v2
-; PWR10LE-NEXT:    xxswapd v3, v2
-; PWR10LE-NEXT:    vadduwm v2, v2, v3
-; PWR10LE-NEXT:    xxspltw v3, v2, 2
-; PWR10LE-NEXT:    vadduwm v2, v2, v3
+; PWR10LE-NEXT:    xxspltib v4, 1
+; PWR10LE-NEXT:    xxlxor v3, v3, v3
+; PWR10LE-NEXT:    li r3, 12
+; PWR10LE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR10LE-NEXT:    vsumsws v2, v2, v3
 ; PWR10LE-NEXT:    vextuwrx r3, r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i8tov16i32_zero:
 ; PWR10BE:       # %bb.0: # %entry
-; PWR10BE-NEXT:    addis r3, r2, .LCPI18_0@toc@ha
-; PWR10BE-NEXT:    xxlxor v4, v4, v4
-; PWR10BE-NEXT:    addi r3, r3, .LCPI18_0@toc@l
-; PWR10BE-NEXT:    lxv v3, 0(r3)
-; PWR10BE-NEXT:    addis r3, r2, .LCPI18_1@toc@ha
-; PWR10BE-NEXT:    addi r3, r3, .LCPI18_1@toc@l
-; PWR10BE-NEXT:    lxv v5, 0(r3)
-; PWR10BE-NEXT:    addis r3, r2, .LCPI18_2@toc@ha
-; PWR10BE-NEXT:    addi r3, r3, .LCPI18_2@toc@l
-; PWR10BE-NEXT:    vperm v3, v4, v2, v3
-; PWR10BE-NEXT:    lxv v0, 0(r3)
-; PWR10BE-NEXT:    addis r3, r2, .LCPI18_3@toc@ha
-; PWR10BE-NEXT:    addi r3, r3, .LCPI18_3@toc@l
-; PWR10BE-NEXT:    vperm v5, v4, v2, v5
-; PWR10BE-NEXT:    lxv v1, 0(r3)
-; PWR10BE-NEXT:    li r3, 0
-; PWR10BE-NEXT:    vadduwm v3, v5, v3
-; PWR10BE-NEXT:    vperm v0, v4, v2, v0
-; PWR10BE-NEXT:    vperm v2, v4, v2, v1
-; PWR10BE-NEXT:    vadduwm v2, v2, v0
-; PWR10BE-NEXT:    vadduwm v2, v3, v2
-; PWR10BE-NEXT:    xxswapd v3, v2
-; PWR10BE-NEXT:    vadduwm v2, v2, v3
-; PWR10BE-NEXT:    xxspltw v3, v2, 1
-; PWR10BE-NEXT:    vadduwm v2, v2, v3
+; PWR10BE-NEXT:    xxspltib v4, 1
+; PWR10BE-NEXT:    xxlxor v3, v3, v3
+; PWR10BE-NEXT:    li r3, 12
+; PWR10BE-NEXT:    vmsumubm v2, v2, v4, v3
+; PWR10BE-NEXT:    vsumsws v2, v2, v3
 ; PWR10BE-NEXT:    vextuwlx r3, r3, v2
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = zext <16 x i8> %a to <16 x i32>
   %1 = call i32 @llvm.vector.reduce.add.v16i32(<16 x i32> %0)
   ret i32 %1
 }
 
 declare i32 @llvm.vector.reduce.add.v2i32(<2 x i32>) #0
 declare i32 @llvm.vector.reduce.add.v4i32(<4 x i32>) #0
 declare i32 @llvm.vector.reduce.add.v8i32(<8 x i32>) #0
 declare i32 @llvm.vector.reduce.add.v16i32(<16 x i32>) #0
 declare i32 @llvm.vector.reduce.add.v32i32(<32 x i32>) #0
 
 ;;
 ;; Vectors of i64
 ;;
 define dso_local i64 @v2i64(<2 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v2i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v2i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vaddudm v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v2i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v2i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    vaddudm v2, v2, v3
 ; PWR10BE-NEXT:    mfvsrd r3, v2
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v4i64(<4 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v4i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v4i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vaddudm v2, v2, v3
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vaddudm v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v4i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v4i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vaddudm v2, v2, v3
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    vaddudm v2, v2, v3
 ; PWR10BE-NEXT:    mfvsrd r3, v2
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v8i64(<8 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v8i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    vaddudm v3, v3, v5
 ; PWR9LE-NEXT:    vaddudm v2, v2, v4
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v8i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vaddudm v3, v3, v5
 ; PWR9BE-NEXT:    vaddudm v2, v2, v4
 ; PWR9BE-NEXT:    vaddudm v2, v2, v3
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vaddudm v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v8i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    vaddudm v3, v3, v5
 ; PWR10LE-NEXT:    vaddudm v2, v2, v4
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v8i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vaddudm v3, v3, v5
 ; PWR10BE-NEXT:    vaddudm v2, v2, v4
 ; PWR10BE-NEXT:    vaddudm v2, v2, v3
 ; PWR10BE-NEXT:    xxswapd v3, v2

 ; PWR9LE-NEXT:    vaddudm v2, v2, v6
 ; PWR9LE-NEXT:    vaddudm v5, v5, v9
 ; PWR9LE-NEXT:    vaddudm v3, v3, v7
 ; PWR9LE-NEXT:    vaddudm v3, v3, v5
 ; PWR9LE-NEXT:    vaddudm v2, v2, v4
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vaddudm v4, v4, v8
 ; PWR9BE-NEXT:    vaddudm v2, v2, v6
 ; PWR9BE-NEXT:    vaddudm v5, v5, v9
 ; PWR9BE-NEXT:    vaddudm v3, v3, v7

 ; PWR10LE-NEXT:    vaddudm v5, v5, v9
 ; PWR10LE-NEXT:    vaddudm v3, v3, v7
 ; PWR10LE-NEXT:    vaddudm v3, v3, v5
 ; PWR10LE-NEXT:    vaddudm v2, v2, v6
 ; PWR10LE-NEXT:    vaddudm v2, v2, v4
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vaddudm v4, v4, v8
 ; PWR10BE-NEXT:    vaddudm v5, v5, v9
 ; PWR10BE-NEXT:    vaddudm v3, v3, v7
 ; PWR10BE-NEXT:    vaddudm v3, v3, v5

 ; PWR9LE-NEXT:    vextsb2d v7, v7
 ; PWR9LE-NEXT:    vperm v2, v2, v2, v8
 ; PWR9LE-NEXT:    vextsb2d v2, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v7
 ; PWR9LE-NEXT:    vaddudm v2, v1, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i8tov16i64_sign:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    addis r3, r2, .LCPI23_0@toc@ha
 ; PWR9BE-NEXT:    addi r3, r3, .LCPI23_0@toc@l
 ; PWR9BE-NEXT:    lxv v3, 0(r3)
 ; PWR9BE-NEXT:    addis r3, r2, .LCPI23_1@toc@ha

 ; PWR10LE-NEXT:    vaddudm v5, v0, v5
 ; PWR10LE-NEXT:    vaddudm v3, v4, v3
 ; PWR10LE-NEXT:    vaddudm v3, v3, v5
 ; PWR10LE-NEXT:    vaddudm v4, v6, v1
 ; PWR10LE-NEXT:    vaddudm v2, v4, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i8tov16i64_sign:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    addis r3, r2, .LCPI23_0@toc@ha
 ; PWR10BE-NEXT:    addi r3, r3, .LCPI23_0@toc@l
 ; PWR10BE-NEXT:    lxv v3, 0(r3)
 ; PWR10BE-NEXT:    addis r3, r2, .LCPI23_1@toc@ha

 ; PWR9LE-NEXT:    vperm v8, v4, v2, v8
 ; PWR9LE-NEXT:    vperm v2, v4, v2, v9
 ; PWR9LE-NEXT:    vaddudm v4, v7, v6
 ; PWR9LE-NEXT:    vaddudm v2, v2, v8
 ; PWR9LE-NEXT:    vaddudm v2, v4, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i8tov16i64_zero:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    addis r3, r2, .LCPI24_0@toc@ha
 ; PWR9BE-NEXT:    xxlxor v4, v4, v4
 ; PWR9BE-NEXT:    addi r3, r3, .LCPI24_0@toc@l
 ; PWR9BE-NEXT:    lxv v3, 0(r3)

 ; PWR10LE-NEXT:    vaddudm v4, v1, v0
 ; PWR10LE-NEXT:    vaddudm v3, v5, v3
 ; PWR10LE-NEXT:    vaddudm v3, v3, v4
 ; PWR10LE-NEXT:    vaddudm v4, v7, v6
 ; PWR10LE-NEXT:    vaddudm v2, v4, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i8tov16i64_zero:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    addis r3, r2, .LCPI24_0@toc@ha
 ; PWR10BE-NEXT:    xxlxor v4, v4, v4
 ; PWR10BE-NEXT:    addi r3, r3, .LCPI24_0@toc@l
 ; PWR10BE-NEXT:    lxv v3, 0(r3)