[DAG] Fold Op(vecreduce(a), vecreduce(b)) into vecreduce(Op(a,b))

[dmgreen]

Summary

So long as the operation is reassociative, we can reassociate the double vecreduce from for example fadd(vecreduce(a), vecreduce(b)) to vecreduce(fadd(a,b)). This will in general save a few instructions, but some architectures (MVE) require the opposite fold, so a shouldExpandReduction is added to account for it. Only targets that use shouldExpandReduction will be affected.

[spatel]

We probably want this for all associate reductions - fmul, min/max, integer ops, etc. Generalize into a helper function that takes an opcode and maps it to the corresponding reduction opcode? Subsequent patches then just need to add set of tests and another case into a switch or something like that.

[RKSimon]

Should we be doing this at IR level as well? https://simd.godbolt.org/z/aTE8qjMET (to be clear - I'm not expecting this patch to address it)

[barannikov88]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

15695

[barannikov88]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

15695

Never mind, I misunderstood the transformation.
Please copy a comment showing the transformation from the description to code.

[dmgreen]

In D141870#4057159, @RKSimon wrote:

Should we be doing this at IR level as well? https://simd.godbolt.org/z/aTE8qjMET (to be clear - I'm not expecting this patch to address it)

The case that was reported to me was a <16 x float> reduction and a <4 x float> reduction added together, which needs to be handled after legalization. Similar to fadd_reduct_reassoc_v4v8f32 but with a 16x and more going on.

I'm not sure that certain architectures would like the combine in for all reductions. It can depend on whether the reduction+add is cheap. MVE might prefer multiple reductions for integer adds for example, which can require quite precise cost modelling, unfortunately.

In D141870#4057153, @spatel wrote:

We probably want this for all associate reductions - fmul, min/max, integer ops, etc. Generalize into a helper function that takes an opcode and maps it to the corresponding reduction opcode? Subsequent patches then just need to add set of tests and another case into a switch or something like that.

Sounds good. I can take a look, and see if it needs a target-hook to prevent the transform for any backends.

[dmgreen]

I've update this to now handle all the reduction types, and added tests for various architectures.

[spatel]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

1325

Flags param is not used?
They could propagate with:

SelectionDAG::FlagInserter FlagsInserter(DAG, Flags);

[dmgreen]

Thanks - Now using a FlagInserter to copy the flags. The flags for integer ops are not passed through, as nsw/nuw/etc may not apply to the new operations.

[spatel]

LGTM

In D141870#4109408, @dmgreen wrote:

Thanks - Now using a FlagInserter to copy the flags. The flags for integer ops are not passed through, as nsw/nuw/etc may not apply to the new operations.

Ah, right - I was only thinking of FMF flags in this context. That's worth a header comment on reassociateReduction. It looks like we drop all flags in reassociateOps, but that's not called from FP opcodes.

We could also make it less likely to go wrong by using a default flags param and/or naming that param "FMFFlags" or something like that.
For example, we have functions like this:

SDValue getMemBasePlusOffset(SDValue Base, SDValue Offset, const SDLoc &DL,
                             const SDNodeFlags Flags = SDNodeFlags());

[dzhidzhoev]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5521

Should here be "min/max/...(vecreduce..."?

[dmgreen]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5521

Yep! Thanks. Updated in 14914fb1573f0393979492238735beecde65b3bb.

llvm/include/llvm/CodeGen/TargetLowering.h

public:
   }
 
   /// Return true if the @llvm.get.active.lane.mask intrinsic should be expanded
   /// using generic code in SelectionDAGBuilder.
   virtual bool shouldExpandGetActiveLaneMask(EVT VT, EVT OpVT) const {
     return true;
   }
 
+  // Return true if op(vecreduce(x), vecreduce(y)) should be reassociated to
+  // vecreduce(op(x, y)) for the reduction opcode RedOpc.
+  virtual bool shouldReassociateReduction(unsigned RedOpc, EVT VT) const {
+    return true;
+  }
+
   /// Return true if it is profitable to convert a select of FP constants into
   /// a constant pool load whose address depends on the select condition. The
   /// parameter may be used to differentiate a select with FP compare from
   /// integer compare.
   virtual bool reduceSelectOfFPConstantLoads(EVT CmpOpVT) const {
     return true;
   }
 

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

bool reassociationCanBreakAddressingModePattern(unsigned Opc,
                                                     const SDLoc &DL,
                                                     SDNode *N,
                                                     SDValue N0,
                                                     SDValue N1);
     SDValue reassociateOpsCommutative(unsigned Opc, const SDLoc &DL, SDValue N0,
                                       SDValue N1);
     SDValue reassociateOps(unsigned Opc, const SDLoc &DL, SDValue N0,
                            SDValue N1, SDNodeFlags Flags);
+    SDValue reassociateReduction(unsigned ResOpc, unsigned Opc, const SDLoc &DL,
+                                 EVT VT, SDValue N0, SDValue N1,
+                                 SDNodeFlags Flags = SDNodeFlags());
 
     SDValue visitShiftByConstant(SDNode *N);
 
     SDValue foldSelectOfConstants(SDNode *N);
     SDValue foldVSelectOfConstants(SDNode *N);
     SDValue foldBinOpIntoSelect(SDNode *BO);
     bool SimplifySelectOps(SDNode *SELECT, SDValue LHS, SDValue RHS);
     SDValue hoistLogicOpWithSameOpcodeHands(SDNode *N);
SDValue DAGCombiner::reassociateOps(unsigned Opc, const SDLoc &DL, SDValue N0,
 
   if (SDValue Combined = reassociateOpsCommutative(Opc, DL, N0, N1))
     return Combined;
   if (SDValue Combined = reassociateOpsCommutative(Opc, DL, N1, N0))
     return Combined;
   return SDValue();
 }
 
+// Try to fold Opc(vecreduce(x), vecreduce(y)) -> vecreduce(Opc(x, y))
+// Note that we only expect Flags to be passed from FP operations. For integer
+// operations they need to be dropped.
+SDValue DAGCombiner::reassociateReduction(unsigned RedOpc, unsigned Opc,
+                                          const SDLoc &DL, EVT VT, SDValue N0,
+                                          SDValue N1, SDNodeFlags Flags) {
+  if (N0.getOpcode() == RedOpc && N1.getOpcode() == RedOpc &&
+      N0.getOperand(0).getValueType() == N1.getOperand(0).getValueType() &&
+      N0->hasOneUse() && N1->hasOneUse() &&
+      TLI.isOperationLegalOrCustom(Opc, N0.getOperand(0).getValueType()) &&
+      TLI.shouldReassociateReduction(RedOpc, N0.getOperand(0).getValueType())) {
+    SelectionDAG::FlagInserter FlagsInserter(DAG, Flags);
+    return DAG.getNode(RedOpc, DL, VT,
+                       DAG.getNode(Opc, DL, N0.getOperand(0).getValueType(),
+                                   N0.getOperand(0), N1.getOperand(0)));
+  }
+  return SDValue();
+}
+
 SDValue DAGCombiner::CombineTo(SDNode *N, const SDValue *To, unsigned NumTo,
                                bool AddTo) {
   assert(N->getNumValues() == NumTo && "Broken CombineTo call!");
   ++NodesCombined;
   LLVM_DEBUG(dbgs() << "\nReplacing.1 "; N->dump(&DAG); dbgs() << "\nWith: ";
              To[0].dump(&DAG);
              dbgs() << " and " << NumTo - 1 << " other values\n");
   for (unsigned i = 0, e = NumTo; i != e; ++i)
auto ReassociateAddOr = [&](SDValue N0, SDValue N1) {
                            N0.getOperand(1));
       }
       return SDValue();
     };
     if (SDValue Add = ReassociateAddOr(N0, N1))
       return Add;
     if (SDValue Add = ReassociateAddOr(N1, N0))
       return Add;
+
+    // Fold add(vecreduce(x), vecreduce(y)) -> vecreduce(add(x, y))
+    if (SDValue SD =
+            reassociateReduction(ISD::VECREDUCE_ADD, ISD::ADD, DL, VT, N0, N1))
+      return SD;
   }
   // fold ((0-A) + B) -> B-A
   if (N0.getOpcode() == ISD::SUB && isNullOrNullSplat(N0.getOperand(0)))
     return DAG.getNode(ISD::SUB, DL, VT, N1, N0.getOperand(1));
 
   // fold (A + (0-B)) -> A-B
   if (N1.getOpcode() == ISD::SUB && isNullOrNullSplat(N1.getOperand(0)))
     return DAG.getNode(ISD::SUB, DL, VT, N0, N1.getOperand(1));
if ((!LegalOperations || TLI.isOperationLegalOrCustom(ISD::AND, VT)) &&
       return DAG.getNode(ISD::AND, DL, VT, N0, DAG.getBuildVector(VT, DL, Mask));
     }
   }
 
   // reassociate mul
   if (SDValue RMUL = reassociateOps(ISD::MUL, DL, N0, N1, N->getFlags()))
     return RMUL;
 
+  // Fold mul(vecreduce(x), vecreduce(y)) -> vecreduce(mul(x, y))
+  if (SDValue SD =
+          reassociateReduction(ISD::VECREDUCE_MUL, ISD::MUL, DL, VT, N0, N1))
+    return SD;
+
   // Simplify the operands using demanded-bits information.
   if (SimplifyDemandedBits(SDValue(N, 0)))
     return SDValue(N, 0);
 
   return SDValue();
 }
 
 /// Return true if divmod libcall is available.
SDValue DAGCombiner::visitIMINMAX(SDNode *N) {
   if (Opcode == ISD::SMIN || Opcode == ISD::SMAX)
     if (SDValue S = PerformMinMaxFpToSatCombine(
             N0, N1, N0, N1, Opcode == ISD::SMIN ? ISD::SETLT : ISD::SETGT, DAG))
       return S;
   if (Opcode == ISD::UMIN)
     if (SDValue S = PerformUMinFpToSatCombine(N0, N1, N0, N1, ISD::SETULT, DAG))
       return S;
 
+  // Fold and(vecreduce(x), vecreduce(y)) -> vecreduce(and(x, y))
+  auto ReductionOpcode = [](unsigned Opcode) {
+    switch (Opcode) {
+    case ISD::SMIN:
+      return ISD::VECREDUCE_SMIN;
+    case ISD::SMAX:
+      return ISD::VECREDUCE_SMAX;
+    case ISD::UMIN:
+      return ISD::VECREDUCE_UMIN;
+    case ISD::UMAX:
+      return ISD::VECREDUCE_UMAX;
+    default:
+      llvm_unreachable("Unexpected opcode");
+    }
+  };
+  if (SDValue SD = reassociateReduction(ReductionOpcode(Opcode), Opcode,
+                                        SDLoc(N), VT, N0, N1))
+    return SD;
+
   // Simplify the operands using demanded-bits information.
   if (SimplifyDemandedBits(SDValue(N, 0)))
     return SDValue(N, 0);
 
   return SDValue();
 }
 
 /// If this is a bitwise logic instruction and both operands have the same
SDValue DAGCombiner::visitAND(SDNode *N) {
 
   if (SDValue NewSel = foldBinOpIntoSelect(N))
     return NewSel;
 
   // reassociate and
   if (SDValue RAND = reassociateOps(ISD::AND, SDLoc(N), N0, N1, N->getFlags()))
     return RAND;
 
+  // Fold and(vecreduce(x), vecreduce(y)) -> vecreduce(and(x, y))
+  if (SDValue SD = reassociateReduction(ISD::VECREDUCE_AND, ISD::AND, SDLoc(N),
+                                        VT, N0, N1))
+    return SD;
+
   // fold (and (or x, C), D) -> D if (C & D) == D
   auto MatchSubset = [](ConstantSDNode *LHS, ConstantSDNode *RHS) {
     return RHS->getAPIntValue().isSubsetOf(LHS->getAPIntValue());
   };
   if (N0.getOpcode() == ISD::OR &&
       ISD::matchBinaryPredicate(N0.getOperand(1), N1, MatchSubset))
     return N1;
 
if (SDValue BSwap = MatchBSwapHWord(N, N0, N1))
     return BSwap;
   if (SDValue BSwap = MatchBSwapHWordLow(N, N0, N1))
     return BSwap;
 
   // reassociate or
   if (SDValue ROR = reassociateOps(ISD::OR, SDLoc(N), N0, N1, N->getFlags()))
     return ROR;
 
+  // Fold or(vecreduce(x), vecreduce(y)) -> vecreduce(or(x, y))
+  if (SDValue SD = reassociateReduction(ISD::VECREDUCE_OR, ISD::OR, SDLoc(N),
+                                        VT, N0, N1))
+    return SD;
+
   // Canonicalize (or (and X, c1), c2) -> (and (or X, c2), c1|c2)
   // iff (c1 & c2) != 0 or c1/c2 are undef.
   auto MatchIntersect = [](ConstantSDNode *C1, ConstantSDNode *C2) {
     return !C1 || !C2 || C1->getAPIntValue().intersects(C2->getAPIntValue());
   };
   if (N0.getOpcode() == ISD::AND && N0->hasOneUse() &&
       ISD::matchBinaryPredicate(N0.getOperand(1), N1, MatchIntersect, true)) {
     if (SDValue COR = DAG.FoldConstantArithmetic(ISD::OR, SDLoc(N1), VT,
SDValue DAGCombiner::visitXOR(SDNode *N) {
 
   if (SDValue NewSel = foldBinOpIntoSelect(N))
     return NewSel;
 
   // reassociate xor
   if (SDValue RXOR = reassociateOps(ISD::XOR, DL, N0, N1, N->getFlags()))
     return RXOR;
 
+  // Fold xor(vecreduce(x), vecreduce(y)) -> vecreduce(xor(x, y))
+  if (SDValue SD =
+          reassociateReduction(ISD::VECREDUCE_XOR, ISD::XOR, DL, VT, N0, N1))
+    return SD;
+
   // fold (a^b) -> (a|b) iff a and b share no bits.
   if ((!LegalOperations || TLI.isOperationLegal(ISD::OR, VT)) &&
       DAG.haveNoCommonBitsSet(N0, N1))
     return DAG.getNode(ISD::OR, DL, VT, N0, N1);
 
   // look for 'add-like' folds:
   // XOR(N0,MIN_SIGNED_VALUE) == ADD(N0,MIN_SIGNED_VALUE)
   if ((!LegalOperations || TLI.isOperationLegal(ISD::ADD, VT)) &&
if (TLI.isOperationLegalOrCustom(ISD::FMUL, VT) && !N0CFP && !N1CFP) {
       if (N0.getOpcode() == ISD::FADD && N1.getOpcode() == ISD::FADD &&
           N0.getOperand(0) == N0.getOperand(1) &&
           N1.getOperand(0) == N1.getOperand(1) &&
           N0.getOperand(0) == N1.getOperand(0)) {
         return DAG.getNode(ISD::FMUL, DL, VT, N0.getOperand(0),
                            DAG.getConstantFP(4.0, DL, VT));
       }
     }
+
+    // Fold fadd(vecreduce(x), vecreduce(y)) -> vecreduce(fadd(x, y))
+    if (SDValue SD = reassociateReduction(ISD::VECREDUCE_FADD, ISD::FADD, DL,
+                                          VT, N0, N1, Flags))
+      return SD;
   } // enable-unsafe-fp-math
 
   // FADD -> FMA combines:
   if (SDValue Fused = visitFADDForFMACombine<EmptyMatchContext>(N)) {
     AddToWorklist(Fused.getNode());
     return Fused;
   }
   return SDValue();
 }
if (Options.UnsafeFPMath || Flags.hasAllowReassociation()) {
     // Match a special-case: we convert X * 2.0 into fadd.
     // fmul (fadd X, X), C -> fmul X, 2.0 * C
     if (N0.getOpcode() == ISD::FADD && N0.hasOneUse() &&
         N0.getOperand(0) == N0.getOperand(1)) {
       const SDValue Two = DAG.getConstantFP(2.0, DL, VT);
       SDValue MulConsts = DAG.getNode(ISD::FMUL, DL, VT, Two, N1);
       return DAG.getNode(ISD::FMUL, DL, VT, N0.getOperand(0), MulConsts);
     }
+
+    // Fold fmul(vecreduce(x), vecreduce(y)) -> vecreduce(fmul(x, y))
+    if (SDValue SD = reassociateReduction(ISD::VECREDUCE_FMUL, ISD::FMUL, DL,
+                                          VT, N0, N1, Flags))
+      return SD;
   }
 
   // fold (fmul X, 2.0) -> (fadd X, X)
   if (N1CFP && N1CFP->isExactlyValue(+2.0))
     return DAG.getNode(ISD::FADD, DL, VT, N0, N0);
 
   // fold (fmul X, -1.0) -> (fsub -0.0, X)
   if (N1CFP && N1CFP->isExactlyValue(-1.0)) {
if (AF.isInfinity() || (Flags.hasNoInfs() && AF.isLargest())) {
       // maxnum(X, -inf) -> X if nnan
       // minimum(X, +inf) -> X
       // maximum(X, -inf) -> X
       if (IsMin != AF.isNegative() && (PropagatesNaN || Flags.hasNoNaNs()))
         return N->getOperand(0);
     }
   }
 
+  const TargetOptions &Options = DAG.getTarget().Options;
+  if ((Options.UnsafeFPMath && Options.NoSignedZerosFPMath) ||
+      (Flags.hasAllowReassociation() && Flags.hasNoSignedZeros()))
+    if (SDValue SD = reassociateReduction(IsMin ? ISD::VECREDUCE_FMIN
+                                                : ISD::VECREDUCE_FMAX,
+                                          Opc, SDLoc(N), VT, N0, N1, Flags))
+      return SD;
+
   return SDValue();
 }
 
 SDValue DAGCombiner::visitFABS(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   EVT VT = N->getValueType(0);
 
   // fold (fabs c1) -> fabs(c1)

llvm/lib/Target/ARM/ARMISelLowering.h

bool isExtractSubvectorCheap(EVT ResVT, EVT SrcVT,
                                  unsigned Index) const override;
 
     bool shouldFormOverflowOp(unsigned Opcode, EVT VT,
                               bool MathUsed) const override {
       // Using overflow ops for overflow checks only should beneficial on ARM.
       return TargetLowering::shouldFormOverflowOp(Opcode, VT, true);
     }
 
+    bool shouldReassociateReduction(unsigned Opc, EVT VT) const override {
+      return Opc != ISD::VECREDUCE_ADD;
+    }
+
     /// Returns true if an argument of type Ty needs to be passed in a
     /// contiguous block of registers in calling convention CallConv.
     bool functionArgumentNeedsConsecutiveRegisters(
         Type *Ty, CallingConv::ID CallConv, bool isVarArg,
         const DataLayout &DL) const override;
 
     /// If a physical register, this returns the register that receives the
     /// exception address on entry to an EH pad.

llvm/test/CodeGen/AArch64/aarch64-addv.ll

; CHECK-NEXT:    ret
   %bin.rdx = load <16 x i32>, ptr %arr
   %r = call i32 @llvm.vector.reduce.add.v16i32(<16 x i32> %bin.rdx)
   ret i32 %r
 }
 
 define i8 @addv_combine_i8(<8 x i8> %a1, <8 x i8> %a2) {
 ; CHECK-LABEL: addv_combine_i8:
 ; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    add v0.8b, v0.8b, v1.8b
 ; CHECK-NEXT:    addv b0, v0.8b
-; CHECK-NEXT:    addv b1, v1.8b
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    fmov w9, s1
-; CHECK-NEXT:    add w0, w8, w9
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
 entry:
   %rdx.1 = call i8 @llvm.vector.reduce.add.v8i8(<8 x i8> %a1)
   %rdx.2 = call i8 @llvm.vector.reduce.add.v8i8(<8 x i8> %a2)
   %r = add i8 %rdx.1, %rdx.2
   ret i8 %r
 }
 
 define i16 @addv_combine_i16(<4 x i16> %a1, <4 x i16> %a2) {
 ; CHECK-LABEL: addv_combine_i16:
 ; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    add v0.4h, v0.4h, v1.4h
 ; CHECK-NEXT:    addv h0, v0.4h
-; CHECK-NEXT:    addv h1, v1.4h
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    fmov w9, s1
-; CHECK-NEXT:    add w0, w8, w9
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
 entry:
   %rdx.1 = call i16 @llvm.vector.reduce.add.v4i16(<4 x i16> %a1)
   %rdx.2 = call i16 @llvm.vector.reduce.add.v4i16(<4 x i16> %a2)
   %r = add i16 %rdx.1, %rdx.2
   ret i16 %r
 }
 

llvm/test/CodeGen/AArch64/double_reduct.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc --mtriple=aarch64-eabi < %s | FileCheck %s
 
 define float @add_f32(<8 x float> %a, <4 x float> %b) {
 ; CHECK-LABEL: add_f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fadd v0.4s, v0.4s, v1.4s
-; CHECK-NEXT:    faddp v2.4s, v2.4s, v2.4s
+; CHECK-NEXT:    fadd v0.4s, v0.4s, v2.4s
 ; CHECK-NEXT:    faddp v0.4s, v0.4s, v0.4s
-; CHECK-NEXT:    faddp s1, v2.2s
 ; CHECK-NEXT:    faddp s0, v0.2s
-; CHECK-NEXT:    fadd s0, s0, s1
 ; CHECK-NEXT:    ret
   %r1 = call fast float @llvm.vector.reduce.fadd.f32.v8f32(float -0.0, <8 x float> %a)
   %r2 = call fast float @llvm.vector.reduce.fadd.f32.v4f32(float -0.0, <4 x float> %b)
   %r = fadd fast float %r1, %r2
   ret float %r
 }
 
 define float @fmul_f32(<8 x float> %a, <4 x float> %b) {
 ; CHECK-LABEL: fmul_f32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ext v3.16b, v2.16b, v2.16b, #8
 ; CHECK-NEXT:    fmul v0.4s, v0.4s, v1.4s
-; CHECK-NEXT:    fmul v1.2s, v2.2s, v3.2s
-; CHECK-NEXT:    ext v2.16b, v0.16b, v0.16b, #8
-; CHECK-NEXT:    fmul s1, s1, v1.s[1]
-; CHECK-NEXT:    fmul v0.2s, v0.2s, v2.2s
+; CHECK-NEXT:    fmul v0.4s, v0.4s, v2.4s
+; CHECK-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:    fmul v0.2s, v0.2s, v1.2s
 ; CHECK-NEXT:    fmul s0, s0, v0.s[1]
-; CHECK-NEXT:    fmul s0, s0, s1
 ; CHECK-NEXT:    ret
   %r1 = call fast float @llvm.vector.reduce.fmul.f32.v8f32(float 1.0, <8 x float> %a)
   %r2 = call fast float @llvm.vector.reduce.fmul.f32.v4f32(float 1.0, <4 x float> %b)
   %r = fmul fast float %r1, %r2
   ret float %r
 }
 
 define float @fmin_f32(<8 x float> %a, <4 x float> %b) {
; CHECK-NEXT:    ret
   %r2 = call i32 @llvm.vector.reduce.add.i32.v4i32(<4 x i32> %b)
   %r = add i32 %r1, %r2
   ret i32 %r
 }
 
 define i16 @add_ext_i16(<16 x i8> %a, <16 x i8> %b) {
 ; CHECK-LABEL: add_ext_i16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    uaddlv h0, v0.16b
-; CHECK-NEXT:    uaddlv h1, v1.16b
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    fmov w9, s1
-; CHECK-NEXT:    add w0, w8, w9
+; CHECK-NEXT:    uaddlp v1.8h, v1.16b
+; CHECK-NEXT:    uadalp v1.8h, v0.16b
+; CHECK-NEXT:    addv h0, v1.8h
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %ae = zext <16 x i8> %a to <16 x i16>
   %be = zext <16 x i8> %b to <16 x i16>
   %r1 = call i16 @llvm.vector.reduce.add.i16.v16i16(<16 x i16> %ae)
   %r2 = call i16 @llvm.vector.reduce.add.i16.v16i16(<16 x i16> %be)
   %r = add i16 %r1, %r2
   ret i16 %r
 }
 
 define i16 @add_ext_v32i16(<32 x i8> %a, <16 x i8> %b) {
 ; CHECK-LABEL: add_ext_v32i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    uaddl2 v3.8h, v0.16b, v1.16b
 ; CHECK-NEXT:    uaddl v0.8h, v0.8b, v1.8b
-; CHECK-NEXT:    uaddlv h2, v2.16b
 ; CHECK-NEXT:    add v0.8h, v0.8h, v3.8h
-; CHECK-NEXT:    fmov w9, s2
+; CHECK-NEXT:    uadalp v0.8h, v2.16b
 ; CHECK-NEXT:    addv h0, v0.8h
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    add w0, w8, w9
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %ae = zext <32 x i8> %a to <32 x i16>
   %be = zext <16 x i8> %b to <16 x i16>
   %r1 = call i16 @llvm.vector.reduce.add.i16.v32i16(<32 x i16> %ae)
   %r2 = call i16 @llvm.vector.reduce.add.i16.v16i16(<16 x i16> %be)
   %r = add i16 %r1, %r2
   ret i16 %r
 }
 
 define i32 @mul_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: mul_i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ext v3.16b, v2.16b, v2.16b, #8
 ; CHECK-NEXT:    mul v0.4s, v0.4s, v1.4s
-; CHECK-NEXT:    mul v1.2s, v2.2s, v3.2s
-; CHECK-NEXT:    ext v2.16b, v0.16b, v0.16b, #8
-; CHECK-NEXT:    mov w9, v1.s[1]
-; CHECK-NEXT:    fmov w11, s1
-; CHECK-NEXT:    mul v0.2s, v0.2s, v2.2s
-; CHECK-NEXT:    mul w9, w11, w9
+; CHECK-NEXT:    mul v0.4s, v0.4s, v2.4s
+; CHECK-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:    mul v0.2s, v0.2s, v1.2s
 ; CHECK-NEXT:    mov w8, v0.s[1]
-; CHECK-NEXT:    fmov w10, s0
-; CHECK-NEXT:    mul w8, w10, w8
-; CHECK-NEXT:    mul w0, w8, w9
+; CHECK-NEXT:    fmov w9, s0
+; CHECK-NEXT:    mul w0, w9, w8
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.mul.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.mul.i32.v4i32(<4 x i32> %b)
   %r = mul i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @and_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: and_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
-; CHECK-NEXT:    ext v3.16b, v2.16b, v2.16b, #8
+; CHECK-NEXT:    and v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
-; CHECK-NEXT:    and v2.8b, v2.8b, v3.8b
 ; CHECK-NEXT:    and v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov w8, v2.s[1]
-; CHECK-NEXT:    mov w9, v0.s[1]
-; CHECK-NEXT:    fmov w10, s0
-; CHECK-NEXT:    fmov w11, s2
-; CHECK-NEXT:    and w9, w10, w9
-; CHECK-NEXT:    and w8, w11, w8
+; CHECK-NEXT:    mov w8, v0.s[1]
+; CHECK-NEXT:    fmov w9, s0
 ; CHECK-NEXT:    and w0, w9, w8
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.and.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.and.i32.v4i32(<4 x i32> %b)
   %r = and i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @or_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: or_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
-; CHECK-NEXT:    ext v3.16b, v2.16b, v2.16b, #8
+; CHECK-NEXT:    orr v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
-; CHECK-NEXT:    orr v2.8b, v2.8b, v3.8b
 ; CHECK-NEXT:    orr v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov w8, v2.s[1]
-; CHECK-NEXT:    mov w9, v0.s[1]
-; CHECK-NEXT:    fmov w10, s0
-; CHECK-NEXT:    fmov w11, s2
-; CHECK-NEXT:    orr w9, w10, w9
-; CHECK-NEXT:    orr w8, w11, w8
+; CHECK-NEXT:    mov w8, v0.s[1]
+; CHECK-NEXT:    fmov w9, s0
 ; CHECK-NEXT:    orr w0, w9, w8
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.or.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.or.i32.v4i32(<4 x i32> %b)
   %r = or i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @xor_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: xor_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
-; CHECK-NEXT:    ext v3.16b, v2.16b, v2.16b, #8
+; CHECK-NEXT:    eor v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
-; CHECK-NEXT:    eor v2.8b, v2.8b, v3.8b
 ; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov w8, v2.s[1]
-; CHECK-NEXT:    mov w9, v0.s[1]
-; CHECK-NEXT:    fmov w10, s0
-; CHECK-NEXT:    fmov w11, s2
-; CHECK-NEXT:    eor w9, w10, w9
-; CHECK-NEXT:    eor w8, w11, w8
+; CHECK-NEXT:    mov w8, v0.s[1]
+; CHECK-NEXT:    fmov w9, s0
 ; CHECK-NEXT:    eor w0, w9, w8
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.xor.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.xor.i32.v4i32(<4 x i32> %b)
   %r = xor i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @umin_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: umin_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    umin v0.4s, v0.4s, v1.4s
-; CHECK-NEXT:    uminv s2, v2.4s
+; CHECK-NEXT:    umin v0.4s, v0.4s, v2.4s
 ; CHECK-NEXT:    uminv s0, v0.4s
-; CHECK-NEXT:    fmov w8, s2
-; CHECK-NEXT:    fmov w9, s0
-; CHECK-NEXT:    cmp w9, w8
-; CHECK-NEXT:    csel w0, w9, w8, lo
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.umin.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.umin.i32.v4i32(<4 x i32> %b)
   %r = call i32 @llvm.umin.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 define i32 @umax_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: umax_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    umax v0.4s, v0.4s, v1.4s
-; CHECK-NEXT:    umaxv s2, v2.4s
+; CHECK-NEXT:    umax v0.4s, v0.4s, v2.4s
 ; CHECK-NEXT:    umaxv s0, v0.4s
-; CHECK-NEXT:    fmov w8, s2
-; CHECK-NEXT:    fmov w9, s0
-; CHECK-NEXT:    cmp w9, w8
-; CHECK-NEXT:    csel w0, w9, w8, hi
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.umax.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.umax.i32.v4i32(<4 x i32> %b)
   %r = call i32 @llvm.umax.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 define i32 @smin_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: smin_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    smin v0.4s, v0.4s, v1.4s
-; CHECK-NEXT:    sminv s2, v2.4s
+; CHECK-NEXT:    smin v0.4s, v0.4s, v2.4s
 ; CHECK-NEXT:    sminv s0, v0.4s
-; CHECK-NEXT:    fmov w8, s2
-; CHECK-NEXT:    fmov w9, s0
-; CHECK-NEXT:    cmp w9, w8
-; CHECK-NEXT:    csel w0, w9, w8, lt
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.smin.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.smin.i32.v4i32(<4 x i32> %b)
   %r = call i32 @llvm.smin.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 define i32 @smax_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: smax_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    smax v0.4s, v0.4s, v1.4s
-; CHECK-NEXT:    smaxv s2, v2.4s
+; CHECK-NEXT:    smax v0.4s, v0.4s, v2.4s
 ; CHECK-NEXT:    smaxv s0, v0.4s
-; CHECK-NEXT:    fmov w8, s2
-; CHECK-NEXT:    fmov w9, s0
-; CHECK-NEXT:    cmp w9, w8
-; CHECK-NEXT:    csel w0, w9, w8, gt
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.smax.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.smax.i32.v4i32(<4 x i32> %b)
   %r = call i32 @llvm.smax.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 declare float @llvm.vector.reduce.fadd.f32.v8f32(float, <8 x float>)

llvm/test/CodeGen/AArch64/sve-doublereduct.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc --mtriple=aarch64-eabi -mattr=+sve2 < %s | FileCheck %s
 
 define float @add_f32(<vscale x 8 x float> %a, <vscale x 4 x float> %b) {
 ; CHECK-LABEL: add_f32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.s
 ; CHECK-NEXT:    fadd z0.s, z0.s, z1.s
-; CHECK-NEXT:    faddv s2, p0, z2.s
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    fadd z0.s, z0.s, z2.s
 ; CHECK-NEXT:    faddv s0, p0, z0.s
-; CHECK-NEXT:    fadd s0, s0, s2
+; CHECK-NEXT:    // kill: def $s0 killed $s0 killed $z0
 ; CHECK-NEXT:    ret
   %r1 = call fast float @llvm.vector.reduce.fadd.f32.nxv8f32(float -0.0, <vscale x 8 x float> %a)
   %r2 = call fast float @llvm.vector.reduce.fadd.f32.nxv4f32(float -0.0, <vscale x 4 x float> %b)
   %r = fadd fast float %r1, %r2
   ret float %r
 }
 
 ;define float @fmul_f32(<vscale x 8 x float> %a, <vscale x 4 x float> %b) {
; CHECK-NEXT:    ret
   %r = call float @llvm.maxnum.f32(float %r1, float %r2)
   ret float %r
 }
 
 
 define i32 @add_i32(<vscale x 8 x i32> %a, <vscale x 4 x i32> %b) {
 ; CHECK-LABEL: add_i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.s
 ; CHECK-NEXT:    add z0.s, z0.s, z1.s
-; CHECK-NEXT:    uaddv d2, p0, z2.s
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    add z0.s, z0.s, z2.s
 ; CHECK-NEXT:    uaddv d0, p0, z0.s
-; CHECK-NEXT:    fmov x8, d0
-; CHECK-NEXT:    fmov x9, d2
-; CHECK-NEXT:    add w0, w8, w9
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.add.i32.nxv8i32(<vscale x 8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.add.i32.nxv4i32(<vscale x 4 x i32> %b)
   %r = add i32 %r1, %r2
   ret i32 %r
 }
 
 define i16 @add_ext_i16(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b) {
 ; CHECK-LABEL: add_ext_i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    uunpkhi z2.h, z0.b
 ; CHECK-NEXT:    uunpklo z0.h, z0.b
 ; CHECK-NEXT:    uunpkhi z3.h, z1.b
 ; CHECK-NEXT:    uunpklo z1.h, z1.b
-; CHECK-NEXT:    ptrue p0.h
 ; CHECK-NEXT:    add z0.h, z0.h, z2.h
 ; CHECK-NEXT:    add z1.h, z1.h, z3.h
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    add z0.h, z0.h, z1.h
 ; CHECK-NEXT:    uaddv d0, p0, z0.h
-; CHECK-NEXT:    uaddv d1, p0, z1.h
-; CHECK-NEXT:    fmov x8, d0
-; CHECK-NEXT:    fmov x9, d1
-; CHECK-NEXT:    add w0, w8, w9
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
 ; CHECK-NEXT:    ret
   %ae = zext <vscale x 16 x i8> %a to <vscale x 16 x i16>
   %be = zext <vscale x 16 x i8> %b to <vscale x 16 x i16>
   %r1 = call i16 @llvm.vector.reduce.add.i16.nxv16i16(<vscale x 16 x i16> %ae)
   %r2 = call i16 @llvm.vector.reduce.add.i16.nxv16i16(<vscale x 16 x i16> %be)
   %r = add i16 %r1, %r2
   ret i16 %r
 }
 
 define i16 @add_ext_v32i16(<vscale x 32 x i8> %a, <vscale x 16 x i8> %b) {
 ; CHECK-LABEL: add_ext_v32i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    uunpklo z3.h, z1.b
 ; CHECK-NEXT:    uunpklo z4.h, z0.b
 ; CHECK-NEXT:    uunpkhi z1.h, z1.b
 ; CHECK-NEXT:    uunpkhi z0.h, z0.b
 ; CHECK-NEXT:    uunpkhi z5.h, z2.b
 ; CHECK-NEXT:    uunpklo z2.h, z2.b
 ; CHECK-NEXT:    add z0.h, z0.h, z1.h
 ; CHECK-NEXT:    add z1.h, z4.h, z3.h
-; CHECK-NEXT:    ptrue p0.h
 ; CHECK-NEXT:    add z0.h, z1.h, z0.h
 ; CHECK-NEXT:    add z1.h, z2.h, z5.h
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    add z0.h, z0.h, z1.h
 ; CHECK-NEXT:    uaddv d0, p0, z0.h
-; CHECK-NEXT:    uaddv d1, p0, z1.h
-; CHECK-NEXT:    fmov x8, d0
-; CHECK-NEXT:    fmov x9, d1
-; CHECK-NEXT:    add w0, w8, w9
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
 ; CHECK-NEXT:    ret
   %ae = zext <vscale x 32 x i8> %a to <vscale x 32 x i16>
   %be = zext <vscale x 16 x i8> %b to <vscale x 16 x i16>
   %r1 = call i16 @llvm.vector.reduce.add.i16.nxv32i16(<vscale x 32 x i16> %ae)
   %r2 = call i16 @llvm.vector.reduce.add.i16.nxv16i16(<vscale x 16 x i16> %be)
   %r = add i16 %r1, %r2
   ret i16 %r
 }
 
 ;define i32 @mul_i32(<vscale x 8 x i32> %a, <vscale x 4 x i32> %b) {
 ;  %r1 = call i32 @llvm.vector.reduce.mul.i32.nxv8i32(<vscale x 8 x i32> %a)
 ;  %r2 = call i32 @llvm.vector.reduce.mul.i32.nxv4i32(<vscale x 4 x i32> %b)
 ;  %r = mul i32 %r1, %r2
 ;  ret i32 %r
 ;}
 
 define i32 @and_i32(<vscale x 8 x i32> %a, <vscale x 4 x i32> %b) {
 ; CHECK-LABEL: and_i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.s
 ; CHECK-NEXT:    and z0.d, z0.d, z1.d
-; CHECK-NEXT:    andv s2, p0, z2.s
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    and z0.d, z0.d, z2.d
 ; CHECK-NEXT:    andv s0, p0, z0.s
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    fmov w9, s2
-; CHECK-NEXT:    and w0, w8, w9
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.and.i32.nxv8i32(<vscale x 8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.and.i32.nxv4i32(<vscale x 4 x i32> %b)
   %r = and i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @or_i32(<vscale x 8 x i32> %a, <vscale x 4 x i32> %b) {
 ; CHECK-LABEL: or_i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.s
 ; CHECK-NEXT:    orr z0.d, z0.d, z1.d
-; CHECK-NEXT:    orv s2, p0, z2.s
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    orr z0.d, z0.d, z2.d
 ; CHECK-NEXT:    orv s0, p0, z0.s
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    fmov w9, s2
-; CHECK-NEXT:    orr w0, w8, w9
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.or.i32.nxv8i32(<vscale x 8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.or.i32.nxv4i32(<vscale x 4 x i32> %b)
   %r = or i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @xor_i32(<vscale x 8 x i32> %a, <vscale x 4 x i32> %b) {
 ; CHECK-LABEL: xor_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    eor z0.d, z0.d, z1.d
-; CHECK-NEXT:    eorv s2, p0, z2.s
+; CHECK-NEXT:    eor3 z0.d, z0.d, z1.d, z2.d
 ; CHECK-NEXT:    eorv s0, p0, z0.s
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    fmov w9, s2
-; CHECK-NEXT:    eor w0, w8, w9
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.xor.i32.nxv8i32(<vscale x 8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.xor.i32.nxv4i32(<vscale x 4 x i32> %b)
   %r = xor i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @umin_i32(<vscale x 8 x i32> %a, <vscale x 4 x i32> %b) {
 ; CHECK-LABEL: umin_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s
 ; CHECK-NEXT:    umin z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    uminv s2, p0, z2.s
+; CHECK-NEXT:    umin z0.s, p0/m, z0.s, z2.s
 ; CHECK-NEXT:    uminv s0, p0, z0.s
-; CHECK-NEXT:    fmov w9, s2
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    cmp w8, w9
-; CHECK-NEXT:    csel w0, w8, w9, lo
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.umin.i32.nxv8i32(<vscale x 8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.umin.i32.nxv4i32(<vscale x 4 x i32> %b)
   %r = call i32 @llvm.umin.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 define i32 @umax_i32(<vscale x 8 x i32> %a, <vscale x 4 x i32> %b) {
 ; CHECK-LABEL: umax_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s
 ; CHECK-NEXT:    umax z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    umaxv s2, p0, z2.s
+; CHECK-NEXT:    umax z0.s, p0/m, z0.s, z2.s
 ; CHECK-NEXT:    umaxv s0, p0, z0.s
-; CHECK-NEXT:    fmov w9, s2
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    cmp w8, w9
-; CHECK-NEXT:    csel w0, w8, w9, hi
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.umax.i32.nxv8i32(<vscale x 8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.umax.i32.nxv4i32(<vscale x 4 x i32> %b)
   %r = call i32 @llvm.umax.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 define i32 @smin_i32(<vscale x 8 x i32> %a, <vscale x 4 x i32> %b) {
 ; CHECK-LABEL: smin_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s
 ; CHECK-NEXT:    smin z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    sminv s2, p0, z2.s
+; CHECK-NEXT:    smin z0.s, p0/m, z0.s, z2.s
 ; CHECK-NEXT:    sminv s0, p0, z0.s
-; CHECK-NEXT:    fmov w9, s2
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    cmp w8, w9
-; CHECK-NEXT:    csel w0, w8, w9, lt
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.smin.i32.nxv8i32(<vscale x 8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.smin.i32.nxv4i32(<vscale x 4 x i32> %b)
   %r = call i32 @llvm.smin.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 define i32 @smax_i32(<vscale x 8 x i32> %a, <vscale x 4 x i32> %b) {
 ; CHECK-LABEL: smax_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s
 ; CHECK-NEXT:    smax z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    smaxv s2, p0, z2.s
+; CHECK-NEXT:    smax z0.s, p0/m, z0.s, z2.s
 ; CHECK-NEXT:    smaxv s0, p0, z0.s
-; CHECK-NEXT:    fmov w9, s2
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    cmp w8, w9
-; CHECK-NEXT:    csel w0, w8, w9, gt
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.smax.i32.nxv8i32(<vscale x 8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.smax.i32.nxv4i32(<vscale x 4 x i32> %b)
   %r = call i32 @llvm.smax.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 declare float @llvm.vector.reduce.fadd.f32.nxv8f32(float, <vscale x 8 x float>)

llvm/test/CodeGen/AArch64/sve-fp-reduce.ll

 ; CHECK-NEXT:    ret
   %res = call double @llvm.vector.reduce.fmin.nxv2f64(<vscale x 2 x double> %a)
   ret double %res
 }
 
 define float @fadd_reduct_reassoc_v4v8f32(<vscale x 4 x float> %a, <vscale x 8 x float> %b) {
 ; CHECK-LABEL: fadd_reduct_reassoc_v4v8f32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.s
 ; CHECK-NEXT:    fadd z1.s, z1.s, z2.s
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    fadd z0.s, z0.s, z1.s
 ; CHECK-NEXT:    faddv s0, p0, z0.s
-; CHECK-NEXT:    faddv s1, p0, z1.s
-; CHECK-NEXT:    fadd s0, s0, s1
+; CHECK-NEXT:    // kill: def $s0 killed $s0 killed $z0
 ; CHECK-NEXT:    ret
   %r1 = call fast float @llvm.vector.reduce.fadd.nxv4f32(float -0.0, <vscale x 4 x float> %a)
   %r2 = call fast float @llvm.vector.reduce.fadd.nxv8f32(float -0.0, <vscale x 8 x float> %b)
   %r = fadd fast float %r1, %r2
   ret float %r
 }
 
 declare half @llvm.vector.reduce.fadd.nxv2f16(half, <vscale x 2 x half>)

llvm/test/CodeGen/AArch64/vecreduce-add.ll

entry:
   %z2 = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> %y)
   %z = add i32 %z1, %z2
   ret i32 %z
 }
 
 define i64 @add_pair_v4i32_v4i64_zext(<4 x i32> %x, <4 x i32> %y) {
 ; CHECK-LABEL: add_pair_v4i32_v4i64_zext:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    uaddlp v0.2d, v0.4s
-; CHECK-NEXT:    uadalp v0.2d, v1.4s
-; CHECK-NEXT:    addp d0, v0.2d
+; CHECK-NEXT:    uaddlp v1.2d, v1.4s
+; CHECK-NEXT:    uadalp v1.2d, v0.4s
+; CHECK-NEXT:    addp d0, v1.2d
 ; CHECK-NEXT:    fmov x0, d0
 ; CHECK-NEXT:    ret
 entry:
   %xx = zext <4 x i32> %x to <4 x i64>
   %z1 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> %xx)
   %yy = zext <4 x i32> %y to <4 x i64>
   %z2 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> %yy)
   %z = add i64 %z1, %z2
   ret i64 %z
 }
 
 define i64 @add_pair_v4i32_v4i64_sext(<4 x i32> %x, <4 x i32> %y) {
 ; CHECK-LABEL: add_pair_v4i32_v4i64_sext:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    saddlp v0.2d, v0.4s
-; CHECK-NEXT:    sadalp v0.2d, v1.4s
-; CHECK-NEXT:    addp d0, v0.2d
+; CHECK-NEXT:    saddlp v1.2d, v1.4s
+; CHECK-NEXT:    sadalp v1.2d, v0.4s
+; CHECK-NEXT:    addp d0, v1.2d
 ; CHECK-NEXT:    fmov x0, d0
 ; CHECK-NEXT:    ret
 entry:
   %xx = sext <4 x i32> %x to <4 x i64>
   %z1 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> %xx)
   %yy = sext <4 x i32> %y to <4 x i64>
   %z2 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> %yy)
   %z = add i64 %z1, %z2
entry:
   %z2 = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> %yy)
   %z = add i64 %z1, %z2
   ret i64 %z
 }
 
 define i32 @add_pair_v8i16_v8i32_zext(<8 x i16> %x, <8 x i16> %y) {
 ; CHECK-LABEL: add_pair_v8i16_v8i32_zext:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    uaddlp v0.4s, v0.8h
-; CHECK-NEXT:    uadalp v0.4s, v1.8h
-; CHECK-NEXT:    addv s0, v0.4s
+; CHECK-NEXT:    uaddlp v1.4s, v1.8h
+; CHECK-NEXT:    uadalp v1.4s, v0.8h
+; CHECK-NEXT:    addv s0, v1.4s
 ; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
 entry:
   %xx = zext <8 x i16> %x to <8 x i32>
   %z1 = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> %xx)
   %yy = zext <8 x i16> %y to <8 x i32>
   %z2 = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> %yy)
   %z = add i32 %z1, %z2
   ret i32 %z
 }
 
 define i32 @add_pair_v8i16_v8i32_sext(<8 x i16> %x, <8 x i16> %y) {
 ; CHECK-LABEL: add_pair_v8i16_v8i32_sext:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    saddlp v0.4s, v0.8h
-; CHECK-NEXT:    sadalp v0.4s, v1.8h
-; CHECK-NEXT:    addv s0, v0.4s
+; CHECK-NEXT:    saddlp v1.4s, v1.8h
+; CHECK-NEXT:    sadalp v1.4s, v0.8h
+; CHECK-NEXT:    addv s0, v1.4s
 ; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
 entry:
   %xx = sext <8 x i16> %x to <8 x i32>
   %z1 = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> %xx)
   %yy = sext <8 x i16> %y to <8 x i32>
   %z2 = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> %yy)
   %z = add i32 %z1, %z2
entry:
   %z2 = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> %yy)
   %z = add i32 %z1, %z2
   ret i32 %z
 }
 
 define zeroext i16 @add_pair_v8i16_v8i16(<8 x i16> %x, <8 x i16> %y) {
 ; CHECK-LABEL: add_pair_v8i16_v8i16:
 ; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    add v0.8h, v0.8h, v1.8h
 ; CHECK-NEXT:    addv h0, v0.8h
-; CHECK-NEXT:    addv h1, v1.8h
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    fmov w9, s1
-; CHECK-NEXT:    add w8, w8, w9
-; CHECK-NEXT:    and w0, w8, #0xffff
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
 entry:
   %z1 = call i16 @llvm.vector.reduce.add.v8i16(<8 x i16> %x)
   %z2 = call i16 @llvm.vector.reduce.add.v8i16(<8 x i16> %y)
   %z = add i16 %z1, %z2
   ret i16 %z
 }
 
entry:
   %z2 = call i64 @llvm.vector.reduce.add.v8i64(<8 x i64> %yy)
   %z = add i64 %z1, %z2
   ret i64 %z
 }
 
 define i64 @add_pair_v4i16_v4i64_zext(<4 x i16> %x, <4 x i16> %y) {
 ; CHECK-LABEL: add_pair_v4i16_v4i64_zext:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ushll v0.4s, v0.4h, #0
 ; CHECK-NEXT:    ushll v1.4s, v1.4h, #0
-; CHECK-NEXT:    uaddlp v0.2d, v0.4s
-; CHECK-NEXT:    uadalp v0.2d, v1.4s
-; CHECK-NEXT:    addp d0, v0.2d
+; CHECK-NEXT:    ushll v0.4s, v0.4h, #0
+; CHECK-NEXT:    uaddlp v1.2d, v1.4s
+; CHECK-NEXT:    uadalp v1.2d, v0.4s
+; CHECK-NEXT:    addp d0, v1.2d
 ; CHECK-NEXT:    fmov x0, d0
 ; CHECK-NEXT:    ret
 entry:
   %xx = zext <4 x i16> %x to <4 x i64>
   %z1 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> %xx)
   %yy = zext <4 x i16> %y to <4 x i64>
   %z2 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> %yy)
   %z = add i64 %z1, %z2
   ret i64 %z
 }
 
 define i64 @add_pair_v4i16_v4i64_sext(<4 x i16> %x, <4 x i16> %y) {
 ; CHECK-LABEL: add_pair_v4i16_v4i64_sext:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    sshll v0.4s, v0.4h, #0
 ; CHECK-NEXT:    sshll v1.4s, v1.4h, #0
-; CHECK-NEXT:    saddlp v0.2d, v0.4s
-; CHECK-NEXT:    sadalp v0.2d, v1.4s
-; CHECK-NEXT:    addp d0, v0.2d
+; CHECK-NEXT:    sshll v0.4s, v0.4h, #0
+; CHECK-NEXT:    saddlp v1.2d, v1.4s
+; CHECK-NEXT:    sadalp v1.2d, v0.4s
+; CHECK-NEXT:    addp d0, v1.2d
 ; CHECK-NEXT:    fmov x0, d0
 ; CHECK-NEXT:    ret
 entry:
   %xx = sext <4 x i16> %x to <4 x i64>
   %z1 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> %xx)
   %yy = sext <4 x i16> %y to <4 x i64>
   %z2 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> %yy)
   %z = add i64 %z1, %z2
entry:
   %z2 = call i32 @llvm.vector.reduce.add.v16i32(<16 x i32> %yy)
   %z = add i32 %z1, %z2
   ret i32 %z
 }
 
 define i32 @add_pair_v8i8_v8i32_zext(<8 x i8> %x, <8 x i8> %y) {
 ; CHECK-BASE-LABEL: add_pair_v8i8_v8i32_zext:
 ; CHECK-BASE:       // %bb.0: // %entry
-; CHECK-BASE-NEXT:    ushll v0.8h, v0.8b, #0
 ; CHECK-BASE-NEXT:    ushll v1.8h, v1.8b, #0
-; CHECK-BASE-NEXT:    uaddlp v0.4s, v0.8h
-; CHECK-BASE-NEXT:    uadalp v0.4s, v1.8h
-; CHECK-BASE-NEXT:    addv s0, v0.4s
+; CHECK-BASE-NEXT:    ushll v0.8h, v0.8b, #0
+; CHECK-BASE-NEXT:    uaddlp v1.4s, v1.8h
+; CHECK-BASE-NEXT:    uadalp v1.4s, v0.8h
+; CHECK-BASE-NEXT:    addv s0, v1.4s
 ; CHECK-BASE-NEXT:    fmov w0, s0
 ; CHECK-BASE-NEXT:    ret
 ;
 ; CHECK-DOT-LABEL: add_pair_v8i8_v8i32_zext:
 ; CHECK-DOT:       // %bb.0: // %entry
 ; CHECK-DOT-NEXT:    movi v2.8b, #1
 ; CHECK-DOT-NEXT:    movi v3.2d, #0000000000000000
 ; CHECK-DOT-NEXT:    udot v3.2s, v1.8b, v2.8b
 ; CHECK-DOT-NEXT:    udot v3.2s, v0.8b, v2.8b
 ; CHECK-DOT-NEXT:    addp v0.2s, v3.2s, v3.2s
 ; CHECK-DOT-NEXT:    fmov w0, s0
 ; CHECK-DOT-NEXT:    ret
 entry:
   %xx = zext <8 x i8> %x to <8 x i32>
   %z1 = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> %xx)
   %yy = zext <8 x i8> %y to <8 x i32>
   %z2 = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> %yy)
   %z = add i32 %z1, %z2
   ret i32 %z
 }
 
 define i32 @add_pair_v8i8_v8i32_sext(<8 x i8> %x, <8 x i8> %y) {
 ; CHECK-BASE-LABEL: add_pair_v8i8_v8i32_sext:
 ; CHECK-BASE:       // %bb.0: // %entry
-; CHECK-BASE-NEXT:    sshll v0.8h, v0.8b, #0
 ; CHECK-BASE-NEXT:    sshll v1.8h, v1.8b, #0
-; CHECK-BASE-NEXT:    saddlp v0.4s, v0.8h
-; CHECK-BASE-NEXT:    sadalp v0.4s, v1.8h
-; CHECK-BASE-NEXT:    addv s0, v0.4s
+; CHECK-BASE-NEXT:    sshll v0.8h, v0.8b, #0
+; CHECK-BASE-NEXT:    saddlp v1.4s, v1.8h
+; CHECK-BASE-NEXT:    sadalp v1.4s, v0.8h
+; CHECK-BASE-NEXT:    addv s0, v1.4s
 ; CHECK-BASE-NEXT:    fmov w0, s0
 ; CHECK-BASE-NEXT:    ret
 ;
 ; CHECK-DOT-LABEL: add_pair_v8i8_v8i32_sext:
 ; CHECK-DOT:       // %bb.0: // %entry
 ; CHECK-DOT-NEXT:    movi v2.8b, #1
 ; CHECK-DOT-NEXT:    movi v3.2d, #0000000000000000
 ; CHECK-DOT-NEXT:    sdot v3.2s, v1.8b, v2.8b
entry:
   %z2 = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> %yy)
   %z = add i32 %z1, %z2
   ret i32 %z
 }
 
 define zeroext i16 @add_pair_v16i8_v16i16_zext(<16 x i8> %x, <16 x i8> %y) {
 ; CHECK-LABEL: add_pair_v16i8_v16i16_zext:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    uaddlv h0, v0.16b
-; CHECK-NEXT:    uaddlv h1, v1.16b
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    fmov w9, s1
-; CHECK-NEXT:    add w8, w8, w9
-; CHECK-NEXT:    and w0, w8, #0xffff
+; CHECK-NEXT:    uaddlp v1.8h, v1.16b
+; CHECK-NEXT:    uadalp v1.8h, v0.16b
+; CHECK-NEXT:    addv h0, v1.8h
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
 entry:
   %xx = zext <16 x i8> %x to <16 x i16>
   %z1 = call i16 @llvm.vector.reduce.add.v16i16(<16 x i16> %xx)
   %yy = zext <16 x i8> %y to <16 x i16>
   %z2 = call i16 @llvm.vector.reduce.add.v16i16(<16 x i16> %yy)
   %z = add i16 %z1, %z2
   ret i16 %z
 }
 
 define signext i16 @add_pair_v16i8_v16i16_sext(<16 x i8> %x, <16 x i8> %y) {
 ; CHECK-LABEL: add_pair_v16i8_v16i16_sext:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    saddlv h0, v0.16b
-; CHECK-NEXT:    saddlv h1, v1.16b
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    fmov w9, s1
-; CHECK-NEXT:    add w8, w8, w9
-; CHECK-NEXT:    sxth w0, w8
+; CHECK-NEXT:    saddlp v1.8h, v1.16b
+; CHECK-NEXT:    sadalp v1.8h, v0.16b
+; CHECK-NEXT:    addv h0, v1.8h
+; CHECK-NEXT:    smov w0, v0.h[0]
 ; CHECK-NEXT:    ret
 entry:
   %xx = sext <16 x i8> %x to <16 x i16>
   %z1 = call i16 @llvm.vector.reduce.add.v16i16(<16 x i16> %xx)
   %yy = sext <16 x i8> %y to <16 x i16>
   %z2 = call i16 @llvm.vector.reduce.add.v16i16(<16 x i16> %yy)
   %z = add i16 %z1, %z2
   ret i16 %z
 }
 
 define zeroext i16 @add_pair_v8i8_v8i16_zext(<8 x i8> %x, <8 x i8> %y) {
 ; CHECK-LABEL: add_pair_v8i8_v8i16_zext:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-NEXT:    ushll v1.8h, v1.8b, #0
+; CHECK-NEXT:    uaddl v0.8h, v0.8b, v1.8b
 ; CHECK-NEXT:    addv h0, v0.8h
-; CHECK-NEXT:    addv h1, v1.8h
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    fmov w9, s1
-; CHECK-NEXT:    add w8, w8, w9
-; CHECK-NEXT:    and w0, w8, #0xffff
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
 entry:
   %xx = zext <8 x i8> %x to <8 x i16>
   %z1 = call i16 @llvm.vector.reduce.add.v8i16(<8 x i16> %xx)
   %yy = zext <8 x i8> %y to <8 x i16>
   %z2 = call i16 @llvm.vector.reduce.add.v8i16(<8 x i16> %yy)
   %z = add i16 %z1, %z2
   ret i16 %z
 }
 
 define signext i16 @add_pair_v8i8_v8i16_sext(<8 x i8> %x, <8 x i8> %y) {
 ; CHECK-LABEL: add_pair_v8i8_v8i16_sext:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    sshll v0.8h, v0.8b, #0
-; CHECK-NEXT:    sshll v1.8h, v1.8b, #0
+; CHECK-NEXT:    saddl v0.8h, v0.8b, v1.8b
 ; CHECK-NEXT:    addv h0, v0.8h
-; CHECK-NEXT:    addv h1, v1.8h
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    fmov w9, s1
-; CHECK-NEXT:    add w8, w8, w9
-; CHECK-NEXT:    sxth w0, w8
+; CHECK-NEXT:    smov w0, v0.h[0]
 ; CHECK-NEXT:    ret
 entry:
   %xx = sext <8 x i8> %x to <8 x i16>
   %z1 = call i16 @llvm.vector.reduce.add.v8i16(<8 x i16> %xx)
   %yy = sext <8 x i8> %y to <8 x i16>
   %z2 = call i16 @llvm.vector.reduce.add.v8i16(<8 x i16> %yy)
   %z = add i16 %z1, %z2
   ret i16 %z
 }
 
 define zeroext i8 @add_pair_v16i8_v16i8(<16 x i8> %x, <16 x i8> %y) {
 ; CHECK-LABEL: add_pair_v16i8_v16i8:
 ; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    add v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    addv b0, v0.16b
-; CHECK-NEXT:    addv b1, v1.16b
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    fmov w9, s1
-; CHECK-NEXT:    add w8, w8, w9
-; CHECK-NEXT:    and w0, w8, #0xff
+; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
 entry:
   %z1 = call i8 @llvm.vector.reduce.add.v16i8(<16 x i8> %x)
   %z2 = call i8 @llvm.vector.reduce.add.v16i8(<16 x i8> %y)
   %z = add i8 %z1, %z2
   ret i8 %z
 }
 
entry:
   %z2 = call i64 @llvm.vector.reduce.add.v8i64(<8 x i64> %yy)
   %z = add i64 %z1, %z2
   ret i64 %z
 }
 
 define i64 @add_pair_v4i8_v4i64_zext(<4 x i8> %x, <4 x i8> %y) {
 ; CHECK-LABEL: add_pair_v4i8_v4i64_zext:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    bic v0.4h, #255, lsl #8
 ; CHECK-NEXT:    bic v1.4h, #255, lsl #8
-; CHECK-NEXT:    ushll v0.4s, v0.4h, #0
+; CHECK-NEXT:    bic v0.4h, #255, lsl #8
 ; CHECK-NEXT:    ushll v1.4s, v1.4h, #0
-; CHECK-NEXT:    uaddlp v0.2d, v0.4s
-; CHECK-NEXT:    uadalp v0.2d, v1.4s
-; CHECK-NEXT:    addp d0, v0.2d
+; CHECK-NEXT:    ushll v0.4s, v0.4h, #0
+; CHECK-NEXT:    uaddlp v1.2d, v1.4s
+; CHECK-NEXT:    uadalp v1.2d, v0.4s
+; CHECK-NEXT:    addp d0, v1.2d
 ; CHECK-NEXT:    fmov x0, d0
 ; CHECK-NEXT:    ret
 entry:
   %xx = zext <4 x i8> %x to <4 x i64>
   %z1 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> %xx)
   %yy = zext <4 x i8> %y to <4 x i64>
   %z2 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> %yy)
   %z = add i64 %z1, %z2
entry:
   %z2 = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> %yy)
   %z = add i64 %z1, %z2
   ret i64 %z
 }
 
 define i32 @add_pair_v8i8_v8i32_double_sext_zext(<8 x i8> %ax, <8 x i8> %ay, <8 x i8> %bx, <8 x i8> %by) {
 ; CHECK-BASE-LABEL: add_pair_v8i8_v8i32_double_sext_zext:
 ; CHECK-BASE:       // %bb.0: // %entry
-; CHECK-BASE-NEXT:    sshll v2.8h, v2.8b, #0
-; CHECK-BASE-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-BASE-NEXT:    saddlp v2.4s, v2.8h
-; CHECK-BASE-NEXT:    uaddlp v0.4s, v0.8h
-; CHECK-BASE-NEXT:    ushll v1.8h, v1.8b, #0
 ; CHECK-BASE-NEXT:    sshll v3.8h, v3.8b, #0
-; CHECK-BASE-NEXT:    uadalp v0.4s, v1.8h
-; CHECK-BASE-NEXT:    sadalp v2.4s, v3.8h
-; CHECK-BASE-NEXT:    add v0.4s, v0.4s, v2.4s
+; CHECK-BASE-NEXT:    ushll v1.8h, v1.8b, #0
+; CHECK-BASE-NEXT:    saddlp v3.4s, v3.8h
+; CHECK-BASE-NEXT:    uaddlp v1.4s, v1.8h
+; CHECK-BASE-NEXT:    ushll v0.8h, v0.8b, #0
+; CHECK-BASE-NEXT:    sshll v2.8h, v2.8b, #0
+; CHECK-BASE-NEXT:    uadalp v1.4s, v0.8h
+; CHECK-BASE-NEXT:    sadalp v3.4s, v2.8h
+; CHECK-BASE-NEXT:    add v0.4s, v3.4s, v1.4s
 ; CHECK-BASE-NEXT:    addv s0, v0.4s
 ; CHECK-BASE-NEXT:    fmov w0, s0
 ; CHECK-BASE-NEXT:    ret
 ;
 ; CHECK-DOT-LABEL: add_pair_v8i8_v8i32_double_sext_zext:
 ; CHECK-DOT:       // %bb.0: // %entry
 ; CHECK-DOT-NEXT:    movi v4.2d, #0000000000000000
 ; CHECK-DOT-NEXT:    movi v5.8b, #1

 }
 
 define i32 @full(ptr %p1, i32 noundef %s1, ptr %p2, i32 noundef %s2) {
 ; CHECK-BASE-LABEL: full:
 ; CHECK-BASE:       // %bb.0: // %entry
 ; CHECK-BASE-NEXT:    // kill: def $w3 killed $w3 def $x3
 ; CHECK-BASE-NEXT:    // kill: def $w1 killed $w1 def $x1
 ; CHECK-BASE-NEXT:    sxtw x8, w1
-; CHECK-BASE-NEXT:    sxtw x9, w3
-; CHECK-BASE-NEXT:    add x10, x0, x8
-; CHECK-BASE-NEXT:    add x11, x2, x9
-; CHECK-BASE-NEXT:    ldr d2, [x0]
-; CHECK-BASE-NEXT:    ldr d3, [x2]
-; CHECK-BASE-NEXT:    ldr d0, [x10]
-; CHECK-BASE-NEXT:    add x10, x10, x8
-; CHECK-BASE-NEXT:    ldr d1, [x11]
-; CHECK-BASE-NEXT:    add x11, x11, x9
+; CHECK-BASE-NEXT:    sxtw x10, w3
+; CHECK-BASE-NEXT:    add x9, x0, x8
+; CHECK-BASE-NEXT:    ldr d0, [x0]
+; CHECK-BASE-NEXT:    ldr d1, [x2]
+; CHECK-BASE-NEXT:    add x11, x2, x10
+; CHECK-BASE-NEXT:    ldr d2, [x9]
+; CHECK-BASE-NEXT:    add x9, x9, x8
 ; CHECK-BASE-NEXT:    uabdl v0.8h, v0.8b, v1.8b
-; CHECK-BASE-NEXT:    uabdl v1.8h, v2.8b, v3.8b
-; CHECK-BASE-NEXT:    ldr d2, [x10]
-; CHECK-BASE-NEXT:    ldr d3, [x11]
-; CHECK-BASE-NEXT:    add x10, x10, x8
+; CHECK-BASE-NEXT:    ldr d1, [x11]
+; CHECK-BASE-NEXT:    add x11, x11, x10
 ; CHECK-BASE-NEXT:    uaddlp v0.4s, v0.8h
-; CHECK-BASE-NEXT:    add x11, x11, x9
+; CHECK-BASE-NEXT:    uabdl v1.8h, v2.8b, v1.8b
+; CHECK-BASE-NEXT:    ldr d2, [x9]
+; CHECK-BASE-NEXT:    ldr d3, [x11]
+; CHECK-BASE-NEXT:    add x9, x9, x8
+; CHECK-BASE-NEXT:    add x11, x11, x10
 ; CHECK-BASE-NEXT:    uadalp v0.4s, v1.8h
 ; CHECK-BASE-NEXT:    uabdl v1.8h, v2.8b, v3.8b
-; CHECK-BASE-NEXT:    ldr d2, [x10]
+; CHECK-BASE-NEXT:    ldr d2, [x9]
 ; CHECK-BASE-NEXT:    ldr d3, [x11]
-; CHECK-BASE-NEXT:    add x10, x10, x8
-; CHECK-BASE-NEXT:    add x11, x11, x9
+; CHECK-BASE-NEXT:    add x9, x9, x8
+; CHECK-BASE-NEXT:    add x11, x11, x10
 ; CHECK-BASE-NEXT:    uadalp v0.4s, v1.8h
 ; CHECK-BASE-NEXT:    uabdl v1.8h, v2.8b, v3.8b
-; CHECK-BASE-NEXT:    ldr d2, [x10]
+; CHECK-BASE-NEXT:    ldr d2, [x9]
 ; CHECK-BASE-NEXT:    ldr d3, [x11]
-; CHECK-BASE-NEXT:    add x10, x10, x8
-; CHECK-BASE-NEXT:    add x11, x11, x9
+; CHECK-BASE-NEXT:    add x9, x9, x8
+; CHECK-BASE-NEXT:    add x11, x11, x10
 ; CHECK-BASE-NEXT:    uadalp v0.4s, v1.8h
 ; CHECK-BASE-NEXT:    uabdl v1.8h, v2.8b, v3.8b
-; CHECK-BASE-NEXT:    ldr d2, [x10]
+; CHECK-BASE-NEXT:    ldr d2, [x9]
 ; CHECK-BASE-NEXT:    ldr d3, [x11]
-; CHECK-BASE-NEXT:    add x10, x10, x8
-; CHECK-BASE-NEXT:    add x11, x11, x9
+; CHECK-BASE-NEXT:    add x9, x9, x8
+; CHECK-BASE-NEXT:    add x11, x11, x10
 ; CHECK-BASE-NEXT:    uadalp v0.4s, v1.8h
 ; CHECK-BASE-NEXT:    uabdl v1.8h, v2.8b, v3.8b
-; CHECK-BASE-NEXT:    ldr d2, [x10]
+; CHECK-BASE-NEXT:    ldr d2, [x9]
 ; CHECK-BASE-NEXT:    ldr d3, [x11]
 ; CHECK-BASE-NEXT:    uadalp v0.4s, v1.8h
-; CHECK-BASE-NEXT:    ldr d1, [x10, x8]
+; CHECK-BASE-NEXT:    ldr d1, [x9, x8]
 ; CHECK-BASE-NEXT:    uabdl v2.8h, v2.8b, v3.8b
-; CHECK-BASE-NEXT:    ldr d3, [x11, x9]
+; CHECK-BASE-NEXT:    ldr d3, [x11, x10]
 ; CHECK-BASE-NEXT:    uadalp v0.4s, v2.8h
 ; CHECK-BASE-NEXT:    uabdl v1.8h, v1.8b, v3.8b
 ; CHECK-BASE-NEXT:    uadalp v0.4s, v1.8h
 ; CHECK-BASE-NEXT:    addv s0, v0.4s
 ; CHECK-BASE-NEXT:    fmov w0, s0
 ; CHECK-BASE-NEXT:    ret
 ;
 ; CHECK-DOT-LABEL: full:

llvm/test/CodeGen/AArch64/vecreduce-fadd.ll

 exit:
   ret half %red.next
 }
 
 
 define half @fadd_reduct_reassoc_v8f16(<8 x half> %a, <8 x half> %b) {
 ; FULLFP16-LABEL: fadd_reduct_reassoc_v8f16:
 ; FULLFP16:       // %bb.0:
-; FULLFP16-NEXT:    faddp v2.8h, v0.8h, v0.8h
-; FULLFP16-NEXT:    faddp v3.8h, v1.8h, v1.8h
-; FULLFP16-NEXT:    faddp v0.8h, v2.8h, v0.8h
-; FULLFP16-NEXT:    faddp v1.8h, v3.8h, v1.8h
+; FULLFP16-NEXT:    fadd v0.8h, v0.8h, v1.8h
+; FULLFP16-NEXT:    faddp v1.8h, v0.8h, v0.8h
+; FULLFP16-NEXT:    faddp v0.8h, v1.8h, v0.8h
 ; FULLFP16-NEXT:    faddp h0, v0.2h
-; FULLFP16-NEXT:    faddp h1, v1.2h
-; FULLFP16-NEXT:    fadd h0, h0, h1
 ; FULLFP16-NEXT:    ret
 ;
 ; CHECKNOFP16-LABEL: fadd_reduct_reassoc_v8f16:
 ; CHECKNOFP16:       // %bb.0:
 ; CHECKNOFP16-NEXT:    mov h2, v0.h[1]
 ; CHECKNOFP16-NEXT:    mov h3, v1.h[1]
 ; CHECKNOFP16-NEXT:    fcvt s4, h0
 ; CHECKNOFP16-NEXT:    fcvt s5, h1
; CHECKNOFP16-NEXT:    ret
   ret half %r
 }
 
 define float @fadd_reduct_reassoc_v8f32(<8 x float> %a, <8 x float> %b) {
 ; CHECK-LABEL: fadd_reduct_reassoc_v8f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fadd v2.4s, v2.4s, v3.4s
 ; CHECK-NEXT:    fadd v0.4s, v0.4s, v1.4s
-; CHECK-NEXT:    faddp v1.4s, v2.4s, v2.4s
+; CHECK-NEXT:    fadd v0.4s, v0.4s, v2.4s
 ; CHECK-NEXT:    faddp v0.4s, v0.4s, v0.4s
-; CHECK-NEXT:    faddp s1, v1.2s
 ; CHECK-NEXT:    faddp s0, v0.2s
-; CHECK-NEXT:    fadd s0, s0, s1
 ; CHECK-NEXT:    ret
   %r1 = call fast float @llvm.vector.reduce.fadd.f32.v8f32(float -0.0, <8 x float> %a)
   %r2 = call fast float @llvm.vector.reduce.fadd.f32.v8f32(float -0.0, <8 x float> %b)
   %r = fadd fast float %r1, %r2
   ret float %r
 }
 
 define float @fadd_reduct_reassoc_v4f32(<4 x float> %a, <4 x float> %b) {
 ; CHECK-LABEL: fadd_reduct_reassoc_v4f32:
 ; CHECK:       // %bb.0:
+; CHECK-NEXT:    fadd v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    faddp v0.4s, v0.4s, v0.4s
-; CHECK-NEXT:    faddp v1.4s, v1.4s, v1.4s
 ; CHECK-NEXT:    faddp s0, v0.2s
-; CHECK-NEXT:    faddp s1, v1.2s
-; CHECK-NEXT:    fadd s0, s0, s1
 ; CHECK-NEXT:    ret
   %r1 = call fast float @llvm.vector.reduce.fadd.f32.v4f32(float -0.0, <4 x float> %a)
   %r2 = call fast float @llvm.vector.reduce.fadd.f32.v4f32(float -0.0, <4 x float> %b)
   %r = fadd fast float %r1, %r2
   ret float %r
 }
 
 define float @fadd_reduct_reassoc_v4f32_init(float %i, <4 x float> %a, <4 x float> %b) {
; CHECK-NEXT:    ret
   %r = fadd fast float %r1, %r2
   ret float %r
 }
 
 define float @fadd_reduct_reassoc_v4v8f32(<4 x float> %a, <8 x float> %b) {
 ; CHECK-LABEL: fadd_reduct_reassoc_v4v8f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fadd v1.4s, v1.4s, v2.4s
+; CHECK-NEXT:    fadd v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    faddp v0.4s, v0.4s, v0.4s
-; CHECK-NEXT:    faddp v1.4s, v1.4s, v1.4s
 ; CHECK-NEXT:    faddp s0, v0.2s
-; CHECK-NEXT:    faddp s1, v1.2s
-; CHECK-NEXT:    fadd s0, s0, s1
 ; CHECK-NEXT:    ret
   %r1 = call fast float @llvm.vector.reduce.fadd.f32.v4f32(float -0.0, <4 x float> %a)
   %r2 = call fast float @llvm.vector.reduce.fadd.f32.v8f32(float -0.0, <8 x float> %b)
   %r = fadd fast float %r1, %r2
   ret float %r
 }
 
 define double @fadd_reduct_reassoc_v4f64(<4 x double> %a, <4 x double> %b) {
 ; CHECK-LABEL: fadd_reduct_reassoc_v4f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fadd v2.2d, v2.2d, v3.2d
 ; CHECK-NEXT:    fadd v0.2d, v0.2d, v1.2d
-; CHECK-NEXT:    faddp d1, v2.2d
+; CHECK-NEXT:    fadd v0.2d, v0.2d, v2.2d
 ; CHECK-NEXT:    faddp d0, v0.2d
-; CHECK-NEXT:    fadd d0, d0, d1
 ; CHECK-NEXT:    ret
   %r1 = call fast double @llvm.vector.reduce.fadd.f64.v4f64(double -0.0, <4 x double> %a)
   %r2 = call fast double @llvm.vector.reduce.fadd.f64.v4f64(double -0.0, <4 x double> %b)
   %r = fadd fast double %r1, %r2
   ret double %r
 }
 
 define float @fadd_reduct_reassoc_v4f32_extrause(<4 x float> %a, <4 x float> %b) {

llvm/test/CodeGen/RISCV/double_reduct.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=riscv32 -mattr=+d,+zfh,+experimental-zvfh,+v,+m -target-abi=ilp32d \
 ; RUN:     -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV32
 ; RUN: llc -mtriple=riscv64 -mattr=+d,+zfh,+experimental-zvfh,+v,+m -target-abi=lp64d \
 ; RUN:     -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV64
 
 define float @add_f32(<4 x float> %a, <4 x float> %b) {
 ; CHECK-LABEL: add_f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
-; CHECK-NEXT:    vmv.s.x v10, zero
-; CHECK-NEXT:    vfredusum.vs v8, v8, v10
-; CHECK-NEXT:    vfmv.f.s ft0, v8
-; CHECK-NEXT:    vfredusum.vs v8, v9, v10
-; CHECK-NEXT:    vfmv.f.s ft1, v8
-; CHECK-NEXT:    fadd.s fa0, ft0, ft1
+; CHECK-NEXT:    vfadd.vv v8, v8, v9
+; CHECK-NEXT:    vmv.s.x v9, zero
+; CHECK-NEXT:    vfredusum.vs v8, v8, v9
+; CHECK-NEXT:    vfmv.f.s fa0, v8
 ; CHECK-NEXT:    ret
   %r1 = call fast float @llvm.vector.reduce.fadd.f32.v4f32(float -0.0, <4 x float> %a)
   %r2 = call fast float @llvm.vector.reduce.fadd.f32.v4f32(float -0.0, <4 x float> %b)
   %r = fadd fast float %r1, %r2
   ret float %r
 }
 
 define float @fmul_f32(<4 x float> %a, <4 x float> %b) {
; CHECK-NEXT:    ret
   %r1 = call fast float @llvm.vector.reduce.fmax.v4f32(<4 x float> %a)
   %r2 = call fast float @llvm.vector.reduce.fmax.v4f32(<4 x float> %b)
   %r = call float @llvm.maxnum.f32(float %r1, float %r2)
   ret float %r
 }
 
 
 define i32 @add_i32(<4 x i32> %a, <4 x i32> %b) {
-; RV32-LABEL: add_i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
-; RV32-NEXT:    vmv.s.x v10, zero
-; RV32-NEXT:    vredsum.vs v8, v8, v10
-; RV32-NEXT:    vmv.x.s a0, v8
-; RV32-NEXT:    vredsum.vs v8, v9, v10
-; RV32-NEXT:    vmv.x.s a1, v8
-; RV32-NEXT:    add a0, a0, a1
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: add_i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
-; RV64-NEXT:    vmv.s.x v10, zero
-; RV64-NEXT:    vredsum.vs v8, v8, v10
-; RV64-NEXT:    vmv.x.s a0, v8
-; RV64-NEXT:    vredsum.vs v8, v9, v10
-; RV64-NEXT:    vmv.x.s a1, v8
-; RV64-NEXT:    addw a0, a0, a1
-; RV64-NEXT:    ret
+; CHECK-LABEL: add_i32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
+; CHECK-NEXT:    vmv.s.x v10, zero
+; CHECK-NEXT:    vadd.vv v8, v8, v9
+; CHECK-NEXT:    vredsum.vs v8, v8, v10
+; CHECK-NEXT:    vmv.x.s a0, v8
+; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.add.i32.v4i32(<4 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.add.i32.v4i32(<4 x i32> %b)
   %r = add i32 %r1, %r2
   ret i32 %r
 }
 
 define i16 @add_ext_i16(<16 x i8> %a, <16 x i8> %b) {
 ; CHECK-LABEL: add_ext_i16:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 16, e16, m1, ta, ma
 ; CHECK-NEXT:    vmv.s.x v10, zero
 ; CHECK-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
-; CHECK-NEXT:    vwredsumu.vs v8, v8, v10
-; CHECK-NEXT:    vsetivli zero, 0, e16, m1, ta, ma
+; CHECK-NEXT:    vwaddu.vv v12, v8, v9
+; CHECK-NEXT:    vsetvli zero, zero, e16, m2, ta, ma
+; CHECK-NEXT:    vredsum.vs v8, v12, v10
 ; CHECK-NEXT:    vmv.x.s a0, v8
-; CHECK-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
-; CHECK-NEXT:    vwredsumu.vs v8, v9, v10
-; CHECK-NEXT:    vsetivli zero, 0, e16, m1, ta, ma
-; CHECK-NEXT:    vmv.x.s a1, v8
-; CHECK-NEXT:    add a0, a0, a1
 ; CHECK-NEXT:    ret
   %ae = zext <16 x i8> %a to <16 x i16>
   %be = zext <16 x i8> %b to <16 x i16>
   %r1 = call i16 @llvm.vector.reduce.add.i16.v16i16(<16 x i16> %ae)
   %r2 = call i16 @llvm.vector.reduce.add.i16.v16i16(<16 x i16> %be)
   %r = add i16 %r1, %r2
   ret i16 %r
 }
; RV64-NEXT:    ret
   %r = mul i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @and_i32(<4 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: and_i32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
-; CHECK-NEXT:    vmv.v.i v10, -1
-; CHECK-NEXT:    vredand.vs v8, v8, v10
+; CHECK-NEXT:    vand.vv v8, v8, v9
+; CHECK-NEXT:    vmv.v.i v9, -1
+; CHECK-NEXT:    vredand.vs v8, v8, v9
 ; CHECK-NEXT:    vmv.x.s a0, v8
-; CHECK-NEXT:    vredand.vs v8, v9, v10
-; CHECK-NEXT:    vmv.x.s a1, v8
-; CHECK-NEXT:    and a0, a0, a1
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.and.i32.v4i32(<4 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.and.i32.v4i32(<4 x i32> %b)
   %r = and i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @or_i32(<4 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: or_i32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
 ; CHECK-NEXT:    vmv.s.x v10, zero
+; CHECK-NEXT:    vor.vv v8, v8, v9
 ; CHECK-NEXT:    vredor.vs v8, v8, v10
 ; CHECK-NEXT:    vmv.x.s a0, v8
-; CHECK-NEXT:    vredor.vs v8, v9, v10
-; CHECK-NEXT:    vmv.x.s a1, v8
-; CHECK-NEXT:    or a0, a0, a1
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.or.i32.v4i32(<4 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.or.i32.v4i32(<4 x i32> %b)
   %r = or i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @xor_i32(<4 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: xor_i32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
 ; CHECK-NEXT:    vmv.s.x v10, zero
+; CHECK-NEXT:    vxor.vv v8, v8, v9
 ; CHECK-NEXT:    vredxor.vs v8, v8, v10
 ; CHECK-NEXT:    vmv.x.s a0, v8
-; CHECK-NEXT:    vredxor.vs v8, v9, v10
-; CHECK-NEXT:    vmv.x.s a1, v8
-; CHECK-NEXT:    xor a0, a0, a1
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.xor.i32.v4i32(<4 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.xor.i32.v4i32(<4 x i32> %b)
   %r = xor i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @umin_i32(<4 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: umin_i32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
-; CHECK-NEXT:    vmv.v.i v10, -1
-; CHECK-NEXT:    vredminu.vs v8, v8, v10
+; CHECK-NEXT:    vminu.vv v8, v8, v9
+; CHECK-NEXT:    vmv.v.i v9, -1
+; CHECK-NEXT:    vredminu.vs v8, v8, v9
 ; CHECK-NEXT:    vmv.x.s a0, v8
-; CHECK-NEXT:    vredminu.vs v8, v9, v10
-; CHECK-NEXT:    vmv.x.s a1, v8
-; CHECK-NEXT:    bltu a0, a1, .LBB11_2
-; CHECK-NEXT:  # %bb.1:
-; CHECK-NEXT:    mv a0, a1
-; CHECK-NEXT:  .LBB11_2:
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.umin.i32.v4i32(<4 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.umin.i32.v4i32(<4 x i32> %b)
   %r = call i32 @llvm.umin.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 define i32 @umax_i32(<4 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: umax_i32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
 ; CHECK-NEXT:    vmv.s.x v10, zero
+; CHECK-NEXT:    vmaxu.vv v8, v8, v9
 ; CHECK-NEXT:    vredmaxu.vs v8, v8, v10
 ; CHECK-NEXT:    vmv.x.s a0, v8
-; CHECK-NEXT:    vredmaxu.vs v8, v9, v10
-; CHECK-NEXT:    vmv.x.s a1, v8
-; CHECK-NEXT:    bltu a1, a0, .LBB12_2
-; CHECK-NEXT:  # %bb.1:
-; CHECK-NEXT:    mv a0, a1
-; CHECK-NEXT:  .LBB12_2:
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.umax.i32.v4i32(<4 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.umax.i32.v4i32(<4 x i32> %b)
   %r = call i32 @llvm.umax.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 define i32 @smin_i32(<4 x i32> %a, <4 x i32> %b) {
 ; RV32-LABEL: smin_i32:
 ; RV32:       # %bb.0:
+; RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
+; RV32-NEXT:    vmin.vv v8, v8, v9
 ; RV32-NEXT:    lui a0, 524288
 ; RV32-NEXT:    addi a0, a0, -1
-; RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
-; RV32-NEXT:    vmv.s.x v10, a0
-; RV32-NEXT:    vredmin.vs v8, v8, v10
+; RV32-NEXT:    vmv.s.x v9, a0
+; RV32-NEXT:    vredmin.vs v8, v8, v9
 ; RV32-NEXT:    vmv.x.s a0, v8
-; RV32-NEXT:    vredmin.vs v8, v9, v10
-; RV32-NEXT:    vmv.x.s a1, v8
-; RV32-NEXT:    blt a0, a1, .LBB13_2
-; RV32-NEXT:  # %bb.1:
-; RV32-NEXT:    mv a0, a1
-; RV32-NEXT:  .LBB13_2:
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: smin_i32:
 ; RV64:       # %bb.0:
+; RV64-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
+; RV64-NEXT:    vmin.vv v8, v8, v9
 ; RV64-NEXT:    lui a0, 524288
 ; RV64-NEXT:    addiw a0, a0, -1
-; RV64-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
-; RV64-NEXT:    vmv.s.x v10, a0
-; RV64-NEXT:    vredmin.vs v8, v8, v10
+; RV64-NEXT:    vmv.s.x v9, a0
+; RV64-NEXT:    vredmin.vs v8, v8, v9
 ; RV64-NEXT:    vmv.x.s a0, v8
-; RV64-NEXT:    vredmin.vs v8, v9, v10
-; RV64-NEXT:    vmv.x.s a1, v8
-; RV64-NEXT:    blt a0, a1, .LBB13_2
-; RV64-NEXT:  # %bb.1:
-; RV64-NEXT:    mv a0, a1
-; RV64-NEXT:  .LBB13_2:
 ; RV64-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.smin.i32.v4i32(<4 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.smin.i32.v4i32(<4 x i32> %b)
   %r = call i32 @llvm.smin.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 define i32 @smax_i32(<4 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: smax_i32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    lui a0, 524288
 ; CHECK-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
 ; CHECK-NEXT:    vmv.s.x v10, a0
+; CHECK-NEXT:    vmax.vv v8, v8, v9
 ; CHECK-NEXT:    vredmax.vs v8, v8, v10
 ; CHECK-NEXT:    vmv.x.s a0, v8
-; CHECK-NEXT:    vredmax.vs v8, v9, v10
-; CHECK-NEXT:    vmv.x.s a1, v8
-; CHECK-NEXT:    blt a1, a0, .LBB14_2
-; CHECK-NEXT:  # %bb.1:
-; CHECK-NEXT:    mv a0, a1
-; CHECK-NEXT:  .LBB14_2:
 ; CHECK-NEXT:    ret
   %r1 = call i32 @llvm.vector.reduce.smax.i32.v4i32(<4 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.smax.i32.v4i32(<4 x i32> %b)
   %r = call i32 @llvm.smax.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 declare float @llvm.vector.reduce.fadd.f32.v4f32(float, <4 x float>)

llvm/test/CodeGen/Thumb2/mve-doublereduct.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve.fp -float-abi=hard -verify-machineinstrs %s -o - | FileCheck %s
 
 define float @add_f32(<8 x float> %a, <4 x float> %b) {
 ; CHECK-LABEL: add_f32:
 ; CHECK:       @ %bb.0:
 ; CHECK-NEXT:    vadd.f32 q0, q0, q1
-; CHECK-NEXT:    vadd.f32 s4, s10, s11
+; CHECK-NEXT:    vadd.f32 q0, q0, q2
 ; CHECK-NEXT:    vadd.f32 s2, s2, s3
 ; CHECK-NEXT:    vadd.f32 s0, s0, s1
-; CHECK-NEXT:    vadd.f32 s6, s8, s9
-; CHECK-NEXT:    vadd.f32 s0, s0, s2
-; CHECK-NEXT:    vadd.f32 s2, s6, s4
 ; CHECK-NEXT:    vadd.f32 s0, s0, s2
 ; CHECK-NEXT:    bx lr
   %r1 = call fast float @llvm.vector.reduce.fadd.f32.v8f32(float -0.0, <8 x float> %a)
   %r2 = call fast float @llvm.vector.reduce.fadd.f32.v4f32(float -0.0, <4 x float> %b)
   %r = fadd fast float %r1, %r2
   ret float %r
 }
 
 define float @fmul_f32(<8 x float> %a, <4 x float> %b) {
 ; CHECK-LABEL: fmul_f32:
 ; CHECK:       @ %bb.0:
 ; CHECK-NEXT:    vmul.f32 q0, q0, q1
-; CHECK-NEXT:    vmul.f32 s4, s10, s11
+; CHECK-NEXT:    vmul.f32 q0, q0, q2
 ; CHECK-NEXT:    vmul.f32 s2, s2, s3
 ; CHECK-NEXT:    vmul.f32 s0, s0, s1
-; CHECK-NEXT:    vmul.f32 s6, s8, s9
-; CHECK-NEXT:    vmul.f32 s0, s0, s2
-; CHECK-NEXT:    vmul.f32 s2, s6, s4
 ; CHECK-NEXT:    vmul.f32 s0, s0, s2
 ; CHECK-NEXT:    bx lr
   %r1 = call fast float @llvm.vector.reduce.fmul.f32.v8f32(float 1.0, <8 x float> %a)
   %r2 = call fast float @llvm.vector.reduce.fmul.f32.v4f32(float 1.0, <4 x float> %b)
   %r = fmul fast float %r1, %r2
   ret float %r
 }
 
; CHECK-NEXT:    bx lr
   %r2 = call i16 @llvm.vector.reduce.add.i16.v16i16(<16 x i16> %be)
   %r = add i16 %r1, %r2
   ret i16 %r
 }
 
 define i32 @mul_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: mul_i32:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    .save {r4, r5, r6, lr}
-; CHECK-NEXT:    push {r4, r5, r6, lr}
 ; CHECK-NEXT:    vmul.i32 q0, q0, q1
-; CHECK-NEXT:    vmov r0, r1, d5
-; CHECK-NEXT:    vmov r6, r3, d0
-; CHECK-NEXT:    vmov r12, lr, d1
-; CHECK-NEXT:    vmov r4, r5, d4
+; CHECK-NEXT:    vmul.i32 q0, q0, q2
+; CHECK-NEXT:    vmov r0, r1, d1
+; CHECK-NEXT:    vmov r2, r3, d0
 ; CHECK-NEXT:    muls r0, r1, r0
-; CHECK-NEXT:    mul r2, r12, lr
-; CHECK-NEXT:    muls r3, r6, r3
-; CHECK-NEXT:    mul r1, r4, r5
-; CHECK-NEXT:    muls r2, r3, r2
+; CHECK-NEXT:    mul r1, r2, r3
 ; CHECK-NEXT:    muls r0, r1, r0
-; CHECK-NEXT:    muls r0, r2, r0
-; CHECK-NEXT:    pop {r4, r5, r6, pc}
+; CHECK-NEXT:    bx lr
   %r1 = call i32 @llvm.vector.reduce.mul.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.mul.i32.v4i32(<4 x i32> %b)
   %r = mul i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @and_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: and_i32:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    .save {r4, r5, r6, lr}
-; CHECK-NEXT:    push {r4, r5, r6, lr}
 ; CHECK-NEXT:    vand q0, q0, q1
-; CHECK-NEXT:    vmov r6, r1, d5
+; CHECK-NEXT:    vand q0, q0, q2
+; CHECK-NEXT:    vmov r0, r1, d1
 ; CHECK-NEXT:    vmov r2, r3, d0
-; CHECK-NEXT:    vmov r12, lr, d1
-; CHECK-NEXT:    vmov r4, r5, d4
-; CHECK-NEXT:    ands r1, r6
-; CHECK-NEXT:    ands r2, r3
-; CHECK-NEXT:    and.w r0, r12, lr
-; CHECK-NEXT:    ands r0, r2
-; CHECK-NEXT:    and.w r2, r4, r5
-; CHECK-NEXT:    ands r1, r2
 ; CHECK-NEXT:    ands r0, r1
-; CHECK-NEXT:    pop {r4, r5, r6, pc}
+; CHECK-NEXT:    and.w r1, r2, r3
+; CHECK-NEXT:    ands r0, r1
+; CHECK-NEXT:    bx lr
   %r1 = call i32 @llvm.vector.reduce.and.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.and.i32.v4i32(<4 x i32> %b)
   %r = and i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @or_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: or_i32:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    .save {r4, r5, r6, lr}
-; CHECK-NEXT:    push {r4, r5, r6, lr}
 ; CHECK-NEXT:    vorr q0, q0, q1
-; CHECK-NEXT:    vmov r6, r1, d5
+; CHECK-NEXT:    vorr q0, q0, q2
+; CHECK-NEXT:    vmov r0, r1, d1
 ; CHECK-NEXT:    vmov r2, r3, d0
-; CHECK-NEXT:    vmov r12, lr, d1
-; CHECK-NEXT:    vmov r4, r5, d4
-; CHECK-NEXT:    orrs r1, r6
-; CHECK-NEXT:    orrs r2, r3
-; CHECK-NEXT:    orr.w r0, r12, lr
-; CHECK-NEXT:    orrs r0, r2
-; CHECK-NEXT:    orr.w r2, r4, r5
-; CHECK-NEXT:    orrs r1, r2
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    pop {r4, r5, r6, pc}
+; CHECK-NEXT:    orr.w r1, r2, r3
+; CHECK-NEXT:    orrs r0, r1
+; CHECK-NEXT:    bx lr
   %r1 = call i32 @llvm.vector.reduce.or.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.or.i32.v4i32(<4 x i32> %b)
   %r = or i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @xor_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: xor_i32:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    .save {r4, r5, r6, lr}
-; CHECK-NEXT:    push {r4, r5, r6, lr}
 ; CHECK-NEXT:    veor q0, q0, q1
-; CHECK-NEXT:    vmov r6, r1, d5
+; CHECK-NEXT:    veor q0, q0, q2
+; CHECK-NEXT:    vmov r0, r1, d1
 ; CHECK-NEXT:    vmov r2, r3, d0
-; CHECK-NEXT:    vmov r12, lr, d1
-; CHECK-NEXT:    vmov r4, r5, d4
-; CHECK-NEXT:    eors r1, r6
-; CHECK-NEXT:    eors r2, r3
-; CHECK-NEXT:    eor.w r0, r12, lr
-; CHECK-NEXT:    eors r0, r2
-; CHECK-NEXT:    eor.w r2, r4, r5
-; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    eors r0, r1
-; CHECK-NEXT:    pop {r4, r5, r6, pc}
+; CHECK-NEXT:    eor.w r1, r2, r3
+; CHECK-NEXT:    eors r0, r1
+; CHECK-NEXT:    bx lr
   %r1 = call i32 @llvm.vector.reduce.xor.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.xor.i32.v4i32(<4 x i32> %b)
   %r = xor i32 %r1, %r2
   ret i32 %r
 }
 
 define i32 @umin_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: umin_i32:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    mov.w r0, #-1
 ; CHECK-NEXT:    vmin.u32 q0, q0, q1
-; CHECK-NEXT:    mov.w r1, #-1
-; CHECK-NEXT:    vminv.u32 r0, q2
-; CHECK-NEXT:    vminv.u32 r1, q0
-; CHECK-NEXT:    cmp r1, r0
-; CHECK-NEXT:    csel r0, r1, r0, lo
+; CHECK-NEXT:    mov.w r0, #-1
+; CHECK-NEXT:    vmin.u32 q0, q0, q2
+; CHECK-NEXT:    vminv.u32 r0, q0
 ; CHECK-NEXT:    bx lr
   %r1 = call i32 @llvm.vector.reduce.umin.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.umin.i32.v4i32(<4 x i32> %b)
   %r = call i32 @llvm.umin.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 define i32 @umax_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: umax_i32:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    vmax.u32 q0, q0, q1
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    vmaxv.u32 r0, q2
-; CHECK-NEXT:    vmaxv.u32 r1, q0
-; CHECK-NEXT:    cmp r1, r0
-; CHECK-NEXT:    csel r0, r1, r0, hi
+; CHECK-NEXT:    movs r0, #0
+; CHECK-NEXT:    vmax.u32 q0, q0, q2
+; CHECK-NEXT:    vmaxv.u32 r0, q0
 ; CHECK-NEXT:    bx lr
   %r1 = call i32 @llvm.vector.reduce.umax.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.umax.i32.v4i32(<4 x i32> %b)
   %r = call i32 @llvm.umax.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 define i32 @smin_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: smin_i32:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    mvn r0, #-2147483648
 ; CHECK-NEXT:    vmin.s32 q0, q0, q1
-; CHECK-NEXT:    mvn r1, #-2147483648
-; CHECK-NEXT:    vminv.s32 r0, q2
-; CHECK-NEXT:    vminv.s32 r1, q0
-; CHECK-NEXT:    cmp r1, r0
-; CHECK-NEXT:    csel r0, r1, r0, lt
+; CHECK-NEXT:    mvn r0, #-2147483648
+; CHECK-NEXT:    vmin.s32 q0, q0, q2
+; CHECK-NEXT:    vminv.s32 r0, q0
 ; CHECK-NEXT:    bx lr
   %r1 = call i32 @llvm.vector.reduce.smin.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.smin.i32.v4i32(<4 x i32> %b)
   %r = call i32 @llvm.smin.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 define i32 @smax_i32(<8 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: smax_i32:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    mov.w r0, #-2147483648
 ; CHECK-NEXT:    vmax.s32 q0, q0, q1
-; CHECK-NEXT:    mov.w r1, #-2147483648
-; CHECK-NEXT:    vmaxv.s32 r0, q2
-; CHECK-NEXT:    vmaxv.s32 r1, q0
-; CHECK-NEXT:    cmp r1, r0
-; CHECK-NEXT:    csel r0, r1, r0, gt
+; CHECK-NEXT:    mov.w r0, #-2147483648
+; CHECK-NEXT:    vmax.s32 q0, q0, q2
+; CHECK-NEXT:    vmaxv.s32 r0, q0
 ; CHECK-NEXT:    bx lr
   %r1 = call i32 @llvm.vector.reduce.smax.i32.v8i32(<8 x i32> %a)
   %r2 = call i32 @llvm.vector.reduce.smax.i32.v4i32(<4 x i32> %b)
   %r = call i32 @llvm.smax.i32(i32 %r1, i32 %r2)
   ret i32 %r
 }
 
 declare float @llvm.vector.reduce.fadd.f32.v8f32(float, <8 x float>)