[SVE][CodeGen] Lower scalable integer vector reductions

[kmclaughlin]

Summary

This patch uses the existing LowerFixedLengthReductionToSVE function to also lower
scalable vector reductions. A separate function has been added to lower VECREDUCE_AND
& VECREDUCE_OR operations with predicate types using ptest.

Lowering scalable floating-point reductions will be addressed in a follow up patch,
for now these will hit the assertion added to expandVecReduce() in TargetLowering.

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

1037–1038

FYI: I guess we'll just promote[1] things like VECREDCUDE_ADD but I think [u/s][min/max] are just variants of AND and OR when it comes to i1 types.

[1] Depending on the cost of ptest and the fact VECREDUCE_AND requires the extra xor, it might end up better to just promote all i1 base operations. Eitherway I'm happy with your current approach I'm just making you aware of a potential future change.

16179

Is OverrideNEON needed here? I ask because when SrcVT is a fixed length vector the value of OverrideNEON seems irrelevant as by calling this function the only safe action is to pack the vector into a scalable type and thus you may as well just always pass in true.

16186–16187

My preference would be to not have any interdependence between these two lowering functions. If you agree and considering LowerPredReductionToSVE has it's own switch statement, can this be moved into LowerVECREDUCE within the if (SrcVT.isScalableVector... block and have LowerPredReductionToSVE create its own predicate..

llvm/test/CodeGen/AArch64/sve-int-reduce.ll

50

This is personal preference so feel free to ignore but considering they go down different code paths and have structurally different output I'd prefer to have the predicate tests within their own ll file.

[kmclaughlin]

• Lower vecreduce_[s|u]max & vecreduce_[s|u]min to AND and OR for i1 types
• Add lowering for vecreduce_xor to LowerPredReductionToSVE using the cntp intrinsic
• Split predicate reduction tests into separate .ll files

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

3906–3907

On reflection and since you're going the extra mile, I'd rather have this transformation (and the matching reduction ones) at the SelectionDAG::getNode() level so that we can just force a canonical representation for all targets as early as possible and then it'll never be a concern from a legalisation point of view.

[kmclaughlin]

• Moved the transformations of i1 [s|u]min & [s|u]max -> and/or to SelectionDAG::getNode()
• Removed custom lowering of vecreduce_[s|u]min & vecreduce_[s|u]max for predicate types

[kmclaughlin]

• Changed getNode() to check if the operand type of vecreduce min/max is i1 instead of the result type
• Fixed a mistake with the changes to getNode() in the previous patch, where the transformations of [s|u]min & [s|u]max would also apply to other operations

[paulwalker-arm]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

3324 ↗

(On Diff #299730)

I think you want to just reject extracts from scalable vectors here. There's code at the top of this function which normally does the job, but it relies on the result type and so EXTRACT_VECTOR-ELT slips through. FYI: Eli has the same fix in D87651 so I'd match that and crown whoever lands first the winner.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

9758

Given LowerPredReductionToSVE has it's own switch block can this be done universally regardless of the opcode. i.e.

if (SrcVT.isScalableVector() ||
      useSVEForFixedLengthVectorVT(SrcVT, OverrideNEON)) {

  if (SrcVT.getVectorElementType() == MVT::i1)
    return LowerPredReductionToSVE(Op, DAG);

  switch (Op.getOpcode()) {

16184–16187

As you're moving this, can you put it immediately before the SDValue Rdx line so it has some company?

llvm/test/CodeGen/AArch64/vecreduce-umax-legalization.ll

145–152 ↗

(On Diff #299730)

Presumably this is down to the canonicalisation? I'm not sure there's any immediate performance concerns here, and I stand behind canonicalisation being the correct solution. I guess we can see if anybody disagrees with this.

The worst that can happen is more custom lowering for NEON is required but because the existing lowering doesn't do anything special for OR, and MIN/MAX operations on i1 is weird anyways, I'd rather not do that in this patch unless we really have to.

[kmclaughlin]

• Changed fix for the warning in computeKnownBits for extract_vector_elt to match D87651
• Moved check for i1 types in LowerVECREDUCE outside of the switch statement

[kmclaughlin]

llvm/test/CodeGen/AArch64/vecreduce-umax-legalization.ll

145–152 ↗

(On Diff #299730)

It is, I needed to change this test after I moved the transformation of vecreduce_umax to getNode()

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

16151

Shouldn't this be ||?

16168

This will cause a compiler warning if anybody adds a case statement after ISD::VECREDUCE_XOR. Personally I'd just wrap all the case blocks in {} and move AArch64CC::CondCode Cond into the blocks that need it.

16172–16174

The AND is not required because ISD::VECREDUCE_XOR only says the bottom N bits are defined, where N is the size of the operand's vector element.

The call to getAnyExtOrTrunc should be using ReduceOp.getValueType() and not hardwiring MVT::i32.

Given this change ReduceOp.getValueType() will be used three times so I'd just have EVT VT = ReduceOp.getValueType(); at the top of the function, which might have the affect that AArch64CC::####_ACTIVE can be used in place and still fit on the line and thus remove the need for {} for those entries.

[kmclaughlin]

Changes to LowerPredReductionToSVE:

• Fixed if statement which should be using ||
• Removed unnecessary And when lowering VECREDUCE_XOR
• Wrapped case blocks in {} where necessary

[kmclaughlin]

Thanks for reviewing this, @paulwalker-arm!

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

 }
 
 SDValue DAGCombiner::visitVECREDUCE(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   EVT VT = N0.getValueType();
   unsigned Opcode = N->getOpcode();
 
   // VECREDUCE over 1-element vector is just an extract.
-  if (VT.getVectorNumElements() == 1) {
+  if (VT.getVectorElementCount().isScalar()) {
     SDLoc dl(N);
     SDValue Res =
         DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, VT.getVectorElementType(), N0,
                     DAG.getVectorIdxConstant(0, dl));
     if (Res.getValueType() != N->getValueType(0))
       Res = DAG.getNode(ISD::ANY_EXTEND, dl, N->getValueType(0), Res);
     return Res;
   }

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

SDValue TargetLowering::expandVecReduce(SDNode *Node, SelectionDAG &DAG) const {
   case ISD::VECREDUCE_UMIN: BaseOpcode = ISD::UMIN; break;
   case ISD::VECREDUCE_FMAX: BaseOpcode = ISD::FMAXNUM; break;
   case ISD::VECREDUCE_FMIN: BaseOpcode = ISD::FMINNUM; break;
   }
 
   SDValue Op = Node->getOperand(0);
   EVT VT = Op.getValueType();
 
+  if (VT.isScalableVector())
+    report_fatal_error(
+        "Expanding reductions for scalable vectors is undefined.");
+
   // Try to use a shuffle reduction for power of two vectors.
   if (VT.isPow2VectorType()) {
     while (VT.getVectorNumElements() > 1) {
       EVT HalfVT = VT.getHalfNumVectorElementsVT(*DAG.getContext());
       if (!isOperationLegalOrCustom(BaseOpcode, HalfVT))
         break;
 
       SDValue Lo, Hi;

llvm/lib/Target/AArch64/AArch64ISelLowering.h

private:
   SDValue LowerSVEStructLoad(unsigned Intrinsic, ArrayRef<SDValue> LoadOps,
                              EVT VT, SelectionDAG &DAG, const SDLoc &DL) const;
 
   SDValue LowerFixedLengthVectorIntDivideToSVE(SDValue Op,
                                                SelectionDAG &DAG) const;
   SDValue LowerFixedLengthVectorIntExtendToSVE(SDValue Op,
                                                SelectionDAG &DAG) const;
   SDValue LowerFixedLengthVectorLoadToSVE(SDValue Op, SelectionDAG &DAG) const;
-  SDValue LowerFixedLengthReductionToSVE(unsigned Opcode, SDValue ScalarOp,
-                                         SelectionDAG &DAG) const;
+  SDValue LowerPredReductionToSVE(unsigned Opcode, SDValue ScalarOp, SDValue Pg,
+  SDValue LowerReductionToSVE(unsigned Opcode, SDValue ScalarOp,
+                              SelectionDAG &DAG, bool OverrideNEON) const;
   SDValue LowerFixedLengthVectorSelectToSVE(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFixedLengthVectorSetccToSVE(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFixedLengthVectorStoreToSVE(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFixedLengthVectorTruncateToSVE(SDValue Op,
                                               SelectionDAG &DAG) const;
 
   SDValue BuildSDIVPow2(SDNode *N, const APInt &Divisor, SelectionDAG &DAG,
                         SmallVectorImpl<SDNode *> &Created) const override;
   SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
                           int &ExtraSteps, bool &UseOneConst,
                           bool Reciprocal) const override;

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

for (auto VT : {MVT::nxv16i8, MVT::nxv8i16, MVT::nxv4i32, MVT::nxv2i64}) {
       setOperationAction(ISD::UDIV, VT, Custom);
       setOperationAction(ISD::SMIN, VT, Custom);
       setOperationAction(ISD::UMIN, VT, Custom);
       setOperationAction(ISD::SMAX, VT, Custom);
       setOperationAction(ISD::UMAX, VT, Custom);
       setOperationAction(ISD::SHL, VT, Custom);
       setOperationAction(ISD::SRL, VT, Custom);
       setOperationAction(ISD::SRA, VT, Custom);
+      setOperationAction(ISD::VECREDUCE_ADD, VT, Custom);
+      setOperationAction(ISD::VECREDUCE_AND, VT, Custom);
+      setOperationAction(ISD::VECREDUCE_OR, VT, Custom);
+      setOperationAction(ISD::VECREDUCE_XOR, VT, Custom);
+      setOperationAction(ISD::VECREDUCE_UMIN, VT, Custom);
+      setOperationAction(ISD::VECREDUCE_UMAX, VT, Custom);
+      setOperationAction(ISD::VECREDUCE_SMIN, VT, Custom);
+      setOperationAction(ISD::VECREDUCE_SMAX, VT, Custom);
     }
 
     // Illegal unpacked integer vector types.
     for (auto VT : {MVT::nxv8i8, MVT::nxv4i16, MVT::nxv2i32}) {
       setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Custom);
       setOperationAction(ISD::INSERT_SUBVECTOR, VT, Custom);
     }
 
     for (auto VT : {MVT::nxv16i1, MVT::nxv8i1, MVT::nxv4i1, MVT::nxv2i1}) {
       setOperationAction(ISD::CONCAT_VECTORS, VT, Custom);
       setOperationAction(ISD::SELECT, VT, Custom);
       setOperationAction(ISD::SETCC, VT, Custom);
       setOperationAction(ISD::SPLAT_VECTOR, VT, Custom);
       setOperationAction(ISD::TRUNCATE, VT, Custom);
+      setOperationAction(ISD::VECREDUCE_AND, VT, Custom);
+      setOperationAction(ISD::VECREDUCE_OR, VT, Custom);
 
       // There are no legal MVT::nxv16f## based types.
       if (VT != MVT::nxv16i1) {
         setOperationAction(ISD::SINT_TO_FP, VT, Promote);
         AddPromotedToType(ISD::SINT_TO_FP, VT, getPromotedVTForPredicate(VT));
         setOperationAction(ISD::UINT_TO_FP, VT, Promote);
         AddPromotedToType(ISD::UINT_TO_FP, VT, getPromotedVTForPredicate(VT));
       }
SDValue AArch64TargetLowering::LowerOperation(SDValue Op,
   case ISD::EXTRACT_SUBVECTOR:
     return LowerEXTRACT_SUBVECTOR(Op, DAG);
   case ISD::INSERT_SUBVECTOR:
     return LowerINSERT_SUBVECTOR(Op, DAG);
   case ISD::SDIV:
   case ISD::UDIV:
     return LowerDIV(Op, DAG);
   case ISD::SMIN:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::SMIN_PRED,
                                /*OverrideNEON=*/true);
   case ISD::UMIN:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::UMIN_PRED,
                                /*OverrideNEON=*/true);
   case ISD::SMAX:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::SMAX_PRED,
                                /*OverrideNEON=*/true);
   case ISD::UMAX:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::UMAX_PRED,
SDValue AArch64TargetLowering::LowerVECREDUCE(SDValue Op,
 
   // Try to lower fixed length reductions to SVE.
   EVT SrcVT = Src.getValueType();
   bool OverrideNEON = Op.getOpcode() == ISD::VECREDUCE_AND ||
                       Op.getOpcode() == ISD::VECREDUCE_OR ||
                       Op.getOpcode() == ISD::VECREDUCE_XOR ||
                       (Op.getOpcode() != ISD::VECREDUCE_ADD &&
                        SrcVT.getVectorElementType() == MVT::i64);
-  if (useSVEForFixedLengthVectorVT(SrcVT, OverrideNEON)) {
+  if (SrcVT.isScalableVector() ||
+      useSVEForFixedLengthVectorVT(SrcVT, OverrideNEON)) {
     switch (Op.getOpcode()) {
     case ISD::VECREDUCE_ADD:
-      return LowerFixedLengthReductionToSVE(AArch64ISD::UADDV_PRED, Op, DAG);
+      return LowerReductionToSVE(AArch64ISD::UADDV_PRED, Op, DAG, OverrideNEON);
     case ISD::VECREDUCE_AND:
-      return LowerFixedLengthReductionToSVE(AArch64ISD::ANDV_PRED, Op, DAG);
+      return LowerReductionToSVE(AArch64ISD::ANDV_PRED, Op, DAG, OverrideNEON);
     case ISD::VECREDUCE_OR:
-      return LowerFixedLengthReductionToSVE(AArch64ISD::ORV_PRED, Op, DAG);
+      return LowerReductionToSVE(AArch64ISD::ORV_PRED, Op, DAG, OverrideNEON);
     case ISD::VECREDUCE_SMAX:
-      return LowerFixedLengthReductionToSVE(AArch64ISD::SMAXV_PRED, Op, DAG);
+      return LowerReductionToSVE(AArch64ISD::SMAXV_PRED, Op, DAG, OverrideNEON);
     case ISD::VECREDUCE_SMIN:
-      return LowerFixedLengthReductionToSVE(AArch64ISD::SMINV_PRED, Op, DAG);
+      return LowerReductionToSVE(AArch64ISD::SMINV_PRED, Op, DAG, OverrideNEON);
     case ISD::VECREDUCE_UMAX:
-      return LowerFixedLengthReductionToSVE(AArch64ISD::UMAXV_PRED, Op, DAG);
+      return LowerReductionToSVE(AArch64ISD::UMAXV_PRED, Op, DAG, OverrideNEON);
     case ISD::VECREDUCE_UMIN:
-      return LowerFixedLengthReductionToSVE(AArch64ISD::UMINV_PRED, Op, DAG);
+      return LowerReductionToSVE(AArch64ISD::UMINV_PRED, Op, DAG, OverrideNEON);
     case ISD::VECREDUCE_XOR:
-      return LowerFixedLengthReductionToSVE(AArch64ISD::EORV_PRED, Op, DAG);
+      return LowerReductionToSVE(AArch64ISD::EORV_PRED, Op, DAG, OverrideNEON);
     case ISD::VECREDUCE_FMAX:
-      return LowerFixedLengthReductionToSVE(AArch64ISD::FMAXNMV_PRED, Op, DAG);
+      return LowerReductionToSVE(AArch64ISD::FMAXNMV_PRED, Op, DAG,
+                                 OverrideNEON);
     case ISD::VECREDUCE_FMIN:
-      return LowerFixedLengthReductionToSVE(AArch64ISD::FMINNMV_PRED, Op, DAG);
+      return LowerReductionToSVE(AArch64ISD::FMINNMV_PRED, Op, DAG,
+                                 OverrideNEON);
     default:
       llvm_unreachable("Unhandled fixed length reduction");
     }
   }
 
   // Lower NEON reductions.
   SDLoc dl(Op);
   switch (Op.getOpcode()) {
assert(useSVEForFixedLengthVectorVT(V.getValueType()) &&
            "Only fixed length vectors are supported!");
     Ops.push_back(convertToScalableVector(DAG, ContainerVT, V));
   }
 
   auto ScalableRes = DAG.getNode(Op.getOpcode(), SDLoc(Op), ContainerVT, Ops);
   return convertFromScalableVector(DAG, VT, ScalableRes);
 }
 
-SDValue AArch64TargetLowering::LowerFixedLengthReductionToSVE(unsigned Opcode,
-    SDValue ScalarOp, SelectionDAG &DAG) const {
+SDValue AArch64TargetLowering::LowerPredReductionToSVE(unsigned Opcode,
+                                                       SDValue ScalarOp,
+                                                       SDValue Pg,
+                                                       SelectionDAG &DAG) const {
+  SDValue Op = ScalarOp.getOperand(0);
+  EVT OpVT = Op.getValueType();
+
+  if (!OpVT.isScalableVector() && OpVT.getVectorElementType() != MVT::i1)
+    return SDValue();
+
+  AArch64CC::CondCode Cond;
+  switch (ScalarOp.getOpcode()) {
+  default:
+    return SDValue();
+  case ISD::VECREDUCE_OR:
+    Cond = AArch64CC::ANY_ACTIVE;
+    break;
+  case ISD::VECREDUCE_AND:
+    Cond = AArch64CC::NONE_ACTIVE;
+    Op = DAG.getNode(ISD::XOR, SDLoc(ScalarOp), OpVT, Op, Pg);
+    break;
+  }
+
+  return getPTest(DAG, ScalarOp.getValueType(), Pg, Op, Cond);
+}
+
+SDValue AArch64TargetLowering::LowerReductionToSVE(unsigned Opcode,
+                                                   SDValue ScalarOp,
+                                                   SelectionDAG &DAG,
+                                                   bool OverrideNEON) const {
   SDLoc DL(ScalarOp);
   SDValue VecOp = ScalarOp.getOperand(0);
   EVT SrcVT = VecOp.getValueType();
 
-  SDValue Pg = getPredicateForVector(DAG, DL, SrcVT);
+  if (useSVEForFixedLengthVectorVT(SrcVT, OverrideNEON)) {
     EVT ContainerVT = getContainerForFixedLengthVector(DAG, SrcVT);
     VecOp = convertToScalableVector(DAG, ContainerVT, VecOp);
+  }
+
+  SDValue Pg = getPredicateForVector(DAG, DL, SrcVT);
+
+  if (SrcVT.getVectorElementType() == MVT::i1)
+    return LowerPredReductionToSVE(Opcode, ScalarOp, Pg, DAG);
 
   // UADDV always returns an i64 result.
   EVT ResVT = (Opcode == AArch64ISD::UADDV_PRED) ? MVT::i64 :
                                                    SrcVT.getVectorElementType();
 
   SDValue Rdx = DAG.getNode(Opcode, DL, getPackedSVEVectorVT(ResVT), Pg, VecOp);
   SDValue Res = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, ResVT,
                             Rdx, DAG.getConstant(0, DL, MVT::i64));
 
   // The VEC_REDUCE nodes expect an element size result.
   if (ResVT != ScalarOp.getValueType())
     Res = DAG.getAnyExtOrTrunc(Res, DL, ScalarOp.getValueType());
 
   return Res;
 }
 
 SDValue
 AArch64TargetLowering::LowerFixedLengthVectorSelectToSVE(SDValue Op,
     SelectionDAG &DAG) const {
   EVT VT = Op.getValueType();
   SDLoc DL(Op);
 
   EVT InVT = Op.getOperand(1).getValueType();
   EVT ContainerVT = getContainerForFixedLengthVector(DAG, InVT);
   SDValue Op1 = convertToScalableVector(DAG, ContainerVT, Op->getOperand(1));
   SDValue Op2 = convertToScalableVector(DAG, ContainerVT, Op->getOperand(2));
 
   // Convert the mask to a predicated (NOTE: We don't need to worry about
   // inactive lanes since VSELECT is safe when given undefined elements).
AArch64TargetLowering::LowerFixedLengthVectorSelectToSVE(SDValue Op,
   return convertFromScalableVector(DAG, VT, ScalableRes);
 }
 
 SDValue AArch64TargetLowering::LowerFixedLengthVectorSetccToSVE(
     SDValue Op, SelectionDAG &DAG) const {
   SDLoc DL(Op);
   EVT InVT = Op.getOperand(0).getValueType();
   EVT ContainerVT = getContainerForFixedLengthVector(DAG, InVT);
 
   assert(useSVEForFixedLengthVectorVT(InVT) &&
          "Only expected to lower fixed length vector operation!");
   assert(Op.getValueType() == InVT.changeTypeToInteger() &&
          "Expected integer result of the same bit length as the inputs!");
 
   // Expand floating point vector comparisons.
   if (InVT.isFloatingPoint())
     return SDValue();

llvm/test/CodeGen/AArch64/sve-int-reduce.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
+
+; ANDV
+
+define i8 @andv_nxv16i8(<vscale x 16 x i8> %a) {
+; CHECK-LABEL: andv_nxv16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    andv b0, p0, z0.b
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i8 @llvm.vector.reduce.and.nxv16i8(<vscale x 16 x i8> %a)
+  ret i8 %res
+}
+
+define i16 @andv_nxv8i16(<vscale x 8 x i16> %a) {
+; CHECK-LABEL: andv_nxv8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    andv h0, p0, z0.h
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i16 @llvm.vector.reduce.and.nxv8i16(<vscale x 8 x i16> %a)
+  ret i16 %res
+}
+
+define i32 @andv_nxv4i32(<vscale x 4 x i32> %a) {
+; CHECK-LABEL: andv_nxv4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    andv s0, p0, z0.s
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.and.nxv4i32(<vscale x 4 x i32> %a)
+  ret i32 %res
+}
+
+define i64 @andv_nxv2i64(<vscale x 2 x i64> %a) {
+; CHECK-LABEL: andv_nxv2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    andv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    ret
+  %res = call i64 @llvm.vector.reduce.and.nxv2i64(<vscale x 2 x i64> %a)
+  ret i64 %res
+}
+
+define i1 @reduce_and_nxv16i1(<vscale x 16 x i1> %vec) {
+; CHECK-LABEL: reduce_and_nxv16i1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.b
+; CHECK-NEXT:    not p0.b, p1/z, p0.b
+; CHECK-NEXT:    ptest p1, p0.b
+; CHECK-NEXT:    cset w0, eq
+; CHECK-NEXT:    ret
+  %res = call i1 @llvm.vector.reduce.and.i1.nxv16i1(<vscale x 16 x i1> %vec)
+  ret i1 %res
+}
+
+define i1 @reduce_and_nxv8i1(<vscale x 8 x i1> %vec) {
+; CHECK-LABEL: reduce_and_nxv8i1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.h
+; CHECK-NEXT:    not p0.b, p1/z, p0.b
+; CHECK-NEXT:    ptest p1, p0.b
+; CHECK-NEXT:    cset w0, eq
+; CHECK-NEXT:    ret
+  %res = call i1 @llvm.vector.reduce.and.i1.nxv8i1(<vscale x 8 x i1> %vec)
+  ret i1 %res
+}
+
+define i1 @reduce_and_nxv4i1(<vscale x 4 x i1> %vec) {
+; CHECK-LABEL: reduce_and_nxv4i1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.s
+; CHECK-NEXT:    not p0.b, p1/z, p0.b
+; CHECK-NEXT:    ptest p1, p0.b
+; CHECK-NEXT:    cset w0, eq
+; CHECK-NEXT:    ret
+  %res = call i1 @llvm.vector.reduce.and.i1.nxv4i1(<vscale x 4 x i1> %vec)
+  ret i1 %res
+}
+
+define i1 @reduce_and_nxv2i1(<vscale x 2 x i1> %vec) {
+; CHECK-LABEL: reduce_and_nxv2i1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.d
+; CHECK-NEXT:    not p0.b, p1/z, p0.b
+; CHECK-NEXT:    ptest p1, p0.b
+; CHECK-NEXT:    cset w0, eq
+; CHECK-NEXT:    ret
+  %res = call i1 @llvm.vector.reduce.and.i1.nxv2i1(<vscale x 2 x i1> %vec)
+  ret i1 %res
+}
+
+; ORV
+
+define i8 @orv_nxv16i8(<vscale x 16 x i8> %a) {
+; CHECK-LABEL: orv_nxv16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    orv b0, p0, z0.b
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i8 @llvm.vector.reduce.or.nxv16i8(<vscale x 16 x i8> %a)
+  ret i8 %res
+}
+
+define i16 @orv_nxv8i16(<vscale x 8 x i16> %a) {
+; CHECK-LABEL: orv_nxv8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    orv h0, p0, z0.h
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i16 @llvm.vector.reduce.or.nxv8i16(<vscale x 8 x i16> %a)
+  ret i16 %res
+}
+
+define i32 @orv_nxv4i32(<vscale x 4 x i32> %a) {
+; CHECK-LABEL: orv_nxv4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    orv s0, p0, z0.s
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.or.nxv4i32(<vscale x 4 x i32> %a)
+  ret i32 %res
+}
+
+define i64 @orv_nxv2i64(<vscale x 2 x i64> %a) {
+; CHECK-LABEL: orv_nxv2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    orv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    ret
+  %res = call i64 @llvm.vector.reduce.or.nxv2i64(<vscale x 2 x i64> %a)
+  ret i64 %res
+}
+
+define i1 @reduce_or_nxv16i1(<vscale x 16 x i1> %vec) {
+; CHECK-LABEL: reduce_or_nxv16i1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.b
+; CHECK-NEXT:    ptest p1, p0.b
+; CHECK-NEXT:    cset w0, ne
+; CHECK-NEXT:    ret
+  %res = call i1 @llvm.vector.reduce.or.i1.nxv16i1(<vscale x 16 x i1> %vec)
+  ret i1 %res
+}
+
+define i1 @reduce_or_nxv8i1(<vscale x 8 x i1> %vec) {
+; CHECK-LABEL: reduce_or_nxv8i1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.h
+; CHECK-NEXT:    ptest p1, p0.b
+; CHECK-NEXT:    cset w0, ne
+; CHECK-NEXT:    ret
+  %res = call i1 @llvm.vector.reduce.or.i1.nxv8i1(<vscale x 8 x i1> %vec)
+  ret i1 %res
+}
+
+define i1 @reduce_or_nxv4i1(<vscale x 4 x i1> %vec) {
+; CHECK-LABEL: reduce_or_nxv4i1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.s
+; CHECK-NEXT:    ptest p1, p0.b
+; CHECK-NEXT:    cset w0, ne
+; CHECK-NEXT:    ret
+  %res = call i1 @llvm.vector.reduce.or.i1.nxv4i1(<vscale x 4 x i1> %vec)
+  ret i1 %res
+}
+
+define i1 @reduce_or_nxv2i1(<vscale x 2 x i1> %vec) {
+; CHECK-LABEL: reduce_or_nxv2i1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p1.d
+; CHECK-NEXT:    ptest p1, p0.b
+; CHECK-NEXT:    cset w0, ne
+; CHECK-NEXT:    ret
+  %res = call i1 @llvm.vector.reduce.or.i1.nxv2i1(<vscale x 2 x i1> %vec)
+  ret i1 %res
+}
+
+; XORV
+
+define i8 @xorv_nxv16i8(<vscale x 16 x i8> %a) {
+; CHECK-LABEL: xorv_nxv16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    eorv b0, p0, z0.b
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i8 @llvm.vector.reduce.xor.nxv16i8(<vscale x 16 x i8> %a)
+  ret i8 %res
+}
+
+define i16 @xorv_nxv8i16(<vscale x 8 x i16> %a) {
+; CHECK-LABEL: xorv_nxv8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    eorv h0, p0, z0.h
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i16 @llvm.vector.reduce.xor.nxv8i16(<vscale x 8 x i16> %a)
+  ret i16 %res
+}
+
+define i32 @xorv_nxv4i32(<vscale x 4 x i32> %a) {
+; CHECK-LABEL: xorv_nxv4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    eorv s0, p0, z0.s
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.xor.nxv4i32(<vscale x 4 x i32> %a)
+  ret i32 %res
+}
+
+define i64 @xorv_nxv2i64(<vscale x 2 x i64> %a) {
+; CHECK-LABEL: xorv_nxv2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    eorv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    ret
+  %res = call i64 @llvm.vector.reduce.xor.nxv2i64(<vscale x 2 x i64> %a)
+  ret i64 %res
+}
+
+; UADDV
+
+define i8 @uaddv_nxv16i8(<vscale x 16 x i8> %a) {
+; CHECK-LABEL: uaddv_nxv16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    uaddv d0, p0, z0.b
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    ret
+  %res = call i8 @llvm.vector.reduce.add.nxv16i8(<vscale x 16 x i8> %a)
+  ret i8 %res
+}
+
+define i16 @uaddv_nxv8i16(<vscale x 8 x i16> %a) {
+; CHECK-LABEL: uaddv_nxv8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    uaddv d0, p0, z0.h
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    ret
+  %res = call i16 @llvm.vector.reduce.add.nxv8i16(<vscale x 8 x i16> %a)
+  ret i16 %res
+}
+
+define i32 @uaddv_nxv4i32(<vscale x 4 x i32> %a) {
+; CHECK-LABEL: uaddv_nxv4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    uaddv d0, p0, z0.s
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.add.nxv4i32(<vscale x 4 x i32> %a)
+  ret i32 %res
+}
+
+define i64 @uaddv_nxv2i64(<vscale x 2 x i64> %a) {
+; CHECK-LABEL: uaddv_nxv2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    uaddv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    ret
+  %res = call i64 @llvm.vector.reduce.add.nxv2i64(<vscale x 2 x i64> %a)
+  ret i64 %res
+}
+
+; UMINV
+
+define i8 @umin_nxv16i8(<vscale x 16 x i8> %a) {
+; CHECK-LABEL: umin_nxv16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    uminv b0, p0, z0.b
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i8 @llvm.vector.reduce.umin.nxv16i8(<vscale x 16 x i8> %a)
+  ret i8 %res
+}
+
+define i16 @umin_nxv8i16(<vscale x 8 x i16> %a) {
+; CHECK-LABEL: umin_nxv8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    uminv h0, p0, z0.h
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i16 @llvm.vector.reduce.umin.nxv8i16(<vscale x 8 x i16> %a)
+  ret i16 %res
+}
+
+define i32 @umin_nxv4i32(<vscale x 4 x i32> %a) {
+; CHECK-LABEL: umin_nxv4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    uminv s0, p0, z0.s
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.umin.nxv4i32(<vscale x 4 x i32> %a)
+  ret i32 %res
+}
+
+define i64 @umin_nxv2i64(<vscale x 2 x i64> %a) {
+; CHECK-LABEL: umin_nxv2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    uminv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    ret
+  %res = call i64 @llvm.vector.reduce.umin.nxv2i64(<vscale x 2 x i64> %a)
+  ret i64 %res
+}
+
+; SMINV
+
+define i8 @smin_nxv16i8(<vscale x 16 x i8> %a) {
+; CHECK-LABEL: smin_nxv16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    sminv b0, p0, z0.b
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i8 @llvm.vector.reduce.smin.nxv16i8(<vscale x 16 x i8> %a)
+  ret i8 %res
+}
+
+define i16 @smin_nxv8i16(<vscale x 8 x i16> %a) {
+; CHECK-LABEL: smin_nxv8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    sminv h0, p0, z0.h
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i16 @llvm.vector.reduce.smin.nxv8i16(<vscale x 8 x i16> %a)
+  ret i16 %res
+}
+
+define i32 @smin_nxv4i32(<vscale x 4 x i32> %a) {
+; CHECK-LABEL: smin_nxv4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    sminv s0, p0, z0.s
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.smin.nxv4i32(<vscale x 4 x i32> %a)
+  ret i32 %res
+}
+
+define i64 @smin_nxv2i64(<vscale x 2 x i64> %a) {
+; CHECK-LABEL: smin_nxv2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    sminv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    ret
+  %res = call i64 @llvm.vector.reduce.smin.nxv2i64(<vscale x 2 x i64> %a)
+  ret i64 %res
+}
+
+; UMAXV
+
+define i8 @umax_nxv16i8(<vscale x 16 x i8> %a) {
+; CHECK-LABEL: umax_nxv16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    umaxv b0, p0, z0.b
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i8 @llvm.vector.reduce.umax.nxv16i8(<vscale x 16 x i8> %a)
+  ret i8 %res
+}
+
+define i16 @umax_nxv8i16(<vscale x 8 x i16> %a) {
+; CHECK-LABEL: umax_nxv8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    umaxv h0, p0, z0.h
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i16 @llvm.vector.reduce.umax.nxv8i16(<vscale x 8 x i16> %a)
+  ret i16 %res
+}
+
+define i32 @umax_nxv4i32(<vscale x 4 x i32> %a) {
+; CHECK-LABEL: umax_nxv4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    umaxv s0, p0, z0.s
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.umax.nxv4i32(<vscale x 4 x i32> %a)
+  ret i32 %res
+}
+
+define i64 @umax_nxv2i64(<vscale x 2 x i64> %a) {
+; CHECK-LABEL: umax_nxv2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    umaxv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    ret
+  %res = call i64 @llvm.vector.reduce.umax.nxv2i64(<vscale x 2 x i64> %a)
+  ret i64 %res
+}
+
+; SMAXV
+
+define i8 @smax_nxv16i8(<vscale x 16 x i8> %a) {
+; CHECK-LABEL: smax_nxv16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    smaxv b0, p0, z0.b
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i8 @llvm.vector.reduce.smax.nxv16i8(<vscale x 16 x i8> %a)
+  ret i8 %res
+}
+
+define i16 @smax_nxv8i16(<vscale x 8 x i16> %a) {
+; CHECK-LABEL: smax_nxv8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    smaxv h0, p0, z0.h
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i16 @llvm.vector.reduce.smax.nxv8i16(<vscale x 8 x i16> %a)
+  ret i16 %res
+}
+
+define i32 @smax_nxv4i32(<vscale x 4 x i32> %a) {
+; CHECK-LABEL: smax_nxv4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    smaxv s0, p0, z0.s
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.smax.nxv4i32(<vscale x 4 x i32> %a)
+  ret i32 %res
+}
+
+define i64 @smax_nxv2i64(<vscale x 2 x i64> %a) {
+; CHECK-LABEL: smax_nxv2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    smaxv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    ret
+  %res = call i64 @llvm.vector.reduce.smax.nxv2i64(<vscale x 2 x i64> %a)
+  ret i64 %res
+}
+
+declare i8 @llvm.vector.reduce.and.nxv16i8(<vscale x 16 x i8>)
+declare i16 @llvm.vector.reduce.and.nxv8i16(<vscale x 8 x i16>)
+declare i32 @llvm.vector.reduce.and.nxv4i32(<vscale x 4 x i32>)
+declare i64 @llvm.vector.reduce.and.nxv2i64(<vscale x 2 x i64>)
+
+declare i1 @llvm.vector.reduce.and.i1.nxv16i1(<vscale x 16 x i1> %vec)
+declare i1 @llvm.vector.reduce.and.i1.nxv8i1(<vscale x 8 x i1> %vec)
+declare i1 @llvm.vector.reduce.and.i1.nxv4i1(<vscale x 4 x i1> %vec)
+declare i1 @llvm.vector.reduce.and.i1.nxv2i1(<vscale x 2 x i1> %vec)
+
+declare i8 @llvm.vector.reduce.or.nxv16i8(<vscale x 16 x i8>)
+declare i16 @llvm.vector.reduce.or.nxv8i16(<vscale x 8 x i16>)
+declare i32 @llvm.vector.reduce.or.nxv4i32(<vscale x 4 x i32>)
+declare i64 @llvm.vector.reduce.or.nxv2i64(<vscale x 2 x i64>)
+
+declare i1 @llvm.vector.reduce.or.i1.nxv16i1(<vscale x 16 x i1> %vec)
+declare i1 @llvm.vector.reduce.or.i1.nxv8i1(<vscale x 8 x i1> %vec)
+declare i1 @llvm.vector.reduce.or.i1.nxv4i1(<vscale x 4 x i1> %vec)
+declare i1 @llvm.vector.reduce.or.i1.nxv2i1(<vscale x 2 x i1> %vec)
+
+declare i8 @llvm.vector.reduce.xor.nxv16i8(<vscale x 16 x i8>)
+declare i16 @llvm.vector.reduce.xor.nxv8i16(<vscale x 8 x i16>)
+declare i32 @llvm.vector.reduce.xor.nxv4i32(<vscale x 4 x i32>)
+declare i64 @llvm.vector.reduce.xor.nxv2i64(<vscale x 2 x i64>)
+
+declare i8 @llvm.vector.reduce.add.nxv16i8(<vscale x 16 x i8>)
+declare i16 @llvm.vector.reduce.add.nxv8i16(<vscale x 8 x i16>)
+declare i32 @llvm.vector.reduce.add.nxv4i32(<vscale x 4 x i32>)
+declare i64 @llvm.vector.reduce.add.nxv2i64(<vscale x 2 x i64>)
+
+declare i8 @llvm.vector.reduce.umin.nxv16i8(<vscale x 16 x i8>)
+declare i16 @llvm.vector.reduce.umin.nxv8i16(<vscale x 8 x i16>)
+declare i32 @llvm.vector.reduce.umin.nxv4i32(<vscale x 4 x i32>)
+declare i64 @llvm.vector.reduce.umin.nxv2i64(<vscale x 2 x i64>)
+
+declare i8 @llvm.vector.reduce.smin.nxv16i8(<vscale x 16 x i8>)
+declare i16 @llvm.vector.reduce.smin.nxv8i16(<vscale x 8 x i16>)
+declare i32 @llvm.vector.reduce.smin.nxv4i32(<vscale x 4 x i32>)
+declare i64 @llvm.vector.reduce.smin.nxv2i64(<vscale x 2 x i64>)
+
+declare i8 @llvm.vector.reduce.umax.nxv16i8(<vscale x 16 x i8>)
+declare i16 @llvm.vector.reduce.umax.nxv8i16(<vscale x 8 x i16>)
+declare i32 @llvm.vector.reduce.umax.nxv4i32(<vscale x 4 x i32>)
+declare i64 @llvm.vector.reduce.umax.nxv2i64(<vscale x 2 x i64>)
+
+declare i8 @llvm.vector.reduce.smax.nxv16i8(<vscale x 16 x i8>)
+declare i16 @llvm.vector.reduce.smax.nxv8i16(<vscale x 8 x i16>)
+declare i32 @llvm.vector.reduce.smax.nxv4i32(<vscale x 4 x i32>)
+declare i64 @llvm.vector.reduce.smax.nxv2i64(<vscale x 2 x i64>)

llvm/test/CodeGen/AArch64/sve-split-int-reduce.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
+
+; ANDV
+
+define i8 @andv_nxv8i8(<vscale x 8 x i8> %a) {
+; CHECK-LABEL: andv_nxv8i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    andv h0, p0, z0.h
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i8 @llvm.vector.reduce.and.nxv8i8(<vscale x 8 x i8> %a)
+  ret i8 %res
+}
+
+define i32 @andv_nxv8i32(<vscale x 8 x i32> %a) {
+; CHECK-LABEL: andv_nxv8i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and z0.d, z0.d, z1.d
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    andv s0, p0, z0.s
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.and.nxv8i32(<vscale x 8 x i32> %a)
+  ret i32 %res
+}
+
+define i1 @andv_nxv32i1(<vscale x 32 x i1> %a) {
+; CHECK-LABEL: andv_nxv32i1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p2.b
+; CHECK-NEXT:    and p0.b, p2/z, p0.b, p1.b
+; CHECK-NEXT:    not p0.b, p2/z, p0.b
+; CHECK-NEXT:    ptest p2, p0.b
+; CHECK-NEXT:    cset w0, eq
+; CHECK-NEXT:    ret
+  %res = call i1 @llvm.vector.reduce.and.nxv32i1(<vscale x 32 x i1> %a)
+  ret i1 %res
+}
+
+define i1 @andv_nxv64i1(<vscale x 64 x i1> %a) {
+; CHECK-LABEL: andv_nxv64i1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    str x29, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT:    addvl sp, sp, #-1
+; CHECK-NEXT:    str p4, [sp, #7, mul vl] // 2-byte Folded Spill
+; CHECK-NEXT:    .cfi_escape 0x0f, 0x0c, 0x8f, 0x00, 0x11, 0x10, 0x22, 0x11, 0x08, 0x92, 0x2e, 0x00, 0x1e, 0x22 // sp + 16 + 8 * VG
+; CHECK-NEXT:    .cfi_offset w29, -16
+; CHECK-NEXT:    ptrue p4.b
+; CHECK-NEXT:    and p1.b, p4/z, p1.b, p3.b
+; CHECK-NEXT:    and p0.b, p4/z, p0.b, p2.b
+; CHECK-NEXT:    and p0.b, p4/z, p0.b, p1.b
+; CHECK-NEXT:    not p0.b, p4/z, p0.b
+; CHECK-NEXT:    ptest p4, p0.b
+; CHECK-NEXT:    ldr p4, [sp, #7, mul vl] // 2-byte Folded Reload
+; CHECK-NEXT:    cset w0, eq
+; CHECK-NEXT:    addvl sp, sp, #1
+; CHECK-NEXT:    ldr x29, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    ret
+  %res = call i1 @llvm.vector.reduce.and.nxv64i1(<vscale x 64 x i1> %a)
+  ret i1 %res
+}
+
+; ORV
+
+define i32 @orv_nxv2i32(<vscale x 2 x i32> %a) {
+; CHECK-LABEL: orv_nxv2i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    orv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.or.nxv2i32(<vscale x 2 x i32> %a)
+  ret i32 %res
+}
+
+define i64 @orv_nxv8i64(<vscale x 8 x i64> %a) {
+; CHECK-LABEL: orv_nxv8i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    orr z1.d, z1.d, z3.d
+; CHECK-NEXT:    orr z0.d, z0.d, z2.d
+; CHECK-NEXT:    orr z0.d, z0.d, z1.d
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    orv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    ret
+  %res = call i64 @llvm.vector.reduce.or.nxv8i64(<vscale x 8 x i64> %a)
+  ret i64 %res
+}
+
+define i1 @orv_nxv32i1(<vscale x 32 x i1> %a) {
+; CHECK-LABEL: orv_nxv32i1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p2.b
+; CHECK-NEXT:    orr p0.b, p2/z, p0.b, p1.b
+; CHECK-NEXT:    ptest p2, p0.b
+; CHECK-NEXT:    cset w0, ne
+; CHECK-NEXT:    ret
+  %res = call i1 @llvm.vector.reduce.or.nxv32i1(<vscale x 32 x i1> %a)
+  ret i1 %res
+}
+
+; XORV
+
+define i16 @xorv_nxv2i16(<vscale x 2 x i16> %a) {
+; CHECK-LABEL: xorv_nxv2i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    eorv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    ret
+  %res = call i16 @llvm.vector.reduce.xor.nxv2i16(<vscale x 2 x i16> %a)
+  ret i16 %res
+}
+
+define i32 @xorv_nxv8i32(<vscale x 8 x i32> %a) {
+; CHECK-LABEL: xorv_nxv8i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    eor z0.d, z0.d, z1.d
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    eorv s0, p0, z0.s
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.xor.nxv8i32(<vscale x 8 x i32> %a)
+  ret i32 %res
+}
+
+; UADDV
+
+define i16 @uaddv_nxv4i16(<vscale x 4 x i16> %a) {
+; CHECK-LABEL: uaddv_nxv4i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    uaddv d0, p0, z0.s
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    ret
+  %res = call i16 @llvm.vector.reduce.add.nxv4i16(<vscale x 4 x i16> %a)
+  ret i16 %res
+}
+
+define i16 @uaddv_nxv16i16(<vscale x 16 x i16> %a) {
+; CHECK-LABEL: uaddv_nxv16i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    add z0.h, z0.h, z1.h
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    uaddv d0, p0, z0.h
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    ret
+  %res = call i16 @llvm.vector.reduce.add.nxv16i16(<vscale x 16 x i16> %a)
+  ret i16 %res
+}
+
+define i32 @uaddv_nxv16i32(<vscale x 16 x i32> %a) {
+; CHECK-LABEL: uaddv_nxv16i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    add z1.s, z1.s, z3.s
+; CHECK-NEXT:    add z0.s, z0.s, z2.s
+; CHECK-NEXT:    add z0.s, z0.s, z1.s
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    uaddv d0, p0, z0.s
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.add.nxv16i32(<vscale x 16 x i32> %a)
+  ret i32 %res
+}
+
+; UMINV
+
+define i32 @umin_nxv2i32(<vscale x 2 x i32> %a) {
+; CHECK-LABEL: umin_nxv2i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    and z0.d, z0.d, #0xffffffff
+; CHECK-NEXT:    uminv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.umin.nxv2i32(<vscale x 2 x i32> %a)
+  ret i32 %res
+}
+
+define i64 @umin_nxv4i64(<vscale x 4 x i64> %a) {
+; CHECK-LABEL: umin_nxv4i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    umin z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    uminv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    ret
+  %res = call i64 @llvm.vector.reduce.umin.nxv4i64(<vscale x 4 x i64> %a)
+  ret i64 %res
+}
+
+; SMINV
+
+define i8 @smin_nxv4i8(<vscale x 4 x i8> %a) {
+; CHECK-LABEL: smin_nxv4i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    sxtb z0.s, p0/m, z0.s
+; CHECK-NEXT:    sminv s0, p0, z0.s
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i8 @llvm.vector.reduce.smin.nxv4i8(<vscale x 4 x i8> %a)
+  ret i8 %res
+}
+
+define i32 @smin_nxv8i32(<vscale x 8 x i32> %a) {
+; CHECK-LABEL: smin_nxv8i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    smin z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    sminv s0, p0, z0.s
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i32 @llvm.vector.reduce.smin.nxv8i32(<vscale x 8 x i32> %a)
+  ret i32 %res
+}
+
+; UMAXV
+
+define i16 @smin_nxv16i16(<vscale x 16 x i16> %a) {
+; CHECK-LABEL: smin_nxv16i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    umax z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    umaxv h0, p0, z0.h
+; CHECK-NEXT:    fmov w0, s0
+; CHECK-NEXT:    ret
+  %res = call i16 @llvm.vector.reduce.umax.nxv16i16(<vscale x 16 x i16> %a)
+  ret i16 %res
+}
+
+; SMAXV
+
+define i64 @smin_nxv8i64(<vscale x 8 x i64> %a) {
+; CHECK-LABEL: smin_nxv8i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    smax z1.d, p0/m, z1.d, z3.d
+; CHECK-NEXT:    smax z0.d, p0/m, z0.d, z2.d
+; CHECK-NEXT:    smax z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    smaxv d0, p0, z0.d
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    ret
+  %res = call i64 @llvm.vector.reduce.smax.nxv8i64(<vscale x 8 x i64> %a)
+  ret i64 %res
+}
+
+declare i8 @llvm.vector.reduce.and.nxv8i8(<vscale x 8 x i8>)
+declare i32 @llvm.vector.reduce.and.nxv8i32(<vscale x 8 x i32>)
+declare i1 @llvm.vector.reduce.and.nxv32i1(<vscale x 32 x i1>)
+declare i1 @llvm.vector.reduce.and.nxv64i1(<vscale x 64 x i1>)
+
+declare i32 @llvm.vector.reduce.or.nxv2i32(<vscale x 2 x i32>)
+declare i64 @llvm.vector.reduce.or.nxv8i64(<vscale x 8 x i64>)
+declare i1 @llvm.vector.reduce.or.nxv32i1(<vscale x 32 x i1>)
+
+declare i16 @llvm.vector.reduce.xor.nxv2i16(<vscale x 2 x i16>)
+declare i32 @llvm.vector.reduce.xor.nxv8i32(<vscale x 8 x i32>)
+
+declare i16 @llvm.vector.reduce.add.nxv4i16(<vscale x 4 x i16>)
+declare i16 @llvm.vector.reduce.add.nxv16i16(<vscale x 16 x i16>)
+declare i32 @llvm.vector.reduce.add.nxv16i32(<vscale x 16 x i32>)
+
+declare i32 @llvm.vector.reduce.umin.nxv2i32(<vscale x 2 x i32>)
+declare i64 @llvm.vector.reduce.umin.nxv4i64(<vscale x 4 x i64>)
+
+declare i8 @llvm.vector.reduce.smin.nxv4i8(<vscale x 4 x i8>)
+declare i32 @llvm.vector.reduce.smin.nxv8i32(<vscale x 8 x i32>)
+
+declare i16 @llvm.vector.reduce.umax.nxv16i16(<vscale x 16 x i16>)
+
+declare i64 @llvm.vector.reduce.smax.nxv8i64(<vscale x 8 x i64>)