[AArch64][SVE]Add cost model for vector reduce for scalable vector

[CarolineConcatto]

Summary

It uses MaxVScale to compute the maximum size of the scalable vector,
and assumes vector.reduce.<operand> uses tree-wise reduction algorithm
to compute the cost model.

Depends on D93030

[sdesmalen]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1107

My understanding is that the PairWise form is not something that can be expressed with the LLVM IR intrinsic and is therefore specific to fixed-width vectors which can use shufflevector.
Perhaps you can just assert IsPairWise is false. @david-arm any thoughts?

1110

The scalable property should match for Ty and CondTy, so that can be an assert. With the if-condition changed to only check Ty.

1114

remove TODO.

1117

nit: s/getMaxVScale().getValue()/*getMaxVScale()/

1122–1129

nit: unsigned CmpOpcode = Ty->isFPOrFPVectorTy() ? Instruction::FCmp : Instruction::ICmp;

1135

I think this should be:

Log2(LT.first) * MinMaxVectorCost + NumReduxLevels * MinMaxScalarCost

Because the number of legalized vectors will do min/max operations amongst the vectors (to end up with a single vector), before it will do the min/max reduction on the elements within the vector.
That also means distinguishing the cost for CmpSel on vectors and CmpSel on scalars.

1141

remove TODO.

1148

nit: You can just as well inline this into the expression below.

[ctetreau]

Have a couple nits, but seems reasonable to me. Somebody else should review as well.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1118

please name the type

1139

ScalableVectorType

1141

why not just return invalid cost now and assert at the call site?

1166

NIT: isa<ScalableVectorType>(ValTy)

[david-arm]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1107

I think we can probably assert IsPairwiseForm=false for now.

1114

Remove TODO here as max vscale should always be a valid value for AArch64 + scalable types?

1117

I don't think we need to use max vscale at all here to be honest. I've discussed this with Carol privately and getTypeLegalizationCost below already returns the number of parts needed (LT.first) along with the MVT for the legal vector type (LT.second). Since we have instructions that do both min/max operations (SMIN,etc.) as well as the horizontal vector reductions for min/max (SMINV, etc.) we can just do something similar to what X86 did and have a very simple cost lookup table for the legal horizontal reduction (SMINV), then add on a legalisation cost. i.e. something like:

unsigned LegalizationCost = 0;
if (not legal) {

// reduce to a single vector, i.e.
// res0 = SMIN part0, part1
// res1 = SMIN res0, part2
// ...
LegalizationCost = getMinMaxCost(); // SMIN, SMAX, etc.
LegalizationCost *= LT.first - 1;

}

// do cost lookup for SMINV, etc.
return LegalizationCost + LegalReductionCost;

1121

As mentioned above, I think we can avoid using max vscale and calculating redux levels.

1135

Similar to my comment above about max vscale, I think we can do this in a much simpler way. Rather than try to calculate a complex cost in this way using selects and compares, etc., I think we can just do it the same way that X86 does. We can have a simple cost lookup table for legal reductions (SMIN/MAX,UMIN/MAX, etc.), then add on a legalisation factor.

1141

Remove TODO?

1160

Same comments as for the MinMax case. We can avoid all reference to max vscale and redux levels, by treating this as the summation of:

LegalizationCost (sequence of FADD,ADD,etc. to reduce to single vector) + LegalReductionCost (single horizontal reduction instruction, e.g. FADDV).

I think the only case we don't support here is mul reductions, but we can just put in a high cost estimate for now, right?

[CarolineConcatto]

-Split the cost on legalization and horizontal vector reduction cost as
suggested in the review

[sdesmalen]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1121–1123

I don't think this condition is necessary because LT.first > 1 already implies splitting is required.

1124

You can use EVT::getTypeForEVT instead.

1135

Sure, we can do that. I was more thinking that we'd then lose scaling of the cost if we know something about vscale. Reductions requires successive inter-lane operations (as opposed to e.g. fadd, which can be done in parallel), so the number of lanes probably has some impact on the cost. If we make it dependent on MaxVscale, the cost-model could return a lower cost if MaxVL=128bits than if MaxVL is 1024bits (so we can tweak the costs with -msve-vector-bits=<bits>). X86 doesn't have this problem, because the vector size is always fixed, so it can use a look-up-table.
I'm not saying that I'm completely against a simple fixed cost initially, just sharing my thoughts.

1147–1148

I don't know what default could be here other than llvm_unreachable. And if that is unreachable, then you can just as well return the final cost on line 1137.

1166–1168

same as above, I think this is implied by LT.first > 1.

1169–1170

Use EVT::getTypeForEVT for LT.second ?

1176

move this to the default in switch below?

1189

For Min/Max you use a fixed-cost of 1 and here you use 2 and 16. Does that mean Min/Max should also be 2?
And should 16 actually be Invalid when this function will use InstructionCost because it cannot be expanded for SVE? If so, can you add a TODO?

[CarolineConcatto]

• use EVT instead of MVT to get the vector type
• use unreachable for the MinMax switch
• add TODO for Arithmetic cost when the ISD is not supported
• change MinMax horizontal reduction cost to be 2

[CarolineConcatto]

-remove redundant code when computing Type for MinMax Cost

[CarolineConcatto]

Thank you @sdesmalen for helping to refine the code.
I hope it is near to the end now.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1147–1148

I've added unreachable to the switch for default as it is not expected to be reached at any time.
Without the default case the compiler complains:
warning: control reaches end of non-void function
as I did not added no return after the switch.

1189

So, I have changed the cost of MinMax and Arithmetic to be 2.
I imagine it is fine for them to be the same.

[sdesmalen]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1119

We already know Ty is a VectorType (see line 1104), so this cast is redundant.

1120

Nit: can you call this LegalVTy?

With the two comments above, that would become:

Type *LegalVTy = EVT(LT.second).getTypeForEVT(Ty->getContext();

1131–1138

The switch statement below and selection of the ISD seems unnecessary if the cost is the same, you can just write:

return LegalizationCost + /*Cost of reduction*/ 2;

1159

We already know Ty is a VectorType (see line 1151), so this cast is redundant.

1166

nit: s/getArithmeticReductionCostScalableVectorType/getArithmeticReductionCostSVE

[sdesmalen]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1166

@CarolineConcatto pointed out to me that earlier on in this patch @ctetreau suggested the opposite.

My argument for having 'SVE' in the name over 'ScalableVectorType' is that the reduction cost is different when SVE is available, regardless of whether the type passed is a scalable or fixed-width vector. For fixed-width vectors, the compiler may still choose to use one of the SVE reduction instructions if available. The code in that function is not specific to scalable vectors either.

[CarolineConcatto]

-remove redundant code
-change getArithmeticReductionCostScalableVectorType to getArithmeticReductionCostSVE
-replace unsigned by int

[sdesmalen]

Thanks for the changes so far, I think the patch is nearly there now.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1153–1155

SVE has no instructions for MUL/FMUL reductions, so these should fall under default (return Invalid).

llvm/test/Analysis/CostModel/AArch64/sve-getIntrinsicInstrCost-vector-reduce.ll

15

Can you make two tests for each reduction:

• One with a legal type (<vscale x 2 x i64> (or <vscale x 2 x double> for fp reductions))
• One with an illegal type that needs splitting (<vscale x 8 x i64> (or <vscale x 8 x double> for fp reductions))

[CarolineConcatto]

• remove cost for vector.reduce.{mul"fmul}
• add regression tests for not legal vectors

[CarolineConcatto]

-add missing test for fmax,fadd with integer

[CarolineConcatto]

@Thank you Sander,
I added more tests. Now we have tests for legal and not legal, besides the test with integer for FP operands.

[CarolineConcatto]

• change vector size to 128bits

[sdesmalen]

llvm/test/Analysis/CostModel/AArch64/sve-getIntrinsicInstrCost-vector-reduce.ll

92–99

These tests can be removed because Floating Point reductions need an FP vector as their input, not an integer vector.

154

I was actually hoping for the test file to be organised in pairs of tests with a legal type, followed by an illegal (too wide) type, as follows:

define i64 @mul.i64.nxv2i64(<vscale x 2 x i64> %v) {
; CHECK-LABEL: 'mul.i64.nxv2i64'
; CHECK-NEXT: Cost Model: Found an estimated cost of .... for instruction:   %r = call i64 @llvm.vector.reduce.mul.nxv2i64(<vscale x 2 x i64> %v)
; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i64 %r

  %r = call i64 @llvm.vector.reduce.mul.nxv2i64(<vscale x 2 x i64> %v)
  ret i64 %r
}

define i64 @mul.i64.nxv8i64(<vscale x 8 x i64> %v) {
; CHECK-LABEL: 'mul.i64.nxv8i64'
; CHECK-NEXT: Cost Model: Found an estimated cost of 16 for instruction:   %r = call i64 @llvm.vector.reduce.mul.nxv8i64(<vscale x 8 x i64> %v)
; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i64 %r

  %r = call i64 @llvm.vector.reduce.mul.nxv8i64(<vscale x 8 x i64> %v)
  ret i64 %r
}

where the only difference is the number of elements (not the element type). This then clearly tests the cost on a legal type followed by a test where the code for legalization cost is exercised.

[CarolineConcatto]

-re-arrange the test
-remove f<operand> tests with integer

[sdesmalen]

LGTM

llvm/test/Analysis/CostModel/AArch64/sve-getIntrinsicInstrCost-vector-reduce.ll

19

This is different from what I suggested (increasing the number of elements), but the result of increasing the element-width is the same for the cost function, so I guess it's okay.

llvm/include/llvm/CodeGen/BasicTTIImpl.h

unsigned getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
     case Intrinsic::vector_reduce_or:
     case Intrinsic::vector_reduce_xor:
     case Intrinsic::vector_reduce_smax:
     case Intrinsic::vector_reduce_smin:
     case Intrinsic::vector_reduce_fmax:
     case Intrinsic::vector_reduce_fmin:
     case Intrinsic::vector_reduce_umax:
     case Intrinsic::vector_reduce_umin: {
-      if (isa<ScalableVectorType>(RetTy))
-        return BaseT::getIntrinsicInstrCost(ICA, CostKind);
       IntrinsicCostAttributes Attrs(IID, RetTy, Args[0]->getType(), FMF, 1, I);
       return getTypeBasedIntrinsicInstrCost(Attrs, CostKind);
     }
     case Intrinsic::vector_reduce_fadd:
     case Intrinsic::vector_reduce_fmul: {
-      if (isa<ScalableVectorType>(RetTy))
-        return BaseT::getIntrinsicInstrCost(ICA, CostKind);
       IntrinsicCostAttributes Attrs(
           IID, RetTy, {Args[0]->getType(), Args[1]->getType()}, FMF, 1, I);
       return getTypeBasedIntrinsicInstrCost(Attrs, CostKind);
     }
     case Intrinsic::fshl:
     case Intrinsic::fshr: {
       if (isa<ScalableVectorType>(RetTy))
         return BaseT::getIntrinsicInstrCost(ICA, CostKind);

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

public:
 
   int getExtractWithExtendCost(unsigned Opcode, Type *Dst, VectorType *VecTy,
                                unsigned Index);
 
   unsigned getCFInstrCost(unsigned Opcode, TTI::TargetCostKind CostKind);
 
   int getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index);
 
+  int getMinMaxReductionCost(VectorType *Ty, VectorType *CondTy,
+                             bool IsPairwise, bool IsUnsigned,
+                             TTI::TargetCostKind CostKind);
+
+  int getArithmeticReductionCostSVE(unsigned Opcode, VectorType *ValTy,
+                                    bool IsPairwiseForm,
+                                    TTI::TargetCostKind CostKind);
+
   int getArithmeticInstrCost(
       unsigned Opcode, Type *Ty,
       TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput,
       TTI::OperandValueKind Opd1Info = TTI::OK_AnyValue,
       TTI::OperandValueKind Opd2Info = TTI::OK_AnyValue,
       TTI::OperandValueProperties Opd1PropInfo = TTI::OP_None,
       TTI::OperandValueProperties Opd2PropInfo = TTI::OP_None,
       ArrayRef<const Value *> Args = ArrayRef<const Value *>(),

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

bool AArch64TTIImpl::useReductionIntrinsic(unsigned Opcode, Type *Ty,
   case Instruction::FCmp:
     return Flags.NoNaN;
   default:
     llvm_unreachable("Unhandled reduction opcode");
   }
   return false;
 }
 
+int AArch64TTIImpl::getMinMaxReductionCost(VectorType *Ty, VectorType *CondTy,
+                                           bool IsPairwise, bool IsUnsigned,
+                                           TTI::TargetCostKind CostKind) {
+  if (!isa<ScalableVectorType>(Ty))
+    return BaseT::getMinMaxReductionCost(Ty, CondTy, IsPairwise, IsUnsigned,
+                                         CostKind);
+  assert((isa<ScalableVectorType>(Ty) && isa<ScalableVectorType>(CondTy)) &&
+         "Both vector needs to be scalable");
+
+  std::pair<int, MVT> LT = TLI->getTypeLegalizationCost(DL, Ty);
+  int LegalizationCost = 0;
+  if (LT.first > 1) {
+    Type *LegalVTy = EVT(LT.second).getTypeForEVT(Ty->getContext());
+    unsigned CmpOpcode =
+        Ty->isFPOrFPVectorTy() ? Instruction::FCmp : Instruction::ICmp;
+    LegalizationCost =
+        getCmpSelInstrCost(CmpOpcode, LegalVTy, LegalVTy,
+                           CmpInst::BAD_ICMP_PREDICATE, CostKind) +
+        getCmpSelInstrCost(Instruction::Select, LegalVTy, LegalVTy,
+                           CmpInst::BAD_ICMP_PREDICATE, CostKind);
+    LegalizationCost *= LT.first - 1;
+  }
+
+  return LegalizationCost + /*Cost of horizontal reduction*/ 2;
+}
+
+int AArch64TTIImpl::getArithmeticReductionCostSVE(
+    unsigned Opcode, VectorType *ValTy, bool IsPairwise,
+    TTI::TargetCostKind CostKind) {
+  assert(!IsPairwise && "Cannot be pair wise to continue");
+
+  std::pair<int, MVT> LT = TLI->getTypeLegalizationCost(DL, ValTy);
+  int LegalizationCost = 0;
+  if (LT.first > 1) {
+    Type *LegalVTy = EVT(LT.second).getTypeForEVT(ValTy->getContext());
+    LegalizationCost = getArithmeticInstrCost(Opcode, LegalVTy, CostKind);
+    LegalizationCost *= LT.first - 1;
+  }
+
+  int ISD = TLI->InstructionOpcodeToISD(Opcode);
+  assert(ISD && "Invalid opcode");
+  // Add the final reduction cost for the legal horizontal reduction
+  switch (ISD) {
+  case ISD::ADD:
+  case ISD::AND:
+  case ISD::OR:
+  case ISD::XOR:
+  case ISD::FADD:
+    return LegalizationCost + 2;
+  default:
+    // TODO: Replace for invalid when InstructionCost is used
+    // cases not supported by SVE
+    return 16;
+  }
+}
+
 int AArch64TTIImpl::getArithmeticReductionCost(unsigned Opcode,
                                                VectorType *ValTy,
                                                bool IsPairwiseForm,
                                                TTI::TargetCostKind CostKind) {
 
+  if (isa<ScalableVectorType>(ValTy))
+    return getArithmeticReductionCostSVE(Opcode, ValTy, IsPairwiseForm,
+                                         CostKind);
   if (IsPairwiseForm)
     return BaseT::getArithmeticReductionCost(Opcode, ValTy, IsPairwiseForm,
                                              CostKind);
 
   std::pair<int, MVT> LT = TLI->getTypeLegalizationCost(DL, ValTy);
   MVT MTy = LT.second;
   int ISD = TLI->InstructionOpcodeToISD(Opcode);
   assert(ISD && "Invalid opcode");
 
   // Horizontal adds can use the 'addv' instruction. We model the cost of these
   // instructions as normal vector adds. This is the only arithmetic vector
   // reduction operation for which we have an instruction.
   static const CostTblEntry CostTblNoPairwise[]{
       {ISD::ADD, MVT::v8i8,  1},
       {ISD::ADD, MVT::v16i8, 1},
       {ISD::ADD, MVT::v4i16, 1},
       {ISD::ADD, MVT::v8i16, 1},
       {ISD::ADD, MVT::v4i32, 1},
   };
 
   if (const auto *Entry = CostTableLookup(CostTblNoPairwise, ISD, MTy))
     return LT.first * Entry->Cost;
 
   return BaseT::getArithmeticReductionCost(Opcode, ValTy, IsPairwiseForm,
                                            CostKind);
 }
 
 int AArch64TTIImpl::getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp,
                                    int Index, VectorType *SubTp) {
   if (Kind == TTI::SK_Broadcast || Kind == TTI::SK_Transpose ||

llvm/test/Analysis/CostModel/AArch64/sve-getIntrinsicInstrCost-vector-reduce.ll

+; Check getIntrinsicInstrCost in BasicTTIImpl.h with SVE for vector.reduce.<operand>
+; Checks legal and not legal vector size
+
+; RUN: opt -cost-model -analyze -mtriple=aarch64--linux-gnu -mattr=+sve  < %s 2>%t | FileCheck %s
+
+
+; If this check fails please read test/CodeGen/AArch64/README for instructions on how to resolve it.
+; WARN-NOT: warning
+
+define i32 @add.i32.nxv4i32(<vscale x 4 x i32> %v) {
+; CHECK-LABEL: 'add.i32.nxv4i32'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %r = call i32 @llvm.vector.reduce.add.nxv4i32(<vscale x 4 x i32> %v)
+; CHECK-NEXT:Cost Model: Found an estimated cost of 0 for instruction:   ret i32 %r
+
+  %r = call i32 @llvm.vector.reduce.add.nxv4i32(<vscale x 4 x i32> %v)
+  ret i32 %r
+}
+
+define i64 @add.i64.nxv4i64(<vscale x 4 x i64> %v) {
+; CHECK-LABEL: 'add.i64.nxv4i64'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 3 for instruction:   %r = call i64 @llvm.vector.reduce.add.nxv4i64(<vscale x 4 x i64> %v)
+; CHECK-NEXT:Cost Model: Found an estimated cost of 0 for instruction:   ret i64 %r
+
+  %r = call i64 @llvm.vector.reduce.add.nxv4i64(<vscale x 4 x i64> %v)
+  ret i64 %r
+}
+
+define i32 @mul.i32.nxv4i32(<vscale x 4 x i32> %v) {
+; CHECK-LABEL: 'mul.i32.nxv4i32'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 16 for instruction:   %r = call i32 @llvm.vector.reduce.mul.nxv4i32(<vscale x 4 x i32> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i32 %r
+
+  %r = call i32 @llvm.vector.reduce.mul.nxv4i32(<vscale x 4 x i32> %v)
+  ret i32 %r
+}
+
+define i64 @mul.i64.nxv4i64(<vscale x 4 x i64> %v) {
+; CHECK-LABEL: 'mul.i64.nxv4i64'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 16 for instruction:   %r = call i64 @llvm.vector.reduce.mul.nxv4i64(<vscale x 4 x i64> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i64 %r
+
+  %r = call i64 @llvm.vector.reduce.mul.nxv4i64(<vscale x 4 x i64> %v)
+  ret i64 %r
+}
+
+define i32 @and.i32.nxv4i32(<vscale x 4 x i32> %v) {
+; CHECK-LABEL: 'and.i32.nxv4i32'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %r = call i32 @llvm.vector.reduce.and.nxv4i32(<vscale x 4 x i32> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i32 %r
+
+  %r = call i32 @llvm.vector.reduce.and.nxv4i32(<vscale x 4 x i32> %v)
+  ret i32 %r
+}
+
+define i64 @and.i64.nxv4i64(<vscale x 4 x i64> %v) {
+; CHECK-LABEL: 'and.i64.nxv4i64'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 3 for instruction:   %r = call i64 @llvm.vector.reduce.and.nxv4i64(<vscale x 4 x i64> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i64 %r
+
+  %r = call i64 @llvm.vector.reduce.and.nxv4i64(<vscale x 4 x i64> %v)
+  ret i64 %r
+}
+
+define i32 @or.i32.nxv4i32(<vscale x 4 x i32> %v) {
+; CHECK-LABEL: 'or.i32.nxv4i32'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %r = call i32 @llvm.vector.reduce.or.nxv4i32(<vscale x 4 x i32> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i32 %r
+
+  %r = call i32 @llvm.vector.reduce.or.nxv4i32(<vscale x 4 x i32> %v)
+  ret i32 %r
+}
+
+define i64 @or.i64.nxv4i64(<vscale x 4 x i64> %v) {
+; CHECK-LABEL: 'or.i64.nxv4i64'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 3 for instruction:   %r = call i64 @llvm.vector.reduce.or.nxv4i64(<vscale x 4 x i64> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i64 %r
+
+  %r = call i64 @llvm.vector.reduce.or.nxv4i64(<vscale x 4 x i64> %v)
+  ret i64 %r
+}
+
+define i32 @xor.i32.nxv4i32(<vscale x 4 x i32> %v) {
+; CHECK-LABEL: 'xor.i32.nxv4i32'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %r = call i32 @llvm.vector.reduce.xor.nxv4i32(<vscale x 4 x i32> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i32 %r
+
+  %r = call i32 @llvm.vector.reduce.xor.nxv4i32(<vscale x 4 x i32> %v)
+  ret i32 %r
+}
+
+define i64 @xor.i64.nxv4i64(<vscale x 4 x i64> %v) {
+; CHECK-LABEL: 'xor.i64.nxv4i64'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 3 for instruction:   %r = call i64 @llvm.vector.reduce.xor.nxv4i64(<vscale x 4 x i64> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i64 %r
+
+  %r = call i64 @llvm.vector.reduce.xor.nxv4i64(<vscale x 4 x i64> %v)
+  ret i64 %r
+}
+
+define i32 @umin.i32.nxv4i32(<vscale x 4 x i32> %v) {
+; CHECK-LABEL: 'umin.i32.nxv4i32'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %r = call i32 @llvm.vector.reduce.umin.nxv4i32(<vscale x 4 x i32> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i32 %r
+
+  %r = call i32 @llvm.vector.reduce.umin.nxv4i32(<vscale x 4 x i32> %v)
+  ret i32 %r
+}
+
+define i64 @umin.i64.nxv4i64(<vscale x 4 x i64> %v) {
+; CHECK-LABEL: 'umin.i64.nxv4i64'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 4 for instruction:   %r = call i64 @llvm.vector.reduce.umin.nxv4i64(<vscale x 4 x i64> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i64 %r
+
+  %r = call i64 @llvm.vector.reduce.umin.nxv4i64(<vscale x 4 x i64> %v)
+  ret i64 %r
+}
+
+define float @fmax.f32.nxv4f32(<vscale x 4 x float> %v) {
+; CHECK-LABEL: 'fmax.f32.nxv4f32'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %r = call float @llvm.vector.reduce.fmax.nxv4f32(<vscale x 4 x float> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret float %r
+
+  %r = call float @llvm.vector.reduce.fmax.nxv4f32(<vscale x 4 x float> %v)
+  ret float %r
+}
+
+define double @fmax.f64.nxv4f64(<vscale x 4 x double> %v) {
+; CHECK-LABEL: 'fmax.f64.nxv4f64'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 4 for instruction:   %r = call double @llvm.vector.reduce.fmax.nxv4f64(<vscale x 4 x double> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret double %r
+
+  %r = call double @llvm.vector.reduce.fmax.nxv4f64(<vscale x 4 x double> %v)
+  ret double %r
+}
+
+define float @fmin.f32.nxv4f32(<vscale x 4 x float> %v) {
+; CHECK-LABEL: 'fmin.f32.nxv4f32'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %r = call float @llvm.vector.reduce.fmin.nxv4f32(<vscale x 4 x float> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret float %r
+
+  %r = call float @llvm.vector.reduce.fmin.nxv4f32(<vscale x 4 x float> %v)
+  ret float %r
+}
+
+define double @fmin.f64.nxv4f64(<vscale x 4 x double> %v) {
+; CHECK-LABEL: 'fmin.f64.nxv4f64'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 4 for instruction:   %r = call double @llvm.vector.reduce.fmin.nxv4f64(<vscale x 4 x double> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret double %r
+
+  %r = call double @llvm.vector.reduce.fmin.nxv4f64(<vscale x 4 x double> %v)
+  ret double %r
+}
+
+define i32 @umax.i32.nxv4i32(<vscale x 4 x i32> %v) {
+; CHECK-LABEL: 'umax.i32.nxv4i32'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %r = call i32 @llvm.vector.reduce.umax.nxv4i32(<vscale x 4 x i32> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i32 %r
+
+  %r = call i32 @llvm.vector.reduce.umax.nxv4i32(<vscale x 4 x i32> %v)
+  ret i32 %r
+}
+
+define i64 @umax.i64.nxv4i64(<vscale x 4 x i64> %v) {
+; CHECK-LABEL: 'umax.i64.nxv4i64'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 4 for instruction:   %r = call i64 @llvm.vector.reduce.umax.nxv4i64(<vscale x 4 x i64> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i64 %r
+
+  %r = call i64 @llvm.vector.reduce.umax.nxv4i64(<vscale x 4 x i64> %v)
+  ret i64 %r
+}
+
+define i32 @smin.i32.nxv4i32(<vscale x 4 x i32> %v) {
+; CHECK-LABEL: 'smin.i32.nxv4i32'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %r = call i32 @llvm.vector.reduce.smin.nxv4i32(<vscale x 4 x i32> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i32 %r
+
+  %r = call i32 @llvm.vector.reduce.smin.nxv4i32(<vscale x 4 x i32> %v)
+  ret i32 %r
+}
+
+define i64 @smin.i64.nxv4i64(<vscale x 4 x i64> %v) {
+; CHECK-LABEL: 'smin.i64.nxv4i64'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 4 for instruction:   %r = call i64 @llvm.vector.reduce.smin.nxv4i64(<vscale x 4 x i64> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i64 %r
+
+  %r = call i64 @llvm.vector.reduce.smin.nxv4i64(<vscale x 4 x i64> %v)
+  ret i64 %r
+}
+
+define i32 @smax.i32.nxv4i32(<vscale x 4 x i32> %v) {
+; CHECK-LABEL: 'smax.i32.nxv4i32'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %r = call i32 @llvm.vector.reduce.smax.nxv4i32(<vscale x 4 x i32> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i32 %r
+
+  %r = call i32 @llvm.vector.reduce.smax.nxv4i32(<vscale x 4 x i32> %v)
+  ret i32 %r
+}
+
+define i64 @smax.i64.nxv4i64(<vscale x 4 x i64> %v) {
+; CHECK-LABEL: 'smax.i64.nxv4i64'
+; CHECK-NEXT: Cost Model: Found an estimated cost of 4 for instruction:   %r = call i64 @llvm.vector.reduce.smax.nxv4i64(<vscale x 4 x i64> %v)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret i64 %r
+
+  %r = call i64 @llvm.vector.reduce.smax.nxv4i64(<vscale x 4 x i64> %v)
+  ret i64 %r
+}
+
+define float @fadda_nxv4f32(float %start, <vscale x 4 x float> %a) #0 {
+; CHECK-LABEL: 'fadda_nxv4f32
+; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %res = call float @llvm.vector.reduce.fadd.nxv4f32(float %start, <vscale x 4 x float> %a)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret float %res
+
+  %res = call float @llvm.vector.reduce.fadd.nxv4f32(float %start, <vscale x 4 x float> %a)
+  ret float %res
+}
+
+define double @fadda_nxv4f64(double %start, <vscale x 4 x double> %a) #0 {
+; CHECK-LABEL: 'fadda_nxv4f64
+; CHECK-NEXT: Cost Model: Found an estimated cost of 6 for instruction:   %res = call double @llvm.vector.reduce.fadd.nxv4f64(double %start, <vscale x 4 x double> %a)
+; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction:   ret double %res
+
+  %res = call double @llvm.vector.reduce.fadd.nxv4f64(double %start, <vscale x 4 x double> %a)
+  ret double %res
+}
+
+
+declare i32 @llvm.vector.reduce.add.nxv4i32(<vscale x 4 x i32>)
+declare i32 @llvm.vector.reduce.mul.nxv4i32(<vscale x 4 x i32>)
+declare i32 @llvm.vector.reduce.and.nxv4i32(<vscale x 4 x i32>)
+declare i32 @llvm.vector.reduce.or.nxv4i32(<vscale x 4 x i32>)
+declare i32 @llvm.vector.reduce.xor.nxv4i32(<vscale x 4 x i32>)
+declare float @llvm.vector.reduce.fmax.nxv4f32(<vscale x 4 x float>)
+declare float @llvm.vector.reduce.fmin.nxv4f32(<vscale x 4 x float>)
+declare i32 @llvm.vector.reduce.fmin.nxv4i32(<vscale x 4 x i32>)
+declare i32 @llvm.vector.reduce.umin.nxv4i32(<vscale x 4 x i32>)
+declare i32 @llvm.vector.reduce.umax.nxv4i32(<vscale x 4 x i32>)
+declare i32 @llvm.vector.reduce.smin.nxv4i32(<vscale x 4 x i32>)
+declare i32 @llvm.vector.reduce.smax.nxv4i32(<vscale x 4 x i32>)
+declare float @llvm.vector.reduce.fadd.nxv4f32(float, <vscale x 4 x float>)
+declare i64 @llvm.vector.reduce.add.nxv4i64(<vscale x 4 x i64>)
+declare i64 @llvm.vector.reduce.mul.nxv4i64(<vscale x 4 x i64>)
+declare i64 @llvm.vector.reduce.and.nxv4i64(<vscale x 4 x i64>)
+declare i64 @llvm.vector.reduce.or.nxv4i64(<vscale x 4 x i64>)
+declare i64 @llvm.vector.reduce.xor.nxv4i64(<vscale x 4 x i64>)
+declare double @llvm.vector.reduce.fmax.nxv4f64(<vscale x 4 x double>)
+declare double @llvm.vector.reduce.fmin.nxv4f64(<vscale x 4 x double>)
+declare i64 @llvm.vector.reduce.umin.nxv4i64(<vscale x 4 x i64>)
+declare i64 @llvm.vector.reduce.umax.nxv4i64(<vscale x 4 x i64>)
+declare i64 @llvm.vector.reduce.smin.nxv4i64(<vscale x 4 x i64>)
+declare i64 @llvm.vector.reduce.smax.nxv4i64(<vscale x 4 x i64>)
+declare double @llvm.vector.reduce.fadd.nxv4f64(double, <vscale x 4 x double>)