[AArch64] Implement getIntrinsicInstrCost, handle min/max intrinsics.

[fhahn]

Summary

This patch adds a specialized implementation of getIntrinsicInstrCost
and add initial cost-modeling for min/max vector intrinsics.

AArch64 NEON support umin/smin/umax/smax for vectors
<8 x i8>, <16 x i8>, <4 x i16>, <8 x i16>, <2 x i32> and <4 x i32>.
Notably, it does not support vectors with i64 elements.

This change by itself should have very little impact on codegen, but in
follow-up patches I plan to teach the vectorizers to consider using
those intrinsics on platforms where it is profitable, e.g. because there
is no general 'select'-like instruction.

The current cost returned should be better for throughput, latency and size.

[dmgreen]

Yeah, this sounds useful. I know we've hit problems with min/max costs in the past (the cmp/select kind, I don't know if they were float). It would be good to see them costed more accurately.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

222

v4i16?

224

use auto instead? It's common to use CostTableLookup too, but I would guess that makes this more verbose?

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

100

What does this using do?

[fhahn]

Add missing v4i16, use auto, remove unncessary using.

[fhahn]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

222

Yeah I missed that one initially. Should be fixed now.

224

I think CostTableLookup is overkill, because at the moment this just uses the same cost for each supported type. Updated to use auto.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

100

IIRC there used to be some compilers that had trouble with calling BaseT::getIntrinsicInstrCost without this, but it builds fine on my system without it. Let's see if any bot complains.

[dmgreen]

I presume float versions would be useful too? Do you plan to add them?

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

222

Look like it would be worth making sure there are tests too.

224

Yeah I agree. Identical costs between sizes and opcodes don't make it very useful here.

[fhahn]

Add/update tests for v4i16, v2i32.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

222

Agreed, added tests for the remaining missing cases (v2i32, v4i16).

[dmgreen]

Thanks. LGTM

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

unsigned getNumberOfRegisters(unsigned ClassID) const {
     if (Vector) {
       if (ST->hasNEON())
         return 32;
       return 0;
     }
     return 31;
   }
 
+  unsigned getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
+                                 TTI::TargetCostKind CostKind);
+
   unsigned getRegisterBitWidth(bool Vector) const {
     if (Vector) {
       if (ST->hasSVE())
         return std::max(ST->getMinSVEVectorSizeInBits(), 128u);
       if (ST->hasNEON())
         return 128;
       return 0;
     }

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

 AArch64TTIImpl::getPopcntSupport(unsigned TyWidth) {
   assert(isPowerOf2_32(TyWidth) && "Ty width must be power of 2");
   if (TyWidth == 32 || TyWidth == 64)
     return TTI::PSK_FastHardware;
   // TODO: AArch64TargetLowering::LowerCTPOP() supports 128bit popcount.
   return TTI::PSK_Software;
 }
 
+unsigned
+AArch64TTIImpl::getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
+                                      TTI::TargetCostKind CostKind) {
+  auto *RetTy = ICA.getReturnType();
+  switch (ICA.getID()) {
+  case Intrinsic::smin:
+  case Intrinsic::umin:
+  case Intrinsic::smax:
+  case Intrinsic::umax: {
+    static const auto ValidMinMaxTys = {MVT::v8i8,  MVT::v16i8, MVT::v4i16,
+                                        MVT::v8i16, MVT::v2i32, MVT::v4i32};
+    auto LT = TLI->getTypeLegalizationCost(DL, RetTy);
+    if (any_of(ValidMinMaxTys, [&LT](MVT M) { return M == LT.second; }))
+      return LT.first;
+    break;
+  }
+  default:
+    break;
+  }
+  return BaseT::getIntrinsicInstrCost(ICA, CostKind);
+}
+
 bool AArch64TTIImpl::isWideningInstruction(Type *DstTy, unsigned Opcode,
                                            ArrayRef<const Value *> Args) {
 
   // A helper that returns a vector type from the given type. The number of
   // elements in type Ty determine the vector width.
   auto toVectorTy = [&](Type *ArgTy) {
     return FixedVectorType::get(ArgTy->getScalarType(),
                                 cast<FixedVectorType>(DstTy)->getNumElements());

llvm/test/Analysis/CostModel/AArch64/min-max.ll

 ; RUN: opt < %s -mtriple=aarch64--linux-gnu -cost-model -analyze | FileCheck %s --check-prefix=COST
 ; RUN: llc < %s -mtriple=aarch64--linux-gnu | FileCheck %s --check-prefix=CODE
 
 ; COST-LABEL: umin.v8i8
-; COST-NEXT: Cost Model: Found an estimated cost of 30 for instruction:   %res = call <8 x i8> @llvm.umin.v8i8(<8 x i8> %v0, <8 x i8> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <8 x i8> @llvm.umin.v8i8(<8 x i8> %v0, <8 x i8> %v1)
 
 ; CODE-LABEL: umin.v8i8
 ; CODE:       bb.0
 ; CODE-NEXT:   umin v{{.*}}.8b, v{{.*}}.8b, v{{.*}}.8b
 ; CODE-NEXT:   ret
 
 declare <8 x i8> @llvm.umin.v8i8(<8 x i8>, <8 x i8>)
 define <8 x i8> @umin.v8i8(<8 x i8> %v0, <8 x i8> %v1) {
   %res = call <8 x i8> @llvm.umin.v8i8(<8 x i8> %v0, <8 x i8> %v1)
   ret <8 x i8> %res
 }
 
 ; COST-LABEL: umin.v9i8
-; COST-NEXT: Cost Model: Found an estimated cost of 34 for instruction:   %res = call <9 x i8> @llvm.umin.v9i8(<9 x i8> %v0, <9 x i8> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <9 x i8> @llvm.umin.v9i8(<9 x i8> %v0, <9 x i8> %v1)
 
 ; CODE-LABEL: umin.v9i8
 ; CODE:       bb.0
 ; CODE-NEXT:   umin v{{.*}}.16b, v{{.*}}.16b, v{{.*}}.16b
 ; CODE-NEXT:   ret
 
 declare <9 x i8> @llvm.umin.v9i8(<9 x i8>, <9 x i8>)
 define <9 x i8> @umin.v9i8(<9 x i8> %v0, <9 x i8> %v1) {
   %res = call <9 x i8> @llvm.umin.v9i8(<9 x i8> %v0, <9 x i8> %v1)
   ret <9 x i8> %res
 }
 
 ; COST-LABEL: umin.v4i16
-; COST-NEXT: Cost Model: Found an estimated cost of 14 for instruction:   %res = call <4 x i16> @llvm.umin.v4i16(<4 x i16> %v0, <4 x i16> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <4 x i16> @llvm.umin.v4i16(<4 x i16> %v0, <4 x i16> %v1)
 
 ; CODE-LABEL: umin.v4i16
 ; CODE:       bb.0
 ; CODE-NEXT:   umin v{{.*}}.4h, v{{.*}}.4h, v{{.*}}.4h
 ; CODE-NEXT:   ret
 
 declare <4 x i16> @llvm.umin.v4i16(<4 x i16>, <4 x i16>)
 define <4 x i16> @umin.v4i16(<4 x i16> %v0, <4 x i16> %v1) {
   %res = call <4 x i16> @llvm.umin.v4i16(<4 x i16> %v0, <4 x i16> %v1)
   ret <4 x i16> %res
 }
 
 ; COST-LABEL: umin.v16i8
-; COST-NEXT: Cost Model: Found an estimated cost of 62 for instruction:   %res = call <16 x i8> @llvm.umin.v16i8(<16 x i8> %v0, <16 x i8> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <16 x i8> @llvm.umin.v16i8(<16 x i8> %v0, <16 x i8> %v1)
 
 ; CODE-LABEL: umin.v16i8
 ; CODE:       bb.0
 ; CODE-NEXT:   umin v{{.*}}.16b, v{{.*}}.16b, v{{.*}}.16b
 ; CODE-NEXT:   ret
 
 declare <16 x i8> @llvm.umin.v16i8(<16 x i8>, <16 x i8>)
 define <16 x i8> @umin.v16i8(<16 x i8> %v0, <16 x i8> %v1) {
   %res = call <16 x i8> @llvm.umin.v16i8(<16 x i8> %v0, <16 x i8> %v1)
   ret <16 x i8> %res
 }
 
 ; COST-LABEL: umin.v8i16
-; COST-NEXT: Cost Model: Found an estimated cost of 30 for instruction:   %res = call <8 x i16> @llvm.umin.v8i16(<8 x i16> %v0, <8 x i16> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <8 x i16> @llvm.umin.v8i16(<8 x i16> %v0, <8 x i16> %v1)
 
 ; CODE-LABEL: umin.v8i16
 ; CODE:       bb.0
 ; CODE-NEXT:   umin v{{.*}}.8h, v{{.*}}.8h, v{{.*}}.8h
 ; CODE-NEXT:   ret
 
 declare <8 x i16> @llvm.umin.v8i16(<8 x i16>, <8 x i16>)
 define <8 x i16> @umin.v8i16(<8 x i16> %v0, <8 x i16> %v1) {
   %res = call <8 x i16> @llvm.umin.v8i16(<8 x i16> %v0, <8 x i16> %v1)
   ret <8 x i16> %res
 }
 
 ; COST-LABEL: umin.v2i32
-; COST-NEXT: Cost Model: Found an estimated cost of 6 for instruction:   %res = call <2 x i32> @llvm.umin.v2i32(<2 x i32> %v0, <2 x i32> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <2 x i32> @llvm.umin.v2i32(<2 x i32> %v0, <2 x i32> %v1)
 
 ; CODE-LABEL: umin.v2i32
 ; CODE:       bb.0
 ; CODE-NEXT:   umin v{{.*}}.2s, v{{.*}}.2s, v{{.*}}.2s
 ; CODE-NEXT:   ret
 
 declare <2 x i32> @llvm.umin.v2i32(<2 x i32>, <2 x i32>)
 define <2 x i32> @umin.v2i32(<2 x i32> %v0, <2 x i32> %v1) {
   %res = call <2 x i32> @llvm.umin.v2i32(<2 x i32> %v0, <2 x i32> %v1)
   ret <2 x i32> %res
 }
 
 ; COST-LABEL: umin.v4i32
-; COST-NEXT: Cost Model: Found an estimated cost of 14 for instruction:   %res = call <4 x i32> @llvm.umin.v4i32(<4 x i32> %v0, <4 x i32> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <4 x i32> @llvm.umin.v4i32(<4 x i32> %v0, <4 x i32> %v1)
 
 ; CODE-LABEL: umin.v4i32
 ; CODE:       bb.0
 ; CODE-NEXT:   umin v{{.*}}.4s, v{{.*}}.4s, v{{.*}}.4s
 ; CODE-NEXT:   ret
 
 declare <4 x i32> @llvm.umin.v4i32(<4 x i32>, <4 x i32>)
 define <4 x i32> @umin.v4i32(<4 x i32> %v0, <4 x i32> %v1) {
   %res = call <4 x i32> @llvm.umin.v4i32(<4 x i32> %v0, <4 x i32> %v1)
   ret <4 x i32> %res
 }
 
 ; COST-LABEL: umin.v8i32
-; COST-NEXT: Cost Model: Found an estimated cost of 10 for instruction:   %res = call <8 x i32> @llvm.umin.v8i32(<8 x i32> %v0, <8 x i32> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %res = call <8 x i32> @llvm.umin.v8i32(<8 x i32> %v0, <8 x i32> %v1)
 
 ; CODE-LABEL: umin.v8i32
 ; CODE:       bb.0
 ; CODE-NEXT:   umin v{{.*}}.4s, v{{.*}}.4s, v{{.*}}.4s
 ; CODE-NEXT:   umin v{{.*}}.4s, v{{.*}}.4s, v{{.*}}.4s
 ; CODE-NEXT:   ret
 
 declare <8 x i32> @llvm.umin.v8i32(<8 x i32>, <8 x i32>)

 
 declare <2 x i64> @llvm.umin.v2i64(<2 x i64>, <2 x i64>)
 define <2 x i64> @umin.v2i64(<2 x i64> %v0, <2 x i64> %v1) {
   %res = call <2 x i64> @llvm.umin.v2i64(<2 x i64> %v0, <2 x i64> %v1)
   ret <2 x i64> %res
 }
 
 ; COST-LABEL: smin.v8i8
-; COST-NEXT: Cost Model: Found an estimated cost of 30 for instruction:   %res = call <8 x i8> @llvm.smin.v8i8(<8 x i8> %v0, <8 x i8> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <8 x i8> @llvm.smin.v8i8(<8 x i8> %v0, <8 x i8> %v1)
 
 ; CODE-LABEL: smin.v8i8
 ; CODE:       bb.0
 ; CODE-NEXT:   smin v{{.*}}.8b, v{{.*}}.8b, v{{.*}}.8b
 ; CODE-NEXT:   ret
 
 declare <8 x i8> @llvm.smin.v8i8(<8 x i8>, <8 x i8>)
 define <8 x i8> @smin.v8i8(<8 x i8> %v0, <8 x i8> %v1) {
   %res = call <8 x i8> @llvm.smin.v8i8(<8 x i8> %v0, <8 x i8> %v1)
   ret <8 x i8> %res
 }
 
 ; COST-LABEL: smin.v9i8
-; COST-NEXT: Cost Model: Found an estimated cost of 34 for instruction:   %res = call <9 x i8> @llvm.smin.v9i8(<9 x i8> %v0, <9 x i8> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <9 x i8> @llvm.smin.v9i8(<9 x i8> %v0, <9 x i8> %v1)
 
 ; CODE-LABEL: smin.v9i8
 ; CODE:       bb.0
 ; CODE-NEXT:   smin v{{.*}}.16b, v{{.*}}.16b, v{{.*}}.16b
 ; CODE-NEXT:   ret
 
 declare <9 x i8> @llvm.smin.v9i8(<9 x i8>, <9 x i8>)
 define <9 x i8> @smin.v9i8(<9 x i8> %v0, <9 x i8> %v1) {
   %res = call <9 x i8> @llvm.smin.v9i8(<9 x i8> %v0, <9 x i8> %v1)
   ret <9 x i8> %res
 }
 
 ; COST-LABEL: smin.v16i8
-; COST-NEXT: Cost Model: Found an estimated cost of 62 for instruction:   %res = call <16 x i8> @llvm.smin.v16i8(<16 x i8> %v0, <16 x i8> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <16 x i8> @llvm.smin.v16i8(<16 x i8> %v0, <16 x i8> %v1)
 
 ; CODE-LABEL: smin.v16i8
 ; CODE:       bb.0
 ; CODE-NEXT:   smin v{{.*}}.16b, v{{.*}}.16b, v{{.*}}.16b
 ; CODE-NEXT:   ret
 
 declare <16 x i8> @llvm.smin.v16i8(<16 x i8>, <16 x i8>)
 define <16 x i8> @smin.v16i8(<16 x i8> %v0, <16 x i8> %v1) {
   %res = call <16 x i8> @llvm.smin.v16i8(<16 x i8> %v0, <16 x i8> %v1)
   ret <16 x i8> %res
 }
 
 ; COST-LABEL: smin.v4i16
-; COST-NEXT: Cost Model: Found an estimated cost of 14 for instruction:   %res = call <4 x i16> @llvm.smin.v4i16(<4 x i16> %v0, <4 x i16> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <4 x i16> @llvm.smin.v4i16(<4 x i16> %v0, <4 x i16> %v1)
 
 ; CODE-LABEL: smin.v4i16
 ; CODE:       bb.0
 ; CODE-NEXT:   smin v{{.*}}.4h, v{{.*}}.4h, v{{.*}}.4h
 ; CODE-NEXT:   ret
 
 declare <4 x i16> @llvm.smin.v4i16(<4 x i16>, <4 x i16>)
 define <4 x i16> @smin.v4i16(<4 x i16> %v0, <4 x i16> %v1) {
   %res = call <4 x i16> @llvm.smin.v4i16(<4 x i16> %v0, <4 x i16> %v1)
   ret <4 x i16> %res
 }
 
 ; COST-LABEL: smin.v8i16
-; COST-NEXT: Cost Model: Found an estimated cost of 30 for instruction:   %res = call <8 x i16> @llvm.smin.v8i16(<8 x i16> %v0, <8 x i16> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <8 x i16> @llvm.smin.v8i16(<8 x i16> %v0, <8 x i16> %v1)
 
 ; CODE-LABEL: smin.v8i16
 ; CODE:       bb.0
 ; CODE-NEXT:   smin v{{.*}}.8h, v{{.*}}.8h, v{{.*}}.8h
 ; CODE-NEXT:   ret
 
 declare <8 x i16> @llvm.smin.v8i16(<8 x i16>, <8 x i16>)
 define <8 x i16> @smin.v8i16(<8 x i16> %v0, <8 x i16> %v1) {
   %res = call <8 x i16> @llvm.smin.v8i16(<8 x i16> %v0, <8 x i16> %v1)
   ret <8 x i16> %res
 }
 
 ; COST-LABEL: smin.v2i32
-; COST-NEXT: Cost Model: Found an estimated cost of 6 for instruction:   %res = call <2 x i32> @llvm.smin.v2i32(<2 x i32> %v0, <2 x i32> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <2 x i32> @llvm.smin.v2i32(<2 x i32> %v0, <2 x i32> %v1)
 
 ; CODE-LABEL: smin.v2i32
 ; CODE:       bb.0
 ; CODE-NEXT:   smin v{{.*}}.2s, v{{.*}}.2s, v{{.*}}.2s
 ; CODE-NEXT:   ret
 
 declare <2 x i32> @llvm.smin.v2i32(<2 x i32>, <2 x i32>)
 define <2 x i32> @smin.v2i32(<2 x i32> %v0, <2 x i32> %v1) {
   %res = call <2 x i32> @llvm.smin.v2i32(<2 x i32> %v0, <2 x i32> %v1)
   ret <2 x i32> %res
 }
 
 ; COST-LABEL: smin.v4i32
-; COST-NEXT: Cost Model: Found an estimated cost of 14 for instruction:   %res = call <4 x i32> @llvm.smin.v4i32(<4 x i32> %v0, <4 x i32> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <4 x i32> @llvm.smin.v4i32(<4 x i32> %v0, <4 x i32> %v1)
 
 ; CODE-LABEL: smin.v4i32
 ; CODE:       bb.0
 ; CODE-NEXT:   smin v{{.*}}.4s, v{{.*}}.4s, v{{.*}}.4s
 ; CODE-NEXT:   ret
 
 declare <4 x i32> @llvm.smin.v4i32(<4 x i32>, <4 x i32>)
 define <4 x i32> @smin.v4i32(<4 x i32> %v0, <4 x i32> %v1) {
   %res = call <4 x i32> @llvm.smin.v4i32(<4 x i32> %v0, <4 x i32> %v1)
   ret <4 x i32> %res
 }
 
 ; COST-LABEL: smin.v8i32
-; COST-NEXT: Cost Model: Found an estimated cost of 10 for instruction:   %res = call <8 x i32> @llvm.smin.v8i32(<8 x i32> %v0, <8 x i32> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %res = call <8 x i32> @llvm.smin.v8i32(<8 x i32> %v0, <8 x i32> %v1)
 
 ; CODE-LABEL: smin.v8i32
 ; CODE:       bb.0
 ; CODE-NEXT:   smin v{{.*}}.4s, v{{.*}}.4s, v{{.*}}.4s
 ; CODE-NEXT:   smin v{{.*}}.4s, v{{.*}}.4s, v{{.*}}.4s
 ; CODE-NEXT:   ret
 
 declare <8 x i32> @llvm.smin.v8i32(<8 x i32>, <8 x i32>)

 
 declare <2 x i64> @llvm.smin.v2i64(<2 x i64>, <2 x i64>)
 define <2 x i64> @smin.v2i64(<2 x i64> %v0, <2 x i64> %v1) {
   %res = call <2 x i64> @llvm.smin.v2i64(<2 x i64> %v0, <2 x i64> %v1)
   ret <2 x i64> %res
 }
 
 ; COST-LABEL: umax.v8i8
-; COST-NEXT: Cost Model: Found an estimated cost of 30 for instruction:   %res = call <8 x i8> @llvm.umax.v8i8(<8 x i8> %v0, <8 x i8> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <8 x i8> @llvm.umax.v8i8(<8 x i8> %v0, <8 x i8> %v1)
 
 ; CODE-LABEL: umax.v8i8
 ; CODE:       bb.0
 ; CODE-NEXT:   umax v{{.*}}.8b, v{{.*}}.8b, v{{.*}}.8b
 ; CODE-NEXT:   ret
 
 declare <8 x i8> @llvm.umax.v8i8(<8 x i8>, <8 x i8>)
 define <8 x i8> @umax.v8i8(<8 x i8> %v0, <8 x i8> %v1) {
   %res = call <8 x i8> @llvm.umax.v8i8(<8 x i8> %v0, <8 x i8> %v1)
   ret <8 x i8> %res
 }
 
 ; COST-LABEL: umax.v9i8
-; COST-NEXT: Cost Model: Found an estimated cost of 34 for instruction:   %res = call <9 x i8> @llvm.umax.v9i8(<9 x i8> %v0, <9 x i8> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <9 x i8> @llvm.umax.v9i8(<9 x i8> %v0, <9 x i8> %v1)
 
 ; CODE-LABEL: umax.v9i8
 ; CODE:       bb.0
 ; CODE-NEXT:   umax v{{.*}}.16b, v{{.*}}.16b, v{{.*}}.16b
 ; CODE-NEXT:   ret
 
 declare <9 x i8> @llvm.umax.v9i8(<9 x i8>, <9 x i8>)
 define <9 x i8> @umax.v9i8(<9 x i8> %v0, <9 x i8> %v1) {
   %res = call <9 x i8> @llvm.umax.v9i8(<9 x i8> %v0, <9 x i8> %v1)
   ret <9 x i8> %res
 }
 
-
 ; COST-LABEL: umax.v16i8
-; COST-NEXT: Cost Model: Found an estimated cost of 62 for instruction:   %res = call <16 x i8> @llvm.umax.v16i8(<16 x i8> %v0, <16 x i8> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <16 x i8> @llvm.umax.v16i8(<16 x i8> %v0, <16 x i8> %v1)
 
 ; CODE-LABEL: umax.v16i8
 ; CODE:       bb.0
 ; CODE-NEXT:   umax v{{.*}}.16b, v{{.*}}.16b, v{{.*}}.16b
 ; CODE-NEXT:   ret
 
 declare <16 x i8> @llvm.umax.v16i8(<16 x i8>, <16 x i8>)
 define <16 x i8> @umax.v16i8(<16 x i8> %v0, <16 x i8> %v1) {
   %res = call <16 x i8> @llvm.umax.v16i8(<16 x i8> %v0, <16 x i8> %v1)
   ret <16 x i8> %res
 }
 
 ; COST-LABEL: umax.v4i16
-; COST-NEXT: Cost Model: Found an estimated cost of 14 for instruction:   %res = call <4 x i16> @llvm.umax.v4i16(<4 x i16> %v0, <4 x i16> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <4 x i16> @llvm.umax.v4i16(<4 x i16> %v0, <4 x i16> %v1)
 
 ; CODE-LABEL: umax.v4i16
 ; CODE:       bb.0
 ; CODE-NEXT:   umax v{{.*}}.4h, v{{.*}}.4h, v{{.*}}.4h
 ; CODE-NEXT:   ret
 
 declare <4 x i16> @llvm.umax.v4i16(<4 x i16>, <4 x i16>)
 define <4 x i16> @umax.v4i16(<4 x i16> %v0, <4 x i16> %v1) {
   %res = call <4 x i16> @llvm.umax.v4i16(<4 x i16> %v0, <4 x i16> %v1)
   ret <4 x i16> %res
 }
 
 ; COST-LABEL: umax.v8i16
-; COST-NEXT: Cost Model: Found an estimated cost of 30 for instruction:   %res = call <8 x i16> @llvm.umax.v8i16(<8 x i16> %v0, <8 x i16> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <8 x i16> @llvm.umax.v8i16(<8 x i16> %v0, <8 x i16> %v1)
 
 ; CODE-LABEL: umax.v8i16
 ; CODE:       bb.0
 ; CODE-NEXT:   umax v{{.*}}.8h, v{{.*}}.8h, v{{.*}}.8h
 ; CODE-NEXT:   ret
 
 declare <8 x i16> @llvm.umax.v8i16(<8 x i16>, <8 x i16>)
 define <8 x i16> @umax.v8i16(<8 x i16> %v0, <8 x i16> %v1) {
   %res = call <8 x i16> @llvm.umax.v8i16(<8 x i16> %v0, <8 x i16> %v1)
   ret <8 x i16> %res
 }
 
 ; COST-LABEL: umax.v2i32
-; COST-NEXT: Cost Model: Found an estimated cost of 6 for instruction:   %res = call <2 x i32> @llvm.umax.v2i32(<2 x i32> %v0, <2 x i32> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <2 x i32> @llvm.umax.v2i32(<2 x i32> %v0, <2 x i32> %v1)
 
 ; CODE-LABEL: umax.v2i32
 ; CODE:       bb.0
 ; CODE-NEXT:   umax v{{.*}}.2s, v{{.*}}.2s, v{{.*}}.2s
 ; CODE-NEXT:   ret
 
 declare <2 x i32> @llvm.umax.v2i32(<2 x i32>, <2 x i32>)
 define <2 x i32> @umax.v2i32(<2 x i32> %v0, <2 x i32> %v1) {
   %res = call <2 x i32> @llvm.umax.v2i32(<2 x i32> %v0, <2 x i32> %v1)
   ret <2 x i32> %res
 }
 
 ; COST-LABEL: umax.v4i32
-; COST-NEXT: Cost Model: Found an estimated cost of 14 for instruction:   %res = call <4 x i32> @llvm.umax.v4i32(<4 x i32> %v0, <4 x i32> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <4 x i32> @llvm.umax.v4i32(<4 x i32> %v0, <4 x i32> %v1)
 
 ; CODE-LABEL: umax.v4i32
 ; CODE:       bb.0
 ; CODE-NEXT:   umax v{{.*}}.4s, v{{.*}}.4s, v{{.*}}.4s
 ; CODE-NEXT:   ret
 
 declare <4 x i32> @llvm.umax.v4i32(<4 x i32>, <4 x i32>)
 define <4 x i32> @umax.v4i32(<4 x i32> %v0, <4 x i32> %v1) {
   %res = call <4 x i32> @llvm.umax.v4i32(<4 x i32> %v0, <4 x i32> %v1)
   ret <4 x i32> %res
 }
 
 ; COST-LABEL: umax.v8i32
-; COST-NEXT: Cost Model: Found an estimated cost of 10 for instruction:   %res = call <8 x i32> @llvm.umax.v8i32(<8 x i32> %v0, <8 x i32> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %res = call <8 x i32> @llvm.umax.v8i32(<8 x i32> %v0, <8 x i32> %v1)
 
 ; CODE-LABEL: umax.v8i32
 ; CODE:       bb.0
 ; CODE-NEXT:   umax v{{.*}}.4s, v{{.*}}.4s, v{{.*}}.4s
 ; CODE-NEXT:   umax v{{.*}}.4s, v{{.*}}.4s, v{{.*}}.4s
 ; CODE-NEXT:   ret
 
 declare <8 x i32> @llvm.umax.v8i32(<8 x i32>, <8 x i32>)

 
 declare <2 x i64> @llvm.umax.v2i64(<2 x i64>, <2 x i64>)
 define <2 x i64> @umax.v2i64(<2 x i64> %v0, <2 x i64> %v1) {
   %res = call <2 x i64> @llvm.umax.v2i64(<2 x i64> %v0, <2 x i64> %v1)
   ret <2 x i64> %res
 }
 
 ; COST-LABEL: smax.v8i8
-; COST-NEXT: Cost Model: Found an estimated cost of 30 for instruction:   %res = call <8 x i8> @llvm.smax.v8i8(<8 x i8> %v0, <8 x i8> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <8 x i8> @llvm.smax.v8i8(<8 x i8> %v0, <8 x i8> %v1)
 
 ; CODE-LABEL: smax.v8i8
 ; CODE:       bb.0
 ; CODE-NEXT:   smax v{{.*}}.8b, v{{.*}}.8b, v{{.*}}.8b
 ; CODE-NEXT:   ret
 
 declare <8 x i8> @llvm.smax.v8i8(<8 x i8>, <8 x i8>)
 define <8 x i8> @smax.v8i8(<8 x i8> %v0, <8 x i8> %v1) {
   %res = call <8 x i8> @llvm.smax.v8i8(<8 x i8> %v0, <8 x i8> %v1)
   ret <8 x i8> %res
 }
 
 ; COST-LABEL: smax.v9i8
-; COST-NEXT: Cost Model: Found an estimated cost of 34 for instruction:   %res = call <9 x i8> @llvm.smax.v9i8(<9 x i8> %v0, <9 x i8> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <9 x i8> @llvm.smax.v9i8(<9 x i8> %v0, <9 x i8> %v1)
 
 ; CODE-LABEL: smax.v9i8
 ; CODE:       bb.0
 ; CODE-NEXT:   smax v{{.*}}.16b, v{{.*}}.16b, v{{.*}}.16b
 ; CODE-NEXT:   ret
 
 declare <9 x i8> @llvm.smax.v9i8(<9 x i8>, <9 x i8>)
 define <9 x i8> @smax.v9i8(<9 x i8> %v0, <9 x i8> %v1) {
   %res = call <9 x i8> @llvm.smax.v9i8(<9 x i8> %v0, <9 x i8> %v1)
   ret <9 x i8> %res
 }
 
 ; COST-LABEL: smax.v16i8
-; COST-NEXT: Cost Model: Found an estimated cost of 62 for instruction:   %res = call <16 x i8> @llvm.smax.v16i8(<16 x i8> %v0, <16 x i8> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <16 x i8> @llvm.smax.v16i8(<16 x i8> %v0, <16 x i8> %v1)
 
 ; CODE-LABEL: smax.v16i8
 ; CODE:       bb.0
 ; CODE-NEXT:   smax v{{.*}}.16b, v{{.*}}.16b, v{{.*}}.16b
 ; CODE-NEXT:   ret
 
 declare <16 x i8> @llvm.smax.v16i8(<16 x i8>, <16 x i8>)
 define <16 x i8> @smax.v16i8(<16 x i8> %v0, <16 x i8> %v1) {
   %res = call <16 x i8> @llvm.smax.v16i8(<16 x i8> %v0, <16 x i8> %v1)
   ret <16 x i8> %res
 }
 
 ; COST-LABEL: smax.v4i16
-; COST-NEXT: Cost Model: Found an estimated cost of 14 for instruction:   %res = call <4 x i16> @llvm.smax.v4i16(<4 x i16> %v0, <4 x i16> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <4 x i16> @llvm.smax.v4i16(<4 x i16> %v0, <4 x i16> %v1)
 
 ; CODE-LABEL: smax.v4i16
 ; CODE:       bb.0
 ; CODE-NEXT:   smax v{{.*}}.4h, v{{.*}}.4h, v{{.*}}.4h
 ; CODE-NEXT:   ret
 
 declare <4 x i16> @llvm.smax.v4i16(<4 x i16>, <4 x i16>)
 define <4 x i16> @smax.v4i16(<4 x i16> %v0, <4 x i16> %v1) {
   %res = call <4 x i16> @llvm.smax.v4i16(<4 x i16> %v0, <4 x i16> %v1)
   ret <4 x i16> %res
 }
 
 ; COST-LABEL: smax.v8i16
-; COST-NEXT: Cost Model: Found an estimated cost of 30 for instruction:   %res = call <8 x i16> @llvm.smax.v8i16(<8 x i16> %v0, <8 x i16> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <8 x i16> @llvm.smax.v8i16(<8 x i16> %v0, <8 x i16> %v1)
 
 ; CODE-LABEL: smax.v8i16
 ; CODE:       bb.0
 ; CODE-NEXT:   smax v{{.*}}.8h, v{{.*}}.8h, v{{.*}}.8h
 ; CODE-NEXT:   ret
 
 declare <8 x i16> @llvm.smax.v8i16(<8 x i16>, <8 x i16>)
 define <8 x i16> @smax.v8i16(<8 x i16> %v0, <8 x i16> %v1) {
   %res = call <8 x i16> @llvm.smax.v8i16(<8 x i16> %v0, <8 x i16> %v1)
   ret <8 x i16> %res
 }
 
 ; COST-LABEL: smax.v2i32
-; COST-NEXT: Cost Model: Found an estimated cost of 6 for instruction:   %res = call <2 x i32> @llvm.smax.v2i32(<2 x i32> %v0, <2 x i32> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <2 x i32> @llvm.smax.v2i32(<2 x i32> %v0, <2 x i32> %v1)
 
 ; CODE-LABEL: smax.v2i32
 ; CODE:       bb.0
 ; CODE-NEXT:   smax v{{.*}}.2s, v{{.*}}.2s, v{{.*}}.2s
 ; CODE-NEXT:   ret
 
 declare <2 x i32> @llvm.smax.v2i32(<2 x i32>, <2 x i32>)
 define <2 x i32> @smax.v2i32(<2 x i32> %v0, <2 x i32> %v1) {
   %res = call <2 x i32> @llvm.smax.v2i32(<2 x i32> %v0, <2 x i32> %v1)
   ret <2 x i32> %res
 }
 
 ; COST-LABEL: smax.v4i32
-; COST-NEXT: Cost Model: Found an estimated cost of 14 for instruction:   %res = call <4 x i32> @llvm.smax.v4i32(<4 x i32> %v0, <4 x i32> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 1 for instruction:   %res = call <4 x i32> @llvm.smax.v4i32(<4 x i32> %v0, <4 x i32> %v1)
 
 ; CODE-LABEL: smax.v4i32
 ; CODE:       bb.0
 ; CODE-NEXT:   smax v{{.*}}.4s, v{{.*}}.4s, v{{.*}}.4s
 ; CODE-NEXT:   ret
 
 declare <4 x i32> @llvm.smax.v4i32(<4 x i32>, <4 x i32>)
 define <4 x i32> @smax.v4i32(<4 x i32> %v0, <4 x i32> %v1) {
   %res = call <4 x i32> @llvm.smax.v4i32(<4 x i32> %v0, <4 x i32> %v1)
   ret <4 x i32> %res
 }
 
 ; COST-LABEL: smax.v8i32
-; COST-NEXT: Cost Model: Found an estimated cost of 10 for instruction:   %res = call <8 x i32> @llvm.smax.v8i32(<8 x i32> %v0, <8 x i32> %v1)
+; COST-NEXT: Cost Model: Found an estimated cost of 2 for instruction:   %res = call <8 x i32> @llvm.smax.v8i32(<8 x i32> %v0, <8 x i32> %v1)
 
 ; CODE-LABEL: smax.v8i32
 ; CODE:       bb.0
 ; CODE-NEXT:   smax v{{.*}}.4s, v{{.*}}.4s, v{{.*}}.4s
 ; CODE-NEXT:   smax v{{.*}}.4s, v{{.*}}.4s, v{{.*}}.4s
 ; CODE-NEXT:   ret
 
 declare <8 x i32> @llvm.smax.v8i32(<8 x i32>, <8 x i32>)