[SVE] Remove invalid calls to VectorType::getNumElements from BasicTTIImpl

[ctetreau]

Summary

Most of these operations are reasonable for scalable vectors. Due to
this, we have decided not to change the interface to specifically take
FixedVectorType despite the fact that the current implementations make
fixed width assumptions. Instead, we cast to FixedVectorType and assert
in the body. If a developer makes some change in the future that causes
one of these asserts to fire, they should either change their code or
make the function they are trying to call handle scalable vectors.

[ctetreau]

llvm/include/llvm/CodeGen/BasicTTIImpl.h

90

I think it's reasonable to bikeshed this name. I picked FIXME_ScalableVectorNotSupported because I wanted a name that is very aggressive in expressing that it's a fixme conversion. However, I recognize that it doesn't strictly conform to our documented coding style.

[efriedma]

I agree it makes sense to keep the TargetTransformInfo API as-is; we almost certainly want it to take a generic VectorType.

I'm not sure the FIXME_ScalableVectorNotSupported thing is useful here, as opposed to just writing cast<FixedVectorType>(). Most of these default cost implementations are working under the assumption that it makes sense to scalarize the operation, and estimating costs based on that. That's not generally useful for scalable vectors, and I suspect that we're just going to override all these methods in the AArch64 target, rather than try to fix the target-independent cost estimation. (At least in the short term.) If we do end up crashing anyway, it should be obvious why it's happening.

[ctetreau]

I'm not sure the FIXME_ScalableVectorNotSupported thing is useful here, as opposed to just writing cast<FixedVectorType>().

I like it for the reasons I mentioned. Strictly speaking, it is equivalent to just doing the cast, but I think it provides documentation value. If people feel strongly about removing it, I can do so.

[ctetreau]

Catch some stragglers, remove helper

[ctetreau]

actually remove the helper

[ctetreau]

llvm/include/llvm/CodeGen/BasicTTIImpl.h

90

I went ahead and removed the helper.

[ctetreau]

remove superfluous casts and potential null dereferences

[RKSimon]

llvm/include/llvm/CodeGen/BasicTTIImpl.h

84

We'd be better off just changing the function to take FixedVectorType, same for all the other cases below.

[ctetreau]

llvm/include/llvm/CodeGen/BasicTTIImpl.h

84

For the private methods, or all of them?

I think changing the private methods to take FixedVectorType is fine. I prefer to leave the public methods as VectorType so that scalable architectures can use this interface.

[ctetreau]

address code review issues

[efriedma]

LGTM

llvm/include/llvm/CodeGen/BasicTTIImpl.h

 template <typename T>
 class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
 private:
   using BaseT = TargetTransformInfoImplCRTPBase<T>;
   using TTI = TargetTransformInfo;
 
   /// Estimate a cost of Broadcast as an extract and sequence of insert
   /// operations.
-  unsigned getBroadcastShuffleOverhead(VectorType *VTy) {
+  unsigned getBroadcastShuffleOverhead(VectorType *InVTy) {
+    auto *VTy = cast<FixedVectorType>(InVTy);
+
     unsigned Cost = 0;
     // Broadcast cost is equal to the cost of extracting the zero'th element
     // plus the cost of inserting it into every element of the result vector.
     Cost += static_cast<T *>(this)->getVectorInstrCost(
         Instruction::ExtractElement, VTy, 0);
 
     for (int i = 0, e = VTy->getNumElements(); i < e; ++i) {
       Cost += static_cast<T *>(this)->getVectorInstrCost(
           Instruction::InsertElement, VTy, i);
     }
     return Cost;
   }
 
   /// Estimate a cost of shuffle as a sequence of extract and insert
   /// operations.
-  unsigned getPermuteShuffleOverhead(VectorType *VTy) {
+  unsigned getPermuteShuffleOverhead(VectorType *InVTy) {
+    auto *VTy = cast<FixedVectorType>(InVTy);
+
     unsigned Cost = 0;
     // Shuffle cost is equal to the cost of extracting element from its argument
     // plus the cost of inserting them onto the result vector.
 
     // e.g. <4 x float> has a mask of <0,5,2,7> i.e we need to extract from
     // index 0 of first vector, index 1 of second vector,index 2 of first
     // vector and finally index 3 of second vector and insert them at index
     // <0,1,2,3> of result vector.
     for (int i = 0, e = VTy->getNumElements(); i < e; ++i) {
       Cost += static_cast<T *>(this)->getVectorInstrCost(
           Instruction::InsertElement, VTy, i);
       Cost += static_cast<T *>(this)->getVectorInstrCost(
           Instruction::ExtractElement, VTy, i);
     }
     return Cost;
   }
 
   /// Estimate a cost of subvector extraction as a sequence of extract and
   /// insert operations.
-  unsigned getExtractSubvectorOverhead(VectorType *VTy, int Index,
-                                       VectorType *SubVTy) {
+  unsigned getExtractSubvectorOverhead(VectorType *InVTy, int Index,
+                                       VectorType *InSubVTy) {
+    auto *VTy = cast<FixedVectorType>(InVTy);
+    auto *SubVTy = cast<FixedVectorType>(InSubVTy);
+
     assert(VTy && SubVTy &&
            "Can only extract subvectors from vectors");
     int NumSubElts = SubVTy->getNumElements();
     assert((Index + NumSubElts) <= (int)VTy->getNumElements() &&
            "SK_ExtractSubvector index out of range");
 
     unsigned Cost = 0;
     // Subvector extraction cost is equal to the cost of extracting element from
     // the source type plus the cost of inserting them into the result vector
     // type.
     for (int i = 0; i != NumSubElts; ++i) {
       Cost += static_cast<T *>(this)->getVectorInstrCost(
           Instruction::ExtractElement, VTy, i + Index);
       Cost += static_cast<T *>(this)->getVectorInstrCost(
           Instruction::InsertElement, SubVTy, i);
     }
     return Cost;
   }
 
   /// Estimate a cost of subvector insertion as a sequence of extract and
   /// insert operations.
-  unsigned getInsertSubvectorOverhead(VectorType *VTy, int Index,
-                                      VectorType *SubVTy) {
+  unsigned getInsertSubvectorOverhead(VectorType *InVTy, int Index,
+                                      VectorType *InSubVTy) {
+    auto *VTy = cast<FixedVectorType>(InVTy);
+    auto *SubVTy = cast<FixedVectorType>(InSubVTy);
+
     assert(VTy && SubVTy &&
            "Can only insert subvectors into vectors");
     int NumSubElts = SubVTy->getNumElements();
     assert((Index + NumSubElts) <= (int)VTy->getNumElements() &&
            "SK_InsertSubvector index out of range");
 
     unsigned Cost = 0;
     // Subvector insertion cost is equal to the cost of extracting element from
public:
   /// \name Vector TTI Implementations
   /// @{
 
   unsigned getRegisterBitWidth(bool Vector) const { return 32; }
 
   /// Estimate the overhead of scalarizing an instruction. Insert and Extract
   /// are set if the demanded result elements need to be inserted and/or
   /// extracted from vectors.
-  unsigned getScalarizationOverhead(VectorType *Ty, const APInt &DemandedElts,
+  unsigned getScalarizationOverhead(VectorType *InTy, const APInt &DemandedElts,
                                     bool Insert, bool Extract) {
+    /// FIXME: a bitfield is not a reasonable abstraction for talking about
+    /// which elements are needed from a scalable vector
+    auto *Ty = cast<FixedVectorType>(InTy);
+
     assert(DemandedElts.getBitWidth() == Ty->getNumElements() &&
            "Vector size mismatch");
 
     unsigned Cost = 0;
 
     for (int i = 0, e = Ty->getNumElements(); i < e; ++i) {
       if (!DemandedElts[i])
         continue;
       if (Insert)
         Cost += static_cast<T *>(this)->getVectorInstrCost(
             Instruction::InsertElement, Ty, i);
       if (Extract)
         Cost += static_cast<T *>(this)->getVectorInstrCost(
             Instruction::ExtractElement, Ty, i);
     }
 
     return Cost;
   }
 
   /// Helper wrapper for the DemandedElts variant of getScalarizationOverhead.
-  unsigned getScalarizationOverhead(VectorType *Ty, bool Insert, bool Extract) {
+  unsigned getScalarizationOverhead(VectorType *InTy, bool Insert,
+                                    bool Extract) {
+    auto *Ty = cast<FixedVectorType>(InTy);
+
     APInt DemandedElts = APInt::getAllOnesValue(Ty->getNumElements());
     return static_cast<T *>(this)->getScalarizationOverhead(Ty, DemandedElts,
                                                             Insert, Extract);
   }
 
   /// Estimate the overhead of scalarizing an instructions unique
   /// non-constant operands. The types of the arguments are ordinarily
   /// scalar, in which case the costs are multiplied with VF.
   unsigned getOperandsScalarizationOverhead(ArrayRef<const Value *> Args,
                                             unsigned VF) {
     unsigned Cost = 0;
     SmallPtrSet<const Value*, 4> UniqueOperands;
     for (const Value *A : Args) {
       if (!isa<Constant>(A) && UniqueOperands.insert(A).second) {
         auto *VecTy = dyn_cast<VectorType>(A->getType());
         if (VecTy) {
           // If A is a vector operand, VF should be 1 or correspond to A.
-          assert((VF == 1 || VF == VecTy->getNumElements()) &&
+          assert((VF == 1 ||
+                  VF == cast<FixedVectorType>(VecTy)->getNumElements()) &&
                  "Vector argument does not match VF");
         }
         else
-          VecTy = VectorType::get(A->getType(), VF);
+          VecTy = FixedVectorType::get(A->getType(), VF);
 
         Cost += getScalarizationOverhead(VecTy, false, true);
       }
     }
 
     return Cost;
   }
 
-  unsigned getScalarizationOverhead(VectorType *Ty, ArrayRef<const Value *> Args) {
+  unsigned getScalarizationOverhead(VectorType *InTy,
+                                    ArrayRef<const Value *> Args) {
+    auto *Ty = cast<FixedVectorType>(InTy);
+
     unsigned Cost = 0;
 
     Cost += getScalarizationOverhead(Ty, true, false);
     if (!Args.empty())
       Cost += getOperandsScalarizationOverhead(Args, Ty->getNumElements());
     else
       // When no information on arguments is provided, we add the cost
       // associated with one argument as a heuristic.
if (!TLI->isOperationExpand(ISD, LT.second)) {
       // as expensive.
       return LT.first * 2 * OpCost;
     }
 
     // Else, assume that we need to scalarize this op.
     // TODO: If one of the types get legalized by splitting, handle this
     // similarly to what getCastInstrCost() does.
     if (auto *VTy = dyn_cast<VectorType>(Ty)) {
-      unsigned Num = VTy->getNumElements();
+      unsigned Num = cast<FixedVectorType>(VTy)->getNumElements();
       unsigned Cost = static_cast<T *>(this)->getArithmeticInstrCost(
           Opcode, VTy->getScalarType(), CostKind);
       // Return the cost of multiple scalar invocation plus the cost of
       // inserting and extracting the values.
       return getScalarizationOverhead(VTy, Args) + Num * Cost;
     }
 
     // We don't know anything about this scalar instruction.
     return OpCost;
   }
 
   unsigned getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp, int Index,
                           VectorType *SubTp) {
+
     switch (Kind) {
     case TTI::SK_Broadcast:
       return getBroadcastShuffleOverhead(Tp);
     case TTI::SK_Select:
     case TTI::SK_Reverse:
     case TTI::SK_Transpose:
     case TTI::SK_PermuteSingleSrc:
     case TTI::SK_PermuteTwoSrc:
if (DstVTy && SrcVTy) {
       // cost of the split itself. Count that as 1, to be consistent with
       // TLI->getTypeLegalizationCost().
       bool SplitSrc =
           TLI->getTypeAction(Src->getContext(), TLI->getValueType(DL, Src)) ==
           TargetLowering::TypeSplitVector;
       bool SplitDst =
           TLI->getTypeAction(Dst->getContext(), TLI->getValueType(DL, Dst)) ==
           TargetLowering::TypeSplitVector;
-      if ((SplitSrc || SplitDst) && SrcVTy->getNumElements() > 1 &&
-          DstVTy->getNumElements() > 1) {
-        Type *SplitDstTy = VectorType::get(DstVTy->getElementType(),
-                                           DstVTy->getNumElements() / 2);
-        Type *SplitSrcTy = VectorType::get(SrcVTy->getElementType(),
-                                           SrcVTy->getNumElements() / 2);
+      if ((SplitSrc || SplitDst) &&
+          cast<FixedVectorType>(SrcVTy)->getNumElements() > 1 &&
+          cast<FixedVectorType>(DstVTy)->getNumElements() > 1) {
+        Type *SplitDstTy = VectorType::getHalfElementsVectorType(DstVTy);
+        Type *SplitSrcTy = VectorType::getHalfElementsVectorType(SrcVTy);
         T *TTI = static_cast<T *>(this);
         // If both types need to be split then the split is free.
         unsigned SplitCost =
             (!SplitSrc || !SplitDst) ? TTI->getVectorSplitCost() : 0;
         return SplitCost +
                (2 * TTI->getCastInstrCost(Opcode, SplitDstTy, SplitSrcTy,
                                           CostKind, I));
       }
 
       // In other cases where the source or destination are illegal, assume
       // the operation will get scalarized.
-      unsigned Num = DstVTy->getNumElements();
+      unsigned Num = cast<FixedVectorType>(DstVTy)->getNumElements();
       unsigned Cost = static_cast<T *>(this)->getCastInstrCost(
           Opcode, Dst->getScalarType(), Src->getScalarType(),
           CostKind, I);
 
       // Return the cost of multiple scalar invocation plus the cost of
       // inserting and extracting the values.
       return getScalarizationOverhead(DstVTy, true, true) + Num * Cost;
     }
if (!(ValTy->isVectorTy() && !LT.second.isVector()) &&
       // by the type-legalization overhead.
       return LT.first * 1;
     }
 
     // Otherwise, assume that the cast is scalarized.
     // TODO: If one of the types get legalized by splitting, handle this
     // similarly to what getCastInstrCost() does.
     if (auto *ValVTy = dyn_cast<VectorType>(ValTy)) {
-      unsigned Num = ValVTy->getNumElements();
+      unsigned Num = cast<FixedVectorType>(ValVTy)->getNumElements();
       if (CondTy)
         CondTy = CondTy->getScalarType();
       unsigned Cost = static_cast<T *>(this)->getCmpSelInstrCost(
           Opcode, ValVTy->getScalarType(), CondTy, CostKind, I);
 
       // Return the cost of multiple scalar invocation plus the cost of
       // inserting and extracting the values.
       return getScalarizationOverhead(ValVTy, true, false) + Num * Cost;
public:
 
   unsigned getInterleavedMemoryOpCost(unsigned Opcode, Type *VecTy,
                                       unsigned Factor,
                                       ArrayRef<unsigned> Indices,
                                       unsigned Alignment, unsigned AddressSpace,
                                       TTI::TargetCostKind CostKind,
                                       bool UseMaskForCond = false,
                                       bool UseMaskForGaps = false) {
-    auto *VT = cast<VectorType>(VecTy);
+    auto *VT = cast<FixedVectorType>(VecTy);
 
     unsigned NumElts = VT->getNumElements();
     assert(Factor > 1 && NumElts % Factor == 0 && "Invalid interleave factor");
 
     unsigned NumSubElts = NumElts / Factor;
-    VectorType *SubVT = VectorType::get(VT->getElementType(), NumSubElts);
+    auto *SubVT = FixedVectorType::get(VT->getElementType(), NumSubElts);
 
     // Firstly, the cost of load/store operation.
     unsigned Cost;
     if (UseMaskForCond || UseMaskForGaps)
       Cost = static_cast<T *>(this)->getMaskedMemoryOpCost(
           Opcode, VecTy, Alignment, AddressSpace, CostKind);
     else
       Cost = static_cast<T *>(this)->getMemoryOpCost(
if (Opcode == Instruction::Load) {
         Cost += static_cast<T *>(this)
                     ->getVectorInstrCost(Instruction::InsertElement, VT, i);
     }
 
     if (!UseMaskForCond)
       return Cost;
 
     Type *I8Type = Type::getInt8Ty(VT->getContext());
-    VectorType *MaskVT = VectorType::get(I8Type, NumElts);
-    SubVT = VectorType::get(I8Type, NumSubElts);
+    auto *MaskVT = FixedVectorType::get(I8Type, NumElts);
+    SubVT = FixedVectorType::get(I8Type, NumSubElts);
 
     // The Mask shuffling cost is extract all the elements of the Mask
     // and insert each of them Factor times into the wide vector:
     //
     // E.g. an interleaved group with factor 3:
     //    %mask = icmp ult <8 x i32> %vec1, %vec2
     //    %interleaved.mask = shufflevector <8 x i1> %mask, <8 x i1> undef,
     //        <24 x i32> <0,0,0,1,1,1,2,2,2,3,3,3,4,4,4,5,5,5,6,6,6,7,7,7>
unsigned getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
 
     // TODO: Combine these two logic paths.
     if (ICA.isTypeBasedOnly())
       return getTypeBasedIntrinsicInstrCost(ICA, CostKind);
 
     Type *RetTy = ICA.getReturnType();
     unsigned VF = ICA.getVectorFactor();
     unsigned RetVF =
-        (RetTy->isVectorTy() ? cast<VectorType>(RetTy)->getNumElements() : 1);
+        (RetTy->isVectorTy() ? cast<FixedVectorType>(RetTy)->getNumElements()
+                             : 1);
     assert((RetVF == 1 || VF == 1) && "VF > 1 and RetVF is a vector type");
     const IntrinsicInst *I = ICA.getInst();
     const SmallVectorImpl<Value *> &Args = ICA.getArgs();
     FastMathFlags FMF = ICA.getFlags();
 
     switch (IID) {
     default: {
       // Assume that we need to scalarize this intrinsic.
       SmallVector<Type *, 4> Types;
       for (Value *Op : Args) {
         Type *OpTy = Op->getType();
         assert(VF == 1 || !OpTy->isVectorTy());
-        Types.push_back(VF == 1 ? OpTy : VectorType::get(OpTy, VF));
+        Types.push_back(VF == 1 ? OpTy : FixedVectorType::get(OpTy, VF));
       }
 
       if (VF > 1 && !RetTy->isVoidTy())
-        RetTy = VectorType::get(RetTy, VF);
+        RetTy = FixedVectorType::get(RetTy, VF);
 
       // Compute the scalarization overhead based on Args for a vector
       // intrinsic. A vectorizer will pass a scalar RetTy and VF > 1, while
       // CostModel will pass a vector RetTy and VF is 1.
       unsigned ScalarizationCost = std::numeric_limits<unsigned>::max();
       if (RetVF > 1 || VF > 1) {
         ScalarizationCost = 0;
         if (!RetTy->isVoidTy())
unsigned getTypeBasedIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
     default: {
       // Assume that we need to scalarize this intrinsic.
       unsigned ScalarizationCost = ScalarizationCostPassed;
       unsigned ScalarCalls = 1;
       Type *ScalarRetTy = RetTy;
       if (auto *RetVTy = dyn_cast<VectorType>(RetTy)) {
         if (!SkipScalarizationCost)
           ScalarizationCost = getScalarizationOverhead(RetVTy, true, false);
-        ScalarCalls = std::max(ScalarCalls, RetVTy->getNumElements());
+        ScalarCalls = std::max(ScalarCalls,
+                               cast<FixedVectorType>(RetVTy)->getNumElements());
         ScalarRetTy = RetTy->getScalarType();
       }
       SmallVector<Type *, 4> ScalarTys;
       for (unsigned i = 0, ie = Tys.size(); i != ie; ++i) {
         Type *Ty = Tys[i];
         if (auto *VTy = dyn_cast<VectorType>(Ty)) {
           if (!SkipScalarizationCost)
             ScalarizationCost += getScalarizationOverhead(VTy, false, true);
-          ScalarCalls = std::max(ScalarCalls, VTy->getNumElements());
+          ScalarCalls = std::max(ScalarCalls,
+                                 cast<FixedVectorType>(VTy)->getNumElements());
           Ty = Ty->getScalarType();
         }
         ScalarTys.push_back(Ty);
       }
       if (ScalarCalls == 1)
         return 1; // Return cost of a scalar intrinsic. Assume it to be cheap.
 
       IntrinsicCostAttributes ScalarAttrs(IID, ScalarRetTy, ScalarTys, FMF);
unsigned getTypeBasedIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
 
     // Else, assume that we need to scalarize this intrinsic. For math builtins
     // this will emit a costly libcall, adding call overhead and spills. Make it
     // very expensive.
     if (auto *RetVTy = dyn_cast<VectorType>(RetTy)) {
       unsigned ScalarizationCost = SkipScalarizationCost ?
         ScalarizationCostPassed : getScalarizationOverhead(RetVTy, true, false);
 
-      unsigned ScalarCalls = RetVTy->getNumElements();
+      unsigned ScalarCalls = cast<FixedVectorType>(RetVTy)->getNumElements();
       SmallVector<Type *, 4> ScalarTys;
       for (unsigned i = 0, ie = Tys.size(); i != ie; ++i) {
         Type *Ty = Tys[i];
         if (Ty->isVectorTy())
           Ty = Ty->getScalarType();
         ScalarTys.push_back(Ty);
       }
       IntrinsicCostAttributes Attrs(IID, RetTy->getScalarType(), ScalarTys, FMF);
       unsigned ScalarCost = ConcreteTTI->getIntrinsicInstrCost(Attrs, CostKind);
       for (unsigned i = 0, ie = Tys.size(); i != ie; ++i) {
         if (auto *VTy = dyn_cast<VectorType>(Tys[i])) {
           if (!ICA.skipScalarizationCost())
             ScalarizationCost += getScalarizationOverhead(VTy, false, true);
-          ScalarCalls = std::max(ScalarCalls, VTy->getNumElements());
+          ScalarCalls = std::max(ScalarCalls,
+                                 cast<FixedVectorType>(VTy)->getNumElements());
         }
       }
       return ScalarCalls * ScalarCost + ScalarizationCost;
     }
 
     // This is going to be turned into a library call, make it expensive.
     return SingleCallCost;
   }

   /// vector %red1 is <n/2 x t> vector.
   ///
   /// The cost model should take into account that the actual length of the
   /// vector is reduced on each iteration.
   unsigned getArithmeticReductionCost(unsigned Opcode, VectorType *Ty,
                                       bool IsPairwise,
                                       TTI::TargetCostKind CostKind) {
     Type *ScalarTy = Ty->getElementType();
-    unsigned NumVecElts = Ty->getNumElements();
+    unsigned NumVecElts = cast<FixedVectorType>(Ty)->getNumElements();
     unsigned NumReduxLevels = Log2_32(NumVecElts);
     unsigned ArithCost = 0;
     unsigned ShuffleCost = 0;
     auto *ConcreteTTI = static_cast<T *>(this);
     std::pair<unsigned, MVT> LT =
         ConcreteTTI->getTLI()->getTypeLegalizationCost(DL, Ty);
     unsigned LongVectorCount = 0;
     unsigned MVTLen =
         LT.second.isVector() ? LT.second.getVectorNumElements() : 1;
     while (NumVecElts > MVTLen) {
       NumVecElts /= 2;
-      VectorType *SubTy = VectorType::get(ScalarTy, NumVecElts);
+      VectorType *SubTy = FixedVectorType::get(ScalarTy, NumVecElts);
       // Assume the pairwise shuffles add a cost.
       ShuffleCost += (IsPairwise + 1) *
                      ConcreteTTI->getShuffleCost(TTI::SK_ExtractSubvector, Ty,
                                                  NumVecElts, SubTy);
       ArithCost += ConcreteTTI->getArithmeticInstrCost(Opcode, SubTy, CostKind);
       Ty = SubTy;
       ++LongVectorCount;
     }

 
   /// Try to calculate op costs for min/max reduction operations.
   /// \param CondTy Conditional type for the Select instruction.
   unsigned getMinMaxReductionCost(VectorType *Ty, VectorType *CondTy,
                                   bool IsPairwise, bool IsUnsigned,
                                   TTI::TargetCostKind CostKind) {
     Type *ScalarTy = Ty->getElementType();
     Type *ScalarCondTy = CondTy->getElementType();
-    unsigned NumVecElts = Ty->getNumElements();
+    unsigned NumVecElts = cast<FixedVectorType>(Ty)->getNumElements();
     unsigned NumReduxLevels = Log2_32(NumVecElts);
     unsigned CmpOpcode;
     if (Ty->isFPOrFPVectorTy()) {
       CmpOpcode = Instruction::FCmp;
     } else {
       assert(Ty->isIntOrIntVectorTy() &&
              "expecting floating point or integer type for min/max reduction");
       CmpOpcode = Instruction::ICmp;
     }
     unsigned MinMaxCost = 0;
     unsigned ShuffleCost = 0;
     auto *ConcreteTTI = static_cast<T *>(this);
     std::pair<unsigned, MVT> LT =
         ConcreteTTI->getTLI()->getTypeLegalizationCost(DL, Ty);
     unsigned LongVectorCount = 0;
     unsigned MVTLen =
         LT.second.isVector() ? LT.second.getVectorNumElements() : 1;
     while (NumVecElts > MVTLen) {
       NumVecElts /= 2;
-      VectorType *SubTy = VectorType::get(ScalarTy, NumVecElts);
-      CondTy = VectorType::get(ScalarCondTy, NumVecElts);
+      auto *SubTy = FixedVectorType::get(ScalarTy, NumVecElts);
+      CondTy = FixedVectorType::get(ScalarCondTy, NumVecElts);
 
       // Assume the pairwise shuffles add a cost.
       ShuffleCost += (IsPairwise + 1) *
                      ConcreteTTI->getShuffleCost(TTI::SK_ExtractSubvector, Ty,
                                                  NumVecElts, SubTy);
       MinMaxCost +=
           ConcreteTTI->getCmpSelInstrCost(CmpOpcode, SubTy, CondTy, CostKind) +
           ConcreteTTI->getCmpSelInstrCost(Instruction::Select, SubTy, CondTy,