[SVE] Make ElementCount and TypeSize use a new PolySize class

[david-arm]

Summary

I have introduced a new template PolySize class, where the template
parameter determines the type of quantity, i.e. for an element
count this is just an unsigned value. The ElementCount class is
now just a simple derivation of PolySize<unsigned>, whereas TypeSize
is more complicated because it still needs to contain the uint64_t
cast operator, since there are still many places in the code that
rely upon this implicit cast. As such the class also still needs
some of it's own operators.

I've tried to minimise the amount of code in the base PolySize
class, which led to a couple of changes:

1. In some places we were relying on '==' operator comparisons

between ElementCounts and the scalar value 1. I didn't put this
operator in the new PolySize class, and thought it was actually
clearer to use the isScalar() function instead.

2. I removed the isByteSized function and replaced it with calls

to isKnownMultipleOf(8).

I've also renamed NextPowerOf2 to be coefficientNextPowerOf2 so
that it's more consistent with coefficientDivideBy.

[ctetreau]

llvm/include/llvm/Support/TypeSize.h

50

Why does there need to be a default constructor?

[david-arm]

llvm/include/llvm/Support/TypeSize.h

50

I think because there are places in the code base that rely upon it - this isn't something new as it was already in ElementCount. I think there is one example in llvm/include/llvm/IR/Intrinsics.h where the ElementCount is a member of a union. I'm pretty sure that removing the default constructor causes compilation errors, but I can try again to make sure.

[ctetreau]

llvm/include/llvm/Support/TypeSize.h

50

Assuming it's necessary to have this constructor, what should it do? Should it explicitly be defined to be {0, false}? Is it a garbage value?

I suppose {0, false} sort of is already a garbage value, so maybe everything can just assert that it's not this?

[david-arm]

llvm/include/llvm/Support/TypeSize.h

50

I think it's just like someone declaring a variable and then initialising it later, no different from a normal integer type that has a default constructor, i.e.

uint64_t SomeVal;
...
SomeVal = name == "Bob" ? 3 : 7;

If we want to remove the default constructor we could do it, but it will require some work to fix up cases where it's relied upon. I can certainly see what breaks when I remove it?

[david-arm]

Hi @ctetreau so when I remove the default constructor I get this build error:

llvm/include/llvm/IR/Intrinsics.h: In static member function ‘static llvm::Intrinsic::IITDescriptor llvm::Intrinsic::IITDescriptor::getVector(unsigned int, bool)’:
llvm/include/llvm/IR/Intrinsics.h:191:21: error: use of deleted function ‘llvm::Intrinsic::IITDescriptor::IITDescriptor()’

191 |       IITDescriptor Result;
    |                     ^~~~~~

llvm/include/llvm/IR/Intrinsics.h:101:10: note: ‘llvm::Intrinsic::IITDescriptor::IITDescriptor()’ is implicitly deleted because the default definition would be ill-formed:

101 |   struct IITDescriptor {

llvm/include/llvm/IR/Intrinsics.h:101:10: error: no matching function for call to ‘llvm::PolySize<unsigned int>::PolySize()’

I think it's because we've included ElementCount within another struct. Ideally I'd like to avoid having to add asserts in PolySize member functions to ensure it's not a default value. I'll see if it's possible to change the IITDescriptor struct in some way so that this doesn't happen.

[david-arm]

OK I can remove the default constructor by rewriting the IITDescriptor in such a way that it's no longer needed.

[ctetreau]

llvm/include/llvm/Support/TypeSize.h

50

I guess it might be nice to be able to do this. When ElementCount and TypeSize were just PODs, I was more OK with expecting the user to not mess it up, but since we're making them proper classes now...

I think if you make isScalable() and getKnownMinValue() assert that it's not {0, false}, then we can leave the default constructor for ergonomic purposes. If everything else were implemented in terms of these functions, then everything would be covered by the assert.

[sdesmalen]

llvm/include/llvm/Support/TypeSize.h

31

You've implemented PolySize more like the current TypeSize than an actual polynomial. That's fine I guess, but I'm currently looking at some interfaces that could use a general StackOffset. Since that currently lives in the AArch64 target, I might create a follow-up patch that generalises this further using a PolyBase class.

58–59

isScalar() only makes sense in the context of ElementCount and doesn't make sense in the context of TypeSize, so I believe this shouldn't live here.

[david-arm]

llvm/include/llvm/Support/TypeSize.h

31

Yeah, perhaps I remembered wrong, but I thought this was inline with what we'd discussed previously. This patch is mainly a refactoring effort to introduce the new class and is just the first step in the evolution. It seemed like refactoring + changing the fundamental nature in one patch might be too confusing that's all. For example, the isKnownXY functions will need rewriting to cope with full polynomials. We might even want to audit places where isScalable() is called to ensure the caller knows that returning true doesn't mean there are no fixed width elements. We might want to reverse the question to something like isFixed() perhaps?

58–59

Yes I think you're right. A polynomial definitely can be scalar, but that doesn't mean the scalar value is one.

[ctetreau]

llvm/include/llvm/Support/TypeSize.h

31

I think it makes sense to implement the actual abstract polynomial, and then have ElementCount and TypeSize inherit from them. So you could have something like:

template <typename NumT, typename VarT> class PolynomialTerm {
private:
  NumT Coefficient;
  VarT Var;
  // TODO: 2 or more vars in one term, assert that this doesn't happen for now
  // TODO: exponents
public:
  NumT getCoefficient() const;
  void setCoefficient(NumT N);
  VarT getVar() const;
  void setVar(VarT V);


  PolynomialTerm operator*(const PolynomialTerm& N) const;
  std::vector<PolynomialTerm> operator+(const PolynomialTerm& N) const;
  // any operator that makes sense for the abstract polynomial term

  bool isKnownLT(const PolynomialTerm& RHS) const;
  bool maybeLT(const PolynomialTerm& RHS) const;
  // all rest of the relations ...
};

And then your ElementCount looks like this:

class ElementCount : public PolynomialTerm<unsigned, bool> {
private:
  ElementCount(unsigned ElementQuantity, bool Scalable)
    : PolynomialTerm(ElementQuantity, Scalable) {}
public:
  bool isScalable() {
    return !getVar();
  }

  static ElementCount getFixed(unsigned N) {
    return ElementCount(N, false);
  }

  ElementCount divideCoefficientBy(unsigned N) {
    return ElementCount(getCoefficient() / N, getVar());
  }

  // etc... All the functions that make sense for ElementCount but not abstract polynomials
};

This will give us a reusable polynomial that can be extended in the future. If ElementCount and TypeSize are well-behaved, then they will just continue to work if the polynomial is extended in the future.

[ctetreau]

llvm/include/llvm/Support/TypeSize.h

31

the implementation of isScalable (should be return getVar();) in my example is wrong, but the point stands.

[vkmr]

llvm/include/llvm/Support/TypeSize.h

49–50

It might be useful to add an explicit getNone() (or getZero()) function that simply returns {0, false} value. For instance, there are places using Optional<ElementCount>for VF (and Optional<unsigned> where VF is not yet ported to ElementCount) and use None to represent VectorizationFailure. We already have isZero() and isNonZero() functions so a Zero getter should not be out of tune.

[david-arm]

llvm/include/llvm/Support/TypeSize.h

31

Hi @ctetreau I personally only intended this particular patch to be a refactoring effort where we can remove some duplication between the two classes and introduce some level of consistency. This first patch then leaves the door open for the introduction of a real polynomial later on. I know that not everyone is keen on using polynomials for ElementCount and TypeSize so I didn't want to make that decision without a strong consensus. The maths and the comparisons become a lot trickier for true polynomials too, not to mention the interfaces and class structures will need changing. I thought it best to only do that work when and where it was truly needed, and perhaps then we'll have a better understanding of the requirements. Is that ok?

[sdesmalen]

llvm/include/llvm/Support/TypeSize.h

31

I'd be okay with you keeping the scope of this patch as-is, I just wanted to let you know I'll probably work on a more generalised polynomial as a follow-up to implement a more generic class for StackOffset. The main concern we wanted to address was removing some duplication, which is what this patch achieves.
Do please add a TODO comment that explains there is future work required as this class doesn't yet implement a polynomial.

[david-arm]

llvm/include/llvm/Support/TypeSize.h

50

I tried replacing "PolySize() = default" with something explicit like "PolySize() : MinVal(0), IsScalable(false) {}", but sadly the compiler says this is not good enough for IITDescriptor. Since we can't be sure exactly what the default value is (I assume it's {0,false} but perhaps someone more knowledgeable can confirm?) I'd have to add asserts in isScalable, getKnownMinValue, getFixedSize and so on along the lines of:

assert(*this != PolySize())

[ctetreau]

llvm/include/llvm/Support/TypeSize.h

31

@david-arm Sorry it took me so long to respond.

If this is is meant to be an temporary intermediate step with the goal of reducing code duplication, then I am fine with it. I agree with Sander that a TODO comment describing the current state and ultimate goal of the work should be added.

[ctetreau]

llvm/include/llvm/Support/TypeSize.h

49–50

Bikeshedding: Maybe it should be named something like getNull() or getInvalid()? You propose using getZero() instead of having an Optional<ElementCount>. I think this usage is fine, but we should be clear that the thing is a garbage value, not zero in the mathematical sense. This lets us make the garbage value be something different in the future if there ends up being a better option.

50

I'm surprised that PolySize() : MinVal(0), IsScalable(false) {} doesn't work, what error does it give? Some issue with assigning 0 to the template field?

Regardless, I think you can assign values at the field headers, and it will be taken as the default. So do:

T MinVal = 0;            // The minimum value that it could be.
bool IsScalable = false; // If true, the total value is determined by multiplying

[vkmr]

llvm/include/llvm/Support/TypeSize.h

49–50

Good point @ctetreau! I am not too keen on one name over other, but if I have to, my vote is for getNull(); I feel it best conveys the idea.

[david-arm]

llvm/include/llvm/Support/TypeSize.h

49–50

Thanks for the suggestion @ctetreau, I'll change the name to getNull()!

50

Hi @ctetreau, so having that constructor is fine, it's just that the compilation of IITDescriptor still fails with the same complaint about a lack of default constructors. :( Sometimes I feel C++ can be incredibly powerful, sometimes it just seems to thwart us at every corner! I'll try your suggestion of assigning default values.

[david-arm]

Hi @ctetreau, I decided to remove the default constructors as we only need a one or two line change llvm/include/llvm/IR/Intrinsics.h. This removes the need for asserts that we have a properly constructed object. Does this seem reasonable?

[ctetreau]

In D88409#2314381, @david-arm wrote:

Hi @ctetreau, I decided to remove the default constructors as we only need a one or two line change llvm/include/llvm/IR/Intrinsics.h. This removes the need for asserts that we have a properly constructed object. Does this seem reasonable?

I think that would be idea.

[ctetreau]

llvm/include/llvm/Support/TypeSize.h

55

NIT: Implement in terms of isZero

153

NIT: prefer static_cast

174

Why provide this implicit conversion from base PolySize to ElementCount?

This could potentially lead to bugs, especially in the future when we introduce true a Polynomial type with exponents and/or multiple variables.

Is this neccesary to support cases like ElementCount EC = PolySize::getScalable(4)?

194

same as above, this implicit conversion could be dangerous

[sdesmalen]

llvm/include/llvm/Support/TypeSize.h

174

This is needed for the result from the base class' operators which will be of type PolySize<unsigned>. With this conversion, these can be freely converted to ElementCount and avoids having to re-specify all the operators. I've taken a similar approach in D88982 and wasn't entirely sure if this was actually unsafe.

The alternative is redefining the operators and calling the base-class, and making the conversion function private/protected to the deriving class.

I actually tried deriving from some other class, something like:

 template<typename Derived, typename Base>
class Operators {
  Derived operator+(const Derived &Other) { return (Derived) Base::operator+(Other); }
};

template<typename T> class Derived : public PolyBase<...>,
              public Operators<Derived, PolyBase> { ... };

but I got C++ template errors that I wasn't able to work around. Not sure if this should even be possible :)

[david-arm]

• Changed PolySize::isNonZero() to return !isZero()
• Used static_cast in coefficientNextPowerOf2

[sdesmalen]

Thanks @david-arm, I think I'm happy enough with the changes you've made. Also some of these changes will be followed up further in D88982.
Please also give @ctetreau a chance to give his blessing before you commit.

[ctetreau]

looks good to me

[david-arm]

I should be adding a fix for the build failure very soon. Will replace the initializer for IITDescriptor with {Vector, {0}}.

llvm/include/llvm/CodeGen/TargetLowering.h

public:
 
   /// Return true if the target has BitExtract instructions.
   bool hasExtractBitsInsn() const { return HasExtractBitsInsn; }
 
   /// Return the preferred vector type legalization action.
   virtual TargetLoweringBase::LegalizeTypeAction
   getPreferredVectorAction(MVT VT) const {
     // The default action for one element vectors is to scalarize
-    if (VT.getVectorElementCount() == 1)
+    if (VT.getVectorElementCount().isScalar())
       return TypeScalarizeVector;
     // The default action for an odd-width vector is to widen.
     if (!VT.isPow2VectorType())
       return TypeWidenVector;
     // The default action for other vectors is to promote
     return TypePromoteInteger;
   }
 

llvm/include/llvm/CodeGen/ValueTypes.h

public:
     }
 
     /// Return true if this is an overloaded type for TableGen.
     bool isOverloaded() const {
       return (V==MVT::iAny || V==MVT::fAny || V==MVT::vAny || V==MVT::iPTRAny);
     }
 
     /// Return true if the bit size is a multiple of 8.
-    bool isByteSized() const {
-      return getSizeInBits().isByteSized();
-    }
+    bool isByteSized() const { return getSizeInBits().isKnownMultipleOf(8); }
 
     /// Return true if the size is a power-of-two number of bytes.
     bool isRound() const {
       if (isScalableVector())
         return false;
       unsigned BitSize = getSizeInBits();
       return BitSize >= 8 && !(BitSize & (BitSize - 1));
     }

llvm/include/llvm/IR/Intrinsics.h

struct IITDescriptor {
     static IITDescriptor get(IITDescriptorKind K, unsigned short Hi,
                              unsigned short Lo) {
       unsigned Field = Hi << 16 | Lo;
       IITDescriptor Result = {K, {Field}};
       return Result;
     }
 
     static IITDescriptor getVector(unsigned Width, bool IsScalable) {
-      IITDescriptor Result;
-      Result.Kind = Vector;
+      IITDescriptor Result = {Vector, 0};
       Result.Vector_Width = ElementCount::get(Width, IsScalable);
       return Result;
     }
   };
 
   /// Return the IIT table descriptor for the specified intrinsic into an array
   /// of IITDescriptors.
   void getIntrinsicInfoTableEntries(ID id, SmallVectorImpl<IITDescriptor> &T);

llvm/include/llvm/Support/MachineValueType.h

public:
     /// be set and the runtime size will be a positive integer multiple of the
     /// base size.
     TypeSize getStoreSizeInBits() const {
       return getStoreSize() * 8;
     }
 
     /// Returns true if the number of bits for the type is a multiple of an
     /// 8-bit byte.
-    bool isByteSized() const {
-      return getSizeInBits().isByteSized();
-    }
+    bool isByteSized() const { return getSizeInBits().isKnownMultipleOf(8); }
 
     /// Return true if we know at compile time this has more bits than VT.
     bool knownBitsGT(MVT VT) const {
       return TypeSize::isKnownGT(getSizeInBits(), VT.getSizeInBits());
     }
 
     /// Return true if we know at compile time this has more than or the same
     /// bits as VT.

llvm/include/llvm/Support/TypeSize.h

 
 #include <cstdint>
 #include <cassert>
 
 namespace llvm {
 
 template <typename T> struct DenseMapInfo;
 
-class ElementCount {
-private:
-  unsigned Min;  // Minimum number of vector elements.
-  bool Scalable; // If true, NumElements is a multiple of 'Min' determined
-                 // at runtime rather than compile time.
-
-  /// Prevent code from using initializer-list contructors like
-  /// ElementCount EC = {<unsigned>, <bool>}. The static `get*`
-  /// methods below are preferred, as users should always make a
-  /// conscious choice on the type of `ElementCount` they are
-  /// requesting.
-  ElementCount(unsigned Min, bool Scalable) : Min(Min), Scalable(Scalable) {}
+// TODO: This class will be redesigned in a later patch that introduces full
+// polynomial behaviour, i.e. the ability to have composites made up of both
+// fixed and scalable sizes.
+template <typename T> class PolySize {
+protected:
+  T MinVal;        // The minimum value that it could be.
+  bool IsScalable; // If true, the total value is determined by multiplying
+                   // 'MinVal' by a runtime determinded quantity, 'vscale'.
 
-public:
-  ElementCount() = default;
-
-  ElementCount operator*(unsigned RHS) {
-    return { Min * RHS, Scalable };
-  }
-
-  friend ElementCount operator-(const ElementCount &LHS,
-                                const ElementCount &RHS) {
-    assert(LHS.Scalable == RHS.Scalable &&
-           "Arithmetic using mixed scalable and fixed types");
-    return {LHS.Min - RHS.Min, LHS.Scalable};
-  }
-
-  bool operator==(const ElementCount& RHS) const {
-    return Min == RHS.Min && Scalable == RHS.Scalable;
-  }
-  bool operator!=(const ElementCount& RHS) const {
-    return !(*this == RHS);
-  }
-  bool operator==(unsigned RHS) const { return Min == RHS && !Scalable; }
-  bool operator!=(unsigned RHS) const { return !(*this == RHS); }
-
-  ElementCount &operator*=(unsigned RHS) {
-    Min *= RHS;
-    return *this;
-  }
-
-  /// We do not provide the '/' operator here because division for polynomial
-  /// types does not work in the same way as for normal integer types. We can
-  /// only divide the minimum value (or coefficient) by RHS, which is not the
-  /// same as
-  ///   (Min * Vscale) / RHS
-  /// The caller is recommended to use this function in combination with
-  /// isKnownMultipleOf(RHS), which lets the caller know if it's possible to
-  /// perform a lossless divide by RHS.
-  ElementCount divideCoefficientBy(unsigned RHS) const {
-    return ElementCount(Min / RHS, Scalable);
-  }
+  constexpr PolySize(T MinVal, bool IsScalable)
+      : MinVal(MinVal), IsScalable(IsScalable) {}
 
-  ElementCount NextPowerOf2() const {
-    return {(unsigned)llvm::NextPowerOf2(Min), Scalable};
-  }
+public:
 
-  /// This function tells the caller whether the element count is known at
-  /// compile time to be a multiple of the scalar value RHS.
-  bool isKnownMultipleOf(unsigned RHS) const {
-    return Min % RHS == 0;
+  static constexpr PolySize getFixed(T MinVal) { return {MinVal, false}; }
+  static constexpr PolySize getScalable(T MinVal) { return {MinVal, true}; }
+  static constexpr PolySize get(T MinVal, bool IsScalable) {
+    return {MinVal, IsScalable};
   }
 
-  static ElementCount getFixed(unsigned Min) { return {Min, false}; }
-  static ElementCount getScalable(unsigned Min) { return {Min, true}; }
-  static ElementCount get(unsigned Min, bool Scalable) {
-    return {Min, Scalable};
-  }
+  static constexpr PolySize getNull() { return {0, false}; }
 
-  /// Printing function.
-  void print(raw_ostream &OS) const {
-    if (Scalable)
-      OS << "vscale x ";
-    OS << Min;
-  }
   /// Counting predicates.
   ///
-  /// Notice that Min = 1 and Scalable = true is considered more than
-  /// one element.
-  ///
   ///@{ No elements..
-  bool isZero() const { return Min == 0; }
+  bool isZero() const { return MinVal == 0; }
   /// At least one element.
-  bool isNonZero() const { return Min != 0; }
+  bool isNonZero() const { return MinVal != 0; }
   /// A return value of true indicates we know at compile time that the number
   /// of elements (vscale * Min) is definitely even. However, returning false
   /// does not guarantee that the total number of elements is odd.
-  bool isKnownEven() const { return (Min & 0x1) == 0; }
-  /// Exactly one element.
-  bool isScalar() const { return !Scalable && Min == 1; }
-  /// One or more elements.
-  bool isVector() const { return (Scalable && Min != 0) || Min > 1; }
+  bool isKnownEven() const { return (MinVal & 0x1) == 0; }
   ///@}
 
-  unsigned getKnownMinValue() const { return Min; }
+  T getKnownMinValue() const { return MinVal; }
 
   // Return the minimum value with the assumption that the count is exact.
   // Use in places where a scalable count doesn't make sense (e.g. non-vector
   // types, or vectors in backends which don't support scalable vectors).
-  unsigned getFixedValue() const {
-    assert(!Scalable &&
+  T getFixedValue() const {
+    assert(!IsScalable &&
            "Request for a fixed element count on a scalable object");
-    return Min;
-  }
-
-  bool isScalable() const { return Scalable; }
-};
-
-/// Stream operator function for `ElementCount`.
-inline raw_ostream &operator<<(raw_ostream &OS, const ElementCount &EC) {
-  EC.print(OS);
-  return OS;
-}
-
-// This class is used to represent the size of types. If the type is of fixed
-// size, it will represent the exact size. If the type is a scalable vector,
-// it will represent the known minimum size.
-class TypeSize {
-  uint64_t MinSize;   // The known minimum size.
-  bool IsScalable;    // If true, then the runtime size is an integer multiple
-                      // of MinSize.
-
-public:
-  constexpr TypeSize(uint64_t MinSize, bool Scalable)
-    : MinSize(MinSize), IsScalable(Scalable) {}
-
-  static constexpr TypeSize Fixed(uint64_t Size) {
-    return TypeSize(Size, /*Scalable=*/false);
+    return MinVal;
   }
 
-  static constexpr TypeSize Scalable(uint64_t MinSize) {
-    return TypeSize(MinSize, /*Scalable=*/true);
-  }
+  bool isScalable() const { return IsScalable; }
 
-  // Scalable vector types with the same minimum size as a fixed size type are
-  // not guaranteed to be the same size at runtime, so they are never
-  // considered to be equal.
-  bool operator==(const TypeSize &RHS) const {
-    return MinSize == RHS.MinSize && IsScalable == RHS.IsScalable;
+  bool operator==(const PolySize &RHS) const {
+    return MinVal == RHS.MinVal && IsScalable == RHS.IsScalable;
   }
 
-  bool operator!=(const TypeSize &RHS) const { return !(*this == RHS); }
+  bool operator!=(const PolySize &RHS) const { return !(*this == RHS); }
 
   // For some cases, size ordering between scalable and fixed size types cannot
   // be determined at compile time, so such comparisons aren't allowed.
   //
   // e.g. <vscale x 2 x i16> could be bigger than <4 x i32> with a runtime
   // vscale >= 5, equal sized with a vscale of 4, and smaller with
   // a vscale <= 3.
   //
   // All the functions below make use of the fact vscale is always >= 1, which
   // means that <vscale x 4 x i32> is guaranteed to be >= <4 x i32>, etc.
 
-  static bool isKnownLT(const TypeSize &LHS, const TypeSize &RHS) {
+  static bool isKnownLT(const PolySize &LHS, const PolySize &RHS) {
     if (!LHS.IsScalable || RHS.IsScalable)
-      return LHS.MinSize < RHS.MinSize;
+      return LHS.MinVal < RHS.MinVal;
 
     // LHS.IsScalable = true, RHS.IsScalable = false
     return false;
   }
 
-  static bool isKnownGT(const TypeSize &LHS, const TypeSize &RHS) {
+  static bool isKnownGT(const PolySize &LHS, const PolySize &RHS) {
     if (LHS.IsScalable || !RHS.IsScalable)
-      return LHS.MinSize > RHS.MinSize;
+      return LHS.MinVal > RHS.MinVal;
 
     // LHS.IsScalable = false, RHS.IsScalable = true
     return false;
   }
 
-  static bool isKnownLE(const TypeSize &LHS, const TypeSize &RHS) {
+  static bool isKnownLE(const PolySize &LHS, const PolySize &RHS) {
     if (!LHS.IsScalable || RHS.IsScalable)
-      return LHS.MinSize <= RHS.MinSize;
+      return LHS.MinVal <= RHS.MinVal;
 
     // LHS.IsScalable = true, RHS.IsScalable = false
     return false;
   }
 
-  static bool isKnownGE(const TypeSize &LHS, const TypeSize &RHS) {
+  static bool isKnownGE(const PolySize &LHS, const PolySize &RHS) {
     if (LHS.IsScalable || !RHS.IsScalable)
-      return LHS.MinSize >= RHS.MinSize;
+      return LHS.MinVal >= RHS.MinVal;
 
     // LHS.IsScalable = false, RHS.IsScalable = true
     return false;
   }
 
+  PolySize operator*(T RHS) { return {MinVal * RHS, IsScalable}; }
+
+  PolySize &operator*=(T RHS) {
+    MinVal *= RHS;
+    return *this;
+  }
+
+  friend PolySize operator-(const PolySize &LHS, const PolySize &RHS) {
+    assert(LHS.IsScalable == RHS.IsScalable &&
+           "Arithmetic using mixed scalable and fixed types");
+    return {LHS.MinVal - RHS.MinVal, LHS.IsScalable};
+  }
+
+  /// This function tells the caller whether the element count is known at
+  /// compile time to be a multiple of the scalar value RHS.
+  bool isKnownMultipleOf(T RHS) const { return MinVal % RHS == 0; }
+
+  /// We do not provide the '/' operator here because division for polynomial
+  /// types does not work in the same way as for normal integer types. We can
+  /// only divide the minimum value (or coefficient) by RHS, which is not the
+  /// same as
+  ///   (Min * Vscale) / RHS
+  /// The caller is recommended to use this function in combination with
+  /// isKnownMultipleOf(RHS), which lets the caller know if it's possible to
+  /// perform a lossless divide by RHS.
+  PolySize divideCoefficientBy(T RHS) const {
+    return PolySize(MinVal / RHS, IsScalable);
+  }
+
+  PolySize coefficientNextPowerOf2() const {
+    return PolySize((T)llvm::NextPowerOf2(MinVal), IsScalable);
+  }
+
+  /// Printing function.
+  void print(raw_ostream &OS) const {
+    if (IsScalable)
+      OS << "vscale x ";
+    OS << MinVal;
+  }
+};
+
+/// Stream operator function for `PolySize`.
+template <typename T>
+inline raw_ostream &operator<<(raw_ostream &OS, const PolySize<T> &PS) {
+  PS.print(OS);
+  return OS;
+}
+
+class ElementCount : public PolySize<unsigned> {
+public:
+
+  constexpr ElementCount(PolySize<unsigned> V) : PolySize(V) {}
+
+  /// Counting predicates.
+  ///
+  /// Notice that MinVal = 1 and IsScalable = true is considered more than
+  /// one element.
+  ///
+  ///@{ No elements..
+  /// Exactly one element.
+  bool isScalar() const { return !IsScalable && MinVal == 1; }
+  /// One or more elements.
+  bool isVector() const { return (IsScalable && MinVal != 0) || MinVal > 1; }
+  ///@}
+};
+
+// This class is used to represent the size of types. If the type is of fixed
+// size, it will represent the exact size. If the type is a scalable vector,
+// it will represent the known minimum size.
+class TypeSize : public PolySize<uint64_t> {
+public:
+  constexpr TypeSize(PolySize<uint64_t> V) : PolySize(V) {}
+
+  constexpr TypeSize(uint64_t MinVal, bool IsScalable)
+      : PolySize(MinVal, IsScalable) {}
+
+  static constexpr TypeSize Fixed(uint64_t MinVal) {
+    return TypeSize(MinVal, false);
+  }
+  static constexpr TypeSize Scalable(uint64_t MinVal) {
+    return TypeSize(MinVal, true);
+  }
+
+  uint64_t getFixedSize() const { return getFixedValue(); }
+  uint64_t getKnownMinSize() const { return getKnownMinValue(); }
+
   friend bool operator<(const TypeSize &LHS, const TypeSize &RHS) {
     assert(LHS.IsScalable == RHS.IsScalable &&
            "Ordering comparison of scalable and fixed types");
 
-    return LHS.MinSize < RHS.MinSize;
+    return LHS.MinVal < RHS.MinVal;
   }
 
   friend bool operator>(const TypeSize &LHS, const TypeSize &RHS) {
     return RHS < LHS;
   }
 
   friend bool operator<=(const TypeSize &LHS, const TypeSize &RHS) {
     return !(RHS < LHS);
   }
 
   friend bool operator>=(const TypeSize &LHS, const TypeSize& RHS) {
     return !(LHS < RHS);
   }
 
-  // Convenience operators to obtain relative sizes independently of
-  // the scalable flag.
-  TypeSize operator*(unsigned RHS) const {
-    return { MinSize * RHS, IsScalable };
-  }
-
-  friend TypeSize operator*(const unsigned LHS, const TypeSize &RHS) {
-    return { LHS * RHS.MinSize, RHS.IsScalable };
-  }
-
-  /// We do not provide the '/' operator here because division for polynomial
-  /// types does not work in the same way as for normal integer types. We can
-  /// only divide the minimum value (or coefficient) by RHS, which is not the
-  /// same as
-  ///   (MinSize * Vscale) / RHS
-  /// The caller is recommended to use this function in combination with
-  /// isKnownMultipleOf(RHS), which lets the caller know if it's possible to
-  /// perform a lossless divide by RHS.
-  TypeSize divideCoefficientBy(uint64_t RHS) const {
-    return {MinSize / RHS, IsScalable};
-  }
-
   TypeSize &operator-=(TypeSize RHS) {
     assert(IsScalable == RHS.IsScalable &&
            "Subtraction using mixed scalable and fixed types");
-    MinSize -= RHS.MinSize;
+    MinVal -= RHS.MinVal;
     return *this;
   }
 
   TypeSize &operator+=(TypeSize RHS) {
     assert(IsScalable == RHS.IsScalable &&
            "Addition using mixed scalable and fixed types");
-    MinSize += RHS.MinSize;
+    MinVal += RHS.MinVal;
     return *this;
   }
 
   friend TypeSize operator-(const TypeSize &LHS, const TypeSize &RHS) {
     assert(LHS.IsScalable == RHS.IsScalable &&
            "Arithmetic using mixed scalable and fixed types");
-    return {LHS.MinSize - RHS.MinSize, LHS.IsScalable};
-  }
-
-  // Return the minimum size with the assumption that the size is exact.
-  // Use in places where a scalable size doesn't make sense (e.g. non-vector
-  // types, or vectors in backends which don't support scalable vectors).
-  uint64_t getFixedSize() const {
-    assert(!IsScalable && "Request for a fixed size on a scalable object");
-    return MinSize;
-  }
-
-  // Return the known minimum size. Use in places where the scalable property
-  // doesn't matter (e.g. determining alignment) or in conjunction with the
-  // isScalable method below.
-  uint64_t getKnownMinSize() const {
-    return MinSize;
-  }
-
-  // Return whether or not the size is scalable.
-  bool isScalable() const {
-    return IsScalable;
-  }
-
-  // Returns true if the number of bits is a multiple of an 8-bit byte.
-  bool isByteSized() const {
-    return (MinSize & 7) == 0;
+    return {LHS.MinVal - RHS.MinVal, LHS.IsScalable};
   }
 
-  // Returns true if the type size is non-zero.
-  bool isNonZero() const { return MinSize != 0; }
-
-  // Returns true if the type size is zero.
-  bool isZero() const { return MinSize == 0; }
-
-  /// This function tells the caller whether the type size is known at
-  /// compile time to be a multiple of the scalar value RHS.
-  bool isKnownMultipleOf(uint64_t RHS) const { return MinSize % RHS == 0; }
-
   // Casts to a uint64_t if this is a fixed-width size.
   //
   // This interface is deprecated and will be removed in a future version
   // of LLVM in favour of upgrading uses that rely on this implicit conversion
   // to uint64_t. Calls to functions that return a TypeSize should use the
   // proper interfaces to TypeSize.
   // In practice this is mostly calls to MVT/EVT::getSizeInBits().
   //
   // To determine how to upgrade the code:
   //
   //   if (<algorithm works for both scalable and fixed-width vectors>)
-  //     use getKnownMinSize()
+  //     use getKnownMinValue()
   //   else if (<algorithm works only for fixed-width vectors>) {
   //     if <algorithm can be adapted for both scalable and fixed-width vectors>
-  //       update the algorithm and use getKnownMinSize()
+  //       update the algorithm and use getKnownMinValue()
   //     else
-  //       bail out early for scalable vectors and use getFixedSize()
+  //       bail out early for scalable vectors and use getFixedValue()
   //   }
   operator uint64_t() const {
 #ifdef STRICT_FIXED_SIZE_VECTORS
-    return getFixedSize();
+    return getFixedValue();
 #else
     if (isScalable())
       WithColor::warning() << "Compiler has made implicit assumption that "
                               "TypeSize is not scalable. This may or may not "
                               "lead to broken code.\n";
-    return getKnownMinSize();
+    return getKnownMinValue();
 #endif
   }
 
+  // Convenience operators to obtain relative sizes independently of
+  // the scalable flag.
+  TypeSize operator*(unsigned RHS) const { return {MinVal * RHS, IsScalable}; }
+
+  friend TypeSize operator*(const unsigned LHS, const TypeSize &RHS) {
+    return {LHS * RHS.MinVal, RHS.IsScalable};
+  }
+
   // Additional convenience operators needed to avoid ambiguous parses.
   // TODO: Make uint64_t the default operator?
-  TypeSize operator*(uint64_t RHS) const {
-    return { MinSize * RHS, IsScalable };
-  }
+  TypeSize operator*(uint64_t RHS) const { return {MinVal * RHS, IsScalable}; }
 
-  TypeSize operator*(int RHS) const {
-    return { MinSize * RHS, IsScalable };
-  }
+  TypeSize operator*(int RHS) const { return {MinVal * RHS, IsScalable}; }
 
-  TypeSize operator*(int64_t RHS) const {
-    return { MinSize * RHS, IsScalable };
-  }
+  TypeSize operator*(int64_t RHS) const { return {MinVal * RHS, IsScalable}; }
 
   friend TypeSize operator*(const uint64_t LHS, const TypeSize &RHS) {
-    return { LHS * RHS.MinSize, RHS.IsScalable };
+    return {LHS * RHS.MinVal, RHS.IsScalable};
   }
 
   friend TypeSize operator*(const int LHS, const TypeSize &RHS) {
-    return { LHS * RHS.MinSize, RHS.IsScalable };
+    return {LHS * RHS.MinVal, RHS.IsScalable};
   }
 
   friend TypeSize operator*(const int64_t LHS, const TypeSize &RHS) {
-    return { LHS * RHS.MinSize, RHS.IsScalable };
-  }
-
-  TypeSize NextPowerOf2() const {
-    return TypeSize(llvm::NextPowerOf2(MinSize), IsScalable);
+    return {LHS * RHS.MinVal, RHS.IsScalable};
   }
 };
 
 /// Returns a TypeSize with a known minimum size that is the next integer
 /// (mod 2**64) that is greater than or equal to \p Value and is a multiple
 /// of \p Align. \p Align must be non-zero.
 ///
 /// Similar to the alignTo functions in MathExtras.h
 inline TypeSize alignTo(TypeSize Size, uint64_t Align) {
   assert(Align != 0u && "Align must be non-zero");
-  return {(Size.getKnownMinSize() + Align - 1) / Align * Align,
+  return {(Size.getKnownMinValue() + Align - 1) / Align * Align,
           Size.isScalable()};
 }
 
 template <> struct DenseMapInfo<ElementCount> {
   static inline ElementCount getEmptyKey() {
     return ElementCount::getScalable(~0U);
   }
   static inline ElementCount getTombstoneKey() {

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

if ((DestNumElts % SrcNumElts) == 0) {
               EVT::getVectorVT(*DAG.getContext(), ScalarVT, NewExtEC);
           if (TLI.isOperationLegalOrCustom(ISD::EXTRACT_SUBVECTOR, NewExtVT)) {
             SDValue NewIndex = DAG.getVectorIdxConstant(IndexValScaled, DL);
             SDValue NewExtract =
                 DAG.getNode(ISD::EXTRACT_SUBVECTOR, DL, NewExtVT,
                             V.getOperand(0), NewIndex);
             return DAG.getBitcast(NVT, NewExtract);
           }
-          if (NewExtEC == 1 &&
+          if (NewExtEC.isScalar() &&
               TLI.isOperationLegalOrCustom(ISD::EXTRACT_VECTOR_ELT, ScalarVT)) {
             SDValue NewIndex = DAG.getVectorIdxConstant(IndexValScaled, DL);
             SDValue NewExtract =
                 DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, ScalarVT,
                             V.getOperand(0), NewIndex);
             return DAG.getBitcast(NVT, NewExtract);
           }
         }

llvm/lib/CodeGen/TargetLoweringBase.cpp

return LegalizeKind(TypeExpandInteger,
                         EVT::getIntegerVT(Context, VT.getSizeInBits() / 2));
   }
 
   // Handle vector types.
   ElementCount NumElts = VT.getVectorElementCount();
   EVT EltVT = VT.getVectorElementType();
 
   // Vectors with only one element are always scalarized.
-  if (NumElts == 1)
+  if (NumElts.isScalar())
     return LegalizeKind(TypeScalarizeVector, EltVT);
 
   if (VT.getVectorElementCount() == ElementCount::getScalable(1))
     report_fatal_error("Cannot legalize this vector");
 
   // Try to widen vector elements until the element type is a power of two and
   // promote it to a legal type later on, for example:
   // <3 x i8> -> <4 x i8> -> <4 x i32>
   if (EltVT.isInteger()) {
     // Vectors with a number of elements that is not a power of two are always
     // widened, for example <3 x i8> -> <4 x i8>.
     if (!VT.isPow2VectorType()) {
-      NumElts = NumElts.NextPowerOf2();
+      NumElts = NumElts.coefficientNextPowerOf2();
       EVT NVT = EVT::getVectorVT(Context, EltVT, NumElts);
       return LegalizeKind(TypeWidenVector, NVT);
     }
 
     // Examine the element type.
     LegalizeKind LK = getTypeConversion(Context, EltVT);
 
     // If type is to be expanded, split the vector.
if (EltVT.isInteger()) {
     // type with a promoted vector element type.
     EltVT = OldEltVT;
   }
 
   // Try to widen the vector until a legal type is found.
   // If there is no wider legal type, split the vector.
   while (true) {
     // Round up to the next power of 2.
-    NumElts = NumElts.NextPowerOf2();
+    NumElts = NumElts.coefficientNextPowerOf2();
 
     // If there is no simple vector type with this many elements then there
     // cannot be a larger legal vector type.  Note that this assumes that
     // there are no skipped intermediate vector types in the simple types.
     if (!EltVT.isSimple())
       break;
     MVT LargerVector = MVT::getVectorVT(EltVT.getSimpleVT(), NumElts);
     if (LargerVector == MVT())
unsigned TargetLoweringBase::getVectorTypeBreakdown(LLVMContext &Context, EVT VT,
 
   MVT DestVT = getRegisterType(Context, NewVT);
   RegisterVT = DestVT;
 
   if (EVT(DestVT).bitsLT(NewVT)) {  // Value is expanded, e.g. i64 -> i16.
     TypeSize NewVTSize = NewVT.getSizeInBits();
     // Convert sizes such as i33 to i64.
     if (!isPowerOf2_32(NewVTSize.getKnownMinSize()))
-      NewVTSize = NewVTSize.NextPowerOf2();
+      NewVTSize = NewVTSize.coefficientNextPowerOf2();
     return NumVectorRegs*(NewVTSize/DestVT.getSizeInBits());
   }
 
   // Otherwise, promotion or legal types use the same number of registers as
   // the vector decimated to the appropriate level.
   return NumVectorRegs;
 }
 

llvm/lib/Support/LowLevelType.cpp

 LLT::LLT(MVT VT) {
   if (VT.isVector()) {
     init(/*IsPointer=*/false, VT.getVectorNumElements() > 1,
          VT.getVectorNumElements(), VT.getVectorElementType().getSizeInBits(),
          /*AddressSpace=*/0);
   } else if (VT.isValid()) {
     // Aggregates are no different from real scalars as far as GlobalISel is
     // concerned.
-    assert(VT.getSizeInBits() != 0 && "invalid zero-sized type");
+    assert(VT.getSizeInBits().isNonZero() && "invalid zero-sized type");
     init(/*IsPointer=*/false, /*IsVector=*/false, /*NumElements=*/0,
          VT.getSizeInBits(), /*AddressSpace=*/0);
   } else {
     IsPointer = false;
     IsVector = false;
     RawData = 0;
   }
 }

llvm/lib/Target/X86/X86InterleavedAccess.cpp

if (VecLength == 768 || VecLength == 1536) {
     VecBasePtr = Builder.CreateBitCast(LI->getPointerOperand(), VecBasePtrTy);
     NumLoads = NumSubVectors * (VecLength / 384);
   } else {
     VecBaseTy = SubVecTy;
     VecBasePtrTy = VecBaseTy->getPointerTo(LI->getPointerAddressSpace());
     VecBasePtr = Builder.CreateBitCast(LI->getPointerOperand(), VecBasePtrTy);
   }
   // Generate N loads of T type.
-  assert(VecBaseTy->getPrimitiveSizeInBits().isByteSized() &&
+  assert(VecBaseTy->getPrimitiveSizeInBits().isKnownMultipleOf(8) &&
          "VecBaseTy's size must be a multiple of 8");
   const Align FirstAlignment = LI->getAlign();
   const Align SubsequentAlignment = commonAlignment(
       FirstAlignment, VecBaseTy->getPrimitiveSizeInBits().getFixedSize() / 8);
   Align Alignment = FirstAlignment;
   for (unsigned i = 0; i < NumLoads; i++) {
     // TODO: Support inbounds GEP.
     Value *NewBasePtr =

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

 /// type is irregular if its allocated size doesn't equal the store size of an
 /// element of the corresponding vector type at the given vectorization factor.
 static bool hasIrregularType(Type *Ty, const DataLayout &DL, ElementCount VF) {
   assert(!VF.isScalable() && "scalable vectors not yet supported.");
   // Determine if an array of VF elements of type Ty is "bitcast compatible"
   // with a <VF x Ty> vector.
   if (VF.isVector()) {
     auto *VectorTy = VectorType::get(Ty, VF);
-    return VF * DL.getTypeAllocSize(Ty) != DL.getTypeStoreSize(VectorTy);
+    return TypeSize::get(VF.getKnownMinValue() *
+                             DL.getTypeAllocSize(Ty).getFixedValue(),
+                         VF.isScalable()) != DL.getTypeStoreSize(VectorTy);
   }
 
   // If the vectorization factor is one, we just check if an array of type Ty
   // requires padding between elements.
   return DL.getTypeAllocSizeInBits(Ty) != DL.getTypeSizeInBits(Ty);
 }
 
 /// A helper function that returns the reciprocal of the block probability of
Value *InnerLoopVectorizer::getOrCreateVectorValue(Value *V, unsigned Part) {
   if (VectorLoopValueMap.hasAnyScalarValue(V)) {
     Value *ScalarValue = VectorLoopValueMap.getScalarValue(V, {Part, 0});
 
     // If we've scalarized a value, that value should be an instruction.
     auto *I = cast<Instruction>(V);
 
     // If we aren't vectorizing, we can just copy the scalar map values over to
     // the vector map.
-    if (VF == 1) {
+    if (VF.isScalar()) {
       VectorLoopValueMap.setVectorValue(V, Part, ScalarValue);
       return ScalarValue;
     }
 
     // Get the last scalar instruction we generated for V and Part. If the value
     // is known to be uniform after vectorization, this corresponds to lane zero
     // of the Part unroll iteration. Otherwise, the last instruction is the one
     // we created for the last vector lane of the Part unroll iteration.
if (VectorLoopValueMap.hasScalarValue(V, Instance))
     return VectorLoopValueMap.getScalarValue(V, Instance);
 
   // If the value has not been scalarized, get its entry in VectorLoopValueMap
   // for the given unroll part. If this entry is not a vector type (i.e., the
   // vectorization factor is one), there is no need to generate an
   // extractelement instruction.
   auto *U = getOrCreateVectorValue(V, Instance.Part);
   if (!U->getType()->isVectorTy()) {
-    assert(VF == 1 && "Value not scalarized has non-vector type");
+    assert(VF.isScalar() && "Value not scalarized has non-vector type");
     return U;
   }
 
   // Otherwise, the value from the original loop has been vectorized and is
   // represented by UF vector values. Extract and return the requested scalar
   // value from the appropriate vector lane.
   return Builder.CreateExtractElement(U, Builder.getInt32(Instance.Lane));
 }
void InnerLoopVectorizer::fixReduction(PHINode *Phi) {
 
   // Find the reduction identity variable. Zero for addition, or, xor,
   // one for multiplication, -1 for And.
   Value *Identity;
   Value *VectorStart;
   if (RK == RecurrenceDescriptor::RK_IntegerMinMax ||
       RK == RecurrenceDescriptor::RK_FloatMinMax) {
     // MinMax reduction have the start value as their identify.
-    if (VF == 1 || IsInLoopReductionPhi) {
+    if (VF.isScalar() || IsInLoopReductionPhi) {
       VectorStart = Identity = ReductionStartValue;
     } else {
       VectorStart = Identity =
         Builder.CreateVectorSplat(VF, ReductionStartValue, "minmax.ident");
     }
   } else {
     // Handle other reduction kinds:
     Constant *Iden = RecurrenceDescriptor::getRecurrenceIdentity(
         RK, VecTy->getScalarType());
-    if (VF == 1 || IsInLoopReductionPhi) {
+    if (VF.isScalar() || IsInLoopReductionPhi) {
       Identity = Iden;
       // This vector is the Identity vector where the first element is the
       // incoming scalar reduction.
       VectorStart = ReductionStartValue;
     } else {
       Identity = ConstantVector::getSplat(VF, Iden);
 
       // This vector is the Identity vector where the first element is the
for (unsigned Part = 0; Part < UF; ++Part) {
 
       // Create the new GEP. Note that this GEP may be a scalar if VF == 1,
       // but it should be a vector, otherwise.
       auto *NewGEP =
           GEP->isInBounds()
               ? Builder.CreateInBoundsGEP(GEP->getSourceElementType(), Ptr,
                                           Indices)
               : Builder.CreateGEP(GEP->getSourceElementType(), Ptr, Indices);
-      assert((VF == 1 || NewGEP->getType()->isVectorTy()) &&
+      assert((VF.isScalar() || NewGEP->getType()->isVectorTy()) &&
              "NewGEP is not a pointer vector");
       VectorLoopValueMap.setVectorValue(GEP, Part, NewGEP);
       addMetadata(NewGEP, GEP);
     }
   }
 }
 
 void InnerLoopVectorizer::widenPHIInstruction(Instruction *PN, unsigned UF,
#endif /* NDEBUG */
   bool VectorizeLoop = true, InterleaveLoop = true;
   if (Requirements.doesNotMeet(F, L, Hints)) {
     LLVM_DEBUG(dbgs() << "LV: Not vectorizing: loop did not meet vectorization "
                          "requirements.\n");
     Hints.emitRemarkWithHints();
     return false;
   }
 
-  if (VF.Width == 1) {
+  if (VF.Width.isScalar()) {
     LLVM_DEBUG(dbgs() << "LV: Vectorization is possible but not beneficial.\n");
     VecDiagMsg = std::make_pair(
         "VectorizationNotBeneficial",
         "the cost-model indicates that vectorization is not beneficial");
     VectorizeLoop = false;
   }
 
   if (!MaybeVF && UserIC > 1) {

llvm/lib/Transforms/Vectorize/VPlan.cpp

void VPlan::execute(VPTransformState *State) {
   // -1. Check if the backedge taken count is needed, and if so build it.
   if (BackedgeTakenCount && BackedgeTakenCount->getNumUsers()) {
     Value *TC = State->TripCount;
     IRBuilder<> Builder(State->CFG.PrevBB->getTerminator());
     auto *TCMO = Builder.CreateSub(TC, ConstantInt::get(TC->getType(), 1),
                                    "trip.count.minus.1");
     auto VF = State->VF;
     Value *VTCMO =
-        VF == 1 ? TCMO : Builder.CreateVectorSplat(VF, TCMO, "broadcast");
+        VF.isScalar() ? TCMO : Builder.CreateVectorSplat(VF, TCMO, "broadcast");
     for (unsigned Part = 0, UF = State->UF; Part < UF; ++Part)
       State->set(BackedgeTakenCount, VTCMO, Part);
   }
 
   // 0. Set the reverse mapping from VPValues to Values for code generation.
   for (auto &Entry : Value2VPValue)
     State->VPValue2Value[Entry.second] = Entry.first;
 
Value *VStart = VF.isScalar()
                                                   CanonicalIV, "broadcast");
   for (unsigned Part = 0, UF = State.UF; Part < UF; ++Part) {
     SmallVector<Constant *, 8> Indices;
     for (unsigned Lane = 0; Lane < VF.getKnownMinValue(); ++Lane)
       Indices.push_back(
           ConstantInt::get(STy, Part * VF.getKnownMinValue() + Lane));
     // If VF == 1, there is only one iteration in the loop above, thus the
     // element pushed back into Indices is ConstantInt::get(STy, Part)
-    Constant *VStep = VF == 1 ? Indices.back() : ConstantVector::get(Indices);
+    Constant *VStep =
+        VF.isScalar() ? Indices.back() : ConstantVector::get(Indices);
     // Add the consecutive indices to the vector value.
     Value *CanonicalVectorIV = Builder.CreateAdd(VStart, VStep, "vec.iv");
     State.set(getVPValue(), CanonicalVectorIV, Part);
   }
 }
 
 void VPWidenCanonicalIVRecipe::print(raw_ostream &O, const Twine &Indent,
                                      VPSlotTracker &SlotTracker) const {