[SVE] MVT scalable size queries

[huntergr]

Summary

Size queries for MVTs, split out from D53137.

Contains a fix for FindMemType to avoid using scalable vector types to contain non-scalable types.

[cameron.mcinally]

LGTM

llvm/include/llvm/CodeGen/ValueTypes.h

294

Nit-picky: would it make sense to add an explicit !scalable assert to all these functions?

I see that this particular function is guarded by the asserts in getSizeInBits() and getExtendedSizeInBits(), but there's really no guarantee that future modifications will preserve this behavior.

To be clear, I don't feel strongly about this. Just thinking aloud...

[huntergr]

In D66871#1649148, @cameron.mcinally wrote:

LGTM

Thanks for the review; there's still D53137 and D66339 as prerequisites so I won't commit this yet.

llvm/include/llvm/CodeGen/ValueTypes.h

294

I can certainly add asserts more widely to make it obvious instead of relying on lower functions to assert and just commenting on it.

[huntergr]

• Refactored to match the same approach used in D53137 -- TypeSize returned for all size queries instead of introducing separate interfaces and relying on implicit conversion to be (mostly) compatible with existing code.
• Added 'isByteSized()' interface to TypeSize.
• Minor bugfix to DAGCombiner::ForwardStoreValueToDirectLoad, where implicit conversions allowed an unsigned number to wrap and then be converted to a larger signed integer; with TypeSize using uint64_t instead, this showed up during a make check-all run with ubsan active. Can't write a test for that one since the transform bailed out when Offset was non-zero.

[rovka]

llvm/include/llvm/CodeGen/ValueTypes.h

319

That's a weird change...

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

520

Why does LE care about isVector and LT doesn't?

In any case, I think both functions would benefit from being rewritten using std::tie and the corresponding operators.

[huntergr]

llvm/include/llvm/CodeGen/ValueTypes.h

319

I think that was a copy-paste from elsewhere. Will revert.

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

520

LT seems to be used with filtered min/max functions below and will only operate on either only vector or only scalar MVTs, so never compares between them.

LE is then explicitly used to remove MVTs from Typesets that may have mixed scalar and vector MVTs, so needs to avoid comparing them.

I'll try reworking them with std::tie.

[sdesmalen]

llvm/lib/Target/AArch64/AArch64StackOffset.h

59

Should this be more explicit here?
e.g.

const TypeSize &Size = Other.second.getSizeInBits();
if (Size.isScalable())
  ScalableBytes += Other.first * Size.getScalableSize() / 8;
else
  Bytes += Other.first * Size.getFixedSize() /8;

(Note that this also assumes adding a getScalableSize() to TypeSize)

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

506

Is this a good argument to add an interface for isKnownGreater and isKnownGreaterOrEqual (vis-a-vis isKnownSmaller/IsKnownSmallerOrEqual) to TypeSize as suggested in https://reviews.llvm.org/D53137#1621351 ?

[huntergr]

• Switched the lambdas in CodeGenDAGPatterns to use std::tie for ordering. This needed the TypeSize objects to be constant (I get a "non-const lvalue reference" error otherwise), so I changed all query methods to return const objects.
• Changed stack offset to deliberately use a TypeSize. This hit another problem with implicit promotion and signed v. unsigned casting, so I've added explicit casts.

[huntergr]

I also changed the SelectionDAGBuilder code for masked loads/stores/gathers/scatters to use the known min size when creating a MachineMemoryOperand, since MMO isn't aware of scalable types yet. I've left TODOs as reminders.

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

506

After thinking about it for a bit, I don't think so.

isKnownSmaller would return false for v8f32 vs. nxv4f32, and we would then proceed with a direct comparison of scalable and non-scalable type sizes, which will assert. Using that as the only comparison would also ignore the different ordering in this function, where a v4f32 would be considered less than an i64 since the scalar size is compared first.

It's possible that a different canonical ordering would work as well, but I'm not sure how many places in CodeGenDAGPatterns assume things about the order. Using the std::tie comparisons changes things slightly as it is, as noted below.

520

I've reworked them to use std::tie, which meant I had to make the size query methods return const TypeSize. Everything still seems to work (make check-all, plus an LNT run).

I don't think the std::tie comparisons are strictly equivalent to what was there before -- if A.getScalarSizeInBits() is greater than B.getScalarSizeInBits() in LT then I think the std::tie comparison would return false immediately instead of progressing to the second comparison as the previous code would.

[huntergr]

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

520

...actually, I think it works out the same anyway, since you would only get a true result from LT in the second case if the scalar sizes were equal.

[rovka]

Regarding the change to return const, I'm not convinced that's a good idea (we actually have a clang-tidy check that warns about that). I think it would be better to either name those temporaries or use std::make_tuple instead of std::tie (whichever you prefer).

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

13929

Could you be a bit more specific about why you need to cast to signed? It was unsigned in the original code too.

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

523

I was actually thinking of including isScalableVector and isVector below in the tie (since bool is an integral type and compares the way you'd expect).

[huntergr]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

13929

This code was buggy before, but didn't crash as getStoreSizeInBits returned a 32 bit unsigned, vs. the uint64_t we now return as an implicit cast.

For the failing case, LDMemType's size was larger than STMemType's size, so wrapped to a very large uint64_t value. That was then divided by 8 (which unset the top bit due to being an unsigned divide), implicitly cast to an int64_t so that Offset (which was zero) could be subtracted.

The next bit of code multiplies Offset by 8, which wraps for a signed value (which UBSan caught).

It didn't happen for the 32 bit result since the top bits were clear at the time of the implicit cast to int64_t.

I suspect the bug would have been found quickly once cases with a non-zero Offset were implemented, but the code below currently bails out in that case.

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

523

Yeah, I thought of that too, but ran into problems when trying to make it const (maybe because bool is a primitive?).

I'll try with make_tuple and see what happens.

[huntergr]

• Removed const qualifier from queries
• Swapped to std::make_tuple for the comparisons in CodeGenDAGPatterns.cpp

[rovka]

I don't see anything else wrong with this. LGTM if you rename the LE predicate.

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

520

Ok, this looks a bit better, but I'm still not happy about the state of this code. I'm not going to hold your patch hostage because of this, since it was a problem of the existing code, but LE is a really really misleading name for this predicate. It suggests that it's defining a partial order, which it isn't. If A is a vector and B is a scalar (or the other way around), we get false for both A <= B and B <= A. That never happens for a mathematical A <= B (since that implies that B < A, which means B <= A should return true). OTOH, fixing it to be a proper partial order would make us remove too much. I think the best solution here would be to use a different name for the predicate - SameKindLE maybe?

llvm/include/llvm/CodeGen/SelectionDAGNodes.h

 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/Metadata.h"
 #include "llvm/IR/Operator.h"
 #include "llvm/Support/AlignOf.h"
 #include "llvm/Support/AtomicOrdering.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/MachineValueType.h"
+#include "llvm/Support/TypeSize.h"
 #include <algorithm>
 #include <cassert>
 #include <climits>
 #include <cstddef>
 #include <cstdint>
 #include <cstring>
 #include <iterator>
 #include <string>
public:
   inline EVT getValueType() const;
 
   /// Return the simple ValueType of the referenced return value.
   MVT getSimpleValueType() const {
     return getValueType().getSimpleVT();
   }
 
   /// Returns the size of the value in bits.
-  unsigned getValueSizeInBits() const {
+  ///
+  /// If the value type is a scalable vector type, the scalable property will
+  /// be set and the runtime size will be a positive integer multiple of the
+  /// base size.
+  TypeSize getValueSizeInBits() const {
     return getValueType().getSizeInBits();
   }
 
-  unsigned getScalarValueSizeInBits() const {
+  TypeSize getScalarValueSizeInBits() const {
     return getValueType().getScalarType().getSizeInBits();
   }
 
   // Forwarding methods - These forward to the corresponding methods in SDNode.
   inline unsigned getOpcode() const;
   inline unsigned getNumOperands() const;
   inline const SDValue &getOperand(unsigned i) const;
   inline uint64_t getConstantOperandVal(unsigned i) const;
public:
   }
 
   /// Return the type of a specified result as a simple type.
   MVT getSimpleValueType(unsigned ResNo) const {
     return getValueType(ResNo).getSimpleVT();
   }
 
   /// Returns MVT::getSizeInBits(getValueType(ResNo)).
-  unsigned getValueSizeInBits(unsigned ResNo) const {
+  ///
+  /// If the value type is a scalable vector type, the scalable property will
+  /// be set and the runtime size will be a positive integer multiple of the
+  /// base size.
+  TypeSize getValueSizeInBits(unsigned ResNo) const {
     return getValueType(ResNo).getSizeInBits();
   }
 
   using value_iterator = const EVT *;
 
   value_iterator value_begin() const { return ValueList; }
   value_iterator value_end() const { return ValueList+NumValues; }
 

llvm/include/llvm/CodeGen/ValueTypes.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_VALUETYPES_H
 #define LLVM_CODEGEN_VALUETYPES_H
 
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/MachineValueType.h"
 #include "llvm/Support/MathExtras.h"
+#include "llvm/Support/TypeSize.h"
 #include <cassert>
 #include <cstdint>
 #include <string>
 
 namespace llvm {
 
   class LLVMContext;
   class Type;
public:
 
     /// Return true if this is an overloaded type for TableGen.
     bool isOverloaded() const {
       return (V==MVT::iAny || V==MVT::fAny || V==MVT::vAny || V==MVT::iPTRAny);
     }
 
     /// Return true if the bit size is a multiple of 8.
     bool isByteSized() const {
-      return (getSizeInBits() & 7) == 0;
+      return getSizeInBits().isByteSized();
     }
 
     /// Return true if the size is a power-of-two number of bytes.
     bool isRound() const {
+      if (isScalableVector())
+        return false;
       unsigned BitSize = getSizeInBits();
       return BitSize >= 8 && !(BitSize & (BitSize - 1));
     }
 
     /// Return true if this has the same number of bits as VT.
     bool bitsEq(EVT VT) const {
       if (EVT::operator==(VT)) return true;
       return getSizeInBits() == VT.getSizeInBits();
ElementCount getVectorElementCount() const {
         return V.getVectorElementCount();
 
       assert(!isScalableVector() &&
              "We don't support extended scalable types yet");
       return {getExtendedVectorNumElements(), false};
     }
 
     /// Return the size of the specified value type in bits.
-    unsigned getSizeInBits() const {
+    ///
+    /// If the value type is a scalable vector type, the scalable property will
+    /// be set and the runtime size will be a positive integer multiple of the
+    /// base size.
+    TypeSize getSizeInBits() const {
       if (isSimple())
         return V.getSizeInBits();
       return getExtendedSizeInBits();
     }
 
-    unsigned getScalarSizeInBits() const {
+    TypeSize getScalarSizeInBits() const {
       return getScalarType().getSizeInBits();
     }
 
     /// Return the number of bytes overwritten by a store of the specified value
     /// type.
-    unsigned getStoreSize() const {
-      return (getSizeInBits() + 7) / 8;
+    ///
+    /// If the value type is a scalable vector type, the scalable property will
+    /// be set and the runtime size will be a positive integer multiple of the
+    /// base size.
+    TypeSize getStoreSize() const {
+      TypeSize BaseSize = getSizeInBits();
+      return {(BaseSize.getKnownMinSize() + 7) / 8, BaseSize.isScalable()};
     }
 
-    /// Return the number of bits overwritten by a store of the specified value
+    /// Returns the number of bits overwritten by a store of the specified value
     /// type.
-    unsigned getStoreSizeInBits() const {
+    ///
+    /// If the value type is a scalable vector type, the scalable property will
+    /// be set and the runtime size will be a positive integer multiple of the
+    /// base size.
+    TypeSize getStoreSizeInBits() const {
       return getStoreSize() * 8;
     }
 
     /// Rounds the bit-width of the given integer EVT up to the nearest power of
     /// two (and at least to eight), and returns the integer EVT with that
     /// number of bits.
     EVT getRoundIntegerType(LLVMContext &Context) const {
       assert(isInteger() && !isVector() && "Invalid integer type!");
private:
     bool isExtended64BitVector() const LLVM_READONLY;
     bool isExtended128BitVector() const LLVM_READONLY;
     bool isExtended256BitVector() const LLVM_READONLY;
     bool isExtended512BitVector() const LLVM_READONLY;
     bool isExtended1024BitVector() const LLVM_READONLY;
     bool isExtended2048BitVector() const LLVM_READONLY;
     EVT getExtendedVectorElementType() const;
     unsigned getExtendedVectorNumElements() const LLVM_READONLY;
-    unsigned getExtendedSizeInBits() const LLVM_READONLY;
+    TypeSize getExtendedSizeInBits() const LLVM_READONLY;
   };
 
 } // end namespace llvm
 
 #endif // LLVM_CODEGEN_VALUETYPES_H

llvm/include/llvm/Support/MachineValueType.h

unsigned getVectorNumElements() const {
       case nxv1f64: return 1;
       }
     }
 
     ElementCount getVectorElementCount() const {
       return { getVectorNumElements(), isScalableVector() };
     }
 
-    unsigned getSizeInBits() const {
+    /// Returns the size of the specified MVT in bits.
+    ///
+    /// If the value type is a scalable vector type, the scalable property will
+    /// be set and the runtime size will be a positive integer multiple of the
+    /// base size.
+    TypeSize getSizeInBits() const {
       switch (SimpleTy) {
       default:
         llvm_unreachable("getSizeInBits called on extended MVT.");
       case Other:
         llvm_unreachable("Value type is non-standard value, Other.");
       case iPTR:
         llvm_unreachable("Value type size is target-dependent. Ask TLI.");
       case iPTRAny:
       case iAny:
       case fAny:
       case vAny:
       case Any:
         llvm_unreachable("Value type is overloaded.");
       case token:
         llvm_unreachable("Token type is a sentinel that cannot be used "
                          "in codegen and has no size");
       case Metadata:
         llvm_unreachable("Value type is metadata.");
       case i1:
-      case v1i1:
-      case nxv1i1: return 1;
-      case v2i1:
-      case nxv2i1: return 2;
-      case v4i1:
-      case nxv4i1: return 4;
+      case v1i1: return TypeSize::Fixed(1);
+      case nxv1i1: return TypeSize::Scalable(1);
+      case v2i1: return TypeSize::Fixed(2);
+      case nxv2i1: return TypeSize::Scalable(2);
+      case v4i1: return TypeSize::Fixed(4);
+      case nxv4i1: return TypeSize::Scalable(4);
       case i8  :
       case v1i8:
-      case v8i1:
+      case v8i1: return TypeSize::Fixed(8);
       case nxv1i8:
-      case nxv8i1: return 8;
+      case nxv8i1: return TypeSize::Scalable(8);
       case i16 :
       case f16:
       case v16i1:
       case v2i8:
-      case v1i16:
+      case v1i16: return TypeSize::Fixed(16);
       case nxv16i1:
       case nxv2i8:
-      case nxv1i16: return 16;
+      case nxv1i16: return TypeSize::Scalable(16);
       case f32 :
       case i32 :
       case v32i1:
       case v4i8:
       case v2i16:
       case v2f16:
       case v1f32:
-      case v1i32:
+      case v1i32: return TypeSize::Fixed(32);
       case nxv32i1:
       case nxv4i8:
       case nxv2i16:
       case nxv1i32:
       case nxv2f16:
-      case nxv1f32: return 32;
+      case nxv1f32: return TypeSize::Scalable(32);
       case v3i16:
-      case v3f16: return 48;
+      case v3f16: return TypeSize::Fixed(48);
       case x86mmx:
       case f64 :
       case i64 :
       case v64i1:
       case v8i8:
       case v4i16:
       case v2i32:
       case v1i64:
       case v4f16:
       case v2f32:
-      case v1f64:
+      case v1f64: return TypeSize::Fixed(64);
       case nxv8i8:
       case nxv4i16:
       case nxv2i32:
       case nxv1i64:
       case nxv4f16:
       case nxv2f32:
-      case nxv1f64: return 64;
-      case f80 :  return 80;
+      case nxv1f64: return TypeSize::Scalable(64);
+      case f80 :  return TypeSize::Fixed(80);
       case v3i32:
-      case v3f32: return 96;
+      case v3f32: return TypeSize::Fixed(96);
       case f128:
       case ppcf128:
       case i128:
       case v128i1:
       case v16i8:
       case v8i16:
       case v4i32:
       case v2i64:
       case v1i128:
       case v8f16:
       case v4f32:
-      case v2f64:
+      case v2f64: return TypeSize::Fixed(128);
       case nxv16i8:
       case nxv8i16:
       case nxv4i32:
       case nxv2i64:
       case nxv8f16:
       case nxv4f32:
-      case nxv2f64: return 128;
+      case nxv2f64: return TypeSize::Scalable(128);
       case v5i32:
-      case v5f32: return 160;
+      case v5f32: return TypeSize::Fixed(160);
       case v256i1:
       case v32i8:
       case v16i16:
       case v8i32:
       case v4i64:
       case v16f16:
       case v8f32:
-      case v4f64:
+      case v4f64: return TypeSize::Fixed(256);
       case nxv32i8:
       case nxv16i16:
       case nxv8i32:
       case nxv4i64:
       case nxv8f32:
-      case nxv4f64: return 256;
+      case nxv4f64: return TypeSize::Scalable(256);
       case v512i1:
       case v64i8:
       case v32i16:
       case v16i32:
       case v8i64:
       case v32f16:
       case v16f32:
-      case v8f64:
+      case v8f64: return TypeSize::Fixed(512);
       case nxv32i16:
       case nxv16i32:
       case nxv8i64:
       case nxv16f32:
-      case nxv8f64: return 512;
+      case nxv8f64: return TypeSize::Scalable(512);
       case v1024i1:
       case v128i8:
       case v64i16:
       case v32i32:
       case v16i64:
-      case v32f32:
+      case v32f32: return TypeSize::Fixed(1024);
       case nxv32i32:
-      case nxv16i64: return 1024;
+      case nxv16i64: return TypeSize::Scalable(1024);
       case v256i8:
       case v128i16:
       case v64i32:
       case v32i64:
-      case v64f32:
-      case nxv32i64: return 2048;
+      case v64f32: return TypeSize::Fixed(2048);
+      case nxv32i64: return TypeSize::Scalable(2048);
       case v128i32:
-      case v128f32:  return 4096;
+      case v128f32:  return TypeSize::Fixed(4096);
       case v256i32:
-      case v256f32:  return 8192;
+      case v256f32:  return TypeSize::Fixed(8192);
       case v512i32:
-      case v512f32:  return 16384;
+      case v512f32:  return TypeSize::Fixed(16384);
       case v1024i32:
-      case v1024f32:  return 32768;
+      case v1024f32:  return TypeSize::Fixed(32768);
       case v2048i32:
-      case v2048f32:  return 65536;
-      case exnref: return 0; // opaque type
+      case v2048f32:  return TypeSize::Fixed(65536);
+      case exnref: return TypeSize::Fixed(0); // opaque type
       }
     }
 
-    unsigned getScalarSizeInBits() const {
+    TypeSize getScalarSizeInBits() const {
       return getScalarType().getSizeInBits();
     }
 
     /// Return the number of bytes overwritten by a store of the specified value
     /// type.
-    unsigned getStoreSize() const {
-      return (getSizeInBits() + 7) / 8;
+    ///
+    /// If the value type is a scalable vector type, the scalable property will
+    /// be set and the runtime size will be a positive integer multiple of the
+    /// base size.
+    TypeSize getStoreSize() const {
+      TypeSize BaseSize = getSizeInBits();
+      return {(BaseSize.getKnownMinSize() + 7) / 8, BaseSize.isScalable()};
     }
 
     /// Return the number of bits overwritten by a store of the specified value
     /// type.
-    unsigned getStoreSizeInBits() const {
+    ///
+    /// If the value type is a scalable vector type, the scalable property will
+    /// be set and the runtime size will be a positive integer multiple of the
+    /// base size.
+    TypeSize getStoreSizeInBits() const {
       return getStoreSize() * 8;
     }
 
+    /// Returns true if the number of bits for the type is a multiple of an
+    /// 8-bit byte.
+    bool isByteSized() const {
+      return getSizeInBits().isByteSized();
+    }
+
     /// Return true if this has more bits than VT.
     bool bitsGT(MVT VT) const {
       return getSizeInBits() > VT.getSizeInBits();
     }
 
     /// Return true if this has no less bits than VT.
     bool bitsGE(MVT VT) const {
       return getSizeInBits() >= VT.getSizeInBits();

llvm/include/llvm/Support/TypeSize.h

uint64_t getKnownMinSize() const {
     return MinSize;
   }
 
   // Return whether or not the size is scalable.
   bool isScalable() const {
     return IsScalable;
   }
 
+  // Returns true if the number of bits is a multiple of an 8-bit byte.
+  bool isByteSized() const {
+    return (MinSize & 7) == 0;
+  }
+
   // Casts to a uint64_t if this is a fixed-width size.
   //
   // NOTE: This interface is obsolete and will be removed in a future version
   // of LLVM in favour of calling getFixedSize() directly.
   operator uint64_t() const {
     return getFixedSize();
   }
 

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

namespace {
 
   public:
     DAGCombiner(SelectionDAG &D, AliasAnalysis *AA, CodeGenOpt::Level OL)
         : DAG(D), TLI(D.getTargetLoweringInfo()), Level(BeforeLegalizeTypes),
           OptLevel(OL), AA(AA) {
       ForCodeSize = DAG.getMachineFunction().getFunction().hasOptSize();
 
       MaximumLegalStoreInBits = 0;
+      // We use the minimum store size here, since that's all we can guarantee
+      // for the scalable vector types.
       for (MVT VT : MVT::all_valuetypes())
         if (EVT(VT).isSimple() && VT != MVT::Other &&
             TLI.isTypeLegal(EVT(VT)) &&
-            VT.getSizeInBits() >= MaximumLegalStoreInBits)
-          MaximumLegalStoreInBits = VT.getSizeInBits();
+            VT.getSizeInBits().getKnownMinSize() >= MaximumLegalStoreInBits)
+          MaximumLegalStoreInBits = VT.getSizeInBits().getKnownMinSize();
     }
 
     void ConsiderForPruning(SDNode *N) {
       // Mark this for potential pruning.
       PruningList.insert(N);
     }
 
     /// Add to the worklist making sure its instance is at the back (next to be
SDValue DAGCombiner::ForwardStoreValueToDirectLoad(LoadSDNode *LD) {
   if (!BasePtrST.equalBaseIndex(BasePtrLD, DAG, Offset))
     return SDValue();
 
   // Normalize for Endianness. After this Offset=0 will denote that the least
   // significant bit in the loaded value maps to the least significant bit in
   // the stored value). With Offset=n (for n > 0) the loaded value starts at the
   // n:th least significant byte of the stored value.
   if (DAG.getDataLayout().isBigEndian())
-    Offset = (STMemType.getStoreSizeInBits() -
-              LDMemType.getStoreSizeInBits()) / 8 - Offset;
+    Offset = ((int64_t)STMemType.getStoreSizeInBits() -
+              (int64_t)LDMemType.getStoreSizeInBits()) / 8 - Offset;
 
   // Check that the stored value cover all bits that are loaded.
   bool STCoversLD =
       (Offset >= 0) &&
       (Offset * 8 + LDMemType.getSizeInBits() <= STMemType.getSizeInBits());
 
   auto ReplaceLd = [&](LoadSDNode *LD, SDValue Val, SDValue Chain) -> SDValue {
     if (LD->isIndexed()) {
bool DAGCombiner::MergeStoresOfConstantsOrVecElts(
     bool IsConstantSrc, bool UseVector, bool UseTrunc) {
   // Make sure we have something to merge.
   if (NumStores < 2)
     return false;
 
   // The latest Node in the DAG.
   SDLoc DL(StoreNodes[0].MemNode);
 
-  int64_t ElementSizeBits = MemVT.getStoreSizeInBits();
+  TypeSize ElementSizeBits = MemVT.getStoreSizeInBits();
   unsigned SizeInBits = NumStores * ElementSizeBits;
   unsigned NumMemElts = MemVT.isVector() ? MemVT.getVectorNumElements() : 1;
 
   EVT StoreTy;
   if (UseVector) {
     unsigned Elts = NumStores * NumMemElts;
     // Get the type for the merged vector store.
     StoreTy = EVT::getVectorVT(*DAG.getContext(), MemVT.getScalarType(), Elts);
bool DAGCombiner::MergeConsecutiveStores(StoreSDNode *St) {
 
   if (MemVT.getSizeInBits() * 2 > MaximumLegalStoreInBits)
     return false;
 
   bool NoVectors = DAG.getMachineFunction().getFunction().hasFnAttribute(
       Attribute::NoImplicitFloat);
 
   // This function cannot currently deal with non-byte-sized memory sizes.
-  if (ElementSizeBytes * 8 != MemVT.getSizeInBits())
+  if (ElementSizeBytes * 8 != (int64_t)MemVT.getSizeInBits())
     return false;
 
   if (!MemVT.isSimple())
     return false;
 
   // Perform an early exit check. Do not bother looking at stored values that
   // are not constants, loads, or extracted vector elements.
   SDValue StoredVal = peekThroughBitcasts(St->getValue());

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

 // <128 x f32> operations in terms of two <64 x f32> operations.
 //
 //===----------------------------------------------------------------------===//
 
 #include "LegalizeTypes.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/raw_ostream.h"
+#include "llvm/Support/TypeSize.h"
 using namespace llvm;
 
 #define DEBUG_TYPE "legalize-types"
 
 //===----------------------------------------------------------------------===//
 //  Result Vector Scalarization: <1 x ty> -> ty.
 //===----------------------------------------------------------------------===//
 

 //  WidenVT:   The widen vector type to load to/store from
 //  Align:     If 0, don't allow use of a wider type
 //  WidenEx:   If Align is not 0, the amount additional we can load/store from.
 
 static EVT FindMemType(SelectionDAG& DAG, const TargetLowering &TLI,
                        unsigned Width, EVT WidenVT,
                        unsigned Align = 0, unsigned WidenEx = 0) {
   EVT WidenEltVT = WidenVT.getVectorElementType();
-  unsigned WidenWidth = WidenVT.getSizeInBits();
+  const bool Scalable = WidenVT.isScalableVector();
+  unsigned WidenWidth = WidenVT.getSizeInBits().getKnownMinSize();
   unsigned WidenEltWidth = WidenEltVT.getSizeInBits();
   unsigned AlignInBits = Align*8;
 
   // If we have one element to load/store, return it.
   EVT RetVT = WidenEltVT;
   if (Width == WidenEltWidth)
     return RetVT;
 
   // See if there is larger legal integer than the element type to load/store.
   unsigned VT;
+  // Don't bother looking for an integer type if the vector is scalable, skip
+  // to vector types.
+  if (!Scalable) {
     for (VT = (unsigned)MVT::LAST_INTEGER_VALUETYPE;
          VT >= (unsigned)MVT::FIRST_INTEGER_VALUETYPE; --VT) {
       EVT MemVT((MVT::SimpleValueType) VT);
       unsigned MemVTWidth = MemVT.getSizeInBits();
       if (MemVT.getSizeInBits() <= WidenEltWidth)
         break;
       auto Action = TLI.getTypeAction(*DAG.getContext(), MemVT);
       if ((Action == TargetLowering::TypeLegal ||
            Action == TargetLowering::TypePromoteInteger) &&
           (WidenWidth % MemVTWidth) == 0 &&
           isPowerOf2_32(WidenWidth / MemVTWidth) &&
           (MemVTWidth <= Width ||
            (Align!=0 && MemVTWidth<=AlignInBits && MemVTWidth<=Width+WidenEx))) {
         if (MemVTWidth == WidenWidth)
           return MemVT;
         RetVT = MemVT;
         break;
       }
     }
+  }
 
   // See if there is a larger vector type to load/store that has the same vector
   // element type and is evenly divisible with the WidenVT.
   for (VT = (unsigned)MVT::LAST_VECTOR_VALUETYPE;
        VT >= (unsigned)MVT::FIRST_VECTOR_VALUETYPE; --VT) {
     EVT MemVT = (MVT::SimpleValueType) VT;
-    unsigned MemVTWidth = MemVT.getSizeInBits();
+    // Skip vector MVTs which don't match the scalable property of WidenVT.
+    if (Scalable != MemVT.isScalableVector())
+      continue;
+    unsigned MemVTWidth = MemVT.getSizeInBits().getKnownMinSize();
     auto Action = TLI.getTypeAction(*DAG.getContext(), MemVT);
     if ((Action == TargetLowering::TypeLegal ||
          Action == TargetLowering::TypePromoteInteger) &&
         WidenEltVT == MemVT.getVectorElementType() &&
         (WidenWidth % MemVTWidth) == 0 &&
         isPowerOf2_32(WidenWidth / MemVTWidth) &&
         (MemVTWidth <= Width ||
          (Align!=0 && MemVTWidth<=AlignInBits && MemVTWidth<=Width+WidenEx))) {

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

for (unsigned Value = 0, NumValues = ValueVTs.size(); Value != NumValues;
           Flags.setReturned();
       }
 
       getCopyToParts(CLI.DAG, CLI.DL, Op, &Parts[0], NumParts, PartVT,
                      CLI.CS.getInstruction(), CLI.CallConv, ExtendKind);
 
       for (unsigned j = 0; j != NumParts; ++j) {
         // if it isn't first piece, alignment must be 1
+        // For scalable vectors the scalable part is currently handled
+        // by individual targets, so we just use the known minimum size here.
         ISD::OutputArg MyFlags(Flags, Parts[j].getValueType(), VT,
-                               i < CLI.NumFixedArgs,
-                               i, j*Parts[j].getValueType().getStoreSize());
+                    i < CLI.NumFixedArgs, i,
+                    j*Parts[j].getValueType().getStoreSize().getKnownMinSize());
         if (NumParts > 1 && j == 0)
           MyFlags.Flags.setSplit();
         else if (j != 0) {
           MyFlags.Flags.setOrigAlign(Align::None());
           if (j == NumParts - 1)
             MyFlags.Flags.setSplitEnd();
         }
 
for (unsigned Value = 0, NumValues = ValueVTs.size();
       if (Arg.hasAttribute(Attribute::Returned))
         Flags.setReturned();
 
       MVT RegisterVT = TLI->getRegisterTypeForCallingConv(
           *CurDAG->getContext(), F.getCallingConv(), VT);
       unsigned NumRegs = TLI->getNumRegistersForCallingConv(
           *CurDAG->getContext(), F.getCallingConv(), VT);
       for (unsigned i = 0; i != NumRegs; ++i) {
+        // For scalable vectors, use the minimum size; individual targets
+        // are responsible for handling scalable vector arguments and
+        // return values.
         ISD::InputArg MyFlags(Flags, RegisterVT, VT, isArgValueUsed,
-                              ArgNo, PartBase+i*RegisterVT.getStoreSize());
+                 ArgNo, PartBase+i*RegisterVT.getStoreSize().getKnownMinSize());
         if (NumRegs > 1 && i == 0)
           MyFlags.Flags.setSplit();
         // if it isn't first piece, alignment must be 1
         else if (i > 0) {
           MyFlags.Flags.setOrigAlign(Align::None());
           if (i == NumRegs - 1)
             MyFlags.Flags.setSplitEnd();
         }
         Ins.push_back(MyFlags);
       }
       if (NeedsRegBlock && Value == NumValues - 1)
         Ins[Ins.size() - 1].Flags.setInConsecutiveRegsLast();
-      PartBase += VT.getStoreSize();
+      PartBase += VT.getStoreSize().getKnownMinSize();
     }
   }
 
   // Call the target to set up the argument values.
   SmallVector<SDValue, 8> InVals;
   SDValue NewRoot = TLI->LowerFormalArguments(
       DAG.getRoot(), F.getCallingConv(), F.isVarArg(), Ins, dl, DAG, InVals);
 

llvm/lib/CodeGen/ValueTypes.cpp

 //===----------- ValueTypes.cpp - Implementation of EVT methods -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/CodeGen/ValueTypes.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Type.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/TypeSize.h"
 using namespace llvm;
 
 EVT EVT::changeExtendedTypeToInteger() const {
   LLVMContext &Context = LLVMTy->getContext();
   return getIntegerVT(Context, getSizeInBits());
 }
 
 EVT EVT::changeExtendedVectorElementTypeToInteger() const {
EVT EVT::getExtendedVectorElementType() const {
   return EVT::getEVT(cast<VectorType>(LLVMTy)->getElementType());
 }
 
 unsigned EVT::getExtendedVectorNumElements() const {
   assert(isExtended() && "Type is not extended!");
   return cast<VectorType>(LLVMTy)->getNumElements();
 }
 
-unsigned EVT::getExtendedSizeInBits() const {
+TypeSize EVT::getExtendedSizeInBits() const {
   assert(isExtended() && "Type is not extended!");
   if (IntegerType *ITy = dyn_cast<IntegerType>(LLVMTy))
-    return ITy->getBitWidth();
+    return TypeSize::Fixed(ITy->getBitWidth());
   if (VectorType *VTy = dyn_cast<VectorType>(LLVMTy))
-    return VTy->getBitWidth();
+    return VTy->getPrimitiveSizeInBits();
   llvm_unreachable("Unrecognized extended type!");
 }
 
 /// getEVTString - This function returns value type as a string, e.g. "i32".
 std::string EVT::getEVTString() const {
   switch (V.SimpleTy) {
   default:
     if (isVector())

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

static SDValue performBitcastCombine(SDNode *N,
   // For example,
   //    (v4i16 (bitconvert
   //             (extract_subvector (v2i64 (bitconvert (v8i16 ...)), (i64 1)))))
   //  becomes
   //    (extract_subvector ((v8i16 ...), (i64 4)))
 
   // Only interested in 64-bit vectors as the ultimate result.
   EVT VT = N->getValueType(0);
-  if (!VT.isVector())
+  if (!VT.isVector() || VT.isScalableVector())
     return SDValue();
   if (VT.getSimpleVT().getSizeInBits() != 64)
     return SDValue();
   // Is the operand an extract_subvector starting at the beginning or halfway
   // point of the vector? A low half may also come through as an
   // EXTRACT_SUBREG, so look for that, too.
   SDValue Op0 = N->getOperand(0);
   if (Op0->getOpcode() != ISD::EXTRACT_SUBVECTOR &&

llvm/lib/Target/AArch64/AArch64StackOffset.h

class StackOffset {
   explicit operator int() const;
 
 public:
   using Part = std::pair<int64_t, MVT>;
 
   StackOffset() : Bytes(0), ScalableBytes(0) {}
 
   StackOffset(int64_t Offset, MVT::SimpleValueType T) : StackOffset() {
-    assert(MVT(T).getSizeInBits() % 8 == 0 &&
-           "Offset type is not a multiple of bytes");
+    assert(MVT(T).isByteSized() && "Offset type is not a multiple of bytes");
     *this += Part(Offset, T);
   }
 
   StackOffset(const StackOffset &Other)
       : Bytes(Other.Bytes), ScalableBytes(Other.ScalableBytes) {}
 
   StackOffset &operator=(const StackOffset &) = default;
 
   StackOffset &operator+=(const StackOffset::Part &Other) {
-    int64_t OffsetInBytes = Other.first * (Other.second.getSizeInBits() / 8);
+    int64_t OffsetInBytes = Other.first *
+                           (Other.second.getSizeInBits().getKnownMinSize() / 8);
     if (Other.second.isScalableVector())
       ScalableBytes += OffsetInBytes;
     else
       Bytes += OffsetInBytes;
     return *this;
   }
 
   StackOffset &operator+=(const StackOffset &Other) {

llvm/lib/Target/ARM/ARMISelLowering.cpp

if (VT.isVector() && VT.isFloatingPoint() && Subtarget->hasMVEIntegerOps() &&
     return false;
 
   bool IsNeg = false;
   if (V < 0) {
     IsNeg = true;
     V = -V;
   }
 
-  unsigned NumBytes = std::max(VT.getSizeInBits() / 8, 1U);
+  unsigned NumBytes = std::max((unsigned)VT.getSizeInBits() / 8, 1U);
 
   // MVE: size * imm7
   if (VT.isVector() && Subtarget->hasMVEIntegerOps()) {
     switch (VT.getSimpleVT().getVectorElementType().SimpleTy) {
     case MVT::i32:
     case MVT::f32:
       return isShiftedUInt<7,2>(V);
     case MVT::i16:

llvm/lib/Target/Hexagon/HexagonISelLowering.cpp

for (unsigned i = 0, e = ArgLocs.size(); i != e; ++i) {
 
     if (VA.isMemLoc()) {
       unsigned LocMemOffset = VA.getLocMemOffset();
       SDValue MemAddr = DAG.getConstant(LocMemOffset, dl,
                                         StackPtr.getValueType());
       MemAddr = DAG.getNode(ISD::ADD, dl, MVT::i32, StackPtr, MemAddr);
       if (ArgAlign)
         LargestAlignSeen = std::max(LargestAlignSeen,
-                                    VA.getLocVT().getStoreSizeInBits() >> 3);
+                             (unsigned)VA.getLocVT().getStoreSizeInBits() >> 3);
       if (Flags.isByVal()) {
         // The argument is a struct passed by value. According to LLVM, "Arg"
         // is a pointer.
         MemOpChains.push_back(CreateCopyOfByValArgument(Arg, MemAddr, Chain,
                                                         Flags, DAG, dl));
       } else {
         MachinePointerInfo LocPI = MachinePointerInfo::getStack(
             DAG.getMachineFunction(), LocMemOffset);

llvm/lib/Target/Mips/MipsISelLowering.cpp

MVT MipsTargetLowering::getRegisterTypeForCallingConv(LLVMContext &Context,
   }
   return MipsTargetLowering::getRegisterType(Context, VT);
 }
 
 unsigned MipsTargetLowering::getNumRegistersForCallingConv(LLVMContext &Context,
                                                            CallingConv::ID CC,
                                                            EVT VT) const {
   if (VT.isVector())
-    return std::max((VT.getSizeInBits() / (Subtarget.isABI_O32() ? 32 : 64)),
+    return std::max(((unsigned)VT.getSizeInBits() /
+                     (Subtarget.isABI_O32() ? 32 : 64)),
                     1U);
   return MipsTargetLowering::getNumRegisters(Context, VT);
 }
 
 unsigned MipsTargetLowering::getVectorTypeBreakdownForCallingConv(
     LLVMContext &Context, CallingConv::ID CC, EVT VT, EVT &IntermediateVT,
     unsigned &NumIntermediates, MVT &RegisterVT) const {
   // Break down vector types to either 2 i64s or 4 i32s.

llvm/lib/Target/NVPTX/NVPTXISelDAGToDAG.cpp

bool NVPTXDAGToDAGISel::tryLoad(SDNode *N) {
   //
   // Sign   : ISD::SEXTLOAD
   // Unsign : ISD::ZEXTLOAD, ISD::NON_EXTLOAD or ISD::EXTLOAD and the
   //          type is integer
   // Float  : ISD::NON_EXTLOAD or ISD::EXTLOAD and the type is float
   MVT SimpleVT = LoadedVT.getSimpleVT();
   MVT ScalarVT = SimpleVT.getScalarType();
   // Read at least 8 bits (predicates are stored as 8-bit values)
-  unsigned fromTypeWidth = std::max(8U, ScalarVT.getSizeInBits());
+  unsigned fromTypeWidth = std::max(8U, (unsigned)ScalarVT.getSizeInBits());
   unsigned int fromType;
 
   // Vector Setting
   unsigned vecType = NVPTX::PTXLdStInstCode::Scalar;
   if (SimpleVT.isVector()) {
     assert(LoadedVT == MVT::v2f16 && "Unexpected vector type");
     // v2f16 is loaded using ld.b32
     fromTypeWidth = 32;
bool NVPTXDAGToDAGISel::tryLoadVector(SDNode *N) {
   // Type Setting: fromType + fromTypeWidth
   //
   // Sign   : ISD::SEXTLOAD
   // Unsign : ISD::ZEXTLOAD, ISD::NON_EXTLOAD or ISD::EXTLOAD and the
   //          type is integer
   // Float  : ISD::NON_EXTLOAD or ISD::EXTLOAD and the type is float
   MVT ScalarVT = SimpleVT.getScalarType();
   // Read at least 8 bits (predicates are stored as 8-bit values)
-  unsigned FromTypeWidth = std::max(8U, ScalarVT.getSizeInBits());
+  unsigned FromTypeWidth = std::max(8U, (unsigned)ScalarVT.getSizeInBits());
   unsigned int FromType;
   // The last operand holds the original LoadSDNode::getExtensionType() value
   unsigned ExtensionType = cast<ConstantSDNode>(
       N->getOperand(N->getNumOperands() - 1))->getZExtValue();
   if (ExtensionType == ISD::SEXTLOAD)
     FromType = NVPTX::PTXLdStInstCode::Signed;
   else if (ScalarVT.isFloatingPoint())
     FromType = ScalarVT.SimpleTy == MVT::f16 ? NVPTX::PTXLdStInstCode::Untyped

llvm/lib/Target/PowerPC/PPCISelDAGToDAG.cpp

for (SDNode::use_iterator Use = N->use_begin(), UseEnd = N->use_end();
     unsigned Opc =
       Use->isMachineOpcode() ? Use->getMachineOpcode() : Use->getOpcode();
     switch (Opc) {
     default: return 0;
     case ISD::TRUNCATE:
       if (Use->isMachineOpcode())
         return 0;
       MaxTruncation =
-        std::max(MaxTruncation, Use->getValueType(0).getSizeInBits());
+        std::max(MaxTruncation, (unsigned)Use->getValueType(0).getSizeInBits());
       continue;
     case ISD::STORE: {
       if (Use->isMachineOpcode())
         return 0;
       StoreSDNode *STN = cast<StoreSDNode>(*Use);
       unsigned MemVTSize = STN->getMemoryVT().getSizeInBits();
       if (MemVTSize == 64 || Use.getOperandNo() != 0)
         return 0;

llvm/lib/Target/X86/X86ISelLowering.cpp

static SDValue getExtendInVec(unsigned Opcode, const SDLoc &DL, EVT VT,
 
   // For 256-bit vectors, we only need the lower (128-bit) input half.
   // For 512-bit vectors, we only need the lower input half or quarter.
   if (InVT.getSizeInBits() > 128) {
     assert(VT.getSizeInBits() == InVT.getSizeInBits() &&
            "Expected VTs to be the same size!");
     unsigned Scale = VT.getScalarSizeInBits() / InVT.getScalarSizeInBits();
     In = extractSubVector(In, 0, DAG, DL,
-                          std::max(128U, VT.getSizeInBits() / Scale));
+                          std::max(128U, (unsigned)VT.getSizeInBits() / Scale));
     InVT = In.getValueType();
   }
 
   if (VT.getVectorNumElements() != InVT.getVectorNumElements())
     Opcode = getOpcode_EXTEND_VECTOR_INREG(Opcode);
 
   return DAG.getNode(Opcode, DL, VT, In);
 }
static SDValue LowerBUILD_VECTORvXi1(SDValue Op, SelectionDAG &DAG,
   if (HasConstElts) {
     if (VT == MVT::v64i1 && !Subtarget.is64Bit()) {
       SDValue ImmL = DAG.getConstant(Lo_32(Immediate), dl, MVT::i32);
       SDValue ImmH = DAG.getConstant(Hi_32(Immediate), dl, MVT::i32);
       ImmL = DAG.getBitcast(MVT::v32i1, ImmL);
       ImmH = DAG.getBitcast(MVT::v32i1, ImmH);
       DstVec = DAG.getNode(ISD::CONCAT_VECTORS, dl, MVT::v64i1, ImmL, ImmH);
     } else {
-      MVT ImmVT = MVT::getIntegerVT(std::max(VT.getSizeInBits(), 8U));
+      MVT ImmVT = MVT::getIntegerVT(std::max((unsigned)VT.getSizeInBits(), 8U));
       SDValue Imm = DAG.getConstant(Immediate, dl, ImmVT);
       MVT VecVT = VT.getSizeInBits() >= 8 ? VT : MVT::v8i1;
       DstVec = DAG.getBitcast(VecVT, Imm);
       DstVec = DAG.getNode(ISD::EXTRACT_SUBVECTOR, dl, VT, DstVec,
                            DAG.getIntPtrConstant(0, dl));
     }
   } else
     DstVec = DAG.getUNDEF(VT);
for (unsigned i = 0; i != NumInputs; ++i) {
     unsigned &Offset = Offsets[i];
     Src = peekThroughBitcasts(Src);
     EVT BaseVT = Src.getValueType();
     while (Src.getOpcode() == ISD::EXTRACT_SUBVECTOR &&
            isa<ConstantSDNode>(Src.getOperand(1))) {
       Offset += Src.getConstantOperandVal(1);
       Src = Src.getOperand(0);
     }
-    WideSizeInBits = std::max(WideSizeInBits, Src.getValueSizeInBits());
+    WideSizeInBits = std::max(WideSizeInBits,
+                              (unsigned)Src.getValueSizeInBits());
     assert((Offset % BaseVT.getVectorNumElements()) == 0 &&
            "Unexpected subvector extraction");
     Offset /= BaseVT.getVectorNumElements();
     Offset *= NumMaskElts;
   }
 
   // Bail if we're always extracting from the lowest subvectors,
   // combineX86ShuffleChain should match this for the current width.

 
 // Given two zexts of <k x i8> to <k x i32>, create a PSADBW of the inputs
 // to these zexts.
 static SDValue createPSADBW(SelectionDAG &DAG, const SDValue &Zext0,
                             const SDValue &Zext1, const SDLoc &DL,
                             const X86Subtarget &Subtarget) {
   // Find the appropriate width for the PSADBW.
   EVT InVT = Zext0.getOperand(0).getValueType();
-  unsigned RegSize = std::max(128u, InVT.getSizeInBits());
+  unsigned RegSize = std::max(128u, (unsigned)InVT.getSizeInBits());
 
   // "Zero-extend" the i8 vectors. This is not a per-element zext, rather we
   // fill in the missing vector elements with 0.
   unsigned NumConcat = RegSize / InVT.getSizeInBits();
   SmallVector<SDValue, 16> Ops(NumConcat, DAG.getConstant(0, DL, InVT));
   Ops[0] = Zext0.getOperand(0);
   MVT ExtendedVT = MVT::getVectorVT(MVT::i8, RegSize / 8);
   SDValue SadOp0 = DAG.getNode(ISD::CONCAT_VECTORS, DL, ExtendedVT, Ops);

llvm/unittests/CodeGen/ScalableVectorMVTsTest.cpp

TEST(ScalableVectorMVTsTest, VTToIRTranslation) {
   Type *Ty = Enxv4f64.getTypeForEVT(Ctx);
   VectorType *ScV4Float64Ty = cast<VectorType>(Ty);
   ASSERT_TRUE(ScV4Float64Ty->isScalable());
   ASSERT_EQ(Enxv4f64.getVectorElementCount(), ScV4Float64Ty->getElementCount());
   ASSERT_EQ(Enxv4f64.getScalarType().getTypeForEVT(Ctx),
             ScV4Float64Ty->getElementType());
 }
 
+TEST(ScalableVectorMVTsTest, SizeQueries) {
+  LLVMContext Ctx;
+
+  EVT nxv4i32 = EVT::getVectorVT(Ctx, MVT::i32, 4, /*Scalable=*/ true);
+  EVT nxv2i32 = EVT::getVectorVT(Ctx, MVT::i32, 2, /*Scalable=*/ true);
+  EVT nxv2i64 = EVT::getVectorVT(Ctx, MVT::i64, 2, /*Scalable=*/ true);
+  EVT nxv2f64 = EVT::getVectorVT(Ctx, MVT::f64, 2, /*Scalable=*/ true);
+
+  EVT v4i32 = EVT::getVectorVT(Ctx, MVT::i32, 4);
+  EVT v2i32 = EVT::getVectorVT(Ctx, MVT::i32, 2);
+  EVT v2i64 = EVT::getVectorVT(Ctx, MVT::i64, 2);
+  EVT v2f64 = EVT::getVectorVT(Ctx, MVT::f64, 2);
+
+  // Check equivalence and ordering on scalable types.
+  EXPECT_EQ(nxv4i32.getSizeInBits(), nxv2i64.getSizeInBits());
+  EXPECT_EQ(nxv2f64.getSizeInBits(), nxv2i64.getSizeInBits());
+  EXPECT_NE(nxv2i32.getSizeInBits(), nxv4i32.getSizeInBits());
+  EXPECT_LT(nxv2i32.getSizeInBits(), nxv2i64.getSizeInBits());
+  EXPECT_LE(nxv4i32.getSizeInBits(), nxv2i64.getSizeInBits());
+  EXPECT_GT(nxv4i32.getSizeInBits(), nxv2i32.getSizeInBits());
+  EXPECT_GE(nxv2i64.getSizeInBits(), nxv4i32.getSizeInBits());
+
+  // Check equivalence and ordering on fixed types.
+  EXPECT_EQ(v4i32.getSizeInBits(), v2i64.getSizeInBits());
+  EXPECT_EQ(v2f64.getSizeInBits(), v2i64.getSizeInBits());
+  EXPECT_NE(v2i32.getSizeInBits(), v4i32.getSizeInBits());
+  EXPECT_LT(v2i32.getSizeInBits(), v2i64.getSizeInBits());
+  EXPECT_LE(v4i32.getSizeInBits(), v2i64.getSizeInBits());
+  EXPECT_GT(v4i32.getSizeInBits(), v2i32.getSizeInBits());
+  EXPECT_GE(v2i64.getSizeInBits(), v4i32.getSizeInBits());
+
+  // Check that scalable and non-scalable types with the same minimum size
+  // are not considered equal.
+  ASSERT_TRUE(v4i32.getSizeInBits() != nxv4i32.getSizeInBits());
+  ASSERT_FALSE(v2i64.getSizeInBits() == nxv2f64.getSizeInBits());
+
+  // Check that we can obtain a known-exact size from a non-scalable type.
+  EXPECT_EQ(v4i32.getSizeInBits(), 128U);
+  EXPECT_EQ(v2i64.getSizeInBits().getFixedSize(), 128U);
+
+  // Check that we can query the known minimum size for both scalable and
+  // fixed length types.
+  EXPECT_EQ(nxv2i32.getSizeInBits().getKnownMinSize(), 64U);
+  EXPECT_EQ(nxv2f64.getSizeInBits().getKnownMinSize(), 128U);
+  EXPECT_EQ(v2i32.getSizeInBits().getKnownMinSize(),
+            nxv2i32.getSizeInBits().getKnownMinSize());
+
+  // Check scalable property.
+  ASSERT_FALSE(v4i32.getSizeInBits().isScalable());
+  ASSERT_TRUE(nxv4i32.getSizeInBits().isScalable());
+
+  // Check convenience size scaling methods.
+  EXPECT_EQ(v2i32.getSizeInBits() * 2, v4i32.getSizeInBits());
+  EXPECT_EQ(2 * nxv2i32.getSizeInBits(), nxv4i32.getSizeInBits());
+  EXPECT_EQ(nxv2f64.getSizeInBits() / 2, nxv2i32.getSizeInBits());
+}
+
 } // end anonymous namespace

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

for (unsigned M : Modes) {
 
     if (none_of(S, isVector) || none_of(B, isVector)) {
       Changed |= berase_if(S, isVector) |
                  berase_if(B, isVector);
     }
   }
 
   auto LT = [](MVT A, MVT B) -> bool {
+    // Always treat non-scalable MVTs as smaller than scalable MVTs for the
+    // purposes of ordering.
+    if (A.isScalableVector() && !B.isScalableVector())
+      return false;
+    if (!A.isScalableVector() && B.isScalableVector())
+      return true;
+
     return A.getScalarSizeInBits() < B.getScalarSizeInBits() ||
            (A.getScalarSizeInBits() == B.getScalarSizeInBits() &&
             A.getSizeInBits() < B.getSizeInBits());
   };
   auto LE = [&LT](MVT A, MVT B) -> bool {
     // This function is used when removing elements: when a vector is compared
     // to a non-vector, it should return false (to avoid removal).
     if (A.isVector() != B.isVector())
       return false;
 
+    // We also don't want to remove elements when they're both vectors with the
+    // same minimum number of lanes, but one is scalable and the other not.
+    if (A.isScalableVector() != B.isScalableVector())
+      return false;
+
     return LT(A, B) || (A.getScalarSizeInBits() == B.getScalarSizeInBits() &&
                         A.getSizeInBits() == B.getSizeInBits());
   };
 
   for (unsigned M : Modes) {
     TypeSetByHwMode::SetType &S = Small.get(M);
     TypeSetByHwMode::SetType &B = Big.get(M);
     // MinS = min scalar in Small, remove all scalars from Big that are