[ValueTypes] Add support for scalable EVTs

[c-rhodes]

Summary

Changes:

• Remove a bunch of asserts checking for unsupported scalable types and add some more now that they are supported.
• Propagate the scalable flag where necessary.
• Add another EVT::getExtendedVectorVT method that takes an ElementCount parameter.
• Add EVT::isExtendedScalableVector and EVT::getExtendedVectorElementCount - latter is currently unused.

[c-rhodes]

Fix warnings reported by Harbor.

[efriedma]

Please post patches with full context (-U10000)

llvm/include/llvm/CodeGen/ValueTypes.h

265

Not sure changing getVectorNumElements like this makes sense.

[c-rhodes]

Added context.

[c-rhodes]

Remove check for isScalableVector in getVectorNumElements.

[c-rhodes]

In D75672#1907847, @efriedma wrote:

Please post patches with full context (-U10000)

I've updated the patch, sorry about that.

llvm/include/llvm/CodeGen/ValueTypes.h

265

I've removed this change, there was just one place in SelectionDAGBuilder I had to fix up to call getVectorElementCount. I had a chat with Graham and he suggested it would be good to check if it's a fixed-length vector here, maybe by having something similar to the MVT isFixedLengthVector on the EVT which could call MVT::isFixedLengthVector if it's a SimplyTy or check it's not a scalable vector otherwise.

[efriedma]

llvm/include/llvm/CodeGen/ValueTypes.h

265

Yes, we want to catch incorrect use of getNumElements in both IR and in SelectionDAG.

[c-rhodes]

llvm/include/llvm/CodeGen/ValueTypes.h

265

Is this something we can implement going forward or should this be handled before landing this patch?

[efriedma]

llvm/include/llvm/CodeGen/ValueTypes.h

265

If it's easy, might as well handle it here, if there's some non-obvious complication, I'd be fine with putting it off.

[c-rhodes]

llvm/include/llvm/CodeGen/ValueTypes.h

265

I've tested the following patch downstream:

diff --git a/llvm/include/llvm/CodeGen/ValueTypes.h b/llvm/include/llvm/CodeGen/ValueTypes.h
index 72e0cc8..c2db003 100644
--- a/llvm/include/llvm/CodeGen/ValueTypes.h
+++ b/llvm/include/llvm/CodeGen/ValueTypes.h
@@ -152,6 +152,10 @@ namespace llvm {
       return isSimple() ? V.isScalableVector() : isExtendedScalableVector();
     }

+    bool isFixedLengthVector() const {
+      return isSimple() ? V.isFixedLengthVector() : isExtendedFixedLengthVector();
+    }
+
     /// Return true if this is a 16-bit vector type.
     bool is16BitVector() const {
       return isSimple() ? V.is16BitVector() : isExtended16BitVector();
@@ -262,7 +266,7 @@ namespace llvm {

     /// Given a vector type, return the number of elements it contains.
     unsigned getVectorNumElements() const {
-      assert(isVector() && "Invalid vector type!");
+      assert(isFixedLengthVector() && "Invalid vector type!");
       if (isSimple())
         return V.getVectorNumElements();
       return getExtendedVectorNumElements();
@@ -432,6 +436,7 @@ namespace llvm {
     bool isExtended1024BitVector() const LLVM_READONLY;
     bool isExtended2048BitVector() const LLVM_READONLY;
     bool isExtendedScalableVector() const LLVM_READONLY;
+    bool isExtendedFixedLengthVector() const LLVM_READONLY;
     EVT getExtendedVectorElementType() const;
     unsigned getExtendedVectorNumElements() const LLVM_READONLY;
     ElementCount getExtendedVectorElementCount() const LLVM_READONLY;
diff --git a/llvm/lib/CodeGen/ValueTypes.cpp b/llvm/lib/CodeGen/ValueTypes.cpp
index dcb3a88..a2a35e2 100644
--- a/llvm/lib/CodeGen/ValueTypes.cpp
+++ b/llvm/lib/CodeGen/ValueTypes.cpp
@@ -106,6 +106,10 @@ bool EVT::isExtendedScalableVector() const {
   return isExtendedVector() && cast<VectorType>(LLVMTy)->isScalable();
 }

+bool EVT::isExtendedFixedLengthVector() const {
+  return isExtendedVector() && !cast<VectorType>(LLVMTy)->isScalable();
+}
+
 EVT EVT::getExtendedVectorElementType() const {
   assert(isExtended() && "Type is not extended!");
   return EVT::getEVT(cast<VectorType>(LLVMTy)->getElementType());

After running check-all there are a lot of failures. Ideally we want to get rid of getVectorNumElements but a quick grep shows there's >450 uses of it in lib/CodeGen and lib/Target/AArch64 alone. All of these probably aren't EVT::getVectorNumElements but having looked through some of the errors downstream there's cases like those in SelectionDAGBuilder::visitShuffleVector that need fixing up.

I think this assert is similar to the implicit cast of TypeSize -> uint64_t which Sander addressed with a warning in D75297. Being restrictive here is also going to trigger a lot of asserts, perhaps we can also emit a warning here?

[efriedma]

llvm/include/llvm/CodeGen/ValueTypes.h

265

That makes sense, but let's do it as a separate patch.

[c-rhodes]

llvm/include/llvm/CodeGen/ValueTypes.h

265

I've created a separate patch D76376.

[efriedma]

LGTM with one minor comment

llvm/include/llvm/CodeGen/ValueTypes.h

263

isVector() || isScalableVector() is the same thing as isVector(), I think?

[c-rhodes]

Thanks for reviewing Eli!

llvm/include/llvm/CodeGen/ValueTypes.h

263

Yeah isVector() looks sufficient, I'll fix before merging.

llvm/include/llvm/CodeGen/ValueTypes.h

public:
 
     /// Returns the EVT that represents a vector NumElements in length, where
     /// each element is of type VT.
     static EVT getVectorVT(LLVMContext &Context, EVT VT, unsigned NumElements,
                            bool IsScalable = false) {
       MVT M = MVT::getVectorVT(VT.V, NumElements, IsScalable);
       if (M.SimpleTy != MVT::INVALID_SIMPLE_VALUE_TYPE)
         return M;
-
-      assert(!IsScalable && "We don't support extended scalable types yet");
-      return getExtendedVectorVT(Context, VT, NumElements);
+      return getExtendedVectorVT(Context, VT, NumElements, IsScalable);
     }
 
     /// Returns the EVT that represents a vector EC.Min elements in length,
     /// where each element is of type VT.
     static EVT getVectorVT(LLVMContext &Context, EVT VT, ElementCount EC) {
       MVT M = MVT::getVectorVT(VT.V, EC);
       if (M.SimpleTy != MVT::INVALID_SIMPLE_VALUE_TYPE)
         return M;
-      assert (!EC.Scalable && "We don't support extended scalable types yet");
-      return getExtendedVectorVT(Context, VT, EC.Min);
+      return getExtendedVectorVT(Context, VT, EC);
     }
 
     /// Return a vector with the same number of elements as this vector, but
     /// with the element type converted to an integer type with the same
     /// bitwidth.
     EVT changeVectorElementTypeToInteger() const {
-      if (!isSimple()) {
-        assert (!isScalableVector() &&
-                "We don't support extended scalable types yet");
+      if (!isSimple())
         return changeExtendedVectorElementTypeToInteger();
-      }
       MVT EltTy = getSimpleVT().getVectorElementType();
       unsigned BitWidth = EltTy.getSizeInBits();
       MVT IntTy = MVT::getIntegerVT(BitWidth);
       MVT VecTy = MVT::getVectorVT(IntTy, getVectorNumElements(),
                                    isScalableVector());
       assert(VecTy.SimpleTy != MVT::INVALID_SIMPLE_VALUE_TYPE &&
              "Simple vector VT not representable by simple integer vector VT!");
       return VecTy;
public:
     /// Return true if this is a vector value type.
     bool isVector() const {
       return isSimple() ? V.isVector() : isExtendedVector();
     }
 
     /// Return true if this is a vector type where the runtime
     /// length is machine dependent
     bool isScalableVector() const {
-      // FIXME: We don't support extended scalable types yet, because the
-      // matching IR type doesn't exist. Once it has been added, this can
-      // be changed to call isExtendedScalableVector.
-      if (!isSimple())
-        return false;
-      return V.isScalableVector();
+      return isSimple() ? V.isScalableVector() : isExtendedScalableVector();
     }
 
     bool isFixedLengthVector() const {
       return isSimple() ? V.isFixedLengthVector()
                         : isExtendedFixedLengthVector();
     }
 
     /// Return true if this is a 16-bit vector type.
public:
     /// If this is a vector type, return the element type, otherwise return
     /// this.
     EVT getScalarType() const {
       return isVector() ? getVectorElementType() : *this;
     }
 
     /// Given a vector type, return the type of each element.
     EVT getVectorElementType() const {
       assert(isVector() && "Invalid vector type!");
       if (isSimple())
         return V.getVectorElementType();
       return getExtendedVectorElementType();
     }
 
     /// Given a vector type, return the number of elements it contains.
     unsigned getVectorNumElements() const {
 #ifdef STRICT_FIXED_SIZE_VECTORS
       assert(isFixedLengthVector() && "Invalid vector type!");
 #else
#endif
     }
 
     // Given a (possibly scalable) vector type, return the ElementCount
     ElementCount getVectorElementCount() const {
       assert((isVector()) && "Invalid vector type!");
       if (isSimple())
         return V.getVectorElementCount();
 
-      assert(!isScalableVector() &&
-             "We don't support extended scalable types yet");
-      return {getExtendedVectorNumElements(), false};
+      return {getExtendedVectorNumElements(), isExtendedScalableVector()};
     }
 
     /// Return the size of the specified value type in bits.
     ///
     /// If the value type is a scalable vector type, the scalable property will
     /// be set and the runtime size will be a positive integer multiple of the
     /// base size.
     TypeSize getSizeInBits() const {
#endif
 
   private:
     // Methods for handling the Extended-type case in functions above.
     // These are all out-of-line to prevent users of this header file
     // from having a dependency on Type.h.
     EVT changeExtendedTypeToInteger() const;
     EVT changeExtendedVectorElementTypeToInteger() const;
     static EVT getExtendedIntegerVT(LLVMContext &C, unsigned BitWidth);
-    static EVT getExtendedVectorVT(LLVMContext &C, EVT VT,
-                                   unsigned NumElements);
+    static EVT getExtendedVectorVT(LLVMContext &C, EVT VT, unsigned NumElements,
+                                   bool IsScalable);
+    static EVT getExtendedVectorVT(LLVMContext &Context, EVT VT,
+                                   ElementCount EC);
     bool isExtendedFloatingPoint() const LLVM_READONLY;
     bool isExtendedInteger() const LLVM_READONLY;
     bool isExtendedScalarInteger() const LLVM_READONLY;
     bool isExtendedVector() const LLVM_READONLY;
     bool isExtended16BitVector() const LLVM_READONLY;
     bool isExtended32BitVector() const LLVM_READONLY;
     bool isExtended64BitVector() const LLVM_READONLY;
     bool isExtended128BitVector() const LLVM_READONLY;
     bool isExtended256BitVector() const LLVM_READONLY;
     bool isExtended512BitVector() const LLVM_READONLY;
     bool isExtended1024BitVector() const LLVM_READONLY;
     bool isExtended2048BitVector() const LLVM_READONLY;
     bool isExtendedFixedLengthVector() const LLVM_READONLY;
+    bool isExtendedScalableVector() const LLVM_READONLY;
     EVT getExtendedVectorElementType() const;
     unsigned getExtendedVectorNumElements() const LLVM_READONLY;
+    ElementCount getExtendedVectorElementCount() const LLVM_READONLY;
     TypeSize getExtendedSizeInBits() const LLVM_READONLY;
   };
 
 } // end namespace llvm
 
 #endif // LLVM_CODEGEN_VALUETYPES_H

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

if (NumIntermediates == NumParts) {
       for (unsigned i = 0; i != NumIntermediates; ++i)
         Ops[i] = getCopyFromParts(DAG, DL, &Parts[i * Factor], Factor,
                                   PartVT, IntermediateVT, V);
     }
 
     // Build a vector with BUILD_VECTOR or CONCAT_VECTORS from the
     // intermediate operands.
     EVT BuiltVectorTy =
-        EVT::getVectorVT(*DAG.getContext(), IntermediateVT.getScalarType(),
-                         (IntermediateVT.isVector()
-                              ? IntermediateVT.getVectorNumElements() * NumParts
-                              : NumIntermediates));
+        IntermediateVT.isVector()
+            ? EVT::getVectorVT(
+                  *DAG.getContext(), IntermediateVT.getScalarType(),
+                  IntermediateVT.getVectorElementCount() * NumParts)
+            : EVT::getVectorVT(*DAG.getContext(),
+                               IntermediateVT.getScalarType(),
+                               NumIntermediates);
     Val = DAG.getNode(IntermediateVT.isVector() ? ISD::CONCAT_VECTORS
                                                 : ISD::BUILD_VECTOR,
                       DL, BuiltVectorTy, Ops);
   }
 
   // There is now one part, held in Val.  Correct it to match ValueVT.
   EVT PartEVT = Val.getValueType();
 

llvm/lib/CodeGen/ValueTypes.cpp

 EVT EVT::changeExtendedTypeToInteger() const {
   LLVMContext &Context = LLVMTy->getContext();
   return getIntegerVT(Context, getSizeInBits());
 }
 
 EVT EVT::changeExtendedVectorElementTypeToInteger() const {
   LLVMContext &Context = LLVMTy->getContext();
   EVT IntTy = getIntegerVT(Context, getScalarSizeInBits());
-  return getVectorVT(Context, IntTy, getVectorNumElements());
+  return getVectorVT(Context, IntTy, getVectorNumElements(),
+                     isScalableVector());
 }
 
 EVT EVT::getExtendedIntegerVT(LLVMContext &Context, unsigned BitWidth) {
   EVT VT;
   VT.LLVMTy = IntegerType::get(Context, BitWidth);
   assert(VT.isExtended() && "Type is not extended!");
   return VT;
 }
 
-EVT EVT::getExtendedVectorVT(LLVMContext &Context, EVT VT,
-                             unsigned NumElements) {
+EVT EVT::getExtendedVectorVT(LLVMContext &Context, EVT VT, unsigned NumElements,
+                             bool IsScalable) {
   EVT ResultVT;
-  ResultVT.LLVMTy = VectorType::get(VT.getTypeForEVT(Context), NumElements);
+  ResultVT.LLVMTy =
+      VectorType::get(VT.getTypeForEVT(Context), NumElements, IsScalable);
+  assert(ResultVT.isExtended() && "Type is not extended!");
+  return ResultVT;
+}
+
+EVT EVT::getExtendedVectorVT(LLVMContext &Context, EVT VT, ElementCount EC) {
+  EVT ResultVT;
+  ResultVT.LLVMTy =
+      VectorType::get(VT.getTypeForEVT(Context), {EC.Min, EC.Scalable});
   assert(ResultVT.isExtended() && "Type is not extended!");
   return ResultVT;
 }
 
 bool EVT::isExtendedFloatingPoint() const {
   assert(isExtended() && "Type is not extended!");
   return LLVMTy->isFPOrFPVectorTy();
 }

 bool EVT::isExtended2048BitVector() const {
   return isExtendedVector() && getExtendedSizeInBits() == 2048;
 }
 
 bool EVT::isExtendedFixedLengthVector() const {
   return isExtendedVector() && !cast<VectorType>(LLVMTy)->isScalable();
 }
 
+bool EVT::isExtendedScalableVector() const {
+  return isExtendedVector() && cast<VectorType>(LLVMTy)->isScalable();
+}
+
 EVT EVT::getExtendedVectorElementType() const {
   assert(isExtended() && "Type is not extended!");
   return EVT::getEVT(cast<VectorType>(LLVMTy)->getElementType());
 }
 
 unsigned EVT::getExtendedVectorNumElements() const {
   assert(isExtended() && "Type is not extended!");
   return cast<VectorType>(LLVMTy)->getNumElements();
 }
 
+ElementCount EVT::getExtendedVectorElementCount() const {
+  assert(isExtended() && "Type is not extended!");
+  return cast<VectorType>(LLVMTy)->getElementCount();
+}
+
 TypeSize EVT::getExtendedSizeInBits() const {
   assert(isExtended() && "Type is not extended!");
   if (IntegerType *ITy = dyn_cast<IntegerType>(LLVMTy))
     return TypeSize::Fixed(ITy->getBitWidth());
   if (VectorType *VTy = dyn_cast<VectorType>(LLVMTy))
     return VTy->getPrimitiveSizeInBits();
   llvm_unreachable("Unrecognized extended type!");
 }