[POC][SVE] Allow code generation for fixed length vectorised loops [Patch 1/2].

[paulwalker-arm]

Summary

No expectations for review at this stage unless you are super keen.

This is a proof of concept patch to show how SVE can be used to code generate fixed length vectors. It represents the minimum amount of SVE instructions (ld1, st1, uzp, uunpklo) required to get most workloads to run.

The general idea is to make all fixed length vector types that fit within a user specified size legal and custom lower all fixed length vector operations to scalable vector operations that use a suitably created predicate. After legalisation there should be no vector operations that operate on fixed length vectors beyond insert/extract_subvector and various extends and truncate. The reason for special case these operations is to maximum our ability for DAG combine to remove them. Those still around at isel get custom selected within ISelDagToDag.

[efriedma]

I've spent a little time considering alternatives. There are basically a few possibilities here:

1. Say the types aren't legal, and convert the types as part of type legalization.
2. Say the types are legal, but the operations aren't, and custom-legalize (or teach LegalizeDAG to legalize) all the operations.
3. Say the types and patterns are legal, and add a bunch of extra patterns to match the instructions.

Any of these options have rough edges... but type legalization seems more natural to me. That would let you rewrite all the types, so you aren't stuck with "fake" types for values that are live across basic blocks. And it would let you take more advantage of existing type legalization infrastructure: you don't have to have a custom legalization handler for every SVE operation, and you probably get better handling of cases where the fixed->scalable transform still leaves some illegal types. And the code would be naturally shared with other targets.

You haven't really demonstrated how you plan to handle i1 vectors, in particular, <8 x i1> might be <vscale x 8 x i1> or <vscale x 4 x i1> depending on how it's generated/used. You can probably make it work with your current approach; I guess you end up inserting predicate pack/unpack operations in Select(), and we should be able to eliminate most of the redundant operations? Not completely sure how that works out. (If an i1 vector is live across basic blocks, the generated code gets really messy, but that's not really specific to SVE.)

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

3139

If we're going to use this approach, we probably want to custom legalize TRUNCATE and ANY_EXTEND, and leave only the INSERT_SUBVECTOR/EXTRACT_SUBVECTOR operations to be handled in Select().

[paulwalker-arm]

Rebasing to reflect the majority of the functionality is now in master. What remains is likely to be abandoned in favour of function attributes but it's here for those who want to experiment.

[paulwalker-arm]

Rebase.

[paulwalker-arm]

The intention of this patch is now complete. All work is available in master with the exception of the hook into -msve-vector-bits which is not necessarily the direction we'll use once function attributes are available.

clang/include/clang/Driver/Options.td

def fcall_saved_x#i : Flag<["-"], "fcall-saved-x"#i>, Group<m_aarch64_Features_Group>,
     HelpText<"Make the x"#i#" register call-saved (AArch64 only)">;
 
 def msign_return_address_EQ : Joined<["-"], "msign-return-address=">,
   Flags<[CC1Option]>, Group<m_Group>, Values<"none,all,non-leaf">,
   HelpText<"Select return address signing scope">;
 def mbranch_protection_EQ : Joined<["-"], "mbranch-protection=">,
   HelpText<"Enforce targets of indirect branches and function returns">;
 
+def msve_vector_bits : Joined<["-"], "msve-vector-bits=">,
+  Group<m_aarch64_Features_Group>,
+  HelpText<"Specify the size in bits of an SVE vector register."
+           " Has no effect unless SVE is enabled. (Default is \"scalable\")">;
+
 def msimd128 : Flag<["-"], "msimd128">, Group<m_wasm_Features_Group>;
 def munimplemented_simd128 : Flag<["-"], "munimplemented-simd128">, Group<m_wasm_Features_Group>;
 def mno_unimplemented_simd128 : Flag<["-"], "mno-unimplemented-simd128">, Group<m_wasm_Features_Group>;
 def mno_simd128 : Flag<["-"], "mno-simd128">, Group<m_wasm_Features_Group>;
 def mnontrapping_fptoint : Flag<["-"], "mnontrapping-fptoint">, Group<m_wasm_Features_Group>;
 def mno_nontrapping_fptoint : Flag<["-"], "mno-nontrapping-fptoint">, Group<m_wasm_Features_Group>;
 def msign_ext : Flag<["-"], "msign-ext">, Group<m_wasm_Features_Group>;
 def mno_sign_ext : Flag<["-"], "mno-sign-ext">, Group<m_wasm_Features_Group>;

clang/lib/Driver/ToolChains/Clang.cpp

if (Arg *A = Args.getLastArg(options::OPT_msign_return_address_EQ,
 
     CmdArgs.push_back(
         Args.MakeArgString(Twine("-msign-return-address=") + Scope));
     CmdArgs.push_back(
         Args.MakeArgString(Twine("-msign-return-address-key=") + Key));
     if (IndirectBranches)
       CmdArgs.push_back("-mbranch-target-enforce");
   }
+
+  if (any_of(CmdArgs,
+             [](const char* Arg){return (strcmp(Arg, "+sve") == 0 ||
+                                         strcmp(Arg, "+sve2") == 0);})) {
+    if (Arg *A = Args.getLastArg(options::OPT_msve_vector_bits)) {
+      StringRef Bits = A->getValue();
+      if (Bits != "scalable") {
+        CmdArgs.push_back("-mllvm");
+        CmdArgs.push_back(Args.MakeArgString("-aarch64-sve-vector-bits-min="
+                                             + Bits));
+        //CmdArgs.push_back("-mllvm");
+        //CmdArgs.push_back(Args.MakeArgString("-aarch64-sve-vector-bits-max="
+        //                                     + Bits));
+      }
+    }
+  }
 }
 
 void Clang::AddMIPSTargetArgs(const ArgList &Args,
                               ArgStringList &CmdArgs) const {
   const Driver &D = getToolChain().getDriver();
   StringRef CPUName;
   StringRef ABIName;
   const llvm::Triple &Triple = getToolChain().getTriple();

llvm/include/llvm/CodeGen/ValueTypes.h

bool isScalableVector() const {
       // FIXME: We don't support extended scalable types yet, because the
       // matching IR type doesn't exist. Once it has been added, this can
       // be changed to call isExtendedScalableVector.
       if (!isSimple())
         return false;
       return V.isScalableVector();
     }
 
+    /// Return true if this is a vector type where the runtime
+    /// length is machine independent
+    bool isFixedVector() const {
+      return isVector() && !isScalableVector();
+    }
+
     /// Return true if this is a 16-bit vector type.
     bool is16BitVector() const {
       return isSimple() ? V.is16BitVector() : isExtended16BitVector();
     }
 
     /// Return true if this is a 32-bit vector type.
     bool is32BitVector() const {
       return isSimple() ? V.is32BitVector() : isExtended32BitVector();

llvm/include/llvm/Support/MachineValueType.h

bool isScalableVector() const {
               SimpleTy <= MVT::LAST_SCALABLE_VECTOR_VALUETYPE);
     }
 
     bool isFixedLengthVector() const {
       return (SimpleTy >= MVT::FIRST_FIXEDLEN_VECTOR_VALUETYPE &&
               SimpleTy <= MVT::LAST_FIXEDLEN_VECTOR_VALUETYPE);
     }
 
+    /// Return true if this is a vector value type where the
+    /// runtime length is machine independent
+    bool isFixedVector() const {
+      return isVector() && !isScalableVector();
+    }
+
     /// Return true if this is a 16-bit vector type.
     bool is16BitVector() const {
       return (SimpleTy == MVT::v2i8  || SimpleTy == MVT::v1i16 ||
               SimpleTy == MVT::v16i1);
     }
 
     /// Return true if this is a 32-bit vector type.
     bool is32BitVector() const {

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

 }
 
 SDValue DAGCombiner::visitSCALAR_TO_VECTOR(SDNode *N) {
   SDValue InVal = N->getOperand(0);
   EVT VT = N->getValueType(0);
 
   // Replace a SCALAR_TO_VECTOR(EXTRACT_VECTOR_ELT(V,C0)) pattern
   // with a VECTOR_SHUFFLE and possible truncate.
-  if (InVal.getOpcode() == ISD::EXTRACT_VECTOR_ELT) {
+  if ((InVal.getOpcode() == ISD::EXTRACT_VECTOR_ELT) &&
+      !VT.isScalableVector()) {
     SDValue InVec = InVal->getOperand(0);
     SDValue EltNo = InVal->getOperand(1);
     auto InVecT = InVec.getValueType();
     if (ConstantSDNode *C0 = dyn_cast<ConstantSDNode>(EltNo)) {
       SmallVector<int, 8> NewMask(InVecT.getVectorNumElements(), -1);
       int Elt = C0->getZExtValue();
       NewMask[0] = Elt;
       // If we have an implict truncate do truncate here as long as it's legal.

llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

SDValue SelectionDAGLegalize::ExpandInsertToVectorThroughStack(SDValue Op) {
   // Store the subvector.
   Ch = DAG.getStore(Ch, dl, Part, SubStackPtr, MachinePointerInfo());
 
   // Finally, load the updated vector.
   return DAG.getLoad(Op.getValueType(), dl, Ch, StackPtr, PtrInfo);
 }
 
 SDValue SelectionDAGLegalize::ExpandVectorBuildThroughStack(SDNode* Node) {
+  bool IsConcat = Node->getOpcode() == ISD::CONCAT_VECTORS;
+
   // We can't handle this case efficiently.  Allocate a sufficiently
-  // aligned object on the stack, store each element into it, then load
+  // aligned object on the stack, store each operand into it, then load
   // the result as a vector.
   // Create the stack frame object.
   EVT VT = Node->getValueType(0);
-  EVT EltVT = VT.getVectorElementType();
+  EVT MemVT = IsConcat ? Node->getOperand(0).getValueType() :
+                         VT.getVectorElementType();
   SDLoc dl(Node);
   SDValue FIPtr = DAG.CreateStackTemporary(VT);
   int FI = cast<FrameIndexSDNode>(FIPtr.getNode())->getIndex();
   MachinePointerInfo PtrInfo =
       MachinePointerInfo::getFixedStack(DAG.getMachineFunction(), FI);
 
   // Emit a store of each element to the stack slot.
   SmallVector<SDValue, 8> Stores;
-  unsigned TypeByteSize = EltVT.getSizeInBits() / 8;
+  unsigned TypeByteSize = MemVT.getSizeInBits() / 8;
   assert(TypeByteSize > 0 && "Vector element type too small for stack store!");
   // Store (in the right endianness) the elements to memory.
   for (unsigned i = 0, e = Node->getNumOperands(); i != e; ++i) {
     // Ignore undef elements.
     if (Node->getOperand(i).isUndef()) continue;
 
     unsigned Offset = TypeByteSize*i;
 
     SDValue Idx = DAG.getConstant(Offset, dl, FIPtr.getValueType());
     Idx = DAG.getMemBasePlusOffset(FIPtr, Idx, dl);
 
     // If the destination vector element type is narrower than the source
     // element type, only store the bits necessary.
-    if (EltVT.bitsLT(Node->getOperand(i).getValueType().getScalarType())) {
+    if (MemVT.bitsLT(Node->getOperand(i).getValueType())) {
       Stores.push_back(DAG.getTruncStore(DAG.getEntryNode(), dl,
                                          Node->getOperand(i), Idx,
-                                         PtrInfo.getWithOffset(Offset), EltVT));
+                                         PtrInfo.getWithOffset(Offset), MemVT));
     } else
       Stores.push_back(DAG.getStore(DAG.getEntryNode(), dl, Node->getOperand(i),
                                     Idx, PtrInfo.getWithOffset(Offset)));
   }
 
   SDValue StoreChain;
   if (!Stores.empty())    // Not all undef elements?
     StoreChain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, Stores);

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

if (auto *C = dyn_cast<ConstantFPSDNode>(Op)) {
     Known.One = C->getValueAPF().bitcastToAPInt();
     Known.Zero = ~Known.One;
     return Known;
   }
 
   if (Depth >= MaxRecursionDepth)
     return Known;  // Limit search depth.
 
+  if (Op.getValueType().isScalableVector())
+    return Known; // Unknown number of elts so assume we don't know anything.
+
   KnownBits Known2;
   unsigned NumElts = DemandedElts.getBitWidth();
   assert((!Op.getValueType().isVector() ||
           NumElts == Op.getValueType().getVectorNumElements()) &&
          "Unexpected vector size");
 
   if (!DemandedElts)
     return Known;  // No demanded elts, better to assume we don't know anything.
if (VT.isSimple() && N1.getValueType().isSimple()) {
       assert(VT.isVector() && N1.getValueType().isVector() &&
              "Extract subvector VTs must be a vectors!");
       assert(VT.getVectorElementType() ==
              N1.getValueType().getVectorElementType() &&
              "Extract subvector VTs must have the same element type!");
       assert(VT.getSimpleVT() <= N1.getSimpleValueType() &&
              "Extract subvector must be from larger vector to smaller vector!");
 
-      if (N2C) {
+      if (N2C && !N1.getValueType().isScalableVector()) {
         assert((VT.getVectorNumElements() + N2C->getZExtValue()
                 <= N1.getValueType().getVectorNumElements())
                && "Extract subvector overflow!");
       }
 
       // Trivial extraction.
       if (VT.getSimpleVT() == N1.getSimpleValueType())
         return N1;
if (VT.isSimple() && N1.getValueType().isSimple()
         && N2.getValueType().isSimple()) {
       assert(VT.isVector() && N1.getValueType().isVector() &&
              N2.getValueType().isVector() &&
              "Insert subvector VTs must be a vectors");
       assert(VT == N1.getValueType() &&
              "Dest and insert subvector source types must match!");
       assert(N2.getSimpleValueType() <= N1.getSimpleValueType() &&
              "Insert subvector must be from smaller vector to larger vector!");
-      if (isa<ConstantSDNode>(Index)) {
+      if (isa<ConstantSDNode>(Index) &&
+          !N1.getValueType().isScalableVector() &&
+          !N2.getValueType().isScalableVector()) {
         assert((N2.getValueType().getVectorNumElements() +
                 cast<ConstantSDNode>(Index)->getZExtValue()
                 <= VT.getVectorNumElements())
                && "Insert subvector overflow!");
       }
 
       // Trivial insertion.
       if (VT.getSimpleVT() == N2.getSimpleValueType())

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

if (!!DemandedVecElts) {
       Known.One &= KnownVec.One;
       Known.Zero &= KnownVec.Zero;
     }
 
     return false;
   }
   case ISD::INSERT_SUBVECTOR: {
     SDValue Base = Op.getOperand(0);
+    if (Base.getValueType().isScalableVector())
+      break;
     SDValue Sub = Op.getOperand(1);
     EVT SubVT = Sub.getValueType();
     unsigned NumSubElts = SubVT.getVectorNumElements();
 
     // If index isn't constant, assume we need the original demanded base
     // elements and ALL the inserted subvector elements.
     APInt BaseElts = DemandedElts;
     APInt SubElts = APInt::getAllOnesValue(NumSubElts);
if (!!BaseElts) {
         Known.One &= KnownBase.One;
         Known.Zero &= KnownBase.Zero;
     }
     break;
   }
   case ISD::EXTRACT_SUBVECTOR: {
     // If index isn't constant, assume we need all the source vector elements.
     SDValue Src = Op.getOperand(0);
+    if (Src.getValueType().isScalableVector())
+      break;
     ConstantSDNode *SubIdx = dyn_cast<ConstantSDNode>(Op.getOperand(1));
     unsigned NumSrcElts = Src.getValueType().getVectorNumElements();
     APInt SrcElts = APInt::getAllOnesValue(NumSrcElts);
     if (SubIdx && SubIdx->getAPIntValue().ule(NumSrcElts - NumElts)) {
       // Offset the demanded elts by the subvector index.
       uint64_t Idx = SubIdx->getZExtValue();
       SrcElts = DemandedElts.zextOrSelf(NumSrcElts).shl(Idx);
     }
for (unsigned i = 0; i != NumSubVecs; ++i) {
       KnownZero.insertBits(SubZero, i * NumSubElts);
     }
     break;
   }
   case ISD::INSERT_SUBVECTOR: {
     if (!isa<ConstantSDNode>(Op.getOperand(2)))
       break;
     SDValue Base = Op.getOperand(0);
+    if (Base.getValueType().isScalableVector())
+      break;
     SDValue Sub = Op.getOperand(1);
     EVT SubVT = Sub.getValueType();
     unsigned NumSubElts = SubVT.getVectorNumElements();
     const APInt &Idx = Op.getConstantOperandAPInt(2);
     if (Idx.ugt(NumElts - NumSubElts))
       break;
     unsigned SubIdx = Idx.getZExtValue();
     APInt SubElts = DemandedElts.extractBits(NumSubElts, SubIdx);
if (SimplifyDemandedVectorElts(Base, BaseElts, KnownUndef, KnownZero, TLO,
                                    Depth + 1))
       return true;
     KnownUndef.insertBits(SubUndef, SubIdx);
     KnownZero.insertBits(SubZero, SubIdx);
     break;
   }
   case ISD::EXTRACT_SUBVECTOR: {
     SDValue Src = Op.getOperand(0);
+    if (Src.getValueType().isScalableVector())
+      break;
     ConstantSDNode *SubIdx = dyn_cast<ConstantSDNode>(Op.getOperand(1));
     unsigned NumSrcElts = Src.getValueType().getVectorNumElements();
     if (SubIdx && SubIdx->getAPIntValue().ule(NumSrcElts - NumElts)) {
       // Offset the demanded elts by the subvector index.
       uint64_t Idx = SubIdx->getZExtValue();
       APInt SrcElts = DemandedElts.zextOrSelf(NumSrcElts).shl(Idx);
       APInt SrcUndef, SrcZero;
       if (SimplifyDemandedVectorElts(Src, SrcElts, SrcUndef, SrcZero, TLO,

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

SDNode *N2 = CurDAG->getMachineNode(AArch64::ADDXrr, DL, MVT::i64,
                                       {SDValue(N1, 0), N->getOperand(2)});
   SDNode *N3 = CurDAG->getMachineNode(
       AArch64::ADDG, DL, MVT::i64,
       {SDValue(N2, 0), CurDAG->getTargetConstant(0, DL, MVT::i64),
        CurDAG->getTargetConstant(TagOffset, DL, MVT::i64)});
   ReplaceNode(N, N3);
 }
 
+static SDNode* ExtractFromSVEReg(SelectionDAG *DAG, EVT VT, SDValue V) {
+  assert(VT.isFixedVector() && "Expected fixed length vector type!");
+  SDLoc DL(V);
+  switch (VT.getSizeInBits()) {
+  case 64: {
+    auto SubReg = DAG->getTargetConstant(AArch64::dsub, DL, MVT::i32);
+    return DAG->getMachineNode(TargetOpcode::EXTRACT_SUBREG, DL, VT, V, SubReg);
+  }
+  case 128: {
+    auto SubReg = DAG->getTargetConstant(AArch64::zsub, DL, MVT::i32);
+    return DAG->getMachineNode(TargetOpcode::EXTRACT_SUBREG, DL, VT, V, SubReg);
+  }
+  default: {
+    auto RC = DAG->getTargetConstant(AArch64::ZPRRegClassID, DL, MVT::i64);
+    return DAG->getMachineNode(TargetOpcode::COPY_TO_REGCLASS, DL, VT, V, RC);
+  }
+  }
+}
+
+static SDNode* InsertIntoSVEReg(SelectionDAG *DAG, EVT VT, SDValue V) {
+  assert(V.getValueType().isFixedVector() &&
+         "Expected fixed length vector type!");
+  SDLoc DL(V);
+  switch (V.getValueType().getSizeInBits()) {
+  case 64: {
+    auto SubReg = DAG->getTargetConstant(AArch64::dsub, DL, MVT::i32);
+    auto Container = DAG->getMachineNode(TargetOpcode::IMPLICIT_DEF, DL, VT);
+    return DAG->getMachineNode(TargetOpcode::INSERT_SUBREG, DL, VT,
+                               SDValue(Container, 0), V, SubReg);
+  }
+  case 128: {
+    auto SubReg = DAG->getTargetConstant(AArch64::zsub, DL, MVT::i32);
+    auto Container = DAG->getMachineNode(TargetOpcode::IMPLICIT_DEF, DL, VT);
+    return DAG->getMachineNode(TargetOpcode::INSERT_SUBREG, DL, VT,
+                               SDValue(Container, 0), V, SubReg);
+  }
+  default: {
+    auto RC = DAG->getTargetConstant(AArch64::ZPRRegClassID, DL, MVT::i64);
+    return DAG->getMachineNode(TargetOpcode::COPY_TO_REGCLASS, DL, VT, V, RC);
+  }
+  }
+}
+
 void AArch64DAGToDAGISel::Select(SDNode *Node) {
   // If we have a custom node, we already have selected!
   if (Node->isMachineOpcode()) {
     LLVM_DEBUG(errs() << "== "; Node->dump(CurDAG); errs() << "\n");
     Node->setNodeId(-1);
     return;
   }
 
if (tryHighFPExt(Node))
       return;
     break;
 
   case ISD::OR:
     if (tryBitfieldInsertOp(Node))
       return;
     break;
 
+  case ISD::EXTRACT_SUBVECTOR: {
+    // Bail when not a "cast" like extract_subvector.
+    auto *LaneNode = dyn_cast<ConstantSDNode>(Node->getOperand(1));
+    if (!LaneNode || (LaneNode->getZExtValue() != 0))
+      break;
+
+    // Bail when normal isel can do the job.
+    EVT InVT = Node->getOperand(0).getValueType();
+    if (VT.isScalableVector() || !InVT.isScalableVector())
+      break;
+
+    // NOTE: We can only get here when doing fixed length SVE code generation.
+    // We do manual selection because the types involved are not linked to real
+    // registers (despite being legal) and must be coerced into SVE registers.
+
+    ReplaceNode(Node, ExtractFromSVEReg(CurDAG, VT, Node->getOperand(0)));
+    return;
+  }
+
+  case ISD::INSERT_SUBVECTOR: {
+    // Bail when not a "cast" like insert_subvector.
+    auto *LaneNode = dyn_cast<ConstantSDNode>(Node->getOperand(2));
+    if (!LaneNode || (LaneNode->getZExtValue() != 0))
+      break;
+
+    // Bail when normal isel should do the job.
+    EVT InVT = Node->getOperand(1).getValueType();
+    if (!VT.isScalableVector() || InVT.isScalableVector())
+      break;
+
+    // NOTE: We can only get here when doing fixed length SVE code generation.
+    // We do manual selection because the types involved are not linked to real
+    // registers (despite being legal) and must be coerced into SVE registers.
+
+    // Bail when inserting into real data. (HACK)
+    if (!Node->getOperand(0).isUndef())
+      break;
+
+    ReplaceNode(Node, InsertIntoSVEReg(CurDAG, VT, Node->getOperand(1)));
+    return;
+  }
+
+  case ISD::TRUNCATE: {
+    EVT InVT = Node->getOperand(0).getValueType();
+    // Only intercept combinations not handled by tablegen based isel patterns.
+    if (!VT.isVector() || VT.isScalableVector())
+      break;
+    if ((VT.getSizeInBits() == 64) && (InVT.getSizeInBits() == 128))
+      break;
+
+    // NOTE: We can only get here when doing fixed length SVE code generation.
+    // We do manual selection because the types involved are not linked to real
+    // registers (despite being legal) and must be coerced into SVE registers.
+    SDLoc DL(Node);
+
+    EVT ContainerVT;
+    switch (InVT.getVectorElementType().getSimpleVT().SimpleTy) {
+    default: llvm_unreachable("unimplemented container type");
+    case MVT::i16: ContainerVT = EVT(MVT::nxv8i16); break;
+    case MVT::i32: ContainerVT = EVT(MVT::nxv4i32); break;
+    case MVT::i64: ContainerVT = EVT(MVT::nxv2i64); break;
+    }
+
+    // Repeatedly truncate operand until the result is of the desired type.
+    auto Op = InsertIntoSVEReg(CurDAG, ContainerVT, Node->getOperand(0));
+    while (ContainerVT.getVectorElementType() != VT.getVectorElementType()) {
+      unsigned Opc;
+      switch (ContainerVT.getSimpleVT().SimpleTy) {
+      default:
+        llvm_unreachable("unimplemented container type");
+      case MVT::nxv8i16:
+        Opc = AArch64::UZP1_ZZZ_B;
+        ContainerVT = MVT::nxv16i8;
+        break;
+      case MVT::nxv4i32:
+        Opc = AArch64::UZP1_ZZZ_H;
+        ContainerVT = MVT::nxv8i16;
+        break;
+      case MVT::nxv2i64:
+        Opc = AArch64::UZP1_ZZZ_S;
+        ContainerVT = MVT::nxv4i32;
+        break;
+      }
+
+      SDValue Src = SDValue(Op, 0);
+      Op = CurDAG->getMachineNode(Opc, DL, ContainerVT, Src, Src);
+    }
+
+    ReplaceNode(Node, ExtractFromSVEReg(CurDAG, VT, SDValue(Op, 0)));
+    return;
+  }
+
+  case ISD::ANY_EXTEND:
+  case ISD::ZERO_EXTEND: {
+    EVT InVT = Node->getOperand(0).getValueType();
+    // Only intercept combinations not handled by tablegen based isel patterns.
+    if (!VT.isVector() || VT.isScalableVector())
+      break;
+    if ((VT.getSizeInBits() == 128) && (InVT.getSizeInBits() == 64))
+      break;
+
+    // NOTE: We can only get here when doing fixed length SVE code generation.
+    // We do manual selection because the types involved are not linked to real
+    // registers (despite being legal) and must be coerced into SVE registers.
+    SDLoc DL(Node);
+
+    EVT ContainerVT;
+    switch (InVT.getVectorElementType().getSimpleVT().SimpleTy) {
+    default: llvm_unreachable("unimplemented container type");
+    case MVT::i8:  ContainerVT = EVT(MVT::nxv16i8); break;
+    case MVT::i16: ContainerVT = EVT(MVT::nxv8i16); break;
+    case MVT::i32: ContainerVT = EVT(MVT::nxv4i32); break;
+    }
+
+    // Repeatedly zero-extend operand until the result is of the desired type.
+    auto Op = InsertIntoSVEReg(CurDAG, ContainerVT, Node->getOperand(0));
+    while (ContainerVT.getVectorElementType() != VT.getVectorElementType()) {
+      unsigned Opc;
+      switch (ContainerVT.getSimpleVT().SimpleTy) {
+      default:
+        llvm_unreachable("unimplemented container type");
+      case MVT::nxv16i8:
+        Opc = AArch64::UUNPKLO_ZZ_H;
+        ContainerVT = MVT::nxv8i16;
+        break;
+      case MVT::nxv8i16:
+        Opc = AArch64::UUNPKLO_ZZ_S;
+        ContainerVT = MVT::nxv4i32;
+        break;
+      case MVT::nxv4i32:
+        Opc = AArch64::UUNPKLO_ZZ_D;
+        ContainerVT = MVT::nxv2i64;
+        break;
+      }
+
+      Op = CurDAG->getMachineNode(Opc, DL, ContainerVT, SDValue(Op, 0));
+    }
+
+    ReplaceNode(Node, ExtractFromSVEReg(CurDAG, VT, SDValue(Op, 0)));
+    return;
+  }
+
   case ISD::Constant: {
     // Materialize zero constants as copies from WZR/XZR.  This allows
     // the coalescer to propagate these into other instructions.
     ConstantSDNode *ConstNode = cast<ConstantSDNode>(Node);
     if (ConstNode->isNullValue()) {
       if (VT == MVT::i32) {
         SDValue New = CurDAG->getCopyFromReg(
             CurDAG->getEntryNode(), SDLoc(Node), AArch64::WZR, MVT::i32);

llvm/lib/Target/AArch64/AArch64ISelLowering.h

bool allowsMisalignedMemoryAccesses(
       bool *Fast = nullptr) const override;
   /// LLT variant.
   bool allowsMisalignedMemoryAccesses(
     LLT Ty, unsigned AddrSpace, unsigned Align, MachineMemOperand::Flags Flags,
     bool *Fast = nullptr) const override;
 
   /// Provide custom lowering hooks for some operations.
   SDValue LowerOperation(SDValue Op, SelectionDAG &DAG) const override;
+  SDValue LowerOperationDefault(SDValue Op, SelectionDAG &DAG) const;
 
   const char *getTargetNodeName(unsigned Opcode) const override;
 
   SDValue PerformDAGCombine(SDNode *N, DAGCombinerInfo &DCI) const override;
 
   /// Returns true if a cast between SrcAS and DestAS is a noop.
   bool isNoopAddrSpaceCast(unsigned SrcAS, unsigned DestAS) const override {
     // Addrspacecasts are always noops.
public:
 
   MachineMemOperand::Flags getMMOFlags(const Instruction &I) const override;
 
   bool functionArgumentNeedsConsecutiveRegisters(Type *Ty,
                                                  CallingConv::ID CallConv,
                                                  bool isVarArg) const override;
   /// Used for exception handling on Win64.
   bool needsFixedCatchObjects() const override;
+
+  bool
+  shouldExpandBuildVectorWithShuffles(EVT VT,
+                                      unsigned DefinedValues) const override;
+
+  bool useSVEForFixedLengthVectors() const;
+
 private:
   /// Keep a pointer to the AArch64Subtarget around so that we can
   /// make the right decision when generating code for different targets.
   const AArch64Subtarget *Subtarget;
 
   bool isExtFreeImpl(const Instruction *Ext) const override;
 
   void addTypeForNEON(MVT VT, MVT PromotedBitwiseVT);
+  void addTypeForFixedLengthSVE(MVT VT);
   void addDRTypeForNEON(MVT VT);
   void addQRTypeForNEON(MVT VT);
 
   SDValue LowerFormalArguments(SDValue Chain, CallingConv::ID CallConv,
                                bool isVarArg,
                                const SmallVectorImpl<ISD::InputArg> &Ins,
                                const SDLoc &DL, SelectionDAG &DAG,
                                SmallVectorImpl<SDValue> &InVals) const override;

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

 STATISTIC(NumShiftInserts, "Number of vector shift inserts");
 STATISTIC(NumOptimizedImms, "Number of times immediates were optimized");
 
 static cl::opt<bool>
 EnableAArch64SlrGeneration("aarch64-shift-insert-generation", cl::Hidden,
                            cl::desc("Allow AArch64 SLI/SRI formation"),
                            cl::init(false));
 
+static cl::opt<bool>
+UseSVEForAll("aarch64-stress-fixed-length-sve", cl::Hidden,
+    cl::desc("Use SVE for fixed length vectors that would ordinarily use NEON"),
+    cl::init(true));
+
 // FIXME: The necessary dtprel relocations don't seem to be supported
 // well in the GNU bfd and gold linkers at the moment. Therefore, by
 // default, for now, fall back to GeneralDynamic code generation.
 cl::opt<bool> EnableAArch64ELFLocalDynamicTLSGeneration(
     "aarch64-elf-ldtls-generation", cl::Hidden,
     cl::desc("Allow AArch64 Local Dynamic TLS code generation"),
     cl::init(false));
 
if (Subtarget->hasSVE()) {
 
     addRegisterClass(MVT::nxv2f16, &AArch64::ZPRRegClass);
     addRegisterClass(MVT::nxv4f16, &AArch64::ZPRRegClass);
     addRegisterClass(MVT::nxv8f16, &AArch64::ZPRRegClass);
     addRegisterClass(MVT::nxv2f32, &AArch64::ZPRRegClass);
     addRegisterClass(MVT::nxv4f32, &AArch64::ZPRRegClass);
     addRegisterClass(MVT::nxv2f64, &AArch64::ZPRRegClass);
 
+    if (useSVEForFixedLengthVectors()) {
+      for (MVT VT : MVT::integer_fixedlen_vector_valuetypes())
+        // Make everything that can fit within an SVE register legal, unless...
+        if (VT.getSizeInBits() <= STI.getMinSVEVectorSizeInBits())
+          // Fixed length predicates are promoted to i8.
+          if ((VT.getVectorElementType() != MVT::i1) && VT.isPow2VectorType())
+            // Ensure Neon MVTs only belong to a single register class.
+            if (VT.getSizeInBits() > 128)
+                addRegisterClass(VT, &AArch64::ZPRRegClass);
+
+      for (MVT VT : MVT::fp_fixedlen_vector_valuetypes())
+        // Make everything that can fit within an SVE register legal, unless...
+        if (VT.getSizeInBits() <= STI.getMinSVEVectorSizeInBits())
+          if (VT.isPow2VectorType())
+            // Ensure Neon MVTs only belong to a single register class.
+            if (VT.getSizeInBits() > 128)
+              addRegisterClass(VT, &AArch64::ZPRRegClass);
+    }
+
     for (auto VT : { MVT::nxv16i8, MVT::nxv8i16, MVT::nxv4i32, MVT::nxv2i64 }) {
       setOperationAction(ISD::SADDSAT, VT, Legal);
       setOperationAction(ISD::UADDSAT, VT, Legal);
       setOperationAction(ISD::SSUBSAT, VT, Legal);
       setOperationAction(ISD::USUBSAT, VT, Legal);
     }
   }
 
if (Subtarget->hasSVE()) {
     // splat of 0 or undef) once vector selects supported in SVE codegen. See
     // D68877 for more details.
     for (MVT VT : MVT::integer_scalable_vector_valuetypes()) {
       if (isTypeLegal(VT))
         setOperationAction(ISD::SPLAT_VECTOR, VT, Custom);
     }
     setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::i8, Custom);
     setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::i16, Custom);
+
+    // See TODO in addTypeForFixedLengthSVE for why this is here.
+    if (useSVEForFixedLengthVectors()) {
+      for (MVT VT : MVT::integer_fixedlen_vector_valuetypes())
+        addTypeForFixedLengthSVE(VT);
+      for (MVT VT : MVT::fp_fixedlen_vector_valuetypes())
+        addTypeForFixedLengthSVE(VT);
+    }
   }
 
   PredictableSelectIsExpensive = Subtarget->predictableSelectIsExpensive();
 }
 
 void AArch64TargetLowering::addTypeForNEON(MVT VT, MVT PromotedBitwiseVT) {
   assert(VT.isVector() && "VT should be a vector type");
 
if (Subtarget->isLittleEndian()) {
     for (unsigned im = (unsigned)ISD::PRE_INC;
          im != (unsigned)ISD::LAST_INDEXED_MODE; ++im) {
       setIndexedLoadAction(im, VT, Legal);
       setIndexedStoreAction(im, VT, Legal);
     }
   }
 }
 
+void AArch64TargetLowering::addTypeForFixedLengthSVE(MVT VT) {
+  assert(VT.isFixedVector() && "Expected fixed length vector type!");
+
+// TODO: Enable this when we can be called before computeRegisterProperties.
+//  // Make everything that can fit within an SVE register legal, unless...
+//  if (VT.getSizeInBits() <= Subtarget->getMinSVEVectorSizeInBits())
+//    // Fixed length predicates are promoted to i8.
+//    if ((VT.getVectorElementType() != MVT::i1) && VT.isPow2VectorType())
+//      // Ensure Neon MVTs only belong to a single register class.
+//      if (VT.getSizeInBits() > 128)
+//        addRegisterClass(VT, &AArch64::ZPRRegClass);
+
+  // By default everything requires expanding.
+  for (unsigned Op = 0; Op < ISD::BUILTIN_OP_END; ++Op)
+    setOperationAction(Op, VT, Expand);
+
+  for (MVT InnerVT : MVT::integer_fixedlen_vector_valuetypes()) {
+    setTruncStoreAction(VT, InnerVT, Expand);
+    setLoadExtAction(ISD::SEXTLOAD, VT, InnerVT, Expand);
+    setLoadExtAction(ISD::ZEXTLOAD, VT, InnerVT, Expand);
+    setLoadExtAction(ISD::EXTLOAD, VT, InnerVT, Expand);
+  }
+
+  // Use common code for all type legalisation.
+  if (!isTypeLegal(VT))
+    return;
+
+  // These are used to legalise operations and represent the only operations
+  // that are legal to operate on fixed length vectors.
+  setOperationAction(ISD::ANY_EXTEND, VT, Legal);
+  setOperationAction(ISD::TRUNCATE, VT, Legal);
+  setOperationAction(ISD::ZERO_EXTEND, VT, Legal);
+  setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Custom);
+  setOperationAction(ISD::INSERT_SUBVECTOR, VT, Custom);
+
+  // Lower fixed length vector operations to scalable equivalents.
+  setOperationAction(ISD::LOAD, VT, Custom);
+  setOperationAction(ISD::STORE, VT, Custom);
+}
+
 void AArch64TargetLowering::addDRTypeForNEON(MVT VT) {
   addRegisterClass(VT, &AArch64::FPR64RegClass);
   addTypeForNEON(VT, MVT::v2i32);
 }
 
 void AArch64TargetLowering::addQRTypeForNEON(MVT VT) {
   addRegisterClass(VT, &AArch64::FPR128RegClass);
   addTypeForNEON(VT, MVT::v4i32);
SDValue Result = DAG.getMemIntrinsicNode(
         {StoreNode->getChain(), Lo, Hi, StoreNode->getBasePtr()},
         StoreNode->getMemoryVT(), StoreNode->getMemOperand());
     return Result;
   }
 
   return SDValue();
 }
 
-SDValue AArch64TargetLowering::LowerOperation(SDValue Op,
+SDValue AArch64TargetLowering::LowerOperationDefault(SDValue Op,
                                                      SelectionDAG &DAG) const {
   LLVM_DEBUG(dbgs() << "Custom lowering: ");
   LLVM_DEBUG(Op.dump());
 
   switch (Op.getOpcode()) {
   default:
     llvm_unreachable("unimplemented operand");
     return SDValue();
   case ISD::BITCAST:
case ISD::ATOMIC_LOAD_SUB:
     return LowerATOMIC_LOAD_SUB(Op, DAG);
   case ISD::ATOMIC_LOAD_AND:
     return LowerATOMIC_LOAD_AND(Op, DAG);
   case ISD::DYNAMIC_STACKALLOC:
     return LowerDYNAMIC_STACKALLOC(Op, DAG);
   }
 }
 
+// Return the largest legal scalable vector type that matches VT's element type.
+static EVT getContainerForFixedVector(SelectionDAG &DAG, EVT VT) {
+  assert(VT.isFixedVector() && DAG.getTargetLoweringInfo().isTypeLegal(VT) &&
+         "Expected legal fixed length vector!");
+  switch (VT.getVectorElementType().getSimpleVT().SimpleTy) {
+  default: llvm_unreachable("unimplemented operand");
+  case MVT::i8:  return EVT(MVT::nxv16i8);
+  case MVT::i16: return EVT(MVT::nxv8i16);
+  case MVT::i32: return EVT(MVT::nxv4i32);
+  case MVT::i64: return EVT(MVT::nxv2i64);
+  case MVT::f16: return EVT(MVT::nxv8f16);
+  case MVT::f32: return EVT(MVT::nxv4f32);
+  case MVT::f64: return EVT(MVT::nxv2f64);
+  }
+}
+
+// Return a PTRUE with active lanes corresponding to the extent of VT.
+static SDValue CreatePredicateForFixedVector(SelectionDAG &DAG, SDLoc &DL,
+                                             EVT VT) {
+  assert(VT.isFixedVector() && DAG.getTargetLoweringInfo().isTypeLegal(VT) &&
+         "Expected legal fixed length vector!");
+
+  int PgPattern;
+  switch (VT.getVectorNumElements()) {
+  default: llvm_unreachable("unimplemented operand");
+  case 32: PgPattern = AArch64SVEPredPattern::vl32; break;
+  case 16: PgPattern = AArch64SVEPredPattern::vl16; break;
+  case  8: PgPattern = AArch64SVEPredPattern::vl8; break;
+  case  4: PgPattern = AArch64SVEPredPattern::vl4; break;
+  case  2: PgPattern = AArch64SVEPredPattern::vl2; break;
+  case  1: PgPattern = AArch64SVEPredPattern::vl1; break;
+  }
+
+#if 0
+  // This allows the use of unpredicated instructions when available.
+  if (VT.getSizeInBits() == Subtarget->getMaxSVEVectorSizeInBits())
+    PgPattern = AArch64SVEPredPattern::all;
+#endif
+
+  MVT MaskVT;
+  switch (VT.getVectorElementType().getSimpleVT().SimpleTy) {
+  default: llvm_unreachable("unimplemented operand");
+  case MVT::i8:  MaskVT = MVT::nxv16i1; break;
+  case MVT::i16:
+  case MVT::f16: MaskVT = MVT::nxv8i1; break;
+  case MVT::i32:
+  case MVT::f32: MaskVT = MVT::nxv4i1; break;
+  case MVT::i64:
+  case MVT::f64: MaskVT = MVT::nxv2i1; break;
+  }
+
+  return DAG.getNode(AArch64ISD::PTRUE, DL, MaskVT,
+                     DAG.getTargetConstant(PgPattern, DL, MVT::i64));
+}
+
+// Grow V to consume an entire SVE register.
+static SDValue ConvertToScalableVector(SelectionDAG &DAG, EVT VT, SDValue V) {
+  assert(VT.isScalableVector() &&
+         "Expected to convert into a scalable vector!");
+  assert(V.getValueType().isVector() && !V.getValueType().isScalableVector() &&
+         "Expected a fixed length vector operand!");
+  SDLoc DL(V);
+  SDValue Zero = DAG.getConstant(0, DL, MVT::i64);
+  return DAG.getNode(ISD::INSERT_SUBVECTOR, DL, VT, DAG.getUNDEF(VT), V, Zero);
+}
+
+// Shrink V so it's just big enough to maintain a VT's worth of data.
+static SDValue ConvertFromScalableVector(SelectionDAG &DAG, EVT VT, SDValue V) {
+  assert(VT.isFixedVector() &&
+         "Expected to convert into a fixed length vector!");
+  assert(V.getValueType().isScalableVector() &&
+         "Expected a scalable vector operand!");
+  SDLoc DL(V);
+  SDValue Zero = DAG.getConstant(0, DL, MVT::i64);
+  return DAG.getNode(ISD::EXTRACT_SUBVECTOR, DL, VT, V, Zero);
+}
+
+static SDValue LowerFixedVectorLoadToSVE(SelectionDAG &DAG, SDValue Op) {
+  auto Load = cast<LoadSDNode>(Op);
+
+  SDLoc DL(Op);
+  EVT VT = Op.getValueType();
+  EVT ContainerVT = getContainerForFixedVector(DAG, VT);
+
+  auto NewLoad = DAG.getMaskedLoad(ContainerVT, DL,
+                                   Load->getChain(),
+                                   Load->getBasePtr(),
+                                   Load->getOffset(),
+                                   CreatePredicateForFixedVector(DAG, DL, VT),
+                                   DAG.getUNDEF(ContainerVT),
+                                   Load->getMemoryVT(),
+                                   Load->getMemOperand(),
+                                   Load->getAddressingMode(),
+                                   Load->getExtensionType());
+
+  auto Result = ConvertFromScalableVector(DAG, VT, NewLoad);
+  SDValue MergedValues[2] = { Result, Load->getChain() };
+  return DAG.getMergeValues(MergedValues, DL);
+}
+
+static SDValue LowerFixedVectorStoreToSVE(SelectionDAG &DAG, SDValue Op) {
+  auto Store = cast<StoreSDNode>(Op);
+
+  SDLoc DL(Op);
+  EVT VT = Store->getValue().getValueType();
+  EVT ContainerVT = getContainerForFixedVector(DAG, VT);
+
+  auto NewValue = ConvertToScalableVector(DAG, ContainerVT, Store->getValue());
+  return DAG.getMaskedStore(Store->getChain(), DL,
+                            NewValue,
+                            Store->getBasePtr(),
+                            Store->getOffset(),
+                            CreatePredicateForFixedVector(DAG, DL, VT),
+                            Store->getMemoryVT(),
+                            Store->getMemOperand(),
+                            Store->getAddressingMode(),
+                            Store->isTruncatingStore());
+}
+
+SDValue AArch64TargetLowering::LowerOperation(SDValue Op,
+                                              SelectionDAG &DAG) const {
+  if (!useSVEForFixedLengthVectors())
+    return LowerOperationDefault(Op, DAG);
+
+  SDLoc DL(Op);
+
+  // If we get here then we have a vector operation whose operands are legal
+  // but too big to be implemented using Neon.  Lower the operation to SVE.
+
+  LLVM_DEBUG(dbgs() << "SVE lowering: ");
+  LLVM_DEBUG(Op.dump());
+
+  switch (Op.getOpcode()) {
+    default:
+      return LowerOperationDefault(Op, DAG);
+
+    // EXTRACT_SUBVECTOR/INSERT_SUBVECTOR are used to "cast" between scalable
+    // and fixed length vector types.  These represent legal operations with
+    // everything else requiring custom handling.
+    case ISD::EXTRACT_SUBVECTOR: {
+      EVT VT = Op.getValueType();
+      EVT InVT = Op.getOperand(0).getValueType();
+
+      if (VT.isFixedVector() && InVT.isFixedVector()) {
+        if (!UseSVEForAll && InVT.getSizeInBits() <= 128)
+          return Op; // Use Neon
+
+        return SDValue(); // Expand fixed length extract...
+      }
+      return Op;          // ...with everything else being legal.
+    }
+    case ISD::INSERT_SUBVECTOR: {
+      EVT VT = Op.getValueType();
+      EVT InVT = Op.getOperand(1).getValueType();
+
+      if (VT.isFixedVector() && InVT.isFixedVector()) {
+        if (!UseSVEForAll && VT.getSizeInBits() <= 128)
+          return Op; // Use Neon.
+
+        return SDValue(); // Expand fixed length inserts...
+      }
+      return Op;          // ...with everything else being legal.
+    }
+
+    // Convert all vector loads to masked_loads.
+    case ISD::LOAD: {
+      EVT VT = Op.getValueType();
+      if (!VT.isFixedVector())
+        return LowerOperationDefault(Op, DAG);
+
+      if (!UseSVEForAll && VT.getSizeInBits() <= 128)
+        return Op; // Use Neon.
+      return LowerFixedVectorLoadToSVE(DAG, Op);
+    }
+
+    // Convert all vector stores to masked_stores.
+    case ISD::STORE: {
+      EVT InVT = cast<StoreSDNode>(Op)->getValue().getValueType();
+      if (!InVT.isFixedVector())
+        return LowerOperationDefault(Op, DAG);
+
+      if (!UseSVEForAll && InVT.getSizeInBits() <= 128)
+        return Op; // Use Neon.
+      return LowerFixedVectorStoreToSVE(DAG, Op);
+    }
+  }
+}
+
+// Should we expand the build vector with shuffles?
+bool AArch64TargetLowering::shouldExpandBuildVectorWithShuffles(EVT VT,
+      unsigned DefinedValues) const {
+  if (useSVEForFixedLengthVectors())
+    return false;
+
+  return TargetLowering::shouldExpandBuildVectorWithShuffles(VT, DefinedValues);
+}
+
+bool AArch64TargetLowering::useSVEForFixedLengthVectors() const {
+  // Prefer Neon unless larger SVE registers are available.
+  return Subtarget->hasSVE() && Subtarget->getMinSVEVectorSizeInBits() >= 256;
+}
+
 //===----------------------------------------------------------------------===//
 //                      Calling Convention Implementation
 //===----------------------------------------------------------------------===//
 
 /// Selects the correct CCAssignFn for a given CallingConvention value.
 CCAssignFn *AArch64TargetLowering::CCAssignFnForCall(CallingConv::ID CC,
                                                      bool IsVarArg) const {
   switch (CC) {

llvm/lib/Target/AArch64/AArch64Subtarget.h

bool isCallingConvWin64(CallingConv::ID CC) const {
     case CallingConv::Win64:
       return true;
     default:
       return false;
     }
   }
 
   void mirFileLoaded(MachineFunction &MF) const override;
+
+  unsigned getMaxSVEVectorSizeInBits() const;
+  unsigned getMinSVEVectorSizeInBits() const;
 };
 } // End llvm namespace
 
 #endif

llvm/lib/Target/AArch64/AArch64Subtarget.cpp

 UseAddressTopByteIgnored("aarch64-use-tbi", cl::desc("Assume that top byte of "
                          "an address is ignored"), cl::init(false), cl::Hidden);
 
 static cl::opt<bool>
     UseNonLazyBind("aarch64-enable-nonlazybind",
                    cl::desc("Call nonlazybind functions via direct GOT load"),
                    cl::init(false), cl::Hidden);
 
+static cl::opt<unsigned>
+SVEVectorBitsMax("aarch64-sve-vector-bits-max",
+                 cl::desc("Assume SVE vector registers are at most this big, "
+                          "with zero meaning no maximum size is assumed."),
+                 cl::init(0), cl::Hidden);
+
+static cl::opt<unsigned>
+SVEVectorBitsMin("aarch64-sve-vector-bits-min",
+                 cl::desc("Assume SVE vector registers are at least this big, "
+                          "with zero meaning no minimum size is assumed."),
+                 cl::init(0), cl::Hidden);
+
 AArch64Subtarget &
 AArch64Subtarget::initializeSubtargetDependencies(StringRef FS,
                                                   StringRef CPUString) {
   // Determine default and user-specified characteristics
 
   if (CPUString.empty())
     CPUString = "generic";
 
void AArch64Subtarget::mirFileLoaded(MachineFunction &MF) const {
   // We usually compute max call frame size after ISel. Do the computation now
   // if the .mir file didn't specify it. Note that this will probably give you
   // bogus values after PEI has eliminated the callframe setup/destroy pseudo
   // instructions, specify explicitly if you need it to be correct.
   MachineFrameInfo &MFI = MF.getFrameInfo();
   if (!MFI.isMaxCallFrameSizeComputed())
     MFI.computeMaxCallFrameSize(MF);
 }
+
+unsigned AArch64Subtarget::getMaxSVEVectorSizeInBits() const {
+  assert(HasSVE && "Tried to get SVE vector length without SVE support!");
+  assert(((SVEVectorBitsMax >= SVEVectorBitsMin) || (SVEVectorBitsMax == 0)) &&
+         "Minimum SVE vector size cannot be larger that its maximum!");
+  return SVEVectorBitsMax;
+}
+
+unsigned AArch64Subtarget::getMinSVEVectorSizeInBits() const {
+  assert(HasSVE && "Tried to get SVE vector length without SVE support!");
+  assert(((SVEVectorBitsMax >= SVEVectorBitsMin) || (SVEVectorBitsMax == 0)) &&
+         "Minimum SVE vector size cannot be larger that its maximum!");
+  return SVEVectorBitsMin;
+}

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

if (Vector) {
         return 32;
       return 0;
     }
     return 31;
   }
 
   unsigned getRegisterBitWidth(bool Vector) const {
     if (Vector) {
+      if (ST->hasSVE())
+        return std::max(ST->getMinSVEVectorSizeInBits(), 128u);
       if (ST->hasNEON())
         return 128;
       return 0;
     }
     return 64;
   }
 
   unsigned getMinVectorRegisterBitWidth() {
Value *getOrCreateResultFromMemIntrinsic(IntrinsicInst *Inst,
                                            Type *ExpectedType);
 
   bool getTgtMemIntrinsic(IntrinsicInst *Inst, MemIntrinsicInfo &Info);
 
   bool isLegalMaskedLoadStore(Type *DataType, MaybeAlign Alignment) {
     if (!isa<VectorType>(DataType) || !ST->hasSVE())
       return false;
 
+    if (ST->getMinSVEVectorSizeInBits() != 0)
+      return false; // HACK
+
     Type *Ty = DataType->getVectorElementType();
     if (Ty->isHalfTy() || Ty->isFloatTy() || Ty->isDoubleTy())
       return true;
 
     if (Ty->isIntegerTy(8) || Ty->isIntegerTy(16) ||
         Ty->isIntegerTy(32) || Ty->isIntegerTy(64))
       return true;