[AArch64][SME2] Add CodeGen support for target("aarch64.svcount").

[sdesmalen]

Summary

This patch adds AArch64 CodeGen support such that the type can be passed
and returned to/from functions, and also adds support to use this type in
load/store operations and PHI nodes.

[sdesmalen]

Rebased patch to use the target("aarch64.svcount") type.

[paulwalker-arm]

My class PPRClass related comment is possibly problematic. My preference is to have a clean split between the two predicate types but I realise this is the start of the journey so if making the change results is too much churn then I'm happy to defer it to a refactoring exercise once the initial support has settled. That said, I'd still prefer to see how custom lowering works out rather than duplicating isel patterns.

I'm just going to throw this out there without giving it much thought but what about enabling the ability to ISD::BITCAST between svcount and nxv16i1? I think this better reflects what's going on compared to using reinterpret_cast. It also means we might get "free" operation legalisation by using setOperationPromotedToType.

llvm/include/llvm/Support/MachineValueType.h

407–412

This is more extensible but would return SimpleTy == MVT::aarch64svcount; be so bad?

llvm/lib/CodeGen/CodeGenPrepare.cpp

7699

Do you think it's worth Type having an isScalable() function? if this will be a common idiom.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

17862–17863

What is it about this combine that's problematic? Can you add a code comment for the rational.

Is it specially related to scalable target types or just target types in general?

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

6482–6483

Is this type specific or is it worth adding VT.isScalable()?

9309–9321

Can this be simplified to:

REINTERPRET_CAST(SELECT(CCVal, REINTERPRET_CAST(TVAL), REINTERPRET_CAST(FVAL)))

Then we'll renter this function and reuse the existing predicate lowering code. Perhaps you tried this and it didn't work out?

llvm/lib/Target/AArch64/AArch64RegisterInfo.td

894

Would it be overkill for aarch64svcount to have its own register class? I'm thinking it'll be nice to have isel protect us from incorrectly mixing predicate types and instructions. I'm also not sure if things like SDTCisSameNumEltsAs will cause is trouble, although I guess you've not hit anything so far.

The sequence below is using the non-count naming. Does this matter?

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

2799

Can the loads and stores (well anything that isn't a native svcount instruction) be done during lower?

[sdesmalen]

• Added EVT::isScalableVT() and refactored code to use it
• Use BITCAST instead of REINTERPRET_CAST
• Custom lower load/store of svcount to use nxv16i1
• Simplified lowering code in LowerSELECT
• Replaced the blanket bailout in visitLOAD/visitSTORE in DAGCombiner with more specific bailouts

[sdesmalen]

Thanks for the review @paulwalker-arm, I think I've addressed most of your comments, with the exception of giving predicate-as-counter its own register class, as that one probably makes more sense to pull out into a separate patch.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

17862–17863

This bailout was a bit too rigorous and I've removed it in favour of more specific bailouts.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

9309–9321

Thanks for the suggestion, using SELECT worked fine.

llvm/lib/Target/AArch64/AArch64RegisterInfo.td

894

I tried to give aarch64svcount it's own register class, but this requires changes (to add a new register class, etc) that complicate this patch quite a bit. I can follow this up in a separate patch.

The sequence below is using the non-count naming. Does this matter?

No that's fine, because it uses the same P registers.

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

420–421

I've tried and now there is isel for ISD::BITCAST this can be

setOperationPromotedToType(ISD::LOAD, MVT::aarch64svcount, MVT::nxv16i1);
setOperationPromotedToType(ISD::STORE, MVT::aarch64svcount, MVT::nxv16i1);

422–423

Do code generation tests for these exist?

llvm/lib/Target/AArch64/AArch64RegisterInfo.td

894

Thanks for the investigation. Sure, having this as a follow on patch works for me.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

2537

Does PNR exist? or perhaps it's a typo? When experimenting with setOperationPromotedToType I hit build errors and had to change these entries.

[sdesmalen]

• Use PromoteToType for svcount_t -> nxv16i1
• PNR -> PPR in one of the patterns
• Added tests for select

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

420–421

I'm surprised this worked, as the term 'Promote' makes me think it only works on promoted (element) types, not a bitcast. I've removed the custom lowering code now.

422–423

Thanks for the reminder, I previously added them as part of D136863. I've added them to this patch now.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

2537

Something probably went wrong with splitting up the patches after I tried adding the predicate-as-counter register class, fixed now.

[paulwalker-arm]

A couple of minor issues but otherwise looks good.

llvm/include/llvm/CodeGen/ValueTypes.h

125

Can you use isZero() here?

llvm/test/CodeGen/AArch64/sme-aarch64-svcount.ll

169–173

Commented out test.

llvm/include/llvm/CodeGen/ValueTypes.h

EVT changeTypeToInteger() {
       if (isSimple())
         return getSimpleVT().changeTypeToInteger();
       return changeExtendedTypeToInteger();
     }
 
     /// Test if the given EVT has zero size, this will fail if called on a
     /// scalable type
     bool isZeroSized() const {
-      return !isScalableVector() && getSizeInBits() == 0;
+      return getSizeInBits().getKnownMinValue() == 0;
     }
 
     /// Test if the given EVT is simple (as opposed to being extended).
     bool isSimple() const {
       return V.SimpleTy != MVT::INVALID_SIMPLE_VALUE_TYPE;
     }
 
     /// Test if the given EVT is extended (as opposed to being simple).
bool isInteger() const {
       return isSimple() ? V.isInteger() : isExtendedInteger();
     }
 
     /// Return true if this is an integer, but not a vector.
     bool isScalarInteger() const {
       return isSimple() ? V.isScalarInteger() : isExtendedScalarInteger();
     }
 
+    /// Return true if this is a vector type where the runtime
+    /// length is machine dependent
+    bool isScalableTargetExtVT() const {
+      return isSimple() && V.isScalableTargetExtVT();
+    }
+
     /// Return true if this is a vector value type.
     bool isVector() const {
       return isSimple() ? V.isVector() : isExtendedVector();
     }
 
     /// Return true if this is a vector type where the runtime
     /// length is machine dependent
     bool isScalableVector() const {
       return isSimple() ? V.isScalableVector() : isExtendedScalableVector();
     }
 
     bool isFixedLengthVector() const {
       return isSimple() ? V.isFixedLengthVector()
                         : isExtendedFixedLengthVector();
     }
 
+    /// Return true if the type is a scalable type.
+    bool isScalableVT() const {
+      return isScalableVector() || isScalableTargetExtVT();
+    }
+
     /// Return true if this is a 16-bit vector type.
     bool is16BitVector() const {
       return isSimple() ? V.is16BitVector() : isExtended16BitVector();
     }
 
     /// Return true if this is a 32-bit vector type.
     bool is32BitVector() const {
       return isSimple() ? V.is32BitVector() : isExtended32BitVector();

llvm/include/llvm/CodeGen/ValueTypes.td

 def x86mmx    : ValueType<64,   188>;  // X86 MMX value
 def FlagVT    : ValueType<0,    189>;  // Pre-RA sched glue
 def isVoid    : ValueType<0,    190>;  // Produces no value
 def untyped   : ValueType<8,    191>;  // Produces an untyped value
 def funcref   : ValueType<0,    192>;  // WebAssembly's funcref type
 def externref : ValueType<0,    193>;  // WebAssembly's externref type
 def x86amx    : ValueType<8192, 194>;  // X86 AMX value
 def i64x8     : ValueType<512,  195>;  // 8 Consecutive GPRs (AArch64)
+def aarch64svcount
+              : ValueType<16,   196>;  // AArch64 predicate-as-counter
 
 def token      : ValueType<0, 248>;  // TokenTy
 def MetadataVT : ValueType<0, 249>;  // Metadata
 
 // Pseudo valuetype mapped to the current pointer size to any address space.
 // Should only be used in TableGen.
 def iPTRAny    : ValueType<0, 250>;
 

llvm/include/llvm/Support/MachineValueType.h

enum SimpleValueType : uint8_t {
       Untyped        = 191,    // This value takes a register, but has
                                // unspecified type.  The register class
                                // will be determined by the opcode.
 
       funcref        = 192,    // WebAssembly's funcref type
       externref      = 193,    // WebAssembly's externref type
       x86amx         = 194,    // This is an X86 AMX value
       i64x8          = 195,    // 8 Consecutive GPRs (AArch64)
+      aarch64svcount = 196,    // AArch64 predicate-as-counter
 
       FIRST_VALUETYPE =  1,    // This is always the beginning of the list.
-      LAST_VALUETYPE = i64x8,  // This always remains at the end of the list.
+      LAST_VALUETYPE = aarch64svcount, // This always remains at the end of the list.
       VALUETYPE_SIZE = LAST_VALUETYPE + 1,
 
       // This is the current maximum for LAST_VALUETYPE.
       // MVT::MAX_ALLOWED_VALUETYPE is used for asserts and to size bit vectors
       // This value must be a multiple of 32.
       MAX_ALLOWED_VALUETYPE = 224,
 
       // A value of type llvm::TokenTy
public:
 
     /// Return true if this is a vector value type where the
     /// runtime length is machine dependent
     bool isScalableVector() const {
       return (SimpleTy >= MVT::FIRST_SCALABLE_VECTOR_VALUETYPE &&
               SimpleTy <= MVT::LAST_SCALABLE_VECTOR_VALUETYPE);
     }
 
+    /// Return true if this is a custom target type that has a scalable size.
+    bool isScalableTargetExtVT() const {
+      return SimpleTy == MVT::aarch64svcount;
+    }
+
+    /// Return true if the type is a scalable type.
+    bool isScalableVT() const {
+      return isScalableVector() || isScalableTargetExtVT();
+    }
+
     bool isFixedLengthVector() const {
       return (SimpleTy >= MVT::FIRST_FIXEDLEN_VECTOR_VALUETYPE &&
               SimpleTy <= MVT::LAST_FIXEDLEN_VECTOR_VALUETYPE);
     }
 
     /// Return true if this is a 16-bit vector type.
     bool is16BitVector() const {
       return (SimpleTy == MVT::v2i8  || SimpleTy == MVT::v1i16 ||
TypeSize getSizeInBits() const {
       case nxv8i1: return TypeSize::Scalable(8);
       case i16 :
       case f16:
       case bf16:
       case v16i1:
       case v2i8:
       case v1i16:
       case v1f16: return TypeSize::Fixed(16);
+      case aarch64svcount:
       case nxv16i1:
       case nxv2i8:
       case nxv1i16:
       case nxv1bf16:
       case nxv1f16: return TypeSize::Scalable(16);
       case f32 :
       case i32 :
       case v32i1:

llvm/lib/Analysis/Loads.cpp

 }
 
 bool llvm::isDereferenceableAndAlignedPointer(
     const Value *V, Type *Ty, Align Alignment, const DataLayout &DL,
     const Instruction *CtxI, AssumptionCache *AC, const DominatorTree *DT,
     const TargetLibraryInfo *TLI) {
   // For unsized types or scalable vectors we don't know exactly how many bytes
   // are dereferenced, so bail out.
-  if (!Ty->isSized() || isa<ScalableVectorType>(Ty))
+  if (!Ty->isSized() || Ty->isScalableTy())
     return false;
 
   // When dereferenceability information is provided by a dereferenceable
   // attribute, we know exactly how many bytes are dereferenceable. If we can
   // determine the exact offset to the attributed variable, we can use that
   // information here.
 
   APInt AccessSize(DL.getPointerTypeSizeInBits(V->getType()),

llvm/lib/CodeGen/CodeGenPrepare.cpp

static bool splitMergedValStore(StoreInst &SI, const DataLayout &DL,
                                 const TargetLowering &TLI) {
   // Handle simple but common cases only.
   Type *StoreType = SI.getValueOperand()->getType();
 
   // The code below assumes shifting a value by <number of bits>,
   // whereas scalable vectors would have to be shifted by
   // <2log(vscale) + number of bits> in order to store the
   // low/high parts. Bailing out for now.
-  if (isa<ScalableVectorType>(StoreType))
+  if (StoreType->isScalableTy())
     return false;
 
   if (!DL.typeSizeEqualsStoreSize(StoreType) ||
       DL.getTypeSizeInBits(StoreType) == 0)
     return false;
 
   unsigned HalfValBitSize = DL.getTypeSizeInBits(StoreType) / 2;
   Type *SplitStoreType = Type::getIntNTy(SI.getContext(), HalfValBitSize);

llvm/lib/CodeGen/LowLevelType.cpp

if (auto VTy = dyn_cast<VectorType>(&Ty)) {
     return LLT::vector(EC, ScalarTy);
   }
 
   if (auto PTy = dyn_cast<PointerType>(&Ty)) {
     unsigned AddrSpace = PTy->getAddressSpace();
     return LLT::pointer(AddrSpace, DL.getPointerSizeInBits(AddrSpace));
   }
 
-  if (Ty.isSized()) {
+  if (Ty.isSized() && !Ty.isScalableTargetExtTy()) {
     // Aggregates are no different from real scalars as far as GlobalISel is
     // concerned.
     auto SizeInBits = DL.getTypeSizeInBits(&Ty);
     assert(SizeInBits != 0 && "invalid zero-sized type");
     return LLT::scalar(SizeInBits);
   }
 
   return LLT();

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::ForwardStoreValueToDirectLoad(LoadSDNode *LD) {
 
   // There are two cases to consider here:
   //  1. The store is fixed width and the load is scalable. In this case we
   //     don't know at compile time if the store completely envelops the load
   //     so we abandon the optimisation.
   //  2. The store is scalable and the load is fixed width. We could
   //     potentially support a limited number of cases here, but there has been
   //     no cost-benefit analysis to prove it's worth it.
-  bool LdStScalable = LDMemType.isScalableVector();
-  if (LdStScalable != STMemType.isScalableVector())
+  bool LdStScalable = LDMemType.isScalableVT();
+  if (LdStScalable != STMemType.isScalableVT())
     return SDValue();
 
   // If we are dealing with scalable vectors on a big endian platform the
   // calculation of offsets below becomes trickier, since we do not know at
   // compile time the absolute size of the vector. Until we've done more
   // analysis on big-endian platforms it seems better to bail out for now.
   if (LdStScalable && DAG.getDataLayout().isBigEndian())
     return SDValue();
SDValue DAGCombiner::ForwardStoreValueToDirectLoad(LoadSDNode *LD) {
   return SDValue();
 }
 
 SDValue DAGCombiner::visitLOAD(SDNode *N) {
   LoadSDNode *LD  = cast<LoadSDNode>(N);
   SDValue Chain = LD->getChain();
   SDValue Ptr   = LD->getBasePtr();
 
   // If load is not volatile and there are no uses of the loaded value (and
   // the updated indexed value in case of indexed loads), change uses of the
   // chain value into uses of the chain input (i.e. delete the dead load).
   // TODO: Allow this for unordered atomics (see D66309)
   if (LD->isSimple()) {
     if (N->getValueType(1) == MVT::Other) {
       // Unindexed loads.
       if (!N->hasAnyUseOfValue(0)) {
         // It's not safe to use the two value CombineTo variant here. e.g.
         // v1, chain2 = load chain1, loc
bool DAGCombiner::mergeConsecutiveStores(StoreSDNode *St) {
   if (OptLevel == CodeGenOpt::None || !EnableStoreMerging)
     return false;
 
   // TODO: Extend this function to merge stores of scalable vectors.
   // (i.e. two <vscale x 8 x i8> stores can be merged to one <vscale x 16 x i8>
   // store since we know <vscale x 16 x i8> is exactly twice as large as
   // <vscale x 8 x i8>). Until then, bail out for scalable vectors.
   EVT MemVT = St->getMemoryVT();
-  if (MemVT.isScalableVector())
+  if (MemVT.isScalableVT())
     return false;
   if (!MemVT.isSimple() || MemVT.getSizeInBits() * 2 > MaximumLegalStoreInBits)
     return false;
 
   // This function cannot currently deal with non-byte-sized memory sizes.
   int64_t ElementSizeBytes = MemVT.getStoreSize();
   if (ElementSizeBytes * 8 != (int64_t)MemVT.getSizeInBits())
     return false;
bool DAGCombiner::parallelizeChainedStores(StoreSDNode *St) {
 
   // Do not handle stores to opaque types
   if (St->getMemoryVT().isZeroSized())
     return false;
 
   // BaseIndexOffset assumes that offsets are fixed-size, which
   // is not valid for scalable vectors where the offsets are
   // scaled by `vscale`, so bail out early.
-  if (St->getMemoryVT().isScalableVector())
+  if (St->getMemoryVT().isScalableVT())
     return false;
 
   // Add ST's interval.
   Intervals.insert(0, (St->getMemoryVT().getSizeInBits() + 7) / 8,
                    std::monostate{});
 
   while (StoreSDNode *Chain = dyn_cast<StoreSDNode>(STChain->getChain())) {
     if (Chain->getMemoryVT().isScalableVector())

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

if (TLI.splitValueIntoRegisterParts(DAG, DL, Val, Parts, NumParts, PartVT,
     return;
   EVT ValueVT = Val.getValueType();
 
   // Handle the vector case separately.
   if (ValueVT.isVector())
     return getCopyToPartsVector(DAG, DL, Val, Parts, NumParts, PartVT, V,
                                 CallConv);
 
-  unsigned PartBits = PartVT.getSizeInBits();
   unsigned OrigNumParts = NumParts;
   assert(DAG.getTargetLoweringInfo().isTypeLegal(PartVT) &&
          "Copying to an illegal type!");
 
   if (NumParts == 0)
     return;
 
   assert(!ValueVT.isVector() && "Vector case handled elsewhere");
   EVT PartEVT = PartVT;
   if (PartEVT == ValueVT) {
     assert(NumParts == 1 && "No-op copy with multiple parts!");
     Parts[0] = Val;
     return;
   }
 
+  unsigned PartBits = PartVT.getSizeInBits();
   if (NumParts * PartBits > ValueVT.getSizeInBits()) {
     // If the parts cover more bits than the value has, promote the value.
     if (PartVT.isFloatingPoint() && ValueVT.isFloatingPoint()) {
       assert(NumParts == 1 && "Do not know what to promote to!");
       Val = DAG.getNode(ISD::FP_EXTEND, DL, PartVT, Val);
     } else {
       if (ValueVT.isFloatingPoint()) {
         // FP values need to be bitcast, then extended if they are being put

llvm/lib/CodeGen/ValueTypes.cpp

std::string EVT::getEVTString() const {
   case MVT::Glue:      return "glue";
   case MVT::x86mmx:    return "x86mmx";
   case MVT::x86amx:    return "x86amx";
   case MVT::i64x8:     return "i64x8";
   case MVT::Metadata:  return "Metadata";
   case MVT::Untyped:   return "Untyped";
   case MVT::funcref:   return "funcref";
   case MVT::externref: return "externref";
+  case MVT::aarch64svcount:
+    return "aarch64svcount";
   }
 }
 
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 void EVT::dump() const {
   print(dbgs());
   dbgs() << "\n";
 }
Type *EVT::getTypeForEVT(LLVMContext &Context) const {
   case MVT::f16:     return Type::getHalfTy(Context);
   case MVT::bf16:    return Type::getBFloatTy(Context);
   case MVT::f32:     return Type::getFloatTy(Context);
   case MVT::f64:     return Type::getDoubleTy(Context);
   case MVT::f80:     return Type::getX86_FP80Ty(Context);
   case MVT::f128:    return Type::getFP128Ty(Context);
   case MVT::ppcf128: return Type::getPPC_FP128Ty(Context);
   case MVT::x86mmx:  return Type::getX86_MMXTy(Context);
+  case MVT::aarch64svcount:
+    return TargetExtType::get(Context, "aarch64.svcount");
   case MVT::x86amx:  return Type::getX86_AMXTy(Context);
   case MVT::i64x8:   return IntegerType::get(Context, 512);
   case MVT::externref:
     // pointer to opaque struct in addrspace(10)
     return PointerType::get(StructType::create(Context), 10);
   case MVT::funcref:
     // pointer to i8 addrspace(20)
     return PointerType::get(Type::getInt8Ty(Context), 20);
MVT MVT::getVT(Type *Ty, bool HandleUnknown){
   case Type::IntegerTyID:
     return getIntegerVT(cast<IntegerType>(Ty)->getBitWidth());
   case Type::HalfTyID:      return MVT(MVT::f16);
   case Type::BFloatTyID:    return MVT(MVT::bf16);
   case Type::FloatTyID:     return MVT(MVT::f32);
   case Type::DoubleTyID:    return MVT(MVT::f64);
   case Type::X86_FP80TyID:  return MVT(MVT::f80);
   case Type::X86_MMXTyID:   return MVT(MVT::x86mmx);
+  case Type::TargetExtTyID:
+    if (cast<TargetExtType>(Ty)->getName() == "aarch64.svcount")
+      return MVT(MVT::aarch64svcount);
+    if (HandleUnknown)
+      return MVT(MVT::Other);
+    llvm_unreachable("Unknown target ext type!");
   case Type::X86_AMXTyID:   return MVT(MVT::x86amx);
   case Type::FP128TyID:     return MVT(MVT::f128);
   case Type::PPC_FP128TyID: return MVT(MVT::ppcf128);
   case Type::PointerTyID:   return MVT(MVT::iPTR);
   case Type::FixedVectorTyID:
   case Type::ScalableVectorTyID: {
     VectorType *VTy = cast<VectorType>(Ty);
     return getVectorVT(

llvm/lib/Support/LowLevelType.cpp

 using namespace llvm;
 
 LLT::LLT(MVT VT) {
   if (VT.isVector()) {
     bool asVector = VT.getVectorMinNumElements() > 1;
     init(/*IsPointer=*/false, asVector, /*IsScalar=*/!asVector,
          VT.getVectorElementCount(), VT.getVectorElementType().getSizeInBits(),
          /*AddressSpace=*/0);
-  } else if (VT.isValid()) {
+  } else if (VT.isValid() && !VT.isScalableTargetExtVT()) {
     // Aggregates are no different from real scalars as far as GlobalISel is
     // concerned.
     init(/*IsPointer=*/false, /*IsVector=*/false, /*IsScalar=*/true,
          ElementCount::getFixed(0), VT.getSizeInBits(), /*AddressSpace=*/0);
   } else {
     IsScalar = false;
     IsPointer = false;
     IsVector = false;

llvm/lib/Target/AArch64/AArch64CallingConvention.td

def CC_AArch64_AAPCS : CallingConv<[
 
   CCIfType<[nxv16i8, nxv8i16, nxv4i32, nxv2i64, nxv2f16, nxv4f16, nxv8f16,
             nxv2bf16, nxv4bf16, nxv8bf16, nxv2f32, nxv4f32, nxv2f64],
            CCAssignToReg<[Z0, Z1, Z2, Z3, Z4, Z5, Z6, Z7]>>,
   CCIfType<[nxv16i8, nxv8i16, nxv4i32, nxv2i64, nxv2f16, nxv4f16, nxv8f16,
             nxv2bf16, nxv4bf16, nxv8bf16, nxv2f32, nxv4f32, nxv2f64],
            CCPassIndirect<i64>>,
 
-  CCIfType<[nxv1i1, nxv2i1, nxv4i1, nxv8i1, nxv16i1],
+  CCIfType<[nxv1i1, nxv2i1, nxv4i1, nxv8i1, nxv16i1, aarch64svcount],
            CCAssignToReg<[P0, P1, P2, P3]>>,
-  CCIfType<[nxv1i1, nxv2i1, nxv4i1, nxv8i1, nxv16i1],
+  CCIfType<[nxv1i1, nxv2i1, nxv4i1, nxv8i1, nxv16i1, aarch64svcount],
            CCPassIndirect<i64>>,
 
   // Handle i1, i8, i16, i32, i64, f32, f64 and v2f64 by passing in registers,
   // up to eight each of GPR and FPR.
   CCIfType<[i1, i8, i16], CCPromoteToType<i32>>,
   CCIfType<[i32], CCAssignToReg<[W0, W1, W2, W3, W4, W5, W6, W7]>>,
   // i128 is split to two i64s, we can't fit half to register X7.
   CCIfType<[i64], CCIfSplit<CCAssignToRegWithShadow<[X0, X2, X4, X6],
CCIfType<[v1i64, v2i32, v4i16, v8i8, v1f64, v2f32, v4f16, v4bf16],
       CCAssignToReg<[D0, D1, D2, D3, D4, D5, D6, D7]>>,
   CCIfType<[f128, v2i64, v4i32, v8i16, v16i8, v4f32, v2f64, v8f16, v8bf16],
       CCAssignToReg<[Q0, Q1, Q2, Q3, Q4, Q5, Q6, Q7]>>,
 
   CCIfType<[nxv16i8, nxv8i16, nxv4i32, nxv2i64, nxv2f16, nxv4f16, nxv8f16,
             nxv2bf16, nxv4bf16, nxv8bf16, nxv2f32, nxv4f32, nxv2f64],
            CCAssignToReg<[Z0, Z1, Z2, Z3, Z4, Z5, Z6, Z7]>>,
 
-  CCIfType<[nxv1i1, nxv2i1, nxv4i1, nxv8i1, nxv16i1],
+  CCIfType<[nxv1i1, nxv2i1, nxv4i1, nxv8i1, nxv16i1, aarch64svcount],
            CCAssignToReg<[P0, P1, P2, P3]>>
 ]>;
 
 // Vararg functions on windows pass floats in integer registers
 let Entry = 1 in
 def CC_AArch64_Win64_VarArg : CallingConv<[
   CCIfType<[f16, bf16], CCBitConvertToType<i16>>,
   CCIfType<[f32], CCBitConvertToType<i32>>,

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (Subtarget->useSVEForFixedLengthVectors()) {
           addRegisterClass(VT, &AArch64::ZPRRegClass);
 
       for (MVT VT : MVT::fp_fixedlen_vector_valuetypes())
         if (useSVEForFixedLengthVectorVT(VT))
           addRegisterClass(VT, &AArch64::ZPRRegClass);
     }
   }
 
+  if (Subtarget->hasSVE2p1() || Subtarget->hasSME2()) {
+    addRegisterClass(MVT::aarch64svcount, &AArch64::PPRRegClass);
+    setOperationAction(ISD::LOAD, MVT::aarch64svcount, Custom);
+    setOperationAction(ISD::STORE, MVT::aarch64svcount, Custom);
+    setOperationAction(ISD::SELECT, MVT::aarch64svcount, Custom);
+    setOperationAction(ISD::SELECT_CC, MVT::aarch64svcount, Expand);
+  }
+
   // Compute derived properties from the register classes
   computeRegisterProperties(Subtarget->getRegisterInfo());
 
   // Provide all sorts of operation actions
   setOperationAction(ISD::GlobalAddress, MVT::i64, Custom);
   setOperationAction(ISD::GlobalTLSAddress, MVT::i64, Custom);
   setOperationAction(ISD::SETCC, MVT::i32, Custom);
   setOperationAction(ISD::SETCC, MVT::i64, Custom);
SDValue AArch64TargetLowering::LowerSTORE(SDValue Op,
   StoreSDNode *StoreNode = cast<StoreSDNode>(Op);
   assert (StoreNode && "Can only custom lower store nodes");
 
   SDValue Value = StoreNode->getValue();
 
   EVT VT = Value.getValueType();
   EVT MemVT = StoreNode->getMemoryVT();
 
+  if (VT == MVT::aarch64svcount) {
+    SDValue StoreVal = DAG.getNode(ISD::BITCAST, Dl,
+                                   MVT::nxv16i1, StoreNode->getValue());
+    return DAG.getStore(StoreNode->getChain(), Dl, StoreVal,
+                        StoreNode->getBasePtr(), StoreNode->getMemOperand());
+  }
+
   if (VT.isVector()) {
     if (useSVEForFixedLengthVectorVT(
             VT,
             /*OverrideNEON=*/Subtarget->useSVEForFixedLengthVectors()))
       return LowerFixedLengthVectorStoreToSVE(Op, DAG);
 
     unsigned AS = StoreNode->getAddressSpace();
     Align Alignment = StoreNode->getAlign();

 }
 
 SDValue AArch64TargetLowering::LowerLOAD(SDValue Op,
                                          SelectionDAG &DAG) const {
   SDLoc DL(Op);
   LoadSDNode *LoadNode = cast<LoadSDNode>(Op);
   assert(LoadNode && "Expected custom lowering of a load node");
 
+  if (LoadNode->getMemoryVT() == MVT::aarch64svcount) {
+    SDValue Load = DAG.getLoad(
+        MVT::nxv16i1, DL, LoadNode->getChain(), LoadNode->getBasePtr(),
+        LoadNode->getPointerInfo(), LoadNode->getOriginalAlign());
+    SDValue Chain = SDValue(Load.getNode(), 1);
+    Load = DAG.getNode(ISD::BITCAST, DL, MVT::aarch64svcount, Load);
+    return DAG.getMergeValues({Load, Chain}, DL);
+  }
+
   if (LoadNode->getMemoryVT() == MVT::i64x8) {
     SmallVector<SDValue, 8> Ops;
     SDValue Base = LoadNode->getBasePtr();
     SDValue Chain = LoadNode->getChain();
     EVT PtrVT = Base.getValueType();
     for (unsigned i = 0; i < 8; i++) {
       SDValue Ptr = DAG.getNode(ISD::ADD, DL, PtrVT, Base,
                                 DAG.getConstant(i * 8, DL, PtrVT));
if (VA.isRegLoc()) {
       else if (RegVT == MVT::f64 || RegVT.is64BitVector())
         RC = &AArch64::FPR64RegClass;
       else if (RegVT == MVT::f128 || RegVT.is128BitVector())
         RC = &AArch64::FPR128RegClass;
       else if (RegVT.isScalableVector() &&
                RegVT.getVectorElementType() == MVT::i1) {
         FuncInfo->setIsSVECC(true);
         RC = &AArch64::PPRRegClass;
+      } else if (RegVT == MVT::aarch64svcount) {
+        FuncInfo->setIsSVECC(true);
+        RC = &AArch64::PPRRegClass;
       } else if (RegVT.isScalableVector()) {
         FuncInfo->setIsSVECC(true);
         RC = &AArch64::ZPRRegClass;
       } else
         llvm_unreachable("RegVT not supported by FORMAL_ARGUMENTS Lowering");
 
       // Transform the arguments in physical registers into virtual ones.
       Register Reg = MF.addLiveIn(VA.getLocReg(), RC);
if (VA.isRegLoc()) {
       // If this is an 8, 16 or 32-bit value, it is really passed promoted
       // to 64 bits.  Insert an assert[sz]ext to capture this, then
       // truncate to the right size.
       switch (VA.getLocInfo()) {
       default:
         llvm_unreachable("Unknown loc info!");
       case CCValAssign::Full:
         break;
       case CCValAssign::Indirect:
-        assert((VA.getValVT().isScalableVector() ||
-                Subtarget->isWindowsArm64EC()) &&
-               "Indirect arguments should be scalable on most subtargets");
+        assert(
+            (VA.getValVT().isScalableVT() || Subtarget->isWindowsArm64EC()) &&
         break;
       case CCValAssign::BCvt:
         ArgValue = DAG.getNode(ISD::BITCAST, DL, VA.getValVT(), ArgValue);
         break;
       case CCValAssign::AExt:
       case CCValAssign::SExt:
       case CCValAssign::ZExt:
         break;
if (VA.isRegLoc()) {
         break;
       }
 
       ArgValue = DAG.getExtLoad(ExtType, DL, VA.getLocVT(), Chain, FIN, PtrInfo,
                                 MemVT);
     }
 
     if (VA.getLocInfo() == CCValAssign::Indirect) {
-      assert(
-          (VA.getValVT().isScalableVector() || Subtarget->isWindowsArm64EC()) &&
+      assert((VA.getValVT().isScalableVT() ||
+              Subtarget->isWindowsArm64EC()) &&
              "Indirect arguments should be scalable on most subtargets");
 
       uint64_t PartSize = VA.getValVT().getStoreSize().getKnownMinValue();
       unsigned NumParts = 1;
       if (Ins[i].Flags.isInConsecutiveRegs()) {
         assert(!Ins[i].Flags.isInConsecutiveRegsLast());
         while (!Ins[i + NumParts - 1].Flags.isInConsecutiveRegsLast())
           ++NumParts;
       }
case CCValAssign::BCvt:
       break;
     case CCValAssign::Trunc:
       Arg = DAG.getZExtOrTrunc(Arg, DL, VA.getLocVT());
       break;
     case CCValAssign::FPExt:
       Arg = DAG.getNode(ISD::FP_EXTEND, DL, VA.getLocVT(), Arg);
       break;
     case CCValAssign::Indirect:
-      bool isScalable = VA.getValVT().isScalableVector();
+      bool isScalable = VA.getValVT().isScalableVT();
       assert((isScalable || Subtarget->isWindowsArm64EC()) &&
              "Indirect arguments should be scalable on most subtargets");
 
       uint64_t StoreSize = VA.getValVT().getStoreSize().getKnownMinValue();
       uint64_t PartSize = StoreSize;
       unsigned NumParts = 1;
       if (Outs[i].Flags.isInConsecutiveRegs()) {
         assert(!Outs[i].Flags.isInConsecutiveRegsLast());

 SDValue AArch64TargetLowering::LowerSELECT(SDValue Op,
                                            SelectionDAG &DAG) const {
   SDValue CCVal = Op->getOperand(0);
   SDValue TVal = Op->getOperand(1);
   SDValue FVal = Op->getOperand(2);
   SDLoc DL(Op);
 
   EVT Ty = Op.getValueType();
+  if (Ty == MVT::aarch64svcount) {
+    TVal = DAG.getNode(ISD::BITCAST, DL, MVT::nxv16i1, TVal);
+    FVal = DAG.getNode(ISD::BITCAST, DL, MVT::nxv16i1, FVal);
+    SDValue Sel =
+        DAG.getNode(ISD::SELECT, DL, MVT::nxv16i1, CCVal, TVal, FVal);
+    return DAG.getNode(ISD::BITCAST, DL, Ty, Sel);
+  }
+
   if (Ty.isScalableVector()) {
     SDValue TruncCC = DAG.getNode(ISD::TRUNCATE, DL, MVT::i1, CCVal);
     MVT PredVT = MVT::getVectorVT(MVT::i1, Ty.getVectorElementCount());
     SDValue SplatPred = DAG.getNode(ISD::SPLAT_VECTOR, DL, PredVT, TruncCC);
     return DAG.getNode(ISD::VSELECT, DL, Ty, SplatPred, TVal, FVal);
   }
 
   if (useSVEForFixedLengthVectorVT(Ty,
                                    Subtarget->forceStreamingCompatibleSVE())) {
     // FIXME: Ideally this would be the same as above using i1 types, however
     // for the moment we can't deal with fixed i1 vector types properly, so
     // instead extend the predicate to a result type sized integer vector.
     MVT SplatValVT = MVT::getIntegerVT(Ty.getScalarSizeInBits());
     MVT PredVT = MVT::getVectorVT(SplatValVT, Ty.getVectorElementCount());
bool AArch64TargetLowering::isLegalAddressingMode(const DataLayout &DL,
   if (AM.BaseGV)
     return false;
 
   // No reg+reg+imm addressing.
   if (AM.HasBaseReg && AM.BaseOffs && AM.Scale)
     return false;
 
   // FIXME: Update this method to support scalable addressing modes.
+  if (Ty->isScalableTargetExtTy())
+    return AM.HasBaseReg && !AM.BaseOffs && !AM.Scale;
+
   if (isa<ScalableVectorType>(Ty)) {
     uint64_t VecElemNumBytes =
         DL.getTypeSizeInBits(cast<VectorType>(Ty)->getElementType()) / 8;
     return AM.HasBaseReg && !AM.BaseOffs &&
            (AM.Scale == 0 || (uint64_t)AM.Scale == VecElemNumBytes);
   }
 
   // check reg + imm case:
static SDValue performSelectCombine(SDNode *N,
                                     TargetLowering::DAGCombinerInfo &DCI) {
   SelectionDAG &DAG = DCI.DAG;
   SDValue N0 = N->getOperand(0);
   EVT ResVT = N->getValueType(0);
 
   if (N0.getOpcode() != ISD::SETCC)
     return SDValue();
 
-  if (ResVT.isScalableVector())
+  if (ResVT.isScalableVT())
     return SDValue();
 
   // Make sure the SETCC result is either i1 (initial DAG), or i32, the lowered
   // scalar SetCCResultType. We also don't expect vectors, because we assume
   // that selects fed by vector SETCCs are canonicalized to VSELECT.
   assert((N0.getValueType() == MVT::i1 || N0.getValueType() == MVT::i32) &&
          "Scalar-SETCC feeding SELECT has unexpected result type!");
 
case AArch64::G_ADD_LOW:
     return true;
   default:
     break;
   }
   return TargetLoweringBase::shouldLocalize(MI, TTI);
 }
 
 bool AArch64TargetLowering::fallBackToDAGISel(const Instruction &Inst) const {
-  if (isa<ScalableVectorType>(Inst.getType()))
+  if (Inst.getType()->isScalableTy())
     return true;
 
   for (unsigned i = 0; i < Inst.getNumOperands(); ++i)
-    if (isa<ScalableVectorType>(Inst.getOperand(i)->getType()))
+    if (Inst.getOperand(i)->getType()->isScalableTy())
       return true;
 
   if (const AllocaInst *AI = dyn_cast<AllocaInst>(&Inst)) {
-    if (isa<ScalableVectorType>(AI->getAllocatedType()))
+    if (AI->getAllocatedType()->isScalableTy())
       return true;
   }
 
   // Checks to allow the use of SME instructions
   if (auto *Base = dyn_cast<CallBase>(&Inst)) {
     auto CallerAttrs = SMEAttrs(*Inst.getFunction());
     auto CalleeAttrs = SMEAttrs(*Base);
     if (CallerAttrs.requiresSMChange(CalleeAttrs,

llvm/lib/Target/AArch64/AArch64RegisterInfo.td

 class ZPRRegOp <string Suffix, AsmOperandClass C, ElementSizeEnum Size,
                 RegisterClass RC> : SVERegOp<Suffix, C, Size, RC> {}
 
 //******************************************************************************
 
 // SVE predicate register classes.
 class PPRClass<int firstreg, int lastreg> : RegisterClass<
                                   "AArch64",
-                                  [ nxv16i1, nxv8i1, nxv4i1, nxv2i1, nxv1i1 ], 16,
+                                  [ nxv16i1, nxv8i1, nxv4i1, nxv2i1, nxv1i1, aarch64svcount ], 16,
                                   (sequence "P%u", firstreg, lastreg)> {
   let Size = 16;
 }
 
 def PPR    : PPRClass<0, 15>;
 def PPR_3b : PPRClass<0, 7>; // Restricted 3 bit SVE predicate register class.
 def PPR_p8to15 : PPRClass<8, 15>;
 

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

let Predicates = [IsLE] in {
 
     def : Pat<(nxv16i8 (bitconvert (nxv8bf16 ZPR:$src))), (nxv16i8 ZPR:$src)>;
     def : Pat<(nxv8i16 (bitconvert (nxv8bf16 ZPR:$src))), (nxv8i16 ZPR:$src)>;
     def : Pat<(nxv4i32 (bitconvert (nxv8bf16 ZPR:$src))), (nxv4i32 ZPR:$src)>;
     def : Pat<(nxv2i64 (bitconvert (nxv8bf16 ZPR:$src))), (nxv2i64 ZPR:$src)>;
     def : Pat<(nxv8f16 (bitconvert (nxv8bf16 ZPR:$src))), (nxv8f16 ZPR:$src)>;
     def : Pat<(nxv4f32 (bitconvert (nxv8bf16 ZPR:$src))), (nxv4f32 ZPR:$src)>;
     def : Pat<(nxv2f64 (bitconvert (nxv8bf16 ZPR:$src))), (nxv2f64 ZPR:$src)>;
+
+    def : Pat<(nxv16i1 (bitconvert (aarch64svcount PNR:$src))), (nxv16i1 PPR:$src)>;
+    def : Pat<(aarch64svcount (bitconvert (nxv16i1 PPR:$src))), (aarch64svcount PNR:$src)>;
   }
 
   // These allow casting from/to unpacked predicate types.
   def : Pat<(nxv16i1 (reinterpret_cast (nxv16i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
   def : Pat<(nxv16i1 (reinterpret_cast (nxv8i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
   def : Pat<(nxv16i1 (reinterpret_cast (nxv4i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
   def : Pat<(nxv16i1 (reinterpret_cast (nxv2i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
   def : Pat<(nxv16i1 (reinterpret_cast (nxv1i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
multiclass unpred_store_predicate<ValueType Ty, Instruction Store> {
     def _fi : Pat<(store (Ty PPR:$val), (am_sve_fi GPR64sp:$base, simm9:$offset)),
                   (Store PPR:$val, GPR64sp:$base, simm9:$offset)>;
 
     def _default : Pat<(store (Ty PPR:$Val), GPR64:$base),
                   (Store PPR:$Val, GPR64:$base, (i64 0))>;
   }
 
   defm Pat_Store_P16 : unpred_store_predicate<nxv16i1, STR_PXI>;
 
   multiclass unpred_load_predicate<ValueType Ty, Instruction Load> {
     def _fi : Pat<(Ty (load (am_sve_fi GPR64sp:$base, simm9:$offset))),
                   (Load GPR64sp:$base, simm9:$offset)>;
 
     def _default : Pat<(Ty (load GPR64:$base)),
                   (Load GPR64:$base, (i64 0))>;
   }
 

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

static void handleMustTailForwardedRegisters(MachineIRBuilder &MIRBuilder,
   for (const auto &F : Forwards) {
     MBB.addLiveIn(F.PReg);
     MIRBuilder.buildCopy(Register(F.VReg), Register(F.PReg));
   }
 }
 
 bool AArch64CallLowering::fallBackToDAGISel(const MachineFunction &MF) const {
   auto &F = MF.getFunction();
-  if (isa<ScalableVectorType>(F.getReturnType()))
-    return true;
-  if (llvm::any_of(F.args(), [](const Argument &A) {
-        return isa<ScalableVectorType>(A.getType());
+  if (F.getReturnType()->isScalableTy() ||
+      llvm::any_of(F.args(), [](const Argument &A) {
+        return A.getType()->isScalableTy();
       }))
     return true;
   const auto &ST = MF.getSubtarget<AArch64Subtarget>();
   if (!ST.hasNEON() || !ST.hasFPARMv8()) {
     LLVM_DEBUG(dbgs() << "Falling back to SDAG because we don't support no-NEON\n");
     return true;
   }
 

llvm/test/CodeGen/AArch64/sme-aarch64-svcount-O3.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -O3 -mtriple=aarch64 -mattr=+sme -S < %s | FileCheck %s
+
+; Test PHI nodes are allowed with opaque scalable types.
+define target("aarch64.svcount") @test_alloca_store_reload(target("aarch64.svcount") %val0, target("aarch64.svcount") %val1, ptr %iptr, ptr %pptr, i64 %N) nounwind {
+; CHECK-LABEL: @test_alloca_store_reload(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    store i64 0, ptr [[IPTR:%.*]], align 4
+; CHECK-NEXT:    store target("aarch64.svcount") [[VAL0:%.*]], ptr [[PPTR:%.*]], align 2
+; CHECK-NEXT:    [[I1_PEEL:%.*]] = icmp eq i64 [[N:%.*]], 0
+; CHECK-NEXT:    br i1 [[I1_PEEL]], label [[LOOP_EXIT:%.*]], label [[LOOP_BODY:%.*]]
+; CHECK:       loop.body:
+; CHECK-NEXT:    [[IND:%.*]] = phi i64 [ [[IND_NEXT:%.*]], [[LOOP_BODY]] ], [ 1, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[IPTR_GEP:%.*]] = getelementptr i64, ptr [[IPTR]], i64 [[IND]]
+; CHECK-NEXT:    store i64 [[IND]], ptr [[IPTR_GEP]], align 4
+; CHECK-NEXT:    store target("aarch64.svcount") [[VAL1:%.*]], ptr [[PPTR]], align 2
+; CHECK-NEXT:    [[IND_NEXT]] = add i64 [[IND]], 1
+; CHECK-NEXT:    [[I1:%.*]] = icmp eq i64 [[IND]], [[N]]
+; CHECK-NEXT:    br i1 [[I1]], label [[LOOP_EXIT]], label [[LOOP_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK:       loop.exit:
+; CHECK-NEXT:    [[PHI_LCSSA:%.*]] = phi target("aarch64.svcount") [ [[VAL0]], [[ENTRY]] ], [ [[VAL1]], [[LOOP_BODY]] ]
+; CHECK-NEXT:    ret target("aarch64.svcount") [[PHI_LCSSA]]
+;
+entry:
+  br label %loop.body
+
+loop.body:
+  %ind = phi i64 [0, %entry], [%ind.next, %loop.body]
+  %phi = phi target("aarch64.svcount") [%val0, %entry], [%val1, %loop.body]
+  %iptr.gep = getelementptr i64, ptr %iptr, i64 %ind
+  store i64 %ind, ptr %iptr.gep
+  store target("aarch64.svcount") %phi, ptr %pptr
+  %ind.next = add i64 %ind, 1
+  %i1 = icmp eq i64 %ind, %N
+  br i1 %i1, label %loop.exit, label %loop.body
+
+loop.exit:
+  ret target("aarch64.svcount") %phi
+}

llvm/test/CodeGen/AArch64/sme-aarch64-svcount.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -O0 -mtriple=aarch64 -mattr=+sve2p1 < %s | FileCheck %s --check-prefixes=CHECK,CHECK-O0
+; RUN: llc -O3 -mtriple=aarch64 -mattr=+sve2p1 < %s | FileCheck %s --check-prefixes=CHECK,CHECK-O3
+
+;
+; Test simple loads, stores and return.
+;
+define target("aarch64.svcount") @test_load(ptr %ptr) nounwind {
+; CHECK-LABEL: test_load:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr p0, [x0]
+; CHECK-NEXT:    ret
+  %res = load target("aarch64.svcount"), ptr %ptr
+  ret target("aarch64.svcount") %res
+}
+
+define void @test_store(ptr %ptr, target("aarch64.svcount") %val) nounwind {
+; CHECK-LABEL: test_store:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    str p0, [x0]
+; CHECK-NEXT:    ret
+  store target("aarch64.svcount") %val, ptr %ptr
+  ret void
+}
+
+define target("aarch64.svcount") @test_alloca_store_reload(target("aarch64.svcount") %val) nounwind {
+; CHECKO0-LABEL: test_alloca_store_reload:
+; CHECKO0:       // %bb.0:
+; CHECKO0-NEXT:    sub sp, sp, #16
+; CHECKO0-NEXT:    add x8, sp, #14
+; CHECKO0-NEXT:    str p0, [x8]
+; CHECKO0-NEXT:    ldr p0, [x8]
+; CHECKO0-NEXT:    add sp, sp, #16
+; CHECKO0-NEXT:    ret
+;
+; CHECKO3-LABEL: test_alloca_store_reload:
+; CHECKO3:       // %bb.0:
+; CHECKO3-NEXT:    sub sp, sp, #16
+; CHECKO3-NEXT:    add x8, sp, #14
+; CHECKO3-NEXT:    str p0, [x8]
+; CHECKO3-NEXT:    add sp, sp, #16
+; CHECKO3-NEXT:    ret
+; CHECK-O0-LABEL: test_alloca_store_reload:
+; CHECK-O0:       // %bb.0:
+; CHECK-O0-NEXT:    sub sp, sp, #16
+; CHECK-O0-NEXT:    add x8, sp, #14
+; CHECK-O0-NEXT:    str p0, [x8]
+; CHECK-O0-NEXT:    ldr p0, [x8]
+; CHECK-O0-NEXT:    add sp, sp, #16
+; CHECK-O0-NEXT:    ret
+;
+; CHECK-O3-LABEL: test_alloca_store_reload:
+; CHECK-O3:       // %bb.0:
+; CHECK-O3-NEXT:    sub sp, sp, #16
+; CHECK-O3-NEXT:    add x8, sp, #14
+; CHECK-O3-NEXT:    str p0, [x8]
+; CHECK-O3-NEXT:    add sp, sp, #16
+; CHECK-O3-NEXT:    ret
+  %ptr = alloca target("aarch64.svcount"), align 1
+  store target("aarch64.svcount") %val, ptr %ptr
+  %res = load target("aarch64.svcount"), ptr %ptr
+  ret target("aarch64.svcount") %res
+}
+
+;
+; Test passing as arguments (from perspective of callee)
+;
+
+define target("aarch64.svcount") @test_return_arg1(target("aarch64.svcount") %arg0, target("aarch64.svcount") %arg1) nounwind {
+; CHECK-LABEL: test_return_arg1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov p0.b, p1.b
+; CHECK-NEXT:    ret
+  ret target("aarch64.svcount") %arg1
+}
+
+define target("aarch64.svcount") @test_return_arg4(target("aarch64.svcount") %arg0, target("aarch64.svcount") %arg1, target("aarch64.svcount") %arg2, target("aarch64.svcount") %arg3, target("aarch64.svcount") %arg4) nounwind {
+; CHECK-LABEL: test_return_arg4:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr p0, [x0]
+; CHECK-NEXT:    ret
+  ret target("aarch64.svcount") %arg4
+}
+
+;
+; Test passing as arguments (from perspective of caller)
+;
+
+declare void @take_svcount_1(target("aarch64.svcount") %arg)
+define void @test_pass_1arg(target("aarch64.svcount") %arg) nounwind {
+; CHECK-LABEL: test_pass_1arg:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT:    bl take_svcount_1
+; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    ret
+  call void @take_svcount_1(target("aarch64.svcount") %arg)
+  ret void
+}
+
+declare void @take_svcount_5(target("aarch64.svcount") %arg0, target("aarch64.svcount") %arg1, target("aarch64.svcount") %arg2, target("aarch64.svcount") %arg3, target("aarch64.svcount") %arg4)
+define void @test_pass_5args(target("aarch64.svcount") %arg) nounwind {
+; CHECKO0-LABEL: test_pass_5args:
+; CHECKO0:       // %bb.0:
+; CHECKO0-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECKO0-NEXT:    addvl sp, sp, #-1
+; CHECKO0-NEXT:    mov p3.b, p0.b
+; CHECKO0-NEXT:    str p3, [sp, #7, mul vl]
+; CHECKO0-NEXT:    addpl x0, sp, #7
+; CHECKO0-NEXT:    mov p0.b, p3.b
+; CHECKO0-NEXT:    mov p1.b, p3.b
+; CHECKO0-NEXT:    mov p2.b, p3.b
+; CHECKO0-NEXT:    bl take_svcount_5
+; CHECKO0-NEXT:    addvl sp, sp, #1
+; CHECKO0-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECKO0-NEXT:    ret
+;
+; CHECKO3-LABEL: test_pass_5args:
+; CHECKO3:       // %bb.0:
+; CHECKO3-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECKO3-NEXT:    addvl sp, sp, #-1
+; CHECKO3-NEXT:    addpl x0, sp, #7
+; CHECKO3-NEXT:    mov p1.b, p0.b
+; CHECKO3-NEXT:    mov p2.b, p0.b
+; CHECKO3-NEXT:    mov p3.b, p0.b
+; CHECKO3-NEXT:    str p0, [sp, #7, mul vl]
+; CHECKO3-NEXT:    bl take_svcount_5
+; CHECKO3-NEXT:    addvl sp, sp, #1
+; CHECKO3-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECKO3-NEXT:    ret
+; CHECK-O0-LABEL: test_pass_5args:
+; CHECK-O0:       // %bb.0:
+; CHECK-O0-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-O0-NEXT:    addvl sp, sp, #-1
+; CHECK-O0-NEXT:    mov p3.b, p0.b
+; CHECK-O0-NEXT:    str p3, [sp, #7, mul vl]
+; CHECK-O0-NEXT:    addpl x0, sp, #7
+; CHECK-O0-NEXT:    mov p0.b, p3.b
+; CHECK-O0-NEXT:    mov p1.b, p3.b
+; CHECK-O0-NEXT:    mov p2.b, p3.b
+; CHECK-O0-NEXT:    bl take_svcount_5
+; CHECK-O0-NEXT:    addvl sp, sp, #1
+; CHECK-O0-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-O0-NEXT:    ret
+;
+; CHECK-O3-LABEL: test_pass_5args:
+; CHECK-O3:       // %bb.0:
+; CHECK-O3-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-O3-NEXT:    addvl sp, sp, #-1
+; CHECK-O3-NEXT:    addpl x0, sp, #7
+; CHECK-O3-NEXT:    mov p1.b, p0.b
+; CHECK-O3-NEXT:    mov p2.b, p0.b
+; CHECK-O3-NEXT:    mov p3.b, p0.b
+; CHECK-O3-NEXT:    str p0, [sp, #7, mul vl]
+; CHECK-O3-NEXT:    bl take_svcount_5
+; CHECK-O3-NEXT:    addvl sp, sp, #1
+; CHECK-O3-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-O3-NEXT:    ret
+  call void @take_svcount_5(target("aarch64.svcount") %arg, target("aarch64.svcount") %arg, target("aarch64.svcount") %arg, target("aarch64.svcount") %arg, target("aarch64.svcount") %arg)
+  ret void
+}

llvm/utils/TableGen/CodeGenTarget.cpp

StringRef llvm::getEnumName(MVT::SimpleValueType T) {
   case MVT::bf16:     return "MVT::bf16";
   case MVT::f32:      return "MVT::f32";
   case MVT::f64:      return "MVT::f64";
   case MVT::f80:      return "MVT::f80";
   case MVT::f128:     return "MVT::f128";
   case MVT::ppcf128:  return "MVT::ppcf128";
   case MVT::x86mmx:   return "MVT::x86mmx";
   case MVT::x86amx:   return "MVT::x86amx";
+  case MVT::aarch64svcount:   return "MVT::aarch64svcount";
   case MVT::i64x8:    return "MVT::i64x8";
   case MVT::Glue:     return "MVT::Glue";
   case MVT::isVoid:   return "MVT::isVoid";
   case MVT::v1i1:     return "MVT::v1i1";
   case MVT::v2i1:     return "MVT::v2i1";
   case MVT::v4i1:     return "MVT::v4i1";
   case MVT::v8i1:     return "MVT::v8i1";
   case MVT::v16i1:    return "MVT::v16i1";