[AArch64][SME2] Add CodeGen support for target("aarch64.svcount").

[sdesmalen]

Summary

This patch adds AArch64 CodeGen support such that the type can be passed
and returned to/from functions, and also adds support to use this type in
load/store operations and PHI nodes.

[sdesmalen]

Rebased patch to use the target("aarch64.svcount") type.

[paulwalker-arm]

My class PPRClass related comment is possibly problematic. My preference is to have a clean split between the two predicate types but I realise this is the start of the journey so if making the change results is too much churn then I'm happy to defer it to a refactoring exercise once the initial support has settled. That said, I'd still prefer to see how custom lowering works out rather than duplicating isel patterns.

I'm just going to throw this out there without giving it much thought but what about enabling the ability to ISD::BITCAST between svcount and nxv16i1? I think this better reflects what's going on compared to using reinterpret_cast. It also means we might get "free" operation legalisation by using setOperationPromotedToType.

llvm/include/llvm/Support/MachineValueType.h

391–396

This is more extensible but would return SimpleTy == MVT::aarch64svcount; be so bad?

llvm/lib/CodeGen/CodeGenPrepare.cpp

7699

Do you think it's worth Type having an isScalable() function? if this will be a common idiom.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

17164–17165

What is it about this combine that's problematic? Can you add a code comment for the rational.

Is it specially related to scalable target types or just target types in general?

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

6154–6156

Is this type specific or is it worth adding VT.isScalable()?

8920–8932

Can this be simplified to:

REINTERPRET_CAST(SELECT(CCVal, REINTERPRET_CAST(TVAL), REINTERPRET_CAST(FVAL)))

Then we'll renter this function and reuse the existing predicate lowering code. Perhaps you tried this and it didn't work out?

llvm/lib/Target/AArch64/AArch64RegisterInfo.td

876

Would it be overkill for aarch64svcount to have its own register class? I'm thinking it'll be nice to have isel protect us from incorrectly mixing predicate types and instructions. I'm also not sure if things like SDTCisSameNumEltsAs will cause is trouble, although I guess you've not hit anything so far.

The sequence below is using the non-count naming. Does this matter?

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

2706

Can the loads and stores (well anything that isn't a native svcount instruction) be done during lower?

[sdesmalen]

• Added EVT::isScalableVT() and refactored code to use it
• Use BITCAST instead of REINTERPRET_CAST
• Custom lower load/store of svcount to use nxv16i1
• Simplified lowering code in LowerSELECT
• Replaced the blanket bailout in visitLOAD/visitSTORE in DAGCombiner with more specific bailouts

[sdesmalen]

Thanks for the review @paulwalker-arm, I think I've addressed most of your comments, with the exception of giving predicate-as-counter its own register class, as that one probably makes more sense to pull out into a separate patch.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

17164–17165

This bailout was a bit too rigorous and I've removed it in favour of more specific bailouts.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8920–8932

Thanks for the suggestion, using SELECT worked fine.

llvm/lib/Target/AArch64/AArch64RegisterInfo.td

876

I tried to give aarch64svcount it's own register class, but this requires changes (to add a new register class, etc) that complicate this patch quite a bit. I can follow this up in a separate patch.

The sequence below is using the non-count naming. Does this matter?

No that's fine, because it uses the same P registers.

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

380–381

I've tried and now there is isel for ISD::BITCAST this can be

setOperationPromotedToType(ISD::LOAD, MVT::aarch64svcount, MVT::nxv16i1);
setOperationPromotedToType(ISD::STORE, MVT::aarch64svcount, MVT::nxv16i1);

382–383

Do code generation tests for these exist?

llvm/lib/Target/AArch64/AArch64RegisterInfo.td

876

Thanks for the investigation. Sure, having this as a follow on patch works for me.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

2447

Does PNR exist? or perhaps it's a typo? When experimenting with setOperationPromotedToType I hit build errors and had to change these entries.

[sdesmalen]

• Use PromoteToType for svcount_t -> nxv16i1
• PNR -> PPR in one of the patterns
• Added tests for select

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

380–381

I'm surprised this worked, as the term 'Promote' makes me think it only works on promoted (element) types, not a bitcast. I've removed the custom lowering code now.

382–383

Thanks for the reminder, I previously added them as part of D136863. I've added them to this patch now.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

2447

Something probably went wrong with splitting up the patches after I tried adding the predicate-as-counter register class, fixed now.

[paulwalker-arm]

A couple of minor issues but otherwise looks good.

llvm/include/llvm/CodeGen/ValueTypes.h

125

Can you use isZero() here?

llvm/test/CodeGen/AArch64/sme-aarch64-svcount.ll

169–173

Commented out test.

llvm/include/llvm/Analysis/ScalarEvolution.h

public:
   const SCEV *getOne(Type *Ty) { return getConstant(Ty, 1); }
 
   /// Return a SCEV for the constant -1 of a specific type.
   const SCEV *getMinusOne(Type *Ty) {
     return getConstant(Ty, -1, /*isSigned=*/true);
   }
 
   /// Return an expression for sizeof ScalableTy that is type IntTy, where
-  /// ScalableTy is a scalable vector type.
-  const SCEV *getSizeOfScalableVectorExpr(Type *IntTy,
-                                          ScalableVectorType *ScalableTy);
+  /// ScalableTy is a scalable vector type or an AArch64 predicate-as-counter
+  /// (opaque) type.
+  const SCEV *getSizeOfScalableTypeExpr(Type *IntTy, Type *ScalableTy);
 
   /// Return an expression for the alloc size of AllocTy that is type IntTy
   const SCEV *getSizeOfExpr(Type *IntTy, Type *AllocTy);
 
   /// Return an expression for the store size of StoreTy that is type IntTy
   const SCEV *getStoreSizeOfExpr(Type *IntTy, Type *StoreTy);
 
   /// Return an expression for offsetof on the given field with type IntTy

llvm/include/llvm/CodeGen/ValueTypes.h

EVT changeTypeToInteger() {
       if (isSimple())
         return getSimpleVT().changeTypeToInteger();
       return changeExtendedTypeToInteger();
     }
 
     /// Test if the given EVT has zero size, this will fail if called on a
     /// scalable type
     bool isZeroSized() const {
-      return !isScalableVector() && getSizeInBits() == 0;
+      return getSizeInBits().getKnownMinValue() == 0;
     }
 
     /// Test if the given EVT is simple (as opposed to being extended).
     bool isSimple() const {
       return V.SimpleTy != MVT::INVALID_SIMPLE_VALUE_TYPE;
     }
 
     /// Test if the given EVT is extended (as opposed to being simple).

llvm/include/llvm/IR/Type.h

public:
 
   /// Return true if it makes sense to take the size of this type. To get the
   /// actual size for a particular target, it is reasonable to use the
   /// DataLayout subsystem to do this.
   bool isSized(SmallPtrSetImpl<Type*> *Visited = nullptr) const {
     // If it's a primitive, it is always sized.
     if (getTypeID() == IntegerTyID || isFloatingPointTy() ||
         getTypeID() == PointerTyID || getTypeID() == X86_MMXTyID ||
-        getTypeID() == X86_AMXTyID)
+        getTypeID() == X86_AMXTyID || getTypeID() == AArch64SvcountTyID)
       return true;
     // If it is not something that can have a size (e.g. a function or label),
     // it doesn't have a size.
     if (getTypeID() != StructTyID && getTypeID() != ArrayTyID && !isVectorTy())
       return false;
     // Otherwise we have to try harder to decide.
     return isSizedDerivedType(Visited);
   }

llvm/include/llvm/Support/MachineValueType.h

enum SimpleValueType : uint8_t {
       Untyped        = 182,    // This value takes a register, but has
                                // unspecified type.  The register class
                                // will be determined by the opcode.
 
       funcref        = 183,    // WebAssembly's funcref type
       externref      = 184,    // WebAssembly's externref type
       x86amx         = 185,    // This is an X86 AMX value
       i64x8          = 186,    // 8 Consecutive GPRs (AArch64)
+      aarch64svcount = 187,    // AArch64 predicate-as-counter
 
       FIRST_VALUETYPE =  1,    // This is always the beginning of the list.
-      LAST_VALUETYPE = i64x8,  // This always remains at the end of the list.
+      LAST_VALUETYPE = aarch64svcount, // This always remains at the end of the list.
       VALUETYPE_SIZE = LAST_VALUETYPE + 1,
 
       // This is the current maximum for LAST_VALUETYPE.
       // MVT::MAX_ALLOWED_VALUETYPE is used for asserts and to size bit vectors
       // This value must be a multiple of 32.
       MAX_ALLOWED_VALUETYPE = 192,
 
       // A value of type llvm::TokenTy
public:
     /// runtime length is machine dependent
     bool isScalableVector() const {
       return (SimpleTy >= MVT::FIRST_SCALABLE_VECTOR_VALUETYPE &&
               SimpleTy <= MVT::LAST_SCALABLE_VECTOR_VALUETYPE);
     }
 
     bool isFixedLengthVector() const {
       return (SimpleTy >= MVT::FIRST_FIXEDLEN_VECTOR_VALUETYPE &&
               SimpleTy <= MVT::LAST_FIXEDLEN_VECTOR_VALUETYPE);
     }
 
     /// Return true if this is a 16-bit vector type.
     bool is16BitVector() const {
       return (SimpleTy == MVT::v2i8  || SimpleTy == MVT::v1i16 ||
               SimpleTy == MVT::v16i1 || SimpleTy == MVT::v1f16);
     }
 
     /// Return true if this is a 32-bit vector type.
     bool is32BitVector() const {
       return (SimpleTy == MVT::v32i1 || SimpleTy == MVT::v4i8   ||
               SimpleTy == MVT::v2i16 || SimpleTy == MVT::v1i32  ||
               SimpleTy == MVT::v2f16 || SimpleTy == MVT::v2bf16 ||
TypeSize getSizeInBits() const {
       case nxv8i1: return TypeSize::Scalable(8);
       case i16 :
       case f16:
       case bf16:
       case v16i1:
       case v2i8:
       case v1i16:
       case v1f16: return TypeSize::Fixed(16);
+      case aarch64svcount:
       case nxv16i1:
       case nxv2i8:
       case nxv1i16:
       case nxv1bf16:
       case nxv1f16: return TypeSize::Scalable(16);
       case f32 :
       case i32 :
       case v32i1:

llvm/lib/Analysis/Loads.cpp

 }
 
 bool llvm::isDereferenceableAndAlignedPointer(
     const Value *V, Type *Ty, Align Alignment, const DataLayout &DL,
     const Instruction *CtxI, AssumptionCache *AC, const DominatorTree *DT,
     const TargetLibraryInfo *TLI) {
   // For unsized types or scalable vectors we don't know exactly how many bytes
   // are dereferenced, so bail out.
-  if (!Ty->isSized() || isa<ScalableVectorType>(Ty))
+  if (!Ty->isSized() || isa<ScalableVectorType>(Ty) || Ty->isAArch64SvcountTy())
     return false;
 
   // When dereferenceability information is provided by a dereferenceable
   // attribute, we know exactly how many bytes are dereferenceable. If we can
   // determine the exact offset to the attributed variable, we can use that
   // information here.
 
   APInt AccessSize(DL.getPointerTypeSizeInBits(V->getType()),

llvm/lib/Analysis/ScalarEvolution.cpp

 
 const SCEV *ScalarEvolution::getUMinExpr(SmallVectorImpl<const SCEV *> &Ops,
                                          bool Sequential) {
   return Sequential ? getSequentialMinMaxExpr(scSequentialUMinExpr, Ops)
                     : getMinMaxExpr(scUMinExpr, Ops);
 }
 
 const SCEV *
-ScalarEvolution::getSizeOfScalableVectorExpr(Type *IntTy,
-                                             ScalableVectorType *ScalableTy) {
+ScalarEvolution::getSizeOfScalableTypeExpr(Type *IntTy, Type *ScalableTy) {
+  assert((isa<ScalableVectorType>(ScalableTy) ||
+          ScalableTy->isAArch64SvcountTy()) &&
+         "Expected a scalable type");
   Constant *NullPtr = Constant::getNullValue(ScalableTy->getPointerTo());
   Constant *One = ConstantInt::get(IntTy, 1);
   Constant *GEP = ConstantExpr::getGetElementPtr(ScalableTy, NullPtr, One);
   // Note that the expression we created is the final expression, we don't
   // want to simplify it any further Also, if we call a normal getSCEV(),
   // we'll end up in an endless recursion. So just create an SCEVUnknown.
   return getUnknown(ConstantExpr::getPtrToInt(GEP, IntTy));
 }
 
 const SCEV *ScalarEvolution::getSizeOfExpr(Type *IntTy, Type *AllocTy) {
-  if (auto *ScalableAllocTy = dyn_cast<ScalableVectorType>(AllocTy))
-    return getSizeOfScalableVectorExpr(IntTy, ScalableAllocTy);
+  if (isa<ScalableVectorType>(AllocTy) || AllocTy->isAArch64SvcountTy())
+    return getSizeOfScalableTypeExpr(IntTy, AllocTy);
+
   // We can bypass creating a target-independent constant expression and then
   // folding it back into a ConstantInt. This is just a compile-time
   // optimization.
   return getConstant(IntTy, getDataLayout().getTypeAllocSize(AllocTy));
 }
 
 const SCEV *ScalarEvolution::getStoreSizeOfExpr(Type *IntTy, Type *StoreTy) {
-  if (auto *ScalableStoreTy = dyn_cast<ScalableVectorType>(StoreTy))
-    return getSizeOfScalableVectorExpr(IntTy, ScalableStoreTy);
+  if (isa<ScalableVectorType>(StoreTy) || StoreTy->isAArch64SvcountTy())
+    return getSizeOfScalableTypeExpr(IntTy, StoreTy);
+
   // We can bypass creating a target-independent constant expression and then
   // folding it back into a ConstantInt. This is just a compile-time
   // optimization.
   return getConstant(IntTy, getDataLayout().getTypeStoreSize(StoreTy));
 }
 
 const SCEV *ScalarEvolution::getOffsetOfExpr(Type *IntTy,
                                              StructType *STy,

llvm/lib/CodeGen/CodeGenPrepare.cpp

static bool splitMergedValStore(StoreInst &SI, const DataLayout &DL,
                                 const TargetLowering &TLI) {
   // Handle simple but common cases only.
   Type *StoreType = SI.getValueOperand()->getType();
 
   // The code below assumes shifting a value by <number of bits>,
   // whereas scalable vectors would have to be shifted by
   // <2log(vscale) + number of bits> in order to store the
   // low/high parts. Bailing out for now.
-  if (isa<ScalableVectorType>(StoreType))
+  if (isa<ScalableVectorType>(StoreType) || StoreType->isAArch64SvcountTy())
     return false;
 
   if (!DL.typeSizeEqualsStoreSize(StoreType) ||
       DL.getTypeSizeInBits(StoreType) == 0)
     return false;
 
   unsigned HalfValBitSize = DL.getTypeSizeInBits(StoreType) / 2;
   Type *SplitStoreType = Type::getIntNTy(SI.getContext(), HalfValBitSize);

llvm/lib/CodeGen/LowLevelType.cpp

if (auto VTy = dyn_cast<VectorType>(&Ty)) {
     return LLT::vector(EC, ScalarTy);
   }
 
   if (auto PTy = dyn_cast<PointerType>(&Ty)) {
     unsigned AddrSpace = PTy->getAddressSpace();
     return LLT::pointer(AddrSpace, DL.getPointerSizeInBits(AddrSpace));
   }
 
-  if (Ty.isSized()) {
+  if (Ty.isSized() && !Ty.isAArch64SvcountTy()) {
     // Aggregates are no different from real scalars as far as GlobalISel is
     // concerned.
     auto SizeInBits = DL.getTypeSizeInBits(&Ty);
     assert(SizeInBits != 0 && "invalid zero-sized type");
     return LLT::scalar(SizeInBits);
   }
 
   return LLT();

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::ForwardStoreValueToDirectLoad(LoadSDNode *LD) {
   return SDValue();
 }
 
 SDValue DAGCombiner::visitLOAD(SDNode *N) {
   LoadSDNode *LD  = cast<LoadSDNode>(N);
   SDValue Chain = LD->getChain();
   SDValue Ptr   = LD->getBasePtr();
 
+  if (N->getValueType(0) == MVT::aarch64svcount)
+    return SDValue();
+
   // If load is not volatile and there are no uses of the loaded value (and
   // the updated indexed value in case of indexed loads), change uses of the
   // chain value into uses of the chain input (i.e. delete the dead load).
   // TODO: Allow this for unordered atomics (see D66309)
   if (LD->isSimple()) {
     if (N->getValueType(1) == MVT::Other) {
       // Unindexed loads.
       if (!N->hasAnyUseOfValue(0)) {

 }
 
 SDValue DAGCombiner::visitSTORE(SDNode *N) {
   StoreSDNode *ST  = cast<StoreSDNode>(N);
   SDValue Chain = ST->getChain();
   SDValue Value = ST->getValue();
   SDValue Ptr   = ST->getBasePtr();
 
+  if (Value.getValueType() == MVT::aarch64svcount)
+    return SDValue();
+
   // If this is a store of a bit convert, store the input value if the
   // resultant store does not need a higher alignment than the original.
   if (Value.getOpcode() == ISD::BITCAST && !ST->isTruncatingStore() &&
       ST->isUnindexed()) {
     EVT SVT = Value.getOperand(0).getValueType();
     // If the store is volatile, we only want to change the store type if the
     // resulting store is legal. Otherwise we might increase the number of
     // memory accesses. We don't care if the original type was legal or not

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

if (TLI.splitValueIntoRegisterParts(DAG, DL, Val, Parts, NumParts, PartVT,
     return;
   EVT ValueVT = Val.getValueType();
 
   // Handle the vector case separately.
   if (ValueVT.isVector())
     return getCopyToPartsVector(DAG, DL, Val, Parts, NumParts, PartVT, V,
                                 CallConv);
 
-  unsigned PartBits = PartVT.getSizeInBits();
   unsigned OrigNumParts = NumParts;
   assert(DAG.getTargetLoweringInfo().isTypeLegal(PartVT) &&
          "Copying to an illegal type!");
 
   if (NumParts == 0)
     return;
 
   assert(!ValueVT.isVector() && "Vector case handled elsewhere");
   EVT PartEVT = PartVT;
   if (PartEVT == ValueVT) {
     assert(NumParts == 1 && "No-op copy with multiple parts!");
     Parts[0] = Val;
     return;
   }
 
+  unsigned PartBits = PartVT.getSizeInBits();
   if (NumParts * PartBits > ValueVT.getSizeInBits()) {
     // If the parts cover more bits than the value has, promote the value.
     if (PartVT.isFloatingPoint() && ValueVT.isFloatingPoint()) {
       assert(NumParts == 1 && "Do not know what to promote to!");
       Val = DAG.getNode(ISD::FP_EXTEND, DL, PartVT, Val);
     } else {
       if (ValueVT.isFloatingPoint()) {
         // FP values need to be bitcast, then extended if they are being put

llvm/lib/CodeGen/ValueTypes.cpp

std::string EVT::getEVTString() const {
   case MVT::Glue:      return "glue";
   case MVT::x86mmx:    return "x86mmx";
   case MVT::x86amx:    return "x86amx";
   case MVT::i64x8:     return "i64x8";
   case MVT::Metadata:  return "Metadata";
   case MVT::Untyped:   return "Untyped";
   case MVT::funcref:   return "funcref";
   case MVT::externref: return "externref";
+  case MVT::aarch64svcount:
+    return "aarch64svcount";
   }
 }
 
 /// getTypeForEVT - This method returns an LLVM type corresponding to the
 /// specified EVT.  For integer types, this returns an unsigned type.  Note
 /// that this will abort for types that cannot be represented.
 Type *EVT::getTypeForEVT(LLVMContext &Context) const {
   // clang-format off
Type *EVT::getTypeForEVT(LLVMContext &Context) const {
   case MVT::f16:     return Type::getHalfTy(Context);
   case MVT::bf16:    return Type::getBFloatTy(Context);
   case MVT::f32:     return Type::getFloatTy(Context);
   case MVT::f64:     return Type::getDoubleTy(Context);
   case MVT::f80:     return Type::getX86_FP80Ty(Context);
   case MVT::f128:    return Type::getFP128Ty(Context);
   case MVT::ppcf128: return Type::getPPC_FP128Ty(Context);
   case MVT::x86mmx:  return Type::getX86_MMXTy(Context);
+  case MVT::aarch64svcount:
+    return Type::getAArch64SvcountTy(Context);
   case MVT::x86amx:  return Type::getX86_AMXTy(Context);
   case MVT::i64x8:   return IntegerType::get(Context, 512);
   case MVT::externref:
     // pointer to opaque struct in addrspace(10)
     return PointerType::get(StructType::create(Context), 10);
   case MVT::funcref:
     // pointer to i8 addrspace(20)
     return PointerType::get(Type::getInt8Ty(Context), 20);
MVT MVT::getVT(Type *Ty, bool HandleUnknown){
   case Type::IntegerTyID:
     return getIntegerVT(cast<IntegerType>(Ty)->getBitWidth());
   case Type::HalfTyID:      return MVT(MVT::f16);
   case Type::BFloatTyID:    return MVT(MVT::bf16);
   case Type::FloatTyID:     return MVT(MVT::f32);
   case Type::DoubleTyID:    return MVT(MVT::f64);
   case Type::X86_FP80TyID:  return MVT(MVT::f80);
   case Type::X86_MMXTyID:   return MVT(MVT::x86mmx);
+  case Type::AArch64SvcountTyID:
+    return MVT(MVT::aarch64svcount);
   case Type::X86_AMXTyID:   return MVT(MVT::x86amx);
   case Type::FP128TyID:     return MVT(MVT::f128);
   case Type::PPC_FP128TyID: return MVT(MVT::ppcf128);
   case Type::PointerTyID:   return MVT(MVT::iPTR);
   case Type::FixedVectorTyID:
   case Type::ScalableVectorTyID: {
     VectorType *VTy = cast<VectorType>(Ty);
     return getVectorVT(

llvm/lib/IR/DataLayout.cpp

case Type::X86_FP80TyID: {
     // If we still couldn't find a reasonable default alignment, fall back
     // to a simple heuristic that the alignment is the first power of two
     // greater-or-equal to the store size of the type.  This is a reasonable
     // approximation of reality, and if the user wanted something less
     // less conservative, they should have specified it explicitly in the data
     // layout.
     return Align(PowerOf2Ceil(BitWidth / 8));
   }
+  case Type::AArch64SvcountTyID:
   case Type::X86_MMXTyID:
   case Type::FixedVectorTyID:
   case Type::ScalableVectorTyID: {
     unsigned BitWidth = getTypeSizeInBits(Ty).getKnownMinSize();
     auto I = findAlignmentLowerBound(VECTOR_ALIGN, BitWidth);
     if (I != Alignments.end() && I->AlignType == VECTOR_ALIGN &&
         I->TypeBitWidth == BitWidth)
       return abi_or_pref ? I->ABIAlign : I->PrefAlign;

llvm/lib/Support/LowLevelType.cpp

 using namespace llvm;
 
 LLT::LLT(MVT VT) {
   if (VT.isVector()) {
     bool asVector = VT.getVectorMinNumElements() > 1;
     init(/*IsPointer=*/false, asVector, /*IsScalar=*/!asVector,
          VT.getVectorElementCount(), VT.getVectorElementType().getSizeInBits(),
          /*AddressSpace=*/0);
-  } else if (VT.isValid()) {
+  } else if (VT.isValid() && VT != MVT::aarch64svcount) {
     // Aggregates are no different from real scalars as far as GlobalISel is
     // concerned.
     init(/*IsPointer=*/false, /*IsVector=*/false, /*IsScalar=*/true,
          ElementCount::getFixed(0), VT.getSizeInBits(), /*AddressSpace=*/0);
   } else {
     IsScalar = false;
     IsPointer = false;
     IsVector = false;

llvm/lib/Target/AArch64/AArch64CallingConvention.td

def CC_AArch64_AAPCS : CallingConv<[
 
   CCIfType<[nxv16i8, nxv8i16, nxv4i32, nxv2i64, nxv2f16, nxv4f16, nxv8f16,
             nxv2bf16, nxv4bf16, nxv8bf16, nxv2f32, nxv4f32, nxv2f64],
            CCAssignToReg<[Z0, Z1, Z2, Z3, Z4, Z5, Z6, Z7]>>,
   CCIfType<[nxv16i8, nxv8i16, nxv4i32, nxv2i64, nxv2f16, nxv4f16, nxv8f16,
             nxv2bf16, nxv4bf16, nxv8bf16, nxv2f32, nxv4f32, nxv2f64],
            CCPassIndirect<i64>>,
 
-  CCIfType<[nxv1i1, nxv2i1, nxv4i1, nxv8i1, nxv16i1],
+  CCIfType<[nxv1i1, nxv2i1, nxv4i1, nxv8i1, nxv16i1, aarch64svcount],
            CCAssignToReg<[P0, P1, P2, P3]>>,
-  CCIfType<[nxv1i1, nxv2i1, nxv4i1, nxv8i1, nxv16i1],
+  CCIfType<[nxv1i1, nxv2i1, nxv4i1, nxv8i1, nxv16i1, aarch64svcount],
            CCPassIndirect<i64>>,
 
   // Handle i1, i8, i16, i32, i64, f32, f64 and v2f64 by passing in registers,
   // up to eight each of GPR and FPR.
   CCIfType<[i1, i8, i16], CCPromoteToType<i32>>,
   CCIfType<[i32], CCAssignToReg<[W0, W1, W2, W3, W4, W5, W6, W7]>>,
   // i128 is split to two i64s, we can't fit half to register X7.
   CCIfType<[i64], CCIfSplit<CCAssignToRegWithShadow<[X0, X2, X4, X6],
CCIfType<[v1i64, v2i32, v4i16, v8i8, v1f64, v2f32, v4f16, v4bf16],
       CCAssignToReg<[D0, D1, D2, D3, D4, D5, D6, D7]>>,
   CCIfType<[f128, v2i64, v4i32, v8i16, v16i8, v4f32, v2f64, v8f16, v8bf16],
       CCAssignToReg<[Q0, Q1, Q2, Q3, Q4, Q5, Q6, Q7]>>,
 
   CCIfType<[nxv16i8, nxv8i16, nxv4i32, nxv2i64, nxv2f16, nxv4f16, nxv8f16,
             nxv2bf16, nxv4bf16, nxv8bf16, nxv2f32, nxv4f32, nxv2f64],
            CCAssignToReg<[Z0, Z1, Z2, Z3, Z4, Z5, Z6, Z7]>>,
 
-  CCIfType<[nxv1i1, nxv2i1, nxv4i1, nxv8i1, nxv16i1],
+  CCIfType<[nxv1i1, nxv2i1, nxv4i1, nxv8i1, nxv16i1, aarch64svcount],
            CCAssignToReg<[P0, P1, P2, P3]>>
 ]>;
 
 // Vararg functions on windows pass floats in integer registers
 let Entry = 1 in
 def CC_AArch64_Win64_VarArg : CallingConv<[
   CCIfType<[f16, bf16], CCBitConvertToType<i16>>,
   CCIfType<[f32], CCBitConvertToType<i32>>,

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (Subtarget->useSVEForFixedLengthVectors()) {
           addRegisterClass(VT, &AArch64::ZPRRegClass);
 
       for (MVT VT : MVT::fp_fixedlen_vector_valuetypes())
         if (useSVEForFixedLengthVectorVT(VT))
           addRegisterClass(VT, &AArch64::ZPRRegClass);
     }
   }
 
+  if (Subtarget->hasSVE2p1() || Subtarget->hasSME2()) {
+    addRegisterClass(MVT::aarch64svcount, &AArch64::PPRRegClass);
+  }
+
   // Compute derived properties from the register classes
   computeRegisterProperties(Subtarget->getRegisterInfo());
 
   // Provide all sorts of operation actions
   setOperationAction(ISD::GlobalAddress, MVT::i64, Custom);
   setOperationAction(ISD::GlobalTLSAddress, MVT::i64, Custom);
   setOperationAction(ISD::SETCC, MVT::i32, Custom);
   setOperationAction(ISD::SETCC, MVT::i64, Custom);
   setOperationAction(ISD::SETCC, MVT::f16, Custom);
   setOperationAction(ISD::SETCC, MVT::f32, Custom);
if (VA.isRegLoc()) {
       else if (RegVT == MVT::f64 || RegVT.is64BitVector())
         RC = &AArch64::FPR64RegClass;
       else if (RegVT == MVT::f128 || RegVT.is128BitVector())
         RC = &AArch64::FPR128RegClass;
       else if (RegVT.isScalableVector() &&
                RegVT.getVectorElementType() == MVT::i1) {
         FuncInfo->setIsSVECC(true);
         RC = &AArch64::PPRRegClass;
+      } else if (RegVT == MVT::aarch64svcount) {
+        FuncInfo->setIsSVECC(true);
+        RC = &AArch64::PPRRegClass;
       } else if (RegVT.isScalableVector()) {
         FuncInfo->setIsSVECC(true);
         RC = &AArch64::ZPRRegClass;
       } else
         llvm_unreachable("RegVT not supported by FORMAL_ARGUMENTS Lowering");
 
       // Transform the arguments in physical registers into virtual ones.
       Register Reg = MF.addLiveIn(VA.getLocReg(), RC);
if (VA.isRegLoc()) {
       // to 64 bits.  Insert an assert[sz]ext to capture this, then
       // truncate to the right size.
       switch (VA.getLocInfo()) {
       default:
         llvm_unreachable("Unknown loc info!");
       case CCValAssign::Full:
         break;
       case CCValAssign::Indirect:
         assert((VA.getValVT().isScalableVector() ||
+                VA.getValVT() == MVT::aarch64svcount ||
                 Subtarget->isWindowsArm64EC()) &&
                "Indirect arguments should be scalable on most subtargets");
         break;
       case CCValAssign::BCvt:
         ArgValue = DAG.getNode(ISD::BITCAST, DL, VA.getValVT(), ArgValue);
         break;
       case CCValAssign::AExt:
       case CCValAssign::SExt:
       case CCValAssign::ZExt:
if (VA.isRegLoc()) {
         break;
       }
 
       ArgValue = DAG.getExtLoad(ExtType, DL, VA.getLocVT(), Chain, FIN, PtrInfo,
                                 MemVT);
     }
 
     if (VA.getLocInfo() == CCValAssign::Indirect) {
-      assert(
-          (VA.getValVT().isScalableVector() || Subtarget->isWindowsArm64EC()) &&
+      assert((VA.getValVT().isScalableVector() ||
+              VA.getValVT() == MVT::aarch64svcount ||
+              Subtarget->isWindowsArm64EC()) &&
              "Indirect arguments should be scalable on most subtargets");
 
       uint64_t PartSize = VA.getValVT().getStoreSize().getKnownMinSize();
       unsigned NumParts = 1;
       if (Ins[i].Flags.isInConsecutiveRegs()) {
         assert(!Ins[i].Flags.isInConsecutiveRegsLast());
         while (!Ins[i + NumParts - 1].Flags.isInConsecutiveRegsLast())
           ++NumParts;
       }
case CCValAssign::BCvt:
       break;
     case CCValAssign::Trunc:
       Arg = DAG.getZExtOrTrunc(Arg, DL, VA.getLocVT());
       break;
     case CCValAssign::FPExt:
       Arg = DAG.getNode(ISD::FP_EXTEND, DL, VA.getLocVT(), Arg);
       break;
     case CCValAssign::Indirect:
-      bool isScalable = VA.getValVT().isScalableVector();
+      bool isScalable = VA.getValVT().isScalableVector() ||
+                        VA.getValVT() == MVT::aarch64svcount;
       assert((isScalable || Subtarget->isWindowsArm64EC()) &&
              "Indirect arguments should be scalable on most subtargets");
 
       uint64_t StoreSize = VA.getValVT().getStoreSize().getKnownMinSize();
       uint64_t PartSize = StoreSize;
       unsigned NumParts = 1;
       if (Outs[i].Flags.isInConsecutiveRegs()) {
         assert(!Outs[i].Flags.isInConsecutiveRegsLast());
SDValue AArch64TargetLowering::LowerSELECT(SDValue Op,
                                            SelectionDAG &DAG) const {
   SDValue CCVal = Op->getOperand(0);
   SDValue TVal = Op->getOperand(1);
   SDValue FVal = Op->getOperand(2);
   SDLoc DL(Op);
 
   EVT Ty = Op.getValueType();
   if (Ty.isScalableVector()) {
     SDValue TruncCC = DAG.getNode(ISD::TRUNCATE, DL, MVT::i1, CCVal);
     MVT PredVT = MVT::getVectorVT(MVT::i1, Ty.getVectorElementCount());
     SDValue SplatPred = DAG.getNode(ISD::SPLAT_VECTOR, DL, PredVT, TruncCC);
     return DAG.getNode(ISD::VSELECT, DL, Ty, SplatPred, TVal, FVal);
   }
 
   if (useSVEForFixedLengthVectorVT(Ty)) {
     // FIXME: Ideally this would be the same as above using i1 types, however
     // for the moment we can't deal with fixed i1 vector types properly, so
     // instead extend the predicate to a result type sized integer vector.
     MVT SplatValVT = MVT::getIntegerVT(Ty.getScalarSizeInBits());
     MVT PredVT = MVT::getVectorVT(SplatValVT, Ty.getVectorElementCount());
     SDValue SplatVal = DAG.getSExtOrTrunc(CCVal, DL, SplatValVT);
     SDValue SplatPred = DAG.getNode(ISD::SPLAT_VECTOR, DL, PredVT, SplatVal);
     return DAG.getNode(ISD::VSELECT, DL, Ty, SplatPred, TVal, FVal);
   }
 
   // Optimize {s|u}{add|sub|mul}.with.overflow feeding into a select
   // instruction.
   if (ISD::isOverflowIntrOpRes(CCVal)) {
     // Only lower legal XALUO ops.
bool AArch64TargetLowering::isLegalAddressingMode(const DataLayout &DL,
   if (AM.BaseGV)
     return false;
 
   // No reg+reg+imm addressing.
   if (AM.HasBaseReg && AM.BaseOffs && AM.Scale)
     return false;
 
   // FIXME: Update this method to support scalable addressing modes.
+  if (Ty->isAArch64SvcountTy())
+    return AM.HasBaseReg && !AM.BaseOffs && !AM.Scale;
+
   if (isa<ScalableVectorType>(Ty)) {
     uint64_t VecElemNumBytes =
         DL.getTypeSizeInBits(cast<VectorType>(Ty)->getElementType()) / 8;
     return AM.HasBaseReg && !AM.BaseOffs &&
            (AM.Scale == 0 || (uint64_t)AM.Scale == VecElemNumBytes);
   }
 
   // check reg + imm case:
case AArch64::G_ADD_LOW:
     return true;
   default:
     break;
   }
   return TargetLoweringBase::shouldLocalize(MI, TTI);
 }
 
 bool AArch64TargetLowering::fallBackToDAGISel(const Instruction &Inst) const {
-  if (isa<ScalableVectorType>(Inst.getType()))
+  auto IsScalable = [](const Type *T) {
+    return isa<ScalableVectorType>(T) || T->isAArch64SvcountTy();
+  };
+
+  if (IsScalable(Inst.getType()))
     return true;
 
   for (unsigned i = 0; i < Inst.getNumOperands(); ++i)
-    if (isa<ScalableVectorType>(Inst.getOperand(i)->getType()))
+    if (IsScalable(Inst.getOperand(i)->getType()))
       return true;
 
   if (const AllocaInst *AI = dyn_cast<AllocaInst>(&Inst)) {
-    if (isa<ScalableVectorType>(AI->getAllocatedType()))
+    if (IsScalable(AI->getAllocatedType()))
       return true;
   }
 
   // Checks to allow the use of SME instructions
   if (auto *Base = dyn_cast<CallBase>(&Inst)) {
     auto CallerAttrs = SMEAttrs(*Inst.getFunction());
     auto CalleeAttrs = SMEAttrs(*Base);
     if (CallerAttrs.requiresSMChange(CalleeAttrs,

llvm/lib/Target/AArch64/AArch64RegisterInfo.td

 class ZPRRegOp <string Suffix, AsmOperandClass C, ElementSizeEnum Size,
                 RegisterClass RC> : SVERegOp<Suffix, C, Size, RC> {}
 
 //******************************************************************************
 
 // SVE predicate register classes.
 class PPRClass<int lastreg> : RegisterClass<
                                   "AArch64",
-                                  [ nxv16i1, nxv8i1, nxv4i1, nxv2i1, nxv1i1 ], 16,
+                                  [ nxv16i1, nxv8i1, nxv4i1, nxv2i1, nxv1i1, aarch64svcount ], 16,
                                   (sequence "P%u", 0, lastreg)> {
   let Size = 16;
 }
 
 def PPR    : PPRClass<15>;
 def PPR_3b : PPRClass<7>; // Restricted 3 bit SVE predicate register class.
 
 class PPRAsmOperand <string name, string RegClass, int Width>: AsmOperandClass {

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

let Predicates = [IsLE] in {
     def : Pat<(nxv16i8 (bitconvert (nxv8bf16 ZPR:$src))), (nxv16i8 ZPR:$src)>;
     def : Pat<(nxv8i16 (bitconvert (nxv8bf16 ZPR:$src))), (nxv8i16 ZPR:$src)>;
     def : Pat<(nxv4i32 (bitconvert (nxv8bf16 ZPR:$src))), (nxv4i32 ZPR:$src)>;
     def : Pat<(nxv2i64 (bitconvert (nxv8bf16 ZPR:$src))), (nxv2i64 ZPR:$src)>;
     def : Pat<(nxv8f16 (bitconvert (nxv8bf16 ZPR:$src))), (nxv8f16 ZPR:$src)>;
     def : Pat<(nxv4f32 (bitconvert (nxv8bf16 ZPR:$src))), (nxv4f32 ZPR:$src)>;
     def : Pat<(nxv2f64 (bitconvert (nxv8bf16 ZPR:$src))), (nxv2f64 ZPR:$src)>;
   }
 
   // These allow casting from/to unpacked predicate types.
   def : Pat<(nxv16i1 (reinterpret_cast (nxv16i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
   def : Pat<(nxv16i1 (reinterpret_cast (nxv8i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
   def : Pat<(nxv16i1 (reinterpret_cast (nxv4i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
   def : Pat<(nxv16i1 (reinterpret_cast (nxv2i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
   def : Pat<(nxv16i1 (reinterpret_cast (nxv1i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
   def : Pat<(nxv8i1 (reinterpret_cast (nxv16i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
   def : Pat<(nxv8i1 (reinterpret_cast  (nxv4i1 PPR:$src))), (COPY_TO_REGCLASS PPR:$src, PPR)>;
multiclass unpred_store_predicate<ValueType Ty, Instruction Store> {
     def _fi : Pat<(store (Ty PPR:$val), (am_sve_fi GPR64sp:$base, simm9:$offset)),
                   (Store PPR:$val, GPR64sp:$base, simm9:$offset)>;
 
     def _default : Pat<(store (Ty PPR:$Val), GPR64:$base),
                   (Store PPR:$Val, GPR64:$base, (i64 0))>;
   }
 
   defm Pat_Store_P16 : unpred_store_predicate<nxv16i1, STR_PXI>;
+  defm Pat_Store_PredAsCount : unpred_store_predicate<aarch64svcount, STR_PXI>;
 
   multiclass unpred_load_predicate<ValueType Ty, Instruction Load> {
     def _fi : Pat<(Ty (load (am_sve_fi GPR64sp:$base, simm9:$offset))),
                   (Load GPR64sp:$base, simm9:$offset)>;
 
     def _default : Pat<(Ty (load GPR64:$base)),
                   (Load GPR64:$base, (i64 0))>;
   }
 
   defm Pat_Load_P16 : unpred_load_predicate<nxv16i1, LDR_PXI>;
+  defm Pat_Load_PredAsCount : unpred_load_predicate<aarch64svcount, LDR_PXI>;
 
   multiclass ld1<Instruction RegRegInst, Instruction RegImmInst, ValueType Ty,
                  SDPatternOperator Load, ValueType PredTy, ValueType MemVT, ComplexPattern AddrCP> {
     // reg + reg
     let AddedComplexity = 1 in {
       def : Pat<(Ty (Load  (PredTy PPR:$gp), (AddrCP GPR64:$base, GPR64:$offset), MemVT)),
                 (RegRegInst PPR:$gp, GPR64sp:$base, GPR64:$offset)>;
     }

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

static void handleMustTailForwardedRegisters(MachineIRBuilder &MIRBuilder,
   // Add the forwards to the MachineBasicBlock and MachineFunction.
   for (const auto &F : Forwards) {
     MBB.addLiveIn(F.PReg);
     MIRBuilder.buildCopy(Register(F.VReg), Register(F.PReg));
   }
 }
 
 bool AArch64CallLowering::fallBackToDAGISel(const MachineFunction &MF) const {
+  auto IsScalable = [](const Type *T) {
+    return isa<ScalableVectorType>(T) || T->isAArch64SvcountTy();
+  };
+
   auto &F = MF.getFunction();
-  if (isa<ScalableVectorType>(F.getReturnType()))
-    return true;
-  if (llvm::any_of(F.args(), [](const Argument &A) {
-        return isa<ScalableVectorType>(A.getType());
+  if (IsScalable(F.getReturnType()) ||
+      llvm::any_of(F.args(), [&IsScalable](const Argument &A) {
+        return IsScalable(A.getType());
       }))
     return true;
   const auto &ST = MF.getSubtarget<AArch64Subtarget>();
   if (!ST.hasNEON() || !ST.hasFPARMv8()) {
     LLVM_DEBUG(dbgs() << "Falling back to SDAG because we don't support no-NEON\n");
     return true;
   }
 

llvm/lib/Transforms/Scalar/SROA.cpp

if (AI.use_empty()) {
     AI.eraseFromParent();
     return true;
   }
   const DataLayout &DL = AI.getModule()->getDataLayout();
 
   // Skip alloca forms that this analysis can't handle.
   auto *AT = AI.getAllocatedType();
   if (AI.isArrayAllocation() || !AT->isSized() || isa<ScalableVectorType>(AT) ||
-      DL.getTypeAllocSize(AT).getFixedSize() == 0)
+      AT->isAArch64SvcountTy() || DL.getTypeAllocSize(AT).getFixedSize() == 0)
     return false;
 
   bool Changed = false;
 
   // First, split any FCA loads and stores touching this alloca to promote
   // better splitting and promotion opportunities.
   IRBuilderTy IRB(&AI);
   AggLoadStoreRewriter AggRewriter(DL, IRB);

llvm/test/CodeGen/AArch64/sme-aarch64-svcount-O3.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -O3 -mtriple=aarch64 -mattr=+sme -S < %s | FileCheck %s
+
+; Test PHI nodes are allowed with
+define aarch64_svcount @test_alloca_store_reload(aarch64_svcount %val0, aarch64_svcount %val1, ptr %iptr, ptr %pptr, i64 %N) nounwind {
+; CHECK-LABEL: @test_alloca_store_reload(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    store i64 0, ptr [[IPTR:%.*]], align 4
+; CHECK-NEXT:    store aarch64_svcount [[VAL0:%.*]], ptr [[PPTR:%.*]], align 2
+; CHECK-NEXT:    [[I1_PEEL:%.*]] = icmp eq i64 [[N:%.*]], 0
+; CHECK-NEXT:    br i1 [[I1_PEEL]], label [[LOOP_EXIT:%.*]], label [[LOOP_BODY:%.*]]
+; CHECK:       loop.body:
+; CHECK-NEXT:    [[IND:%.*]] = phi i64 [ [[IND_NEXT:%.*]], [[LOOP_BODY]] ], [ 1, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    [[IPTR_GEP:%.*]] = getelementptr i64, ptr [[IPTR]], i64 [[IND]]
+; CHECK-NEXT:    store i64 [[IND]], ptr [[IPTR_GEP]], align 4
+; CHECK-NEXT:    store aarch64_svcount [[VAL1:%.*]], ptr [[PPTR]], align 2
+; CHECK-NEXT:    [[IND_NEXT]] = add i64 [[IND]], 1
+; CHECK-NEXT:    [[I1:%.*]] = icmp eq i64 [[IND]], [[N]]
+; CHECK-NEXT:    br i1 [[I1]], label [[LOOP_EXIT]], label [[LOOP_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK:       loop.exit:
+; CHECK-NEXT:    [[PHI_LCSSA:%.*]] = phi aarch64_svcount [ [[VAL0]], [[ENTRY]] ], [ [[VAL1]], [[LOOP_BODY]] ]
+; CHECK-NEXT:    ret aarch64_svcount [[PHI_LCSSA]]
+;
+entry:
+  br label %loop.body
+
+loop.body:
+  %ind = phi i64 [0, %entry], [%ind.next, %loop.body]
+  %phi = phi aarch64_svcount [%val0, %entry], [%val1, %loop.body]
+  %iptr.gep = getelementptr i64, ptr %iptr, i64 %ind
+  store i64 %ind, ptr %iptr.gep
+  store aarch64_svcount %phi, ptr %pptr
+  %ind.next = add i64 %ind, 1
+  %i1 = icmp eq i64 %ind, %N
+  br i1 %i1, label %loop.exit, label %loop.body
+
+loop.exit:
+  ret aarch64_svcount %phi
+}

llvm/test/CodeGen/AArch64/sme-aarch64-svcount.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -O0 -mtriple=aarch64 -mattr=+sve2p1 < %s | FileCheck %s --check-prefixes=CHECK,CHECKO0
+; RUN: llc -O3 -mtriple=aarch64 -mattr=+sve2p1 < %s | FileCheck %s --check-prefixes=CHECK,CHECKO3
+
+;
+; Test simple loads, stores and return.
+;
+define aarch64_svcount @test_load(ptr %ptr) nounwind {
+; CHECK-LABEL: test_load:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr p0, [x0]
+; CHECK-NEXT:    ret
+  %res = load aarch64_svcount, ptr %ptr
+  ret aarch64_svcount %res
+}
+
+define void @test_store(ptr %ptr, aarch64_svcount %val) nounwind {
+; CHECK-LABEL: test_store:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    str p0, [x0]
+; CHECK-NEXT:    ret
+  store aarch64_svcount %val, ptr %ptr
+  ret void
+}
+
+define aarch64_svcount @test_alloca_store_reload(aarch64_svcount %val) nounwind {
+; CHECKO0-LABEL: test_alloca_store_reload:
+; CHECKO0:       // %bb.0:
+; CHECKO0-NEXT:    sub sp, sp, #16
+; CHECKO0-NEXT:    add x8, sp, #14
+; CHECKO0-NEXT:    str p0, [x8]
+; CHECKO0-NEXT:    ldr p0, [x8]
+; CHECKO0-NEXT:    add sp, sp, #16
+; CHECKO0-NEXT:    ret
+;
+; CHECKO3-LABEL: test_alloca_store_reload:
+; CHECKO3:       // %bb.0:
+; CHECKO3-NEXT:    sub sp, sp, #16
+; CHECKO3-NEXT:    add x8, sp, #14
+; CHECKO3-NEXT:    str p0, [x8]
+; CHECKO3-NEXT:    add sp, sp, #16
+; CHECKO3-NEXT:    ret
+  %ptr = alloca aarch64_svcount, align 1
+  store aarch64_svcount %val, ptr %ptr
+  %res = load aarch64_svcount, ptr %ptr
+  ret aarch64_svcount %res
+}
+
+;
+; Test passing as arguments (from perspective of callee)
+;
+
+define aarch64_svcount @test_return_arg1(aarch64_svcount %arg0, aarch64_svcount %arg1) nounwind {
+; CHECK-LABEL: test_return_arg1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov p0.b, p1.b
+; CHECK-NEXT:    ret
+  ret aarch64_svcount %arg1
+}
+
+define aarch64_svcount @test_return_arg4(aarch64_svcount %arg0, aarch64_svcount %arg1, aarch64_svcount %arg2, aarch64_svcount %arg3, aarch64_svcount %arg4) nounwind {
+; CHECK-LABEL: test_return_arg4:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr p0, [x0]
+; CHECK-NEXT:    ret
+  ret aarch64_svcount %arg4
+}
+
+;
+; Test passing as arguments (from perspective of caller)
+;
+
+declare void @take_svcount_1(aarch64_svcount %arg)
+define void @test_pass_1arg(aarch64_svcount %arg) nounwind {
+; CHECK-LABEL: test_pass_1arg:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT:    bl take_svcount_1
+; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    ret
+  call void @take_svcount_1(aarch64_svcount %arg)
+  ret void
+}
+
+declare void @take_svcount_5(aarch64_svcount %arg0, aarch64_svcount %arg1, aarch64_svcount %arg2, aarch64_svcount %arg3, aarch64_svcount %arg4)
+define void @test_pass_5args(aarch64_svcount %arg) nounwind {
+; CHECKO0-LABEL: test_pass_5args:
+; CHECKO0:       // %bb.0:
+; CHECKO0-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECKO0-NEXT:    addvl sp, sp, #-1
+; CHECKO0-NEXT:    mov p3.b, p0.b
+; CHECKO0-NEXT:    str p3, [sp, #7, mul vl]
+; CHECKO0-NEXT:    addpl x0, sp, #7
+; CHECKO0-NEXT:    mov p0.b, p3.b
+; CHECKO0-NEXT:    mov p1.b, p3.b
+; CHECKO0-NEXT:    mov p2.b, p3.b
+; CHECKO0-NEXT:    bl take_svcount_5
+; CHECKO0-NEXT:    addvl sp, sp, #1
+; CHECKO0-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECKO0-NEXT:    ret
+;
+; CHECKO3-LABEL: test_pass_5args:
+; CHECKO3:       // %bb.0:
+; CHECKO3-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECKO3-NEXT:    addvl sp, sp, #-1
+; CHECKO3-NEXT:    addpl x0, sp, #7
+; CHECKO3-NEXT:    mov p1.b, p0.b
+; CHECKO3-NEXT:    mov p2.b, p0.b
+; CHECKO3-NEXT:    mov p3.b, p0.b
+; CHECKO3-NEXT:    str p0, [sp, #7, mul vl]
+; CHECKO3-NEXT:    bl take_svcount_5
+; CHECKO3-NEXT:    addvl sp, sp, #1
+; CHECKO3-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECKO3-NEXT:    ret
+  call void @take_svcount_5(aarch64_svcount %arg, aarch64_svcount %arg, aarch64_svcount %arg, aarch64_svcount %arg, aarch64_svcount %arg)
+  ret void
+}

llvm/test/Transforms/InstCombine/AArch64/sme-svcount.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -instcombine -S < %s | FileCheck %s
+
+define aarch64_svcount @test_alloca_store_reload(aarch64_svcount %val) nounwind {
+; CHECK-LABEL: @test_alloca_store_reload(
+; CHECK-NEXT:    ret aarch64_svcount [[VAL:%.*]]
+;
+  %ptr = alloca aarch64_svcount, align 1
+  store aarch64_svcount %val, ptr %ptr
+  %res = load aarch64_svcount, ptr %ptr
+  ret aarch64_svcount %res
+}

llvm/test/Transforms/SROA/aarch64-sme-svcount.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -sroa -S < %s | FileCheck %s
+
+define aarch64_svcount @test_alloca_store_reload(aarch64_svcount %val) nounwind {
+; CHECK-LABEL: @test_alloca_store_reload(
+; CHECK-NEXT:    [[PTR:%.*]] = alloca aarch64_svcount, align 1
+; CHECK-NEXT:    store aarch64_svcount [[VAL:%.*]], ptr [[PTR]], align 2
+; CHECK-NEXT:    [[RES:%.*]] = load aarch64_svcount, ptr [[PTR]], align 2
+; CHECK-NEXT:    ret aarch64_svcount [[RES]]
+;
+  %ptr = alloca aarch64_svcount, align 1
+  store aarch64_svcount %val, ptr %ptr
+  %res = load aarch64_svcount, ptr %ptr
+  ret aarch64_svcount %res
+}

llvm/utils/TableGen/CodeGenTarget.cpp

StringRef llvm::getEnumName(MVT::SimpleValueType T) {
   case MVT::bf16:     return "MVT::bf16";
   case MVT::f32:      return "MVT::f32";
   case MVT::f64:      return "MVT::f64";
   case MVT::f80:      return "MVT::f80";
   case MVT::f128:     return "MVT::f128";
   case MVT::ppcf128:  return "MVT::ppcf128";
   case MVT::x86mmx:   return "MVT::x86mmx";
   case MVT::x86amx:   return "MVT::x86amx";
+  case MVT::aarch64svcount:   return "MVT::aarch64svcount";
   case MVT::i64x8:    return "MVT::i64x8";
   case MVT::Glue:     return "MVT::Glue";
   case MVT::isVoid:   return "MVT::isVoid";
   case MVT::v1i1:     return "MVT::v1i1";
   case MVT::v2i1:     return "MVT::v2i1";
   case MVT::v4i1:     return "MVT::v4i1";
   case MVT::v8i1:     return "MVT::v8i1";
   case MVT::v16i1:    return "MVT::v16i1";