[mlir][RFC] Add scalable dimensions to VectorType

[jsetoain]

Summary

With VectorType supporting scalable dimensions, we don't need many of
the operations currently present in ArmSVE, like mask generation and
basic arithmetic instructions. Therefore, this patch also gets
rid of those.

Having built-in scalable vector support also simplifies the lowering of
scalable vector dialects down to LLVMIR.

Scalable dimensions are indicated with the scalable dimensions
between square brackets:

        vector<[4]xf32>

Is a scalable vector of 4 single precission floating point elements.

More generally, a VectorType can have a set of fixed-length dimensions
followed by a set of scalable dimensions:

        vector<2x[4x4]xf32>

Is a vector with 2 scalable 4x4 vectors of single precission floating
point elements.

The scale of the scalable dimensions can be obtained with the Vector
operation:

        %vs = vector.vscale

This change is being discussed in the discourse RFC:

https://llvm.discourse.group/t/rfc-add-built-in-support-for-scalable-vector-types/4484

Differential Revision: https://reviews.llvm.org/D111819

[aartbik]

This direction makes a lot of sense to me, if we want to avoid code dup between the upcoming vector specific dialects (SVE and RISC-V at the moment).
Since this touches "core", however, I hope others chime in too.

mlir/include/mlir/Dialect/LLVMIR/LLVMTypes.h

449

period at end

450

would IsScalableVectorType be a bit more consistent with naming?
(in sentence you would put scalable at the end, but since we use "ScalableVectorType" as typename this seems a bit better)

mlir/include/mlir/Dialect/StandardOps/IR/Ops.td

1123 ↗

(On Diff #379753)

compile-time

1125 ↗

(On Diff #379753)

comma, no : Also, is this not better an e.g.?
Or just:

For example, ....

mlir/include/mlir/IR/BuiltinTypes.td

907

Perhaps you can add some text calling out that < > is fixed length and << >> is scalable?
Just because it is a new syntax that we have to get used to ;-)

921

period at end

924

period at end

mlir/include/mlir/IR/OpBase.td

651

I wanted to say period at end, but I see that is not really the style in this file

[rriddle]

mlir/include/mlir/Dialect/StandardOps/IR/Ops.td

1118–1120 ↗

(On Diff #379753)

Why would this be here and not in say, the vector dialect?

mlir/lib/Dialect/LLVMIR/IR/LLVMTypes.cpp

554–555

Drop else after return.

568–570

587–588

Drop else after return.

mlir/lib/IR/BuiltinTypes.cpp

298

Use cast if you aren't checking the result, dyn_cast> can return null.

322

Same here.

346

And here, and others.

mlir/lib/Target/LLVMIR/TypeToLLVM.cpp

147–148

Drop else after return.

[jsetoain]

mlir/include/mlir/Dialect/StandardOps/IR/Ops.td

1118–1120 ↗

(On Diff #379753)

Indeed, my first instinct was to put it in Vector, but since this value is tightly coupled with the type itself, which is _not_ part of Vector, it feels a bit out of place there as well. I believe that the Vector type not being part of the Vector dialect is what creates the situation. As things are, if there's a place to put runtime constants, that's where this should go. Alternatively, if there's a way to express runtime properties of a type, say:

%0 = vector<<>>.scale : index

That also looks somewhat right (if a bit ugly). In any case, Standard or Vector, neither place looks better than the other to me, if you see clearly that this makes more sense in Vector, I can move it there and everything else works the same. This is something I actually wanted feedback about, thanks :-)

[rriddle]

mlir/include/mlir/Dialect/StandardOps/IR/Ops.td

1118–1120 ↗

(On Diff #379753)

I don't think it fits well in standard though, given that the standard dialect is going away. It seems like a better home should be found for this.

[jsetoain]

mlir/include/mlir/Dialect/StandardOps/IR/Ops.td

1118–1120 ↗

(On Diff #379753)

Agree. I'll move it to Vector as a non-entirely-terrible option and, should a better place become apparent, we can reconsider in the future. It's not intertwined with anything else, it'd be a quite innocuous change. Thanks!

[mehdi_amini]

This direction makes a lot of sense to me, if we want to avoid code dup between the upcoming vector specific dialects (SVE and RISC-V at the moment).
Since this touches "core", however, I hope others chime in too.

My main concern is that the discussion on Discourse does not seem to have a conclusion right now.

[jsetoain]

In D111819#3081297, @mehdi_amini wrote:

This direction makes a lot of sense to me, if we want to avoid code dup between the upcoming vector specific dialects (SVE and RISC-V at the moment).
Since this touches "core", however, I hope others chime in too.

My main concern is that the discussion on Discourse does not seem to have a conclusion right now.

My apologies. We did have a conversation this week but I didn't update the RFC (I have now). Although some details are still in the air, the need for a built-in type is not in question. This is just the first, most basic implementation possible.

[jsetoain]

Address reviewers' comments

[jsetoain]

I've addressed all the comments.

[jsetoain]

Fixed formatting

[jsetoain]

Rebase on main

[jsetoain]

Rebase on main

[jsetoain]

Rebase on main

[rriddle]

mlir/include/mlir/Dialect/LLVMIR/LLVMTypes.h

450

The get here is a bit weird, why not something like isScalableVectorType?

mlir/include/mlir/Dialect/Vector/VectorOps.td

2388–2389

You could also drop the trailing type if you want, it can be inferred. (i.e. = "attr-dict")

mlir/include/mlir/IR/BuiltinTypes.td

936

Is this wrapped at 80 characters?

mlir/lib/Dialect/ArmSVE/IR/ArmSVEDialect.cpp

51

mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp

161

Why the extra spaces?

mlir/lib/Dialect/LLVMIR/IR/LLVMTypes.cpp

551–557

mlir/lib/IR/BuiltinTypes.cpp

296–299

320

Same here.

344

and here.

mlir/lib/Parser/TypeParser.cpp

454–456

[jsetoain]

Address review comments

[mehdi_amini]

What's the next step on this? Seems like the RFC discussion got to a resolution right?

@nicolasvasilache @ftynse : can you chime in on the change in mlir/include/mlir/IR/BuiltinTypes.td ?

[nicolasvasilache]

My apologies for the long delay, my biggest problem atm is I do not have a good mental model of how RVV and Arm SVE operate in detail.
I had started to read specs but it invariably gets pushed back on the stack as it is not high on my priority list ..

From a pure cleanup perspective, I generally like it.

From a composability perspective, I think I would prefer to have it spelled out as vector<<4>xf32> or vector<4*xf32> or vector<(4s)xf32>.
The rationale is that I think we still want to have n-D scalable vector types in MLIR to allow expressing a statically known number of 1-D scalable vectors that serves as an "unroll-and-jammed vector pack" vector<8x4*xf32>.
This would be more consistent with the design of the rest of the vector dialect.

One thing that is higher priority to me personally these days is that we are also exploring using the vector dialect as a programming model for GPUs.
In this context, vector<4x8x16*x32*xf32> would also make sense for us.

Bottom line, if we avoided anchoring on the current LLVM / HW implementation that only support 1-D scalable vectors and we made it future-proof in that direction, I am fine with proceeding.

[jsetoain]

Generalize the concept of scalable dimensions to support use cases unrelated to scalable vectors

[jsetoain]

mlir/include/mlir/Dialect/Vector/VectorOps.td

2383

I've seen this, I'll take care of it together with any other necessary fix.

2388–2389

It feels a bit "naked", but it might be because I'm used to see it with the return type attached. We can give it a go and see what people think, if people don't care, going "concise" is my preferred option. Is there a "good practices" manual for dialect syntax? I can't find one.

mlir/include/mlir/IR/BuiltinTypes.td

909

And this...

936

Not sure what happened there. Good catch, thanks!

mlir/lib/Parser/TypeParser.cpp

454–456

Arg! That was embarrassing... Sorry about that!

[nicolasvasilache]

Mostly LGTM, I added 3 areas of improvements.
Once these are addressed I'll happily accept.
Thanks for your hard work and patience!

mlir/include/mlir/Dialect/Vector/VectorOps.td

2374

Should we call this vector.vscale ?

2378

I would emphasize that this is for 1-D scalable vectors and that there is currently no way to extract the scale
for a >1-D scalable vectors.
This instruction may be extended in the future to take a position but I am unclear whether this is what we want atm.

I think the global vs local property of vscale should also be discussed here.

I'd maybe even go as far as spelling it vector.scale.global in the future?

Edit: as I read deeper through the PR, I am now unclear whether vector<[2x8]xf32> is the same as vector<[2]x[8]xf32> ?

I think vector<[2x8]xf32> would make sense for SVE in MLIR (and would then get flattened to 1-D going through LLVM).
In the future we may also want vector<[2]x[8]xf32> for GPUs but this is not the same representation?
Is this what you have in mind ?

In any case, please propose a few wording changes to integrate the relevant parts of my comments and disregard/add a TODO for the others :)

mlir/include/mlir/IR/BuiltinTypes.h

323 ↗

(On Diff #391762)

I fear this will prove annoying to use in practice ..

Could we go with unsigned numScalableDims?

Then you can just use APIS such as ArrayRef's shape.take_back(numScalableDims); and friends.

342 ↗

(On Diff #391762)

this would get nicer with numScalableDims.

mlir/include/mlir/IR/BuiltinTypes.td

928

Now that I read this, I am unclear whether vector<[2x8]xf32> is the same as vector<[2]x[8]xf32>, I would think not and the latter form could be a future extension (if so, add a TODO)?
This really depends on whether you think you can make use of vector<[2x8]xf32> in MLIR instead of having to represent as vector<[16]xf32>; I claim you would have a bunch of nice use cases for this (coupled with the shape_cast op once properly extended).

mlir/test/Dialect/ArmSVE/legalize-for-llvm.mlir

1

I would expect to see a test file (somwhere in the builtin stuff) where you have both:

• negative tests for various failure modes of misuses of scalable vectors (with appropriate error messages)
• positive tests with multi-dim multi-scale vector (atm everything I see is 0-dim 1-scale only).

In a followup PR, I'd love to see a 1-dim, 2-scale version of the neon 2d dot (or something equivalent) and see it lower to unrolled LLVM.

[nicolasvasilache]

Also signal boosting for @ThomasRaoux @dcaballe @springerm ; no need to review but be aware that this is coming.

[rriddle]

mlir/include/mlir/IR/BuiltinTypes.td

937

Can we use Optional here instead? -1 is a bit magic.

953–954

Why not just isScalable? The naming here is a bit weird.

mlir/lib/IR/AsmPrinter.cpp

1945

Please cache the end iterator to avoid recomputing it every iteration.

mlir/lib/Parser/TypeParser.cpp

525

Looks like this is missing test coverage.

[jsetoain]

mlir/include/mlir/IR/BuiltinTypes.td

928

It is, indeed, very much not the same. I find it useful to think about something like [2x8] as a series of 2x8 blocks, one after another. Therefore, even though they would have the same memory requirements, [2x8], [8x2], [4x4], and [16] can represent different data arrangements when you're loading your data from memory. From that point of view, [2]x[8] can't be the same as [2x8] even if the scale for both dimensions is the same. In fact, I don't think something like [2]x[8] makes sense in the context of scalable vectors. For GPU thread blocks, the situation is different. I'm not involved with that work so I can't come up with anything on the spot, but I intuit it could have potentially useful cases. As this work progresses, I suspect we will need to come back to it.

[jsetoain]

Addressed latest round of reviews

[jsetoain]

I've also moved a bunch of tests around. Scalable vector tests that are not SVE-specific have been moved either to Arithmetic or Vector, depending on their nature.

mlir/include/mlir/Dialect/Vector/VectorOps.td

2378

That's not exactly right. You can have a 2D scalable vector, and vscale represents its multiplicity, but you can't have a 2D scalable vector with two different scales (which we might want to have for GPUs). As it is, we can't represent those yet, so I don't think we need to clarify that in the description. I've added the multi-dimensional multi-scale vector and local/global scale to a TODO for future reference.

mlir/include/mlir/IR/BuiltinTypes.td

937

Not sure how to use Optional for Types, this is the only way I found to provide a default value in a type builder. In any case, I've changed it to "numScalableDims" as suggested by Nicolas. It makes code a bit less awkward and conveniently replaces a arguably ugly "first dimension = -1" to a more semantically sensible "number of dimensions = 0". If you still find this unacceptable, I can look into adding an "Optional" equivalent for types.

mlir/test/Dialect/ArmSVE/legalize-for-llvm.mlir

1

RE follow-up PR, that was in my low priority TODO list, I'll move it to the main TODO, it should be a quick and easy change.

[rriddle]

LG from my point-of-view, but also get an LGTM from Nicolas as well.

mlir/include/mlir/Dialect/Vector/VectorOps.td

2374

Can you move this into the documentation of the op? This seems useful to expose in the user facing docs.

2388–2389

I don't think we have a "good practices" manual, though that sounds useful.

2399

If a verifier isn't necessary, you can just ignore it.

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp

31

nit: Prefer pre-increment unless you need post increment behavior.

38

Same here.

mlir/lib/IR/AsmPrinter.cpp

1945

Unresolved.

[jsetoain]

Address reviewer comments

[jsetoain]

Addressed comments.

[nicolasvasilache]

Thanks for getting these through @jsetoain !

[jsetoain]

Rebase on main

mlir/include/mlir/Dialect/ArmSVE/ArmSVE.td

 //===----------------------------------------------------------------------===//
 
 #ifndef ARMSVE_OPS
 #define ARMSVE_OPS
 
 include "mlir/Interfaces/SideEffectInterfaces.td"
 include "mlir/Dialect/LLVMIR/LLVMOpBase.td"
 include "mlir/Dialect/Arithmetic/IR/ArithmeticBase.td"
-include "mlir/Dialect/ArmSVE/ArmSVEOpBase.td"
 
 //===----------------------------------------------------------------------===//
 // ArmSVE dialect definition
 //===----------------------------------------------------------------------===//
 
 def ArmSVE_Dialect : Dialect {
   let name = "arm_sve";
   let cppNamespace = "::mlir::arm_sve";
   let summary = "Basic dialect to target Arm SVE architectures";
   let description = [{
     This dialect contains the definitions necessary to target Arm SVE scalable
     vector operations, including a scalable vector type and intrinsics for
     some Arm SVE instructions.
   }];
 }
 
 //===----------------------------------------------------------------------===//
-// ArmSVE type definitions
-//===----------------------------------------------------------------------===//
-
-def ArmSVE_ScalableVectorType : DialectType<ArmSVE_Dialect,
-    CPred<"$_self.isa<ScalableVectorType>()">,
-              "scalable vector type">,
-    BuildableType<"$_builder.getType<ScalableVectorType>()"> {
-  let description = [{
-    `arm_sve.vector` represents vectors that will be processed by a scalable
-    vector architecture.
-  }];
-}
-
-class ArmSVE_Type<string name> : TypeDef<ArmSVE_Dialect, name> { }
-
-def ScalableVectorType : ArmSVE_Type<"ScalableVector"> {
-  let mnemonic = "vector";
-
-  let summary = "Scalable vector type";
-
-  let description = [{
-    A type representing scalable length SIMD vectors. Unlike fixed-length SIMD
-    vectors, whose size is constant and known at compile time, scalable
-    vectors' length is constant but determined by the specific hardware at
-    run time.
-  }];
-
-  let parameters = (ins
-    ArrayRefParameter<"int64_t", "Vector shape">:$shape,
-    "Type":$elementType
-  );
-
-  let printer = [{
-    $_printer << "<";
-    for (int64_t dim : getShape())
-      $_printer << dim << 'x';
-    $_printer << getElementType() << '>';
-  }];
-
-  let parser = [{
-    VectorType vector;
-    if ($_parser.parseType(vector))
-      return Type();
-    return get($_ctxt, vector.getShape(), vector.getElementType());
-  }];
-
-  let extraClassDeclaration = [{
-    bool hasStaticShape() const {
-      return llvm::none_of(getShape(), ShapedType::isDynamic);
-    }
-    int64_t getNumElements() const {
-      assert(hasStaticShape() &&
-             "cannot get element count of dynamic shaped type");
-      ArrayRef<int64_t> shape = getShape();
-      int64_t num = 1;
-      for (auto dim : shape)
-        num *= dim;
-      return num;
-    }
-  }];
-}
-
-//===----------------------------------------------------------------------===//
-// Additional LLVM type constraints
-//===----------------------------------------------------------------------===//
-def LLVMScalableVectorType :
-  Type<CPred<"$_self.isa<::mlir::LLVM::LLVMScalableVectorType>()">,
-       "LLVM dialect scalable vector type">;
-
-//===----------------------------------------------------------------------===//
 // ArmSVE op definitions
 //===----------------------------------------------------------------------===//
 
 class ArmSVE_Op<string mnemonic, list<OpTrait> traits = []> :
   Op<ArmSVE_Dialect, mnemonic, traits> {}
 
-class ArmSVE_NonSVEIntrUnaryOverloadedOp<string mnemonic,
-                                         list<OpTrait> traits =[]> :
-  LLVM_IntrOpBase</*Dialect dialect=*/ArmSVE_Dialect,
-                  /*string opName=*/mnemonic,
-                  /*string enumName=*/mnemonic,
-                  /*list<int> overloadedResults=*/[0],
-                  /*list<int> overloadedOperands=*/[], // defined by result overload
-                  /*list<OpTrait> traits=*/traits,
-                  /*int numResults=*/1>;
-
 class ArmSVE_IntrBinaryOverloadedOp<string mnemonic,
                                     list<OpTrait> traits = []> :
   LLVM_IntrOpBase</*Dialect dialect=*/ArmSVE_Dialect,
                   /*string opName=*/"intr." # mnemonic,
                   /*string enumName=*/"aarch64_sve_" # !subst(".", "_", mnemonic),
                   /*list<int> overloadedResults=*/[0],
                   /*list<int> overloadedOperands=*/[], // defined by result overload
                   /*list<OpTrait> traits=*/traits,
                   /*int numResults=*/1>;
 
-class ScalableFOp<string mnemonic, string op_description,
-                  list<OpTrait> traits = []> :
-  ArmSVE_Op<mnemonic, !listconcat(traits,
-                       [AllTypesMatch<["src1", "src2", "dst"]>])> {
-  let summary = op_description # " for scalable vectors of floats";
-  let description = [{
-    The `arm_sve.}] # mnemonic # [{` operations takes two scalable vectors and
-    returns one scalable vector with the result of the }] # op_description # [{.
-  }];
-  let arguments = (ins
-          ScalableVectorOf<[AnyFloat]>:$src1,
-          ScalableVectorOf<[AnyFloat]>:$src2
-  );
-  let results = (outs ScalableVectorOf<[AnyFloat]>:$dst);
-  let assemblyFormat =
-    "$src1 `,` $src2 attr-dict `:` type($src1)";
-}
-
-class ScalableIOp<string mnemonic, string op_description,
-                  list<OpTrait> traits = []> :
-  ArmSVE_Op<mnemonic, !listconcat(traits,
-                       [AllTypesMatch<["src1", "src2", "dst"]>])> {
-  let summary = op_description # " for scalable vectors of integers";
-  let description = [{
-    The `arm_sve.}] # mnemonic # [{` operation takes two scalable vectors and
-    returns one scalable vector with the result of the }] # op_description # [{.
-  }];
-  let arguments = (ins
-          ScalableVectorOf<[I8, I16, I32, I64]>:$src1,
-          ScalableVectorOf<[I8, I16, I32, I64]>:$src2
-  );
-  let results = (outs ScalableVectorOf<[I8, I16, I32, I64]>:$dst);
-  let assemblyFormat =
-    "$src1 `,` $src2 attr-dict `:` type($src1)";
-}
-
 class ScalableMaskedFOp<string mnemonic, string op_description,
                         list<OpTrait> traits = []> :
   ArmSVE_Op<mnemonic, !listconcat(traits,
                        [AllTypesMatch<["src1", "src2", "res"]>,
                         TypesMatchWith<
                           "mask has i1 element type and same shape as operands",
                           "src1", "mask", "getI1SameShape($_self)">])> {
   let summary = "masked " # op_description # " for scalable vectors of floats";
let arguments = (ins
           ScalableVectorOfLengthAndType<[16], [I8]>:$src1,
           ScalableVectorOfLengthAndType<[16], [I8]>:$src2
   );
   let results = (outs ScalableVectorOfLengthAndType<[4], [I32]>:$dst);
   let assemblyFormat =
     "$acc `,` $src1 `,` $src2 attr-dict `:` type($src1) `to` type($dst)";
 }
 
-def VectorScaleOp : ArmSVE_Op<"vector_scale",
-                 [NoSideEffect]> {
-  let summary = "Load vector scale size";
-  let description = [{
-    The vector_scale op returns the scale of the scalable vectors, a positive
-    integer value that is constant at runtime but unknown at compile time.
-    The scale of the vector indicates the multiplicity of the vectors and
-    vector operations. I.e.: an !arm_sve.vector<4xi32> is equivalent to
-    vector_scale consecutive vector<4xi32>; and an operation on an
-    !arm_sve.vector<4xi32> is equivalent to performing that operation vector_scale
-    times, once on each <4xi32> segment of the scalable vector. The vector_scale
-    op can be used to calculate the step in vector-length agnostic (VLA) loops.
-  }];
-  let results = (outs Index:$res);
-  let assemblyFormat =
-    "attr-dict `:` type($res)";
-}
-
-def ScalableLoadOp : ArmSVE_Op<"load">,
-    Arguments<(ins Arg<AnyMemRef, "", [MemRead]>:$base, Index:$index)>,
-    Results<(outs ScalableVectorOf<[AnyType]>:$result)> {
-  let summary = "Load scalable vector from memory";
-  let description = [{
-    Load a slice of memory into a scalable vector.
-  }];
-  let extraClassDeclaration = [{
-    MemRefType getMemRefType() {
-      return base().getType().cast<MemRefType>();
-    }
-  }];
-  let assemblyFormat = "$base `[` $index `]` attr-dict `:` "
-    "type($result) `from` type($base)";
-}
-
-def ScalableStoreOp : ArmSVE_Op<"store">,
-    Arguments<(ins Arg<AnyMemRef, "", [MemWrite]>:$base, Index:$index,
-               ScalableVectorOf<[AnyType]>:$value)> {
-  let summary = "Store scalable vector into memory";
-  let description = [{
-    Store a scalable vector on a slice of memory.
-  }];
-  let extraClassDeclaration = [{
-    MemRefType getMemRefType() {
-      return base().getType().cast<MemRefType>();
-    }
-  }];
-  let assemblyFormat = "$value `,` $base `[` $index `]` attr-dict `:` "
-    "type($value) `to` type($base)";
-}
-
-def ScalableAddIOp : ScalableIOp<"addi", "addition", [Commutative]>;
-
-def ScalableAddFOp : ScalableFOp<"addf", "addition", [Commutative]>;
-
-def ScalableSubIOp : ScalableIOp<"subi", "subtraction">;
-
-def ScalableSubFOp : ScalableFOp<"subf", "subtraction">;
-
-def ScalableMulIOp : ScalableIOp<"muli", "multiplication", [Commutative]>;
-
-def ScalableMulFOp : ScalableFOp<"mulf", "multiplication", [Commutative]>;
-
-def ScalableSDivIOp : ScalableIOp<"divi_signed", "signed division">;
-
-def ScalableUDivIOp : ScalableIOp<"divi_unsigned", "unsigned division">;
-
-def ScalableDivFOp : ScalableFOp<"divf", "division">;
-
 def ScalableMaskedAddIOp : ScalableMaskedIOp<"masked.addi", "addition",
                                              [Commutative]>;
 
 def ScalableMaskedAddFOp : ScalableMaskedFOp<"masked.addf", "addition",
                             [Commutative]>;
 
 def ScalableMaskedSubIOp : ScalableMaskedIOp<"masked.subi", "subtraction">;
 
 def ScalableMaskedSubFOp : ScalableMaskedFOp<"masked.subf", "subtraction">;
 
 def ScalableMaskedMulIOp : ScalableMaskedIOp<"masked.muli", "multiplication",
                             [Commutative]>;
 
 def ScalableMaskedMulFOp : ScalableMaskedFOp<"masked.mulf", "multiplication",
                             [Commutative]>;
 
 def ScalableMaskedSDivIOp : ScalableMaskedIOp<"masked.divi_signed",
                                               "signed division">;
 
 def ScalableMaskedUDivIOp : ScalableMaskedIOp<"masked.divi_unsigned",
                                               "unsigned division">;
 
 def ScalableMaskedDivFOp : ScalableMaskedFOp<"masked.divf", "division">;
 
-//===----------------------------------------------------------------------===//
-// ScalableCmpFOp
-//===----------------------------------------------------------------------===//
-
-def ScalableCmpFOp : ArmSVE_Op<"cmpf", [NoSideEffect, SameTypeOperands,
-    TypesMatchWith<"result type has i1 element type and same shape as operands",
-    "lhs", "result", "getI1SameShape($_self)">]> {
-  let summary = "floating-point comparison operation for scalable vectors";
-  let description = [{
-    The `arm_sve.cmpf` operation compares two scalable vectors of floating point
-    elements according to the float comparison rules and the predicate specified
-    by the respective attribute. The predicate defines the type of comparison:
-    (un)orderedness, (in)equality and signed less/greater than (or equal to) as
-    well as predicates that are always true or false. The result is a scalable
-    vector of i1 elements. Unlike `arm_sve.cmpi`, the operands are always
-    treated as signed. The u prefix indicates *unordered* comparison, not
-    unsigned comparison, so "une" means unordered not equal. For the sake of
-    readability by humans, custom assembly form for the operation uses a
-    string-typed attribute for the predicate.  The value of this attribute
-    corresponds to lower-cased name of the predicate constant, e.g., "one" means
-    "ordered not equal".  The string representation of the attribute is merely a
-    syntactic sugar and is converted to an integer attribute by the parser.
-
-    Example:
-
-    ```mlir
-    %r = arm_sve.cmpf oeq, %0, %1 : !arm_sve.vector<4xf32>
-    ```
-  }];
-  let arguments = (ins
-    Arith_CmpFPredicateAttr:$predicate,
-    ScalableVectorOf<[AnyFloat]>:$lhs,
-    ScalableVectorOf<[AnyFloat]>:$rhs // TODO: This should support a simple scalar
-  );
-  let results = (outs ScalableVectorOf<[I1]>:$result);
-
-  let builders = [
-    OpBuilder<(ins "arith::CmpFPredicate":$predicate, "Value":$lhs,
-                  "Value":$rhs), [{
-      buildScalableCmpFOp($_builder, $_state, predicate, lhs, rhs);
-    }]>];
-
-  let extraClassDeclaration = [{
-    static StringRef getPredicateAttrName() { return "predicate"; }
-    static arith::CmpFPredicate getPredicateByName(StringRef name);
-
-    arith::CmpFPredicate getPredicate() {
-      return (arith::CmpFPredicate) (*this)->getAttrOfType<IntegerAttr>(
-          getPredicateAttrName()).getInt();
-    }
-  }];
-
-  let verifier = [{ return success(); }];
-
-  let assemblyFormat = "$predicate `,` $lhs `,` $rhs attr-dict `:` type($lhs)";
-}
-
-//===----------------------------------------------------------------------===//
-// ScalableCmpIOp
-//===----------------------------------------------------------------------===//
-
-def ScalableCmpIOp : ArmSVE_Op<"cmpi", [NoSideEffect, SameTypeOperands,
-    TypesMatchWith<"result type has i1 element type and same shape as operands",
-    "lhs", "result", "getI1SameShape($_self)">]> {
-  let summary = "integer comparison operation for scalable vectors";
-  let description = [{
-    The `arm_sve.cmpi` operation compares two scalable vectors of integer
-    elements according to the predicate specified by the respective attribute.
-
-    The predicate defines the type of comparison:
-
-    -   equal (mnemonic: `"eq"`; integer value: `0`)
-    -   not equal (mnemonic: `"ne"`; integer value: `1`)
-    -   signed less than (mnemonic: `"slt"`; integer value: `2`)
-    -   signed less than or equal (mnemonic: `"sle"`; integer value: `3`)
-    -   signed greater than (mnemonic: `"sgt"`; integer value: `4`)
-    -   signed greater than or equal (mnemonic: `"sge"`; integer value: `5`)
-    -   unsigned less than (mnemonic: `"ult"`; integer value: `6`)
-    -   unsigned less than or equal (mnemonic: `"ule"`; integer value: `7`)
-    -   unsigned greater than (mnemonic: `"ugt"`; integer value: `8`)
-    -   unsigned greater than or equal (mnemonic: `"uge"`; integer value: `9`)
-
-    Example:
-
-    ```mlir
-    %r = arm_sve.cmpi uge, %0, %1 : !arm_sve.vector<4xi32>
-    ```
-  }];
-
-  let arguments = (ins
-      Arith_CmpIPredicateAttr:$predicate,
-      ScalableVectorOf<[I8, I16, I32, I64]>:$lhs,
-      ScalableVectorOf<[I8, I16, I32, I64]>:$rhs
-  );
-  let results = (outs ScalableVectorOf<[I1]>:$result);
-
-  let builders = [
-    OpBuilder<(ins "arith::CmpIPredicate":$predicate, "Value":$lhs,
-                 "Value":$rhs), [{
-      buildScalableCmpIOp($_builder, $_state, predicate, lhs, rhs);
-    }]>];
-
-  let extraClassDeclaration = [{
-    static StringRef getPredicateAttrName() { return "predicate"; }
-    static arith::CmpIPredicate getPredicateByName(StringRef name);
-
-    arith::CmpIPredicate getPredicate() {
-      return (arith::CmpIPredicate) (*this)->getAttrOfType<IntegerAttr>(
-          getPredicateAttrName()).getInt();
-    }
-  }];
-
-  let verifier = [{ return success(); }];
-
-  let assemblyFormat = "$predicate `,` $lhs `,` $rhs attr-dict `:` type($lhs)";
-}
-
 def UmmlaIntrOp :
   ArmSVE_IntrBinaryOverloadedOp<"ummla">,
-  Arguments<(ins LLVMScalableVectorType, LLVMScalableVectorType,
-             LLVMScalableVectorType)>;
+  Arguments<(ins AnyScalableVector, AnyScalableVector, AnyScalableVector)>;
 
 def SmmlaIntrOp :
   ArmSVE_IntrBinaryOverloadedOp<"smmla">,
-  Arguments<(ins LLVMScalableVectorType, LLVMScalableVectorType,
-             LLVMScalableVectorType)>;
+  Arguments<(ins AnyScalableVector, AnyScalableVector, AnyScalableVector)>;
 
 def SdotIntrOp :
   ArmSVE_IntrBinaryOverloadedOp<"sdot">,
-  Arguments<(ins LLVMScalableVectorType, LLVMScalableVectorType,
-             LLVMScalableVectorType)>;
+  Arguments<(ins AnyScalableVector, AnyScalableVector, AnyScalableVector)>;
 
 def UdotIntrOp :
   ArmSVE_IntrBinaryOverloadedOp<"udot">,
-  Arguments<(ins LLVMScalableVectorType, LLVMScalableVectorType,
-             LLVMScalableVectorType)>;
+  Arguments<(ins AnyScalableVector, AnyScalableVector, AnyScalableVector)>;
 
 def ScalableMaskedAddIIntrOp :
   ArmSVE_IntrBinaryOverloadedOp<"add">,
-  Arguments<(ins LLVMScalableVectorType, LLVMScalableVectorType,
-             LLVMScalableVectorType)>;
+  Arguments<(ins AnyScalableVector, AnyScalableVector, AnyScalableVector)>;
 
 def ScalableMaskedAddFIntrOp :
   ArmSVE_IntrBinaryOverloadedOp<"fadd">,
-  Arguments<(ins LLVMScalableVectorType, LLVMScalableVectorType,
-             LLVMScalableVectorType)>;
+  Arguments<(ins AnyScalableVector, AnyScalableVector, AnyScalableVector)>;
 
 def ScalableMaskedMulIIntrOp :
   ArmSVE_IntrBinaryOverloadedOp<"mul">,
-  Arguments<(ins LLVMScalableVectorType, LLVMScalableVectorType,
-             LLVMScalableVectorType)>;
+  Arguments<(ins AnyScalableVector, AnyScalableVector, AnyScalableVector)>;
 
 def ScalableMaskedMulFIntrOp :
   ArmSVE_IntrBinaryOverloadedOp<"fmul">,
-  Arguments<(ins LLVMScalableVectorType, LLVMScalableVectorType,
-             LLVMScalableVectorType)>;
+  Arguments<(ins AnyScalableVector, AnyScalableVector, AnyScalableVector)>;
 
 def ScalableMaskedSubIIntrOp :
   ArmSVE_IntrBinaryOverloadedOp<"sub">,
-  Arguments<(ins LLVMScalableVectorType, LLVMScalableVectorType,
-             LLVMScalableVectorType)>;
+  Arguments<(ins AnyScalableVector, AnyScalableVector, AnyScalableVector)>;
 
 def ScalableMaskedSubFIntrOp :
   ArmSVE_IntrBinaryOverloadedOp<"fsub">,
-  Arguments<(ins LLVMScalableVectorType, LLVMScalableVectorType,
-             LLVMScalableVectorType)>;
+  Arguments<(ins AnyScalableVector, AnyScalableVector, AnyScalableVector)>;
 
 def ScalableMaskedSDivIIntrOp :
   ArmSVE_IntrBinaryOverloadedOp<"sdiv">,
-  Arguments<(ins LLVMScalableVectorType, LLVMScalableVectorType,
-             LLVMScalableVectorType)>;
+  Arguments<(ins AnyScalableVector, AnyScalableVector, AnyScalableVector)>;
 
 def ScalableMaskedUDivIIntrOp :
   ArmSVE_IntrBinaryOverloadedOp<"udiv">,
-  Arguments<(ins LLVMScalableVectorType, LLVMScalableVectorType,
-             LLVMScalableVectorType)>;
+  Arguments<(ins AnyScalableVector, AnyScalableVector, AnyScalableVector)>;
 
 def ScalableMaskedDivFIntrOp :
   ArmSVE_IntrBinaryOverloadedOp<"fdiv">,
-  Arguments<(ins LLVMScalableVectorType, LLVMScalableVectorType,
-             LLVMScalableVectorType)>;
-
-def VectorScaleIntrOp:
-  ArmSVE_NonSVEIntrUnaryOverloadedOp<"vscale">;
+  Arguments<(ins AnyScalableVector, AnyScalableVector, AnyScalableVector)>;
 
 #endif // ARMSVE_OPS

mlir/include/mlir/Dialect/ArmSVE/ArmSVEDialect.h

 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Dialect.h"
 #include "mlir/IR/OpDefinition.h"
 #include "mlir/Interfaces/SideEffectInterfaces.h"
 
 #include "mlir/Dialect/ArmSVE/ArmSVEDialect.h.inc"
 #include "mlir/Dialect/StandardOps/IR/Ops.h"
 
-#define GET_TYPEDEF_CLASSES
-#include "mlir/Dialect/ArmSVE/ArmSVETypes.h.inc"
-
 #define GET_OP_CLASSES
 #include "mlir/Dialect/ArmSVE/ArmSVE.h.inc"
 
 #endif // MLIR_DIALECT_ARMSVE_ARMSVEDIALECT_H

mlir/include/mlir/Dialect/ArmSVE/ArmSVEOpBase.td

-//===-- ArmSVEOpBase.td - Base op definitions for ArmSVE ---*- tablegen -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// This is the base operation definition file for ArmSVE scalable vector types.
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef ARMSVE_OP_BASE
-#define ARMSVE_OP_BASE
-
-//===----------------------------------------------------------------------===//
-// ArmSVE scalable vector type constraints
-//===----------------------------------------------------------------------===//
-
-def IsScalableVectorTypePred :
-    CPred<"$_self.isa<::mlir::arm_sve::ScalableVectorType>()">;
-
-class ScalableVectorOf<list<Type> allowedTypes> :
-    ContainerType<AnyTypeOf<allowedTypes>, IsScalableVectorTypePred,
-          "$_self.cast<::mlir::arm_sve::ScalableVectorType>().getElementType()",
-          "scalable vector">;
-
-// Whether the number of elements of a scalable vector is from the given
-// `allowedLengths` list
-class IsScalableVectorOfLengthPred<list<int> allowedLengths> :
-  And<[IsScalableVectorTypePred,
-       Or<!foreach(allowedlength, allowedLengths, CPred<
-          [{$_self.cast<::mlir::arm_sve::ScalableVectorType>().getNumElements() == }]
-          # allowedlength>)>]>;
-
-// Any scalable vector where the number of elements is from the given
-// `allowedLengths` list
-class ScalableVectorOfLength<list<int> allowedLengths> : Type<
-  IsScalableVectorOfLengthPred<allowedLengths>,
-  " of length " # !interleave(allowedLengths, "/"),
-  "::mlir::arm_sve::ScalableVectorType">;
-
-// Any scalable vector where the number of elements is from the given
-// `allowedLengths` list and the type is from the given `allowedTypes` list
-class ScalableVectorOfLengthAndType<list<int> allowedLengths,
-                                    list<Type> allowedTypes> : Type<
-  And<[ScalableVectorOf<allowedTypes>.predicate,
-       ScalableVectorOfLength<allowedLengths>.predicate]>,
-  ScalableVectorOf<allowedTypes>.summary #
-  ScalableVectorOfLength<allowedLengths>.summary,
-  "::mlir::arm_sve::ScalableVectorType">;
-
-#endif // ARMSVE_OP_BASE

mlir/include/mlir/Dialect/LLVMIR/LLVMOps.td

 }
 
 /// Create a call to Masked Compress Store intrinsic.
 def LLVM_masked_compressstore
     : LLVM_IntrOp<"masked.compressstore", [], [0], [], 0> {
   let arguments = (ins LLVM_Type, LLVM_Type, LLVM_Type);
 }
 
-//
+/// Create a call to vscale intrinsic.
+def LLVM_vscale : LLVM_IntrOp<"vscale", [0], [], [], 1>;
+
 // Atomic operations.
 //
 
 def AtomicBinOpXchg : I64EnumAttrCase<"xchg", 0>;
 def AtomicBinOpAdd  : I64EnumAttrCase<"add", 1>;
 def AtomicBinOpSub  : I64EnumAttrCase<"sub", 2>;
 def AtomicBinOpAnd  : I64EnumAttrCase<"_and", 3>;
 def AtomicBinOpNand : I64EnumAttrCase<"nand", 4>;

mlir/include/mlir/Dialect/LLVMIR/LLVMTypes.h

 
 /// Returns the element type of any vector type compatible with the LLVM
 /// dialect.
 Type getVectorElementType(Type type);
 
 /// Returns the element count of any LLVM-compatible vector type.
 llvm::ElementCount getVectorNumElements(Type type);
 
+/// Returns whether a vector type is scalable or not.
+bool getIsScalableVectorType(Type vectorType);
+
+/// Creates an LLVM dialect-compatible vector type with the given element type
+/// and length.
+Type getVectorType(Type elementType, unsigned numElements, bool isScalable);
+
 /// Creates an LLVM dialect-compatible type with the given element type and
 /// length.
 Type getFixedVectorType(Type elementType, unsigned numElements);
 
+/// Creates an LLVM dialect-compatible type with the given element type and
+/// length.
+Type getScalableVectorType(Type elementType, unsigned numElements);
+
 /// Returns the size of the given primitive LLVM dialect-compatible type
 /// (including vectors) in bits, for example, the size of i16 is 16 and
 /// the size of vector<4xi16> is 64. Returns 0 for non-primitive
 /// (aggregates such as struct) or types that don't have a size (such as void).
 llvm::TypeSize getPrimitiveTypeSizeInBits(Type type);
 
 } // namespace LLVM
 } // namespace mlir
 
 #endif // MLIR_DIALECT_LLVMIR_LLVMTYPES_H_

mlir/include/mlir/Dialect/Vector/VectorOps.td

let description = [{
     %1 = vector.flat_transpose %0 { rows = 4: i32, columns = 4: i32 }
        : (vector<16xf32>) -> vector<16xf32>
     ```
   }];
   let verifier = ?;
   let assemblyFormat = "$matrix attr-dict `:` type($matrix) `->` type($res)";
 }
 
+//===----------------------------------------------------------------------===//
+// VectorScaleOp
+//===----------------------------------------------------------------------===//
+
+def VectorScaleOp : Vector_Op<"vector_scale",
+                 [NoSideEffect]> {
+  let summary = "Load vector scale size";
+  let description = [{
+    The vector_scale op returns the scale of the scalable vectors, a positive
+    integer value that is constant at runtime but unknown at compile-time.
+    The scale of the vector indicates the multiplicity of the vectors and
+    vector operations. For example, a vector<<4xi32>> is equivalent to
+    vector_scale consecutive vector<4xi32>; and an operation on a
+    vector<<4xi32>> is equivalent to performing that operation vector_scale
+    times, once on each <4xi32> segment of the scalable vector. The vector_scale
+    op can be used to calculate the step in vector-length agnostic (VLA) loops.
+  }];
+  let results = (outs Index:$res);
+  let assemblyFormat =
+    "attr-dict `:` type($res)";
+  let verifier = [{ return success(); }];
+}
+
 #endif // VECTOR_OPS

mlir/include/mlir/IR/BuiltinTypes.td

 def Builtin_Vector : Builtin_Type<"Vector", [
     DeclareTypeInterfaceMethods<SubElementTypeInterface>
   ], "ShapedType"> {
   let summary = "Multi-dimensional SIMD vector type";
   let description = [{
     Syntax:
 
     ```
-    vector-type ::= `vector` `<` static-dimension-list vector-element-type `>`
+    vector-type ::= fixed-length-vector | scalable-length-vector
+    fixed-length-vector ::= `vector` `<` static-dimension-list vector-element-type `>`
+    scalable-length-vector ::= `vector` `<<` static-dimension-list vector-element-type `>>`
     vector-element-type ::= float-type | integer-type | index-type
 
     static-dimension-list ::= (decimal-literal `x`)*
     ```
 
-    The vector type represents a SIMD style vector, used by target-specific
-    operation sets like AVX. While the most common use is for 1D vectors (e.g.
-    vector<16 x f32>) we also support multidimensional registers on targets that
-    support them (like TPUs).
+    The vector type represents a SIMD style vector, either fixed-length or
+    scalable length, used by target-specific operation sets like AVX or SVE.
+    Fixed-length vectors are represented by single angle brackets (< >), and
+    scalable-length vectors are represented by double angle brackets (<< >>).
+    While the most common use is for 1D vectors (e.g. vector<16 x f32>) we
+    also support multidimensional registers on targets that support them
+    (like TPUs).
 
     Vector shapes must be positive decimal integers. 0D vectors are allowed by
     omitting the dimension: `vector<f32>`.
 
     Note: hexadecimal integer literals are not allowed in vector type
     declarations, `vector<0x42xi32>` is invalid because it is interpreted as a
     2D vector with shape `(0, 42)` and zero shapes are not allowed.
 
     Examples:
 
     ```mlir
+    // A 2D fixed-length vector of 3x42 i32 elements.
     vector<3x42xi32>
+
+    // A 1D scalable-length vector that contains a multiple of 4 f32 elements.
+    vector<<4xf32>>
     ```
   }];
   let parameters = (ins
     ArrayRefParameter<"int64_t">:$shape,
-    "Type":$elementType
+    "Type":$elementType,
+    "bool":$isScalable
   );
 
   let builders = [
     TypeBuilderWithInferredContext<(ins
-      "ArrayRef<int64_t>":$shape, "Type":$elementType
+      "ArrayRef<int64_t>":$shape, "Type":$elementType, CArg<"bool", "false">:$isScalable
     ), [{
-      return $_get(elementType.getContext(), shape, elementType);
+      return $_get(elementType.getContext(), shape, elementType, isScalable);
     }]>
   ];
   let extraClassDeclaration = [{
     /// This is a builder type that keeps local references to arguments.
     /// Arguments that are passed into the builder must outlive the builder.
     class Builder;
 
     /// Returns true of the given type can be used as an element of a vector
     /// type. In particular, vectors can consist of integer, index, or float
     /// primitives.
     static bool isValidElementType(Type t) {
       return t.isa<IntegerType, IndexType, FloatType>();
     }
 
     /// Get or create a new VectorType with the same shape as `this` and an
     /// element type of bitwidth scaled by `scale`.
     /// Return null if the scaled element type cannot be represented.
     VectorType scaleElementBitwidth(unsigned scale);
   }];
   let skipDefaultBuilders = 1;
   let genVerifyDecl = 1;
 }
 
 #endif // BUILTIN_TYPES

mlir/include/mlir/IR/OpBase.td

 
 //===----------------------------------------------------------------------===//
 // Common predicates
 //===----------------------------------------------------------------------===//
 
 // Whether a type is a VectorType.
 def IsVectorTypePred : CPred<"$_self.isa<::mlir::VectorType>()">;
 
+// Whether a type is a fixed-length VectorType.
+def IsFixedVectorTypePred : CPred<[{$_self.isa<::mlir::VectorType>() &&
+                                  !$_self.cast<VectorType>().getIsScalable()}]>;
+
+// Whether a type is a scalable VectorType.
+def IsScalableVectorTypePred : CPred<[{$_self.isa<::mlir::VectorType>() &&
+                                   $_self.cast<VectorType>().getIsScalable()}]>;
+
 // Whether a type is a TensorType.
 def IsTensorTypePred : CPred<"$_self.isa<::mlir::TensorType>()">;
 
 // Whether a type is a MemRefType.
 def IsMemRefTypePred : CPred<"$_self.isa<::mlir::MemRefType>()">;
 
 // Whether a type is an UnrankedMemRefType
 def IsUnrankedMemRefTypePred
Or<!foreach(rank, ranks,
                       # rank>)>]>;
 
 // Vector types.
 
 class VectorOf<list<Type> allowedTypes> :
   ShapedContainerType<allowedTypes, IsVectorTypePred, "vector",
                       "::mlir::VectorType">;
 
+class FixedVectorOf<list<Type> allowedTypes> :
+  ShapedContainerType<allowedTypes, IsFixedVectorTypePred,
+          "fixed-length vector", "::mlir::VectorType">;
+
+class ScalableVectorOf<list<Type> allowedTypes> :
+  ShapedContainerType<allowedTypes, IsScalableVectorTypePred,
+          "scalable vector", "::mlir::VectorType">;
+
 // Whether the number of elements of a vector is from the given
 // `allowedRanks` list
 class IsVectorOfRankPred<list<int> allowedRanks> :
   And<[IsVectorTypePred,
        Or<!foreach(allowedlength, allowedRanks,
                    CPred<[{$_self.cast<::mlir::VectorType>().getRank()
                            == }]
                          # allowedlength>)>]>;

 // `allowedLengths` list
 class IsVectorOfLengthPred<list<int> allowedLengths> :
   And<[IsVectorTypePred,
        Or<!foreach(allowedlength, allowedLengths,
                    CPred<[{$_self.cast<::mlir::VectorType>().getNumElements()
                            == }]
                          # allowedlength>)>]>;
 
+// Whether the number of elements of a fixed-length vector is from the given
+// `allowedLengths` list
+class IsFixedVectorOfLengthPred<list<int> allowedLengths> :
+  And<[IsFixedVectorTypePred,
+       Or<!foreach(allowedlength, allowedLengths,
+                   CPred<[{$_self.cast<::mlir::VectorType>().getNumElements()
+                           == }]
+                         # allowedlength>)>]>;
+
+// Whether the number of elements of a scalable vector is from the given
+// `allowedLengths` list
+class IsScalableVectorOfLengthPred<list<int> allowedLengths> :
+  And<[IsScalableVectorTypePred,
+       Or<!foreach(allowedlength, allowedLengths,
+                   CPred<[{$_self.cast<::mlir::VectorType>().getNumElements()
+                           == }]
+                         # allowedlength>)>]>;
+
 // Any vector where the number of elements is from the given
 // `allowedLengths` list
 class VectorOfLength<list<int> allowedLengths> : Type<
   IsVectorOfLengthPred<allowedLengths>,
   " of length " # !interleave(allowedLengths, "/"),
   "::mlir::VectorType">;
 
+// Any fixed-length vector where the number of elements is from the given
+// `allowedLengths` list
+class FixedVectorOfLength<list<int> allowedLengths> : Type<
+  IsFixedVectorOfLengthPred<allowedLengths>,
+  " of length " # !interleave(allowedLengths, "/"),
+  "::mlir::VectorType">;
+
+// Any scalable vector where the number of elements is from the given
+// `allowedLengths` list
+class ScalableVectorOfLength<list<int> allowedLengths> : Type<
+  IsScalableVectorOfLengthPred<allowedLengths>,
+  " of length " # !interleave(allowedLengths, "/"),
+  "::mlir::VectorType">;
+
 // Any vector where the number of elements is from the given
 // `allowedLengths` list and the type is from the given `allowedTypes`
 // list
 class VectorOfLengthAndType<list<int> allowedLengths,
                             list<Type> allowedTypes> : Type<
   And<[VectorOf<allowedTypes>.predicate,
        VectorOfLength<allowedLengths>.predicate]>,
   VectorOf<allowedTypes>.summary # VectorOfLength<allowedLengths>.summary,
   "::mlir::VectorType">;
 
+// Any fixed-length vector where the number of elements is from the given
+// `allowedLengths` list and the type is from the given `allowedTypes` list
+class FixedVectorOfLengthAndType<list<int> allowedLengths,
+                                    list<Type> allowedTypes> : Type<
+  And<[FixedVectorOf<allowedTypes>.predicate,
+       FixedVectorOfLength<allowedLengths>.predicate]>,
+  FixedVectorOf<allowedTypes>.summary #
+  FixedVectorOfLength<allowedLengths>.summary,
+  "::mlir::VectorType">;
+
+// Any scalable vector where the number of elements is from the given
+// `allowedLengths` list and the type is from the given `allowedTypes` list
+class ScalableVectorOfLengthAndType<list<int> allowedLengths,
+                                    list<Type> allowedTypes> : Type<
+  And<[ScalableVectorOf<allowedTypes>.predicate,
+       ScalableVectorOfLength<allowedLengths>.predicate]>,
+  ScalableVectorOf<allowedTypes>.summary #
+  ScalableVectorOfLength<allowedLengths>.summary,
+  "::mlir::VectorType">;
+
 def AnyVector : VectorOf<[AnyType]>;
 
+def AnyFixedVector : FixedVectorOf<[AnyType]>;
+
+def AnyScalableVector : ScalableVectorOf<[AnyType]>;
+
 // Shaped types.
 
 def AnyShaped: ShapedContainerType<[AnyType], IsShapedTypePred, "shaped",
                                    "::mlir::ShapedType">;
 
 // Tensor types.
 
 // Any tensor type whose element type is from the given `allowedTypes` list

mlir/lib/Conversion/LLVMCommon/TypeConverter.cpp

 
 /// Convert an n-D vector type to an LLVM vector type via (n-1)-D array type
 /// when n > 1. For example, `vector<4 x f32>` remains as is while,
 /// `vector<4x8x16xf32>` converts to `!llvm.array<4xarray<8 x vector<16xf32>>>`.
 Type LLVMTypeConverter::convertVectorType(VectorType type) {
   auto elementType = convertType(type.getElementType());
   if (!elementType)
     return {};
-  Type vectorType = VectorType::get(type.getShape().back(), elementType);
+  Type vectorType = VectorType::get(type.getShape().back(), elementType,
+                                    type.getIsScalable());
   assert(LLVM::isCompatibleVectorType(vectorType) &&
          "expected vector type compatible with the LLVM dialect");
   auto shape = type.getShape();
   for (int i = shape.size() - 2; i >= 0; --i)
     vectorType = LLVM::LLVMArrayType::get(vectorType, shape[i]);
   return vectorType;
 }
 

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp

 #include "mlir/Transforms/DialectConversion.h"
 
 using namespace mlir;
 using namespace mlir::vector;
 
 // Helper to reduce vector type by one rank at front.
 static VectorType reducedVectorTypeFront(VectorType tp) {
   assert((tp.getRank() > 1) && "unlowerable vector type");
-  return VectorType::get(tp.getShape().drop_front(), tp.getElementType());
+  return VectorType::get(tp.getShape().drop_front(), tp.getElementType(),
+                         tp.getIsScalable());
 }
 
 // Helper to reduce vector type by *all* but one rank at back.
 static VectorType reducedVectorTypeBack(VectorType tp) {
   assert((tp.getRank() > 1) && "unlowerable vector type");
-  return VectorType::get(tp.getShape().take_back(), tp.getElementType());
+  return VectorType::get(tp.getShape().take_back(), tp.getElementType(),
+                         tp.getIsScalable());
 }
 
 // Helper that picks the proper sequence for inserting.
 static Value insertOne(ConversionPatternRewriter &rewriter,
                        LLVMTypeConverter &typeConverter, Location loc,
                        Value val1, Value val2, Type llvmType, int64_t rank,
                        int64_t pos) {
   if (rank == 1) {
     auto idxType = rewriter.getIndexType();

 static Value castDataPtr(ConversionPatternRewriter &rewriter, Location loc,
                          Value ptr, MemRefType memRefType, Type vt) {
   auto pType = LLVM::LLVMPointerType::get(vt, memRefType.getMemorySpaceAsInt());
   return rewriter.create<LLVM::BitcastOp>(loc, pType, ptr);
 }
 
 namespace {
 
+/// Trivial Vector to LLVM conversions
+using VectorScaleOpConversion =
+    OneToOneConvertToLLVMPattern<vector::VectorScaleOp, LLVM::vscale>;
+
 /// Conversion pattern for a vector.bitcast.
 class VectorBitCastOpConversion
     : public ConvertOpToLLVMPattern<vector::BitCastOp> {
 public:
   using ConvertOpToLLVMPattern<vector::BitCastOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
   matchAndRewrite(vector::BitCastOp bitCastOp, OpAdaptor adaptor,
void mlir::populateVectorToLLVMConversionPatterns(
   patterns.add<VectorFMAOpNDRewritePattern>(ctx);
   populateVectorInsertExtractStridedSliceTransforms(patterns);
   patterns.add<VectorReductionOpConversion>(converter, reassociateFPReductions);
   patterns
       .add<VectorBitCastOpConversion, VectorShuffleOpConversion,
            VectorExtractElementOpConversion, VectorExtractOpConversion,
            VectorFMAOp1DConversion, VectorInsertElementOpConversion,
            VectorInsertOpConversion, VectorPrintOpConversion,
-           VectorTypeCastOpConversion,
+           VectorTypeCastOpConversion, VectorScaleOpConversion,
            VectorLoadStoreConversion<vector::LoadOp, vector::LoadOpAdaptor>,
            VectorLoadStoreConversion<vector::MaskedLoadOp,
                                      vector::MaskedLoadOpAdaptor>,
            VectorLoadStoreConversion<vector::StoreOp, vector::StoreOpAdaptor>,
            VectorLoadStoreConversion<vector::MaskedStoreOp,
                                      vector::MaskedStoreOpAdaptor>,
            VectorGatherOpConversion, VectorScatterOpConversion,
            VectorExpandLoadOpConversion, VectorCompressStoreOpConversion>(

mlir/lib/Dialect/Arithmetic/IR/ArithmeticOps.cpp

 /// Return the type of the same shape (scalar, vector or tensor) containing i1.
 static Type getI1SameShape(Type type) {
   auto i1Type = IntegerType::get(type.getContext(), 1);
   if (auto tensorType = type.dyn_cast<RankedTensorType>())
     return RankedTensorType::get(tensorType.getShape(), i1Type);
   if (type.isa<UnrankedTensorType>())
     return UnrankedTensorType::get(i1Type);
   if (auto vectorType = type.dyn_cast<VectorType>())
-    return VectorType::get(vectorType.getShape(), i1Type);
+    return VectorType::get(vectorType.getShape(), i1Type,
+                           vectorType.getIsScalable());
   return i1Type;
 }
 
 //===----------------------------------------------------------------------===//
 // CmpIOp
 //===----------------------------------------------------------------------===//
 
 /// Compute `lhs` `pred` `rhs`, where `pred` is one of the known integer

mlir/lib/Dialect/ArmSVE/IR/ArmSVEDialect.cpp

 #include "llvm/ADT/TypeSwitch.h"
 
 using namespace mlir;
 using namespace arm_sve;
 
 #include "mlir/Dialect/ArmSVE/ArmSVEDialect.cpp.inc"
 
 static Type getI1SameShape(Type type);
-static void buildScalableCmpIOp(OpBuilder &build, OperationState &result,
-                                arith::CmpIPredicate predicate, Value lhs,
-                                Value rhs);
-static void buildScalableCmpFOp(OpBuilder &build, OperationState &result,
-                                arith::CmpFPredicate predicate, Value lhs,
-                                Value rhs);
 
 #define GET_OP_CLASSES
 #include "mlir/Dialect/ArmSVE/ArmSVE.cpp.inc"
 
 #define GET_TYPEDEF_CLASSES
 #include "mlir/Dialect/ArmSVE/ArmSVETypes.cpp.inc"
 
 void ArmSVEDialect::initialize() {
   addOperations<
 #define GET_OP_LIST
 #include "mlir/Dialect/ArmSVE/ArmSVE.cpp.inc"
       >();
-  addTypes<
-#define GET_TYPEDEF_LIST
-#include "mlir/Dialect/ArmSVE/ArmSVETypes.cpp.inc"
-      >();
-}
-
-//===----------------------------------------------------------------------===//
-// ScalableVectorType
-//===----------------------------------------------------------------------===//
-
-Type ArmSVEDialect::parseType(DialectAsmParser &parser) const {
-  llvm::SMLoc typeLoc = parser.getCurrentLocation();
-  {
-    Type genType;
-    auto parseResult = generatedTypeParser(parser, "vector", genType);
-    if (parseResult.hasValue())
-      return genType;
-  }
-  parser.emitError(typeLoc, "unknown type in ArmSVE dialect");
-  return Type();
-}
-
-void ArmSVEDialect::printType(Type type, DialectAsmPrinter &os) const {
-  if (failed(generatedTypePrinter(type, os)))
-    llvm_unreachable("unexpected 'arm_sve' type kind");
 }
 
 //===----------------------------------------------------------------------===//
 // ScalableVector versions of general helpers for comparison ops
 //===----------------------------------------------------------------------===//
 
 // Return the scalable vector of the same shape and containing i1.
 static Type getI1SameShape(Type type) {
   auto i1Type = IntegerType::get(type.getContext(), 1);
-  if (auto sVectorType = type.dyn_cast<ScalableVectorType>())
-    return ScalableVectorType::get(type.getContext(), sVectorType.getShape(),
-                                   i1Type);
+  if (auto sVectorType = type.dyn_cast<VectorType>())
+    return VectorType::get(sVectorType.getShape(), i1Type,
+                           /* isScalable = */ true);
   return nullptr;
 }
-
-//===----------------------------------------------------------------------===//
-// CmpFOp
-//===----------------------------------------------------------------------===//
-
-static void buildScalableCmpFOp(OpBuilder &build, OperationState &result,
-                                arith::CmpFPredicate predicate, Value lhs,
-                                Value rhs) {
-  result.addOperands({lhs, rhs});
-  result.types.push_back(getI1SameShape(lhs.getType()));
-  result.addAttribute(ScalableCmpFOp::getPredicateAttrName(),
-                      build.getI64IntegerAttr(static_cast<int64_t>(predicate)));
-}
-
-static void buildScalableCmpIOp(OpBuilder &build, OperationState &result,
-                                arith::CmpIPredicate predicate, Value lhs,
-                                Value rhs) {
-  result.addOperands({lhs, rhs});
-  result.types.push_back(getI1SameShape(lhs.getType()));
-  result.addAttribute(ScalableCmpIOp::getPredicateAttrName(),
-                      build.getI64IntegerAttr(static_cast<int64_t>(predicate)));
-}

mlir/lib/Dialect/ArmSVE/Transforms/LegalizeForLLVMExport.cpp

 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/StandardOps/IR/Ops.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/PatternMatch.h"
 
 using namespace mlir;
 using namespace mlir::arm_sve;
 
-// Extract an LLVM IR type from the LLVM IR dialect type.
-static Type unwrap(Type type) {
-  if (!type)
-    return nullptr;
-  auto *mlirContext = type.getContext();
-  if (!LLVM::isCompatibleType(type))
-    emitError(UnknownLoc::get(mlirContext),
-              "conversion resulted in a non-LLVM type");
-  return type;
-}
-
-static Optional<Type>
-convertScalableVectorTypeToLLVM(ScalableVectorType svType,
-                                LLVMTypeConverter &converter) {
-  auto elementType = unwrap(converter.convertType(svType.getElementType()));
-  if (!elementType)
-    return {};
-
-  auto sVectorType =
-      LLVM::LLVMScalableVectorType::get(elementType, svType.getShape().back());
-  return sVectorType;
-}
-
 template <typename OpTy>
 class ForwardOperands : public OpConversionPattern<OpTy> {
   using OpConversionPattern<OpTy>::OpConversionPattern;
 
   LogicalResult
   matchAndRewrite(OpTy op, typename OpTy::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const final {
     if (adaptor.getOperands().getTypes() == op->getOperands().getTypes())
public:
   matchAndRewrite(ReturnOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const final {
     rewriter.updateRootInPlace(
         op, [&]() { op->setOperands(adaptor.getOperands()); });
     return success();
   }
 };
 
-static Optional<Value> addUnrealizedCast(OpBuilder &builder,
-                                         ScalableVectorType svType,
-                                         ValueRange inputs, Location loc) {
-  if (inputs.size() != 1 ||
-      !inputs[0].getType().isa<LLVM::LLVMScalableVectorType>())
-    return Value();
-  return builder.create<UnrealizedConversionCastOp>(loc, svType, inputs)
-      .getResult(0);
-}
-
 using SdotOpLowering = OneToOneConvertToLLVMPattern<SdotOp, SdotIntrOp>;
 using SmmlaOpLowering = OneToOneConvertToLLVMPattern<SmmlaOp, SmmlaIntrOp>;
 using UdotOpLowering = OneToOneConvertToLLVMPattern<UdotOp, UdotIntrOp>;
 using UmmlaOpLowering = OneToOneConvertToLLVMPattern<UmmlaOp, UmmlaIntrOp>;
-using VectorScaleOpLowering =
-    OneToOneConvertToLLVMPattern<VectorScaleOp, VectorScaleIntrOp>;
 using ScalableMaskedAddIOpLowering =
     OneToOneConvertToLLVMPattern<ScalableMaskedAddIOp,
                                  ScalableMaskedAddIIntrOp>;
 using ScalableMaskedAddFOpLowering =
     OneToOneConvertToLLVMPattern<ScalableMaskedAddFOp,
                                  ScalableMaskedAddFIntrOp>;
 using ScalableMaskedSubIOpLowering =
     OneToOneConvertToLLVMPattern<ScalableMaskedSubIOp,
OneToOneConvertToLLVMPattern<ScalableMaskedSDivIOp,
                                  ScalableMaskedSDivIIntrOp>;
 using ScalableMaskedUDivIOpLowering =
     OneToOneConvertToLLVMPattern<ScalableMaskedUDivIOp,
                                  ScalableMaskedUDivIIntrOp>;
 using ScalableMaskedDivFOpLowering =
     OneToOneConvertToLLVMPattern<ScalableMaskedDivFOp,
                                  ScalableMaskedDivFIntrOp>;
 
-// Load operation is lowered to code that obtains a pointer to the indexed
-// element and loads from it.
-struct ScalableLoadOpLowering : public ConvertOpToLLVMPattern<ScalableLoadOp> {
-  using ConvertOpToLLVMPattern<ScalableLoadOp>::ConvertOpToLLVMPattern;
-
-  LogicalResult
-  matchAndRewrite(ScalableLoadOp loadOp, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    auto type = loadOp.getMemRefType();
-    if (!isConvertibleAndHasIdentityMaps(type))
-      return failure();
-
-    LLVMTypeConverter converter(loadOp.getContext());
-
-    auto resultType = loadOp.result().getType();
-    LLVM::LLVMPointerType llvmDataTypePtr;
-    if (resultType.isa<VectorType>()) {
-      llvmDataTypePtr =
-          LLVM::LLVMPointerType::get(resultType.cast<VectorType>());
-    } else if (resultType.isa<ScalableVectorType>()) {
-      llvmDataTypePtr = LLVM::LLVMPointerType::get(
-          convertScalableVectorTypeToLLVM(resultType.cast<ScalableVectorType>(),
-                                          converter)
-              .getValue());
-    }
-    Value dataPtr = getStridedElementPtr(loadOp.getLoc(), type, adaptor.base(),
-                                         adaptor.index(), rewriter);
-    Value bitCastedPtr = rewriter.create<LLVM::BitcastOp>(
-        loadOp.getLoc(), llvmDataTypePtr, dataPtr);
-    rewriter.replaceOpWithNewOp<LLVM::LoadOp>(loadOp, bitCastedPtr);
-    return success();
-  }
-};
-
-// Store operation is lowered to code that obtains a pointer to the indexed
-// element, and stores the given value to it.
-struct ScalableStoreOpLowering
-    : public ConvertOpToLLVMPattern<ScalableStoreOp> {
-  using ConvertOpToLLVMPattern<ScalableStoreOp>::ConvertOpToLLVMPattern;
-
-  LogicalResult
-  matchAndRewrite(ScalableStoreOp storeOp, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    auto type = storeOp.getMemRefType();
-    if (!isConvertibleAndHasIdentityMaps(type))
-      return failure();
-
-    LLVMTypeConverter converter(storeOp.getContext());
-
-    auto resultType = storeOp.value().getType();
-    LLVM::LLVMPointerType llvmDataTypePtr;
-    if (resultType.isa<VectorType>()) {
-      llvmDataTypePtr =
-          LLVM::LLVMPointerType::get(resultType.cast<VectorType>());
-    } else if (resultType.isa<ScalableVectorType>()) {
-      llvmDataTypePtr = LLVM::LLVMPointerType::get(
-          convertScalableVectorTypeToLLVM(resultType.cast<ScalableVectorType>(),
-                                          converter)
-              .getValue());
-    }
-    Value dataPtr = getStridedElementPtr(storeOp.getLoc(), type, adaptor.base(),
-                                         adaptor.index(), rewriter);
-    Value bitCastedPtr = rewriter.create<LLVM::BitcastOp>(
-        storeOp.getLoc(), llvmDataTypePtr, dataPtr);
-    rewriter.replaceOpWithNewOp<LLVM::StoreOp>(storeOp, adaptor.value(),
-                                               bitCastedPtr);
-    return success();
-  }
-};
-
-static void
-populateBasicSVEArithmeticExportPatterns(LLVMTypeConverter &converter,
-                                         OwningRewritePatternList &patterns) {
-  // clang-format off
-  patterns.add<OneToOneConvertToLLVMPattern<ScalableAddIOp, LLVM::AddOp>,
-               OneToOneConvertToLLVMPattern<ScalableAddFOp, LLVM::FAddOp>,
-               OneToOneConvertToLLVMPattern<ScalableSubIOp, LLVM::SubOp>,
-               OneToOneConvertToLLVMPattern<ScalableSubFOp, LLVM::FSubOp>,
-               OneToOneConvertToLLVMPattern<ScalableMulIOp, LLVM::MulOp>,
-               OneToOneConvertToLLVMPattern<ScalableMulFOp, LLVM::FMulOp>,
-               OneToOneConvertToLLVMPattern<ScalableSDivIOp, LLVM::SDivOp>,
-               OneToOneConvertToLLVMPattern<ScalableUDivIOp, LLVM::UDivOp>,
-               OneToOneConvertToLLVMPattern<ScalableDivFOp, LLVM::FDivOp>
-              >(converter);
-  // clang-format on
-}
-
-static void
-configureBasicSVEArithmeticLegalizations(LLVMConversionTarget &target) {
-  // clang-format off
-  target.addIllegalOp<ScalableAddIOp,
-                      ScalableAddFOp,
-                      ScalableSubIOp,
-                      ScalableSubFOp,
-                      ScalableMulIOp,
-                      ScalableMulFOp,
-                      ScalableSDivIOp,
-                      ScalableUDivIOp,
-                      ScalableDivFOp>();
-  // clang-format on
-}
-
-static void
-populateSVEMaskGenerationExportPatterns(LLVMTypeConverter &converter,
-                                        OwningRewritePatternList &patterns) {
-  // clang-format off
-  patterns.add<OneToOneConvertToLLVMPattern<ScalableCmpFOp, LLVM::FCmpOp>,
-               OneToOneConvertToLLVMPattern<ScalableCmpIOp, LLVM::ICmpOp>
-              >(converter);
-  // clang-format on
-}
-
-static void
-configureSVEMaskGenerationLegalizations(LLVMConversionTarget &target) {
-  // clang-format off
-  target.addIllegalOp<ScalableCmpFOp,
-                      ScalableCmpIOp>();
-  // clang-format on
-}
-
 /// Populate the given list with patterns that convert from ArmSVE to LLVM.
 void mlir::populateArmSVELegalizeForLLVMExportPatterns(
     LLVMTypeConverter &converter, OwningRewritePatternList &patterns) {
   // Populate conversion patterns
-  // Remove any ArmSVE-specific types from function signatures and results.
-  populateFuncOpTypeConversionPattern(patterns, converter);
-  converter.addConversion([&converter](ScalableVectorType svType) {
-    return convertScalableVectorTypeToLLVM(svType, converter);
-  });
-  converter.addSourceMaterialization(addUnrealizedCast);
 
   // clang-format off
   patterns.add<ForwardOperands<CallOp>,
                ForwardOperands<CallIndirectOp>,
                ForwardOperands<ReturnOp>>(converter,
                                           &converter.getContext());
   patterns.add<SdotOpLowering,
                SmmlaOpLowering,
                UdotOpLowering,
                UmmlaOpLowering,
-               VectorScaleOpLowering,
                ScalableMaskedAddIOpLowering,
                ScalableMaskedAddFOpLowering,
                ScalableMaskedSubIOpLowering,
                ScalableMaskedSubFOpLowering,
                ScalableMaskedMulIOpLowering,
                ScalableMaskedMulFOpLowering,
                ScalableMaskedSDivIOpLowering,
                ScalableMaskedUDivIOpLowering,
                ScalableMaskedDivFOpLowering>(converter);
-  patterns.add<ScalableLoadOpLowering,
-               ScalableStoreOpLowering>(converter);
   // clang-format on
-  populateBasicSVEArithmeticExportPatterns(converter, patterns);
-  populateSVEMaskGenerationExportPatterns(converter, patterns);
 }
 
 void mlir::configureArmSVELegalizeForExportTarget(
     LLVMConversionTarget &target) {
   // clang-format off
   target.addLegalOp<SdotIntrOp,
                     SmmlaIntrOp,
                     UdotIntrOp,
                     UmmlaIntrOp,
-                    VectorScaleIntrOp,
                     ScalableMaskedAddIIntrOp,
                     ScalableMaskedAddFIntrOp,
                     ScalableMaskedSubIIntrOp,
                     ScalableMaskedSubFIntrOp,
                     ScalableMaskedMulIIntrOp,
                     ScalableMaskedMulFIntrOp,
                     ScalableMaskedSDivIIntrOp,
                     ScalableMaskedUDivIIntrOp,
                     ScalableMaskedDivFIntrOp>();
   target.addIllegalOp<SdotOp,
                       SmmlaOp,
                       UdotOp,
                       UmmlaOp,
-                      VectorScaleOp,
                       ScalableMaskedAddIOp,
                       ScalableMaskedAddFOp,
                       ScalableMaskedSubIOp,
                       ScalableMaskedSubFOp,
                       ScalableMaskedMulIOp,
                       ScalableMaskedMulFOp,
                       ScalableMaskedSDivIOp,
                       ScalableMaskedUDivIOp,
-                      ScalableMaskedDivFOp,
-                      ScalableLoadOp,
-                      ScalableStoreOp>();
+                      ScalableMaskedDivFOp>();
   // clang-format on
-  auto hasScalableVectorType = [](TypeRange types) {
-    for (Type type : types)
-      if (type.isa<arm_sve::ScalableVectorType>())
-        return true;
-    return false;
-  };
-  target.addDynamicallyLegalOp<FuncOp>([hasScalableVectorType](FuncOp op) {
-    return !hasScalableVectorType(op.getType().getInputs()) &&
-           !hasScalableVectorType(op.getType().getResults());
-  });
-  target.addDynamicallyLegalOp<CallOp, CallIndirectOp, ReturnOp>(
-      [hasScalableVectorType](Operation *op) {
-        return !hasScalableVectorType(op->getOperandTypes()) &&
-               !hasScalableVectorType(op->getResultTypes());
-      });
-  configureBasicSVEArithmeticLegalizations(target);
-  configureSVEMaskGenerationLegalizations(target);
 }

mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp

static ParseResult parseCmpOp(OpAsmParser &parser, OperationState &result) {
 
   // The result type is either i1 or a vector type <? x i1> if the inputs are
   // vectors.
   Type resultType = IntegerType::get(builder.getContext(), 1);
   if (!isCompatibleType(type))
     return parser.emitError(trailingTypeLoc,
                             "expected LLVM dialect-compatible type");
   if (LLVM::isCompatibleVectorType(type)) {
-    if (type.isa<LLVM::LLVMScalableVectorType>()) {
-      resultType = LLVM::LLVMScalableVectorType::get(
-          resultType, LLVM::getVectorNumElements(type).getKnownMinValue());
+    if (LLVM::getIsScalableVectorType(type)) {
+      resultType = LLVM::getVectorType(
+          resultType, LLVM::getVectorNumElements(type).getKnownMinValue(),
+          /* isScalable = */ true);
     } else {
-      resultType = LLVM::getFixedVectorType(
-          resultType, LLVM::getVectorNumElements(type).getFixedValue());
+      resultType = LLVM::getVectorType(
+          resultType, LLVM::getVectorNumElements(type).getFixedValue(),
+          /* isScalable = */ false);
     }
   }
 
   result.addTypes({resultType});
   return success();
 }
 
 //===----------------------------------------------------------------------===//

mlir/lib/Dialect/LLVMIR/IR/LLVMTypes.cpp

return llvm::TypeSwitch<Type, Type>(type)
           [](auto ty) { return ty.getElementType(); })
       .Default([](Type) -> Type {
         llvm_unreachable("incompatible with LLVM vector type");
       });
 }
 
 llvm::ElementCount mlir::LLVM::getVectorNumElements(Type type) {
   return llvm::TypeSwitch<Type, llvm::ElementCount>(type)
-      .Case<LLVMFixedVectorType, VectorType>([](auto ty) {
+      .Case<VectorType>([](auto ty) {
+        if (ty.getIsScalable())
+          return llvm::ElementCount::getScalable(ty.getNumElements());
+        return llvm::ElementCount::getFixed(ty.getNumElements());
+      })
+      .Case<LLVMFixedVectorType>([](auto ty) {
         return llvm::ElementCount::getFixed(ty.getNumElements());
       })
       .Case([](LLVMScalableVectorType ty) {
         return llvm::ElementCount::getScalable(ty.getMinNumElements());
       })
       .Default([](Type) -> llvm::ElementCount {
         llvm_unreachable("incompatible with LLVM vector type");
       });
 }
 
+bool mlir::LLVM::getIsScalableVectorType(Type vectorType) {
+  assert(
+      (vectorType
+           .isa<LLVMFixedVectorType, LLVMScalableVectorType, VectorType>()) &&
+      "expected LLVM-compatible vector type");
+  return !vectorType.isa<LLVMFixedVectorType>() &&
+         (vectorType.isa<LLVMScalableVectorType>() ||
+          vectorType.cast<VectorType>().getIsScalable());
+}
+
+Type mlir::LLVM::getVectorType(Type elementType, unsigned numElements,
+                               bool isScalable) {
+  bool useLLVM = LLVMFixedVectorType::isValidElementType(elementType);
+  bool useBuiltIn = VectorType::isValidElementType(elementType);
+  (void)useBuiltIn;
+  assert((useLLVM ^ useBuiltIn) && "expected LLVM-compatible fixed-vector type "
+                                   "to be either builtin or LLVM dialect type");
+  if (useLLVM) {
+    if (isScalable)
+      return LLVMScalableVectorType::get(elementType, numElements);
+    return LLVMFixedVectorType::get(elementType, numElements);
+  }
+  return VectorType::get(numElements, elementType, isScalable);
+}
+
 Type mlir::LLVM::getFixedVectorType(Type elementType, unsigned numElements) {
   bool useLLVM = LLVMFixedVectorType::isValidElementType(elementType);
   bool useBuiltIn = VectorType::isValidElementType(elementType);
   (void)useBuiltIn;
   assert((useLLVM ^ useBuiltIn) && "expected LLVM-compatible fixed-vector type "
                                    "to be either builtin or LLVM dialect type");
   if (useLLVM)
     return LLVMFixedVectorType::get(elementType, numElements);
   return VectorType::get(numElements, elementType);
 }
 
+Type mlir::LLVM::getScalableVectorType(Type elementType, unsigned numElements) {
+  bool useLLVM = LLVMScalableVectorType::isValidElementType(elementType);
+  bool useBuiltIn = VectorType::isValidElementType(elementType);
+  (void)useBuiltIn;
+  assert((useLLVM ^ useBuiltIn) && "expected LLVM-compatible scalable-vector "
+                                   "type to be either builtin or LLVM dialect "
+                                   "type");
+  if (useLLVM)
+    return LLVMScalableVectorType::get(elementType, numElements);
+  return VectorType::get(numElements, elementType, /* isScalable =*/true);
+}
+
 llvm::TypeSize mlir::LLVM::getPrimitiveTypeSizeInBits(Type type) {
   assert(isCompatibleType(type) &&
          "expected a type compatible with the LLVM dialect");
 
   return llvm::TypeSwitch<Type, llvm::TypeSize>(type)
       .Case<BFloat16Type, Float16Type>(
           [](Type) { return llvm::TypeSize::Fixed(16); })
       .Case<Float32Type>([](Type) { return llvm::TypeSize::Fixed(32); })

mlir/lib/Dialect/StandardOps/IR/Ops.cpp

 // Return the type of the same shape (scalar, vector or tensor) containing i1.
 static Type getI1SameShape(Type type) {
   auto i1Type = IntegerType::get(type.getContext(), 1);
   if (auto tensorType = type.dyn_cast<RankedTensorType>())
     return RankedTensorType::get(tensorType.getShape(), i1Type);
   if (type.isa<UnrankedTensorType>())
     return UnrankedTensorType::get(i1Type);
   if (auto vectorType = type.dyn_cast<VectorType>())
-    return VectorType::get(vectorType.getShape(), i1Type);
+    return VectorType::get(vectorType.getShape(), i1Type,
+                           vectorType.getIsScalable());
   return i1Type;
 }
 
 //===----------------------------------------------------------------------===//
 // CondBranchOp
 //===----------------------------------------------------------------------===//
 
 namespace {

mlir/lib/IR/AsmPrinter.cpp

TypeSwitch<Type>(type)
         } else {
           os << '(';
           interleaveComma(results, [&](Type ty) { printType(ty); });
           os << ')';
         }
       })
       .Case<VectorType>([&](VectorType vectorTy) {
         os << "vector<";
+        if (vectorTy.getIsScalable())
+          os << "<";
         for (int64_t dim : vectorTy.getShape())
           os << dim << 'x';
         printType(vectorTy.getElementType());
         os << '>';
+        if (vectorTy.getIsScalable())
+          os << ">";
       })
       .Case<RankedTensorType>([&](RankedTensorType tensorTy) {
         os << "tensor<";
         for (int64_t dim : tensorTy.getShape()) {
           if (ShapedType::isDynamic(dim))
             os << '?';
           else
             os << dim;
           os << 'x';
         }
         printType(tensorTy.getElementType());
         // Only print the encoding attribute value if set.
         if (tensorTy.getEncoding()) {

mlir/lib/IR/BuiltinTypes.cpp

if (auto other = dyn_cast<UnrankedMemRefType>()) {
     MemRefType::Builder b(shape, elementType);
     b.setMemorySpace(other.getMemorySpace());
     return b;
   }
 
   if (isa<TensorType>())
     return RankedTensorType::get(shape, elementType);
 
   if (isa<VectorType>())
-    return VectorType::get(shape, elementType);
+    return VectorType::get(shape, elementType,
+                           cast<VectorType>().getIsScalable());
 
   llvm_unreachable("Unhandled ShapedType clone case");
 }
 
 ShapedType ShapedType::clone(ArrayRef<int64_t> shape) {
   if (auto other = dyn_cast<MemRefType>()) {
     MemRefType::Builder b(other);
     b.setShape(shape);
     return b;
   }
 
   if (auto other = dyn_cast<UnrankedMemRefType>()) {
     MemRefType::Builder b(shape, other.getElementType());
     b.setShape(shape);
     b.setMemorySpace(other.getMemorySpace());
     return b;
   }
 
   if (isa<TensorType>())
     return RankedTensorType::get(shape, getElementType());
 
   if (isa<VectorType>())
-    return VectorType::get(shape, getElementType());
+    return VectorType::get(shape, getElementType(),
+                           cast<VectorType>().getIsScalable());
 
   llvm_unreachable("Unhandled ShapedType clone case");
 }
 
 ShapedType ShapedType::clone(Type elementType) {
   if (auto other = dyn_cast<MemRefType>()) {
     MemRefType::Builder b(other);
     b.setElementType(elementType);
     return b;
   }
 
   if (auto other = dyn_cast<UnrankedMemRefType>()) {
     return UnrankedMemRefType::get(elementType, other.getMemorySpace());
   }
 
   if (isa<TensorType>()) {
     if (hasRank())
       return RankedTensorType::get(getShape(), elementType);
     return UnrankedTensorType::get(elementType);
   }
 
   if (isa<VectorType>())
-    return VectorType::get(getShape(), elementType);
+    return VectorType::get(getShape(), elementType,
+                           cast<VectorType>().getIsScalable());
 
   llvm_unreachable("Unhandled ShapedType clone hit");
 }
 
 Type ShapedType::getElementType() const {
   return TypeSwitch<Type, Type>(*this)
       .Case<VectorType, RankedTensorType, UnrankedTensorType, MemRefType,
             UnrankedMemRefType>([](auto ty) { return ty.getElementType(); });
bool ShapedType::hasStaticShape(ArrayRef<int64_t> shape) const {
   return hasStaticShape() && getShape() == shape;
 }
 
 //===----------------------------------------------------------------------===//
 // VectorType
 //===----------------------------------------------------------------------===//
 
 LogicalResult VectorType::verify(function_ref<InFlightDiagnostic()> emitError,
-                                 ArrayRef<int64_t> shape, Type elementType) {
+                                 ArrayRef<int64_t> shape, Type elementType,
+                                 bool isScalable) {
   if (!isValidElementType(elementType))
     return emitError()
            << "vector elements must be int/index/float type but got "
            << elementType;
 
   if (any_of(shape, [](int64_t i) { return i <= 0; }))
     return emitError()
            << "vector types must have positive constant sizes but got "

mlir/lib/Parser/TypeParser.cpp

 ///   static-dimension-list ::= (decimal-literal `x`)*
 ///
 VectorType Parser::parseVectorType() {
   consumeToken(Token::kw_vector);
 
   if (parseToken(Token::less, "expected '<' in vector type"))
     return nullptr;
 
+  bool isScalable = false;
+  if (consumeIf(Token::less))
+    isScalable = true;
+
   SmallVector<int64_t, 4> dimensions;
   if (parseDimensionListRanked(dimensions, /*allowDynamic=*/false))
     return nullptr;
   if (any_of(dimensions, [](int64_t i) { return i <= 0; }))
     return emitError(getToken().getLoc(),
                      "vector types must have positive constant sizes"),
            nullptr;
 
   // Parse the element type.
   auto typeLoc = getToken().getLoc();
   auto elementType = parseType();
   if (!elementType || parseToken(Token::greater, "expected '>' in vector type"))
     return nullptr;
+  if (isScalable &&
+      parseToken(Token::greater, "expected extra '>' in scalable vector type"))
+    return nullptr;
   if (!VectorType::isValidElementType(elementType))
     return emitError(typeLoc, "vector elements must be int/index/float type"),
            nullptr;
 
-  return VectorType::get(dimensions, elementType);
+  return VectorType::get(dimensions, elementType, isScalable);
 }
 
 /// Parse a dimension list of a tensor or memref type.  This populates the
 /// dimension list, using -1 for the `?` dimensions if `allowDynamic` is set and
 /// errors out on `?` otherwise.
 ///
 ///   dimension-list-ranked ::= (dimension `x`)*
 ///   dimension ::= `?` | decimal-literal
if (consumeIf(Token::question)) {
         dimensions.push_back((int64_t)dimension.getValue());
         consumeToken(Token::integer);
       }
     }
 
     // Make sure we have an 'x' or something like 'xbf32'.
     if (parseXInDimensionList())
       return failure();
   }
 
   return success();
 }
 
 /// Parse an 'x' token in a dimension list, handling the case where the x is
 /// juxtaposed with an element type, as in "xf32", leaving the "f32" as the next
 /// token.
 ParseResult Parser::parseXInDimensionList() {

mlir/lib/Target/LLVMIR/ModuleTranslation.cpp

if (auto funcAttr = attr.dyn_cast<FlatSymbolRefAttr>())
     return llvm::ConstantExpr::getBitCast(
         moduleTranslation.lookupFunction(funcAttr.getValue()), llvmType);
   if (auto splatAttr = attr.dyn_cast<SplatElementsAttr>()) {
     llvm::Type *elementType;
     uint64_t numElements;
     if (auto *arrayTy = dyn_cast<llvm::ArrayType>(llvmType)) {
       elementType = arrayTy->getElementType();
       numElements = arrayTy->getNumElements();
+    } else if (auto fVectorTy = dyn_cast<llvm::FixedVectorType>(llvmType)) {
+      elementType = fVectorTy->getElementType();
+      numElements = fVectorTy->getNumElements();
+    } else if (auto sVectorTy = dyn_cast<llvm::ScalableVectorType>(llvmType)) {
+      elementType = sVectorTy->getElementType();
+      numElements = sVectorTy->getMinNumElements();
     } else {
-      auto *vectorTy = cast<llvm::FixedVectorType>(llvmType);
-      elementType = vectorTy->getElementType();
-      numElements = vectorTy->getNumElements();
+      llvm_unreachable("unrecognized constant vector type");
     }
     // Splat value is a scalar. Extract it only if the element type is not
     // another sequence type. The recursion terminates because each step removes
     // one outer sequential type.
     bool elementTypeSequential =
         isa<llvm::ArrayType, llvm::VectorType>(elementType);
     llvm::Constant *child = getLLVMConstant(
         elementType,

mlir/lib/Target/LLVMIR/TypeToLLVM.cpp

llvm::Type *translate(LLVM::LLVMStructType type) {
     structType->setBody(subtypes, type.isPacked());
     return structType;
   }
 
   /// Translates the given built-in vector type compatible with LLVM.
   llvm::Type *translate(VectorType type) {
     assert(LLVM::isCompatibleVectorType(type) &&
            "expected compatible with LLVM vector type");
+    if (type.getIsScalable())
+      return llvm::ScalableVectorType::get(translateType(type.getElementType()),
+                                           type.getNumElements());
     return llvm::FixedVectorType::get(translateType(type.getElementType()),
                                       type.getNumElements());
   }
 
   /// Translates the given fixed-vector type.
   llvm::Type *translate(LLVM::LLVMFixedVectorType type) {
     return llvm::FixedVectorType::get(translateType(type.getElementType()),
                                       type.getNumElements());
   }
 
   /// Translates the given scalable-vector type.
   llvm::Type *translate(LLVM::LLVMScalableVectorType type) {
     return llvm::ScalableVectorType::get(translateType(type.getElementType()),
                                          type.getMinNumElements());

mlir/test/Dialect/ArmSVE/legalize-for-llvm.mlir

-// RUN: mlir-opt %s -convert-vector-to-llvm="enable-arm-sve" -convert-std-to-llvm | mlir-opt | FileCheck %s
+// RUN: mlir-opt %s -convert-vector-to-llvm="enable-arm-sve" -convert-std-to-llvm -reconcile-unrealized-casts | mlir-opt | FileCheck %s
 
-func @arm_sve_sdot(%a: !arm_sve.vector<16xi8>,
-                   %b: !arm_sve.vector<16xi8>,
-                   %c: !arm_sve.vector<4xi32>)
-    -> !arm_sve.vector<4xi32> {
+func @arm_sve_sdot(%a: vector<<16xi8>>,
+                   %b: vector<<16xi8>>,
+                   %c: vector<<4xi32>>)
+    -> vector<<4xi32>> {
   // CHECK: arm_sve.intr.sdot
   %0 = arm_sve.sdot %c, %a, %b :
-               !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
-  return %0 : !arm_sve.vector<4xi32>
+               vector<<16xi8>> to vector<<4xi32>>
+  return %0 : vector<<4xi32>>
 }
 
-func @arm_sve_smmla(%a: !arm_sve.vector<16xi8>,
-                    %b: !arm_sve.vector<16xi8>,
-                    %c: !arm_sve.vector<4xi32>)
-    -> !arm_sve.vector<4xi32> {
+func @arm_sve_smmla(%a: vector<<16xi8>>,
+                    %b: vector<<16xi8>>,
+                    %c: vector<<4xi32>>)
+    -> vector<<4xi32>> {
   // CHECK: arm_sve.intr.smmla
   %0 = arm_sve.smmla %c, %a, %b :
-               !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
-  return %0 : !arm_sve.vector<4xi32>
+               vector<<16xi8>> to vector<<4xi32>>
+  return %0 : vector<<4xi32>>
 }
 
-func @arm_sve_udot(%a: !arm_sve.vector<16xi8>,
-                   %b: !arm_sve.vector<16xi8>,
-                   %c: !arm_sve.vector<4xi32>)
-    -> !arm_sve.vector<4xi32> {
+func @arm_sve_udot(%a: vector<<16xi8>>,
+                   %b: vector<<16xi8>>,
+                   %c: vector<<4xi32>>)
+    -> vector<<4xi32>> {
   // CHECK: arm_sve.intr.udot
   %0 = arm_sve.udot %c, %a, %b :
-               !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
-  return %0 : !arm_sve.vector<4xi32>
+               vector<<16xi8>> to vector<<4xi32>>
+  return %0 : vector<<4xi32>>
 }
 
-func @arm_sve_ummla(%a: !arm_sve.vector<16xi8>,
-                    %b: !arm_sve.vector<16xi8>,
-                    %c: !arm_sve.vector<4xi32>)
-    -> !arm_sve.vector<4xi32> {
+func @arm_sve_ummla(%a: vector<<16xi8>>,
+                    %b: vector<<16xi8>>,
+                    %c: vector<<4xi32>>)
+    -> vector<<4xi32>> {
   // CHECK: arm_sve.intr.ummla
   %0 = arm_sve.ummla %c, %a, %b :
-               !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
-  return %0 : !arm_sve.vector<4xi32>
+               vector<<16xi8>> to vector<<4xi32>>
+  return %0 : vector<<4xi32>>
 }
 
-func @arm_sve_arithi(%a: !arm_sve.vector<4xi32>,
-                     %b: !arm_sve.vector<4xi32>,
-                     %c: !arm_sve.vector<4xi32>,
-                     %d: !arm_sve.vector<4xi32>,
-                     %e: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32> {
-  // CHECK: llvm.mul {{.*}}: !llvm.vec<? x 4 x i32>
-  %0 = arm_sve.muli %a, %b : !arm_sve.vector<4xi32>
-  // CHECK: llvm.add {{.*}}: !llvm.vec<? x 4 x i32>
-  %1 = arm_sve.addi %0, %c : !arm_sve.vector<4xi32>
-  // CHECK: llvm.sub {{.*}}: !llvm.vec<? x 4 x i32>
-  %2 = arm_sve.subi %1, %d : !arm_sve.vector<4xi32>
-  // CHECK: llvm.sdiv {{.*}}: !llvm.vec<? x 4 x i32>
-  %3 = arm_sve.divi_signed %2, %e : !arm_sve.vector<4xi32>
-  // CHECK: llvm.udiv {{.*}}: !llvm.vec<? x 4 x i32>
-  %4 = arm_sve.divi_unsigned %2, %e : !arm_sve.vector<4xi32>
-  return %4 : !arm_sve.vector<4xi32>
-}
-
-func @arm_sve_arithf(%a: !arm_sve.vector<4xf32>,
-                     %b: !arm_sve.vector<4xf32>,
-                     %c: !arm_sve.vector<4xf32>,
-                     %d: !arm_sve.vector<4xf32>,
-                     %e: !arm_sve.vector<4xf32>) -> !arm_sve.vector<4xf32> {
-  // CHECK: llvm.fmul {{.*}}: !llvm.vec<? x 4 x f32>
-  %0 = arm_sve.mulf %a, %b : !arm_sve.vector<4xf32>
-  // CHECK: llvm.fadd {{.*}}: !llvm.vec<? x 4 x f32>
-  %1 = arm_sve.addf %0, %c : !arm_sve.vector<4xf32>
-  // CHECK: llvm.fsub {{.*}}: !llvm.vec<? x 4 x f32>
-  %2 = arm_sve.subf %1, %d : !arm_sve.vector<4xf32>
-  // CHECK: llvm.fdiv {{.*}}: !llvm.vec<? x 4 x f32>
-  %3 = arm_sve.divf %2, %e : !arm_sve.vector<4xf32>
-  return %3 : !arm_sve.vector<4xf32>
-}
-
-func @arm_sve_arithi_masked(%a: !arm_sve.vector<4xi32>,
-                            %b: !arm_sve.vector<4xi32>,
-                            %c: !arm_sve.vector<4xi32>,
-                            %d: !arm_sve.vector<4xi32>,
-                            %e: !arm_sve.vector<4xi32>,
-                            %mask: !arm_sve.vector<4xi1>
-                            ) -> !arm_sve.vector<4xi32> {
-  // CHECK: arm_sve.intr.add{{.*}}: (!llvm.vec<? x 4 x i1>, !llvm.vec<? x 4 x i32>, !llvm.vec<? x 4 x i32>) -> !llvm.vec<? x 4 x i32>
-  %0 = arm_sve.masked.addi %mask, %a, %b : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xi32>
-  // CHECK: arm_sve.intr.sub{{.*}}: (!llvm.vec<? x 4 x i1>, !llvm.vec<? x 4 x i32>, !llvm.vec<? x 4 x i32>) -> !llvm.vec<? x 4 x i32>
-  %1 = arm_sve.masked.subi %mask, %0, %c : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xi32>
-  // CHECK: arm_sve.intr.mul{{.*}}: (!llvm.vec<? x 4 x i1>, !llvm.vec<? x 4 x i32>, !llvm.vec<? x 4 x i32>) -> !llvm.vec<? x 4 x i32>
-  %2 = arm_sve.masked.muli %mask, %1, %d : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xi32>
-  // CHECK: arm_sve.intr.sdiv{{.*}}: (!llvm.vec<? x 4 x i1>, !llvm.vec<? x 4 x i32>, !llvm.vec<? x 4 x i32>) -> !llvm.vec<? x 4 x i32>
-  %3 = arm_sve.masked.divi_signed %mask, %2, %e : !arm_sve.vector<4xi1>,
-                                                  !arm_sve.vector<4xi32>
-  // CHECK: arm_sve.intr.udiv{{.*}}: (!llvm.vec<? x 4 x i1>, !llvm.vec<? x 4 x i32>, !llvm.vec<? x 4 x i32>) -> !llvm.vec<? x 4 x i32>
-  %4 = arm_sve.masked.divi_unsigned %mask, %3, %e : !arm_sve.vector<4xi1>,
-                                                    !arm_sve.vector<4xi32>
-  return %4 : !arm_sve.vector<4xi32>
-}
-
-func @arm_sve_arithf_masked(%a: !arm_sve.vector<4xf32>,
-                            %b: !arm_sve.vector<4xf32>,
-                            %c: !arm_sve.vector<4xf32>,
-                            %d: !arm_sve.vector<4xf32>,
-                            %e: !arm_sve.vector<4xf32>,
-                            %mask: !arm_sve.vector<4xi1>
-                            ) -> !arm_sve.vector<4xf32> {
-  // CHECK: arm_sve.intr.fadd{{.*}}: (!llvm.vec<? x 4 x i1>, !llvm.vec<? x 4 x f32>, !llvm.vec<? x 4 x f32>) -> !llvm.vec<? x 4 x f32>
-  %0 = arm_sve.masked.addf %mask, %a, %b : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xf32>
-  // CHECK: arm_sve.intr.fsub{{.*}}: (!llvm.vec<? x 4 x i1>, !llvm.vec<? x 4 x f32>, !llvm.vec<? x 4 x f32>) -> !llvm.vec<? x 4 x f32>
-  %1 = arm_sve.masked.subf %mask, %0, %c : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xf32>
-  // CHECK: arm_sve.intr.fmul{{.*}}: (!llvm.vec<? x 4 x i1>, !llvm.vec<? x 4 x f32>, !llvm.vec<? x 4 x f32>) -> !llvm.vec<? x 4 x f32>
-  %2 = arm_sve.masked.mulf %mask, %1, %d : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xf32>
-  // CHECK: arm_sve.intr.fdiv{{.*}}: (!llvm.vec<? x 4 x i1>, !llvm.vec<? x 4 x f32>, !llvm.vec<? x 4 x f32>) -> !llvm.vec<? x 4 x f32>
-  %3 = arm_sve.masked.divf %mask, %2, %e : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xf32>
-  return %3 : !arm_sve.vector<4xf32>
-}
-
-func @arm_sve_mask_genf(%a: !arm_sve.vector<4xf32>,
-                        %b: !arm_sve.vector<4xf32>)
-                        -> !arm_sve.vector<4xi1> {
-  // CHECK: llvm.fcmp "oeq" {{.*}}: !llvm.vec<? x 4 x f32>
-  %0 = arm_sve.cmpf oeq, %a, %b : !arm_sve.vector<4xf32>
-  return %0 : !arm_sve.vector<4xi1>
-}
-
-func @arm_sve_mask_geni(%a: !arm_sve.vector<4xi32>,
-                        %b: !arm_sve.vector<4xi32>)
-                        -> !arm_sve.vector<4xi1> {
-  // CHECK: llvm.icmp "uge" {{.*}}: !llvm.vec<? x 4 x i32>
-  %0 = arm_sve.cmpi uge, %a, %b : !arm_sve.vector<4xi32>
-  return %0 : !arm_sve.vector<4xi1>
-}
-
-func @arm_sve_abs_diff(%a: !arm_sve.vector<4xi32>,
-                       %b: !arm_sve.vector<4xi32>)
-                       -> !arm_sve.vector<4xi32> {
-  // CHECK: llvm.sub {{.*}}: !llvm.vec<? x 4 x i32>
-  %z = arm_sve.subi %a, %a : !arm_sve.vector<4xi32>
-  // CHECK: llvm.icmp "sge" {{.*}}: !llvm.vec<? x 4 x i32>
-  %agb = arm_sve.cmpi sge, %a, %b : !arm_sve.vector<4xi32>
-  // CHECK: llvm.icmp "slt" {{.*}}: !llvm.vec<? x 4 x i32>
-  %bga = arm_sve.cmpi slt, %a, %b : !arm_sve.vector<4xi32>
-  // CHECK: "arm_sve.intr.sub"{{.*}}: (!llvm.vec<? x 4 x i1>, !llvm.vec<? x 4 x i32>, !llvm.vec<? x 4 x i32>) -> !llvm.vec<? x 4 x i32>
-  %0 = arm_sve.masked.subi %agb, %a, %b : !arm_sve.vector<4xi1>,
-                                          !arm_sve.vector<4xi32>
-  // CHECK: "arm_sve.intr.sub"{{.*}}: (!llvm.vec<? x 4 x i1>, !llvm.vec<? x 4 x i32>, !llvm.vec<? x 4 x i32>) -> !llvm.vec<? x 4 x i32>
-  %1 = arm_sve.masked.subi %bga, %b, %a : !arm_sve.vector<4xi1>,
-                                          !arm_sve.vector<4xi32>
-  // CHECK: "arm_sve.intr.add"{{.*}}: (!llvm.vec<? x 4 x i1>, !llvm.vec<? x 4 x i32>, !llvm.vec<? x 4 x i32>) -> !llvm.vec<? x 4 x i32>
-  %2 = arm_sve.masked.addi %agb, %z, %0 : !arm_sve.vector<4xi1>,
-                                          !arm_sve.vector<4xi32>
-  // CHECK: "arm_sve.intr.add"{{.*}}: (!llvm.vec<? x 4 x i1>, !llvm.vec<? x 4 x i32>, !llvm.vec<? x 4 x i32>) -> !llvm.vec<? x 4 x i32>
-  %3 = arm_sve.masked.addi %bga, %2, %1 : !arm_sve.vector<4xi1>,
-                                          !arm_sve.vector<4xi32>
-  return %3 : !arm_sve.vector<4xi32>
+func @arm_sve_arithi(%a: vector<<4xi32>>,
+                     %b: vector<<4xi32>>,
+                     %c: vector<<4xi32>>,
+                     %d: vector<<4xi32>>,
+                     %e: vector<<4xi32>>) -> vector<<4xi32>> {
+  // CHECK: llvm.mul {{.*}}: vector<<4xi32>>
+  %0 = arith.muli %a, %b : vector<<4xi32>>
+  // CHECK: llvm.sub {{.*}}: vector<<4xi32>>
+  %1 = arith.subi %0, %c : vector<<4xi32>>
+  // CHECK: llvm.sdiv {{.*}}: vector<<4xi32>>
+  %2 = arith.divsi %1, %d : vector<<4xi32>>
+  // CHECK: llvm.udiv {{.*}}: vector<<4xi32>>
+  %3 = arith.divui %1, %e : vector<<4xi32>>
+  // CHECK: llvm.add {{.*}}: vector<<4xi32>>
+  %4 = arith.addi %2, %3 : vector<<4xi32>>
+  return %4 : vector<<4xi32>>
+}
+
+func @arm_sve_arithf(%a: vector<<4xf32>>,
+                     %b: vector<<4xf32>>,
+                     %c: vector<<4xf32>>,
+                     %d: vector<<4xf32>>,
+                     %e: vector<<4xf32>>) -> vector<<4xf32>> {
+  // CHECK: llvm.fmul {{.*}}: vector<<4xf32>>
+  %0 = arith.mulf %a, %b : vector<<4xf32>>
+  // CHECK: llvm.fadd {{.*}}: vector<<4xf32>>
+  %1 = arith.addf %0, %c : vector<<4xf32>>
+  // CHECK: llvm.fsub {{.*}}: vector<<4xf32>>
+  %2 = arith.subf %1, %d : vector<<4xf32>>
+  // CHECK: llvm.fdiv {{.*}}: vector<<4xf32>>
+  %3 = arith.divf %2, %e : vector<<4xf32>>
+  return %3 : vector<<4xf32>>
+}
+
+func @arm_sve_arithi_masked(%a: vector<<4xi32>>,
+                            %b: vector<<4xi32>>,
+                            %c: vector<<4xi32>>,
+                            %d: vector<<4xi32>>,
+                            %e: vector<<4xi32>>,
+                            %mask: vector<<4xi1>>
+                            ) -> vector<<4xi32>> {
+  // CHECK: arm_sve.intr.add{{.*}}: (vector<<4xi1>>, vector<<4xi32>>, vector<<4xi32>>) -> vector<<4xi32>>
+  %0 = arm_sve.masked.addi %mask, %a, %b : vector<<4xi1>>,
+                                           vector<<4xi32>>
+  // CHECK: arm_sve.intr.sub{{.*}}: (vector<<4xi1>>, vector<<4xi32>>, vector<<4xi32>>) -> vector<<4xi32>>
+  %1 = arm_sve.masked.subi %mask, %0, %c : vector<<4xi1>>,
+                                           vector<<4xi32>>
+  // CHECK: arm_sve.intr.mul{{.*}}: (vector<<4xi1>>, vector<<4xi32>>, vector<<4xi32>>) -> vector<<4xi32>>
+  %2 = arm_sve.masked.muli %mask, %1, %d : vector<<4xi1>>,
+                                           vector<<4xi32>>
+  // CHECK: arm_sve.intr.sdiv{{.*}}: (vector<<4xi1>>, vector<<4xi32>>, vector<<4xi32>>) -> vector<<4xi32>>
+  %3 = arm_sve.masked.divi_signed %mask, %2, %e : vector<<4xi1>>,
+                                                  vector<<4xi32>>
+  // CHECK: arm_sve.intr.udiv{{.*}}: (vector<<4xi1>>, vector<<4xi32>>, vector<<4xi32>>) -> vector<<4xi32>>
+  %4 = arm_sve.masked.divi_unsigned %mask, %3, %e : vector<<4xi1>>,
+                                                    vector<<4xi32>>
+  return %4 : vector<<4xi32>>
+}
+
+func @arm_sve_arithf_masked(%a: vector<<4xf32>>,
+                            %b: vector<<4xf32>>,
+                            %c: vector<<4xf32>>,
+                            %d: vector<<4xf32>>,
+                            %e: vector<<4xf32>>,
+                            %mask: vector<<4xi1>>
+                            ) -> vector<<4xf32>> {
+  // CHECK: arm_sve.intr.fadd{{.*}}: (vector<<4xi1>>, vector<<4xf32>>, vector<<4xf32>>) -> vector<<4xf32>>
+  %0 = arm_sve.masked.addf %mask, %a, %b : vector<<4xi1>>,
+                                           vector<<4xf32>>
+  // CHECK: arm_sve.intr.fsub{{.*}}: (vector<<4xi1>>, vector<<4xf32>>, vector<<4xf32>>) -> vector<<4xf32>>
+  %1 = arm_sve.masked.subf %mask, %0, %c : vector<<4xi1>>,
+                                           vector<<4xf32>>
+  // CHECK: arm_sve.intr.fmul{{.*}}: (vector<<4xi1>>, vector<<4xf32>>, vector<<4xf32>>) -> vector<<4xf32>>
+  %2 = arm_sve.masked.mulf %mask, %1, %d : vector<<4xi1>>,
+                                           vector<<4xf32>>
+  // CHECK: arm_sve.intr.fdiv{{.*}}: (vector<<4xi1>>, vector<<4xf32>>, vector<<4xf32>>) -> vector<<4xf32>>
+  %3 = arm_sve.masked.divf %mask, %2, %e : vector<<4xi1>>,
+                                           vector<<4xf32>>
+  return %3 : vector<<4xf32>>
+}
+
+func @arm_sve_mask_genf(%a: vector<<4xf32>>,
+                        %b: vector<<4xf32>>)
+                        -> vector<<4xi1>> {
+  // CHECK: llvm.fcmp "oeq" {{.*}}: vector<<4xf32>>
+  %0 = arith.cmpf oeq, %a, %b : vector<<4xf32>>
+  return %0 : vector<<4xi1>>
+}
+
+func @arm_sve_mask_geni(%a: vector<<4xi32>>,
+                        %b: vector<<4xi32>>)
+                        -> vector<<4xi1>> {
+  // CHECK: llvm.icmp "uge" {{.*}}: vector<<4xi32>>
+  %0 = arith.cmpi uge, %a, %b : vector<<4xi32>>
+  return %0 : vector<<4xi1>>
+}
+
+func @arm_sve_abs_diff(%a: vector<<4xi32>>,
+                       %b: vector<<4xi32>>)
+                       -> vector<<4xi32>> {
+  // CHECK: llvm.mlir.constant(dense<0> : vector<<4xi32>>) : vector<<4xi32>>
+  %z = arith.subi %a, %a : vector<<4xi32>>
+  // CHECK: llvm.icmp "sge" {{.*}}: vector<<4xi32>>
+  %agb = arith.cmpi sge, %a, %b : vector<<4xi32>>
+  // CHECK: llvm.icmp "slt" {{.*}}: vector<<4xi32>>
+  %bga = arith.cmpi slt, %a, %b : vector<<4xi32>>
+  // CHECK: "arm_sve.intr.sub"{{.*}}: (vector<<4xi1>>, vector<<4xi32>>, vector<<4xi32>>) -> vector<<4xi32>>
+  %0 = arm_sve.masked.subi %agb, %a, %b : vector<<4xi1>>,
+                                          vector<<4xi32>>
+  // CHECK: "arm_sve.intr.sub"{{.*}}: (vector<<4xi1>>, vector<<4xi32>>, vector<<4xi32>>) -> vector<<4xi32>>
+  %1 = arm_sve.masked.subi %bga, %b, %a : vector<<4xi1>>,
+                                          vector<<4xi32>>
+  // CHECK: "arm_sve.intr.add"{{.*}}: (vector<<4xi1>>, vector<<4xi32>>, vector<<4xi32>>) -> vector<<4xi32>>
+  %2 = arm_sve.masked.addi %agb, %z, %0 : vector<<4xi1>>,
+                                          vector<<4xi32>>
+  // CHECK: "arm_sve.intr.add"{{.*}}: (vector<<4xi1>>, vector<<4xi32>>, vector<<4xi32>>) -> vector<<4xi32>>
+  %3 = arm_sve.masked.addi %bga, %2, %1 : vector<<4xi1>>,
+                                          vector<<4xi32>>
+  return %3 : vector<<4xi32>>
 }
 
 func @get_vector_scale() -> index {
-  // CHECK: arm_sve.vscale
-  %0 = arm_sve.vector_scale : index
+  // CHECK: llvm.intr.vscale
+  %0 = vector.vector_scale : index
   return %0 : index
 }

mlir/test/Dialect/ArmSVE/memcpy.mlir

-// RUN: mlir-opt %s -convert-vector-to-llvm="enable-arm-sve" | mlir-opt | FileCheck %s
-
-// CHECK: memcopy([[SRC:%arg[0-9]+]]: memref<?xf32>, [[DST:%arg[0-9]+]]
-func @memcopy(%src : memref<?xf32>, %dst : memref<?xf32>, %size : index) {
-  %c0 = arith.constant 0 : index
-  %c4 = arith.constant 4 : index
-  %vs = arm_sve.vector_scale : index
-  %step = arith.muli %c4, %vs : index
-
-  // CHECK: [[SRCMRS:%[0-9]+]] = builtin.unrealized_conversion_cast [[SRC]] : memref<?xf32> to !llvm.struct<(ptr<f32>
-  // CHECK: [[DSTMRS:%[0-9]+]] = builtin.unrealized_conversion_cast [[DST]] : memref<?xf32> to !llvm.struct<(ptr<f32>
-  // CHECK: scf.for [[LOOPIDX:%arg[0-9]+]] = {{.*}}
-  scf.for %i0 = %c0 to %size step %step {
-    // CHECK: [[SRCIDX:%[0-9]+]] = builtin.unrealized_conversion_cast [[LOOPIDX]] : index to i64
-    // CHECK: [[SRCMEM:%[0-9]+]] = llvm.extractvalue [[SRCMRS]][1] : !llvm.struct<(ptr<f32>
-    // CHECK-NEXT: [[SRCPTR:%[0-9]+]] = llvm.getelementptr [[SRCMEM]]{{.}}[[SRCIDX]]{{.}} : (!llvm.ptr<f32>, i64) -> !llvm.ptr<f32>
-    // CHECK-NEXT: [[SRCVPTR:%[0-9]+]] = llvm.bitcast [[SRCPTR]] : !llvm.ptr<f32> to !llvm.ptr<vec<? x 4 x f32>>
-    // CHECK-NEXT: [[LDVAL:%[0-9]+]] = llvm.load [[SRCVPTR]] : !llvm.ptr<vec<? x 4 x f32>>
-    %0 = arm_sve.load %src[%i0] : !arm_sve.vector<4xf32> from memref<?xf32>
-    // CHECK: [[DSTMEM:%[0-9]+]] = llvm.extractvalue [[DSTMRS]][1] : !llvm.struct<(ptr<f32>
-    // CHECK-NEXT: [[DSTPTR:%[0-9]+]] = llvm.getelementptr [[DSTMEM]]{{.}}[[SRCIDX]]{{.}} : (!llvm.ptr<f32>, i64) -> !llvm.ptr<f32>
-    // CHECK-NEXT: [[DSTVPTR:%[0-9]+]] = llvm.bitcast [[DSTPTR]] : !llvm.ptr<f32> to !llvm.ptr<vec<? x 4 x f32>>
-    // CHECK-NEXT: llvm.store [[LDVAL]], [[DSTVPTR]] : !llvm.ptr<vec<? x 4 x f32>>
-    arm_sve.store %0, %dst[%i0] : !arm_sve.vector<4xf32> to memref<?xf32>
-  }
-
-  return
-}

mlir/test/Dialect/ArmSVE/roundtrip.mlir

 // RUN: mlir-opt -verify-diagnostics %s | mlir-opt | FileCheck %s
 
-func @arm_sve_sdot(%a: !arm_sve.vector<16xi8>,
-                   %b: !arm_sve.vector<16xi8>,
-                   %c: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32> {
-  // CHECK: arm_sve.sdot {{.*}}: !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32
+func @arm_sve_sdot(%a: vector<<16xi8>>,
+                   %b: vector<<16xi8>>,
+                   %c: vector<<4xi32>>) -> vector<<4xi32>> {
+  // CHECK: arm_sve.sdot {{.*}}: vector<<16xi8>> to vector<<4xi32
   %0 = arm_sve.sdot %c, %a, %b :
-             !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
-  return %0 : !arm_sve.vector<4xi32>
+             vector<<16xi8>> to vector<<4xi32>>
+  return %0 : vector<<4xi32>>
 }
 
-func @arm_sve_smmla(%a: !arm_sve.vector<16xi8>,
-                    %b: !arm_sve.vector<16xi8>,
-                    %c: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32> {
-  // CHECK: arm_sve.smmla {{.*}}: !arm_sve.vector<16xi8> to !arm_sve.vector<4xi3
+func @arm_sve_smmla(%a: vector<<16xi8>>,
+                    %b: vector<<16xi8>>,
+                    %c: vector<<4xi32>>) -> vector<<4xi32>> {
+  // CHECK: arm_sve.smmla {{.*}}: vector<<16xi8>> to vector<<4xi3
   %0 = arm_sve.smmla %c, %a, %b :
-             !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
-  return %0 : !arm_sve.vector<4xi32>
+             vector<<16xi8>> to vector<<4xi32>>
+  return %0 : vector<<4xi32>>
 }
 
-func @arm_sve_udot(%a: !arm_sve.vector<16xi8>,
-                   %b: !arm_sve.vector<16xi8>,
-                   %c: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32> {
-  // CHECK: arm_sve.udot {{.*}}: !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32
+func @arm_sve_udot(%a: vector<<16xi8>>,
+                   %b: vector<<16xi8>>,
+                   %c: vector<<4xi32>>) -> vector<<4xi32>> {
+  // CHECK: arm_sve.udot {{.*}}: vector<<16xi8>> to vector<<4xi32
   %0 = arm_sve.udot %c, %a, %b :
-             !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
-  return %0 : !arm_sve.vector<4xi32>
+             vector<<16xi8>> to vector<<4xi32>>
+  return %0 : vector<<4xi32>>
 }
 
-func @arm_sve_ummla(%a: !arm_sve.vector<16xi8>,
-                    %b: !arm_sve.vector<16xi8>,
-                    %c: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32> {
-  // CHECK: arm_sve.ummla {{.*}}: !arm_sve.vector<16xi8> to !arm_sve.vector<4xi3
+func @arm_sve_ummla(%a: vector<<16xi8>>,
+                    %b: vector<<16xi8>>,
+                    %c: vector<<4xi32>>) -> vector<<4xi32>> {
+  // CHECK: arm_sve.ummla {{.*}}: vector<<16xi8>> to vector<<4xi3
   %0 = arm_sve.ummla %c, %a, %b :
-             !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
-  return %0 : !arm_sve.vector<4xi32>
+             vector<<16xi8>> to vector<<4xi32>>
+  return %0 : vector<<4xi32>>
 }
 
-func @arm_sve_arithi(%a: !arm_sve.vector<4xi32>,
-                     %b: !arm_sve.vector<4xi32>,
-                     %c: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32> {
-  // CHECK: arm_sve.muli {{.*}}: !arm_sve.vector<4xi32>
-  %0 = arm_sve.muli %a, %b : !arm_sve.vector<4xi32>
-  // CHECK: arm_sve.addi {{.*}}: !arm_sve.vector<4xi32>
-  %1 = arm_sve.addi %0, %c : !arm_sve.vector<4xi32>
-  return %1 : !arm_sve.vector<4xi32>
-}
-
-func @arm_sve_arithf(%a: !arm_sve.vector<4xf32>,
-                     %b: !arm_sve.vector<4xf32>,
-                     %c: !arm_sve.vector<4xf32>) -> !arm_sve.vector<4xf32> {
-  // CHECK: arm_sve.mulf {{.*}}: !arm_sve.vector<4xf32>
-  %0 = arm_sve.mulf %a, %b : !arm_sve.vector<4xf32>
-  // CHECK: arm_sve.addf {{.*}}: !arm_sve.vector<4xf32>
-  %1 = arm_sve.addf %0, %c : !arm_sve.vector<4xf32>
-  return %1 : !arm_sve.vector<4xf32>
-}
-
-func @arm_sve_masked_arithi(%a: !arm_sve.vector<4xi32>,
-                            %b: !arm_sve.vector<4xi32>,
-                            %c: !arm_sve.vector<4xi32>,
-                            %d: !arm_sve.vector<4xi32>,
-                            %e: !arm_sve.vector<4xi32>,
-                            %mask: !arm_sve.vector<4xi1>)
-                            -> !arm_sve.vector<4xi32> {
-  // CHECK: arm_sve.masked.muli {{.*}}: !arm_sve.vector<4xi1>, !arm_sve.vector
-  %0 = arm_sve.masked.muli %mask, %a, %b : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xi32>
-  // CHECK: arm_sve.masked.addi {{.*}}: !arm_sve.vector<4xi1>, !arm_sve.vector
-  %1 = arm_sve.masked.addi %mask, %0, %c : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xi32>
-  // CHECK: arm_sve.masked.subi {{.*}}: !arm_sve.vector<4xi1>, !arm_sve.vector
-  %2 = arm_sve.masked.subi %mask, %1, %d : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xi32>
+func @arm_sve_arithi(%a: vector<<4xi32>>,
+                     %b: vector<<4xi32>>,
+                     %c: vector<<4xi32>>) -> vector<<4xi32>> {
+  // CHECK: muli {{.*}}: vector<<4xi32>>
+  %0 = arith.muli %a, %b : vector<<4xi32>>
+  // CHECK: addi {{.*}}: vector<<4xi32>>
+  %1 = arith.addi %0, %c : vector<<4xi32>>
+  return %1 : vector<<4xi32>>
+}
+
+func @arm_sve_arithf(%a: vector<<4xf32>>,
+                     %b: vector<<4xf32>>,
+                     %c: vector<<4xf32>>) -> vector<<4xf32>> {
+  // CHECK: mulf {{.*}}: vector<<4xf32>>
+  %0 = arith.mulf %a, %b : vector<<4xf32>>
+  // CHECK: addf {{.*}}: vector<<4xf32>>
+  %1 = arith.addf %0, %c : vector<<4xf32>>
+  return %1 : vector<<4xf32>>
+}
+
+func @arm_sve_masked_arithi(%a: vector<<4xi32>>,
+                            %b: vector<<4xi32>>,
+                            %c: vector<<4xi32>>,
+                            %d: vector<<4xi32>>,
+                            %e: vector<<4xi32>>,
+                            %mask: vector<<4xi1>>)
+                            -> vector<<4xi32>> {
+  // CHECK: arm_sve.masked.muli {{.*}}: vector<<4xi1>>, vector<
+  %0 = arm_sve.masked.muli %mask, %a, %b : vector<<4xi1>>,
+                                           vector<<4xi32>>
+  // CHECK: arm_sve.masked.addi {{.*}}: vector<<4xi1>>, vector<
+  %1 = arm_sve.masked.addi %mask, %0, %c : vector<<4xi1>>,
+                                           vector<<4xi32>>
+  // CHECK: arm_sve.masked.subi {{.*}}: vector<<4xi1>>, vector<
+  %2 = arm_sve.masked.subi %mask, %1, %d : vector<<4xi1>>,
+                                           vector<<4xi32>>
   // CHECK: arm_sve.masked.divi_signed
-  %3 = arm_sve.masked.divi_signed %mask, %2, %e : !arm_sve.vector<4xi1>,
-                                                  !arm_sve.vector<4xi32>
+  %3 = arm_sve.masked.divi_signed %mask, %2, %e : vector<<4xi1>>,
+                                                  vector<<4xi32>>
   // CHECK: arm_sve.masked.divi_unsigned
-  %4 = arm_sve.masked.divi_unsigned %mask, %3, %e : !arm_sve.vector<4xi1>,
-                                                    !arm_sve.vector<4xi32>
-  return %2 : !arm_sve.vector<4xi32>
-}
-
-func @arm_sve_masked_arithf(%a: !arm_sve.vector<4xf32>,
-                            %b: !arm_sve.vector<4xf32>,
-                            %c: !arm_sve.vector<4xf32>,
-                            %d: !arm_sve.vector<4xf32>,
-                            %e: !arm_sve.vector<4xf32>,
-                            %mask: !arm_sve.vector<4xi1>)
-                            -> !arm_sve.vector<4xf32> {
-  // CHECK: arm_sve.masked.mulf {{.*}}: !arm_sve.vector<4xi1>, !arm_sve.vector
-  %0 = arm_sve.masked.mulf %mask, %a, %b : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xf32>
-  // CHECK: arm_sve.masked.addf {{.*}}: !arm_sve.vector<4xi1>, !arm_sve.vector
-  %1 = arm_sve.masked.addf %mask, %0, %c : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xf32>
-  // CHECK: arm_sve.masked.subf {{.*}}: !arm_sve.vector<4xi1>, !arm_sve.vector
-  %2 = arm_sve.masked.subf %mask, %1, %d : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xf32>
-  // CHECK: arm_sve.masked.divf {{.*}}: !arm_sve.vector<4xi1>, !arm_sve.vector
-  %3 = arm_sve.masked.divf %mask, %2, %e : !arm_sve.vector<4xi1>,
-                                           !arm_sve.vector<4xf32>
-  return %3 : !arm_sve.vector<4xf32>
-}
-
-func @arm_sve_mask_genf(%a: !arm_sve.vector<4xf32>,
-                        %b: !arm_sve.vector<4xf32>)
-                        -> !arm_sve.vector<4xi1> {
-  // CHECK: arm_sve.cmpf oeq, {{.*}}: !arm_sve.vector<4xf32>
-  %0 = arm_sve.cmpf oeq, %a, %b : !arm_sve.vector<4xf32>
-  return %0 : !arm_sve.vector<4xi1>
-}
-
-func @arm_sve_mask_geni(%a: !arm_sve.vector<4xi32>,
-                        %b: !arm_sve.vector<4xi32>)
-                        -> !arm_sve.vector<4xi1> {
-  // CHECK: arm_sve.cmpi uge, {{.*}}: !arm_sve.vector<4xi32>
-  %0 = arm_sve.cmpi uge, %a, %b : !arm_sve.vector<4xi32>
-  return %0 : !arm_sve.vector<4xi1>
+  %4 = arm_sve.masked.divi_unsigned %mask, %3, %e : vector<<4xi1>>,
+                                                    vector<<4xi32>>
+  return %2 : vector<<4xi32>>
+}
+
+func @arm_sve_masked_arithf(%a: vector<<4xf32>>,
+                            %b: vector<<4xf32>>,
+                            %c: vector<<4xf32>>,
+                            %d: vector<<4xf32>>,
+                            %e: vector<<4xf32>>,
+                            %mask: vector<<4xi1>>)
+                            -> vector<<4xf32>> {
+  // CHECK: arm_sve.masked.mulf {{.*}}: vector<<4xi1>>, vector<
+  %0 = arm_sve.masked.mulf %mask, %a, %b : vector<<4xi1>>,
+                                           vector<<4xf32>>
+  // CHECK: arm_sve.masked.addf {{.*}}: vector<<4xi1>>, vector<
+  %1 = arm_sve.masked.addf %mask, %0, %c : vector<<4xi1>>,
+                                           vector<<4xf32>>
+  // CHECK: arm_sve.masked.subf {{.*}}: vector<<4xi1>>, vector<
+  %2 = arm_sve.masked.subf %mask, %1, %d : vector<<4xi1>>,
+                                           vector<<4xf32>>
+  // CHECK: arm_sve.masked.divf {{.*}}: vector<<4xi1>>, vector<
+  %3 = arm_sve.masked.divf %mask, %2, %e : vector<<4xi1>>,
+                                           vector<<4xf32>>
+  return %3 : vector<<4xf32>>
+}
+
+func @arm_sve_mask_genf(%a: vector<<4xf32>>,
+                        %b: vector<<4xf32>>)
+                        -> vector<<4xi1>> {
+  // CHECK: cmpf oeq, {{.*}}: vector<<4xf32>>
+  %0 = arith.cmpf oeq, %a, %b : vector<<4xf32>>
+  return %0 : vector<<4xi1>>
+}
+
+func @arm_sve_mask_geni(%a: vector<<4xi32>>,
+                        %b: vector<<4xi32>>)
+                        -> vector<<4xi1>> {
+  // CHECK: cmpi uge, {{.*}}: vector<<4xi32>>
+  %0 = arith.cmpi uge, %a, %b : vector<<4xi32>>
+  return %0 : vector<<4xi1>>
 }
 
-func @arm_sve_memory(%v: !arm_sve.vector<4xi32>,
+func @arm_sve_memory(%v: vector<<4xi32>>,
                      %m: memref<?xi32>)
-                     -> !arm_sve.vector<4xi32> {
+                     -> vector<<4xi32>> {
   %c0 = arith.constant 0 : index
-  // CHECK: arm_sve.load {{.*}}: !arm_sve.vector<4xi32> from memref<?xi32>
-  %0 = arm_sve.load %m[%c0] : !arm_sve.vector<4xi32> from memref<?xi32>
-  // CHECK: arm_sve.store {{.*}}: !arm_sve.vector<4xi32> to memref<?xi32>
-  arm_sve.store %v, %m[%c0] : !arm_sve.vector<4xi32> to memref<?xi32>
-  return %0 : !arm_sve.vector<4xi32>
+  // CHECK: vector.load {{.*}}: memref<?xi32>, vector<<4xi32>>
+  %0 = vector.load %m[%c0] : memref<?xi32>, vector<<4xi32>>
+  // CHECK: vector.store {{.*}}: memref<?xi32>, vector<<4xi32>>
+  vector.store %v, %m[%c0] : memref<?xi32>, vector<<4xi32>>
+  return %0 : vector<<4xi32>>
 }
 
 func @get_vector_scale() -> index {
-  // CHECK: arm_sve.vector_scale : index
-  %0 = arm_sve.vector_scale : index
+  // CHECK: vector.vector_scale : index
+  %0 = vector.vector_scale : index
   return %0 : index
 }

mlir/test/Dialect/ArmSVE/scalable-memcpy.mlir

+// RUN: mlir-opt %s -convert-vector-to-llvm | mlir-opt | FileCheck %s
+
+// CHECK: scalable_memcopy([[SRC:%arg[0-9]+]]: memref<?xf32>, [[DST:%arg[0-9]+]]
+func @scalable_memcopy(%src : memref<?xf32>, %dst : memref<?xf32>, %size : index) {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %vs = vector.vector_scale : index
+  %step = arith.muli %c4, %vs : index
+  // CHECK: [[SRCMRS:%[0-9]+]] = builtin.unrealized_conversion_cast [[SRC]] : memref<?xf32> to !llvm.struct<(ptr<f32>
+  // CHECK: [[DSTMRS:%[0-9]+]] = builtin.unrealized_conversion_cast [[DST]] : memref<?xf32> to !llvm.struct<(ptr<f32>
+  // CHECK: scf.for [[LOOPIDX:%arg[0-9]+]] = {{.*}}
+  scf.for %i0 = %c0 to %size step %step {
+    // CHECK: [[DATAIDX:%[0-9]+]] = builtin.unrealized_conversion_cast [[LOOPIDX]] : index to i64
+    // CHECK: [[SRCMEM:%[0-9]+]] = llvm.extractvalue [[SRCMRS]][1] : !llvm.struct<(ptr<f32>
+    // CHECK-NEXT: [[SRCPTR:%[0-9]+]] = llvm.getelementptr [[SRCMEM]]{{.}}[[DATAIDX]]{{.}} : (!llvm.ptr<f32>, i64) -> !llvm.ptr<f32>
+    // CHECK-NEXT: [[SRCVPTR:%[0-9]+]] = llvm.bitcast [[SRCPTR]] : !llvm.ptr<f32> to !llvm.ptr<vector<<4xf32>>>
+    // CHECK-NEXT: [[LDVAL:%[0-9]+]] = llvm.load [[SRCVPTR]]{{.*}}: !llvm.ptr<vector<<4xf32>>>
+    %0 = vector.load %src[%i0] : memref<?xf32>, vector<<4xf32>>
+    // CHECK: [[DSTMEM:%[0-9]+]] = llvm.extractvalue [[DSTMRS]][1] : !llvm.struct<(ptr<f32>
+    // CHECK-NEXT: [[DSTPTR:%[0-9]+]] = llvm.getelementptr [[DSTMEM]]{{.}}[[DATAIDX]]{{.}} : (!llvm.ptr<f32>, i64) -> !llvm.ptr<f32>
+    // CHECK-NEXT: [[DSTVPTR:%[0-9]+]] = llvm.bitcast [[DSTPTR]] : !llvm.ptr<f32> to !llvm.ptr<vector<<4xf32>>>
+    // CHECK-NEXT: llvm.store [[LDVAL]], [[DSTVPTR]]{{.*}}: !llvm.ptr<vector<<4xf32>>>
+    vector.store %0, %dst[%i0] : memref<?xf32>, vector<<4xf32>>
+  }
+
+  return
+}

mlir/test/Target/LLVMIR/arm-sve.mlir

 // RUN: mlir-translate --mlir-to-llvmir %s | FileCheck %s
 
 // CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_sdot
-llvm.func @arm_sve_sdot(%arg0: !llvm.vec<?x16 x i8>,
-                        %arg1: !llvm.vec<?x16 x i8>,
-                        %arg2: !llvm.vec<?x4 x i32>)
-                        -> !llvm.vec<?x4 x i32> {
+llvm.func @arm_sve_sdot(%arg0: vector<<16xi8>>,
+                        %arg1: vector<<16xi8>>,
+                        %arg2: vector<<4xi32>>)
+                        -> vector<<4xi32>> {
   // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.sdot.nxv4i32(<vscale x 4
   %0 = "arm_sve.intr.sdot"(%arg2, %arg0, %arg1) :
-    (!llvm.vec<?x4 x i32>, !llvm.vec<?x16 x i8>, !llvm.vec<?x16 x i8>)
-        -> !llvm.vec<?x4 x i32>
-  llvm.return %0 : !llvm.vec<?x4 x i32>
+    (vector<<4xi32>>, vector<<16xi8>>, vector<<16xi8>>)
+        -> vector<<4xi32>>
+  llvm.return %0 : vector<<4xi32>>
 }
 
 // CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_smmla
-llvm.func @arm_sve_smmla(%arg0: !llvm.vec<?x16 x i8>,
-                         %arg1: !llvm.vec<?x16 x i8>,
-                         %arg2: !llvm.vec<?x4 x i32>)
-                         -> !llvm.vec<?x4 x i32> {
+llvm.func @arm_sve_smmla(%arg0: vector<<16xi8>>,
+                         %arg1: vector<<16xi8>>,
+                         %arg2: vector<<4xi32>>)
+                         -> vector<<4xi32>> {
   // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.smmla.nxv4i32(<vscale x 4
   %0 = "arm_sve.intr.smmla"(%arg2, %arg0, %arg1) :
-    (!llvm.vec<?x4 x i32>, !llvm.vec<?x16 x i8>, !llvm.vec<?x16 x i8>)
-        -> !llvm.vec<?x4 x i32>
-  llvm.return %0 : !llvm.vec<?x4 x i32>
+    (vector<<4xi32>>, vector<<16xi8>>, vector<<16xi8>>)
+        -> vector<<4xi32>>
+  llvm.return %0 : vector<<4xi32>>
 }
 
 // CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_udot
-llvm.func @arm_sve_udot(%arg0: !llvm.vec<?x16 x i8>,
-                        %arg1: !llvm.vec<?x16 x i8>,
-                        %arg2: !llvm.vec<?x4 x i32>)
-                        -> !llvm.vec<?x4 x i32> {
+llvm.func @arm_sve_udot(%arg0: vector<<16xi8>>,
+                        %arg1: vector<<16xi8>>,
+                        %arg2: vector<<4xi32>>)
+                        -> vector<<4xi32>> {
   // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.udot.nxv4i32(<vscale x 4
   %0 = "arm_sve.intr.udot"(%arg2, %arg0, %arg1) :
-    (!llvm.vec<?x4 x i32>, !llvm.vec<?x16 x i8>, !llvm.vec<?x16 x i8>)
-        -> !llvm.vec<?x4 x i32>
-  llvm.return %0 : !llvm.vec<?x4 x i32>
+    (vector<<4xi32>>, vector<<16xi8>>, vector<<16xi8>>)
+        -> vector<<4xi32>>
+  llvm.return %0 : vector<<4xi32>>
 }
 
 // CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_ummla
-llvm.func @arm_sve_ummla(%arg0: !llvm.vec<?x16 x i8>,
-                         %arg1: !llvm.vec<?x16 x i8>,
-                         %arg2: !llvm.vec<?x4 x i32>)
-                         -> !llvm.vec<?x4 x i32> {
+llvm.func @arm_sve_ummla(%arg0: vector<<16xi8>>,
+                         %arg1: vector<<16xi8>>,
+                         %arg2: vector<<4xi32>>)
+                         -> vector<<4xi32>> {
   // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.ummla.nxv4i32(<vscale x 4
   %0 = "arm_sve.intr.ummla"(%arg2, %arg0, %arg1) :
-    (!llvm.vec<?x4 x i32>, !llvm.vec<?x16 x i8>, !llvm.vec<?x16 x i8>)
-        -> !llvm.vec<?x4 x i32>
-  llvm.return %0 : !llvm.vec<?x4 x i32>
+    (vector<<4xi32>>, vector<<16xi8>>, vector<<16xi8>>)
+        -> vector<<4xi32>>
+  llvm.return %0 : vector<<4xi32>>
 }
 
 // CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_arithi
-llvm.func @arm_sve_arithi(%arg0: !llvm.vec<? x 4 x i32>,
-                          %arg1: !llvm.vec<? x 4 x i32>,
-                          %arg2: !llvm.vec<? x 4 x i32>)
-                          -> !llvm.vec<? x 4 x i32> {
+llvm.func @arm_sve_arithi(%arg0: vector<<4xi32>>,
+                          %arg1: vector<<4xi32>>,
+                          %arg2: vector<<4xi32>>)
+                          -> vector<<4xi32>> {
   // CHECK: mul <vscale x 4 x i32>
-  %0 = llvm.mul %arg0, %arg1 : !llvm.vec<? x 4 x i32>
+  %0 = llvm.mul %arg0, %arg1 : vector<<4xi32>>
   // CHECK: add <vscale x 4 x i32>
-  %1 = llvm.add %0, %arg2 : !llvm.vec<? x 4 x i32>
-  llvm.return %1 : !llvm.vec<? x 4 x i32>
+  %1 = llvm.add %0, %arg2 : vector<<4xi32>>
+  llvm.return %1 : vector<<4xi32>>
 }
 
 // CHECK-LABEL: define <vscale x 4 x float> @arm_sve_arithf
-llvm.func @arm_sve_arithf(%arg0: !llvm.vec<? x 4 x f32>,
-                          %arg1: !llvm.vec<? x 4 x f32>,
-                          %arg2: !llvm.vec<? x 4 x f32>)
-                          -> !llvm.vec<? x 4 x f32> {
+llvm.func @arm_sve_arithf(%arg0: vector<<4xf32>>,
+                          %arg1: vector<<4xf32>>,
+                          %arg2: vector<<4xf32>>)
+                          -> vector<<4xf32>> {
   // CHECK: fmul <vscale x 4 x float>
-  %0 = llvm.fmul %arg0, %arg1 : !llvm.vec<? x 4 x f32>
+  %0 = llvm.fmul %arg0, %arg1 : vector<<4xf32>>
   // CHECK: fadd <vscale x 4 x float>
-  %1 = llvm.fadd %0, %arg2 : !llvm.vec<? x 4 x f32>
-  llvm.return %1 : !llvm.vec<? x 4 x f32>
+  %1 = llvm.fadd %0, %arg2 : vector<<4xf32>>
+  llvm.return %1 : vector<<4xf32>>
 }
 
 // CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_arithi_masked
-llvm.func @arm_sve_arithi_masked(%arg0: !llvm.vec<? x 4 x i32>,
-                                 %arg1: !llvm.vec<? x 4 x i32>,
-                                 %arg2: !llvm.vec<? x 4 x i32>,
-                                 %arg3: !llvm.vec<? x 4 x i32>,
-                                 %arg4: !llvm.vec<? x 4 x i32>,
-                                 %arg5: !llvm.vec<? x 4 x i1>)
-                                 -> !llvm.vec<? x 4 x i32> {
+llvm.func @arm_sve_arithi_masked(%arg0: vector<<4xi32>>,
+                                 %arg1: vector<<4xi32>>,
+                                 %arg2: vector<<4xi32>>,
+                                 %arg3: vector<<4xi32>>,
+                                 %arg4: vector<<4xi32>>,
+                                 %arg5: vector<<4xi1>>)
+                                 -> vector<<4xi32>> {
   // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.add.nxv4i32
-  %0 = "arm_sve.intr.add"(%arg5, %arg0, %arg1) : (!llvm.vec<? x 4 x i1>,
-                                                  !llvm.vec<? x 4 x i32>,
-                                                  !llvm.vec<? x 4 x i32>)
-                                                  -> !llvm.vec<? x 4 x i32>
+  %0 = "arm_sve.intr.add"(%arg5, %arg0, %arg1) : (vector<<4xi1>>,
+                                                  vector<<4xi32>>,
+                                                  vector<<4xi32>>)
+                                                  -> vector<<4xi32>>
   // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.sub.nxv4i32
-  %1 = "arm_sve.intr.sub"(%arg5, %0, %arg1) : (!llvm.vec<? x 4 x i1>,
-                                               !llvm.vec<? x 4 x i32>,
-                                               !llvm.vec<? x 4 x i32>)
-                                               -> !llvm.vec<? x 4 x i32>
+  %1 = "arm_sve.intr.sub"(%arg5, %0, %arg1) : (vector<<4xi1>>,
+                                               vector<<4xi32>>,
+                                               vector<<4xi32>>)
+                                               -> vector<<4xi32>>
   // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.mul.nxv4i32
-  %2 = "arm_sve.intr.mul"(%arg5, %1, %arg3) : (!llvm.vec<? x 4 x i1>,
-                                               !llvm.vec<? x 4 x i32>,
-                                               !llvm.vec<? x 4 x i32>)
-                                               -> !llvm.vec<? x 4 x i32>
+  %2 = "arm_sve.intr.mul"(%arg5, %1, %arg3) : (vector<<4xi1>>,
+                                               vector<<4xi32>>,
+                                               vector<<4xi32>>)
+                                               -> vector<<4xi32>>
   // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.sdiv.nxv4i32
-  %3 = "arm_sve.intr.sdiv"(%arg5, %2, %arg4) : (!llvm.vec<? x 4 x i1>,
-                                               !llvm.vec<? x 4 x i32>,
-                                               !llvm.vec<? x 4 x i32>)
-                                               -> !llvm.vec<? x 4 x i32>
+  %3 = "arm_sve.intr.sdiv"(%arg5, %2, %arg4) : (vector<<4xi1>>,
+                                               vector<<4xi32>>,
+                                               vector<<4xi32>>)
+                                               -> vector<<4xi32>>
   // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.udiv.nxv4i32
-  %4 = "arm_sve.intr.udiv"(%arg5, %3, %arg4) : (!llvm.vec<? x 4 x i1>,
-                                               !llvm.vec<? x 4 x i32>,
-                                               !llvm.vec<? x 4 x i32>)
-                                               -> !llvm.vec<? x 4 x i32>
-  llvm.return %4 : !llvm.vec<? x 4 x i32>
+  %4 = "arm_sve.intr.udiv"(%arg5, %3, %arg4) : (vector<<4xi1>>,
+                                               vector<<4xi32>>,
+                                               vector<<4xi32>>)
+                                               -> vector<<4xi32>>
+  llvm.return %4 : vector<<4xi32>>
 }
 
 // CHECK-LABEL: define <vscale x 4 x float> @arm_sve_arithf_masked
-llvm.func @arm_sve_arithf_masked(%arg0: !llvm.vec<? x 4 x f32>,
-                                 %arg1: !llvm.vec<? x 4 x f32>,
-                                 %arg2: !llvm.vec<? x 4 x f32>,
-                                 %arg3: !llvm.vec<? x 4 x f32>,
-                                 %arg4: !llvm.vec<? x 4 x f32>,
-                                 %arg5: !llvm.vec<? x 4 x i1>)
-                                 -> !llvm.vec<? x 4 x f32> {
+llvm.func @arm_sve_arithf_masked(%arg0: vector<<4xf32>>,
+                                 %arg1: vector<<4xf32>>,
+                                 %arg2: vector<<4xf32>>,
+                                 %arg3: vector<<4xf32>>,
+                                 %arg4: vector<<4xf32>>,
+                                 %arg5: vector<<4xi1>>)
+                                 -> vector<<4xf32>> {
   // CHECK: call <vscale x 4 x float> @llvm.aarch64.sve.fadd.nxv4f32
-  %0 = "arm_sve.intr.fadd"(%arg5, %arg0, %arg1) : (!llvm.vec<? x 4 x i1>,
-                                                   !llvm.vec<? x 4 x f32>,
-                                                   !llvm.vec<? x 4 x f32>)
-                                                   -> !llvm.vec<? x 4 x f32>
+  %0 = "arm_sve.intr.fadd"(%arg5, %arg0, %arg1) : (vector<<4xi1>>,
+                                                   vector<<4xf32>>,
+                                                   vector<<4xf32>>)
+                                                   -> vector<<4xf32>>
   // CHECK: call <vscale x 4 x float> @llvm.aarch64.sve.fsub.nxv4f32
-  %1 = "arm_sve.intr.fsub"(%arg5, %0, %arg2) : (!llvm.vec<? x 4 x i1>,
-                                                !llvm.vec<? x 4 x f32>,
-                                                !llvm.vec<? x 4 x f32>)
-                                                -> !llvm.vec<? x 4 x f32>
+  %1 = "arm_sve.intr.fsub"(%arg5, %0, %arg2) : (vector<<4xi1>>,
+                                                vector<<4xf32>>,
+                                                vector<<4xf32>>)
+                                                -> vector<<4xf32>>
   // CHECK: call <vscale x 4 x float> @llvm.aarch64.sve.fmul.nxv4f32
-  %2 = "arm_sve.intr.fmul"(%arg5, %1, %arg3) : (!llvm.vec<? x 4 x i1>,
-                                                !llvm.vec<? x 4 x f32>,
-                                                !llvm.vec<? x 4 x f32>)
-                                                -> !llvm.vec<? x 4 x f32>
+  %2 = "arm_sve.intr.fmul"(%arg5, %1, %arg3) : (vector<<4xi1>>,
+                                                vector<<4xf32>>,
+                                                vector<<4xf32>>)
+                                                -> vector<<4xf32>>
   // CHECK: call <vscale x 4 x float> @llvm.aarch64.sve.fdiv.nxv4f32
-  %3 = "arm_sve.intr.fdiv"(%arg5, %2, %arg4) : (!llvm.vec<? x 4 x i1>,
-                                                !llvm.vec<? x 4 x f32>,
-                                                !llvm.vec<? x 4 x f32>)
-                                                -> !llvm.vec<? x 4 x f32>
-  llvm.return %3 : !llvm.vec<? x 4 x f32>
+  %3 = "arm_sve.intr.fdiv"(%arg5, %2, %arg4) : (vector<<4xi1>>,
+                                                vector<<4xf32>>,
+                                                vector<<4xf32>>)
+                                                -> vector<<4xf32>>
+  llvm.return %3 : vector<<4xf32>>
 }
 
 // CHECK-LABEL: define <vscale x 4 x i1> @arm_sve_mask_genf
-llvm.func @arm_sve_mask_genf(%arg0: !llvm.vec<? x 4 x f32>,
-                             %arg1: !llvm.vec<? x 4 x f32>)
-                             -> !llvm.vec<? x 4 x i1> {
+llvm.func @arm_sve_mask_genf(%arg0: vector<<4xf32>>,
+                             %arg1: vector<<4xf32>>)
+                             -> vector<<4xi1>> {
   // CHECK: fcmp oeq <vscale x 4 x float>
-  %0 = llvm.fcmp "oeq" %arg0, %arg1 : !llvm.vec<? x 4 x f32>
-  llvm.return %0 : !llvm.vec<? x 4 x i1>
+  %0 = llvm.fcmp "oeq" %arg0, %arg1 : vector<<4xf32>>
+  llvm.return %0 : vector<<4xi1>>
 }
 
 // CHECK-LABEL: define <vscale x 4 x i1> @arm_sve_mask_geni
-llvm.func @arm_sve_mask_geni(%arg0: !llvm.vec<? x 4 x i32>,
-                             %arg1: !llvm.vec<? x 4 x i32>)
-                             -> !llvm.vec<? x 4 x i1> {
+llvm.func @arm_sve_mask_geni(%arg0: vector<<4xi32>>,
+                             %arg1: vector<<4xi32>>)
+                             -> vector<<4xi1>> {
   // CHECK: icmp uge <vscale x 4 x i32>
-  %0 = llvm.icmp "uge" %arg0, %arg1 : !llvm.vec<? x 4 x i32>
-  llvm.return %0 : !llvm.vec<? x 4 x i1>
+  %0 = llvm.icmp "uge" %arg0, %arg1 : vector<<4xi32>>
+  llvm.return %0 : vector<<4xi1>>
 }
 
 // CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_abs_diff
-llvm.func @arm_sve_abs_diff(%arg0: !llvm.vec<? x 4 x i32>,
-                            %arg1: !llvm.vec<? x 4 x i32>)
-                            -> !llvm.vec<? x 4 x i32> {
+llvm.func @arm_sve_abs_diff(%arg0: vector<<4xi32>>,
+                            %arg1: vector<<4xi32>>)
+                            -> vector<<4xi32>> {
   // CHECK: sub <vscale x 4 x i32>
-  %0 = llvm.sub %arg0, %arg0  : !llvm.vec<? x 4 x i32>
+  %0 = llvm.sub %arg0, %arg0  : vector<<4xi32>>
   // CHECK: icmp sge <vscale x 4 x i32>
-  %1 = llvm.icmp "sge" %arg0, %arg1 : !llvm.vec<? x 4 x i32>
+  %1 = llvm.icmp "sge" %arg0, %arg1 : vector<<4xi32>>
   // CHECK: icmp slt <vscale x 4 x i32>
-  %2 = llvm.icmp "slt" %arg0, %arg1 : !llvm.vec<? x 4 x i32>
+  %2 = llvm.icmp "slt" %arg0, %arg1 : vector<<4xi32>>
   // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.sub.nxv4i32
-  %3 = "arm_sve.intr.sub"(%1, %arg0, %arg1) : (!llvm.vec<? x 4 x i1>,
-                                               !llvm.vec<? x 4 x i32>,
-                                               !llvm.vec<? x 4 x i32>)
-                                               -> !llvm.vec<? x 4 x i32>
+  %3 = "arm_sve.intr.sub"(%1, %arg0, %arg1) : (vector<<4xi1>>,
+                                               vector<<4xi32>>,
+                                               vector<<4xi32>>)
+                                               -> vector<<4xi32>>
   // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.sub.nxv4i32
-  %4 = "arm_sve.intr.sub"(%2, %arg1, %arg0) : (!llvm.vec<? x 4 x i1>,
-                                               !llvm.vec<? x 4 x i32>,
-                                               !llvm.vec<? x 4 x i32>)
-                                               -> !llvm.vec<? x 4 x i32>
+  %4 = "arm_sve.intr.sub"(%2, %arg1, %arg0) : (vector<<4xi1>>,
+                                               vector<<4xi32>>,
+                                               vector<<4xi32>>)
+                                               -> vector<<4xi32>>
   // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.add.nxv4i32
-  %5 = "arm_sve.intr.add"(%1, %0, %3) : (!llvm.vec<? x 4 x i1>,
-                                         !llvm.vec<? x 4 x i32>,
-                                         !llvm.vec<? x 4 x i32>)
-                                         -> !llvm.vec<? x 4 x i32>
+  %5 = "arm_sve.intr.add"(%1, %0, %3) : (vector<<4xi1>>,
+                                         vector<<4xi32>>,
+                                         vector<<4xi32>>)
+                                         -> vector<<4xi32>>
   // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.add.nxv4i32
-  %6 = "arm_sve.intr.add"(%2, %5, %4) : (!llvm.vec<? x 4 x i1>,
-                                         !llvm.vec<? x 4 x i32>,
-                                         !llvm.vec<? x 4 x i32>)
-                                         -> !llvm.vec<? x 4 x i32>
-  llvm.return %6 : !llvm.vec<? x 4 x i32>
+  %6 = "arm_sve.intr.add"(%2, %5, %4) : (vector<<4xi1>>,
+                                         vector<<4xi32>>,
+                                         vector<<4xi32>>)
+                                         -> vector<<4xi32>>
+  llvm.return %6 : vector<<4xi32>>
 }
 
 // CHECK-LABEL: define void @memcopy
 llvm.func @memcopy(%arg0: !llvm.ptr<f32>, %arg1: !llvm.ptr<f32>,
                    %arg2: i64, %arg3: i64, %arg4: i64,
                    %arg5: !llvm.ptr<f32>, %arg6: !llvm.ptr<f32>,
                    %arg7: i64, %arg8: i64, %arg9: i64,
                    %arg10: i64) {
%10 = llvm.insertvalue %arg8, %9[3, 0] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
                                                          array<1 x i64>,
                                                          array<1 x i64>)>
   %11 = llvm.insertvalue %arg9, %10[4, 0] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
                                                          array<1 x i64>,
                                                          array<1 x i64>)>
   %12 = llvm.mlir.constant(0 : index) : i64
   %13 = llvm.mlir.constant(4 : index) : i64
   // CHECK: [[VL:%[0-9]+]] = call i64 @llvm.vscale.i64()
-  %14 = "arm_sve.vscale"() : () -> i64
+  %14 = "llvm.intr.vscale"() : () -> i64
   // CHECK: mul i64 [[VL]], 4
   %15 = llvm.mul %14, %13  : i64
   llvm.br ^bb1(%12 : i64)
 ^bb1(%16: i64):
   %17 = llvm.icmp "slt" %16, %arg10 : i64
   llvm.cond_br %17, ^bb2, ^bb3
 ^bb2:
   // CHECK: extractvalue { float*, float*, i64, [1 x i64], [1 x i64] }
   %18 = llvm.extractvalue %5[1] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
                                                 array<1 x i64>,
                                                 array<1 x i64>)>
   // CHECK: etelementptr float, float*
   %19 = llvm.getelementptr %18[%16] : (!llvm.ptr<f32>, i64) -> !llvm.ptr<f32>
   // CHECK: bitcast float* %{{[0-9]+}} to <vscale x 4 x float>*
-  %20 = llvm.bitcast %19 : !llvm.ptr<f32> to !llvm.ptr<vec<? x 4 x f32>>
+  %20 = llvm.bitcast %19 : !llvm.ptr<f32> to !llvm.ptr<vector<<4xf32>>>
   // CHECK: load <vscale x 4 x float>, <vscale x 4 x float>*
-  %21 = llvm.load %20 : !llvm.ptr<vec<? x 4 x f32>>
+  %21 = llvm.load %20 : !llvm.ptr<vector<<4xf32>>>
   // CHECK: extractvalue { float*, float*, i64, [1 x i64], [1 x i64] }
   %22 = llvm.extractvalue %11[1] : !llvm.struct<(ptr<f32>, ptr<f32>, i64,
                                                  array<1 x i64>,
                                                  array<1 x i64>)>
   // CHECK: getelementptr float, float* %32
   %23 = llvm.getelementptr %22[%16] : (!llvm.ptr<f32>, i64) -> !llvm.ptr<f32>
   // CHECK: bitcast float* %33 to <vscale x 4 x float>*
-  %24 = llvm.bitcast %23 : !llvm.ptr<f32> to !llvm.ptr<vec<? x 4 x f32>>
+  %24 = llvm.bitcast %23 : !llvm.ptr<f32> to !llvm.ptr<vector<<4xf32>>>
   // CHECK: store <vscale x 4 x float> %{{[0-9]+}}, <vscale x 4 x float>* %{{[0-9]+}}
-  llvm.store %21, %24 : !llvm.ptr<vec<? x 4 x f32>>
+  llvm.store %21, %24 : !llvm.ptr<vector<<4xf32>>>
   %25 = llvm.add %16, %15  : i64
   llvm.br ^bb1(%25 : i64)
 ^bb3:
   llvm.return
 }
 
 // CHECK-LABEL: define i64 @get_vector_scale()
 llvm.func @get_vector_scale() -> i64 {
   // CHECK: call i64 @llvm.vscale.i64()
-  %0 = "arm_sve.vscale"() : () -> i64
+  %0 = "llvm.intr.vscale"() : () -> i64
   llvm.return %0 : i64
 }