[mlir][ArmSVE][RFC] Add an ArmSVE dialect

[jsetoain]

Summary

This revision starts an Arm-specific ArmSVE dialect discussed in the discourse RFC thread:

https://llvm.discourse.group/t/rfc-vector-dialects-neon-and-sve/2284

[rriddle]

What discourse thread was this discussed in?

[rriddle]

mlir/include/mlir/Dialect/LLVMIR/LLVMSVE.td

54 ↗

(On Diff #307827)

Remove commented out code.

mlir/include/mlir/Dialect/SVE/SVE.td

25 ↗

(On Diff #307827)

SVE seems way too generic of a name for some seemingly specific to arm.

71 ↗

(On Diff #307827)

I'd prefer you resolve these todos right now.

155 ↗

(On Diff #307827)

I dont particularly like the idea of hard coding a higher level dialect specifically on what it would target in LLVM.

[nicolasvasilache]

In D92172#2418298, @rriddle wrote:

What discourse thread was this discussed in?

https://llvm.discourse.group/t/rfc-vector-dialects-neon-and-sve/2284 but manually creating http links and synchronizing at sub minute granularity is hard :)

@jsetoain could you please update your commit message with this link?

[rriddle]

In D92172#2418310, @nicolasvasilache wrote:

In D92172#2418298, @rriddle wrote:

What discourse thread was this discussed in?

https://llvm.discourse.group/t/rfc-vector-dialects-neon-and-sve/2284 but manually creating http links and synchronizing at sub minute granularity is hard :)

@jsetoain could you please update your commit message with this link?

Thanks, I just assumed I missed the thread that discussed this.

[jsetoain]

Change to comply with header guard naming style

[jsetoain]

Remove commented code

[jsetoain]

Added missing documentation for SVE intrinsics

[jsetoain]

mlir/include/mlir/Dialect/SVE/SVE.td

25 ↗

(On Diff #307827)

Do you foresee a problem with this specific acronym or are you suggesting we should implement some sort of vendor-specific (or, more accurately, IP-designer-specific) naming convention for hardware-specific dialects? I'm happy to change the name if it makes more sense, but I'd need some guidance.

155 ↗

(On Diff #307827)

I'm not sure I understand the issue. Would you mind to elaborate, please? Notice that, in order to generate scalable vector code, we need a primitive that provides, at run-time, the scale of the vector; this would feed into the step of the outer loop. LLVM provides llvm.vscale for this purpose, and it's conveniently architecture-independent. If there are better alternatives, I'm happy to consider them. If I'm missing something, please let me know. Thank you!

[mehdi_amini]

mlir/include/mlir/Dialect/SVE/SVE.td

25 ↗

(On Diff #307827)

If this is not a vendor agnostic "scalable vector" abstraction but it is specific to the ARM SVE implementation, then what about naming this arm_sve as dialect name instead?

[mehdi_amini]

mlir/include/mlir/Dialect/SVE/SVE.td

115 ↗

(On Diff #307898)

Nice Ascii art :)

I suspect we need some markdown tweak to make sure it displays well after HTML conversion.

mlir/include/mlir/Dialect/SVE/SVEDialect.h

90 ↗

(On Diff #307898)

Can the types be generated from TableGen? https://mlir.llvm.org/docs/OpDefinitions/#type-definitions
(@rriddle may be able to advise here)

mlir/lib/Target/LLVMIR/LLVMSVEIntr.cpp

63 ↗

(On Diff #307898)

I don't understand why we need a specific conversion here. This concerns me, I see that we have this for "mlir-to-nvvmir" and "avx512-mlir-to-llvmir" but I don't understand why it is desirable. I'd expect the export to LLVM IR to be self-contained and not have many entry-points which don't compose with each others.

This looks to me that this would better be implemented using an op interface or similar mechanism instead.

You're not responsible for setting it up this way, but I'd still like to head from @ftynse / @rriddle on this.

[mehdi_amini]

This is really nice to see this, thanks for the contribution!

[rriddle]

mlir/include/mlir/Dialect/SVE/SVE.td

25 ↗

(On Diff #307827)

Apologies for the terse comments, added proper discussion to the RFC. My thoughts basically echo those of Mehdi, in that if we have something targeting a specific target we should ideally have that in the name similarly to how LLVM intrinsics work.

155 ↗

(On Diff #307827)

Is it possible to write the documentation of this operation in a way that can stand on it's own? I see the lowering to a specific llvm intrinsic as an implementation detail of the lowering. I'd prefer to avoid baking a specific lowering into the semantics of an operation. If we bake in the semantics such that we have "This op matches 1-1 with this llvm intrinsic", the dialect is basically just an extension of the LLVM dialect.

[jsetoain]

mlir/include/mlir/Dialect/SVE/SVE.td

115 ↗

(On Diff #307898)

Alternatively, I can remove the ASCII art. It's not in the original ISA documentation, but I find these "visual" explanations clearer than the formal specification, easier to notice the typical pitfalls with scalable vectors. Would that work as a quick and easy fix?

25 ↗

(On Diff #307827)

Don't worry, I took no offence, dry language is fine by me but I wasn't sure what the suggestion was. It's clear now, let's solve this in the RFC :-)

[jsetoain]

Fixed issue with doc formating.
Changed name of vector scale size operation to a better suited one, fixed documentation accordingly.

[jsetoain]

mlir/include/mlir/Dialect/SVE/SVE.td

155 ↗

(On Diff #307827)

Absolutely. I guess I was being lazy, my apologies. I hope changing the name and focusing on function instead of lowering target will avoid the confusion.

[mehdi_amini]

mlir/include/mlir/Dialect/SVE/SVE.td

115 ↗

(On Diff #307898)

ASCII art is fine, I was just pointing out that we need to surround it in-between something like <pre> / </pre> for it to properly display on the website later.

[jsetoain]

Switching to ODS for type generation

[jsetoain]

mlir/include/mlir/Dialect/SVE/SVEDialect.h

90 ↗

(On Diff #307898)

As it turns out, they can (I quite like it). Is this a new thing? I completely missed it and I noticed nobody else is using it. I've noticed something a bit odd, I still need to do the def SVE_ScalableVectorType : DialectType<SVE_Dialect, thing, but I don't see why TypeDef class can't do that automatically. Am I misusing it?

[mehdi_amini]

mlir/include/mlir/Dialect/SVE/SVEDialect.h

90 ↗

(On Diff #307898)

It is fairly recent (2 months maybe?)

I don't know for the other question, @rriddle will be able to help here.

[jsetoain]

Make operations signless.
Rename dialect and move it to a different namespace.

[nicolasvasilache]

Once my comments are addressed I am happy to land this.
I think the additional layer of tablegen discussed in https://llvm.discourse.group/t/exposing-llvm-ops-to-the-mlir-type-system/2290/11 and having everything in the LLVM dialect is a cross-cutting change that should be done separately and unified across all 3 dialects (AVX512, ArmNeon, ArmSVE).

mlir/include/mlir/Conversion/Passes.td

375

@aartbik improved the conversion to drop one-off passes.
See the changes to https://reviews.llvm.org/D92171 after I rebased on those.
Basically the ConvertVectorToLLVMPass gains a target-specific bit and conditionally includes the dependent dialects as well as the relevant conversion patterns.

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

107 ↗

(On Diff #309310)

Not sure what the proper source of truth is here but in the Neon dialect I just copied a part of the description and referred to https://developer.arm.com/architectures/instruction-sets/simd-isas/neon/intrinsics

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVEDialect.h

13 ↗

(On Diff #309310)

To silence the lint warnings, please spell out the whole path using _ in place of /.
e.g. MLIR_DIALECT_TARGET_ARMSVE_ARMSVEDIALECT_H

[nicolasvasilache]

mlir/include/mlir/Dialect/SVE/SVE.td

155 ↗

(On Diff #307827)

@rriddle see https://llvm.discourse.group/t/exposing-llvm-ops-to-the-mlir-type-system/2290/11, ideally we would only have a single dialect and an extra layer of tablegen but that should be done separately.

mlir/lib/Target/LLVMIR/LLVMSVEIntr.cpp

63 ↗

(On Diff #307898)

As usual with MLIR, this is on-demand and the avx512 use case was not justification enough.
We should now layer this better indeed but this is a cross cutting change that is better done separately.
I would not condition landing this on that particular change.

[nicolasvasilache]

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

151 ↗

(On Diff #309310)

Let's make everything signless at this level, you could write something along the lines of
signless integer that the op interprets as signed/ unsigned as appropriate.

[mehdi_amini]

mlir/lib/Target/LLVMIR/LLVMSVEIntr.cpp

63 ↗

(On Diff #307898)

Are you committing to this refactoring? If you'd like to land this before it then I'd like to see a TODO in the code and a commitment from someone to make it happen "soon".

[jsetoain]

Fixed lint problems.
Improved documentation.
Merged with D92614, doesn't require it's own lowering pass anymore.

[jsetoain]

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

107 ↗

(On Diff #309310)

I used the ArmARM, but I've switched to a source similar to yours. I think it's clearer that way.

151 ↗

(On Diff #309310)

Are you proposing to get rid of the distinction between smmla & ummla or is it just about the wording? If it's the former, I'm not sure how I'd go about differentiating between both cases if we come from signless MLIR. If it's the later, let me know if this works or you want me to make it clearer.

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVEDialect.h

13 ↗

(On Diff #309310)

I've fixed this one and the other one. Apologies for the overlook.

[aartbik]

Nice to see all vector dialects coming together!

mlir/include/mlir/Conversion/Passes.td

398–399

SVE -> ArmSVE

405

note that Nicolas made changes here too; race condition on rebasing :-)

423

rather not a default of true here ;-)

[nicolasvasilache]

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

151 ↗

(On Diff #309310)

Just the wording, in line with this: https://llvm.discourse.group/t/rfc-vector-dialects-neon-and-sve/2284/12.
For smmla you'd write signless integer that the op interprets as signed.
For ummla you'd write signless integer that the op interprets as unsigned.

[jsetoain]

Fixed default value for enabling ArmSVE extensions in vector

[jsetoain]

Improve documentation around signedness of operations.

[jsetoain]

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

151 ↗

(On Diff #309310)

Yes, I saw that. Hopefully this wording is clearer.

[jsetoain]

mlir/include/mlir/Conversion/Passes.td

405

I noticed that while I was following your patch for AVX512. Unless there's a canonical way to deal with this, I'll take silver medal and deal with it once D92171 has landed :-)

[jsetoain]

Rebased on top of D92171
Dialect syntax homogenized with ArmNeon dialect
Naming convention aligned with ArmNeon dialect
File location aligned with ArmNeon dialect
Conversion code aligned with ArmNeon dialect
Improved type checking
Added extra scalable vector type traits

[aartbik]

LGTM, but leaving the approval to River since he added some blocking feedback earlier, so I want to make sure he feels everything has been addressed

mlir/test/Conversion/ArmSVEToLLVM/convert-to-llvm.mlir

8

putting { on previous line seems a bit more consistent with most of our tests

[jsetoain]

Align test syntax style with common practise

[rriddle]

mlir/include/mlir/Dialect/ArmSVE/ArmSVE.td

83

nit: Can you use ShapedType::isDynamic here instead?

91

nit: Spell out auto here.

136

Is there any sort of ordering to these ops?

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp

46

nit: The order here is different from the one used elsewhere.

mlir/lib/Dialect/ArmSVE/IR/ArmSVEDialect.cpp

42

Don't use namespaces for method qualification, use arm_sve:: instead (like above).

mlir/lib/Target/LLVMIR/LLVMArmSVEIntr.cpp

40

Only classes should go in anonymous namespaces, methods should be static and at the top level.

[jsetoain]

Reorder operations and initializations alphabetically
Removed useless code
Clean up unnecessary 'auto' declartion
Fixed namespace usage

[jsetoain]

mlir/include/mlir/Dialect/ArmSVE/ArmSVE.td

83

Well... that was embarrassing :">

136

I've switched to alphabetical

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp

46

I've changed the others to leave all the vector extensions in alphabetical order

[jsetoain]

I don't have commit access to the repository. Please, commit when ready.

mlir/include/mlir/Conversion/ArmSVEToLLVM/ArmSVEToLLVM.h

+//===- ArmSVEToLLVM.h - Conversion Patterns from ArmSVE to LLVM -----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_CONVERSION_ARMSVETOLLVM_ARMSVETOLLVM_H_
+#define MLIR_CONVERSION_ARMSVETOLLVM_ARMSVETOLLVM_H_
+
+namespace mlir {
+
+class LLVMTypeConverter;
+class OwningRewritePatternList;
+
+/// Collect a set of patterns to convert from the ArmSVE dialect to LLVM.
+void populateArmSVEToLLVMConversionPatterns(LLVMTypeConverter &converter,
+                                            OwningRewritePatternList &patterns);
+
+} // namespace mlir
+
+#endif // MLIR_CONVERSION_ARMSVETOLLVM_ARMSVETOLLVM_H_

mlir/include/mlir/Conversion/Passes.td

def ConvertStandardToSPIRV : Pass<"convert-std-to-spirv", "ModuleOp"> {
   let constructor = "mlir::createConvertStandardToSPIRVPass()";
   let dependentDialects = ["spirv::SPIRVDialect"];
 }
 
 //===----------------------------------------------------------------------===//
 // VectorToSCF
 //===----------------------------------------------------------------------===//
 
 def ConvertVectorToSCF : FunctionPass<"convert-vector-to-scf"> {
   let summary = "Lower the operations from the vector dialect into the SCF "
                 "dialect";
   let constructor = "mlir::createConvertVectorToSCFPass()";
   let dependentDialects = ["AffineDialect", "scf::SCFDialect"];
   let options = [
     Option<"fullUnroll", "full-unroll", "bool", /*default=*/"false",
            "Perform full unrolling when converting vector transfers to SCF">,
   ];
 }
 
 //===----------------------------------------------------------------------===//
 // VectorToLLVM
 //===----------------------------------------------------------------------===//
 
 def ConvertVectorToLLVM : Pass<"convert-vector-to-llvm", "ModuleOp"> {
   let summary = "Lower the operations from the vector dialect into the LLVM "
                 "dialect";
   let description = [{
 
     Convert operations from the vector dialect into the LLVM IR dialect
     operations. The lowering pass provides several options to control
     the kinds of optimizations that are allowed. It also provides options
     that enable the use of one or more architectural-specific dialects
-    (AVX512, ArmNeon, SVE, etc.) in combination with the architectural-neutral
-    vector dialect lowering.
+    (AVX512, ArmNeon, ArmSVE, etc.) in combination with the
+    architectural-neutral vector dialect lowering.
 
   }];
   let constructor = "mlir::createConvertVectorToLLVMPass()";
   // Override explicitly in C++ to allow conditional dialect dependence.
   // let dependentDialects;
   let options = [
     Option<"reassociateFPReductions", "reassociate-fp-reductions",
            "bool", /*default=*/"false",
            "Allows llvm to reassociate floating-point reductions for speed">,
     Option<"enableIndexOptimizations", "enable-index-optimizations",
            "bool", /*default=*/"true",
            "Allows compiler to assume indices fit in 32-bit if that yields "
      "faster code">,
     Option<"enableAVX512", "enable-avx512",
            "bool", /*default=*/"false",
            "Enables the use of AVX512 dialect while lowering the vector "
      "dialect.">,
     Option<"enableArmNeon", "enable-arm-neon",
            "bool", /*default=*/"false",
            "Enables the use of ArmNeon dialect while lowering the vector "
-     "dialect.">
+     "dialect.">,
+    Option<"enableArmSVE", "enable-arm-sve",
+           "bool", /*default=*/"false",
+           "Enables the use of ArmSVE dialect while lowering the vector "
+       "dialect.">
   ];
 }
 
 //===----------------------------------------------------------------------===//
 // VectorToROCDL
 //===----------------------------------------------------------------------===//
 
 def ConvertVectorToROCDL : Pass<"convert-vector-to-rocdl", "ModuleOp"> {

mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h

 
 /// Options to control Vector to LLVM lowering.
 ///
 /// This should kept in sync with VectorToLLVM options defined for the
 /// ConvertVectorToLLVM pass in include/mlir/Conversion/Passes.td
 struct LowerVectorToLLVMOptions {
   LowerVectorToLLVMOptions()
       : reassociateFPReductions(false), enableIndexOptimizations(true),
-        enableArmNeon(false), enableAVX512(false) {}
+        enableArmNeon(false), enableArmSVE(false), enableAVX512(false) {}
 
   LowerVectorToLLVMOptions &setReassociateFPReductions(bool b) {
     reassociateFPReductions = b;
     return *this;
   }
   LowerVectorToLLVMOptions &setEnableIndexOptimizations(bool b) {
     enableIndexOptimizations = b;
     return *this;
   }
-  LowerVectorToLLVMOptions &setEnableAVX512(bool b) {
-    enableAVX512 = b;
-    return *this;
-  }
   LowerVectorToLLVMOptions &setEnableArmNeon(bool b) {
     enableArmNeon = b;
     return *this;
   }
+  LowerVectorToLLVMOptions &setEnableArmSVE(bool b) {
+    enableArmSVE = b;
+    return *this;
+  }
+  LowerVectorToLLVMOptions &setEnableAVX512(bool b) {
+    enableAVX512 = b;
+    return *this;
+  }
 
   bool reassociateFPReductions;
   bool enableIndexOptimizations;
   bool enableArmNeon;
+  bool enableArmSVE;
   bool enableAVX512;
 };
 
 /// Collect a set of patterns to convert from Vector contractions to LLVM Matrix
 /// Intrinsics. To lower to assembly, the LLVM flag -lower-matrix-intrinsics
 /// will be needed when invoking LLVM.
 void populateVectorToLLVMMatrixConversionPatterns(
     LLVMTypeConverter &converter, OwningRewritePatternList &patterns);

mlir/include/mlir/Dialect/ArmSVE/ArmSVE.td

+//===-- ArmSVE.td - ArmSVE dialect operation definitions ---*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the basic operations for the ArmSVE dialect.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef ARMSVE_OPS
+#define ARMSVE_OPS
+
+include "mlir/Interfaces/SideEffectInterfaces.td"
+
+//===----------------------------------------------------------------------===//
+// ArmSVE dialect definition
+//===----------------------------------------------------------------------===//
+
+def ArmSVE_Dialect : Dialect {
+  let name = "arm_sve";
+  let cppNamespace = "::mlir::arm_sve";
+  let summary = "Basic dialect to target Arm SVE architectures";
+  let description = [{
+    This dialect contains the definitions necessary to target Arm SVE scalable
+    vector operations, including a scalable vector type and intrinsics for
+    some Arm SVE instructions.
+  }];
+}
+
+//===----------------------------------------------------------------------===//
+// ArmSVE type definitions
+//===----------------------------------------------------------------------===//
+
+def ArmSVE_ScalableVectorType : DialectType<ArmSVE_Dialect,
+    CPred<"$_self.isa<ScalableVectorType>()">,
+              "scalable vector type">,
+    BuildableType<"$_builder.getType<ScalableVectorType>()"> {
+  let typeDescription = [{
+    `arm_sve.vector` represents vectors that will be processed by a scalable
+    vector architecture.
+  }];
+}
+
+class ArmSVE_Type<string name> : TypeDef<ArmSVE_Dialect, name> { }
+
+def ScalableVectorType : ArmSVE_Type<"ScalableVector"> {
+  let mnemonic = "vector";
+
+  let summary = "Scalable vector type";
+
+  let description = [{
+    A type representing scalable length SIMD vectors. Unlike fixed-length SIMD
+    vectors, whose size is constant and known at compile time, scalable
+    vectors' length is constant but determined by the specific hardware at
+    run time.
+  }];
+
+  let parameters = (ins
+    ArrayRefParameter<"int64_t", "Vector shape">:$shape,
+    "Type":$elementType
+  );
+
+  let printer = [{
+    $_printer << "vector<";
+    for (int64_t dim : getShape())
+      $_printer << dim << 'x';
+    $_printer << getElementType() << '>';
+  }];
+
+  let parser = [{
+    VectorType vector;
+    if ($_parser.parseType(vector))
+      return Type();
+    return get(ctxt, vector.getShape(), vector.getElementType());
+  }];
+
+  let extraClassDeclaration = [{
+    bool hasStaticShape() const {
+      return llvm::none_of(getShape(), ShapedType::isDynamic);
+    }
+    int64_t getNumElements() const {
+      assert(hasStaticShape() &&
+             "cannot get element count of dynamic shaped type");
+      ArrayRef<int64_t> shape = getShape();
+      int64_t num = 1;
+      for (auto dim : shape)
+        num *= dim;
+      return num;
+    }
+  }];
+}
+
+//===----------------------------------------------------------------------===//
+// ArmSVE type traits
+//===----------------------------------------------------------------------===//
+
+def IsScalableVectorTypePred :
+    CPred<"$_self.isa<::mlir::arm_sve::ScalableVectorType>()">;
+
+class ScalableVectorOf<list<Type> allowedTypes> :
+    ContainerType<AnyTypeOf<allowedTypes>, IsScalableVectorTypePred,
+          "$_self.cast<::mlir::arm_sve::ScalableVectorType>().getElementType()",
+          "scalable vector">;
+
+class IsScalableVectorOfLengthPred<list<int> allowedLengths> :
+  And<[IsScalableVectorTypePred,
+       Or<!foreach(allowedlength, allowedLengths, CPred<
+          [{$_self.cast<::mlir::arm_sve::ScalableVectorType>().getNumElements() == }]
+          # allowedlength>)>]>;
+
+class ScalableVectorOfLength<list<int> allowedLengths> : Type<
+  IsScalableVectorOfLengthPred<allowedLengths>,
+  " of length " # StrJoinInt<allowedLengths, "/">.result>;
+
+class ScalableVectorOfLengthAndType<list<int> allowedLengths,
+                                    list<Type> allowedTypes> : Type<
+  And<[ScalableVectorOf<allowedTypes>.predicate,
+       ScalableVectorOfLength<allowedLengths>.predicate]>,
+  ScalableVectorOf<allowedTypes>.description #
+  ScalableVectorOfLength<allowedLengths>.description>;
+
+//===----------------------------------------------------------------------===//
+// ArmSVE op definitions
+//===----------------------------------------------------------------------===//
+
+class ArmSVE_Op<string mnemonic, list<OpTrait> traits = []> :
+  Op<ArmSVE_Dialect, mnemonic, traits> {}
+
+def SdotOp : ArmSVE_Op<"sdot",
+               [NoSideEffect,
+               AllTypesMatch<["src1", "src2"]>,
+               AllTypesMatch<["acc", "dst"]>,
+             ]> {
+  let summary = "Vector-vector dot product and accumulate op";
+  let description = [{
+    SDOT: Signed integer addition of dot product.
+
+    This function maps to the SDOT instruction, and it takes signless integer
+    operands that the operation interprets as signed. It partitions the second
+    and third vector inputs into groups of four elements. They calculate the dot
+    product of each group (without loss of precision) and then add each result
+    to the overlapping element of the first vector input.
+
+    Source:
+    https://developer.arm.com/documentation/100987/0000
+  }];
+  // Supports either:
+  //   (vector<16xi8>, vector<16xi8>) -> (vector<4xi32>)
+  //   (vector<8xi16>. vector<8xi16>) -> (vector<2xi64>)
+  let arguments = (ins
+          ScalableVectorOfLengthAndType<[4, 2], [I32, I64]>:$acc,
+          ScalableVectorOfLengthAndType<[16, 8], [I8, I16]>:$src1,
+          ScalableVectorOfLengthAndType<[16, 8], [I8, I16]>:$src2
+  );
+  let results = (outs ScalableVectorOfLengthAndType<[4, 2], [I32, I64]>:$dst);
+  let assemblyFormat =
+    "$acc `,` $src1 `,` $src2 attr-dict `:` type($src1) `to` type($dst)";
+}
+
+def SmmlaOp : ArmSVE_Op<"smmla",
+                [NoSideEffect,
+                AllTypesMatch<["src1", "src2"]>,
+                AllTypesMatch<["acc", "dst"]>,
+              ]> {
+  let summary = "Matrix-matrix mutiply and accumulate op";
+  let description = [{
+    SMMLA: Signed integer matrix multiply-accumulate.
+
+    This function maps to the SMMLA instruction, and it takes signless integer
+    operands that the operation interprets as signed. It partitions the inputs
+    into 128-bit quadwords, with the first input containing a row-by-row 2×2
+    matrix of 32-bit integers, the second input containing a row-by-row 2×8
+    matrix of 8-bit integers, and the third input containing a column-by-column
+    8×2 matrix of 8-bit integers. For each quadword, they multiply the second
+    input matrix by the third input matrix using natural arithmetic and then add
+    the result to the first input using modular arithmetic.
+
+    Source:
+    https://developer.arm.com/documentation/100987/0000
+  }];
+  // Supports (vector<16xi8>, vector<16xi8>) -> (vector<4xi32>)
+  let arguments = (ins
+          ScalableVectorOfLengthAndType<[4], [I32]>:$acc,
+          ScalableVectorOfLengthAndType<[16], [I8]>:$src1,
+          ScalableVectorOfLengthAndType<[16], [I8]>:$src2
+  );
+  let results = (outs ScalableVectorOfLengthAndType<[4], [I32]>:$dst);
+  let assemblyFormat =
+    "$acc `,` $src1 `,` $src2 attr-dict `:` type($src1) `to` type($dst)";
+}
+
+def UdotOp : ArmSVE_Op<"udot",
+               [NoSideEffect,
+               AllTypesMatch<["src1", "src2"]>,
+               AllTypesMatch<["acc", "dst"]>,
+             ]> {
+  let summary = "Vector-vector dot product and accumulate op";
+  let description = [{
+    UDOT: Unsigned integer addition of dot product.
+
+    This function maps to the UDOT instruction, and it takes signless integer
+    operands that the operation interprets as unsigned. It partitions the second
+    and third vector inputs into groups of four elements. They calculate the dot
+    product of each group (without loss of precision) and then add each result
+    to the overlapping element of the first vector input.
+
+    Source:
+    https://developer.arm.com/documentation/100987/0000
+  }];
+  // Supports either:
+  //   (vector<16xi8>, vector<16xi8>) -> (vector<4xi32>)
+  //   (vector<8xi16>. vector<8xi16>) -> (vector<2xi64>)
+  let arguments = (ins
+          ScalableVectorOfLengthAndType<[4, 2], [I32, I64]>:$acc,
+          ScalableVectorOfLengthAndType<[16, 8], [I8, I16]>:$src1,
+          ScalableVectorOfLengthAndType<[16, 8], [I8, I16]>:$src2
+  );
+  let results = (outs ScalableVectorOfLengthAndType<[4, 2], [I32, I64]>:$dst);
+  let assemblyFormat =
+    "$acc `,` $src1 `,` $src2 attr-dict `:` type($src1) `to` type($dst)";
+}
+
+def UmmlaOp : ArmSVE_Op<"ummla",
+                [NoSideEffect,
+                AllTypesMatch<["src1", "src2"]>,
+                AllTypesMatch<["acc", "dst"]>,
+              ]> {
+  let summary = "Matrix-matrix mutiply and accumulate op";
+  let description = [{
+    UMMLA: Unsigned integer matrix multiply-accumulate.
+
+    This function maps to the UMMLA instruction, and it takes signless integer
+    operands that the operation interprets as unsigned. It partitions the inputs
+    into 128-bit quadwords, with the first input containing a row-by-row 2×2
+    matrix of 32-bit integers, the second input containing a row-by-row 2×8
+    matrix of 8-bit integers, and the third input containing a column-by-column
+    8×2 matrix of 8-bit integers. For each quadword, they multiply the second
+    input matrix by the third input matrix using natural arithmetic and then add
+    the result to the first input using modular arithmetic.
+
+    Source:
+    https://developer.arm.com/documentation/100987/0000
+  }];
+  // Supports (vector<16xi8>, vector<16xi8>) -> (vector<4xi32>)
+  let arguments = (ins
+          ScalableVectorOfLengthAndType<[4], [I32]>:$acc,
+          ScalableVectorOfLengthAndType<[16], [I8]>:$src1,
+          ScalableVectorOfLengthAndType<[16], [I8]>:$src2
+  );
+  let results = (outs ScalableVectorOfLengthAndType<[4], [I32]>:$dst);
+  let assemblyFormat =
+    "$acc `,` $src1 `,` $src2 attr-dict `:` type($src1) `to` type($dst)";
+}
+
+def VectorScaleOp : ArmSVE_Op<"vector_scale",
+                 [NoSideEffect]> {
+  let summary = "Load vector scale size";
+  let description = [{
+    The vector_scale op returns the scale of the scalable vectors, a positive
+    integer value that is constant at runtime but unknown at compile time.
+    The scale of the vector indicates the multiplicity of the vectors and
+    vector operations. I.e.: an !arm_sve.vector<4xi32> is equivalent to
+    vector_scale consecutive vector<4xi32>; and an operation on an
+    !arm_sve.vector<4xi32> is equivalent to performing that operation vector_scale
+    times, once on each <4xi32> segment of the scalable vector. The vector_scale
+    op can be used to calculate the step in vector-length agnostic (VLA) loops.
+  }];
+  let results = (outs Index:$res);
+  let assemblyFormat =
+    "attr-dict `:` type($res)";
+}
+
+#endif // ARMSVE_OPS

mlir/include/mlir/Dialect/ArmSVE/ArmSVEDialect.h

+//===- ArmSVEDialect.h - MLIR Dialect for Arm SVE ---------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the Target dialect for ArmSVE in MLIR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_ARMSVE_ARMSVEDIALECT_H
+#define MLIR_DIALECT_ARMSVE_ARMSVEDIALECT_H
+
+#include "mlir/IR/Dialect.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/StandardTypes.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+
+#include "mlir/Dialect/ArmSVE/ArmSVEDialect.h.inc"
+
+#define GET_TYPEDEF_CLASSES
+#include "mlir/Dialect/ArmSVE/ArmSVETypes.h.inc"
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/ArmSVE/ArmSVE.h.inc"
+
+#endif // MLIR_DIALECT_ARMSVE_ARMSVEDIALECT_H

mlir/include/mlir/Dialect/ArmSVE/CMakeLists.txt

+add_mlir_dialect(ArmSVE arm_sve ArmSVE)

mlir/include/mlir/Dialect/CMakeLists.txt

 add_subdirectory(Affine)
 add_subdirectory(Async)
 add_subdirectory(ArmNeon)
+add_subdirectory(ArmSVE)
 add_subdirectory(AVX512)
 add_subdirectory(GPU)
 add_subdirectory(Linalg)
 add_subdirectory(LLVMIR)
 add_subdirectory(OpenACC)
 add_subdirectory(OpenMP)
 add_subdirectory(PDL)
 add_subdirectory(PDLInterp)
 add_subdirectory(Quant)
 add_subdirectory(SCF)
 add_subdirectory(Shape)
 add_subdirectory(SPIRV)
 add_subdirectory(StandardOps)
 add_subdirectory(Tensor)
 add_subdirectory(Tosa)
 add_subdirectory(Vector)

mlir/include/mlir/Dialect/LLVMIR/CMakeLists.txt

 mlir_tablegen(LLVMAVX512Conversions.inc -gen-llvmir-conversions)
 add_public_tablegen_target(MLIRLLVMAVX512ConversionsIncGen)
 
 add_mlir_dialect(LLVMArmNeon llvm_arm_neon LLVMArmNeon)
 add_mlir_doc(LLVMArmNeon -gen-dialect-doc LLVMArmNeon Dialects/)
 set(LLVM_TARGET_DEFINITIONS LLVMArmNeon.td)
 mlir_tablegen(LLVMArmNeonConversions.inc -gen-llvmir-conversions)
 add_public_tablegen_target(MLIRLLVMArmNeonConversionsIncGen)
+
+add_mlir_dialect(LLVMArmSVE llvm_arm_sve LLVMArmSVE)
+add_mlir_doc(LLVMArmSVE -gen-dialect-doc LLVMArmSve Dialects/)
+set(LLVM_TARGET_DEFINITIONS LLVMArmSVE.td)
+mlir_tablegen(LLVMArmSVEConversions.inc -gen-llvmir-conversions)
+add_public_tablegen_target(MLIRLLVMArmSVEConversionsIncGen)

mlir/include/mlir/Dialect/LLVMIR/LLVMArmSVE.td

+//===-- LLVMArmSVE.td - LLVMARMSVE dialect op definitions --*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the basic operations for the LLVMArmSVE dialect.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVMIR_ARMSVE_OPS
+#define LLVMIR_ARMSVE_OPS
+
+include "mlir/Dialect/LLVMIR/LLVMOpBase.td"
+
+//===----------------------------------------------------------------------===//
+// LLVMArmSVE dialect definition
+//===----------------------------------------------------------------------===//
+
+def LLVMArmSVE_Dialect : Dialect {
+  let name = "llvm_arm_sve";
+  let cppNamespace = "::mlir::LLVM";
+}
+
+//----------------------------------------------------------------------------//
+// MLIR LLVM Arm SVE intrinsics using the MLIR LLVM Dialect type system
+//----------------------------------------------------------------------------//
+
+class LLVMArmSVE_NonSVEIntrUnaryOverloadedOp<string mnemonic,
+                                             list<OpTrait> traits =[]> :
+  LLVM_IntrOpBase</*Dialect dialect=*/LLVMArmSVE_Dialect,
+                  /*string opName=*/mnemonic,
+                  /*string enumName=*/mnemonic,
+                  /*list<int> overloadedResults=*/[0],
+                  /*list<int> overloadedOperands=*/[], // defined by result overload
+                  /*list<OpTrait> traits=*/traits,
+                  /*int numResults=*/1>;
+
+class LLVMArmSVE_IntrBinaryOverloadedOp<string mnemonic,
+                                        list<OpTrait> traits = []> :
+  LLVM_IntrOpBase</*Dialect dialect=*/LLVMArmSVE_Dialect,
+                  /*string opName=*/mnemonic,
+                  /*string enumName=*/"aarch64_sve_" # !subst(".", "_", mnemonic),
+                  /*list<int> overloadedResults=*/[0],
+                  /*list<int> overloadedOperands=*/[], // defined by result overload
+                  /*list<OpTrait> traits=*/traits,
+                  /*int numResults=*/1>;
+
+def LLVM_aarch64_arm_sve_ummla :
+  LLVMArmSVE_IntrBinaryOverloadedOp<"ummla">,
+  Arguments<(ins LLVM_Type, LLVM_Type, LLVM_Type)>;
+
+def LLVM_aarch64_arm_sve_smmla :
+  LLVMArmSVE_IntrBinaryOverloadedOp<"smmla">,
+  Arguments<(ins LLVM_Type, LLVM_Type, LLVM_Type)>;
+
+def LLVM_aarch64_arm_sve_sdot :
+  LLVMArmSVE_IntrBinaryOverloadedOp<"sdot">,
+  Arguments<(ins LLVM_Type, LLVM_Type, LLVM_Type)>;
+
+def LLVM_aarch64_arm_sve_udot :
+  LLVMArmSVE_IntrBinaryOverloadedOp<"udot">,
+  Arguments<(ins LLVM_Type, LLVM_Type, LLVM_Type)>;
+
+def LLVM_vector_scale :
+  LLVMArmSVE_NonSVEIntrUnaryOverloadedOp<"vscale">;
+
+#endif // ARMSVE_OPS

mlir/include/mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h

+//===- LLVMSVEDialect.h - MLIR Dialect for LLVMSVE --------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the Target dialect for LLVMArmSVE in MLIR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_LLVMIR_LLVMARMSVEDIALECT_H_
+#define MLIR_DIALECT_LLVMIR_LLVMARMSVEDIALECT_H_
+
+#include "mlir/IR/Dialect.h"
+#include "mlir/IR/OpDefinition.h"
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/LLVMIR/LLVMArmSVE.h.inc"
+
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h.inc"
+
+#endif // MLIR_DIALECT_LLVMIR_LLVMARMSVEDIALECT_H_

mlir/include/mlir/InitAllDialects.h

 //===----------------------------------------------------------------------===//
 
 #ifndef MLIR_INITALLDIALECTS_H_
 #define MLIR_INITALLDIALECTS_H_
 
 #include "mlir/Dialect/AVX512/AVX512Dialect.h"
 #include "mlir/Dialect/Affine/IR/AffineOps.h"
 #include "mlir/Dialect/ArmNeon/ArmNeonDialect.h"
+#include "mlir/Dialect/ArmSVE/ArmSVEDialect.h"
 #include "mlir/Dialect/Async/IR/Async.h"
 #include "mlir/Dialect/GPU/GPUDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMAVX512Dialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMArmNeonDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/LLVMIR/NVVMDialect.h"
 #include "mlir/Dialect/LLVMIR/ROCDLDialect.h"
 #include "mlir/Dialect/Linalg/IR/LinalgOps.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
 #include "mlir/Dialect/PDL/IR/PDL.h"
 #include "mlir/Dialect/PDLInterp/IR/PDLInterp.h"
registry.insert<acc::OpenACCDialect,
                   AffineDialect,
                   arm_neon::ArmNeonDialect,
                   async::AsyncDialect,
                   avx512::AVX512Dialect,
                   gpu::GPUDialect,
                   LLVM::LLVMAVX512Dialect,
                   LLVM::LLVMDialect,
                   LLVM::LLVMArmNeonDialect,
+                  LLVM::LLVMArmSVEDialect,
                   linalg::LinalgDialect,
                   scf::SCFDialect,
                   omp::OpenMPDialect,
                   pdl::PDLDialect,
                   pdl_interp::PDLInterpDialect,
                   quant::QuantizationDialect,
                   spirv::SPIRVDialect,
                   StandardOpsDialect,
+                  arm_sve::ArmSVEDialect,
                   vector::VectorDialect,
                   NVVM::NVVMDialect,
                   ROCDL::ROCDLDialect,
                   SDBMDialect,
                   shape::ShapeDialect,
                   tensor::TensorDialect,
                   tosa::TosaDialect>();
   // clang-format on
 }
 
 } // namespace mlir
 
 #endif // MLIR_INITALLDIALECTS_H_

mlir/include/mlir/InitAllTranslations.h

 void registerFromLLVMIRTranslation();
 void registerFromSPIRVTranslation();
 void registerToLLVMIRTranslation();
 void registerToSPIRVTranslation();
 void registerToNVVMIRTranslation();
 void registerToROCDLIRTranslation();
 void registerArmNeonToLLVMIRTranslation();
 void registerAVX512ToLLVMIRTranslation();
+void registerArmSVEToLLVMIRTranslation();
 
 // This function should be called before creating any MLIRContext if one
 // expects all the possible translations to be made available to the context
 // automatically.
 inline void registerAllTranslations() {
   static bool initOnce = []() {
     registerFromLLVMIRTranslation();
     registerFromSPIRVTranslation();
     registerToLLVMIRTranslation();
     registerToSPIRVTranslation();
     registerToNVVMIRTranslation();
     registerToROCDLIRTranslation();
     registerArmNeonToLLVMIRTranslation();
     registerAVX512ToLLVMIRTranslation();
+    registerArmSVEToLLVMIRTranslation();
     return true;
   }();
   (void)initOnce;
 }
 } // namespace mlir
 
 #endif // MLIR_INITALLTRANSLATIONS_H

mlir/lib/Conversion/ArmSVEToLLVM/ArmSVEToLLVM.cpp

+//===- ArmSVEToLLVM.cpp - Convert ArmSVE to the LLVM dialect --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Conversion/ArmSVEToLLVM/ArmSVEToLLVM.h"
+#include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h"
+#include "mlir/Dialect/ArmSVE/ArmSVEDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/StandardOps/IR/Ops.h"
+#include "mlir/Dialect/Vector/VectorOps.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/PatternMatch.h"
+
+using namespace mlir;
+using namespace mlir::arm_sve;
+using namespace mlir::vector;
+
+using SdotOpLowering =
+    OneToOneConvertToLLVMPattern<SdotOp, LLVM::aarch64_arm_sve_sdot>;
+
+using SmmlaOpLowering =
+    OneToOneConvertToLLVMPattern<SmmlaOp, LLVM::aarch64_arm_sve_smmla>;
+
+using UdotOpLowering =
+    OneToOneConvertToLLVMPattern<UdotOp, LLVM::aarch64_arm_sve_udot>;
+
+using UmmlaOpLowering =
+    OneToOneConvertToLLVMPattern<UmmlaOp, LLVM::aarch64_arm_sve_ummla>;
+
+using VectorScaleOpLowering =
+    OneToOneConvertToLLVMPattern<VectorScaleOp, LLVM::vector_scale>;
+
+// Extract an LLVM IR type from the LLVM IR dialect type.
+static LLVM::LLVMType unwrap(Type type) {
+  if (!type)
+    return nullptr;
+  auto *mlirContext = type.getContext();
+  auto wrappedLLVMType = type.dyn_cast<LLVM::LLVMType>();
+  if (!wrappedLLVMType)
+    emitError(UnknownLoc::get(mlirContext),
+              "conversion resulted in a non-LLVM type");
+  return wrappedLLVMType;
+}
+
+static Optional<Type>
+convertScalableVectorTypeToLLVM(ScalableVectorType svType,
+                                LLVMTypeConverter &converter) {
+  auto elementType = unwrap(converter.convertType(svType.getElementType()));
+  if (!elementType)
+    return {};
+
+  auto sVectorType =
+      LLVM::LLVMScalableVectorType::get(elementType, svType.getShape().back());
+  return sVectorType;
+}
+
+/// Populate the given list with patterns that convert from ArmSVE to LLVM.
+void mlir::populateArmSVEToLLVMConversionPatterns(
+    LLVMTypeConverter &converter, OwningRewritePatternList &patterns) {
+  converter.addConversion([&converter](ScalableVectorType svType) {
+    return convertScalableVectorTypeToLLVM(svType, converter);
+  });
+  // clang-format off
+  patterns.insert<SdotOpLowering,
+                  SmmlaOpLowering,
+                  UdotOpLowering,
+                  UmmlaOpLowering,
+                  VectorScaleOpLowering>(converter);
+  // clang-format on
+}

mlir/lib/Conversion/ArmSVEToLLVM/CMakeLists.txt

+add_mlir_conversion_library(MLIRArmSVEToLLVM
+  ArmSVEToLLVM.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Conversion/ArmSVEToLLVM
+
+  DEPENDS
+  MLIRConversionPassIncGen
+
+  LINK_COMPONENTS
+  Core
+
+  LINK_LIBS PUBLIC
+  MLIRArmSVE
+  MLIRLLVMArmSVE
+  MLIRLLVMIR
+  MLIRStandardToLLVM
+  MLIRTransforms
+  )

mlir/lib/Conversion/CMakeLists.txt

 add_subdirectory(SCFToGPU)
 add_subdirectory(SCFToOpenMP)
 add_subdirectory(SCFToSPIRV)
 add_subdirectory(SCFToStandard)
 add_subdirectory(ShapeToStandard)
 add_subdirectory(SPIRVToLLVM)
 add_subdirectory(StandardToLLVM)
 add_subdirectory(StandardToSPIRV)
+add_subdirectory(ArmSVEToLLVM)
 add_subdirectory(VectorToROCDL)
 add_subdirectory(VectorToLLVM)
 add_subdirectory(VectorToSCF)
 add_subdirectory(VectorToSPIRV)

mlir/lib/Conversion/PassDetail.h

 
 namespace gpu {
 class GPUDialect;
 class GPUModuleOp;
 } // end namespace gpu
 
 namespace LLVM {
 class LLVMArmNeonDialect;
+class LLVMArmSVEDialect;
 class LLVMAVX512Dialect;
 class LLVMDialect;
 } // end namespace LLVM
 
 namespace NVVM {
 class NVVMDialect;
 } // end namespace NVVM
 

mlir/lib/Conversion/VectorToLLVM/CMakeLists.txt

add_mlir_conversion_library(MLIRVectorToLLVM
 
   LINK_LIBS PUBLIC
   MLIRArmNeon
   MLIRArmNeonToLLVM
   MLIRAVX512
   MLIRAVX512ToLLVM
   MLIRLLVMArmNeon
   MLIRLLVMAVX512
+  MLIRArmSVE
+  MLIRArmSVEToLLVM
+  MLIRLLVMArmSVE
   MLIRLLVMIR
   MLIRStandardToLLVM
   MLIRTargetLLVMIRModuleTranslation
   MLIRTransforms
   MLIRVector
   )

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp

 //===- VectorToLLVM.cpp - Conversion from Vector to the LLVM dialect ------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h"
 
 #include "../PassDetail.h"
 
 #include "mlir/Conversion/AVX512ToLLVM/ConvertAVX512ToLLVM.h"
 #include "mlir/Conversion/ArmNeonToLLVM/ArmNeonToLLVM.h"
+#include "mlir/Conversion/ArmSVEToLLVM/ArmSVEToLLVM.h"
 #include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h"
 #include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVMPass.h"
 #include "mlir/Dialect/AVX512/AVX512Dialect.h"
 #include "mlir/Dialect/ArmNeon/ArmNeonDialect.h"
+#include "mlir/Dialect/ArmSVE/ArmSVEDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMAVX512Dialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMArmNeonDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/Vector/VectorOps.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 
 using namespace mlir;
 using namespace mlir::vector;
 
 namespace {
 struct LowerVectorToLLVMPass
     : public ConvertVectorToLLVMBase<LowerVectorToLLVMPass> {
   LowerVectorToLLVMPass(const LowerVectorToLLVMOptions &options) {
     this->reassociateFPReductions = options.reassociateFPReductions;
     this->enableIndexOptimizations = options.enableIndexOptimizations;
     this->enableArmNeon = options.enableArmNeon;
+    this->enableArmSVE = options.enableArmSVE;
     this->enableAVX512 = options.enableAVX512;
   }
   // Override explicitly to allow conditional dialect dependence.
   void getDependentDialects(DialectRegistry &registry) const override {
     registry.insert<LLVM::LLVMDialect>();
     if (enableArmNeon)
       registry.insert<LLVM::LLVMArmNeonDialect>();
+    if (enableArmSVE)
+      registry.insert<LLVM::LLVMArmSVEDialect>();
     if (enableAVX512)
       registry.insert<LLVM::LLVMAVX512Dialect>();
   }
   void runOnOperation() override;
 };
 } // namespace
 
 void LowerVectorToLLVMPass::runOnOperation() {
void LowerVectorToLLVMPass::runOnOperation() {
 
   // Architecture specific augmentations.
   LLVMConversionTarget target(getContext());
   if (enableArmNeon) {
     target.addLegalDialect<LLVM::LLVMArmNeonDialect>();
     target.addIllegalDialect<arm_neon::ArmNeonDialect>();
     populateArmNeonToLLVMConversionPatterns(converter, patterns);
   }
+  if (enableArmSVE) {
+    target.addLegalDialect<LLVM::LLVMArmSVEDialect>();
+    target.addIllegalDialect<arm_sve::ArmSVEDialect>();
+    populateArmSVEToLLVMConversionPatterns(converter, patterns);
+  }
   if (enableAVX512) {
     target.addLegalDialect<LLVM::LLVMAVX512Dialect>();
     target.addIllegalDialect<avx512::AVX512Dialect>();
     populateAVX512ToLLVMConversionPatterns(converter, patterns);
   }
 
   if (failed(
           applyPartialConversion(getOperation(), target, std::move(patterns))))
     signalPassFailure();
 }
 
 std::unique_ptr<OperationPass<ModuleOp>>
 mlir::createConvertVectorToLLVMPass(const LowerVectorToLLVMOptions &options) {
   return std::make_unique<LowerVectorToLLVMPass>(options);
 }

mlir/lib/Dialect/ArmSVE/CMakeLists.txt

+add_mlir_dialect_library(MLIRArmSVE
+  IR/ArmSVEDialect.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/ArmSVE
+
+  DEPENDS
+  MLIRArmSVEIncGen
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  MLIRSideEffectInterfaces
+  )

mlir/lib/Dialect/ArmSVE/IR/ArmSVEDialect.cpp

+//===- ArmSVEDialect.cpp - MLIR ArmSVE dialect implementation -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the ArmSVE dialect and its operations.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/ArmSVE/ArmSVEDialect.h"
+#include "mlir/Dialect/Vector/VectorOps.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/DialectImplementation.h"
+#include "mlir/IR/OpImplementation.h"
+#include "mlir/IR/StandardTypes.h"
+#include "mlir/IR/TypeUtilities.h"
+#include "llvm/ADT/TypeSwitch.h"
+
+using namespace mlir;
+
+void arm_sve::ArmSVEDialect::initialize() {
+  addOperations<
+#define GET_OP_LIST
+#include "mlir/Dialect/ArmSVE/ArmSVE.cpp.inc"
+      >();
+  addTypes<
+#define GET_TYPEDEF_LIST
+#include "mlir/Dialect/ArmSVE/ArmSVETypes.cpp.inc"
+      >();
+}
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/ArmSVE/ArmSVE.cpp.inc"
+
+#define GET_TYPEDEF_CLASSES
+#include "mlir/Dialect/ArmSVE/ArmSVETypes.cpp.inc"
+
+//===----------------------------------------------------------------------===//
+// ScalableVectorType
+//===----------------------------------------------------------------------===//
+
+Type arm_sve::ArmSVEDialect::parseType(DialectAsmParser &parser) const {
+  llvm::SMLoc typeLoc = parser.getCurrentLocation();
+  auto genType = generatedTypeParser(getContext(), parser, "vector");
+  if (genType != Type())
+    return genType;
+  parser.emitError(typeLoc, "unknown type in ArmSVE dialect");
+  return Type();
+}
+
+void arm_sve::ArmSVEDialect::printType(Type type, DialectAsmPrinter &os) const {
+  if (failed(generatedTypePrinter(type, os)))
+    llvm_unreachable("unexpected 'arm_sve' type kind");
+}

mlir/lib/Dialect/CMakeLists.txt

 add_subdirectory(Affine)
 add_subdirectory(ArmNeon)
+add_subdirectory(ArmSVE)
 add_subdirectory(Async)
 add_subdirectory(AVX512)
 add_subdirectory(GPU)
 add_subdirectory(Linalg)
 add_subdirectory(LLVMIR)
 add_subdirectory(OpenACC)
 add_subdirectory(OpenMP)
 add_subdirectory(PDL)

mlir/lib/Dialect/LLVMIR/CMakeLists.txt

add_mlir_dialect_library(MLIRLLVMArmNeon
   Core
 
   LINK_LIBS PUBLIC
   MLIRIR
   MLIRLLVMIR
   MLIRSideEffectInterfaces
   )
 
+add_mlir_dialect_library(MLIRLLVMArmSVE
+  IR/LLVMArmSVEDialect.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/LLVMIR
+
+  DEPENDS
+  MLIRLLVMArmSVEIncGen
+  MLIRLLVMArmSVEConversionsIncGen
+  intrinsics_gen
+
+  LINK_COMPONENTS
+  AsmParser
+  Core
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  MLIRLLVMIR
+  MLIRSideEffectInterfaces
+  )
+
 add_mlir_dialect_library(MLIRNVVMIR
   IR/NVVMDialect.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/LLVMIR
 
   DEPENDS
   MLIRNVVMOpsIncGen

mlir/lib/Dialect/LLVMIR/IR/LLVMArmSVEDialect.cpp

+//===- LLVMArmSVEDialect.cpp - MLIR LLVMSVE ops implementation ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the LLVMArmSVE dialect and its operations.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/IR/IntrinsicsAArch64.h"
+
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/OpImplementation.h"
+#include "mlir/IR/TypeUtilities.h"
+
+using namespace mlir;
+
+void LLVM::LLVMArmSVEDialect::initialize() {
+  addOperations<
+#define GET_OP_LIST
+#include "mlir/Dialect/LLVMIR/LLVMArmSVE.cpp.inc"
+      >();
+}
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/LLVMIR/LLVMArmSVE.cpp.inc"

mlir/lib/Target/CMakeLists.txt

add_mlir_translation_library(MLIRTargetArmNeon
 
   LINK_LIBS PUBLIC
   MLIRIR
   MLIRLLVMArmNeon
   MLIRLLVMIR
   MLIRTargetLLVMIRModuleTranslation
   )
 
+add_mlir_translation_library(MLIRTargetArmSVE
+  LLVMIR/LLVMArmSVEIntr.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Target/LLVMIR
+
+  DEPENDS
+  MLIRLLVMArmSVEConversionsIncGen
+
+  LINK_COMPONENTS
+  Core
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  MLIRLLVMArmSVE
+  MLIRLLVMIR
+  MLIRTargetLLVMIRModuleTranslation
+  )
+
 add_mlir_translation_library(MLIRTargetNVVMIR
   LLVMIR/ConvertToNVVMIR.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Target/LLVMIR
 
   DEPENDS
   intrinsics_gen

mlir/lib/Target/LLVMIR/LLVMArmSVEIntr.cpp

+//===- LLVMArmSVEIntr.cpp - Convert MLIR LLVM dialect to LLVM intrinsics --===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements a translation between the MLIR LLVM and ArmSVE dialects
+// and LLVM IR with Arm SVE intrinsics.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h"
+#include "mlir/Target/LLVMIR/ModuleTranslation.h"
+#include "mlir/Translation.h"
+#include "llvm/IR/IntrinsicsAArch64.h"
+
+using namespace mlir;
+
+namespace {
+class LLVMArmSVEModuleTranslation : public LLVM::ModuleTranslation {
+  friend LLVM::ModuleTranslation;
+
+public:
+  using LLVM::ModuleTranslation::ModuleTranslation;
+
+protected:
+  LogicalResult convertOperation(Operation &opInst,
+                                 llvm::IRBuilder<> &builder) override {
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEConversions.inc"
+
+    return LLVM::ModuleTranslation::convertOperation(opInst, builder);
+  }
+};
+} // end namespace
+
+static std::unique_ptr<llvm::Module>
+translateLLVMArmSVEModuleToLLVMIR(Operation *m, llvm::LLVMContext &llvmContext,
+                                  StringRef name) {
+  return LLVM::ModuleTranslation::translateModule<LLVMArmSVEModuleTranslation>(
+      m, llvmContext, name);
+}
+
+namespace mlir {
+void registerArmSVEToLLVMIRTranslation() {
+  TranslateFromMLIRRegistration reg(
+      "arm-sve-mlir-to-llvmir",
+      [](ModuleOp module, raw_ostream &output) {
+        llvm::LLVMContext llvmContext;
+        auto llvmModule = translateLLVMArmSVEModuleToLLVMIR(
+            module, llvmContext, "LLVMDialectModule");
+        if (!llvmModule)
+          return failure();
+
+        llvmModule->print(output, nullptr);
+        return success();
+      },
+      [](DialectRegistry &registry) {
+        registry.insert<LLVM::LLVMArmSVEDialect, LLVM::LLVMDialect>();
+      });
+}
+} // namespace mlir

mlir/test/Conversion/ArmSVEToLLVM/convert-to-llvm.mlir

+// RUN: mlir-opt %s -convert-vector-to-llvm="enable-arm-sve" | mlir-opt | FileCheck %s
+
+func @arm_sve_sdot(%a: !arm_sve.vector<16xi8>,
+                   %b: !arm_sve.vector<16xi8>,
+                   %c: !arm_sve.vector<4xi32>)
+    -> !arm_sve.vector<4xi32> {
+  // CHECK: llvm_arm_sve.sdot
+  %0 = arm_sve.sdot %c, %a, %b :
+               !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @arm_sve_smmla(%a: !arm_sve.vector<16xi8>,
+                    %b: !arm_sve.vector<16xi8>,
+                    %c: !arm_sve.vector<4xi32>)
+    -> !arm_sve.vector<4xi32> {
+  // CHECK: llvm_arm_sve.smmla
+  %0 = arm_sve.smmla %c, %a, %b :
+               !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @arm_sve_udot(%a: !arm_sve.vector<16xi8>,
+                   %b: !arm_sve.vector<16xi8>,
+                   %c: !arm_sve.vector<4xi32>)
+    -> !arm_sve.vector<4xi32> {
+  // CHECK: llvm_arm_sve.udot
+  %0 = arm_sve.udot %c, %a, %b :
+               !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @arm_sve_ummla(%a: !arm_sve.vector<16xi8>,
+                    %b: !arm_sve.vector<16xi8>,
+                    %c: !arm_sve.vector<4xi32>)
+    -> !arm_sve.vector<4xi32> {
+  // CHECK: llvm_arm_sve.ummla
+  %0 = arm_sve.ummla %c, %a, %b :
+               !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @get_vector_scale() -> index {
+  // CHECK: llvm_arm_sve.vscale
+  %0 = arm_sve.vector_scale : index
+  return %0 : index
+}

mlir/test/Dialect/ArmSVE/roundtrip.mlir

+// RUN: mlir-opt -verify-diagnostics %s | mlir-opt | FileCheck %s
+
+func @arm_sve_sdot(%a: !arm_sve.vector<16xi8>,
+                   %b: !arm_sve.vector<16xi8>,
+                   %c: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32> {
+  // CHECK: arm_sve.sdot {{.*}}: !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32
+  %0 = arm_sve.sdot %c, %a, %b :
+             !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @arm_sve_smmla(%a: !arm_sve.vector<16xi8>,
+                    %b: !arm_sve.vector<16xi8>,
+                    %c: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32> {
+  // CHECK: arm_sve.smmla {{.*}}: !arm_sve.vector<16xi8> to !arm_sve.vector<4xi3
+  %0 = arm_sve.smmla %c, %a, %b :
+             !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @arm_sve_udot(%a: !arm_sve.vector<16xi8>,
+                   %b: !arm_sve.vector<16xi8>,
+                   %c: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32> {
+  // CHECK: arm_sve.udot {{.*}}: !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32
+  %0 = arm_sve.udot %c, %a, %b :
+             !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @arm_sve_ummla(%a: !arm_sve.vector<16xi8>,
+                    %b: !arm_sve.vector<16xi8>,
+                    %c: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32> {
+  // CHECK: arm_sve.ummla {{.*}}: !arm_sve.vector<16xi8> to !arm_sve.vector<4xi3
+  %0 = arm_sve.ummla %c, %a, %b :
+             !arm_sve.vector<16xi8> to !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @get_vector_scale() -> index {
+  // CHECK: arm_sve.vector_scale : index
+  %0 = arm_sve.vector_scale : index
+  return %0 : index
+}

mlir/test/Target/arm-sve.mlir

+// RUN: mlir-opt -verify-diagnostics %s | mlir-opt | mlir-translate --arm-sve-mlir-to-llvmir | FileCheck %s
+
+// CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_sdot
+llvm.func @arm_sve_sdot(%arg0: !llvm.vec<? x 16 x i8>,
+                        %arg1: !llvm.vec<? x 16 x i8>,
+                        %arg2: !llvm.vec<? x 4 x i32>)
+                        -> !llvm.vec<? x 4 x i32> {
+  // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.sdot.nxv4i32(<vscale x 4
+  %0 = "llvm_arm_sve.sdot"(%arg2, %arg0, %arg1) :
+    (!llvm.vec<? x 4 x i32>, !llvm.vec<? x 16 x i8>, !llvm.vec<? x 16 x i8>)
+        -> !llvm.vec<? x 4 x i32>
+  llvm.return %0 : !llvm.vec<? x 4 x i32>
+}
+
+// CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_smmla
+llvm.func @arm_sve_smmla(%arg0: !llvm.vec<? x 16 x i8>,
+                         %arg1: !llvm.vec<? x 16 x i8>,
+                         %arg2: !llvm.vec<? x 4 x i32>)
+                         -> !llvm.vec<? x 4 x i32> {
+  // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.smmla.nxv4i32(<vscale x 4
+  %0 = "llvm_arm_sve.smmla"(%arg2, %arg0, %arg1) :
+    (!llvm.vec<? x 4 x i32>, !llvm.vec<? x 16 x i8>, !llvm.vec<? x 16 x i8>)
+        -> !llvm.vec<? x 4 x i32>
+  llvm.return %0 : !llvm.vec<? x 4 x i32>
+}
+
+// CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_udot
+llvm.func @arm_sve_udot(%arg0: !llvm.vec<? x 16 x i8>,
+                        %arg1: !llvm.vec<? x 16 x i8>,
+                        %arg2: !llvm.vec<? x 4 x i32>)
+                        -> !llvm.vec<? x 4 x i32> {
+  // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.udot.nxv4i32(<vscale x 4
+  %0 = "llvm_arm_sve.udot"(%arg2, %arg0, %arg1) :
+    (!llvm.vec<? x 4 x i32>, !llvm.vec<? x 16 x i8>, !llvm.vec<? x 16 x i8>)
+        -> !llvm.vec<? x 4 x i32>
+  llvm.return %0 : !llvm.vec<? x 4 x i32>
+}
+
+// CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_ummla
+llvm.func @arm_sve_ummla(%arg0: !llvm.vec<? x 16 x i8>,
+                         %arg1: !llvm.vec<? x 16 x i8>,
+                         %arg2: !llvm.vec<? x 4 x i32>)
+                         -> !llvm.vec<? x 4 x i32> {
+  // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.ummla.nxv4i32(<vscale x 4
+  %0 = "llvm_arm_sve.ummla"(%arg2, %arg0, %arg1) :
+    (!llvm.vec<? x 4 x i32>, !llvm.vec<? x 16 x i8>, !llvm.vec<? x 16 x i8>)
+        -> !llvm.vec<? x 4 x i32>
+  llvm.return %0 : !llvm.vec<? x 4 x i32>
+}
+
+// CHECK-LABEL: define i64 @get_vector_scale()
+llvm.func @get_vector_scale() -> !llvm.i64 {
+  // CHECK: call i64 @llvm.vscale.i64()
+  %0 = "llvm_arm_sve.vscale"() : () -> !llvm.i64
+  llvm.return %0 : !llvm.i64
+}