[mlir][ArmSVE][RFC] Add an ArmSVE dialect

[jsetoain]

Summary

This revision starts an Arm-specific ArmSVE dialect discussed in the discourse RFC thread:

https://llvm.discourse.group/t/rfc-vector-dialects-neon-and-sve/2284

[rriddle]

What discourse thread was this discussed in?

[rriddle]

mlir/include/mlir/Dialect/LLVMIR/LLVMSVE.td

54 ↗

(On Diff #307827)

Remove commented out code.

mlir/include/mlir/Dialect/SVE/SVE.td

25 ↗

(On Diff #307827)

SVE seems way too generic of a name for some seemingly specific to arm.

71 ↗

(On Diff #307827)

I'd prefer you resolve these todos right now.

155 ↗

(On Diff #307827)

I dont particularly like the idea of hard coding a higher level dialect specifically on what it would target in LLVM.

[nicolasvasilache]

In D92172#2418298, @rriddle wrote:

What discourse thread was this discussed in?

https://llvm.discourse.group/t/rfc-vector-dialects-neon-and-sve/2284 but manually creating http links and synchronizing at sub minute granularity is hard :)

@jsetoain could you please update your commit message with this link?

[rriddle]

In D92172#2418310, @nicolasvasilache wrote:

In D92172#2418298, @rriddle wrote:

What discourse thread was this discussed in?

https://llvm.discourse.group/t/rfc-vector-dialects-neon-and-sve/2284 but manually creating http links and synchronizing at sub minute granularity is hard :)

@jsetoain could you please update your commit message with this link?

Thanks, I just assumed I missed the thread that discussed this.

[jsetoain]

Change to comply with header guard naming style

[jsetoain]

Remove commented code

[jsetoain]

Added missing documentation for SVE intrinsics

[jsetoain]

mlir/include/mlir/Dialect/SVE/SVE.td

25 ↗

(On Diff #307827)

Do you foresee a problem with this specific acronym or are you suggesting we should implement some sort of vendor-specific (or, more accurately, IP-designer-specific) naming convention for hardware-specific dialects? I'm happy to change the name if it makes more sense, but I'd need some guidance.

155 ↗

(On Diff #307827)

I'm not sure I understand the issue. Would you mind to elaborate, please? Notice that, in order to generate scalable vector code, we need a primitive that provides, at run-time, the scale of the vector; this would feed into the step of the outer loop. LLVM provides llvm.vscale for this purpose, and it's conveniently architecture-independent. If there are better alternatives, I'm happy to consider them. If I'm missing something, please let me know. Thank you!

[mehdi_amini]

mlir/include/mlir/Dialect/SVE/SVE.td

25 ↗

(On Diff #307827)

If this is not a vendor agnostic "scalable vector" abstraction but it is specific to the ARM SVE implementation, then what about naming this arm_sve as dialect name instead?

[mehdi_amini]

mlir/include/mlir/Dialect/SVE/SVE.td

115 ↗

(On Diff #307898)

Nice Ascii art :)

I suspect we need some markdown tweak to make sure it displays well after HTML conversion.

mlir/include/mlir/Dialect/SVE/SVEDialect.h

90 ↗

(On Diff #307898)

Can the types be generated from TableGen? https://mlir.llvm.org/docs/OpDefinitions/#type-definitions
(@rriddle may be able to advise here)

mlir/lib/Target/LLVMIR/LLVMSVEIntr.cpp

63 ↗

(On Diff #307898)

I don't understand why we need a specific conversion here. This concerns me, I see that we have this for "mlir-to-nvvmir" and "avx512-mlir-to-llvmir" but I don't understand why it is desirable. I'd expect the export to LLVM IR to be self-contained and not have many entry-points which don't compose with each others.

This looks to me that this would better be implemented using an op interface or similar mechanism instead.

You're not responsible for setting it up this way, but I'd still like to head from @ftynse / @rriddle on this.

[mehdi_amini]

This is really nice to see this, thanks for the contribution!

[rriddle]

mlir/include/mlir/Dialect/SVE/SVE.td

25 ↗

(On Diff #307827)

Apologies for the terse comments, added proper discussion to the RFC. My thoughts basically echo those of Mehdi, in that if we have something targeting a specific target we should ideally have that in the name similarly to how LLVM intrinsics work.

155 ↗

(On Diff #307827)

Is it possible to write the documentation of this operation in a way that can stand on it's own? I see the lowering to a specific llvm intrinsic as an implementation detail of the lowering. I'd prefer to avoid baking a specific lowering into the semantics of an operation. If we bake in the semantics such that we have "This op matches 1-1 with this llvm intrinsic", the dialect is basically just an extension of the LLVM dialect.

[jsetoain]

mlir/include/mlir/Dialect/SVE/SVE.td

25 ↗

(On Diff #307827)

Don't worry, I took no offence, dry language is fine by me but I wasn't sure what the suggestion was. It's clear now, let's solve this in the RFC :-)

115 ↗

(On Diff #307898)

Alternatively, I can remove the ASCII art. It's not in the original ISA documentation, but I find these "visual" explanations clearer than the formal specification, easier to notice the typical pitfalls with scalable vectors. Would that work as a quick and easy fix?

[jsetoain]

Fixed issue with doc formating.
Changed name of vector scale size operation to a better suited one, fixed documentation accordingly.

[jsetoain]

mlir/include/mlir/Dialect/SVE/SVE.td

155 ↗

(On Diff #307827)

Absolutely. I guess I was being lazy, my apologies. I hope changing the name and focusing on function instead of lowering target will avoid the confusion.

[mehdi_amini]

mlir/include/mlir/Dialect/SVE/SVE.td

115 ↗

(On Diff #307898)

ASCII art is fine, I was just pointing out that we need to surround it in-between something like <pre> / </pre> for it to properly display on the website later.

[jsetoain]

Switching to ODS for type generation

[jsetoain]

mlir/include/mlir/Dialect/SVE/SVEDialect.h

90 ↗

(On Diff #307898)

As it turns out, they can (I quite like it). Is this a new thing? I completely missed it and I noticed nobody else is using it. I've noticed something a bit odd, I still need to do the def SVE_ScalableVectorType : DialectType<SVE_Dialect, thing, but I don't see why TypeDef class can't do that automatically. Am I misusing it?

[mehdi_amini]

mlir/include/mlir/Dialect/SVE/SVEDialect.h

90 ↗

(On Diff #307898)

It is fairly recent (2 months maybe?)

I don't know for the other question, @rriddle will be able to help here.

[jsetoain]

Make operations signless.
Rename dialect and move it to a different namespace.

[nicolasvasilache]

Once my comments are addressed I am happy to land this.
I think the additional layer of tablegen discussed in https://llvm.discourse.group/t/exposing-llvm-ops-to-the-mlir-type-system/2290/11 and having everything in the LLVM dialect is a cross-cutting change that should be done separately and unified across all 3 dialects (AVX512, ArmNeon, ArmSVE).

mlir/include/mlir/Conversion/Passes.td

375

@aartbik improved the conversion to drop one-off passes.
See the changes to https://reviews.llvm.org/D92171 after I rebased on those.
Basically the ConvertVectorToLLVMPass gains a target-specific bit and conditionally includes the dependent dialects as well as the relevant conversion patterns.

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

108

Not sure what the proper source of truth is here but in the Neon dialect I just copied a part of the description and referred to https://developer.arm.com/architectures/instruction-sets/simd-isas/neon/intrinsics

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVEDialect.h

14

To silence the lint warnings, please spell out the whole path using _ in place of /.
e.g. MLIR_DIALECT_TARGET_ARMSVE_ARMSVEDIALECT_H

[nicolasvasilache]

mlir/include/mlir/Dialect/SVE/SVE.td

155 ↗

(On Diff #307827)

@rriddle see https://llvm.discourse.group/t/exposing-llvm-ops-to-the-mlir-type-system/2290/11, ideally we would only have a single dialect and an extra layer of tablegen but that should be done separately.

mlir/lib/Target/LLVMIR/LLVMSVEIntr.cpp

63 ↗

(On Diff #307898)

As usual with MLIR, this is on-demand and the avx512 use case was not justification enough.
We should now layer this better indeed but this is a cross cutting change that is better done separately.
I would not condition landing this on that particular change.

[nicolasvasilache]

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

152

Let's make everything signless at this level, you could write something along the lines of
signless integer that the op interprets as signed/ unsigned as appropriate.

[mehdi_amini]

mlir/lib/Target/LLVMIR/LLVMSVEIntr.cpp

63 ↗

(On Diff #307898)

Are you committing to this refactoring? If you'd like to land this before it then I'd like to see a TODO in the code and a commitment from someone to make it happen "soon".

[jsetoain]

Fixed lint problems.
Improved documentation.
Merged with D92614, doesn't require it's own lowering pass anymore.

[jsetoain]

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

108

I used the ArmARM, but I've switched to a source similar to yours. I think it's clearer that way.

152

Are you proposing to get rid of the distinction between smmla & ummla or is it just about the wording? If it's the former, I'm not sure how I'd go about differentiating between both cases if we come from signless MLIR. If it's the later, let me know if this works or you want me to make it clearer.

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVEDialect.h

14

I've fixed this one and the other one. Apologies for the overlook.

[aartbik]

Nice to see all vector dialects coming together!

mlir/include/mlir/Conversion/Passes.td

399

SVE -> ArmSVE

405

note that Nicolas made changes here too; race condition on rebasing :-)

419

rather not a default of true here ;-)

[nicolasvasilache]

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

152

Just the wording, in line with this: https://llvm.discourse.group/t/rfc-vector-dialects-neon-and-sve/2284/12.
For smmla you'd write signless integer that the op interprets as signed.
For ummla you'd write signless integer that the op interprets as unsigned.

[jsetoain]

Fixed default value for enabling ArmSVE extensions in vector

[jsetoain]

Improve documentation around signedness of operations.

[jsetoain]

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

152

Yes, I saw that. Hopefully this wording is clearer.

[jsetoain]

mlir/include/mlir/Conversion/Passes.td

405

I noticed that while I was following your patch for AVX512. Unless there's a canonical way to deal with this, I'll take silver medal and deal with it once D92171 has landed :-)

[jsetoain]

Rebased on top of D92171
Dialect syntax homogenized with ArmNeon dialect
Naming convention aligned with ArmNeon dialect
File location aligned with ArmNeon dialect
Conversion code aligned with ArmNeon dialect
Improved type checking
Added extra scalable vector type traits

[aartbik]

LGTM, but leaving the approval to River since he added some blocking feedback earlier, so I want to make sure he feels everything has been addressed

mlir/test/Conversion/ArmSVEToLLVM/convert-to-llvm.mlir

8

putting { on previous line seems a bit more consistent with most of our tests

[jsetoain]

Align test syntax style with common practise

[rriddle]

mlir/include/mlir/Dialect/ArmSVE/ArmSVE.td

82 ↗

(On Diff #311356)

nit: Can you use ShapedType::isDynamic here instead?

90 ↗

(On Diff #311356)

nit: Spell out auto here.

135 ↗

(On Diff #311356)

Is there any sort of ordering to these ops?

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp

37

nit: The order here is different from the one used elsewhere.

mlir/lib/Dialect/ArmSVE/IR/ArmSVEDialect.cpp

41 ↗

(On Diff #311356)

Don't use namespaces for method qualification, use arm_sve:: instead (like above).

mlir/lib/Target/LLVMIR/LLVMArmSVEIntr.cpp

40

Only classes should go in anonymous namespaces, methods should be static and at the top level.

[jsetoain]

Reorder operations and initializations alphabetically
Removed useless code
Clean up unnecessary 'auto' declartion
Fixed namespace usage

[jsetoain]

mlir/include/mlir/Dialect/ArmSVE/ArmSVE.td

82 ↗

(On Diff #311356)

Well... that was embarrassing :">

135 ↗

(On Diff #311356)

I've switched to alphabetical

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp

37

I've changed the others to leave all the vector extensions in alphabetical order

[jsetoain]

I don't have commit access to the repository. Please, commit when ready.

mlir/include/mlir/Conversion/ArmSVEToLLVM/ConvertArmSVEToLLVM.h

+//===- ConvertArmSVEToLLVM.h - Conversion Patterns from Arm SVE to LLVM ---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_CONVERSION_ARMSVETOLLVM_CONVERTARMSVETOLLVM_H_
+#define MLIR_CONVERSION_ARMSVETOLLVM_CONVERTARMSVETOLLVM_H_
+
+namespace mlir {
+
+class LLVMTypeConverter;
+class OwningRewritePatternList;
+
+/// Collect a set of patterns to convert from the Arm SVE dialect to LLVM.
+void populateArmSVEToLLVMConversionPatterns(LLVMTypeConverter &converter,
+                                            OwningRewritePatternList &patterns);
+
+} // namespace mlir
+
+#endif // MLIR_CONVERSION_ARMSVETOLLVM_CONVERTARMSVETOLLVM_H_

mlir/include/mlir/Conversion/Passes.td

def ConvertStandardToSPIRV : Pass<"convert-std-to-spirv", "ModuleOp"> {
   let constructor = "mlir::createConvertStandardToSPIRVPass()";
   let dependentDialects = ["spirv::SPIRVDialect"];
 }
 
 //===----------------------------------------------------------------------===//
 // VectorToSCF
 //===----------------------------------------------------------------------===//
 
 def ConvertVectorToSCF : FunctionPass<"convert-vector-to-scf"> {
   let summary = "Lower the operations from the vector dialect into the SCF "
                 "dialect";
   let constructor = "mlir::createConvertVectorToSCFPass()";
   let dependentDialects = ["AffineDialect", "scf::SCFDialect"];
   let options = [
     Option<"fullUnroll", "full-unroll", "bool", /*default=*/"false",
            "Perform full unrolling when converting vector transfers to SCF">,
   ];
 }
 
 //===----------------------------------------------------------------------===//
 // VectorToLLVM
 //===----------------------------------------------------------------------===//
 
 def ConvertVectorToLLVM : Pass<"convert-vector-to-llvm", "ModuleOp"> {
   let summary = "Lower the operations from the vector dialect into the LLVM "
                 "dialect";
   let description = [{
 
     Convert operations from the vector dialect into the LLVM IR dialect
     operations. The lowering pass provides several options to control
     the kinds of optimizations that are allowed. It also provides options
     that enable the use of one or more architectural-specific dialects
-    (AVX512, Neon, SVE, etc.) in combination with the architectural-neutral
+    (AVX512, Neon, Arm SVE, etc.) in combination with the architectural-neutral
     vector dialect lowering.
 
   }];
   let constructor = "mlir::createConvertVectorToLLVMPass()";
-  let dependentDialects = ["LLVM::LLVMDialect", "LLVM::LLVMAVX512Dialect"];
+  let dependentDialects = ["LLVM::LLVMDialect", "LLVM::LLVMAVX512Dialect",
+                           "LLVM::LLVMArmSVEDialect"];
   let options = [
     Option<"reassociateFPReductions", "reassociate-fp-reductions",
            "bool", /*default=*/"false",
            "Allows llvm to reassociate floating-point reductions for speed">,
     Option<"enableIndexOptimizations", "enable-index-optimizations",
            "bool", /*default=*/"true",
            "Allows compiler to assume indices fit in 32-bit if that yields "
      "faster code">,
     Option<"enableAVX512", "enable-avx512",
            "bool", /*default=*/"false",
            "Enables the use of AVX512 dialect while lowering the vector "
-     "dialect.">
+     "dialect.">,
+    Option<"enableArmSVE", "enable-arm-sve",
+           "bool", /*default=*/"false",
+           "Enables the use of ArmSVE dialect while lowering the vector "
+       "dialect.">
   ];
 }
 
 //===----------------------------------------------------------------------===//
 // VectorToROCDL
 //===----------------------------------------------------------------------===//
 
 def ConvertVectorToROCDL : Pass<"convert-vector-to-rocdl", "ModuleOp"> {

mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h

 
 /// Options to control Vector to LLVM lowering.
 ///
 /// This should kept in sync with VectorToLLVM options defined for the
 /// ConvertVectorToLLVM pass in include/mlir/Conversion/Passes.td
 struct LowerVectorToLLVMOptions {
   LowerVectorToLLVMOptions()
       : reassociateFPReductions(false), enableIndexOptimizations(true),
-        enableAVX512(false) {}
+        enableAVX512(false), enableArmSVE(false) {}
 
   LowerVectorToLLVMOptions &setReassociateFPReductions(bool b) {
     reassociateFPReductions = b;
     return *this;
   }
   LowerVectorToLLVMOptions &setEnableIndexOptimizations(bool b) {
     enableIndexOptimizations = b;
     return *this;
   }
   LowerVectorToLLVMOptions &setEnableAVX512(bool b) {
     enableAVX512 = b;
     return *this;
   }
+  LowerVectorToLLVMOptions &setEnableArmSVE(bool b) {
+    enableArmSVE = b;
+    return *this;
+  }
 
   bool reassociateFPReductions;
   bool enableIndexOptimizations;
   bool enableAVX512;
+  bool enableArmSVE;
 };
 
 /// Collect a set of patterns to convert from Vector contractions to LLVM Matrix
 /// Intrinsics. To lower to assembly, the LLVM flag -lower-matrix-intrinsics
 /// will be needed when invoking LLVM.
 void populateVectorToLLVMMatrixConversionPatterns(
     LLVMTypeConverter &converter, OwningRewritePatternList &patterns);
 

mlir/include/mlir/Dialect/CMakeLists.txt

 add_subdirectory(Affine)
 add_subdirectory(Async)
 add_subdirectory(AVX512)
 add_subdirectory(GPU)
 add_subdirectory(Linalg)
 add_subdirectory(LLVMIR)
 add_subdirectory(OpenACC)
 add_subdirectory(OpenMP)
 add_subdirectory(PDL)
 add_subdirectory(PDLInterp)
 add_subdirectory(Quant)
 add_subdirectory(SCF)
 add_subdirectory(Shape)
 add_subdirectory(SPIRV)
 add_subdirectory(StandardOps)
+add_subdirectory(Target)
 add_subdirectory(Tosa)
 add_subdirectory(Vector)

mlir/include/mlir/Dialect/LLVMIR/CMakeLists.txt

 mlir_tablegen(ROCDLConversions.inc -gen-llvmir-conversions)
 add_public_tablegen_target(MLIRROCDLConversionsIncGen)
 
 add_mlir_dialect(LLVMAVX512 llvm_avx512 LLVMAVX512)
 
 set(LLVM_TARGET_DEFINITIONS LLVMAVX512.td)
 mlir_tablegen(LLVMAVX512Conversions.inc -gen-llvmir-conversions)
 add_public_tablegen_target(MLIRLLVMAVX512ConversionsIncGen)
+
+add_mlir_dialect(LLVMArmSVE llvm_arm_sve LLVMArmSVE)
+
+set(LLVM_TARGET_DEFINITIONS LLVMArmSVE.td)
+mlir_tablegen(LLVMArmSVEConversions.inc -gen-llvmir-conversions)
+add_public_tablegen_target(MLIRLLVMArmSVEConversionsIncGen)

mlir/include/mlir/Dialect/LLVMIR/LLVMArmSVE.td

+//===-- LLVMArmSVE.td - LLVMARMSVE dialect op definitions --*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the basic operations for the LLVMArmSVE dialect.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVMIR_ARM_SVE_OPS
+#define LLVMIR_ARM_SVE_OPS
+
+include "mlir/Dialect/LLVMIR/LLVMOpBase.td"
+
+//===----------------------------------------------------------------------===//
+// LLVMArmSVE dialect definition
+//===----------------------------------------------------------------------===//
+
+def LLVMArmSVE_Dialect : Dialect {
+  let name = "llvm_arm_sve";
+  let cppNamespace = "::mlir::LLVM";
+}
+
+//----------------------------------------------------------------------------//
+// MLIR LLVM Arm SVE intrinsics using the MLIR LLVM Dialect type system
+//----------------------------------------------------------------------------//
+
+class LLVMArmSVE_NonSVEIntrUnaryOverloadedOp<string mnemonic,
+                                             list<OpTrait> traits =[]> :
+  LLVM_IntrOpBase</*Dialect dialect=*/LLVMArmSVE_Dialect,
+                  /*string opName=*/mnemonic,
+                  /*string enumName=*/mnemonic,
+                  /*list<int> overloadedResults=*/[0],
+                  /*list<int> overloadedOperands=*/[], // defined by result overload
+                  /*list<OpTrait> traits=*/traits,
+                  /*int numResults=*/1>;
+
+class LLVMArmSVE_IntrOp<string mnemonic, list<OpTrait> traits = []> :
+  LLVM_IntrOpBase<LLVMArmSVE_Dialect, mnemonic,
+                  "aarch64_sve_" # !subst(".", "_", mnemonic),
+                  [], [], traits, 1>;
+
+class LLVMArmSVE_IntrBinaryOverloadedOp<string mnemonic,
+                                        list<OpTrait> traits = []> :
+  LLVM_IntrOpBase</*Dialect dialect=*/LLVMArmSVE_Dialect,
+                  /*string opName=*/mnemonic,
+                  /*string enumName=*/"aarch64_sve_" # !subst(".", "_", mnemonic),
+                  /*list<int> overloadedResults=*/[0],
+                  /*list<int> overloadedOperands=*/[], // defined by result overload
+                  /*list<OpTrait> traits=*/traits,
+                  /*int numResults=*/1>;
+
+def LLVM_aarch64_sve_ummla :
+  LLVMArmSVE_IntrBinaryOverloadedOp<"ummla">,
+  Arguments<(ins LLVM_Type, LLVM_Type, LLVM_Type)>;
+
+def LLVM_aarch64_sve_smmla :
+  LLVMArmSVE_IntrBinaryOverloadedOp<"smmla">,
+  Arguments<(ins LLVM_Type, LLVM_Type, LLVM_Type)>;
+
+def LLVM_aarch64_sve_sdot :
+  LLVMArmSVE_IntrBinaryOverloadedOp<"sdot">,
+  Arguments<(ins LLVM_Type, LLVM_Type, LLVM_Type)>;
+
+def LLVM_aarch64_sve_udot :
+  LLVMArmSVE_IntrBinaryOverloadedOp<"udot">,
+  Arguments<(ins LLVM_Type, LLVM_Type, LLVM_Type)>;
+
+def LLVM_vector_scale :
+  LLVMArmSVE_NonSVEIntrUnaryOverloadedOp<"vscale">;
+
+#endif // ARM_SVE_OPS

mlir/include/mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h

+//===- LLVMSVEDialect.h - MLIR Dialect for LLVMSVE --------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the Target dialect for LLVMSVE in MLIR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_LLVMIR_LLVMARMSVEDIALECT_H_
+#define MLIR_DIALECT_LLVMIR_LLVMARMSVEDIALECT_H_
+
+#include "mlir/IR/Dialect.h"
+#include "mlir/IR/OpDefinition.h"
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/LLVMIR/LLVMArmSVE.h.inc"
+
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h.inc"
+
+#endif // MLIR_DIALECT_LLVMIR_LLVMARMSVEDIALECT_H_

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

+//===-- ArmSVE.td - ArmSVE dialect operation definitions ---*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the basic operations for the ArmSVE dialect.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef ARM_SVE_OPS
+#define ARM_SVE_OPS
+
+include "mlir/Interfaces/SideEffectInterfaces.td"
+
+//===----------------------------------------------------------------------===//
+// ArmSVE dialect definition
+//===----------------------------------------------------------------------===//
+
+def ArmSVE_Dialect : Dialect {
+  let name = "arm_sve";
+  let cppNamespace = "::mlir::arm_sve";
+  let summary = "Basic dialect to target Arm SVE architectures";
+  let description = [{
+    This dialect contains the definitions necessary to target Arm SVE scalable
+    vector operations, including a scalable vector type and intrinsics for
+    some Arm SVE instructions.
+  }];
+}
+
+//===----------------------------------------------------------------------===//
+// ArmSVE type definitions
+//===----------------------------------------------------------------------===//
+
+def ArmSVE_ScalableVectorType : DialectType<ArmSVE_Dialect,
+    CPred<"$_self.isa<ScalableVectorType>()">,
+              "scalable vector type">,
+    BuildableType<"$_builder.getType<ScalableVectorType>()"> {
+  let typeDescription = [{
+    `arm_sve.vector` represents vectors that will be processed by a scalable
+    vector architecture.
+  }];
+}
+
+class ArmSVE_Type<string name> : TypeDef<ArmSVE_Dialect, name> { }
+
+def ScalableVectorType : ArmSVE_Type<"ScalableVector"> {
+  let mnemonic = "vector";
+
+  let summary = "Scalable vector type";
+
+  let description = [{
+    A type representing scalable length SIMD vectors. Unlike fixed-length SIMD
+    vectors, whose size is constant and known at compile time, scalable
+    vectors' length is constant but determined by the specific hardware at
+    run time.
+  }];
+
+  let parameters = (ins
+    ArrayRefParameter<"int64_t", "Vector shape">:$shape,
+    "Type":$elementType
+  );
+
+  let printer = [{
+    $_printer << "vector<";
+    for (int64_t dim : getShape())
+      $_printer << dim << 'x';
+    $_printer << getElementType() << '>';
+  }];
+
+  let parser = [{
+    VectorType vector;
+    if ($_parser.parseType(vector))
+      return Type();
+    return get(ctxt, vector.getShape(), vector.getElementType());
+  }];
+}
+
+//===----------------------------------------------------------------------===//
+// ArmSVE type traits
+//===----------------------------------------------------------------------===//
+
+def IsScalableVectorTypePred :
+    CPred<"$_self.isa<::mlir::arm_sve::ScalableVectorType>()">;
+
+class ScalableVectorOf<list<Type> allowedTypes> :
+    ContainerType<AnyTypeOf<allowedTypes>, IsScalableVectorTypePred,
+          "$_self.cast<::mlir::arm_sve::ScalableVectorType>().getElementType()",
+          "scalable vector">;
+
+//===----------------------------------------------------------------------===//
+// ArmSVE op definitions
+//===----------------------------------------------------------------------===//
+
+class ArmSVE_Op<string mnemonic, list<OpTrait> traits = []> :
+  Op<ArmSVE_Dialect, mnemonic, traits> {}
+
+def UmmlaOp : ArmSVE_Op<"ummla",
+                [NoSideEffect,
+                AllTypesMatch<["src1", "src2"]>,
+                AllTypesMatch<["acc", "dst"]>,
+              ]> {
+  let summary = "Matrix-matrix mutiply and accumulate op";
+  let description = [{
+    UMMLA: Unsigned integer matrix multiply-accumulate.
+
+    These function maps to the UMMLA instruction. It partitions the inputs into
+    128-bit quadwords, with the first input containing a row-by-row 2×2 matrix
+    of 32-bit unsigned integers, the second input containing a row-by-row 2×8
+    matrix of 8-bit signed integers, and the third input containing a
+    column-by-column 8×2 matrix of 8-bit unsigned integers. For each quadword,
+    they multiply the second input matrix by the third input matrix using
+    natural arithmetic and then add the result to the first input using modular
+    arithmetic.
+
+    Source:
+    https://developer.arm.com/documentation/100987/0000
+  }];
+  // Supports vector<16xi8>.
+  let arguments = (ins
+          ScalableVectorOf<[I32]>:$acc,
+          ScalableVectorOf<[I8]>:$src1,
+          ScalableVectorOf<[I8]>:$src2
+  );
+  let results = (outs ScalableVectorOf<[I32]>:$dst);
+  let assemblyFormat =
+    "$acc `,` $src1 `,` $src2 attr-dict `:` type($src1) `->` type($dst)";
+}
+
+def SmmlaOp : ArmSVE_Op<"smmla",
+                [NoSideEffect,
+                AllTypesMatch<["src1", "src2"]>,
+                AllTypesMatch<["acc", "dst"]>,
+              ]> {
+  let summary = "Matrix-matrix mutiply and accumulate op";
+  let description = [{
+    SMMLA: Signed integer matrix multiply-accumulate.
+
+    These function maps to the SMMLA instruction. It partitions the inputs into
+    128-bit quadwords, with the first input containing a row-by-row 2×2 matrix
+    of 32-bit signed integers, the second input containing a row-by-row 2×8
+    matrix of 8-bit signed integers, and the third input containing a
+    column-by-column 8×2 matrix of 8-bit signed integers. For each quadword,
+    they multiply the second input matrix by the third input matrix using
+    natural arithmetic and then add the result to the first input using modular
+    arithmetic.
+
+    Source:
+    https://developer.arm.com/documentation/100987/0000
+  }];
+  // Supports vector<16 x si8>.
+  let arguments = (ins
+          ScalableVectorOf<[I32]>:$acc,
+          ScalableVectorOf<[I8]>:$src1,
+          ScalableVectorOf<[I8]>:$src2
+  );
+  let results = (outs ScalableVectorOf<[I32]>:$dst);
+  let assemblyFormat =
+    "$acc `,` $src1 `,` $src2 attr-dict `:` type($src1) `->` type($dst)";
+}
+
+def SdotOp : ArmSVE_Op<"sdot",
+               [NoSideEffect,
+               AllTypesMatch<["src1", "src2"]>,
+               AllTypesMatch<["acc", "dst"]>,
+             ]> {
+  let summary = "Vector-vector dot product and accumulate op";
+  let description = [{
+    SDOT: Signed integer addition of dot product.
+
+    This function partitions the second and third vector inputs into groups of
+    four elements. They calculate the dot product of each group (without loss
+    of precision) and then add each result to the overlapping element of the
+    first vector input. This is the version for signed integers.
+
+    Source:
+    https://developer.arm.com/documentation/100987/0000
+  }];
+  // Supports vector<16xi8> and vector<8xi16>.
+  let arguments = (ins
+          ScalableVectorOf<[I32, I64]>:$acc,
+          ScalableVectorOf<[I8, I16]>:$src1,
+          ScalableVectorOf<[I8, I16]>:$src2
+  );
+  let results = (outs ScalableVectorOf<[I32, I64]>:$dst);
+  let assemblyFormat =
+    "$acc `,` $src1 `,` $src2 attr-dict `:` type($src1) `->` type($dst)";
+}
+
+def UdotOp : ArmSVE_Op<"udot",
+               [NoSideEffect,
+               AllTypesMatch<["src1", "src2"]>,
+               AllTypesMatch<["acc", "dst"]>,
+             ]> {
+  let summary = "Vector-vector dot product and accumulate op";
+  let description = [{
+    UDOT: Unsigned integer addition of dot product.
+
+    This function partitions the second and third vector inputs into groups of
+    four elements. They calculate the dot product of each group (without loss
+    of precision) and then add each result to the overlapping element of the
+    first vector input. This is the version for unsigned integers.
+
+    Source:
+    https://developer.arm.com/documentation/100987/0000
+  }];
+  // Supports vector<16xi8> and vector<8xi16>.
+  let arguments = (ins
+          ScalableVectorOf<[I32, I64]>:$acc,
+          ScalableVectorOf<[I8, I16]>:$src1,
+          ScalableVectorOf<[I8, I16]>:$src2
+  );
+  let results = (outs ScalableVectorOf<[I32, I64]>:$dst);
+  let assemblyFormat =
+    "$acc `,` $src1 `,` $src2 attr-dict `:` type($src1) `->` type($dst)";
+}
+
+def VectorScaleOp : ArmSVE_Op<"vector_scale",
+                 [NoSideEffect]> {
+  let summary = "Load vector scale size";
+  let description = [{
+    The vector_scale op returns the scale of the scalable vectors, a positive
+    integer value that is constant at runtime but unknown at compile time.
+    The scale of the vector indicates the multiplicity of the vectors and
+    vector operations. I.e.: an !arm_sve.vector<4xi32> is equivalent to
+    vector_scale consecutive vector<4xi32>; and an operation on an
+    !arm_sve.vector<4xi32> is equivalent to performing that operation vector_scale
+    times, once on each <4xi32> segment of the scalable vector. The vector_scale
+    op can be used to calculate the step in vector-length agnostic (VLA) loops.
+  }];
+  let results = (outs Index:$res);
+  let assemblyFormat =
+    "attr-dict `:` type($res)";
+}
+
+#endif // ARM_SVE_OPS

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVEDialect.h

+//===- ArmSVEDialect.h - MLIR Dialect for Arm SVE ---------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the Target dialect for Arm SVE in MLIR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_TARGET_ARMSVE_ARMSVEDIALECT_H
+#define MLIR_DIALECT_TARGET_ARMSVE_ARMSVEDIALECT_H
+
+#include "mlir/IR/Dialect.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/StandardTypes.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+
+#include "mlir/Dialect/Target/ArmSVE/ArmSVEDialect.h.inc"
+
+#define GET_TYPEDEF_CLASSES
+#include "mlir/Dialect/Target/ArmSVE/ArmSVETypes.h.inc"
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/Target/ArmSVE/ArmSVE.h.inc"
+
+#endif // MLIR_DIALECT_TARGET_ARMSVE_ARMSVEDIALECT_H

mlir/include/mlir/Dialect/Target/ArmSVE/CMakeLists.txt

+add_mlir_dialect(ArmSVE arm_sve ArmSVE)

mlir/include/mlir/Dialect/Target/CMakeLists.txt

+add_subdirectory(ArmSVE)

mlir/include/mlir/InitAllDialects.h

 #ifndef MLIR_INITALLDIALECTS_H_
 #define MLIR_INITALLDIALECTS_H_
 
 #include "mlir/Dialect/AVX512/AVX512Dialect.h"
 #include "mlir/Dialect/Affine/IR/AffineOps.h"
 #include "mlir/Dialect/Async/IR/Async.h"
 #include "mlir/Dialect/GPU/GPUDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMAVX512Dialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/LLVMIR/NVVMDialect.h"
 #include "mlir/Dialect/LLVMIR/ROCDLDialect.h"
 #include "mlir/Dialect/Linalg/IR/LinalgOps.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
 #include "mlir/Dialect/PDL/IR/PDL.h"
 #include "mlir/Dialect/PDLInterp/IR/PDLInterp.h"
 #include "mlir/Dialect/Quant/QuantOps.h"
 #include "mlir/Dialect/SCF/SCF.h"
 #include "mlir/Dialect/SDBM/SDBMDialect.h"
 #include "mlir/Dialect/SPIRV/SPIRVDialect.h"
 #include "mlir/Dialect/Shape/IR/Shape.h"
 #include "mlir/Dialect/StandardOps/IR/Ops.h"
+#include "mlir/Dialect/Target/ArmSVE/ArmSVEDialect.h"
 #include "mlir/Dialect/Tosa/IR/TosaOps.h"
 #include "mlir/Dialect/Vector/VectorOps.h"
 #include "mlir/IR/Dialect.h"
 
 namespace mlir {
 
 // Add all the MLIR dialects to the provided registry.
 inline void registerAllDialects(DialectRegistry &registry) {
   // clang-format off
   registry.insert<acc::OpenACCDialect,
                   AffineDialect,
                   async::AsyncDialect,
                   avx512::AVX512Dialect,
                   gpu::GPUDialect,
                   LLVM::LLVMAVX512Dialect,
                   LLVM::LLVMDialect,
+                  LLVM::LLVMArmSVEDialect,
                   linalg::LinalgDialect,
                   scf::SCFDialect,
                   omp::OpenMPDialect,
                   pdl::PDLDialect,
                   pdl_interp::PDLInterpDialect,
                   quant::QuantizationDialect,
                   spirv::SPIRVDialect,
                   StandardOpsDialect,
+                  arm_sve::ArmSVEDialect,
                   vector::VectorDialect,
                   NVVM::NVVMDialect,
                   ROCDL::ROCDLDialect,
                   SDBMDialect,
                   shape::ShapeDialect,
                   tosa::TosaDialect>();
   // clang-format on
 }
 
 } // namespace mlir
 
 #endif // MLIR_INITALLDIALECTS_H_

mlir/include/mlir/InitAllTranslations.h

 
 void registerFromLLVMIRTranslation();
 void registerFromSPIRVTranslation();
 void registerToLLVMIRTranslation();
 void registerToSPIRVTranslation();
 void registerToNVVMIRTranslation();
 void registerToROCDLIRTranslation();
 void registerAVX512ToLLVMIRTranslation();
+void registerArmSVEToLLVMIRTranslation();
 
 // This function should be called before creating any MLIRContext if one
 // expects all the possible translations to be made available to the context
 // automatically.
 inline void registerAllTranslations() {
   static bool initOnce = []() {
     registerFromLLVMIRTranslation();
     registerFromSPIRVTranslation();
     registerToLLVMIRTranslation();
     registerToSPIRVTranslation();
     registerToNVVMIRTranslation();
     registerToROCDLIRTranslation();
     registerAVX512ToLLVMIRTranslation();
+    registerArmSVEToLLVMIRTranslation();
     return true;
   }();
   (void)initOnce;
 }
 } // namespace mlir
 
 #endif // MLIR_INITALLTRANSLATIONS_H

mlir/lib/Conversion/ArmSVEToLLVM/CMakeLists.txt

+add_mlir_conversion_library(MLIRArmSVEToLLVM
+  ConvertArmSVEToLLVM.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Conversion/ArmSVEToLLVM
+
+  DEPENDS
+  MLIRConversionPassIncGen
+
+  LINK_COMPONENTS
+  Core
+
+  LINK_LIBS PUBLIC
+  MLIRArmSVE
+  MLIRLLVMArmSVE
+  MLIRLLVMIR
+  MLIRStandardToLLVM
+  MLIRTransforms
+  )

mlir/lib/Conversion/ArmSVEToLLVM/ConvertArmSVEToLLVM.cpp

+//===- ConvertArmSVEToLLVM.cpp - Convert Arm SVE to the LLVM dialect ------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Conversion/ArmSVEToLLVM/ConvertArmSVEToLLVM.h"
+#include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h"
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/StandardOps/IR/Ops.h"
+#include "mlir/Dialect/Target/ArmSVE/ArmSVEDialect.h"
+#include "mlir/Dialect/Vector/VectorOps.h"
+#include "mlir/IR/BuiltinDialect.h"
+#include "mlir/IR/PatternMatch.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::arm_sve;
+
+template <typename OpTy>
+static Type getSrc1VectorElementType(OpTy op) {
+  return op.src1()
+      .getType()
+      .template cast<ScalableVectorType>()
+      .getElementType();
+}
+template <typename OpTy>
+static Type getSrc2VectorElementType(OpTy op) {
+  return op.src2()
+      .getType()
+      .template cast<ScalableVectorType>()
+      .getElementType();
+}
+template <typename OpTy>
+static Type getAccVectorElementType(OpTy op) {
+  return op.acc()
+      .getType()
+      .template cast<ScalableVectorType>()
+      .getElementType();
+}
+
+/// Basic lowering implementation for one-to-one rewriting from Arm SVE Ops to
+/// LLVM Dialect Ops. Convert the type of the result to an LLVM type, pass
+/// operands as is, preserve attributes.
+template <typename SourceOp, typename TargetOp>
+static LogicalResult
+matchAndRewriteOneToOne(const ConvertToLLVMPattern &lowering,
+                        LLVMTypeConverter &typeConverter, Operation *op,
+                        ArrayRef<Value> operands,
+                        ConversionPatternRewriter &rewriter) {
+  unsigned numResults = op->getNumResults();
+
+  Type packedType;
+  if (numResults != 0) {
+    packedType = typeConverter.packFunctionResults(op->getResultTypes());
+    if (!packedType)
+      return failure();
+  }
+
+  auto newOp = rewriter.create<TargetOp>(op->getLoc(), packedType, operands,
+                                         op->getAttrs());
+
+  // If the operation produced 0 or 1 result, return them immediately.
+  if (numResults == 0)
+    return rewriter.eraseOp(op), success();
+  if (numResults == 1)
+    return rewriter.replaceOp(op, newOp.getOperation()->getResult(0)),
+           success();
+
+  // Otherwise, it had been converted to an operation producing a structure.
+  // Extract individual results from the structure and return them as list.
+  SmallVector<Value, 4> results;
+  results.reserve(numResults);
+  for (unsigned i = 0; i < numResults; ++i) {
+    auto type = typeConverter.convertType(op->getResult(i).getType());
+    results.push_back(rewriter.create<LLVM::ExtractValueOp>(
+        op->getLoc(), type, newOp.getOperation()->getResult(0),
+        rewriter.getI64ArrayAttr(i)));
+  }
+  rewriter.replaceOp(op, results);
+  return success();
+}
+
+namespace {
+
+struct UmmlaOpConversion : public ConvertToLLVMPattern {
+  explicit UmmlaOpConversion(MLIRContext *context,
+                             LLVMTypeConverter &typeConverter)
+      : ConvertToLLVMPattern(UmmlaOp::getOperationName(), context,
+                             typeConverter) {}
+
+  LogicalResult
+  matchAndRewrite(Operation *op, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (!(getAccVectorElementType(cast<UmmlaOp>(op)).isInteger(32) &&
+          getSrc1VectorElementType(cast<UmmlaOp>(op)).isInteger(8) &&
+          getSrc2VectorElementType(cast<UmmlaOp>(op)).isInteger(8)))
+      return failure();
+    return matchAndRewriteOneToOne<UmmlaOp, LLVM::aarch64_sve_ummla>(
+        *this, *getTypeConverter(), op, operands, rewriter);
+  }
+};
+
+struct SmmlaOpConversion : public ConvertToLLVMPattern {
+  explicit SmmlaOpConversion(MLIRContext *context,
+                             LLVMTypeConverter &typeConverter)
+      : ConvertToLLVMPattern(SmmlaOp::getOperationName(), context,
+                             typeConverter) {}
+
+  LogicalResult
+  matchAndRewrite(Operation *op, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (!(getAccVectorElementType(cast<SmmlaOp>(op)).isInteger(32) &&
+          getSrc1VectorElementType(cast<SmmlaOp>(op)).isInteger(8) &&
+          getSrc2VectorElementType(cast<SmmlaOp>(op)).isInteger(8)))
+      return failure();
+    return matchAndRewriteOneToOne<SmmlaOp, LLVM::aarch64_sve_smmla>(
+        *this, *getTypeConverter(), op, operands, rewriter);
+  }
+};
+
+struct SdotOpConversion : public ConvertToLLVMPattern {
+  explicit SdotOpConversion(MLIRContext *context,
+                            LLVMTypeConverter &typeConverter)
+      : ConvertToLLVMPattern(SdotOp::getOperationName(), context,
+                             typeConverter) {}
+
+  LogicalResult
+  matchAndRewrite(Operation *op, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (!((getAccVectorElementType(cast<SdotOp>(op)).isInteger(32) &&
+           getSrc1VectorElementType(cast<SdotOp>(op)).isInteger(8) &&
+           getSrc2VectorElementType(cast<SdotOp>(op)).isInteger(8)) ||
+          (getAccVectorElementType(cast<SdotOp>(op)).isInteger(64) &&
+           getSrc1VectorElementType(cast<SdotOp>(op)).isInteger(16) &&
+           getSrc2VectorElementType(cast<SdotOp>(op)).isInteger(16))))
+      return failure();
+    return matchAndRewriteOneToOne<SdotOp, LLVM::aarch64_sve_sdot>(
+        *this, *getTypeConverter(), op, operands, rewriter);
+  }
+};
+
+struct UdotOpConversion : public ConvertToLLVMPattern {
+  explicit UdotOpConversion(MLIRContext *context,
+                            LLVMTypeConverter &typeConverter)
+      : ConvertToLLVMPattern(UdotOp::getOperationName(), context,
+                             typeConverter) {}
+
+  LogicalResult
+  matchAndRewrite(Operation *op, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (!((getAccVectorElementType(cast<UdotOp>(op)).isInteger(32) &&
+           getSrc1VectorElementType(cast<UdotOp>(op)).isInteger(8) &&
+           getSrc2VectorElementType(cast<UdotOp>(op)).isInteger(8)) ||
+          (getAccVectorElementType(cast<UdotOp>(op)).isInteger(64) &&
+           getSrc1VectorElementType(cast<UdotOp>(op)).isInteger(16) &&
+           getSrc2VectorElementType(cast<UdotOp>(op)).isInteger(16))))
+      return failure();
+    return matchAndRewriteOneToOne<UdotOp, LLVM::aarch64_sve_udot>(
+        *this, *getTypeConverter(), op, operands, rewriter);
+  }
+};
+
+struct VectorScaleOpConversion : public ConvertToLLVMPattern {
+  explicit VectorScaleOpConversion(MLIRContext *context,
+                                   LLVMTypeConverter &typeConverter)
+      : ConvertToLLVMPattern(VectorScaleOp::getOperationName(), context,
+                             typeConverter) {}
+
+  LogicalResult
+  matchAndRewrite(Operation *op, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    return matchAndRewriteOneToOne<VectorScaleOp, LLVM::vector_scale>(
+        *this, *getTypeConverter(), op, operands, rewriter);
+  }
+};
+
+} // namespace
+
+// Extract an LLVM IR type from the LLVM IR dialect type.
+static LLVM::LLVMType unwrap(Type type) {
+  if (!type)
+    return nullptr;
+  auto *mlirContext = type.getContext();
+  auto wrappedLLVMType = type.dyn_cast<LLVM::LLVMType>();
+  if (!wrappedLLVMType)
+    emitError(UnknownLoc::get(mlirContext),
+              "conversion resulted in a non-LLVM type");
+  return wrappedLLVMType;
+}
+
+static Optional<Type>
+convertScalableVectorTypeToLLVM(ScalableVectorType svType,
+                                LLVMTypeConverter &converter) {
+  auto elementType = unwrap(converter.convertType(svType.getElementType()));
+  if (!elementType)
+    return {};
+
+  auto sVectorType =
+      LLVM::LLVMScalableVectorType::get(elementType, svType.getShape().back());
+  return sVectorType;
+}
+
+/// Populate the given list with patterns that convert from Arm SVE to LLVM.
+void mlir::populateArmSVEToLLVMConversionPatterns(
+    LLVMTypeConverter &converter, OwningRewritePatternList &patterns) {
+  MLIRContext *ctx = converter.getDialect()->getContext();
+  converter.addConversion([&converter](ScalableVectorType svType) {
+    return convertScalableVectorTypeToLLVM(svType, converter);
+  });
+  // clang-format off
+  patterns.insert<UmmlaOpConversion,
+                  SmmlaOpConversion,
+                  UdotOpConversion,
+                  SdotOpConversion,
+                  VectorScaleOpConversion>(ctx, converter);
+  // clang-format on
+}

mlir/lib/Conversion/CMakeLists.txt

 add_subdirectory(SCFToGPU)
 add_subdirectory(SCFToOpenMP)
 add_subdirectory(SCFToSPIRV)
 add_subdirectory(SCFToStandard)
 add_subdirectory(ShapeToStandard)
 add_subdirectory(SPIRVToLLVM)
 add_subdirectory(StandardToLLVM)
 add_subdirectory(StandardToSPIRV)
+add_subdirectory(ArmSVEToLLVM)
 add_subdirectory(VectorToROCDL)
 add_subdirectory(VectorToLLVM)
 add_subdirectory(VectorToSCF)
 add_subdirectory(VectorToSPIRV)

mlir/lib/Conversion/PassDetail.h

 namespace gpu {
 class GPUDialect;
 class GPUModuleOp;
 } // end namespace gpu
 
 namespace LLVM {
 class LLVMDialect;
 class LLVMAVX512Dialect;
+class LLVMArmSVEDialect;
 } // end namespace LLVM
 
 namespace NVVM {
 class NVVMDialect;
 } // end namespace NVVM
 
 namespace omp {
 class OpenMPDialect;

mlir/lib/Conversion/VectorToLLVM/CMakeLists.txt

add_mlir_conversion_library(MLIRVectorToLLVM
 
   LINK_COMPONENTS
   Core
 
   LINK_LIBS PUBLIC
   MLIRAVX512
   MLIRAVX512ToLLVM
   MLIRLLVMAVX512
+  MLIRArmSVE
+  MLIRArmSVEToLLVM
+  MLIRLLVMArmSVE
   MLIRLLVMIR
   MLIRStandardToLLVM
   MLIRTargetLLVMIRModuleTranslation
   MLIRTransforms
   MLIRVector
   )

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp

 //===- VectorToLLVM.cpp - Conversion from Vector to the LLVM dialect ------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h"
 
 #include "../PassDetail.h"
 
 #include "mlir/Conversion/AVX512ToLLVM/ConvertAVX512ToLLVM.h"
+#include "mlir/Conversion/ArmSVEToLLVM/ConvertArmSVEToLLVM.h"
 #include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h"
 #include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVMPass.h"
 #include "mlir/Dialect/AVX512/AVX512Dialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMAVX512Dialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/Target/ArmSVE/ArmSVEDialect.h"
 #include "mlir/Dialect/Vector/VectorOps.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 
 using namespace mlir;
 using namespace mlir::vector;
 
 namespace {
 struct LowerVectorToLLVMPass
     : public ConvertVectorToLLVMBase<LowerVectorToLLVMPass> {
   LowerVectorToLLVMPass(const LowerVectorToLLVMOptions &options) {
     this->reassociateFPReductions = options.reassociateFPReductions;
     this->enableIndexOptimizations = options.enableIndexOptimizations;
     this->enableAVX512 = options.enableAVX512;
+    this->enableArmSVE = options.enableArmSVE;
   }
   void runOnOperation() override;
 };
 } // namespace
 
 void LowerVectorToLLVMPass::runOnOperation() {
   // Perform progressive lowering of operations on slices and
   // all contraction operations. Also applies folding and DCE.
   {
     OwningRewritePatternList patterns;
void LowerVectorToLLVMPass::runOnOperation() {
 
   // Architecture specific augmentations.
   LLVMConversionTarget target(getContext());
   if (enableAVX512) {
     target.addLegalDialect<LLVM::LLVMAVX512Dialect>();
     target.addIllegalDialect<avx512::AVX512Dialect>();
     populateAVX512ToLLVMConversionPatterns(converter, patterns);
   }
+  if (enableArmSVE) {
+    target.addLegalDialect<LLVM::LLVMArmSVEDialect>();
+    target.addIllegalDialect<arm_sve::ArmSVEDialect>();
+    populateArmSVEToLLVMConversionPatterns(converter, patterns);
+  }
 
   if (failed(
           applyPartialConversion(getOperation(), target, std::move(patterns))))
     signalPassFailure();
 }
 
 std::unique_ptr<OperationPass<ModuleOp>>
 mlir::createConvertVectorToLLVMPass(const LowerVectorToLLVMOptions &options) {
   return std::make_unique<LowerVectorToLLVMPass>(options);
 }

mlir/lib/Dialect/CMakeLists.txt

 add_subdirectory(Affine)
 add_subdirectory(Async)
 add_subdirectory(AVX512)
 add_subdirectory(GPU)
 add_subdirectory(Linalg)
 add_subdirectory(LLVMIR)
 add_subdirectory(OpenACC)
 add_subdirectory(OpenMP)
 add_subdirectory(PDL)
 add_subdirectory(PDLInterp)
 add_subdirectory(Quant)
 add_subdirectory(SCF)
 add_subdirectory(SDBM)
 add_subdirectory(Shape)
 add_subdirectory(SPIRV)
 add_subdirectory(StandardOps)
+add_subdirectory(Target)
 add_subdirectory(Tosa)
 add_subdirectory(Vector)
 
 set(LLVM_OPTIONAL_SOURCES
   Traits.cpp
 )
 
 add_mlir_library(MLIRDialect
   Traits.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect
 
   LINK_LIBS PUBLIC
   MLIRIR
   )

mlir/lib/Dialect/LLVMIR/CMakeLists.txt

add_mlir_dialect_library(MLIRLLVMAVX512
   Core
 
   LINK_LIBS PUBLIC
   MLIRIR
   MLIRLLVMIR
   MLIRSideEffectInterfaces
   )
 
+add_mlir_dialect_library(MLIRLLVMArmSVE
+  IR/LLVMArmSVEDialect.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/LLVMIR
+
+  DEPENDS
+  MLIRLLVMArmSVEIncGen
+  MLIRLLVMArmSVEConversionsIncGen
+  intrinsics_gen
+
+  LINK_COMPONENTS
+  AsmParser
+  Core
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  MLIRLLVMIR
+  MLIRSideEffectInterfaces
+  )
+
 add_mlir_dialect_library(MLIRNVVMIR
   IR/NVVMDialect.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/LLVMIR
 
   DEPENDS
   MLIRNVVMOpsIncGen

mlir/lib/Dialect/LLVMIR/IR/LLVMArmSVEDialect.cpp

+//===- LLVMArmSVEDialect.cpp - MLIR LLVMSVE ops implementation ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the LLVMArmSVE dialect and its operations.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/IR/IntrinsicsAArch64.h"
+
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/OpImplementation.h"
+#include "mlir/IR/TypeUtilities.h"
+
+using namespace mlir;
+
+void LLVM::LLVMArmSVEDialect::initialize() {
+  addOperations<
+#define GET_OP_LIST
+#include "mlir/Dialect/LLVMIR/LLVMArmSVE.cpp.inc"
+      >();
+}
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/LLVMIR/LLVMArmSVE.cpp.inc"

mlir/lib/Dialect/Target/ArmSVE/CMakeLists.txt

+add_mlir_dialect_library(MLIRArmSVE
+  IR/ArmSVEDialect.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/ArmSVE
+
+  DEPENDS
+  MLIRArmSVEIncGen
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  MLIRSideEffectInterfaces
+  )

mlir/lib/Dialect/Target/ArmSVE/IR/ArmSVEDialect.cpp

+//===- ArmSVEDialect.cpp - MLIR ARM SVE dialect implementation ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the Arm SVE dialect and its operations.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Target/ArmSVE/ArmSVEDialect.h"
+#include "mlir/Dialect/Vector/VectorOps.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/DialectImplementation.h"
+#include "mlir/IR/OpImplementation.h"
+#include "mlir/IR/StandardTypes.h"
+#include "mlir/IR/TypeUtilities.h"
+#include "llvm/ADT/TypeSwitch.h"
+
+using namespace mlir;
+
+void arm_sve::ArmSVEDialect::initialize() {
+  addOperations<
+#define GET_OP_LIST
+#include "mlir/Dialect/Target/ArmSVE/ArmSVE.cpp.inc"
+      >();
+  addTypes<
+#define GET_TYPEDEF_LIST
+#include "mlir/Dialect/Target/ArmSVE/ArmSVETypes.cpp.inc"
+      >();
+}
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/Target/ArmSVE/ArmSVE.cpp.inc"
+
+#define GET_TYPEDEF_CLASSES
+#include "mlir/Dialect/Target/ArmSVE/ArmSVETypes.cpp.inc"
+
+namespace mlir {
+namespace arm_sve {
+
+//===----------------------------------------------------------------------===//
+// VectorType
+//===----------------------------------------------------------------------===//
+
+Type ArmSVEDialect::parseType(DialectAsmParser &parser) const {
+  llvm::SMLoc typeLoc = parser.getCurrentLocation();
+  auto genType = generatedTypeParser(getContext(), parser, "vector");
+  if (genType != Type())
+    return genType;
+  parser.emitError(typeLoc, "unknown type in Arm SVE dialect");
+  return Type();
+}
+
+void ArmSVEDialect::printType(Type type, DialectAsmPrinter &os) const {
+  if (failed(generatedTypePrinter(type, os)))
+    llvm_unreachable("unexpected 'arm_sve' type kind");
+}
+
+} // namespace arm_sve
+} // namespace mlir

mlir/lib/Dialect/Target/CMakeLists.txt

+add_subdirectory(ArmSVE)

mlir/lib/Target/CMakeLists.txt

add_mlir_translation_library(MLIRTargetROCDLIR
 
   LINK_LIBS PUBLIC
   MLIRGPU
   MLIRIR
   MLIRLLVMIR
   MLIRROCDLIR
   MLIRTargetLLVMIRModuleTranslation
   )
+
+add_mlir_translation_library(MLIRTargetArmSVE
+  LLVMIR/LLVMArmSVEIntr.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Target/LLVMIR
+
+  DEPENDS
+  MLIRLLVMArmSVEConversionsIncGen
+
+  LINK_COMPONENTS
+  Core
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  MLIRLLVMArmSVE
+  MLIRLLVMIR
+  MLIRTargetLLVMIRModuleTranslation
+  )

mlir/lib/Target/LLVMIR/LLVMArmSVEIntr.cpp

+//===- LLVMArmSVEIntr.cpp - Convert MLIR LLVM dialect to LLVM intrinsics --===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements a translation between the MLIR LLVM and Arm SVE dialects
+// and LLVM IR with Arm SVE intrinsics.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEDialect.h"
+#include "mlir/Target/LLVMIR/ModuleTranslation.h"
+#include "mlir/Translation.h"
+#include "llvm/IR/IntrinsicsAArch64.h"
+
+using namespace mlir;
+
+namespace {
+class LLVMArmSVEModuleTranslation : public LLVM::ModuleTranslation {
+  friend LLVM::ModuleTranslation;
+
+public:
+  using LLVM::ModuleTranslation::ModuleTranslation;
+
+protected:
+  LogicalResult convertOperation(Operation &opInst,
+                                 llvm::IRBuilder<> &builder) override {
+#include "mlir/Dialect/LLVMIR/LLVMArmSVEConversions.inc"
+
+    return LLVM::ModuleTranslation::convertOperation(opInst, builder);
+  }
+};
+
+std::unique_ptr<llvm::Module>
+translateLLVMArmSVEModuleToLLVMIR(Operation *m, llvm::LLVMContext &llvmContext,
+                                  StringRef name) {
+  return LLVM::ModuleTranslation::translateModule<LLVMArmSVEModuleTranslation>(
+      m, llvmContext, name);
+}
+} // end namespace
+
+namespace mlir {
+void registerArmSVEToLLVMIRTranslation() {
+  TranslateFromMLIRRegistration reg(
+      "arm-sve-mlir-to-llvmir",
+      [](ModuleOp module, raw_ostream &output) {
+        llvm::LLVMContext llvmContext;
+        auto llvmModule = translateLLVMArmSVEModuleToLLVMIR(
+            module, llvmContext, "LLVMDialectModule");
+        if (!llvmModule)
+          return failure();
+
+        llvmModule->print(output, nullptr);
+        return success();
+      },
+      [](DialectRegistry &registry) {
+        registry.insert<LLVM::LLVMArmSVEDialect, LLVM::LLVMDialect>();
+      });
+}
+} // namespace mlir

mlir/test/Conversion/ArmSVEToLLVM/convert-to-llvm.mlir

+// RUN: mlir-opt %s -convert-vector-to-llvm="enable-arm-sve" | mlir-opt | FileCheck %s
+
+func @arm_sve_sdot(%a: !arm_sve.vector<16xi8>,
+                   %b: !arm_sve.vector<16xi8>,
+                   %c: !arm_sve.vector<4xi32>)
+    -> !arm_sve.vector<4xi32>
+{
+  // CHECK: llvm_arm_sve.sdot
+  %0 = arm_sve.sdot %c, %a, %b :
+               !arm_sve.vector<16xi8> -> !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @arm_sve_smmla(%a: !arm_sve.vector<16xi8>,
+                    %b: !arm_sve.vector<16xi8>,
+                    %c: !arm_sve.vector<4xi32>)
+    -> !arm_sve.vector<4xi32>
+{
+  // CHECK: llvm_arm_sve.smmla
+  %0 = arm_sve.smmla %c, %a, %b :
+               !arm_sve.vector<16xi8> -> !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @arm_sve_udot(%a: !arm_sve.vector<16xi8>,
+                   %b: !arm_sve.vector<16xi8>,
+                   %c: !arm_sve.vector<4xi32>)
+    -> !arm_sve.vector<4xi32>
+{
+  // CHECK: llvm_arm_sve.udot
+  %0 = arm_sve.udot %c, %a, %b :
+               !arm_sve.vector<16xi8> -> !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @arm_sve_ummla(%a: !arm_sve.vector<16xi8>,
+                    %b: !arm_sve.vector<16xi8>,
+                    %c: !arm_sve.vector<4xi32>)
+    -> !arm_sve.vector<4xi32>
+{
+  // CHECK: llvm_arm_sve.ummla
+  %0 = arm_sve.ummla %c, %a, %b :
+               !arm_sve.vector<16xi8> -> !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @get_vector_scale() -> index
+{
+  // CHECK: llvm_arm_sve.vscale
+  %0 = arm_sve.vector_scale : index
+  return %0 : index
+}

mlir/test/Dialect/Target/ArmSVE/roundtrip.mlir

+// RUN: mlir-opt -verify-diagnostics %s | mlir-opt | FileCheck %s
+
+func @arm_sve_sdot(%a: !arm_sve.vector<16xi8>,
+                   %b: !arm_sve.vector<16xi8>,
+                   %c: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32>
+{
+  // CHECK: arm_sve.sdot {{.*}}: !arm_sve.vector<16xi8> -> !arm_sve.vector<4xi32
+  %0 = arm_sve.sdot %c, %a, %b :
+             !arm_sve.vector<16xi8> -> !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @arm_sve_smmla(%a: !arm_sve.vector<16xi8>,
+                    %b: !arm_sve.vector<16xi8>,
+                    %c: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32>
+{
+  // CHECK: arm_sve.smmla {{.*}}: !arm_sve.vector<16xi8> -> !arm_sve.vector<4xi3
+  %0 = arm_sve.smmla %c, %a, %b :
+             !arm_sve.vector<16xi8> -> !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @arm_sve_udot(%a: !arm_sve.vector<16xi8>,
+                   %b: !arm_sve.vector<16xi8>,
+                   %c: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32>
+{
+  // CHECK: arm_sve.udot {{.*}}: !arm_sve.vector<16xi8> -> !arm_sve.vector<4xi32
+  %0 = arm_sve.udot %c, %a, %b :
+             !arm_sve.vector<16xi8> -> !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @arm_sve_ummla(%a: !arm_sve.vector<16xi8>,
+                    %b: !arm_sve.vector<16xi8>,
+                    %c: !arm_sve.vector<4xi32>) -> !arm_sve.vector<4xi32>
+{
+  // CHECK: arm_sve.ummla {{.*}}: !arm_sve.vector<16xi8> -> !arm_sve.vector<4xi3
+  %0 = arm_sve.ummla %c, %a, %b :
+             !arm_sve.vector<16xi8> -> !arm_sve.vector<4xi32>
+  return %0 : !arm_sve.vector<4xi32>
+}
+
+func @get_vector_scale() -> index
+{
+  // CHECK: arm_sve.vector_scale : index
+  %0 = arm_sve.vector_scale : index
+  return %0 : index
+}

mlir/test/Target/armsve.mlir

+// RUN: mlir-opt -verify-diagnostics %s | mlir-opt | mlir-translate --arm-sve-mlir-to-llvmir | FileCheck %s
+
+// CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_sdot
+llvm.func @arm_sve_sdot(%arg0: !llvm.vec<? x 16 x i8>,
+                        %arg1: !llvm.vec<? x 16 x i8>,
+                        %arg2: !llvm.vec<? x 4 x i32>)
+                        -> !llvm.vec<? x 4 x i32>
+{
+  // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.sdot.nxv4i32(<vscale x 4
+  %0 = "llvm_arm_sve.sdot"(%arg2, %arg0, %arg1) :
+    (!llvm.vec<? x 4 x i32>, !llvm.vec<? x 16 x i8>, !llvm.vec<? x 16 x i8>)
+        -> !llvm.vec<? x 4 x i32>
+  llvm.return %0 : !llvm.vec<? x 4 x i32>
+}
+
+// CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_smmla
+llvm.func @arm_sve_smmla(%arg0: !llvm.vec<? x 16 x i8>,
+                         %arg1: !llvm.vec<? x 16 x i8>,
+                         %arg2: !llvm.vec<? x 4 x i32>)
+                         -> !llvm.vec<? x 4 x i32> {
+  // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.smmla.nxv4i32(<vscale x 4
+  %0 = "llvm_arm_sve.smmla"(%arg2, %arg0, %arg1) :
+    (!llvm.vec<? x 4 x i32>, !llvm.vec<? x 16 x i8>, !llvm.vec<? x 16 x i8>)
+        -> !llvm.vec<? x 4 x i32>
+  llvm.return %0 : !llvm.vec<? x 4 x i32>
+}
+
+// CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_udot
+llvm.func @arm_sve_udot(%arg0: !llvm.vec<? x 16 x i8>,
+                        %arg1: !llvm.vec<? x 16 x i8>,
+                        %arg2: !llvm.vec<? x 4 x i32>)
+                        -> !llvm.vec<? x 4 x i32>
+{
+  // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.udot.nxv4i32(<vscale x 4
+  %0 = "llvm_arm_sve.udot"(%arg2, %arg0, %arg1) :
+    (!llvm.vec<? x 4 x i32>, !llvm.vec<? x 16 x i8>, !llvm.vec<? x 16 x i8>)
+        -> !llvm.vec<? x 4 x i32>
+  llvm.return %0 : !llvm.vec<? x 4 x i32>
+}
+
+// CHECK-LABEL: define <vscale x 4 x i32> @arm_sve_ummla
+llvm.func @arm_sve_ummla(%arg0: !llvm.vec<? x 16 x i8>,
+                         %arg1: !llvm.vec<? x 16 x i8>,
+                         %arg2: !llvm.vec<? x 4 x i32>)
+                         -> !llvm.vec<? x 4 x i32> {
+  // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.ummla.nxv4i32(<vscale x 4
+  %0 = "llvm_arm_sve.ummla"(%arg2, %arg0, %arg1) :
+    (!llvm.vec<? x 4 x i32>, !llvm.vec<? x 16 x i8>, !llvm.vec<? x 16 x i8>)
+        -> !llvm.vec<? x 4 x i32>
+  llvm.return %0 : !llvm.vec<? x 4 x i32>
+}
+
+// CHECK-LABEL: define i64 @get_vector_scale()
+llvm.func @get_vector_scale() -> !llvm.i64 {
+  // CHECK: call i64 @llvm.vscale.i64()
+  %0 = "llvm_arm_sve.vscale"() : () -> !llvm.i64
+  llvm.return %0 : !llvm.i64
+}