[mlir][ArmSVE][RFC] Add an ArmSVE dialect

[jsetoain]

Summary

This revision starts an Arm-specific ArmSVE dialect discussed in the discourse RFC thread:

https://llvm.discourse.group/t/rfc-vector-dialects-neon-and-sve/2284

[rriddle]

What discourse thread was this discussed in?

[rriddle]

mlir/include/mlir/Dialect/LLVMIR/LLVMSVE.td

54

Remove commented out code.

mlir/include/mlir/Dialect/SVE/SVE.td

25

SVE seems way too generic of a name for some seemingly specific to arm.

71

I'd prefer you resolve these todos right now.

155

I dont particularly like the idea of hard coding a higher level dialect specifically on what it would target in LLVM.

[nicolasvasilache]

In D92172#2418298, @rriddle wrote:

What discourse thread was this discussed in?

https://llvm.discourse.group/t/rfc-vector-dialects-neon-and-sve/2284 but manually creating http links and synchronizing at sub minute granularity is hard :)

@jsetoain could you please update your commit message with this link?

[rriddle]

In D92172#2418310, @nicolasvasilache wrote:

In D92172#2418298, @rriddle wrote:

What discourse thread was this discussed in?

https://llvm.discourse.group/t/rfc-vector-dialects-neon-and-sve/2284 but manually creating http links and synchronizing at sub minute granularity is hard :)

@jsetoain could you please update your commit message with this link?

Thanks, I just assumed I missed the thread that discussed this.

[jsetoain]

Change to comply with header guard naming style

[jsetoain]

Remove commented code

[jsetoain]

Added missing documentation for SVE intrinsics

[jsetoain]

mlir/include/mlir/Dialect/SVE/SVE.td

25

Do you foresee a problem with this specific acronym or are you suggesting we should implement some sort of vendor-specific (or, more accurately, IP-designer-specific) naming convention for hardware-specific dialects? I'm happy to change the name if it makes more sense, but I'd need some guidance.

155

I'm not sure I understand the issue. Would you mind to elaborate, please? Notice that, in order to generate scalable vector code, we need a primitive that provides, at run-time, the scale of the vector; this would feed into the step of the outer loop. LLVM provides llvm.vscale for this purpose, and it's conveniently architecture-independent. If there are better alternatives, I'm happy to consider them. If I'm missing something, please let me know. Thank you!

[mehdi_amini]

mlir/include/mlir/Dialect/SVE/SVE.td

25

If this is not a vendor agnostic "scalable vector" abstraction but it is specific to the ARM SVE implementation, then what about naming this arm_sve as dialect name instead?

[mehdi_amini]

mlir/include/mlir/Dialect/SVE/SVE.td

116

Nice Ascii art :)

I suspect we need some markdown tweak to make sure it displays well after HTML conversion.

mlir/include/mlir/Dialect/SVE/SVEDialect.h

91

Can the types be generated from TableGen? https://mlir.llvm.org/docs/OpDefinitions/#type-definitions
(@rriddle may be able to advise here)

mlir/lib/Target/LLVMIR/LLVMSVEIntr.cpp

64

I don't understand why we need a specific conversion here. This concerns me, I see that we have this for "mlir-to-nvvmir" and "avx512-mlir-to-llvmir" but I don't understand why it is desirable. I'd expect the export to LLVM IR to be self-contained and not have many entry-points which don't compose with each others.

This looks to me that this would better be implemented using an op interface or similar mechanism instead.

You're not responsible for setting it up this way, but I'd still like to head from @ftynse / @rriddle on this.

[mehdi_amini]

This is really nice to see this, thanks for the contribution!

[rriddle]

mlir/include/mlir/Dialect/SVE/SVE.td

25

Apologies for the terse comments, added proper discussion to the RFC. My thoughts basically echo those of Mehdi, in that if we have something targeting a specific target we should ideally have that in the name similarly to how LLVM intrinsics work.

155

Is it possible to write the documentation of this operation in a way that can stand on it's own? I see the lowering to a specific llvm intrinsic as an implementation detail of the lowering. I'd prefer to avoid baking a specific lowering into the semantics of an operation. If we bake in the semantics such that we have "This op matches 1-1 with this llvm intrinsic", the dialect is basically just an extension of the LLVM dialect.

[jsetoain]

mlir/include/mlir/Dialect/SVE/SVE.td

25

Don't worry, I took no offence, dry language is fine by me but I wasn't sure what the suggestion was. It's clear now, let's solve this in the RFC :-)

116

Alternatively, I can remove the ASCII art. It's not in the original ISA documentation, but I find these "visual" explanations clearer than the formal specification, easier to notice the typical pitfalls with scalable vectors. Would that work as a quick and easy fix?

[jsetoain]

Fixed issue with doc formating.
Changed name of vector scale size operation to a better suited one, fixed documentation accordingly.

[jsetoain]

mlir/include/mlir/Dialect/SVE/SVE.td

155

Absolutely. I guess I was being lazy, my apologies. I hope changing the name and focusing on function instead of lowering target will avoid the confusion.

[mehdi_amini]

mlir/include/mlir/Dialect/SVE/SVE.td

116

ASCII art is fine, I was just pointing out that we need to surround it in-between something like <pre> / </pre> for it to properly display on the website later.

[jsetoain]

Switching to ODS for type generation

[jsetoain]

mlir/include/mlir/Dialect/SVE/SVEDialect.h

91

As it turns out, they can (I quite like it). Is this a new thing? I completely missed it and I noticed nobody else is using it. I've noticed something a bit odd, I still need to do the def SVE_ScalableVectorType : DialectType<SVE_Dialect, thing, but I don't see why TypeDef class can't do that automatically. Am I misusing it?

[mehdi_amini]

mlir/include/mlir/Dialect/SVE/SVEDialect.h

91

It is fairly recent (2 months maybe?)

I don't know for the other question, @rriddle will be able to help here.

[jsetoain]

Make operations signless.
Rename dialect and move it to a different namespace.

[nicolasvasilache]

Once my comments are addressed I am happy to land this.
I think the additional layer of tablegen discussed in https://llvm.discourse.group/t/exposing-llvm-ops-to-the-mlir-type-system/2290/11 and having everything in the LLVM dialect is a cross-cutting change that should be done separately and unified across all 3 dialects (AVX512, ArmNeon, ArmSVE).

mlir/include/mlir/Conversion/Passes.td

386

@aartbik improved the conversion to drop one-off passes.
See the changes to https://reviews.llvm.org/D92171 after I rebased on those.
Basically the ConvertVectorToLLVMPass gains a target-specific bit and conditionally includes the dependent dialects as well as the relevant conversion patterns.

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

107 ↗

(On Diff #309310)

Not sure what the proper source of truth is here but in the Neon dialect I just copied a part of the description and referred to https://developer.arm.com/architectures/instruction-sets/simd-isas/neon/intrinsics

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVEDialect.h

13 ↗

(On Diff #309310)

To silence the lint warnings, please spell out the whole path using _ in place of /.
e.g. MLIR_DIALECT_TARGET_ARMSVE_ARMSVEDIALECT_H

[nicolasvasilache]

mlir/include/mlir/Dialect/SVE/SVE.td

155

@rriddle see https://llvm.discourse.group/t/exposing-llvm-ops-to-the-mlir-type-system/2290/11, ideally we would only have a single dialect and an extra layer of tablegen but that should be done separately.

mlir/lib/Target/LLVMIR/LLVMSVEIntr.cpp

64

As usual with MLIR, this is on-demand and the avx512 use case was not justification enough.
We should now layer this better indeed but this is a cross cutting change that is better done separately.
I would not condition landing this on that particular change.

[nicolasvasilache]

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

151 ↗

(On Diff #309310)

Let's make everything signless at this level, you could write something along the lines of
signless integer that the op interprets as signed/ unsigned as appropriate.

[mehdi_amini]

mlir/lib/Target/LLVMIR/LLVMSVEIntr.cpp

64

Are you committing to this refactoring? If you'd like to land this before it then I'd like to see a TODO in the code and a commitment from someone to make it happen "soon".

[jsetoain]

Fixed lint problems.
Improved documentation.
Merged with D92614, doesn't require it's own lowering pass anymore.

[jsetoain]

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

107 ↗

(On Diff #309310)

I used the ArmARM, but I've switched to a source similar to yours. I think it's clearer that way.

151 ↗

(On Diff #309310)

Are you proposing to get rid of the distinction between smmla & ummla or is it just about the wording? If it's the former, I'm not sure how I'd go about differentiating between both cases if we come from signless MLIR. If it's the later, let me know if this works or you want me to make it clearer.

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVEDialect.h

13 ↗

(On Diff #309310)

I've fixed this one and the other one. Apologies for the overlook.

[aartbik]

Nice to see all vector dialects coming together!

mlir/include/mlir/Conversion/Passes.td

414–415

SVE -> ArmSVE

417

note that Nicolas made changes here too; race condition on rebasing :-)

425

rather not a default of true here ;-)

[nicolasvasilache]

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

151 ↗

(On Diff #309310)

Just the wording, in line with this: https://llvm.discourse.group/t/rfc-vector-dialects-neon-and-sve/2284/12.
For smmla you'd write signless integer that the op interprets as signed.
For ummla you'd write signless integer that the op interprets as unsigned.

[jsetoain]

Fixed default value for enabling ArmSVE extensions in vector

[jsetoain]

Improve documentation around signedness of operations.

[jsetoain]

mlir/include/mlir/Dialect/Target/ArmSVE/ArmSVE.td

151 ↗

(On Diff #309310)

Yes, I saw that. Hopefully this wording is clearer.

[jsetoain]

mlir/include/mlir/Conversion/Passes.td

417

I noticed that while I was following your patch for AVX512. Unless there's a canonical way to deal with this, I'll take silver medal and deal with it once D92171 has landed :-)

[jsetoain]

Rebased on top of D92171
Dialect syntax homogenized with ArmNeon dialect
Naming convention aligned with ArmNeon dialect
File location aligned with ArmNeon dialect
Conversion code aligned with ArmNeon dialect
Improved type checking
Added extra scalable vector type traits

[aartbik]

LGTM, but leaving the approval to River since he added some blocking feedback earlier, so I want to make sure he feels everything has been addressed

mlir/test/Conversion/ArmSVEToLLVM/convert-to-llvm.mlir

7 ↗

(On Diff #311298)

putting { on previous line seems a bit more consistent with most of our tests

[jsetoain]

Align test syntax style with common practise

[rriddle]

mlir/include/mlir/Dialect/ArmSVE/ArmSVE.td

82 ↗

(On Diff #311356)

nit: Can you use ShapedType::isDynamic here instead?

90 ↗

(On Diff #311356)

nit: Spell out auto here.

135 ↗

(On Diff #311356)

Is there any sort of ordering to these ops?

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp

46 ↗

(On Diff #311356)

nit: The order here is different from the one used elsewhere.

mlir/lib/Dialect/ArmSVE/IR/ArmSVEDialect.cpp

41 ↗

(On Diff #311356)

Don't use namespaces for method qualification, use arm_sve:: instead (like above).

mlir/lib/Target/LLVMIR/LLVMArmSVEIntr.cpp

39 ↗

(On Diff #311356)

Only classes should go in anonymous namespaces, methods should be static and at the top level.

[jsetoain]

Reorder operations and initializations alphabetically
Removed useless code
Clean up unnecessary 'auto' declartion
Fixed namespace usage

[jsetoain]

mlir/include/mlir/Dialect/ArmSVE/ArmSVE.td

82 ↗

(On Diff #311356)

Well... that was embarrassing :">

135 ↗

(On Diff #311356)

I've switched to alphabetical

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp

46 ↗

(On Diff #311356)

I've changed the others to leave all the vector extensions in alphabetical order

[jsetoain]

I don't have commit access to the repository. Please, commit when ready.

mlir/include/mlir/Conversion/Passes.h

 #include "mlir/Conversion/LinalgToSPIRV/LinalgToSPIRVPass.h"
 #include "mlir/Conversion/LinalgToStandard/LinalgToStandard.h"
 #include "mlir/Conversion/OpenMPToLLVM/ConvertOpenMPToLLVM.h"
 #include "mlir/Conversion/PDLToPDLInterp/PDLToPDLInterp.h"
 #include "mlir/Conversion/SCFToGPU/SCFToGPUPass.h"
 #include "mlir/Conversion/SCFToOpenMP/SCFToOpenMP.h"
 #include "mlir/Conversion/SCFToStandard/SCFToStandard.h"
 #include "mlir/Conversion/SPIRVToLLVM/ConvertSPIRVToLLVMPass.h"
+#include "mlir/Conversion/SVEToLLVM/ConvertSVEToLLVM.h"
 #include "mlir/Conversion/ShapeToStandard/ShapeToStandard.h"
 #include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVMPass.h"
 #include "mlir/Conversion/StandardToSPIRV/ConvertStandardToSPIRVPass.h"
 #include "mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h"
 #include "mlir/Conversion/VectorToROCDL/VectorToROCDL.h"
 #include "mlir/Conversion/VectorToSCF/VectorToSCF.h"
 #include "mlir/Conversion/VectorToSPIRV/ConvertVectorToSPIRVPass.h"
 

mlir/include/mlir/Conversion/Passes.td

 
 def ConvertStandardToSPIRV : Pass<"convert-std-to-spirv", "ModuleOp"> {
   let summary = "Convert Standard Ops to SPIR-V dialect";
   let constructor = "mlir::createConvertStandardToSPIRVPass()";
   let dependentDialects = ["spirv::SPIRVDialect"];
 }
 
 //===----------------------------------------------------------------------===//
+// SVEToLLVM
+//===----------------------------------------------------------------------===//
+
+def ConvertSVEToLLVM : Pass<"convert-sve-to-llvm", "ModuleOp"> {
+  let summary = "Convert the operations from the sve dialect into the LLVM "
+                "dialect";
+  let constructor = "mlir::createConvertSVEToLLVMPass()";
+  let dependentDialects = ["LLVM::LLVMDialect", "LLVM::LLVMSVEDialect"];
+}
+
+//===----------------------------------------------------------------------===//
 // VectorToSCF
 //===----------------------------------------------------------------------===//
 
 def ConvertVectorToSCF : FunctionPass<"convert-vector-to-scf"> {
   let summary = "Lower the operations from the vector dialect into the SCF "
                 "dialect";
   let constructor = "mlir::createConvertVectorToSCFPass()";
   let dependentDialects = ["AffineDialect", "scf::SCFDialect"];
   let options = [
     Option<"fullUnroll", "full-unroll", "bool", /*default=*/"false",
            "Perform full unrolling when converting vector transfers to SCF">,
   ];
 }
 
 //===----------------------------------------------------------------------===//
 // VectorToLLVM
 //===----------------------------------------------------------------------===//
 
 def ConvertVectorToLLVM : Pass<"convert-vector-to-llvm", "ModuleOp"> {
   let summary = "Lower the operations from the vector dialect into the LLVM "
                 "dialect";
   let constructor = "mlir::createConvertVectorToLLVMPass()";
   let dependentDialects = ["LLVM::LLVMDialect"];
   let options = [
     Option<"reassociateFPReductions", "reassociate-fp-reductions",
            "bool", /*default=*/"false",
            "Allows llvm to reassociate floating-point reductions for speed">,
     Option<"enableIndexOptimizations", "enable-index-optimizations",
            "bool", /*default=*/"true",
            "Allows compiler to assume indices fit in 32-bit if that yields faster code">
   ];
 }
 
 //===----------------------------------------------------------------------===//
 // VectorToROCDL
 //===----------------------------------------------------------------------===//
 
 def ConvertVectorToROCDL : Pass<"convert-vector-to-rocdl", "ModuleOp"> {
   let summary = "Lower the operations from the vector dialect into the ROCDL "
                 "dialect";

mlir/include/mlir/Conversion/SVEToLLVM/ConvertSVEToLLVM.h

+//===- ConvertSVEToLLVM.h - Conversion Patterns from SVE to LLVM ----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_EDGE_CONVERSION_SVETOLLVM_CONVERTSVETOLLVM_H_
+#define MLIR_EDGE_CONVERSION_SVETOLLVM_CONVERTSVETOLLVM_H_
+
+#include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h"
+#include <memory>
+
+namespace mlir {
+class LLVMTypeConverter;
+class ModuleOp;
+template <typename T>
+class OperationPass;
+class OwningRewritePatternList;
+
+//===----------------------------------------------------------------------===//
+// SVE Scalar Vector Type Conversion
+//===----------------------------------------------------------------------===//
+
+class SVETypeConverter : public LLVMTypeConverter {
+public:
+  explicit SVETypeConverter(MLIRContext *ctx);
+};
+
+/// Collect a set of patterns to convert from the SVE dialect to LLVM.
+void populateSVEToLLVMConversionPatterns(SVETypeConverter &converter,
+                                         OwningRewritePatternList &patterns);
+
+/// Create a pass to convert SVE operations to the LLVMIR dialect.
+std::unique_ptr<OperationPass<ModuleOp>> createConvertSVEToLLVMPass();
+
+} // namespace mlir
+
+#endif // MLIR_EDGE_CONVERSION_SVETOLLVM_CONVERTSVETOLLVM_H_

mlir/include/mlir/Dialect/CMakeLists.txt

 add_subdirectory(Affine)
 add_subdirectory(Async)
 add_subdirectory(AVX512)
 add_subdirectory(GPU)
 add_subdirectory(Linalg)
 add_subdirectory(LLVMIR)
 add_subdirectory(OpenACC)
 add_subdirectory(OpenMP)
 add_subdirectory(PDL)
 add_subdirectory(PDLInterp)
 add_subdirectory(Quant)
 add_subdirectory(SCF)
 add_subdirectory(Shape)
 add_subdirectory(SPIRV)
 add_subdirectory(StandardOps)
+add_subdirectory(SVE)
 add_subdirectory(Tosa)
 add_subdirectory(Vector)

mlir/include/mlir/Dialect/LLVMIR/CMakeLists.txt

 mlir_tablegen(ROCDLConversions.inc -gen-llvmir-conversions)
 add_public_tablegen_target(MLIRROCDLConversionsIncGen)
 
 add_mlir_dialect(LLVMAVX512 llvm_avx512 LLVMAVX512)
 
 set(LLVM_TARGET_DEFINITIONS LLVMAVX512.td)
 mlir_tablegen(LLVMAVX512Conversions.inc -gen-llvmir-conversions)
 add_public_tablegen_target(MLIRLLVMAVX512ConversionsIncGen)
+
+add_mlir_dialect(LLVMSVE llvm_sve LLVMSVE)
+
+set(LLVM_TARGET_DEFINITIONS LLVMSVE.td)
+mlir_tablegen(LLVMSVEConversions.inc -gen-llvmir-conversions)
+add_public_tablegen_target(MLIRLLVMSVEConversionsIncGen)

mlir/include/mlir/Dialect/LLVMIR/LLVMSVE.td

+//===-- LLVMSVE.td - LLVMSVE dialect op definitions --------*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the basic operations for the LLVMSVE dialect.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVMIR_SVE_OPS
+#define LLVMIR_SVE_OPS
+
+include "mlir/Dialect/LLVMIR/LLVMOpBase.td"
+
+//===----------------------------------------------------------------------===//
+// LLVMSVE dialect definition
+//===----------------------------------------------------------------------===//
+
+def LLVMSVE_Dialect : Dialect {
+  let name = "llvm_sve";
+  let cppNamespace = "::mlir::LLVM";
+}
+
+//----------------------------------------------------------------------------//
+// MLIR LLVM SVE intrinsics using the MLIR LLVM Dialect type system
+//----------------------------------------------------------------------------//
+
+class LLVMSVE_NonSVEIntrUnaryOverloadedOp<string mnemonic, list<OpTrait> traits =[]> :
+  LLVM_IntrOpBase</*Dialect dialect=*/LLVMSVE_Dialect,
+                  /*string opName=*/mnemonic,
+                  /*string enumName=*/mnemonic,
+                  /*list<int> overloadedResults=*/[0],
+                  /*list<int> overloadedOperands=*/[], // defined by result overload
+                  /*list<OpTrait> traits=*/traits,
+                  /*int numResults=*/1>;
+
+class LLVMSVE_IntrOp<string mnemonic, list<OpTrait> traits = []> :
+  LLVM_IntrOpBase<LLVMSVE_Dialect, mnemonic,
+                  "aarch64_sve_" # !subst(".", "_", mnemonic),
+                  [], [], traits, 1>;
+
+class LLVMSVE_IntrBinaryOverloadedOp<string mnemonic, list<OpTrait> traits = []> :
+  LLVM_IntrOpBase</*Dialect dialect=*/LLVMSVE_Dialect,
+                  /*string opName=*/mnemonic,
+                  /*string enumName=*/"aarch64_sve_" # !subst(".", "_", mnemonic),
+                  /*list<int> overloadedResults=*/[0],
+                  /*list<int> overloadedOperands=*/[], // defined by result overload
+                  /*list<OpTrait> traits=*/traits,
+                  /*int numResults=*/1>;
+
+//def LLVM_sve_fadd :
+//  LLVMSVE_InstrOp<"fadd">,
+//  Arguments<(ins LLVM_Type, LLVM_Type, LLVM_Type)>;
+
+def LLVM_aarch64_sve_ummla :
+  LLVMSVE_IntrBinaryOverloadedOp<"ummla">,
+  Arguments<(ins LLVM_Type, LLVM_Type, LLVM_Type)>;
+
+def LLVM_aarch64_sve_smmla :
+  LLVMSVE_IntrBinaryOverloadedOp<"smmla">,
+  Arguments<(ins LLVM_Type, LLVM_Type, LLVM_Type)>;
+
+def LLVM_aarch64_sve_sdot :
+  LLVMSVE_IntrBinaryOverloadedOp<"sdot">,
+  Arguments<(ins LLVM_Type, LLVM_Type, LLVM_Type)>;
+
+def LLVM_aarch64_sve_udot :
+  LLVMSVE_IntrBinaryOverloadedOp<"udot">,
+  Arguments<(ins LLVM_Type, LLVM_Type, LLVM_Type)>;
+
+def LLVM_vscale :
+  LLVMSVE_NonSVEIntrUnaryOverloadedOp<"vscale">;
+
+#endif // SVE_OPS

mlir/include/mlir/Dialect/LLVMIR/LLVMSVEDialect.h

+//===- LLVMSVEDialect.h - MLIR Dialect for LLVMSVE --------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the Target dialect for LLVMSVE in MLIR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_LLVMIR_LLVMSVEDIALECT_H_
+#define MLIR_DIALECT_LLVMIR_LLVMSVEDIALECT_H_
+
+#include "mlir/IR/Dialect.h"
+#include "mlir/IR/OpDefinition.h"
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/LLVMIR/LLVMSVE.h.inc"
+
+#include "mlir/Dialect/LLVMIR/LLVMSVEDialect.h.inc"
+
+#endif // MLIR_DIALECT_LLVMIR_LLVMSVEDIALECT_H_

mlir/include/mlir/Dialect/SVE/CMakeLists.txt

+add_mlir_dialect(SVE sve SVE)

mlir/include/mlir/Dialect/SVE/SVE.td

+//===-- SVE.td - SVE dialect operation definitions ---------*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the basic operations for the SVE dialect.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef SVE_OPS
+#define SVE_OPS
+
+include "mlir/Interfaces/SideEffectInterfaces.td"
+
+//===----------------------------------------------------------------------===//
+// SVE dialect definition
+//===----------------------------------------------------------------------===//
+
+def SVE_Dialect : Dialect {
+  let name = "sve";
+  let cppNamespace = "::mlir::sve";
+  let summary = "Basic dialect to target Arm SVE architectures";
+  let description = [{
+    This dialect contains the definitions necessary to target SVE scalable
+    vector operations, including a scalable vector type and intrinsics for
+    some SVE instructions.
+  }];
+}
+
+//===----------------------------------------------------------------------===//
+// SVE scalable vector type definition
+//===----------------------------------------------------------------------===//
+
+def SVE_ScalableVectorType : DialectType<SVE_Dialect,
+    CPred<"$_self.isa<ScalableVectorType>()">,
+              "scalable vector type">,
+    BuildableType<"$_builder.getType<ScalableVectorType>()"> {
+  let typeDescription = [{
+    `sve.vector` represents vectors that will be processed by a scalable
+    vector architecture.
+  }];
+}
+
+def IsScalableVectorTypePred :
+    CPred<"$_self.isa<::mlir::sve::ScalableVectorType>()">;
+
+class ScalableVectorOf<list<Type> allowedTypes> :
+    ContainerType<AnyTypeOf<allowedTypes>, IsScalableVectorTypePred,
+              "$_self.cast<::mlir::sve::ScalableVectorType>().getElementType()",
+              "scalable vector">;
+
+//===----------------------------------------------------------------------===//
+// SVE op definitions
+//===----------------------------------------------------------------------===//
+
+class SVE_Op<string mnemonic, list<OpTrait> traits = []> :
+  Op<SVE_Dialect, mnemonic, traits> {}
+
+def UmmlaOp : SVE_Op<"ummla",
+                [NoSideEffect,
+                AllTypesMatch<["src1", "src2"]>,
+                AllTypesMatch<["acc", "dst"]>,
+              ]> {
+  let summary = "Matrix-matrix mutiply and accumulate op";
+  let description = [{
+    The ummla op is an SVE specific op that can lower to the proper LLVMSVE
+    operation: `llvm.aarch64.sve.ummla` instruction.
+    TODO: Add documentation for ummla.
+  }];
+  // Supports vector<16xi8>.
+  let arguments = (ins
+          ScalableVectorOf<[UI32]>:$acc,
+          ScalableVectorOf<[UI8]>:$src1,
+          ScalableVectorOf<[UI8]>:$src2
+  );
+  let results = (outs ScalableVectorOf<[UI32]>:$dst);
+  let assemblyFormat =
+    "$acc `,` $src1 `,` $src2 attr-dict `:` type($src1) `->` type($dst)";
+}
+
+def SmmlaOp : SVE_Op<"smmla",
+                [NoSideEffect,
+                AllTypesMatch<["src1", "src2"]>,
+                AllTypesMatch<["acc", "dst"]>,
+              ]> {
+  let summary = "Matrix-matrix mutiply and accumulate op";
+  let description = [{
+    The ummla op is an SVE specific op that can lower to the proper LLVMSVE
+    operation: `llvm.aarch64.sve.smmla` instruction.
+    TODO: Add documentation for smmla.
+  }];
+  // Supports vector<16 x i8>.
+  let arguments = (ins
+          ScalableVectorOf<[SI32]>:$acc,
+          ScalableVectorOf<[SI8]>:$src1,
+          ScalableVectorOf<[SI8]>:$src2
+  );
+  let results = (outs ScalableVectorOf<[SI32]>:$dst);
+  let assemblyFormat =
+    "$acc `,` $src1 `,` $src2 attr-dict `:` type($src1) `->` type($dst)";
+}
+
+def SdotOp : SVE_Op<"sdot",
+               [NoSideEffect,
+               AllTypesMatch<["src1", "src2"]>,
+               AllTypesMatch<["acc", "dst"]>,
+             ]> {
+  let summary = "Vector-vector dot product and accumulate op";
+  let description = [{
+    The ummla op is an SVE specific op that can lower to the proper LLVMSVE
+    operation: `llvm.aarch64.sve.sdot`
+    TODO: Add documentation for sdot.
+  }];
+  // Supports vector<16xi8> and vector<8xi16>.
+  let arguments = (ins
+          ScalableVectorOf<[SI32, SI64]>:$acc,
+          ScalableVectorOf<[SI8, SI16]>:$src1,
+          ScalableVectorOf<[SI8, SI16]>:$src2
+  );
+  let results = (outs ScalableVectorOf<[SI32, SI64]>:$dst);
+  let assemblyFormat =
+    "$acc `,` $src1 `,` $src2 attr-dict `:` type($src1) `->` type($dst)";
+}
+
+def UdotOp : SVE_Op<"udot",
+               [NoSideEffect,
+               AllTypesMatch<["src1", "src2"]>,
+               AllTypesMatch<["acc", "dst"]>,
+             ]> {
+  let summary = "Vector-vector dot product and accumulate op";
+  let description = [{
+    The ummla op is an SVE specific op that can lower to the proper LLVMSVE
+    operation: `llvm.aarch64.sve.udot`
+    TODO: Add documentation for sdot.
+  }];
+  // Supports vector<16xi8> and vector<8xi16>.
+  let arguments = (ins
+          ScalableVectorOf<[UI32, UI64]>:$acc,
+          ScalableVectorOf<[UI8, UI16]>:$src1,
+          ScalableVectorOf<[UI8, UI16]>:$src2
+  );
+  let results = (outs ScalableVectorOf<[UI32, UI64]>:$dst);
+  let assemblyFormat =
+    "$acc `,` $src1 `,` $src2 attr-dict `:` type($src1) `->` type($dst)";
+}
+
+def VscaleOp : SVE_Op<"vscale",
+                 [NoSideEffect]> {
+  let summary = "Load vector scale size";
+  let description = [{
+    The vscale op is a scalable vector architecture intrinsic that will
+    lower to `llvm.vscale`. vscale returns the scaling factor for
+    scalable vectors.
+  }];
+  let results = (outs Index:$res);
+  let assemblyFormat =
+    "attr-dict `:` type($res)";
+}
+
+#endif // SVE_OPS

mlir/include/mlir/Dialect/SVE/SVEDialect.h

+//===- SVEDialect.h - MLIR Dialect for SVE ----------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the Target dialect for SVE in MLIR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_SVE_SVEDIALECT_H_
+#define MLIR_DIALECT_SVE_SVEDIALECT_H_
+
+#include "mlir/IR/Dialect.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/StandardTypes.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+
+namespace mlir {
+namespace sve {
+
+///===----------------------------------------------------------------------===//
+///=== ScalableVectorType
+///===----------------------------------------------------------------------===//
+
+struct ScalableVectorTypeStorage : public TypeStorage {
+  ScalableVectorTypeStorage(unsigned shapeSize, Type elementTy,
+                            const int64_t *shapeElements)
+      : shapeElements(shapeElements), shapeSize(shapeSize),
+        elementType(elementTy) {}
+
+  /// Hash key for uniquing
+  using KeyTy = std::pair<ArrayRef<int64_t>, Type>;
+  bool operator==(const KeyTy &key) const {
+    return key == KeyTy(getShape(), elementType);
+  }
+
+  /// Construction.
+  static ScalableVectorTypeStorage *construct(TypeStorageAllocator &allocator,
+                                              const KeyTy &key) {
+    // Copy the shape into the bump pointer.
+    ArrayRef<int64_t> shape = allocator.copyInto(key.first);
+
+    // Initialize the memory using placement new.
+    return new (allocator.allocate<ScalableVectorTypeStorage>())
+        ScalableVectorTypeStorage(shape.size(), key.second, shape.data());
+  }
+
+  ArrayRef<int64_t> getShape() const {
+    return ArrayRef<int64_t>(shapeElements, shapeSize);
+  }
+
+  const int64_t *shapeElements;
+  unsigned shapeSize;
+  Type elementType;
+};
+
+/// Scalable vector types represent multi-dimensional SIMD vectors that will be
+/// processed by a scalable vector length processor. They have a fixed
+/// known constant shape with one or more dimensions.
+class ScalableVectorType : public Type::TypeBase<ScalableVectorType, ShapedType,
+                                                 ScalableVectorTypeStorage> {
+public:
+  using Base::Base;
+
+  /// Get or create a new ScalableVectorType of the provided shape and element
+  /// type. Assumes the arguments define a well-formed ScalableVectorType.
+  static ScalableVectorType get(ArrayRef<int64_t> shape, Type elementType);
+
+  /// Get or create a new ScalableVectorType of the provided shape and element
+  /// type declared at the given, potentially unknown, location. If the
+  /// ScalableVectorType defined by the arguments would be ill-formed, emit
+  /// errors and return nullptr-wrapping type.
+  static ScalableVectorType getChecked(ArrayRef<int64_t> shape,
+                                       Type elementType, Location location);
+
+  static LogicalResult verifyConstructionInvariants(Location loc,
+                                                    ArrayRef<int64_t> shape,
+                                                    Type elementType);
+
+  static bool isValidElementType(Type t) {
+    return t.isa<IntegerType, FloatType>();
+  }
+
+  ArrayRef<int64_t> getShape() const;
+
+  Type getElementType() const;
+};
+
+} // namespace sve
+} // namespace mlir
+
+#include "mlir/Dialect/SVE/SVEDialect.h.inc"
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/SVE/SVE.h.inc"
+
+#endif // MLIR_DIALECT_SVE_SVEDIALECT_H_

mlir/include/mlir/InitAllDialects.h

 #define MLIR_INITALLDIALECTS_H_
 
 #include "mlir/Dialect/AVX512/AVX512Dialect.h"
 #include "mlir/Dialect/Affine/IR/AffineOps.h"
 #include "mlir/Dialect/Async/IR/Async.h"
 #include "mlir/Dialect/GPU/GPUDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMAVX512Dialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMSVEDialect.h"
 #include "mlir/Dialect/LLVMIR/NVVMDialect.h"
 #include "mlir/Dialect/LLVMIR/ROCDLDialect.h"
 #include "mlir/Dialect/Linalg/IR/LinalgOps.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
 #include "mlir/Dialect/PDL/IR/PDL.h"
 #include "mlir/Dialect/PDLInterp/IR/PDLInterp.h"
 #include "mlir/Dialect/Quant/QuantOps.h"
 #include "mlir/Dialect/SCF/SCF.h"
 #include "mlir/Dialect/SDBM/SDBMDialect.h"
 #include "mlir/Dialect/SPIRV/SPIRVDialect.h"
+#include "mlir/Dialect/SVE/SVEDialect.h"
 #include "mlir/Dialect/Shape/IR/Shape.h"
 #include "mlir/Dialect/StandardOps/IR/Ops.h"
 #include "mlir/Dialect/Tosa/IR/TosaOps.h"
 #include "mlir/Dialect/Vector/VectorOps.h"
 #include "mlir/IR/Dialect.h"
 
 namespace mlir {
 
 // Add all the MLIR dialects to the provided registry.
 inline void registerAllDialects(DialectRegistry &registry) {
   // clang-format off
   registry.insert<acc::OpenACCDialect,
                   AffineDialect,
                   async::AsyncDialect,
                   avx512::AVX512Dialect,
                   gpu::GPUDialect,
                   LLVM::LLVMAVX512Dialect,
                   LLVM::LLVMDialect,
+                  LLVM::LLVMSVEDialect,
                   linalg::LinalgDialect,
                   scf::SCFDialect,
                   omp::OpenMPDialect,
                   pdl::PDLDialect,
                   pdl_interp::PDLInterpDialect,
                   quant::QuantizationDialect,
                   spirv::SPIRVDialect,
                   StandardOpsDialect,
+                  sve::SVEDialect,
                   vector::VectorDialect,
                   NVVM::NVVMDialect,
                   ROCDL::ROCDLDialect,
                   SDBMDialect,
                   shape::ShapeDialect,
                   tosa::TosaDialect>();
   // clang-format on
 }
 
 } // namespace mlir
 
 #endif // MLIR_INITALLDIALECTS_H_

mlir/include/mlir/InitAllTranslations.h

 
 void registerFromLLVMIRTranslation();
 void registerFromSPIRVTranslation();
 void registerToLLVMIRTranslation();
 void registerToSPIRVTranslation();
 void registerToNVVMIRTranslation();
 void registerToROCDLIRTranslation();
 void registerAVX512ToLLVMIRTranslation();
+void registerSVEToLLVMIRTranslation();
 
 // This function should be called before creating any MLIRContext if one
 // expects all the possible translations to be made available to the context
 // automatically.
 inline void registerAllTranslations() {
   static bool initOnce = []() {
     registerFromLLVMIRTranslation();
     registerFromSPIRVTranslation();
     registerToLLVMIRTranslation();
     registerToSPIRVTranslation();
     registerToNVVMIRTranslation();
     registerToROCDLIRTranslation();
     registerAVX512ToLLVMIRTranslation();
+    registerSVEToLLVMIRTranslation();
     return true;
   }();
   (void)initOnce;
 }
 } // namespace mlir
 
 #endif // MLIR_INITALLTRANSLATIONS_H

mlir/lib/Conversion/CMakeLists.txt

 add_subdirectory(SCFToGPU)
 add_subdirectory(SCFToOpenMP)
 add_subdirectory(SCFToSPIRV)
 add_subdirectory(SCFToStandard)
 add_subdirectory(ShapeToStandard)
 add_subdirectory(SPIRVToLLVM)
 add_subdirectory(StandardToLLVM)
 add_subdirectory(StandardToSPIRV)
+add_subdirectory(SVEToLLVM)
 add_subdirectory(VectorToROCDL)
 add_subdirectory(VectorToLLVM)
 add_subdirectory(VectorToSCF)
 add_subdirectory(VectorToSPIRV)

mlir/lib/Conversion/PassDetail.h

 namespace gpu {
 class GPUDialect;
 class GPUModuleOp;
 } // end namespace gpu
 
 namespace LLVM {
 class LLVMDialect;
 class LLVMAVX512Dialect;
+class LLVMSVEDialect;
 } // end namespace LLVM
 
 namespace NVVM {
 class NVVMDialect;
 } // end namespace NVVM
 
 namespace omp {
 class OpenMPDialect;

mlir/lib/Conversion/SVEToLLVM/CMakeLists.txt

+add_mlir_conversion_library(MLIRSVEToLLVM
+  ConvertSVEToLLVM.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Conversion/SVEToLLVM
+
+  DEPENDS
+  MLIRConversionPassIncGen
+
+  LINK_COMPONENTS
+  Core
+
+  LINK_LIBS PUBLIC
+  MLIRSVE
+  MLIRLLVMSVE
+  MLIRLLVMIR
+  MLIRStandardToLLVM
+  MLIRTransforms
+  )

mlir/lib/Conversion/SVEToLLVM/ConvertSVEToLLVM.cpp

+//===- ConvertSVEToLLVM.cpp - Convert SVE to the LLVM dialect -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Conversion/SVEToLLVM/ConvertSVEToLLVM.h"
+
+#include "../PassDetail.h"
+#include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h"
+#include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVMPass.h"
+#include "mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h"
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMSVEDialect.h"
+#include "mlir/Dialect/SVE/SVEDialect.h"
+#include "mlir/Dialect/StandardOps/IR/Ops.h"
+#include "mlir/Dialect/Vector/VectorOps.h"
+#include "mlir/IR/BuiltinDialect.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Pass/Pass.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::sve;
+
+template <typename OpTy>
+static Type getSrc1VectorElementType(OpTy op) {
+  return op.src1()
+      .getType()
+      .template cast<ScalableVectorType>()
+      .getElementType();
+}
+template <typename OpTy>
+static Type getSrc2VectorElementType(OpTy op) {
+  return op.src2()
+      .getType()
+      .template cast<ScalableVectorType>()
+      .getElementType();
+}
+template <typename OpTy>
+static Type getAccVectorElementType(OpTy op) {
+  return op.acc()
+      .getType()
+      .template cast<ScalableVectorType>()
+      .getElementType();
+}
+
+/// Basic lowering implementation for one-to-one rewriting from SVE Ops to
+/// LLVM Dialect Ops. Convert the type of the result to an LLVM type, pass
+/// operands as is, preserve attributes.
+template <typename SourceOp, typename TargetOp>
+static LogicalResult
+matchAndRewriteOneToOne(const ConvertToLLVMPattern &lowering,
+                        LLVMTypeConverter &typeConverter, Operation *op,
+                        ArrayRef<Value> operands,
+                        ConversionPatternRewriter &rewriter) {
+  unsigned numResults = op->getNumResults();
+
+  Type packedType;
+  if (numResults != 0) {
+    packedType = typeConverter.packFunctionResults(op->getResultTypes());
+    if (!packedType)
+      return failure();
+  }
+
+  auto newOp = rewriter.create<TargetOp>(op->getLoc(), packedType, operands,
+                                         op->getAttrs());
+
+  // If the operation produced 0 or 1 result, return them immediately.
+  if (numResults == 0)
+    return rewriter.eraseOp(op), success();
+  if (numResults == 1)
+    return rewriter.replaceOp(op, newOp.getOperation()->getResult(0)),
+           success();
+
+  // Otherwise, it had been converted to an operation producing a structure.
+  // Extract individual results from the structure and return them as list.
+  SmallVector<Value, 4> results;
+  results.reserve(numResults);
+  for (unsigned i = 0; i < numResults; ++i) {
+    auto type = typeConverter.convertType(op->getResult(i).getType());
+    results.push_back(rewriter.create<LLVM::ExtractValueOp>(
+        op->getLoc(), type, newOp.getOperation()->getResult(0),
+        rewriter.getI64ArrayAttr(i)));
+  }
+  rewriter.replaceOp(op, results);
+  return success();
+}
+
+namespace {
+
+struct UmmlaOpConversion : public ConvertToLLVMPattern {
+  explicit UmmlaOpConversion(MLIRContext *context,
+                             LLVMTypeConverter &typeConverter)
+      : ConvertToLLVMPattern(UmmlaOp::getOperationName(), context,
+                             typeConverter) {}
+
+  LogicalResult
+  matchAndRewrite(Operation *op, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (!(getAccVectorElementType(cast<UmmlaOp>(op)).isInteger(32) &&
+          getSrc1VectorElementType(cast<UmmlaOp>(op)).isInteger(8) &&
+          getSrc2VectorElementType(cast<UmmlaOp>(op)).isInteger(8)))
+      return failure();
+    return matchAndRewriteOneToOne<UmmlaOp, LLVM::aarch64_sve_ummla>(
+        *this, this->typeConverter, op, operands, rewriter);
+  }
+};
+
+struct SmmlaOpConversion : public ConvertToLLVMPattern {
+  explicit SmmlaOpConversion(MLIRContext *context,
+                             LLVMTypeConverter &typeConverter)
+      : ConvertToLLVMPattern(SmmlaOp::getOperationName(), context,
+                             typeConverter) {}
+
+  LogicalResult
+  matchAndRewrite(Operation *op, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (!(getAccVectorElementType(cast<SmmlaOp>(op)).isInteger(32) &&
+          getSrc1VectorElementType(cast<SmmlaOp>(op)).isInteger(8) &&
+          getSrc2VectorElementType(cast<SmmlaOp>(op)).isInteger(8)))
+      return failure();
+    return matchAndRewriteOneToOne<SmmlaOp, LLVM::aarch64_sve_smmla>(
+        *this, this->typeConverter, op, operands, rewriter);
+  }
+};
+
+struct SdotOpConversion : public ConvertToLLVMPattern {
+  explicit SdotOpConversion(MLIRContext *context,
+                            LLVMTypeConverter &typeConverter)
+      : ConvertToLLVMPattern(SdotOp::getOperationName(), context,
+                             typeConverter) {}
+
+  LogicalResult
+  matchAndRewrite(Operation *op, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (!((getAccVectorElementType(cast<SdotOp>(op)).isInteger(32) &&
+           getSrc1VectorElementType(cast<SdotOp>(op)).isInteger(8) &&
+           getSrc2VectorElementType(cast<SdotOp>(op)).isInteger(8)) ||
+          (getAccVectorElementType(cast<SdotOp>(op)).isInteger(64) &&
+           getSrc1VectorElementType(cast<SdotOp>(op)).isInteger(16) &&
+           getSrc2VectorElementType(cast<SdotOp>(op)).isInteger(16))))
+      return failure();
+    return matchAndRewriteOneToOne<SdotOp, LLVM::aarch64_sve_sdot>(
+        *this, this->typeConverter, op, operands, rewriter);
+  }
+};
+
+struct UdotOpConversion : public ConvertToLLVMPattern {
+  explicit UdotOpConversion(MLIRContext *context,
+                            LLVMTypeConverter &typeConverter)
+      : ConvertToLLVMPattern(UdotOp::getOperationName(), context,
+                             typeConverter) {}
+
+  LogicalResult
+  matchAndRewrite(Operation *op, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (!((getAccVectorElementType(cast<UdotOp>(op)).isInteger(32) &&
+           getSrc1VectorElementType(cast<UdotOp>(op)).isInteger(8) &&
+           getSrc2VectorElementType(cast<UdotOp>(op)).isInteger(8)) ||
+          (getAccVectorElementType(cast<UdotOp>(op)).isInteger(64) &&
+           getSrc1VectorElementType(cast<UdotOp>(op)).isInteger(16) &&
+           getSrc2VectorElementType(cast<UdotOp>(op)).isInteger(16))))
+      return failure();
+    return matchAndRewriteOneToOne<UdotOp, LLVM::aarch64_sve_udot>(
+        *this, this->typeConverter, op, operands, rewriter);
+  }
+};
+
+struct VscaleOpConversion : public ConvertToLLVMPattern {
+  explicit VscaleOpConversion(MLIRContext *context,
+                              LLVMTypeConverter &typeConverter)
+      : ConvertToLLVMPattern(VscaleOp::getOperationName(), context,
+                             typeConverter) {}
+
+  LogicalResult
+  matchAndRewrite(Operation *op, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    return matchAndRewriteOneToOne<VscaleOp, LLVM::vscale>(
+        *this, this->typeConverter, op, operands, rewriter);
+  }
+};
+
+} // namespace
+
+/// Populate the given list with patterns that convert from SVE to LLVM.
+void mlir::populateSVEToLLVMConversionPatterns(
+    SVETypeConverter &converter, OwningRewritePatternList &patterns) {
+  MLIRContext *ctx = converter.getDialect()->getContext();
+  // clang-format off
+  patterns.insert<UmmlaOpConversion,
+                  SmmlaOpConversion,
+                  UdotOpConversion,
+                  SdotOpConversion,
+                  VscaleOpConversion>(ctx, converter);
+  // clang-format on
+}
+
+namespace {
+struct ConvertSVEToLLVMPass
+    : public ConvertSVEToLLVMBase<ConvertSVEToLLVMPass> {
+  void runOnOperation() override;
+};
+} // namespace
+
+// Extract an LLVM IR type from the LLVM IR dialect type.
+static LLVM::LLVMType unwrap(Type type) {
+  if (!type)
+    return nullptr;
+  auto *mlirContext = type.getContext();
+  auto wrappedLLVMType = type.dyn_cast<LLVM::LLVMType>();
+  if (!wrappedLLVMType)
+    emitError(UnknownLoc::get(mlirContext),
+              "conversion resulted in a non-LLVM type");
+  return wrappedLLVMType;
+}
+
+static Optional<Type> convertScalableVectorType(ScalableVectorType svType,
+                                                LLVMTypeConverter &converter) {
+  auto elementType = unwrap(converter.convertType(svType.getElementType()));
+  if (!elementType)
+    return {};
+
+  auto sVectorType =
+      LLVM::LLVMScalableVectorType::get(elementType, svType.getShape().back());
+  return sVectorType;
+}
+
+SVETypeConverter::SVETypeConverter(MLIRContext *ctx) : LLVMTypeConverter(ctx) {
+  addConversion([this](ScalableVectorType svType) {
+    return convertScalableVectorType(svType, *this);
+  });
+}
+
+void ConvertSVEToLLVMPass::runOnOperation() {
+  OwningRewritePatternList patterns;
+  SVETypeConverter sveConverter(&getContext());
+  populateSVEToLLVMConversionPatterns(sveConverter, patterns);
+  populateVectorToLLVMConversionPatterns(sveConverter, patterns);
+  populateStdToLLVMConversionPatterns(sveConverter, patterns);
+
+  ConversionTarget target(getContext());
+  target.addLegalDialect<LLVM::LLVMDialect>();
+  target.addLegalDialect<LLVM::LLVMSVEDialect>();
+  target.addIllegalDialect<sve::SVEDialect>();
+  if (failed(applyPartialConversion(getOperation(), target,
+                                    std::move(patterns)))) {
+    signalPassFailure();
+  }
+}
+
+std::unique_ptr<OperationPass<ModuleOp>> mlir::createConvertSVEToLLVMPass() {
+  return std::make_unique<ConvertSVEToLLVMPass>();
+}

mlir/lib/Dialect/CMakeLists.txt

 add_subdirectory(Affine)
 add_subdirectory(Async)
 add_subdirectory(AVX512)
 add_subdirectory(GPU)
 add_subdirectory(Linalg)
 add_subdirectory(LLVMIR)
 add_subdirectory(OpenACC)
 add_subdirectory(OpenMP)
 add_subdirectory(PDL)
 add_subdirectory(PDLInterp)
 add_subdirectory(Quant)
 add_subdirectory(SCF)
 add_subdirectory(SDBM)
 add_subdirectory(Shape)
 add_subdirectory(SPIRV)
 add_subdirectory(StandardOps)
+add_subdirectory(SVE)
 add_subdirectory(Tosa)
 add_subdirectory(Vector)
 
 set(LLVM_OPTIONAL_SOURCES
   Traits.cpp
 )
 
 add_mlir_library(MLIRDialect
   Traits.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect
 
   LINK_LIBS PUBLIC
   MLIRIR
   )

mlir/lib/Dialect/LLVMIR/CMakeLists.txt

add_mlir_dialect_library(MLIRLLVMAVX512
   Core
 
   LINK_LIBS PUBLIC
   MLIRIR
   MLIRLLVMIR
   MLIRSideEffectInterfaces
   )
 
+add_mlir_dialect_library(MLIRLLVMSVE
+  IR/LLVMSVEDialect.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/LLVMIR
+
+  DEPENDS
+  MLIRLLVMSVEIncGen
+  MLIRLLVMSVEConversionsIncGen
+  intrinsics_gen
+
+  LINK_COMPONENTS
+  AsmParser
+  Core
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  MLIRLLVMIR
+  MLIRSideEffectInterfaces
+  )
+
 add_mlir_dialect_library(MLIRNVVMIR
   IR/NVVMDialect.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/LLVMIR
 
   DEPENDS
   MLIRNVVMOpsIncGen

mlir/lib/Dialect/LLVMIR/IR/LLVMSVEDialect.cpp

+//===- LLVMSVEDialect.cpp - MLIR LLVMSVE ops implementation ---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the LLVMSVE dialect and its operations.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/IR/IntrinsicsAArch64.h"
+
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMSVEDialect.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/OpImplementation.h"
+#include "mlir/IR/TypeUtilities.h"
+
+using namespace mlir;
+
+void LLVM::LLVMSVEDialect::initialize() {
+  addOperations<
+#define GET_OP_LIST
+#include "mlir/Dialect/LLVMIR/LLVMSVE.cpp.inc"
+      >();
+}
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/LLVMIR/LLVMSVE.cpp.inc"

mlir/lib/Dialect/SVE/CMakeLists.txt

+add_mlir_dialect_library(MLIRSVE
+  IR/SVEDialect.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/SVE
+
+  DEPENDS
+  MLIRSVEIncGen
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  MLIRSideEffectInterfaces
+  MLIRVectorToLLVM
+  )

mlir/lib/Dialect/SVE/IR/SVEDialect.cpp

+//===- SVEDialect.cpp - MLIR SVE dialect implementation -------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the SVE dialect and its operations.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SVE/SVEDialect.h"
+#include "mlir/Dialect/Vector/VectorOps.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/DialectImplementation.h"
+#include "mlir/IR/OpImplementation.h"
+#include "mlir/IR/StandardTypes.h"
+#include "mlir/IR/TypeUtilities.h"
+#include "llvm/ADT/TypeSwitch.h"
+
+using namespace mlir;
+
+void sve::SVEDialect::initialize() {
+  addOperations<
+#define GET_OP_LIST
+#include "mlir/Dialect/SVE/SVE.cpp.inc"
+      >();
+  addTypes<ScalableVectorType>();
+}
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/SVE/SVE.cpp.inc"
+
+namespace mlir {
+namespace sve {
+
+//===----------------------------------------------------------------------===//
+// VectorType
+//===----------------------------------------------------------------------===//
+
+ScalableVectorType ScalableVectorType::get(ArrayRef<int64_t> shape,
+                                           Type elementType) {
+  return Base::get(elementType.getContext(), shape, elementType);
+}
+
+ScalableVectorType ScalableVectorType::getChecked(ArrayRef<int64_t> shape,
+                                                  Type elementType,
+                                                  Location location) {
+  return Base::getChecked(location, shape, elementType);
+}
+
+ArrayRef<int64_t> ScalableVectorType::getShape() const {
+  return getImpl()->getShape();
+}
+
+Type ScalableVectorType::getElementType() const {
+  return getImpl()->elementType;
+}
+
+LogicalResult ScalableVectorType::verifyConstructionInvariants(
+    Location loc, ArrayRef<int64_t> shape, Type elementType) {
+  if (shape.empty())
+    return emitError(loc,
+                     "scalable vector types must have at least one dimension");
+
+  if (!isValidElementType(elementType))
+    return emitError(loc, "vector elements must be int or float type");
+
+  if (any_of(shape, [](int64_t i) { return i <= 0; }))
+    return emitError(loc, "vector types must have positive constant sizes");
+
+  return success();
+}
+
+Type SVEDialect::parseType(DialectAsmParser &parser) const {
+  llvm::SMLoc typeLoc = parser.getCurrentLocation();
+  VectorType vectorTypeType;
+  if (parser.parseType(vectorTypeType)) {
+    parser.emitError(typeLoc, "unknown type in SVE dialect");
+    return Type();
+  }
+  return ScalableVectorType::get(vectorTypeType.getShape(),
+                                 vectorTypeType.getElementType());
+}
+
+void SVEDialect::printType(Type type, DialectAsmPrinter &os) const {
+  TypeSwitch<Type>(type)
+      .Case<ScalableVectorType>([&](ScalableVectorType svTy) {
+        os << "vector<";
+        for (int64_t dim : svTy.getShape())
+          os << dim << 'x';
+        os << svTy.getElementType() << '>';
+      })
+      .Default([](Type) { llvm_unreachable("unexpected 'sve' type kind"); });
+}
+
+} // namespace sve
+} // namespace mlir

mlir/lib/Target/CMakeLists.txt

add_mlir_translation_library(MLIRTargetROCDLIR
 
   LINK_LIBS PUBLIC
   MLIRGPU
   MLIRIR
   MLIRLLVMIR
   MLIRROCDLIR
   MLIRTargetLLVMIRModuleTranslation
   )
+
+add_mlir_translation_library(MLIRTargetSVE
+  LLVMIR/LLVMSVEIntr.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Target/LLVMIR
+
+  DEPENDS
+  MLIRLLVMSVEConversionsIncGen
+
+  LINK_COMPONENTS
+  Core
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  MLIRLLVMSVE
+  MLIRLLVMIR
+  MLIRTargetLLVMIRModuleTranslation
+  )

mlir/lib/Target/LLVMIR/LLVMSVEIntr.cpp

+//===- LLVMSVEIntr.cpp - Convert MLIR LLVM dialect to LLVM intrinsics -----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements a translation between the MLIR LLVM and SVE dialects and
+// LLVM IR with SVE intrinsics.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/LLVMIR/LLVMSVEDialect.h"
+#include "mlir/Target/LLVMIR/ModuleTranslation.h"
+#include "mlir/Translation.h"
+#include "llvm/IR/IntrinsicsAArch64.h"
+
+using namespace mlir;
+
+namespace {
+class LLVMSVEModuleTranslation : public LLVM::ModuleTranslation {
+  friend LLVM::ModuleTranslation;
+
+public:
+  using LLVM::ModuleTranslation::ModuleTranslation;
+
+protected:
+  LogicalResult convertOperation(Operation &opInst,
+                                 llvm::IRBuilder<> &builder) override {
+#include "mlir/Dialect/LLVMIR/LLVMSVEConversions.inc"
+
+    return LLVM::ModuleTranslation::convertOperation(opInst, builder);
+  }
+};
+
+std::unique_ptr<llvm::Module>
+translateLLVMSVEModuleToLLVMIR(Operation *m, llvm::LLVMContext &llvmContext,
+                               StringRef name) {
+  return LLVM::ModuleTranslation::translateModule<LLVMSVEModuleTranslation>(
+      m, llvmContext, name);
+}
+} // end namespace
+
+namespace mlir {
+void registerSVEToLLVMIRTranslation() {
+  TranslateFromMLIRRegistration reg(
+      "sve-mlir-to-llvmir",
+      [](ModuleOp module, raw_ostream &output) {
+        llvm::LLVMContext llvmContext;
+        auto llvmModule = translateLLVMSVEModuleToLLVMIR(module, llvmContext,
+                                                         "LLVMDialectModule");
+        if (!llvmModule)
+          return failure();
+
+        llvmModule->print(output, nullptr);
+        return success();
+      },
+      [](DialectRegistry &registry) {
+        registry.insert<LLVM::LLVMSVEDialect, LLVM::LLVMDialect>();
+      });
+}
+} // namespace mlir

mlir/test/Conversion/SVEToLLVM/convert-to-llvm.mlir

+// RUN: mlir-opt %s -convert-sve-to-llvm | mlir-opt | FileCheck %s
+
+func @sve_sdot(%a: !sve.vector<16xsi8>,
+               %b: !sve.vector<16xsi8>,
+               %c: !sve.vector<4xsi32>)
+    -> !sve.vector<4xsi32>
+{
+  // CHECK: llvm_sve.sdot
+  %0 = sve.sdot %c, %a, %b : !sve.vector<16xsi8> -> !sve.vector<4xsi32>
+  return %0 : !sve.vector<4xsi32>
+}
+
+func @sve_smmla(%a: !sve.vector<16xsi8>,
+                %b: !sve.vector<16xsi8>,
+                %c: !sve.vector<4xsi32>)
+    -> !sve.vector<4xsi32>
+{
+  // CHECK: llvm_sve.smmla
+  %0 = sve.smmla %c, %a, %b : !sve.vector<16xsi8> -> !sve.vector<4xsi32>
+  return %0 : !sve.vector<4xsi32>
+}
+
+func @sve_udot(%a: !sve.vector<16xui8>,
+               %b: !sve.vector<16xui8>,
+               %c: !sve.vector<4xui32>)
+    -> !sve.vector<4xui32>
+{
+  // CHECK: llvm_sve.udot
+  %0 = sve.udot %c, %a, %b : !sve.vector<16xui8> -> !sve.vector<4xui32>
+  return %0 : !sve.vector<4xui32>
+}
+
+func @sve_ummla(%a: !sve.vector<16xui8>,
+                %b: !sve.vector<16xui8>,
+                %c: !sve.vector<4xui32>)
+    -> !sve.vector<4xui32>
+{
+  // CHECK: llvm_sve.ummla
+  %0 = sve.ummla %c, %a, %b : !sve.vector<16xui8> -> !sve.vector<4xui32>
+  return %0 : !sve.vector<4xui32>
+}
+
+func @get_vscale() -> index
+{
+  // CHECK: llvm_sve.vscale
+  %0 = sve.vscale : index
+  return %0 : index
+}

mlir/test/Dialect/SVE/roundtrip.mlir

+// RUN: mlir-opt -verify-diagnostics %s | mlir-opt | FileCheck %s
+
+func @sve_sdot(%a: !sve.vector<16xsi8>,
+               %b: !sve.vector<16xsi8>,
+               %c: !sve.vector<4xsi32>) -> !sve.vector<4xsi32>
+{
+  // CHECK: sve.sdot {{.*}}: !sve.vector<16xsi8> -> !sve.vector<4xsi32>
+  %0 = sve.sdot %c, %a, %b : !sve.vector<16xsi8> -> !sve.vector<4xsi32>
+  return %0 : !sve.vector<4xsi32>
+}
+
+func @sve_smmla(%a: !sve.vector<16xsi8>,
+                %b: !sve.vector<16xsi8>,
+                %c: !sve.vector<4xsi32>) -> !sve.vector<4xsi32>
+{
+  // CHECK: sve.smmla {{.*}}: !sve.vector<16xsi8> -> !sve.vector<4xsi32>
+  %0 = sve.smmla %c, %a, %b : !sve.vector<16xsi8> -> !sve.vector<4xsi32>
+  return %0 : !sve.vector<4xsi32>
+}
+
+func @sve_udot(%a: !sve.vector<16xui8>,
+               %b: !sve.vector<16xui8>,
+               %c: !sve.vector<4xui32>) -> !sve.vector<4xui32>
+{
+  // CHECK: sve.udot {{.*}}: !sve.vector<16xui8> -> !sve.vector<4xui32>
+  %0 = sve.udot %c, %a, %b : !sve.vector<16xui8> -> !sve.vector<4xui32>
+  return %0 : !sve.vector<4xui32>
+}
+
+func @sve_ummla(%a: !sve.vector<16xui8>,
+                %b: !sve.vector<16xui8>,
+                %c: !sve.vector<4xui32>) -> !sve.vector<4xui32>
+{
+  // CHECK: sve.ummla {{.*}}: !sve.vector<16xui8> -> !sve.vector<4xui32>
+  %0 = sve.ummla %c, %a, %b : !sve.vector<16xui8> -> !sve.vector<4xui32>
+  return %0 : !sve.vector<4xui32>
+}
+
+func @get_vscale() -> index
+{
+  // CHECK: sve.vscale : index
+  %0 = sve.vscale : index
+  return %0 : index
+}

mlir/test/Target/sve.mlir

+// RUN: mlir-opt -verify-diagnostics %s | mlir-opt | mlir-translate --sve-mlir-to-llvmir | FileCheck %s
+
+// CHECK-LABEL: define <vscale x 4 x i32> @sve_sdot
+llvm.func @sve_sdot(%arg0: !llvm.vec<? x 16 x i8>,
+                    %arg1: !llvm.vec<? x 16 x i8>,
+                    %arg2: !llvm.vec<? x 4 x i32>)
+  -> !llvm.vec<? x 4 x i32>
+{
+  // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.sdot.nxv4i32(<vscale x 4
+  %0 = "llvm_sve.sdot"(%arg2, %arg0, %arg1) :
+    (!llvm.vec<? x 4 x i32>, !llvm.vec<? x 16 x i8>, !llvm.vec<? x 16 x i8>)
+        -> !llvm.vec<? x 4 x i32>
+  llvm.return %0 : !llvm.vec<? x 4 x i32>
+}
+
+// CHECK-LABEL: define <vscale x 4 x i32> @sve_smmla
+llvm.func @sve_smmla(%arg0: !llvm.vec<? x 16 x i8>,
+                     %arg1: !llvm.vec<? x 16 x i8>,
+                     %arg2: !llvm.vec<? x 4 x i32>)
+                     -> !llvm.vec<? x 4 x i32> {
+  // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.smmla.nxv4i32(<vscale x 4
+  %0 = "llvm_sve.smmla"(%arg2, %arg0, %arg1) :
+    (!llvm.vec<? x 4 x i32>, !llvm.vec<? x 16 x i8>, !llvm.vec<? x 16 x i8>)
+        -> !llvm.vec<? x 4 x i32>
+  llvm.return %0 : !llvm.vec<? x 4 x i32>
+}
+
+// CHECK-LABEL: define <vscale x 4 x i32> @sve_udot
+llvm.func @sve_udot(%arg0: !llvm.vec<? x 16 x i8>,
+                    %arg1: !llvm.vec<? x 16 x i8>,
+                    %arg2: !llvm.vec<? x 4 x i32>)
+  -> !llvm.vec<? x 4 x i32>
+{
+  // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.udot.nxv4i32(<vscale x 4
+  %0 = "llvm_sve.udot"(%arg2, %arg0, %arg1) :
+    (!llvm.vec<? x 4 x i32>, !llvm.vec<? x 16 x i8>, !llvm.vec<? x 16 x i8>)
+        -> !llvm.vec<? x 4 x i32>
+  llvm.return %0 : !llvm.vec<? x 4 x i32>
+}
+
+// CHECK-LABEL: define <vscale x 4 x i32> @sve_ummla
+llvm.func @sve_ummla(%arg0: !llvm.vec<? x 16 x i8>,
+                     %arg1: !llvm.vec<? x 16 x i8>,
+                     %arg2: !llvm.vec<? x 4 x i32>)
+                     -> !llvm.vec<? x 4 x i32> {
+  // CHECK: call <vscale x 4 x i32> @llvm.aarch64.sve.ummla.nxv4i32(<vscale x 4
+  %0 = "llvm_sve.ummla"(%arg2, %arg0, %arg1) :
+    (!llvm.vec<? x 4 x i32>, !llvm.vec<? x 16 x i8>, !llvm.vec<? x 16 x i8>)
+        -> !llvm.vec<? x 4 x i32>
+  llvm.return %0 : !llvm.vec<? x 4 x i32>
+}
+
+// CHECK-LABEL: define i64 @get_vscale()
+llvm.func @get_vscale() -> !llvm.i64 {
+  // CHECK: call i64 @llvm.vscale.i64()
+  %0 = "llvm_sve.vscale"() : () -> !llvm.i64
+  llvm.return %0 : !llvm.i64
+}