[mlir][ArmNeon][RFC] Add an ArmNeon dialect

[nicolasvasilache]

Summary

This revision starts an Arm-specific ArmNeon dialect discussed in the discourse RFC thread.

[rriddle]

What thread was this discussed in?

[nicolasvasilache]

Update commit message.

[rriddle]

mlir/include/mlir/Dialect/Neon/Neon.td

41 ↗

(On Diff #307820)

Please resolve these TODOs in this commit.

mlir/lib/IR/StandardTypes.cpp

303 ↗

(On Diff #307820)

Use isa here instead.

[aartbik]

mlir/lib/Conversion/NeonToLLVM/ConvertNeonToLLVM.cpp

112 ↗

(On Diff #307828)

This (and other parts) is directly taken from the AVX512 pass. For most part, that is okay with me, but this actually lowering part in AVX512 reflected in these lines of code did not compose very well with the regular vector dialect lowering (which has grown over time with e.g. contract lowering and friends).

Perhaps we can make this part a bit more forward looking on how we want this to run.

mlir/lib/Target/LLVMIR/LLVMNeonIntr.cpp

10 ↗

(On Diff #307828)

AVX -> Neon ;-)

[aartbik]

mlir/lib/Conversion/NeonToLLVM/ConvertNeonToLLVM.cpp

112 ↗

(On Diff #307828)

As discussed in the syn meeting, let's not make this a full pass (and also remove the AVX512 pass), but simply keep this as "populate" methods that are pulled in on demand by the regular vector dialect pass. That will avoid copy-and-pasting the logic and avoid inconsistencies over time. The tests can have their own driver, they don't need a full pass either.

As agreed, I will do the AVX512 pass removal in the meanwhile to set the right example.

[aartbik]

AVX512 restructuring landed as https://reviews.llvm.org/D92614
Let's follow that structure for all future architectural-dependent vector dialects....

[nicolasvasilache]

Address and rebase

[nicolasvasilache]

Update commit message

[nicolasvasilache]

CMake

[aartbik]

mlir/include/mlir/Conversion/ArmNeonToLLVM/ArmNeonToLLVM.h

10

I see a lint warning on this line. Not sure if it is useful?

mlir/include/mlir/Conversion/Passes.td

404–405

yes, this is a good point, we should only pull in what is needed

mlir/include/mlir/Dialect/ArmNeon/ArmNeonDialect.h

3

this line is way too long now

[mehdi_amini]

mlir/include/mlir/Conversion/Passes.td

407

Just don't add it here and implement the override method manually? That way you can check for the options.

[ftynse]

mlir/lib/Conversion/ArmNeonToLLVM/ArmNeonToLLVM.cpp

25

Could you specify what are the specific adaptations?

70

I don't see "different possible target ops" in the code below. Maybe you could rather take/expose OneToOneLLVMConversionPattern?

[nicolasvasilache]

Address and rebase

[nicolasvasilache]

mlir/lib/Conversion/ArmNeonToLLVM/ArmNeonToLLVM.cpp

25

just old crap, dropped, thanks!

70

thanks, much better, paid some of that AVX512 debt too.

[nicolasvasilache]

Fixes post rebase.

[nicolasvasilache]

CMake

[nicolasvasilache]

.

[ftynse]

mlir/include/mlir/Dialect/LLVMIR/LLVMArmNeon.td

2

This looks too long

mlir/lib/Dialect/LLVMIR/IR/LLVMArmNeonDialect.cpp

2

Overflow comment

[nicolasvasilache]

Line length.

[silvas]

Did we actually get consensus around the direction here? I didn't see a clear conclusion.

mlir/include/mlir/Conversion/ArmNeonToLLVM/ArmNeonToLLVM.h

+//===- ArmNeonToLLVM.h - Conversion Patterns from ArmNeon to LLVM ---------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_CONVERSION_ARMNEONTOLLVM_NEONTOLLVM_H_
+#define MLIR_CONVERSION_ARMNEONTOLLVM_NEONTOLLVM_H_
+
+namespace mlir {
+
+class LLVMTypeConverter;
+class OwningRewritePatternList;
+
+/// Collect a set of patterns to convert from theArmNeon dialect to LLVM.
+void populateArmNeonToLLVMConversionPatterns(
+    LLVMTypeConverter &converter, OwningRewritePatternList &patterns);
+
+} // namespace mlir
+
+#endif // MLIR_CONVERSION_ARMNEONTOLLVM_NEONTOLLVM_H_

mlir/include/mlir/Conversion/Passes.td

def ConvertVectorToLLVM : Pass<"convert-vector-to-llvm", "ModuleOp"> {
   let summary = "Lower the operations from the vector dialect into the LLVM "
                 "dialect";
   let description = [{
 
     Convert operations from the vector dialect into the LLVM IR dialect
     operations. The lowering pass provides several options to control
     the kinds of optimizations that are allowed. It also provides options
     that enable the use of one or more architectural-specific dialects
-    (AVX512, Neon, SVE, etc.) in combination with the architectural-neutral
+    (AVX512, ArmNeon, SVE, etc.) in combination with the architectural-neutral
     vector dialect lowering.
 
   }];
   let constructor = "mlir::createConvertVectorToLLVMPass()";
-  let dependentDialects = ["LLVM::LLVMDialect", "LLVM::LLVMAVX512Dialect"];
+  // Override explicitly in C++ to allow conditional dialect dependence.
+  // let dependentDialects;
   let options = [
     Option<"reassociateFPReductions", "reassociate-fp-reductions",
            "bool", /*default=*/"false",
            "Allows llvm to reassociate floating-point reductions for speed">,
     Option<"enableIndexOptimizations", "enable-index-optimizations",
            "bool", /*default=*/"true",
            "Allows compiler to assume indices fit in 32-bit if that yields "
      "faster code">,
     Option<"enableAVX512", "enable-avx512",
            "bool", /*default=*/"false",
            "Enables the use of AVX512 dialect while lowering the vector "
+     "dialect.">,
+    Option<"enableArmNeon", "enable-arm-neon",
+           "bool", /*default=*/"false",
+           "Enables the use of ArmNeon dialect while lowering the vector "
      "dialect.">
   ];
 }
 
 //===----------------------------------------------------------------------===//
 // VectorToROCDL
 //===----------------------------------------------------------------------===//
 

mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h

 
 /// Options to control Vector to LLVM lowering.
 ///
 /// This should kept in sync with VectorToLLVM options defined for the
 /// ConvertVectorToLLVM pass in include/mlir/Conversion/Passes.td
 struct LowerVectorToLLVMOptions {
   LowerVectorToLLVMOptions()
       : reassociateFPReductions(false), enableIndexOptimizations(true),
-        enableAVX512(false) {}
+        enableArmNeon(false), enableAVX512(false) {}
 
   LowerVectorToLLVMOptions &setReassociateFPReductions(bool b) {
     reassociateFPReductions = b;
     return *this;
   }
   LowerVectorToLLVMOptions &setEnableIndexOptimizations(bool b) {
     enableIndexOptimizations = b;
     return *this;
   }
   LowerVectorToLLVMOptions &setEnableAVX512(bool b) {
     enableAVX512 = b;
     return *this;
   }
+  LowerVectorToLLVMOptions &setEnableArmNeon(bool b) {
+    enableArmNeon = b;
+    return *this;
+  }
 
   bool reassociateFPReductions;
   bool enableIndexOptimizations;
+  bool enableArmNeon;
   bool enableAVX512;
 };
 
 /// Collect a set of patterns to convert from Vector contractions to LLVM Matrix
 /// Intrinsics. To lower to assembly, the LLVM flag -lower-matrix-intrinsics
 /// will be needed when invoking LLVM.
 void populateVectorToLLVMMatrixConversionPatterns(
     LLVMTypeConverter &converter, OwningRewritePatternList &patterns);

mlir/include/mlir/Dialect/ArmNeon/ArmNeon.td

+//===-- ArmNeonOps.td - ArmNeon dialect op definitions -----*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the basic operations for the ArmNeon dialect.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef ARMNEON_OPS
+#define ARMNEON_OPS
+
+include "mlir/Interfaces/SideEffectInterfaces.td"
+
+//===----------------------------------------------------------------------===//
+// ArmNeon dialect definition
+//===----------------------------------------------------------------------===//
+
+def ArmNeon_Dialect : Dialect {
+  let name = "arm_neon";
+  let cppNamespace = "::mlir::arm_neon";
+}
+
+//===----------------------------------------------------------------------===//
+// ArmNeon op definitions
+//===----------------------------------------------------------------------===//
+
+class ArmNeon_Op<string mnemonic, list<OpTrait> traits = []> :
+  Op<ArmNeon_Dialect, mnemonic, traits> {}
+
+def SMullOp : ArmNeon_Op<"smull", [NoSideEffect,
+  AllTypesMatch<["a", "b"]>,
+  TypesMatchWith<
+    "res has same vector shape and element bitwidth scaled by 2 as a",
+    "a", "res", "$_self.cast<VectorType>().scaleElementBitwidth(2)">]> {
+  let summary = "smull roundscale op";
+  let description = [{
+    Signed Multiply Long (vector). This instruction multiplies corresponding
+    signed integer values in the lower or upper half of the vectors of the two
+    source SIMD&FP registers, places the results in a vector, and writes the
+    vector to the destination SIMD&FP register.
+
+    Source:
+    https://developer.arm.com/architectures/instruction-sets/simd-isas/neon/intrinsics
+  }];
+  // Supports either:
+  //   (vector<8xi8>, vector<8xi8>) -> (vector<8xi16>)
+  //   (vector<4xi16>, vector<4xi16>) -> (vector<4xi32>)
+  //   (vector<2xi32>, vector<2xi32>) -> (vector<2xi64>)
+  let arguments = (ins VectorOfLengthAndType<[8, 4, 2], [I8, I16, I32]>:$a,
+                       VectorOfLengthAndType<[8, 4, 2], [I8, I16, I32]>:$b);
+  let results = (outs VectorOfLengthAndType<[8, 4, 2], [I16, I32, I64]>:$res);
+  let assemblyFormat =
+    "$a `,` $b attr-dict `:` type($a) `to` type($res)";
+}
+
+#endif // ARMNEON_OPS

mlir/include/mlir/Dialect/ArmNeon/ArmNeonDialect.h

+//===- ArmNeonDialect.h - MLIR Dialect forArmNeon ---------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the Target dialect for ArmNeon in MLIR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_ARMNEON_ARMNEONDIALECT_H_
+#define MLIR_DIALECT_ARMNEON_ARMNEONDIALECT_H_
+
+#include "mlir/IR/Dialect.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+
+#include "mlir/Dialect/ArmNeon/ArmNeonDialect.h.inc"
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/ArmNeon/ArmNeon.h.inc"
+
+#endif // MLIR_DIALECT_ARMNEON_ARMNEONDIALECT_H_

mlir/include/mlir/Dialect/ArmNeon/CMakeLists.txt

+add_mlir_dialect(ArmNeon arm_neon)
+add_mlir_doc(ArmNeon -gen-dialect-doc ArmNeon Dialects/)

mlir/include/mlir/Dialect/CMakeLists.txt

 add_subdirectory(Affine)
 add_subdirectory(Async)
+add_subdirectory(ArmNeon)
 add_subdirectory(AVX512)
 add_subdirectory(GPU)
 add_subdirectory(Linalg)
 add_subdirectory(LLVMIR)
 add_subdirectory(OpenACC)
 add_subdirectory(OpenMP)
 add_subdirectory(PDL)
 add_subdirectory(PDLInterp)
 add_subdirectory(Quant)
 add_subdirectory(SCF)
 add_subdirectory(Shape)
 add_subdirectory(SPIRV)
 add_subdirectory(StandardOps)
 add_subdirectory(Tosa)
 add_subdirectory(Vector)

mlir/include/mlir/Dialect/LLVMIR/CMakeLists.txt

 add_subdirectory(Transforms)
 
 set(LLVM_TARGET_DEFINITIONS LLVMOps.td)
 mlir_tablegen(LLVMOps.h.inc -gen-op-decls)
 mlir_tablegen(LLVMOps.cpp.inc -gen-op-defs)
 mlir_tablegen(LLVMOpsDialect.h.inc -gen-dialect-decls)
 mlir_tablegen(LLVMOpsEnums.h.inc -gen-enum-decls)
 mlir_tablegen(LLVMOpsEnums.cpp.inc -gen-enum-defs)
 add_public_tablegen_target(MLIRLLVMOpsIncGen)
 
-add_mlir_dialect(NVVMOps nvvm)
-add_mlir_doc(NVVMOps -gen-dialect-doc NVVMDialect Dialects/)
-add_mlir_dialect(ROCDLOps rocdl)
-add_mlir_doc(ROCDLOps -gen-dialect-doc ROCDLDialect Dialects/)
-
 set(LLVM_TARGET_DEFINITIONS LLVMOps.td)
 mlir_tablegen(LLVMConversions.inc -gen-llvmir-conversions)
 mlir_tablegen(LLVMConversionEnumsToLLVM.inc -gen-enum-to-llvmir-conversions)
 mlir_tablegen(LLVMConversionEnumsFromLLVM.inc -gen-enum-from-llvmir-conversions)
 add_public_tablegen_target(MLIRLLVMConversionsIncGen)
+
+add_mlir_dialect(NVVMOps nvvm)
+add_mlir_doc(NVVMOps -gen-dialect-doc NVVMDialect Dialects/)
 set(LLVM_TARGET_DEFINITIONS NVVMOps.td)
 mlir_tablegen(NVVMConversions.inc -gen-llvmir-conversions)
 add_public_tablegen_target(MLIRNVVMConversionsIncGen)
+
+add_mlir_dialect(ROCDLOps rocdl)
+add_mlir_doc(ROCDLOps -gen-dialect-doc ROCDLDialect Dialects/)
 set(LLVM_TARGET_DEFINITIONS ROCDLOps.td)
 mlir_tablegen(ROCDLConversions.inc -gen-llvmir-conversions)
 add_public_tablegen_target(MLIRROCDLConversionsIncGen)
 
 add_mlir_dialect(LLVMAVX512 llvm_avx512 LLVMAVX512)
-
+add_mlir_doc(LLVMAVX512 -gen-dialect-doc LLVMAVX512 Dialects/)
 set(LLVM_TARGET_DEFINITIONS LLVMAVX512.td)
 mlir_tablegen(LLVMAVX512Conversions.inc -gen-llvmir-conversions)
 add_public_tablegen_target(MLIRLLVMAVX512ConversionsIncGen)
+
+add_mlir_dialect(LLVMArmNeon llvm_arm_neon LLVMArmNeon)
+add_mlir_doc(LLVMArmNeon -gen-dialect-doc LLVMArmNeon Dialects/)
+set(LLVM_TARGET_DEFINITIONS LLVMArmNeon.td)
+mlir_tablegen(LLVMArmNeonConversions.inc -gen-llvmir-conversions)
+add_public_tablegen_target(MLIRLLVMArmNeonConversionsIncGen)

mlir/include/mlir/Dialect/LLVMIR/LLVMArmNeon.td

+//===-- LLVMArmNeon.td - LLVMArmNeon dialect op definitions ------*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the basic operations for the LLVMArmNeon dialect.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVMIR_ARMNEON_OPS
+#define LLVMIR_ARMNEON_OPS
+
+include "mlir/Dialect/LLVMIR/LLVMOpBase.td"
+
+//===----------------------------------------------------------------------===//
+// LLVMArmNeon dialect definition
+//===----------------------------------------------------------------------===//
+
+def LLVMArmNeon_Dialect : Dialect {
+  let name = "llvm_arm_neon";
+  let cppNamespace = "::mlir::LLVM";
+}
+
+//----------------------------------------------------------------------------//
+// MLIR LLVMArmNeon intrinsics using the MLIR LLVM Dialect type system
+//----------------------------------------------------------------------------//
+
+class LLVMArmNeon_IntrBinaryOverloadedOp<string mnemonic, list<OpTrait> traits = []> :
+  LLVM_IntrOpBase</*Dialect dialect=*/LLVMArmNeon_Dialect,
+                  /*string opName=*/mnemonic,
+                  /*string enumName=*/"aarch64_neon_" # !subst(".", "_", mnemonic),
+                  /*list<int> overloadedResults=*/[0],
+                  /*list<int> overloadedOperands=*/[], // defined by result overload
+                  /*list<OpTrait> traits=*/traits,
+                  /*int numResults=*/1>;
+
+def LLVM_aarch64_arm_neon_smull :
+  LLVMArmNeon_IntrBinaryOverloadedOp<"smull">, Arguments<(ins LLVM_Type, LLVM_Type)>;
+
+#endif // ARMNEON_OPS

mlir/include/mlir/Dialect/LLVMIR/LLVMArmNeonDialect.h

+//===- LLVMArmNeonDialect.h - MLIR Dialect for LLVMArmNeon ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the Target dialect for LLVMArmNeon in MLIR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_LLVMIR_LLVMARMNEONDIALECT_H_
+#define MLIR_DIALECT_LLVMIR_LLVMARMNEONDIALECT_H_
+
+#include "mlir/IR/Dialect.h"
+#include "mlir/IR/OpDefinition.h"
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/LLVMIR/LLVMArmNeon.h.inc"
+
+#include "mlir/Dialect/LLVMIR/LLVMArmNeonDialect.h.inc"
+
+#endif // MLIR_DIALECT_LLVMIR_LLVMARMNEONDIALECT_H_

mlir/include/mlir/IR/BuiltinTypes.h

public:
 
   /// Return true if this is a signless integer type.
   bool isSignless() const { return getSignedness() == Signless; }
   /// Return true if this is a signed integer type.
   bool isSigned() const { return getSignedness() == Signed; }
   /// Return true if this is an unsigned integer type.
   bool isUnsigned() const { return getSignedness() == Unsigned; }
 
+  /// Get or create a new IntegerType with the same signedness as `this` and a
+  /// bitwidth scaled by `scale`.
+  /// Return null if the scaled element type cannot be represented.
+  IntegerType scaleElementBitwidth(unsigned scale);
+
   /// Integer representation maximal bitwidth.
   static constexpr unsigned kMaxWidth = 4096;
 };
 
 //===----------------------------------------------------------------------===//
 // FloatType
 //===----------------------------------------------------------------------===//
 
 class FloatType : public Type {
 public:
   using Type::Type;
 
   // Convenience factories.
   static FloatType getBF16(MLIRContext *ctx);
   static FloatType getF16(MLIRContext *ctx);
   static FloatType getF32(MLIRContext *ctx);
   static FloatType getF64(MLIRContext *ctx);
 
   /// Methods for support type inquiry through isa, cast, and dyn_cast.
   static bool classof(Type type);
 
   /// Return the bitwidth of this float type.
   unsigned getWidth();
 
+  /// Get or create a new FloatType with bitwidth scaled by `scale`.
+  /// Return null if the scaled element type cannot be represented.
+  FloatType scaleElementBitwidth(unsigned scale);
+
   /// Return the floating semantics of this float type.
   const llvm::fltSemantics &getFloatSemantics();
 };
 
 //===----------------------------------------------------------------------===//
 // BFloat16Type
 
 class BFloat16Type
public:
 
   /// Returns true of the given type can be used as an element of a vector type.
   /// In particular, vectors can consist of integer or float primitives.
   static bool isValidElementType(Type t) {
     return t.isa<IntegerType, FloatType>();
   }
 
   ArrayRef<int64_t> getShape() const;
+
+  /// Get or create a new VectorType with the same shape as `this` and an
+  /// element type of bitwidth scaled by `scale`.
+  /// Return null if the scaled element type cannot be represented.
+  VectorType scaleElementBitwidth(unsigned scale);
 };
 
 //===----------------------------------------------------------------------===//
 // TensorType
 //===----------------------------------------------------------------------===//
 
 /// Tensor types represent multi-dimensional arrays, and have two variants:
 /// RankedTensorType and UnrankedTensorType.

mlir/include/mlir/InitAllDialects.h

 //
 //===----------------------------------------------------------------------===//
 
 #ifndef MLIR_INITALLDIALECTS_H_
 #define MLIR_INITALLDIALECTS_H_
 
 #include "mlir/Dialect/AVX512/AVX512Dialect.h"
 #include "mlir/Dialect/Affine/IR/AffineOps.h"
+#include "mlir/Dialect/ArmNeon/ArmNeonDialect.h"
 #include "mlir/Dialect/Async/IR/Async.h"
 #include "mlir/Dialect/GPU/GPUDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMAVX512Dialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMArmNeonDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/LLVMIR/NVVMDialect.h"
 #include "mlir/Dialect/LLVMIR/ROCDLDialect.h"
 #include "mlir/Dialect/Linalg/IR/LinalgOps.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
 #include "mlir/Dialect/PDL/IR/PDL.h"
 #include "mlir/Dialect/PDLInterp/IR/PDLInterp.h"

 
 namespace mlir {
 
 // Add all the MLIR dialects to the provided registry.
 inline void registerAllDialects(DialectRegistry &registry) {
   // clang-format off
   registry.insert<acc::OpenACCDialect,
                   AffineDialect,
+                  arm_neon::ArmNeonDialect,
                   async::AsyncDialect,
                   avx512::AVX512Dialect,
                   gpu::GPUDialect,
                   LLVM::LLVMAVX512Dialect,
                   LLVM::LLVMDialect,
+                  LLVM::LLVMArmNeonDialect,
                   linalg::LinalgDialect,
                   scf::SCFDialect,
                   omp::OpenMPDialect,
                   pdl::PDLDialect,
                   pdl_interp::PDLInterpDialect,
                   quant::QuantizationDialect,
                   spirv::SPIRVDialect,
                   StandardOpsDialect,

mlir/include/mlir/InitAllTranslations.h

 namespace mlir {
 
 void registerFromLLVMIRTranslation();
 void registerFromSPIRVTranslation();
 void registerToLLVMIRTranslation();
 void registerToSPIRVTranslation();
 void registerToNVVMIRTranslation();
 void registerToROCDLIRTranslation();
+void registerArmNeonToLLVMIRTranslation();
 void registerAVX512ToLLVMIRTranslation();
 
 // This function should be called before creating any MLIRContext if one
 // expects all the possible translations to be made available to the context
 // automatically.
 inline void registerAllTranslations() {
   static bool initOnce = []() {
     registerFromLLVMIRTranslation();
     registerFromSPIRVTranslation();
     registerToLLVMIRTranslation();
     registerToSPIRVTranslation();
     registerToNVVMIRTranslation();
     registerToROCDLIRTranslation();
+    registerArmNeonToLLVMIRTranslation();
     registerAVX512ToLLVMIRTranslation();
     return true;
   }();
   (void)initOnce;
 }
 } // namespace mlir
 
 #endif // MLIR_INITALLTRANSLATIONS_H

mlir/lib/Conversion/AVX512ToLLVM/ConvertAVX512ToLLVM.cpp

 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/PatternMatch.h"
 
 using namespace mlir;
 using namespace mlir::vector;
 using namespace mlir::avx512;
 
 template <typename OpTy>
-static Type getSrcVectorElementType(OpTy op) {
-  return op.src().getType().template cast<VectorType>().getElementType();
-}
-
-// TODO: Code is currently copy-pasted and adapted from the code
-// 1-1 LLVM conversion. It would better if it were properly exposed in core and
-// reusable.
-/// Basic lowering implementation for one-to-one rewriting from AVX512 Ops to
-/// LLVM Dialect Ops. Convert the type of the result to an LLVM type, pass
-/// operands as is, preserve attributes.
-template <typename SourceOp, typename TargetOp>
-static LogicalResult
-matchAndRewriteOneToOne(LLVMTypeConverter &typeConverter, Operation *op,
-                        ArrayRef<Value> operands,
-                        ConversionPatternRewriter &rewriter) {
-  unsigned numResults = op->getNumResults();
-
-  Type packedType;
-  if (numResults != 0) {
-    packedType = typeConverter.packFunctionResults(op->getResultTypes());
-    if (!packedType)
-      return failure();
-  }
-
-  auto newOp = rewriter.create<TargetOp>(op->getLoc(), packedType, operands,
-                                         op->getAttrs());
-
-  // If the operation produced 0 or 1 result, return them immediately.
-  if (numResults == 0)
-    return rewriter.eraseOp(op), success();
-  if (numResults == 1)
-    return rewriter.replaceOp(op, newOp->getResult(0)), success();
-
-  // Otherwise, it had been converted to an operation producing a structure.
-  // Extract individual results from the structure and return them as list.
-  SmallVector<Value, 4> results;
-  results.reserve(numResults);
-  for (unsigned i = 0; i < numResults; ++i) {
-    auto type = typeConverter.convertType(op->getResult(i).getType());
-    results.push_back(rewriter.create<LLVM::ExtractValueOp>(
-        op->getLoc(), type, newOp->getResult(0), rewriter.getI64ArrayAttr(i)));
-  }
-  rewriter.replaceOp(op, results);
-  return success();
+static Type getSrcVectorElementType(Operation *op) {
+  return cast<OpTy>(op)
+      .src()
+      .getType()
+      .template cast<VectorType>()
+      .getElementType();
 }
 
 namespace {
-// TODO: Patterns are too verbose due to the fact that we have 1 op (e.g.
-// MaskRndScaleOp) and different possible target ops. It would be better to take
-// a Functor so that all these conversions become 1-liners.
-struct MaskRndScaleOpPS512Conversion
-    : public ConvertOpToLLVMPattern<MaskRndScaleOp> {
-  using ConvertOpToLLVMPattern<MaskRndScaleOp>::ConvertOpToLLVMPattern;
-
-  LogicalResult
-  matchAndRewrite(MaskRndScaleOp op, ArrayRef<Value> operands,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (!getSrcVectorElementType(op).isF32())
-      return failure();
-    return matchAndRewriteOneToOne<MaskRndScaleOp,
-                                   LLVM::x86_avx512_mask_rndscale_ps_512>(
-        *getTypeConverter(), op, operands, rewriter);
-  }
-};
 
-struct MaskRndScaleOpPD512Conversion
-    : public ConvertOpToLLVMPattern<MaskRndScaleOp> {
-  using ConvertOpToLLVMPattern<MaskRndScaleOp>::ConvertOpToLLVMPattern;
+// TODO: turn these into simpler declarative templated patterns when we've had
+// enough.
+struct MaskRndScaleOp512Conversion : public ConvertToLLVMPattern {
+  explicit MaskRndScaleOp512Conversion(MLIRContext *context,
+                                       LLVMTypeConverter &typeConverter)
+      : ConvertToLLVMPattern(MaskRndScaleOp::getOperationName(), context,
+                             typeConverter) {}
 
   LogicalResult
   matchAndRewrite(MaskRndScaleOp op, ArrayRef<Value> operands,
                   ConversionPatternRewriter &rewriter) const override {
-    if (!getSrcVectorElementType(op).isF64())
-      return failure();
-    return matchAndRewriteOneToOne<MaskRndScaleOp,
-                                   LLVM::x86_avx512_mask_rndscale_pd_512>(
-        *getTypeConverter(), op, operands, rewriter);
-  }
-};
-
-struct ScaleFOpPS512Conversion : public ConvertOpToLLVMPattern<MaskScaleFOp> {
-  using ConvertOpToLLVMPattern<MaskScaleFOp>::ConvertOpToLLVMPattern;
-
-  LogicalResult
-  matchAndRewrite(MaskScaleFOp op, ArrayRef<Value> operands,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (!getSrcVectorElementType(op).isF32())
-      return failure();
-    return matchAndRewriteOneToOne<MaskScaleFOp,
-                                   LLVM::x86_avx512_mask_scalef_ps_512>(
-        *getTypeConverter(), op, operands, rewriter);
+    Type elementType = getSrcVectorElementType<MaskRndScaleOp>(op);
+    if (elementType.isF32())
+      return LLVM::detail::oneToOneRewrite(
+          op, LLVM::x86_avx512_mask_rndscale_ps_512::getOperationName(),
+          operands, *getTypeConverter(), rewriter);
+    if (elementType.isF64())
+      return LLVM::detail::oneToOneRewrite(
+          op, LLVM::x86_avx512_mask_rndscale_pd_512::getOperationName(),
+          operands, *getTypeConverter(), rewriter);
   }
 };
 
-struct ScaleFOpPD512Conversion : public ConvertOpToLLVMPattern<MaskScaleFOp> {
-  using ConvertOpToLLVMPattern<MaskScaleFOp>::ConvertOpToLLVMPattern;
+struct ScaleFOp512Conversion : public ConvertToLLVMPattern {
+  explicit ScaleFOp512Conversion(MLIRContext *context,
+                                 LLVMTypeConverter &typeConverter)
+      : ConvertToLLVMPattern(MaskScaleFOp::getOperationName(), context,
+                             typeConverter) {}
 
   LogicalResult
   matchAndRewrite(MaskScaleFOp op, ArrayRef<Value> operands,
                   ConversionPatternRewriter &rewriter) const override {
-    if (!getSrcVectorElementType(op).isF64())
-      return failure();
-    return matchAndRewriteOneToOne<MaskScaleFOp,
-                                   LLVM::x86_avx512_mask_scalef_pd_512>(
-        *getTypeConverter(), op, operands, rewriter);
+    Type elementType = getSrcVectorElementType<MaskScaleFOp>(op);
+    if (elementType.isF32())
+      return LLVM::detail::oneToOneRewrite(
+          op, LLVM::x86_avx512_mask_scalef_ps_512::getOperationName(), operands,
+          *getTypeConverter(), rewriter);
+    if (elementType.isF64())
+      return LLVM::detail::oneToOneRewrite(
+          op, LLVM::x86_avx512_mask_scalef_pd_512::getOperationName(), operands,
+          *getTypeConverter(), rewriter);
   }
 };
 } // namespace
 
 /// Populate the given list with patterns that convert from AVX512 to LLVM.
 void mlir::populateAVX512ToLLVMConversionPatterns(
     LLVMTypeConverter &converter, OwningRewritePatternList &patterns) {
   // clang-format off
-  patterns.insert<MaskRndScaleOpPS512Conversion,
-                  MaskRndScaleOpPD512Conversion,
-                  ScaleFOpPS512Conversion,
-                  ScaleFOpPD512Conversion>(converter);
+  patterns.insert<MaskRndScaleOp512Conversion,
+                  ScaleFOp512Conversion>(ctx, converter);
   // clang-format on
 }

mlir/lib/Conversion/ArmNeonToLLVM/ArmNeonToLLVM.cpp

+//===- ArmNeonToLLVM.cpp - ArmNeon to the LLVM dialect --------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Conversion/ArmNeonToLLVM/ArmNeonToLLVM.h"
+
+#include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h"
+#include "mlir/Dialect/ArmNeon/ArmNeonDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMArmNeonDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/StandardOps/IR/Ops.h"
+#include "mlir/Dialect/Vector/VectorOps.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/PatternMatch.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::arm_neon;
+
+using SMullOpLowering =
+    OneToOneConvertToLLVMPattern<SMullOp, LLVM::aarch64_arm_neon_smull>;
+
+/// Populate the given list with patterns that convert from ArmNeon to LLVM.
+void mlir::populateArmNeonToLLVMConversionPatterns(
+    LLVMTypeConverter &converter, OwningRewritePatternList &patterns) {
+  patterns.insert<SMullOpLowering>(converter);
+}

mlir/lib/Conversion/ArmNeonToLLVM/CMakeLists.txt

+add_mlir_conversion_library(MLIRArmNeonToLLVM
+  ArmNeonToLLVM.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Conversion/ArmNeonToLLVM
+
+  DEPENDS
+  MLIRConversionPassIncGen
+
+  LINK_COMPONENTS
+  Core
+
+  LINK_LIBS PUBLIC
+  MLIRArmNeon
+  MLIRLLVMArmNeon
+  MLIRLLVMIR
+  MLIRStandardToLLVM
+  MLIRTransforms
+  )

mlir/lib/Conversion/CMakeLists.txt

 add_subdirectory(AffineToStandard)
+add_subdirectory(ArmNeonToLLVM)
 add_subdirectory(AsyncToLLVM)
 add_subdirectory(AVX512ToLLVM)
 add_subdirectory(GPUCommon)
 add_subdirectory(GPUToNVVM)
 add_subdirectory(GPUToROCDL)
 add_subdirectory(GPUToSPIRV)
 add_subdirectory(GPUToVulkan)
 add_subdirectory(LinalgToLLVM)

mlir/lib/Conversion/PassDetail.h

 void registerDialect(DialectRegistry &registry);
 
 namespace gpu {
 class GPUDialect;
 class GPUModuleOp;
 } // end namespace gpu
 
 namespace LLVM {
-class LLVMDialect;
+class LLVMArmNeonDialect;
 class LLVMAVX512Dialect;
+class LLVMDialect;
 } // end namespace LLVM
 
 namespace NVVM {
 class NVVMDialect;
 } // end namespace NVVM
 
 namespace omp {
 class OpenMPDialect;

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp

 //===- VectorToLLVM.cpp - Conversion from Vector to the LLVM dialect ------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h"
 
 #include "../PassDetail.h"
 
 #include "mlir/Conversion/AVX512ToLLVM/ConvertAVX512ToLLVM.h"
+#include "mlir/Conversion/ArmNeonToLLVM/ArmNeonToLLVM.h"
 #include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h"
 #include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVMPass.h"
 #include "mlir/Dialect/AVX512/AVX512Dialect.h"
+#include "mlir/Dialect/ArmNeon/ArmNeonDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMAVX512Dialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMArmNeonDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/Vector/VectorOps.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 
 using namespace mlir;
 using namespace mlir::vector;
 
 namespace {
 struct LowerVectorToLLVMPass
     : public ConvertVectorToLLVMBase<LowerVectorToLLVMPass> {
   LowerVectorToLLVMPass(const LowerVectorToLLVMOptions &options) {
     this->reassociateFPReductions = options.reassociateFPReductions;
     this->enableIndexOptimizations = options.enableIndexOptimizations;
+    this->enableArmNeon = options.enableArmNeon;
     this->enableAVX512 = options.enableAVX512;
   }
+  // Override explicitly to allow conditional dialect dependence.
+  void getDependentDialects(DialectRegistry &registry) const override {
+    registry.insert<LLVM::LLVMDialect>();
+    if (enableArmNeon)
+      registry.insert<LLVM::LLVMArmNeonDialect>();
+    if (enableAVX512)
+      registry.insert<LLVM::LLVMAVX512Dialect>();
+  }
   void runOnOperation() override;
 };
 } // namespace
 
 void LowerVectorToLLVMPass::runOnOperation() {
   // Perform progressive lowering of operations on slices and
   // all contraction operations. Also applies folding and DCE.
   {
void LowerVectorToLLVMPass::runOnOperation() {
   populateVectorToLLVMMatrixConversionPatterns(converter, patterns);
   populateVectorToLLVMConversionPatterns(
       converter, patterns, reassociateFPReductions, enableIndexOptimizations);
   populateVectorToLLVMMatrixConversionPatterns(converter, patterns);
   populateStdToLLVMConversionPatterns(converter, patterns);
 
   // Architecture specific augmentations.
   LLVMConversionTarget target(getContext());
+  if (enableArmNeon) {
+    target.addLegalDialect<LLVM::LLVMArmNeonDialect>();
+    target.addIllegalDialect<arm_neon::ArmNeonDialect>();
+    populateArmNeonToLLVMConversionPatterns(converter, patterns);
+  }
   if (enableAVX512) {
     target.addLegalDialect<LLVM::LLVMAVX512Dialect>();
     target.addIllegalDialect<avx512::AVX512Dialect>();
     populateAVX512ToLLVMConversionPatterns(converter, patterns);
   }
 
   if (failed(
           applyPartialConversion(getOperation(), target, std::move(patterns))))
     signalPassFailure();
 }
 
 std::unique_ptr<OperationPass<ModuleOp>>
 mlir::createConvertVectorToLLVMPass(const LowerVectorToLLVMOptions &options) {
   return std::make_unique<LowerVectorToLLVMPass>(options);
 }

mlir/lib/Dialect/ArmNeon/CMakeLists.txt

+add_mlir_dialect_library(MLIRArmNeon
+  IR/ArmNeonDialect.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/ArmNeon
+
+  DEPENDS
+  MLIRArmNeonIncGen
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  MLIRSideEffectInterfaces
+  )

mlir/lib/Dialect/ArmNeon/IR/ArmNeonDialect.cpp

+//===- ArmNeonOps.cpp - MLIRArmNeon ops implementation --------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the ArmNeon dialect and its operations.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/ArmNeon/ArmNeonDialect.h"
+#include "mlir/Dialect/Vector/VectorOps.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/OpImplementation.h"
+#include "mlir/IR/TypeUtilities.h"
+
+using namespace mlir;
+
+void arm_neon::ArmNeonDialect::initialize() {
+  addOperations<
+#define GET_OP_LIST
+#include "mlir/Dialect/ArmNeon/ArmNeon.cpp.inc"
+      >();
+}
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/ArmNeon/ArmNeon.cpp.inc"

mlir/lib/Dialect/CMakeLists.txt

 add_subdirectory(Affine)
+add_subdirectory(ArmNeon)
 add_subdirectory(Async)
 add_subdirectory(AVX512)
 add_subdirectory(GPU)
 add_subdirectory(Linalg)
 add_subdirectory(LLVMIR)
 add_subdirectory(OpenACC)
 add_subdirectory(OpenMP)
 add_subdirectory(PDL)

mlir/lib/Dialect/LLVMIR/CMakeLists.txt

add_mlir_dialect_library(MLIRLLVMAVX512
   Core
 
   LINK_LIBS PUBLIC
   MLIRIR
   MLIRLLVMIR
   MLIRSideEffectInterfaces
   )
 
+add_mlir_dialect_library(MLIRLLVMArmNeon
+  IR/LLVMArmNeonDialect.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/LLVMIR
+
+  DEPENDS
+  MLIRLLVMArmNeonIncGen
+  MLIRLLVMArmNeonConversionsIncGen
+  intrinsics_gen
+
+  LINK_COMPONENTS
+  AsmParser
+  Core
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  MLIRLLVMIR
+  MLIRSideEffectInterfaces
+  )
+
 add_mlir_dialect_library(MLIRNVVMIR
   IR/NVVMDialect.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/LLVMIR
 
   DEPENDS
   MLIRNVVMOpsIncGen

mlir/lib/Dialect/LLVMIR/IR/LLVMArmNeonDialect.cpp

+//===- LLVMArmNeonDialect.cpp - MLIR LLVMArmNeon ops implementation
+//-------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the LLVMArmNeon dialect and its operations.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/IR/IntrinsicsAArch64.h"
+
+#include "mlir/Dialect/LLVMIR/LLVMArmNeonDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/OpImplementation.h"
+#include "mlir/IR/TypeUtilities.h"
+
+using namespace mlir;
+
+void LLVM::LLVMArmNeonDialect::initialize() {
+  addOperations<
+#define GET_OP_LIST
+#include "mlir/Dialect/LLVMIR/LLVMArmNeon.cpp.inc"
+      >();
+}
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/LLVMIR/LLVMArmNeon.cpp.inc"

mlir/lib/IR/BuiltinTypes.cpp

 }
 
 unsigned IntegerType::getWidth() const { return getImpl()->width; }
 
 IntegerType::SignednessSemantics IntegerType::getSignedness() const {
   return getImpl()->signedness;
 }
 
+IntegerType IntegerType::scaleElementBitwidth(unsigned scale) {
+  if (!scale)
+    return IntegerType();
+  return IntegerType::get(scale * getWidth(), getSignedness(), getContext());
+}
+
 //===----------------------------------------------------------------------===//
 // Float Type
 //===----------------------------------------------------------------------===//
 
 unsigned FloatType::getWidth() {
   if (isa<Float16Type, BFloat16Type>())
     return 16;
   if (isa<Float32Type>())
if (isa<Float16Type>())
     return APFloat::IEEEhalf();
   if (isa<Float32Type>())
     return APFloat::IEEEsingle();
   if (isa<Float64Type>())
     return APFloat::IEEEdouble();
   llvm_unreachable("non-floating point type used");
 }
 
+FloatType FloatType::scaleElementBitwidth(unsigned scale) {
+  if (!scale)
+    return FloatType();
+  MLIRContext *ctx = getContext();
+  if (isF16() || isBF16()) {
+    if (scale == 2)
+      return FloatType::getF32(ctx);
+    if (scale == 4)
+      return FloatType::getF64(ctx);
+  }
+  if (isF32())
+    if (scale == 2)
+      return FloatType::getF64(ctx);
+  return FloatType();
+}
+
 //===----------------------------------------------------------------------===//
 // FunctionType
 //===----------------------------------------------------------------------===//
 
 FunctionType FunctionType::get(TypeRange inputs, TypeRange results,
                                MLIRContext *context) {
   return Base::get(context, inputs, results);
 }
LogicalResult VectorType::verifyConstructionInvariants(Location loc,
   if (any_of(shape, [](int64_t i) { return i <= 0; }))
     return emitError(loc, "vector types must have positive constant sizes");
 
   return success();
 }
 
 ArrayRef<int64_t> VectorType::getShape() const { return getImpl()->getShape(); }
 
+VectorType VectorType::scaleElementBitwidth(unsigned scale) {
+  if (!scale)
+    return VectorType();
+  if (auto et = getElementType().dyn_cast<IntegerType>())
+    if (auto scaledEt = et.scaleElementBitwidth(scale))
+      return VectorType::get(getShape(), scaledEt);
+  if (auto et = getElementType().dyn_cast<FloatType>())
+    if (auto scaledEt = et.scaleElementBitwidth(scale))
+      return VectorType::get(getShape(), scaledEt);
+  return VectorType();
+}
+
 //===----------------------------------------------------------------------===//
 // TensorType
 //===----------------------------------------------------------------------===//
 
 // Check if "elementType" can be an element type of a tensor. Emit errors if
 // location is not nullptr.  Returns failure if check failed.
 static LogicalResult checkTensorElementType(Location location,
                                             Type elementType) {

mlir/lib/Target/CMakeLists.txt

add_mlir_translation_library(MLIRTargetLLVMIR
   LINK_COMPONENTS
   Core
   IRReader
 
   LINK_LIBS PUBLIC
   MLIRTargetLLVMIRModuleTranslation
   )
 
+add_mlir_translation_library(MLIRTargetArmNeon
+  LLVMIR/LLVMArmNeonIntr.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Target/LLVMIR
+
+  DEPENDS
+  MLIRLLVMArmNeonConversionsIncGen
+
+  LINK_COMPONENTS
+  Core
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  MLIRLLVMArmNeon
+  MLIRLLVMIR
+  MLIRTargetLLVMIRModuleTranslation
+  )
+
 add_mlir_translation_library(MLIRTargetNVVMIR
   LLVMIR/ConvertToNVVMIR.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Target/LLVMIR
 
   DEPENDS
   intrinsics_gen

mlir/lib/Target/LLVMIR/LLVMArmNeonIntr.cpp

+//===- ArmNeonIntr.cpp - Convert MLIR LLVM dialect to LLVM intrinsics -----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements a translation between the MLIR LLVM and ArmNeon dialects
+// and LLVM IR with ArmNeon intrinsics.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/LLVMIR/LLVMArmNeonDialect.h"
+#include "mlir/Target/LLVMIR/ModuleTranslation.h"
+#include "mlir/Translation.h"
+#include "llvm/IR/IntrinsicsAArch64.h"
+
+using namespace mlir;
+
+namespace {
+class LLVMArmNeonModuleTranslation : public LLVM::ModuleTranslation {
+  friend LLVM::ModuleTranslation;
+
+public:
+  using LLVM::ModuleTranslation::ModuleTranslation;
+
+protected:
+  LogicalResult convertOperation(Operation &opInst,
+                                 llvm::IRBuilder<> &builder) override {
+#include "mlir/Dialect/LLVMIR/LLVMArmNeonConversions.inc"
+
+    return LLVM::ModuleTranslation::convertOperation(opInst, builder);
+  }
+};
+
+std::unique_ptr<llvm::Module>
+translateLLVMArmNeonModuleToLLVMIR(Operation *m, llvm::LLVMContext &llvmContext,
+                                   StringRef name) {
+  return LLVM::ModuleTranslation::translateModule<LLVMArmNeonModuleTranslation>(
+      m, llvmContext, name);
+}
+} // end namespace
+
+namespace mlir {
+void registerArmNeonToLLVMIRTranslation() {
+  TranslateFromMLIRRegistration reg(
+      "arm-neon-mlir-to-llvmir",
+      [](ModuleOp module, raw_ostream &output) {
+        llvm::LLVMContext llvmContext;
+        auto llvmModule = translateLLVMArmNeonModuleToLLVMIR(
+            module, llvmContext, "LLVMDialectModule");
+        if (!llvmModule)
+          return failure();
+
+        llvmModule->print(output, nullptr);
+        return success();
+      },
+      [](DialectRegistry &registry) {
+        registry.insert<LLVM::LLVMArmNeonDialect, LLVM::LLVMDialect>();
+      });
+}
+} // namespace mlir

mlir/test/Conversion/ArmNeonToLLVM/convert-to-llvm.mlir

+// RUN: mlir-opt %s -convert-vector-to-llvm="enable-arm-neon" | mlir-opt | FileCheck %s
+
+// CHECK-LABEL: arm_neon_smull
+func @arm_neon_smull(%a: vector<8xi8>, %b: vector<8xi8>)
+    -> (vector<8xi16>, vector<4xi32>, vector<2xi64>) {
+  // CHECK: arm_neon.smull{{.*}}: (!llvm.vec<8 x i8>, !llvm.vec<8 x i8>) -> !llvm.vec<8 x i16>
+  %0 = arm_neon.smull %a, %b : vector<8xi8> to vector<8xi16>
+  %00 = vector.extract_strided_slice %0 {offsets = [3], sizes = [4], strides = [1]}:
+    vector<8xi16> to vector<4xi16>
+
+  // CHECK: arm_neon.smull{{.*}}: (!llvm.vec<4 x i16>, !llvm.vec<4 x i16>) -> !llvm.vec<4 x i32>
+  %1 = arm_neon.smull %00, %00 : vector<4xi16> to vector<4xi32>
+  %11 = vector.extract_strided_slice %1 {offsets = [1], sizes = [2], strides = [1]}:
+    vector<4xi32> to vector<2xi32>
+
+  // CHECK: arm_neon.smull{{.*}}: (!llvm.vec<2 x i32>, !llvm.vec<2 x i32>) -> !llvm.vec<2 x i64>
+  %2 = arm_neon.smull %11, %11 : vector<2xi32> to vector<2xi64>
+
+  return %0, %1, %2 : vector<8xi16>, vector<4xi32>, vector<2xi64>
+}

mlir/test/Dialect/ArmNeon/roundtrip.mlir

+// RUN: mlir-opt -verify-diagnostics %s | mlir-opt | FileCheck %s
+
+// CHECK-LABEL: arm_neon_smull
+func @arm_neon_smull(%a: vector<8xi8>, %b: vector<8xi8>)
+    -> (vector<8xi16>, vector<4xi32>, vector<2xi64>) {
+  // CHECK: arm_neon.smull {{.*}}: vector<8xi8> to vector<8xi16>
+  %0 = arm_neon.smull %a, %b : vector<8xi8> to vector<8xi16>
+  %00 = vector.extract_strided_slice %0 {offsets = [3], sizes = [4], strides = [1]}:
+    vector<8xi16> to vector<4xi16>
+
+  // CHECK: arm_neon.smull {{.*}}: vector<4xi16> to vector<4xi32>
+  %1 = arm_neon.smull %00, %00 : vector<4xi16> to vector<4xi32>
+  %11 = vector.extract_strided_slice %1 {offsets = [1], sizes = [2], strides = [1]}:
+    vector<4xi32> to vector<2xi32>
+
+  // CHECK: arm_neon.smull {{.*}}: vector<2xi32> to vector<2xi64>
+  %2 = arm_neon.smull %11, %11 : vector<2xi32> to vector<2xi64>
+
+  return %0, %1, %2 : vector<8xi16>, vector<4xi32>, vector<2xi64>
+}

mlir/test/Target/arm-neon.mlir

+// RUN: mlir-opt -verify-diagnostics %s | mlir-opt | mlir-translate -arm-neon-mlir-to-llvmir | FileCheck %s
+
+// CHECK-LABEL: arm_neon_smull
+llvm.func @arm_neon_smull(%arg0: !llvm.vec<8 x i8>, %arg1: !llvm.vec<8 x i8>) -> !llvm.struct<(vec<8 x i16>, vec<4 x i32>, vec<2 x i64>)> {
+  //      CHECK: %[[V0:.*]] = call <8 x i16> @llvm.aarch64.neon.smull.v8i16(<8 x i8> %{{.*}}, <8 x i8> %{{.*}})
+  // CHECK-NEXT: %[[V00:.*]] = shufflevector <8 x i16> %3, <8 x i16> %[[V0]], <4 x i32> <i32 3, i32 4, i32 5, i32 6>
+  %0 = "llvm_arm_neon.smull"(%arg0, %arg1) : (!llvm.vec<8 x i8>, !llvm.vec<8 x i8>) -> !llvm.vec<8 x i16>
+  %1 = llvm.shufflevector %0, %0 [3, 4, 5, 6] : !llvm.vec<8 x i16>, !llvm.vec<8 x i16>
+
+  // CHECK-NEXT: %[[V1:.*]] = call <4 x i32> @llvm.aarch64.neon.smull.v4i32(<4 x i16> %[[V00]], <4 x i16> %[[V00]])
+  // CHECK-NEXT: %[[V11:.*]] = shufflevector <4 x i32> %[[V1]], <4 x i32> %[[V1]], <2 x i32> <i32 1, i32 2>
+  %2 = "llvm_arm_neon.smull"(%1, %1) : (!llvm.vec<4 x i16>, !llvm.vec<4 x i16>) -> !llvm.vec<4 x i32>
+  %3 = llvm.shufflevector %2, %2 [1, 2] : !llvm.vec<4 x i32>, !llvm.vec<4 x i32>
+
+  // CHECK-NEXT: %[[V1:.*]] = call <2 x i64> @llvm.aarch64.neon.smull.v2i64(<2 x i32> %[[V11]], <2 x i32> %[[V11]])
+  %4 = "llvm_arm_neon.smull"(%3, %3) : (!llvm.vec<2 x i32>, !llvm.vec<2 x i32>) -> !llvm.vec<2 x i64>
+
+  %5 = llvm.mlir.undef : !llvm.struct<(vec<8 x i16>, vec<4 x i32>, vec<2 x i64>)>
+  %6 = llvm.insertvalue %0, %5[0] : !llvm.struct<(vec<8 x i16>, vec<4 x i32>, vec<2 x i64>)>
+  %7 = llvm.insertvalue %2, %6[1] : !llvm.struct<(vec<8 x i16>, vec<4 x i32>, vec<2 x i64>)>
+  %8 = llvm.insertvalue %4, %7[2] : !llvm.struct<(vec<8 x i16>, vec<4 x i32>, vec<2 x i64>)>
+
+  //      CHECK: ret { <8 x i16>, <4 x i32>, <2 x i64> }
+  llvm.return %8 : !llvm.struct<(vec<8 x i16>, vec<4 x i32>, vec<2 x i64>)>
+}