[mlir] X86Vector: Add AVX Rsqrt

[aartbik]

mlir/include/mlir/Dialect/AVX512/AVX512.td

42

So technically you are bringing in AVX into the AVX512 dialect. That is not a big deal, but it makes me wonder, do we want any of the x86_avx512_rsqrtWIDTH versions as well? If so, should we introduce a generic op for this that takes an attribute? (not saying we should since I am not very familiar what you are going to use this for, just thinking out loud here)

275

Please use the section separating comment used above consistently for new instructions

mlir/test/Dialect/AVX512/legalize-for-llvm.mlir

43

this omission is pre-existing, but shouldn't we have CHECK-LABEL at the start of each method (or split-input), to make sure CHECKs are leaking between methods in hte long run

mlir/test/Dialect/AVX512/roundtrip.mlir

48

same question

[ezhulenev]

mlir/include/mlir/Dialect/AVX512/AVX512.td

42

Would it be a good idea to create a "generic" AVX dialect, and lower to concrete AVX version based on vector length and availability? Or too much instructions are sufficiently different to make this practical?

[ftynse]

mlir/include/mlir/Dialect/AVX512/AVX512.td

42

I find it very surprising to have AVX instructions in the AVX512 dialect, these print as avx512.intr.rsqrt.ps.256 but the instruction in question is _not_ defined in the avx512 set. Maybe we should consider renaming AVX512 to X86 (which would be also consistent with how LLVM has it) and allow any instruction set with several prefixes, i.e. x86.avx512...., x86.avx....

As for the "generic" AVX dialect, why stop at AVX? Can't we just have an operation on a generic vector that lowers to the target-specific intrinsic depending on the target features and vector sizes. This sounds like Vector dialect :)

mlir/test/Dialect/AVX512/roundtrip.mlir

52

Please add a newline

[cota]

Address some reviewer comments

mlir/include/mlir/Dialect/AVX512/AVX512.td

42

Maybe we should consider renaming AVX512 to X86 (which would be also consistent with how LLVM has it) and allow any instruction set with several prefixes, i.e. x86.avx512...., x86.avx

This means more work for me, but I have to agree. I'll work on this and send it as a parent patch.

42

do we want any of the x86_avx512_rsqrtWIDTH versions as well?

I have no use case for them since I am only interested in AVX's rsqrt, but I can probably squeeze this change in.

[aartbik]

mlir/include/mlir/Dialect/AVX512/AVX512.td

42

Thanks. When given the choice between putting a generic op in vector dialect (viz. VV) or a target specific op (viz. HWV), we should always favor the VV approach, the HWV is a last (and often temporary) solution for experimenting with idiomatic instructions.

As for the new name, X86 makes sense since it keeps a lot of specifics hidden (SSE vz AVX, AVX2, AVX512 etc), but it does not carry the "vector" semantics part ver ystrong, like ARMNeon and ARMSVE do). So AVX would make some sense, or X86SIMD, X86Vector. Not to bikeshed too much, but what do you think?

[cota]

mlir/include/mlir/Dialect/AVX512/AVX512.td

42

So AVX would make some sense, or X86SIMD, X86Vector.

I like this suggestion -- with just x86 in the name there is a risk this will become more of an an incoherent bag than it needs to be. Plus, it would be confusing since there are other x86-related dialects (e.g. AMX).

My vote is for X86Vector.

[ftynse]

mlir/include/mlir/Dialect/AVX512/AVX512.td

42

X86Vector works for me

[cota]

mlir/include/mlir/Dialect/AVX512/AVX512.td

42

I've introduced X86Vector in https://reviews.llvm.org/D100119
I'll update this patch to apply on top of X86Vector once the X86Vector patch is reviewed.

[aartbik]

can you please rebase so we can continue the review in the context of the new dialect name?

[cota]

Rebase on X86Vector

[cota]

fix newline

[aartbik]

mlir/test/Integration/Dialect/Vector/CPU/X86Vector/test-rsqrt.mlir

18 ↗

(On Diff #336968)

Note that this is old-style (I am responsible for setting the example, but that was in the context of testing the broadcast/insert). But a much better and more readable style for such vector constants is

%v = std.constant dense<[0,125, 0.25 ....]> : vector<8xf32>

In such cases, the CHECKin print also is not needed (which was only added to help the reader see what vectors were constructed.

[cota]

• Use std.constant dense
• Mention https://bugs.llvm.org/show_bug.cgi?id=49906 in test-rsqrt.mlir

mlir/test/Integration/Dialect/Vector/CPU/X86Vector/test-rsqrt.mlir

18 ↗

(On Diff #336968)

Much better! Done.

mlir/include/mlir/Dialect/AVX512/AVX512.td

 class AVX512_Op<string mnemonic, list<OpTrait> traits = []> :
   Op<AVX512_Dialect, mnemonic, traits> {}
 
 class AVX512_IntrOp<string mnemonic, int numResults, list<OpTrait> traits = []> :
   LLVM_IntrOpBase<AVX512_Dialect, "intr." # mnemonic,
                   "x86_avx512_" # !subst(".", "_", mnemonic),
                   [], [], traits, numResults>;
 
+class AVX_IntrOp<string mnemonic, int numResults, list<OpTrait> traits = []> :
+  LLVM_IntrOpBase<AVX512_Dialect, "intr." # mnemonic,
+                  "x86_avx_" # !subst(".", "_", mnemonic),
+                  [], [], traits, numResults>;
+
 // Defined by first result overload. May have to be extended for other
 // instructions in the future.
 class AVX512_IntrOverloadedOp<string mnemonic,
                               list<OpTrait> traits = []> :
   LLVM_IntrOpBase<AVX512_Dialect, "intr." # mnemonic,
                   "x86_avx512_" # !subst(".", "_", mnemonic),
                   /*list<int> overloadedResults=*/[0],
                   /*list<int> overloadedOperands=*/[],

 }
 
 def Vp2IntersectQIntrOp : AVX512_IntrOp<"vp2intersect.q.512", 2, [
   NoSideEffect]> {
   let arguments = (ins VectorOfLengthAndType<[8], [I64]>:$a,
                    VectorOfLengthAndType<[8], [I64]>:$b);
 }
 
+// Rsqrt.
+def RsqrtOp : AVX512_Op<"rsqrt", [NoSideEffect, SameOperandsAndResultType]> {
+  let summary = "Rsqrt";
+  let arguments = (ins VectorOfLengthAndType<[8], [F32]>:$a);
+  let results = (outs VectorOfLengthAndType<[8], [F32]>:$b);
+  let assemblyFormat = "$a attr-dict `:` type($a)";
+}
+
+def RsqrtIntrOp : AVX_IntrOp<"rsqrt.ps.256", 1, [NoSideEffect,
+  SameOperandsAndResultType]> {
+  let arguments = (ins VectorOfLengthAndType<[8], [F32]>:$a);
+}
+
 #endif // AVX512_OPS

mlir/lib/Dialect/AVX512/Transforms/LegalizeForLLVMExport.cpp

matchAndRewrite(MaskCompressOp op, ArrayRef<Value> operands,
 
     rewriter.replaceOpWithNewOp<MaskCompressIntrOp>(op, opType, adaptor.a(),
                                                     src, adaptor.k());
 
     return success();
   }
 };
 
+struct RsqrtOpConversion : public ConvertOpToLLVMPattern<RsqrtOp> {
+  using ConvertOpToLLVMPattern<RsqrtOp>::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(RsqrtOp op, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    RsqrtOp::Adaptor adaptor(operands);
+
+    auto opType = adaptor.a().getType();
+    rewriter.replaceOpWithNewOp<RsqrtIntrOp>(op, opType, adaptor.a());
+    return success();
+  }
+};
+
 /// An entry associating the "main" AVX512 op with its instantiations for
 /// vectors of 32-bit and 64-bit elements.
 template <typename OpTy, typename Intr32OpTy, typename Intr64OpTy>
 struct RegEntry {
   using MainOp = OpTy;
   using Intr32Op = Intr32OpTy;
   using Intr64Op = Intr64OpTy;
 };
using Registry = RegistryImpl<
     RegEntry<Vp2IntersectOp, Vp2IntersectDIntrOp, Vp2IntersectQIntrOp>>;
 
 } // namespace
 
 /// Populate the given list with patterns that convert from AVX512 to LLVM.
 void mlir::populateAVX512LegalizeForLLVMExportPatterns(
     LLVMTypeConverter &converter, RewritePatternSet &patterns) {
   Registry::registerPatterns(converter, patterns);
-  patterns.add<MaskCompressOpConversion>(converter);
+  patterns.add<MaskCompressOpConversion, RsqrtOpConversion>(converter);
 }
 
 void mlir::configureAVX512LegalizeForExportTarget(
     LLVMConversionTarget &target) {
   Registry::configureTarget(target);
   target.addLegalOp<MaskCompressIntrOp>();
   target.addIllegalOp<MaskCompressOp>();
+  target.addLegalOp<RsqrtIntrOp>();
+  target.addIllegalOp<RsqrtOp>();
 }

mlir/test/Dialect/AVX512/legalize-for-llvm.mlir

func @avx512_vp2intersect(%a: vector<16xi32>, %b: vector<8xi64>)
   -> (vector<16xi1>, vector<16xi1>, vector<8xi1>, vector<8xi1>)
 {
   // CHECK: avx512.intr.vp2intersect.d.512
   %0, %1 = avx512.vp2intersect %a, %a : vector<16xi32>
   // CHECK: avx512.intr.vp2intersect.q.512
   %2, %3 = avx512.vp2intersect %b, %b : vector<8xi64>
   return %0, %1, %2, %3 : vector<16xi1>, vector<16xi1>, vector<8xi1>, vector<8xi1>
 }
+
+func @axv512_rsqrt(%a: vector<8xf32>) -> (vector<8xf32>)
+{
+  // CHECK: avx512.intr.rsqrt.ps.256
+  %0 = avx512.rsqrt %a : vector<8xf32>
+  return %0 : vector<8xf32>
+}

mlir/test/Dialect/AVX512/roundtrip.mlir

func @avx512_mask_compress(%k1: vector<16xi1>, %a1: vector<16xf32>,
   // CHECK: avx512.mask.compress {{.*}} : vector<16xf32>
   %0 = avx512.mask.compress %k1, %a1 : vector<16xf32>
   // CHECK: avx512.mask.compress {{.*}} : vector<16xf32>
   %1 = avx512.mask.compress %k1, %a1 {constant_src = dense<5.0> : vector<16xf32>} : vector<16xf32>
   // CHECK: avx512.mask.compress {{.*}} : vector<8xi64>
   %2 = avx512.mask.compress %k2, %a2, %a2 : vector<8xi64>, vector<8xi64>
   return %0, %1, %2 : vector<16xf32>, vector<16xf32>, vector<8xi64>
 }
+
+func @avx512_rsqrt(%a: vector<8xf32>) -> (vector<8xf32>)
+{
+  // CHECK: avx512.rsqrt {{.*}} : vector<8xf32>
+  %0 = avx512.rsqrt %a : vector<8xf32>
+  return %0 : vector<8xf32>
+}
 No newline at end of file

mlir/test/Integration/Dialect/Vector/CPU/AVX512/test-rsqrt.mlir

+// RUN: mlir-opt %s -convert-scf-to-std -convert-vector-to-llvm="enable-avx512" -convert-std-to-llvm  | \
+// RUN: mlir-translate --mlir-to-llvmir | \
+// RUN: %lli --entry-function=entry --mattr="avx512bw" --dlopen=%mlir_integration_test_dir/libmlir_c_runner_utils%shlibext | \
+// RUN: FileCheck %s
+
+func @entry() -> i32 {
+  %i0 = constant 0 : i32
+
+  %f0 = constant 0.125 : f32
+  %f1 = constant 0.25 : f32
+  %f2 = constant 0.5 : f32
+  %f3 = constant 1.0 : f32
+  %f4 = constant 2.0: f32
+  %f5 = constant 4.0: f32
+  %f6 = constant 8.0: f32
+  %f7 = constant 16.0: f32
+
+  %v0 = vector.broadcast %f0 : f32 to vector<8xf32>
+  %v1 = vector.insert %f1, %v0[1] : f32 into vector<8xf32>
+  %v2 = vector.insert %f2, %v1[2] : f32 into vector<8xf32>
+  %v3 = vector.insert %f3, %v2[3] : f32 into vector<8xf32>
+  %v4 = vector.insert %f4, %v3[4] : f32 into vector<8xf32>
+  %v5 = vector.insert %f5, %v4[5] : f32 into vector<8xf32>
+  %v6 = vector.insert %f6, %v5[6] : f32 into vector<8xf32>
+  %v7 = vector.insert %f7, %v6[7] : f32 into vector<8xf32>
+  vector.print %v7 : vector<8xf32>
+  // CHECK: ( 0.125, 0.25, 0.5, 1, 2, 4, 8, 16 )
+
+  %r = avx512.rsqrt %v7 : vector<8xf32>
+  vector.print %r : vector<8xf32>
+  // CHECK: ( 2.82764, 1.99951, 1.41382, 0.999756, 0.706909, 0.499878, 0.353455, 0.249939 )
+
+  return %i0 : i32
+}

mlir/test/Target/LLVMIR/avx512.mlir

 llvm.func @LLVM_x86_vp2intersect_q_512(%a: vector<8xi64>, %b: vector<8xi64>)
   -> !llvm.struct<(vector<8 x i1>, vector<8 x i1>)>
 {
   // CHECK: call { <8 x i1>, <8 x i1> } @llvm.x86.avx512.vp2intersect.q.512(<8 x i64>
   %0 = "avx512.intr.vp2intersect.q.512"(%a, %b) :
     (vector<8xi64>, vector<8xi64>) -> !llvm.struct<(vector<8 x i1>, vector<8 x i1>)>
   llvm.return %0 : !llvm.struct<(vector<8 x i1>, vector<8 x i1>)>
 }
+
+// CHECK-LABEL: define <8 x float> @LLVM_x86_avx512_rsqrt_ps_256
+llvm.func @LLVM_x86_avx512_rsqrt_ps_256(%a: vector <8xf32>) -> vector<8xf32>
+{
+  // CHECK: call <8 x float> @llvm.x86.avx.rsqrt.ps.256(<8 x float>
+  %0 = "avx512.intr.rsqrt.ps.256"(%a) : (vector<8xf32>) -> (vector<8xf32>)
+  llvm.return %0 : vector<8xf32>
+}