AArch64 SVE: add sve patterns to make use of predicated smin, umin, smax, and umax instructions, add ll test file for the new patterns

[david-arm]

Thanks for this @hassnaa-arm, it looks like a useful addition to the patterns! I just spotted an issue with the test output for elements other than i32.

llvm/test/CodeGen/AArch64/sve-min-max-pred.ll

14

I don't quite understand what's going on here, but this doesn't look right I'm afraid! The vector elements are 8-bits so I was expecting to see this instead:

; CHECK: smax z0.b, p0/m, z0.b, z1.b

It's the same problem for all the smax/smin/umax/umin examples here where the element type != i32.

32

This looks right!

[david-arm]

Could you also add a brief description

llvm/test/CodeGen/AArch64/sve-min-max-pred.ll

3

I think you only need one RUN line here and it should specify the SVE attribute, i.e. something like this:

; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve -o - | FileCheck %s

[hassnaa-arm]

Add sve patterns to make use of predicated smin, umin, smax, umax, and its test file.

[david-arm]

Sorry @hassnaa-arm, I accidentally removed @sdesmalen as a reviewer from your patch yesterday!

[sdesmalen]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

379

Can these defs use EitherVSelectOrPassthruPatFrags instead?

[hassnaa-arm]

AArch64 SVE: add sve patterns to make use of predicated smin, umin, smax, and umax instructions, add ll test file for the new patterns

[sdesmalen]

LGTM with nit addressed, thanks @hassnaa-arm!

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

380

nit: can you remove the newlines and move these definitions to line 247? (after previous uses of EitherVSelectOrPassthruPatFrags)

[hassnaa-arm]

AArch64 SVE: add sve patterns to make use of predicated smin, umin, smax, and umax instructions, add sve-min-max.ll test file for the new patterns

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

def AArch64mla_m1 : PatFrags<(ops node:$pred, node:$op1, node:$op2, node:$op3),
                               (add node:$op1, (vselect node:$pred, (AArch64mul_p_oneuse (SVEAllActive), node:$op2, node:$op3), (SVEDup0)))]>;
 def AArch64mls_m1 : PatFrags<(ops node:$pred, node:$op1, node:$op2, node:$op3),
                              [(int_aarch64_sve_mls node:$pred, node:$op1, node:$op2, node:$op3),
                               (sub node:$op1, (AArch64mul_p_oneuse node:$pred, node:$op2, node:$op3)),
                               // sub(a, select(mask, mul(b, c), splat(0))) -> mls(a, mask, b, c)
                               (sub node:$op1, (vselect node:$pred, (AArch64mul_p_oneuse (SVEAllActive), node:$op2, node:$op3), (SVEDup0)))]>;
 
 class fma_patfrags<SDPatternOperator intrinsic, SDPatternOperator sdnode>
     : PatFrags<(ops node:$pred, node:$op1, node:$op2, node:$op3),
                [(intrinsic node:$pred, node:$op1, node:$op2, node:$op3),
                 (sdnode (SVEAllActive), node:$op1, (vselect node:$pred, (AArch64fmul_p_oneuse (SVEAllActive), node:$op2, node:$op3), (SVEDup0)))],
                [{
   if ((N->getOpcode() != AArch64ISD::FADD_PRED) &&
       (N->getOpcode() != AArch64ISD::FSUB_PRED))
     return true;  // it's the intrinsic
   return N->getFlags().hasAllowContract();
 }]>;
 
 def AArch64fmla_m1 : fma_patfrags<int_aarch64_sve_fmla, AArch64fadd_p_nsz>;
 def AArch64fmls_m1 : fma_patfrags<int_aarch64_sve_fmls, AArch64fsub_p>;
 
+def AArch64smax_m1 : EitherVSelectOrPassthruPatFrags<int_aarch64_sve_smax, AArch64smax_p>;
+def AArch64umax_m1 : EitherVSelectOrPassthruPatFrags<int_aarch64_sve_umax, AArch64umax_p>;
+def AArch64smin_m1 : EitherVSelectOrPassthruPatFrags<int_aarch64_sve_smin, AArch64smin_p>;
+def AArch64umin_m1 : EitherVSelectOrPassthruPatFrags<int_aarch64_sve_umin, AArch64umin_p>;
+
 let Predicates = [HasSVE] in {
   defm RDFFR_PPz  : sve_int_rdffr_pred<0b0, "rdffr", int_aarch64_sve_rdffr_z>;
   def  RDFFRS_PPz : sve_int_rdffr_pred<0b1, "rdffrs">;
   defm RDFFR_P    : sve_int_rdffr_unpred<"rdffr", int_aarch64_sve_rdffr>;
   def  SETFFR     : sve_int_setffr<"setffr", int_aarch64_sve_setffr>;
   def  WRFFR      : sve_int_wrffr<"wrffr", int_aarch64_sve_wrffr>;
 } // End HasSVE
 
def : Pat<(nxv16i8 (zext (nxv16i1 (AArch64setcc_z (nxv16i1 (SVEAllActive):$Pg), nxv16i8:$Op2, (SVEDup0), SETEQ)))),
             (CNOT_ZPmZ_B $Op2, $Pg, $Op2)>;
   def : Pat<(nxv8i16 (zext (nxv8i1 (AArch64setcc_z (nxv8i1 (SVEAllActive):$Pg), nxv8i16:$Op2, (SVEDup0), SETEQ)))),
             (CNOT_ZPmZ_H $Op2, $Pg, $Op2)>;
   def : Pat<(nxv4i32 (zext (nxv4i1 (AArch64setcc_z (nxv4i1 (SVEAllActive):$Pg), nxv4i32:$Op2, (SVEDup0), SETEQ)))),
             (CNOT_ZPmZ_S $Op2, $Pg, $Op2)>;
   def : Pat<(nxv2i64 (zext (nxv2i1 (AArch64setcc_z (nxv2i1 (SVEAllActive):$Pg), nxv2i64:$Op2, (SVEDup0), SETEQ)))),
             (CNOT_ZPmZ_D $Op2, $Pg, $Op2)>;
 
-  defm SMAX_ZPmZ : sve_int_bin_pred_arit_1<0b000, "smax", "SMAX_ZPZZ", int_aarch64_sve_smax, DestructiveBinaryComm>;
-  defm UMAX_ZPmZ : sve_int_bin_pred_arit_1<0b001, "umax", "UMAX_ZPZZ", int_aarch64_sve_umax, DestructiveBinaryComm>;
-  defm SMIN_ZPmZ : sve_int_bin_pred_arit_1<0b010, "smin", "SMIN_ZPZZ", int_aarch64_sve_smin, DestructiveBinaryComm>;
-  defm UMIN_ZPmZ : sve_int_bin_pred_arit_1<0b011, "umin", "UMIN_ZPZZ", int_aarch64_sve_umin, DestructiveBinaryComm>;
+  defm SMAX_ZPmZ : sve_int_bin_pred_arit_1<0b000, "smax", "SMAX_ZPZZ", AArch64smax_m1, DestructiveBinaryComm>;
+  defm UMAX_ZPmZ : sve_int_bin_pred_arit_1<0b001, "umax", "UMAX_ZPZZ", AArch64umax_m1, DestructiveBinaryComm>;
+  defm SMIN_ZPmZ : sve_int_bin_pred_arit_1<0b010, "smin", "SMIN_ZPZZ", AArch64smin_m1, DestructiveBinaryComm>;
+  defm UMIN_ZPmZ : sve_int_bin_pred_arit_1<0b011, "umin", "UMIN_ZPZZ", AArch64umin_m1, DestructiveBinaryComm>;
   defm SABD_ZPmZ : sve_int_bin_pred_arit_1<0b100, "sabd", "SABD_ZPZZ", int_aarch64_sve_sabd, DestructiveBinaryComm>;
   defm UABD_ZPmZ : sve_int_bin_pred_arit_1<0b101, "uabd", "UABD_ZPZZ", int_aarch64_sve_uabd, DestructiveBinaryComm>;
 
   defm SMAX_ZPZZ : sve_int_bin_pred_bhsd<AArch64smax_p>;
   defm UMAX_ZPZZ : sve_int_bin_pred_bhsd<AArch64umax_p>;
   defm SMIN_ZPZZ : sve_int_bin_pred_bhsd<AArch64smin_p>;
   defm UMIN_ZPZZ : sve_int_bin_pred_bhsd<AArch64umin_p>;
   defm SABD_ZPZZ : sve_int_bin_pred_bhsd<AArch64sabd_p>;

llvm/test/CodeGen/AArch64/sve-min-max-pred.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mattr=+sve -mtriple=aarch64  %s -o - | FileCheck %s --check-prefixes=CHECK
+
+; These tests just check that the plumbing is in place for @llvm.smax, @llvm.umax,
+; @llvm.smin, @llvm.umin.
+
+; tests for smax:
+
+define <vscale x 16 x i8> @smax_select_i8(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b) {
+; CHECK-LABEL: smax_select_i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    smax z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 16 x i8> @llvm.smax.nxv16i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b)
+  %out = select <vscale x 16 x i1> %pg, <vscale x 16 x i8> %sel, <vscale x 16 x i8> %a
+  ret <vscale x 16 x i8> %out
+}
+
+define <vscale x 8 x i16> @smax_select_i16(<vscale x 8 x i1> %pg, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b) {
+; CHECK-LABEL: smax_select_i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    smax z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 8 x i16> @llvm.smax.nxv8i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b)
+  %out = select <vscale x 8 x i1> %pg, <vscale x 8 x i16> %sel, <vscale x 8 x i16> %a
+  ret <vscale x 8 x i16> %out
+}
+
+define <vscale x 4 x i32> @smax_select_i32(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b) {
+; CHECK-LABEL: smax_select_i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    smax z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 4 x i32> @llvm.smax.nxv4i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b)
+  %out = select <vscale x 4 x i1> %pg, <vscale x 4 x i32> %sel, <vscale x 4 x i32> %a
+  ret <vscale x 4 x i32> %out
+}
+
+define <vscale x 2 x i64> @smax_select_i64(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b) {
+; CHECK-LABEL: smax_select_i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    smax z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 2 x i64> @llvm.smax.nxv2i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b)
+  %out = select <vscale x 2 x i1> %pg, <vscale x 2 x i64> %sel, <vscale x 2 x i64> %a
+  ret <vscale x 2 x i64> %out
+}
+
+; tests for umax:
+
+define <vscale x 16 x i8> @umax_select_i8(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b) {
+; CHECK-LABEL: umax_select_i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    umax z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 16 x i8> @llvm.umax.nxv16i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b)
+  %out = select <vscale x 16 x i1> %pg, <vscale x 16 x i8> %sel, <vscale x 16 x i8> %a
+  ret <vscale x 16 x i8> %out
+}
+
+define <vscale x 8 x i16> @umax_select_i16(<vscale x 8 x i1> %pg, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b) {
+; CHECK-LABEL: umax_select_i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    umax z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 8 x i16> @llvm.umax.nxv8i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b)
+  %out = select <vscale x 8 x i1> %pg, <vscale x 8 x i16> %sel, <vscale x 8 x i16> %a
+  ret <vscale x 8 x i16> %out
+}
+
+define <vscale x 4 x i32> @umax_select_i32(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b) {
+; CHECK-LABEL: umax_select_i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    umax z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 4 x i32> @llvm.umax.nxv4i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b)
+  %out = select <vscale x 4 x i1> %pg, <vscale x 4 x i32> %sel, <vscale x 4 x i32> %a
+  ret <vscale x 4 x i32> %out
+}
+
+define <vscale x 2 x i64> @umax_select_i64(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b) {
+; CHECK-LABEL: umax_select_i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    umax z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 2 x i64> @llvm.umax.nxv2i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b)
+  %out = select <vscale x 2 x i1> %pg, <vscale x 2 x i64> %sel, <vscale x 2 x i64> %a
+  ret <vscale x 2 x i64> %out
+}
+
+; tests for smin:
+
+define <vscale x 16 x i8> @smin_select_i8(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b) {
+; CHECK-LABEL: smin_select_i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    smin z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 16 x i8> @llvm.smin.nxv16i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b)
+  %out = select <vscale x 16 x i1> %pg, <vscale x 16 x i8> %sel, <vscale x 16 x i8> %a
+  ret <vscale x 16 x i8> %out
+}
+
+define <vscale x 8 x i16> @smin_select_i16(<vscale x 8 x i1> %pg, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b) {
+; CHECK-LABEL: smin_select_i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    smin z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 8 x i16> @llvm.smin.nxv8i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b)
+  %out = select <vscale x 8 x i1> %pg, <vscale x 8 x i16> %sel, <vscale x 8 x i16> %a
+  ret <vscale x 8 x i16> %out
+}
+
+define <vscale x 4 x i32> @smin_select_i32(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b) {
+; CHECK-LABEL: smin_select_i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    smin z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 4 x i32> @llvm.smin.nxv4i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b)
+  %out = select <vscale x 4 x i1> %pg, <vscale x 4 x i32> %sel, <vscale x 4 x i32> %a
+  ret <vscale x 4 x i32> %out
+}
+
+define <vscale x 2 x i64> @smin_select_i64(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b) {
+; CHECK-LABEL: smin_select_i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    smin z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 2 x i64> @llvm.smin.nxv2i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b)
+  %out = select <vscale x 2 x i1> %pg, <vscale x 2 x i64> %sel, <vscale x 2 x i64> %a
+  ret <vscale x 2 x i64> %out
+}
+
+; tests for umin:
+
+define <vscale x 16 x i8> @umin_select_i8(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b) {
+; CHECK-LABEL: umin_select_i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    umin z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 16 x i8> @llvm.umin.nxv16i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b)
+  %out = select <vscale x 16 x i1> %pg, <vscale x 16 x i8> %sel, <vscale x 16 x i8> %a
+  ret <vscale x 16 x i8> %out
+}
+
+define <vscale x 8 x i16> @umin_select_i16(<vscale x 8 x i1> %pg, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b) {
+; CHECK-LABEL: umin_select_i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    umin z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 8 x i16> @llvm.umin.nxv8i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b)
+  %out = select <vscale x 8 x i1> %pg, <vscale x 8 x i16> %sel, <vscale x 8 x i16> %a
+  ret <vscale x 8 x i16> %out
+}
+
+define <vscale x 4 x i32> @umin_select_i32(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b) {
+; CHECK-LABEL: umin_select_i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    umin z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 4 x i32> @llvm.umin.nxv4i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b)
+  %out = select <vscale x 4 x i1> %pg, <vscale x 4 x i32> %sel, <vscale x 4 x i32> %a
+  ret <vscale x 4 x i32> %out
+}
+
+define <vscale x 2 x i64> @umin_select_i64(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b) {
+; CHECK-LABEL: umin_select_i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    umin z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    ret
+  %sel = call <vscale x 2 x i64> @llvm.umin.nxv2i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b)
+  %out = select <vscale x 2 x i1> %pg, <vscale x 2 x i64> %sel, <vscale x 2 x i64> %a
+  ret <vscale x 2 x i64> %out
+}
+
+
+declare <vscale x  16 x i8> @llvm.smax.nxv16i8(<vscale x  16 x i8>, <vscale x  16 x i8>)
+declare <vscale x  8 x i16> @llvm.smax.nxv8i16(<vscale x  8 x i16>, <vscale x  8 x i16>)
+declare <vscale x  4 x i32> @llvm.smax.nxv4i32(<vscale x  4 x i32>, <vscale x  4 x i32>)
+declare <vscale x  2 x i64> @llvm.smax.nxv2i64(<vscale x  2 x i64>, <vscale x  2 x i64>)
+
+declare <vscale x  16 x i8> @llvm.umax.nxv16i8(<vscale x  16 x i8>, <vscale x  16 x i8>)
+declare <vscale x  8 x i16> @llvm.umax.nxv8i16(<vscale x  8 x i16>, <vscale x  8 x i16>)
+declare <vscale x  4 x i32> @llvm.umax.nxv4i32(<vscale x  4 x i32>, <vscale x  4 x i32>)
+declare <vscale x  2 x i64> @llvm.umax.nxv2i64(<vscale x  2 x i64>, <vscale x  2 x i64>)
+
+declare <vscale x  16 x i8> @llvm.smin.nxv16i8(<vscale x  16 x i8>, <vscale x  16 x i8>)
+declare <vscale x  8 x i16> @llvm.smin.nxv8i16(<vscale x  8 x i16>, <vscale x  8 x i16>)
+declare <vscale x  4 x i32> @llvm.smin.nxv4i32(<vscale x  4 x i32>, <vscale x  4 x i32>)
+declare <vscale x  2 x i64> @llvm.smin.nxv2i64(<vscale x  2 x i64>, <vscale x  2 x i64>)
+
+declare <vscale x  16 x i8> @llvm.umin.nxv16i8(<vscale x  16 x i8>, <vscale x  16 x i8>)
+declare <vscale x  8 x i16> @llvm.umin.nxv8i16(<vscale x  8 x i16>, <vscale x  8 x i16>)
+declare <vscale x  4 x i32> @llvm.umin.nxv4i32(<vscale x  4 x i32>, <vscale x  4 x i32>)
+declare <vscale x  2 x i64> @llvm.umin.nxv2i64(<vscale x  2 x i64>, <vscale x  2 x i64>)