[SVE] Add SVE2 patterns for unpredicated multiply instructions

[dancgr]

Summary

Add patterns for SVE2 unpredicated multiply instructions:

• mul, smulh, umulh, pmul, sqdmulh, sqrdmulh

Also removed negative tests for immediate multiply that are set up to fail, since the case now can succeed and also is handled in the new test.

[efriedma]

Can we also also add mul patterns for targets that have SVE, but not SVE2?

llvm/include/llvm/IR/IntrinsicsAArch64.td

1207

This name seems strange. Why "z"?

[dancgr]

In D72799#1822588, @efriedma wrote:

Can we also also add mul patterns for targets that have SVE, but not SVE2?

That instruction is restricted to SVE2. Do we have unpredicated vector mul instructions for SVE targets as well?

The way the instruction is implemented did not allow me to match with SVE targets.

llvm/include/llvm/IR/IntrinsicsAArch64.td

1207

I just followed the name similar to the one for logical predicated instructions. I'm not sure if that is the correct way to name predicated intrinsics. I marked the place with a comment.

1562

@efriedma Here!

[efriedma]

In D72799#1822599, @dancgr wrote:

In D72799#1822588, @efriedma wrote:

Can we also also add mul patterns for targets that have SVE, but not SVE2?

That instruction is restricted to SVE2. Do we have unpredicated vector mul instructions for SVE targets as well?

We can synthesize a predicate using ptrue. We do this in a few places currently; for example, to implement bswap.

llvm/include/llvm/IR/IntrinsicsAArch64.td

1562

I think here the "_z" is meant to indicate that the predicate operand is "<Pg>/Z".

[dancgr]

In D72799#1822788, @efriedma wrote:

In D72799#1822599, @dancgr wrote:

In D72799#1822588, @efriedma wrote:

Can we also also add mul patterns for targets that have SVE, but not SVE2?

That instruction is restricted to SVE2. Do we have unpredicated vector mul instructions for SVE targets as well?

We can synthesize a predicate using ptrue. We do this in a few places currently; for example, to implement bswap.

I hadn't thought of that. I will implement SVE fallbacks for the SVE2 instructions using ptrue.

llvm/include/llvm/IR/IntrinsicsAArch64.td

1562

Ok, I will update it to _m to follow the same pattern.

[sdesmalen]

Thanks for efforts on this patch @dancgr!

As you can see, I have requested some changes to the patch to avoid diverging too much while we try to upstream the ACLE.
We also need to make sure we can represent the C/C++ intrinsics with the LLVM IR intrinsics, which is a reason to have predicated smulh/umulh.

llvm/include/llvm/IR/IntrinsicsAArch64.td

1201

smulh needs to be predicated AdvSIMD_Pred2VectorArg_Intrinsic, so that we can implement the predicated ACLE intrinsic with this LLVM IR intrinsic.
You can choose to optimize this predicated form (with ptrue predicate mask) into an unpredicated form using a ISel pattern.

1202

Same for umulh.

1207

Adding _z here implies that the false lanes will be zeroed, which isn't the case, as this is used with merging predication.
These intrinsics are to implement the ACLE, which does the zeroing explicitly with a select. These intrinsics are merging, unless explicitly specified.

1562

The reason that these intrinsics have _z in the name is because they implement an operation with zeroing predication (which maps directly to predicate AND that zeroes the false lanes).

Please don't change these suffixes, as this will give us trouble while we're in the process of implementing the ACLE upstream. We would need to update our Clang implementation to target intrinsics with slightly different names.

llvm/lib/Target/AArch64/SVEInstrFormats.td

2636

nit: sve2_int_mul2 -> sve2_int_mul_single ?

[dancgr]

I'm not sure on some parts, but I have prepared a major update for this patch that I hope will fix most of @sdesmalen concerns.

llvm/include/llvm/IR/IntrinsicsAArch64.td

1201

Ok, but we have already a predicated smulh intrinsic, which is the int_aarch64_sve_smulh_m. Do we need the predicated version to be named exactly int_aarch64_sve_smulh?

1207

Ok, makes sense now.

1562

I'm not planning on changing these, I just wanted to understand the naming convention. Now it is clear to me.

[dancgr]

Tentative changes to address some comments from the reviewers.

[dancgr]

I'm not sure on some parts, but I have prepared a major update for this patch that I hope will fix most of @sdesmalen concerns.

[sdesmalen]

In D72799#1829698, @dancgr wrote:

I'm not sure on some parts, but I have prepared a major update for this patch that I hope will fix most of @sdesmalen concerns.

Great, thanks for updating your patch! The patch is nearly good to go I think, just one more little change needed to remove _m.

llvm/include/llvm/IR/IntrinsicsAArch64.td

1207

Given that merging is the default, we don't explicitly specify the _m in the intrinsic name.

[dancgr]

Remove _m from intrinsics, as it is the default behaviour. Also, add unpredicated patterns for SVE2 smulh, umulh.

[dancgr]

llvm/include/llvm/IR/IntrinsicsAArch64.td

1207

Ok, I will update that too.

[sdesmalen]

Thanks for the updates to the patch @dancgr , LGTM!

llvm/include/llvm/IR/IntrinsicsAArch64.td

 //
 // Integer arithmetic
 //
 
 def int_aarch64_sve_add   : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_sub   : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_subr  : AdvSIMD_Pred2VectorArg_Intrinsic;
 
+def int_aarch64_sve_pmul       : AdvSIMD_2VectorArg_Intrinsic;
+def int_aarch64_sve_sqdmulh    : AdvSIMD_2VectorArg_Intrinsic;
+def int_aarch64_sve_sqrdmulh   : AdvSIMD_2VectorArg_Intrinsic;
+
 def int_aarch64_sve_mul        : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_smulh      : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_umulh      : AdvSIMD_Pred2VectorArg_Intrinsic;
 
 def int_aarch64_sve_sdiv       : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_udiv       : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_sdivr      : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_udivr      : AdvSIMD_Pred2VectorArg_Intrinsic;
 
 def int_aarch64_sve_smax       : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_umax       : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_smin       : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_umin       : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_sabd       : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_uabd       : AdvSIMD_Pred2VectorArg_Intrinsic;
 
 def int_aarch64_sve_mad        : AdvSIMD_Pred3VectorArg_Intrinsic;

 //
 
 def int_aarch64_sve_ptrue : AdvSIMD_SVE_PTRUE_Intrinsic;
 
 //
 // Predicate operations
 //
 
 def int_aarch64_sve_and_z   : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_bic_z   : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_brka    : AdvSIMD_Merged1VectorArg_Intrinsic;
 def int_aarch64_sve_brka_z  : AdvSIMD_Pred1VectorArg_Intrinsic;
 def int_aarch64_sve_brkb    : AdvSIMD_Merged1VectorArg_Intrinsic;
 def int_aarch64_sve_brkb_z  : AdvSIMD_Pred1VectorArg_Intrinsic;
 def int_aarch64_sve_brkn_z  : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_brkpa_z : AdvSIMD_Pred2VectorArg_Intrinsic;
 def int_aarch64_sve_brkpb_z : AdvSIMD_Pred2VectorArg_Intrinsic;

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

let Predicates = [HasSVE] in {
   defm EOR_ZI : sve_int_log_imm<0b01, "eor", "eon", xor>;
   defm AND_ZI : sve_int_log_imm<0b10, "and", "bic", and>;
 
   defm SMAX_ZI   : sve_int_arith_imm1<0b00, "smax", smax>;
   defm SMIN_ZI   : sve_int_arith_imm1<0b10, "smin", smin>;
   defm UMAX_ZI   : sve_int_arith_imm1_unsigned<0b01, "umax", umax>;
   defm UMIN_ZI   : sve_int_arith_imm1_unsigned<0b11, "umin", umin>;
 
-  defm MUL_ZI    : sve_int_arith_imm2<"mul", mul>;
-  defm MUL_ZPmZ   : sve_int_bin_pred_arit_2<0b000, "mul", int_aarch64_sve_mul>;
+  defm MUL_ZI     : sve_int_arith_imm2<"mul", mul>;
+  defm MUL_ZPmZ   : sve_int_bin_pred_arit_2<0b000, "mul",   int_aarch64_sve_mul>;
   defm SMULH_ZPmZ : sve_int_bin_pred_arit_2<0b010, "smulh", int_aarch64_sve_smulh>;
   defm UMULH_ZPmZ : sve_int_bin_pred_arit_2<0b011, "umulh", int_aarch64_sve_umulh>;
 
+  // Add unpredicated alternative for the mul instruction.
+  def : Pat<(mul nxv16i8:$Op1, nxv16i8:$Op2),
+            (MUL_ZPmZ_B (PTRUE_B 31), $Op1, $Op2)>;
+  def : Pat<(mul nxv8i16:$Op1, nxv8i16:$Op2),
+            (MUL_ZPmZ_H (PTRUE_H 31), $Op1, $Op2)>;
+  def : Pat<(mul nxv4i32:$Op1, nxv4i32:$Op2),
+            (MUL_ZPmZ_S (PTRUE_S 31), $Op1, $Op2)>;
+  def : Pat<(mul nxv2i64:$Op1, nxv2i64:$Op2),
+            (MUL_ZPmZ_D (PTRUE_D 31), $Op1, $Op2)>;
+
   defm SDIV_ZPmZ  : sve_int_bin_pred_arit_2_div<0b100, "sdiv", int_aarch64_sve_sdiv>;
   defm UDIV_ZPmZ  : sve_int_bin_pred_arit_2_div<0b101, "udiv", int_aarch64_sve_udiv>;
   defm SDIVR_ZPmZ : sve_int_bin_pred_arit_2_div<0b110, "sdivr", int_aarch64_sve_sdivr>;
   defm UDIVR_ZPmZ : sve_int_bin_pred_arit_2_div<0b111, "udivr", int_aarch64_sve_udivr>;
 
   defm SDOT_ZZZ : sve_intx_dot<0b0, "sdot", int_aarch64_sve_sdot>;
   defm UDOT_ZZZ : sve_intx_dot<0b1, "udot", int_aarch64_sve_udot>;
 
let Predicates = [HasSVE2] in {
   // SVE2 integer multiply (indexed)
   defm MUL_ZZZI : sve2_int_mul_by_indexed_elem<0b1110, "mul">;
 
   // SVE2 saturating multiply high (indexed)
   defm SQDMULH_ZZZI  : sve2_int_mul_by_indexed_elem<0b1100, "sqdmulh">;
   defm SQRDMULH_ZZZI : sve2_int_mul_by_indexed_elem<0b1101, "sqrdmulh">;
 
   // SVE2 signed saturating doubling multiply high (unpredicated)
-  defm SQDMULH_ZZZ  : sve2_int_mul<0b100, "sqdmulh">;
-  defm SQRDMULH_ZZZ : sve2_int_mul<0b101, "sqrdmulh">;
+  defm SQDMULH_ZZZ  : sve2_int_mul<0b100, "sqdmulh",  int_aarch64_sve_sqdmulh>;
+  defm SQRDMULH_ZZZ : sve2_int_mul<0b101, "sqrdmulh", int_aarch64_sve_sqrdmulh>;
 
   // SVE2 integer multiply vectors (unpredicated)
-  defm MUL_ZZZ    : sve2_int_mul<0b000, "mul">;
-  defm SMULH_ZZZ  : sve2_int_mul<0b010, "smulh">;
-  defm UMULH_ZZZ  : sve2_int_mul<0b011, "umulh">;
-  def  PMUL_ZZZ_B : sve2_int_mul<0b00, 0b001, "pmul", ZPR8>;
-
+  defm MUL_ZZZ    : sve2_int_mul<0b000,  "mul",   mul>;
+  defm SMULH_ZZZ  : sve2_int_mul<0b010,  "smulh", null_frag>;
+  defm UMULH_ZZZ  : sve2_int_mul<0b011,  "umulh", null_frag>;
+  defm PMUL_ZZZ   : sve2_int_mul_single<0b001, "pmul",  int_aarch64_sve_pmul>;
+
+  // Add patterns for unpredicated version of smulh and umulh.
+  def : Pat<(nxv16i8 (int_aarch64_sve_smulh (nxv16i1 (AArch64ptrue 31)), nxv16i8:$Op1, nxv16i8:$Op2)),
+            (SMULH_ZZZ_B $Op1, $Op2)>;
+  def : Pat<(nxv8i16 (int_aarch64_sve_smulh (nxv8i1 (AArch64ptrue 31)), nxv8i16:$Op1, nxv8i16:$Op2)),
+            (SMULH_ZZZ_H $Op1, $Op2)>;
+  def : Pat<(nxv4i32 (int_aarch64_sve_smulh (nxv4i1 (AArch64ptrue 31)), nxv4i32:$Op1, nxv4i32:$Op2)),
+            (SMULH_ZZZ_S $Op1, $Op2)>;
+  def : Pat<(nxv2i64 (int_aarch64_sve_smulh (nxv2i1 (AArch64ptrue 31)), nxv2i64:$Op1, nxv2i64:$Op2)),
+            (SMULH_ZZZ_D $Op1, $Op2)>;
+  def : Pat<(nxv16i8 (int_aarch64_sve_umulh (nxv16i1 (AArch64ptrue 31)), nxv16i8:$Op1, nxv16i8:$Op2)),
+            (UMULH_ZZZ_B $Op1, $Op2)>;
+  def : Pat<(nxv8i16 (int_aarch64_sve_umulh (nxv8i1 (AArch64ptrue 31)), nxv8i16:$Op1, nxv8i16:$Op2)),
+            (UMULH_ZZZ_H $Op1, $Op2)>;
+  def : Pat<(nxv4i32 (int_aarch64_sve_umulh (nxv4i1 (AArch64ptrue 31)), nxv4i32:$Op1, nxv4i32:$Op2)),
+            (UMULH_ZZZ_S $Op1, $Op2)>;
+  def : Pat<(nxv2i64 (int_aarch64_sve_umulh (nxv2i1 (AArch64ptrue 31)), nxv2i64:$Op1, nxv2i64:$Op2)),
+            (UMULH_ZZZ_D $Op1, $Op2)>;
   // SVE2 complex integer dot product (indexed)
   defm CDOT_ZZZI : sve2_cintx_dot_by_indexed_elem<"cdot">;
 
   // SVE2 complex integer dot product
   defm CDOT_ZZZ : sve2_cintx_dot<"cdot">;
 
   // SVE2 complex integer multiply-add (indexed)
   defm CMLA_ZZZI      : sve2_cmla_by_indexed_elem<0b0, "cmla">;

llvm/lib/Target/AArch64/SVEInstrFormats.td

: I<(outs zprty:$Zd), (ins zprty:$Zn, zprty:$Zm),
   let Inst{21}    = 0b1;
   let Inst{20-16} = Zm;
   let Inst{15-13} = 0b011;
   let Inst{12-10} = opc;
   let Inst{9-5}   = Zn;
   let Inst{4-0}   = Zd;
 }
 
-multiclass sve2_int_mul<bits<3> opc, string asm> {
+multiclass sve2_int_mul<bits<3> opc, string asm, SDPatternOperator op> {
   def _B : sve2_int_mul<0b00, opc, asm, ZPR8>;
   def _H : sve2_int_mul<0b01, opc, asm, ZPR16>;
   def _S : sve2_int_mul<0b10, opc, asm, ZPR32>;
   def _D : sve2_int_mul<0b11, opc, asm, ZPR64>;
+
+  def : SVE_2_Op_Pat<nxv16i8, op, nxv16i8, nxv16i8, !cast<Instruction>(NAME # _B)>;
+  def : SVE_2_Op_Pat<nxv8i16, op, nxv8i16, nxv8i16, !cast<Instruction>(NAME # _H)>;
+  def : SVE_2_Op_Pat<nxv4i32, op, nxv4i32, nxv4i32, !cast<Instruction>(NAME # _S)>;
+  def : SVE_2_Op_Pat<nxv2i64, op, nxv2i64, nxv2i64, !cast<Instruction>(NAME # _D)>;
+}
+
+multiclass sve2_int_mul_single<bits<3> opc, string asm, SDPatternOperator op> {
+  def _B : sve2_int_mul<0b00, opc, asm, ZPR8>;
+
+  def : SVE_2_Op_Pat<nxv16i8, op, nxv16i8, nxv16i8, !cast<Instruction>(NAME # _B)>;
 }
 
 //===----------------------------------------------------------------------===//
 // SVE2 Integer Multiply - Indexed Group
 //===----------------------------------------------------------------------===//
 
 class sve2_int_mul_by_indexed_elem<bits<2> sz, bits<4> opc, string asm,
                                    ZPRRegOp zprty1, ZPRRegOp zprty2,

llvm/test/CodeGen/AArch64/sve-int-arith-imm.ll

 ; CHECK-LABEL: mul_i64_pos
 ; CHECK: mul z0.d, z0.d, #105
 ; CHECK-NEXT: ret
   %elt = insertelement <vscale x 2 x i64> undef, i64 105, i32 0
   %splat = shufflevector <vscale x 2 x i64> %elt, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
   %res = mul <vscale x 2 x i64> %a, %splat
   ret <vscale x 2 x i64> %res
 }
+
+define <vscale x 8 x i16> @mul_i16_range(<vscale x 8 x i16> %a) {
+; CHECK-LABEL: mul_i16_range
+; CHECK: mov w[[W:[0-9]+]], #255
+; CHECK-NEXT: mov z1.h, w[[W]]
+; CHECK: ptrue p0.h
+; CHECK-NEXT: mul z0.h, p0/m, z0.h, z1.h
+  %elt = insertelement <vscale x 8 x i16> undef, i16 255, i32 0
+  %splat = shufflevector <vscale x 8 x i16> %elt, <vscale x 8 x i16> undef, <vscale x 8 x i32> zeroinitializer
+  %res = mul <vscale x 8 x i16> %a, %splat
+  ret <vscale x 8 x i16> %res
+}
+
+define <vscale x 4 x i32> @mul_i32_range(<vscale x 4 x i32> %a) {
+; CHECK-LABEL: mul_i32_range
+; CHECK: mov w[[W:[0-9]+]], #255
+; CHECK-NEXT: mov z1.s, w[[W]]
+; CHECK: ptrue p0.s
+; CHECK-NEXT: mul z0.s, p0/m, z0.s, z1.s
+  %elt = insertelement <vscale x 4 x i32> undef, i32 255, i32 0
+  %splat = shufflevector <vscale x 4 x i32> %elt, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  %res = mul <vscale x 4 x i32> %a, %splat
+  ret <vscale x 4 x i32> %res
+}
+
+define <vscale x 2 x i64> @mul_i64_range(<vscale x 2 x i64> %a) {
+; CHECK-LABEL: mul_i64_range
+; CHECK: mov w[[W:[0-9]+]], #255
+; CHECK-NEXT: mov z1.d, x[[W]]
+; CHECK: ptrue p0.d
+; CHECK-NEXT: mul z0.d, p0/m, z0.d, z1.d
+  %elt = insertelement <vscale x 2 x i64> undef, i64 255, i32 0
+  %splat = shufflevector <vscale x 2 x i64> %elt, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+  %res = mul <vscale x 2 x i64> %a, %splat
+  ret <vscale x 2 x i64> %res
+}

llvm/test/CodeGen/AArch64/sve-int-mul-pred.ll

; CHECK-NEXT: ret
   ret <vscale x 2 x i64> %out
 }
 
 define <vscale x 16 x i8> @smulh_i8(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b) {
 ; CHECK-LABEL: smulh_i8:
 ; CHECK: smulh z0.b, p0/m, z0.b, z1.b
 ; CHECK-NEXT: ret
   %out = call <vscale x 16 x i8> @llvm.aarch64.sve.smulh.nxv16i8(<vscale x 16 x i1> %pg,
                                                                 <vscale x 16 x i8> %a,
                                                                 <vscale x 16 x i8> %b)
   ret <vscale x 16 x i8> %out
 }
 
 define <vscale x 8 x i16> @smulh_i16(<vscale x 8 x i1> %pg, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b) {
 ; CHECK-LABEL: smulh_i16:
 ; CHECK: smulh z0.h, p0/m, z0.h, z1.h
 ; CHECK-NEXT: ret
   %out = call <vscale x 8 x i16> @llvm.aarch64.sve.smulh.nxv8i16(<vscale x 8 x i1> %pg,
                                                                 <vscale x 8 x i16> %a,
                                                                 <vscale x 8 x i16> %b)
   ret <vscale x 8 x i16> %out
 }
 
 define <vscale x 4 x i32> @smulh_i32(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b) {
 ; CHECK-LABEL: smulh_i32:
 ; CHECK: smulh z0.s, p0/m, z0.s, z1.s
 ; CHECK-NEXT: ret
   %out = call <vscale x 4 x i32> @llvm.aarch64.sve.smulh.nxv4i32(<vscale x 4 x i1> %pg,
                                                                 <vscale x 4 x i32> %a,
                                                                 <vscale x 4 x i32> %b)
   ret <vscale x 4 x i32> %out
 }
 
 define <vscale x 2 x i64> @smulh_i64(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b) {
 ; CHECK-LABEL: smulh_i64:
 ; CHECK: smulh z0.d, p0/m, z0.d, z1.d
 ; CHECK-NEXT: ret
   %out = call <vscale x 2 x i64> @llvm.aarch64.sve.smulh.nxv2i64(<vscale x 2 x i1> %pg,
                                                                 <vscale x 2 x i64> %a,
                                                                 <vscale x 2 x i64> %b)
   ret <vscale x 2 x i64> %out
 }
 
 define <vscale x 16 x i8> @umulh_i8(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b) {
 ; CHECK-LABEL: umulh_i8:
 ; CHECK: umulh z0.b, p0/m, z0.b, z1.b
 ; CHECK-NEXT: ret
   %out = call <vscale x 16 x i8> @llvm.aarch64.sve.umulh.nxv16i8(<vscale x 16 x i1> %pg,
                                                                 <vscale x 16 x i8> %a,
                                                                 <vscale x 16 x i8> %b)
   ret <vscale x 16 x i8> %out
 }
 
 define <vscale x 8 x i16> @umulh_i16(<vscale x 8 x i1> %pg, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b) {
 ; CHECK-LABEL: umulh_i16:
 ; CHECK: umulh z0.h, p0/m, z0.h, z1.h
 ; CHECK-NEXT: ret
   %out = call <vscale x 8 x i16> @llvm.aarch64.sve.umulh.nxv8i16(<vscale x 8 x i1> %pg,
                                                                 <vscale x 8 x i16> %a,
                                                                 <vscale x 8 x i16> %b)
   ret <vscale x 8 x i16> %out
 }
 
 define <vscale x 4 x i32> @umulh_i32(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b) {
 ; CHECK-LABEL: umulh_i32:
 ; CHECK: umulh z0.s, p0/m, z0.s, z1.s
 ; CHECK-NEXT: ret
   %out = call <vscale x 4 x i32> @llvm.aarch64.sve.umulh.nxv4i32(<vscale x 4 x i1> %pg,
                                                                  <vscale x 4 x i32> %a,
                                                                  <vscale x 4 x i32> %b)
   ret <vscale x 4 x i32> %out
 }
 
 define <vscale x 2 x i64> @umulh_i64(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b) {
 ; CHECK-LABEL: umulh_i64:
 ; CHECK: umulh z0.d, p0/m, z0.d, z1.d
 ; CHECK-NEXT: ret
   %out = call <vscale x 2 x i64> @llvm.aarch64.sve.umulh.nxv2i64(<vscale x 2 x i1> %pg,
                                                                  <vscale x 2 x i64> %a,
                                                                  <vscale x 2 x i64> %b)
   ret <vscale x 2 x i64> %out
 }
 
 declare <vscale x 16 x  i8> @llvm.aarch64.sve.mul.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x  i8>, <vscale x 16 x  i8>)
 declare <vscale x  8 x i16> @llvm.aarch64.sve.mul.nxv8i16(<vscale x  8 x i1>, <vscale x  8 x i16>, <vscale x  8 x i16>)
 declare <vscale x  4 x i32> @llvm.aarch64.sve.mul.nxv4i32(<vscale x  4 x i1>, <vscale x  4 x i32>, <vscale x  4 x i32>)
 declare <vscale x  2 x i64> @llvm.aarch64.sve.mul.nxv2i64(<vscale x  2 x i1>, <vscale x  2 x i64>, <vscale x  2 x i64>)
 declare <vscale x 16 x  i8> @llvm.aarch64.sve.smulh.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x  i8>, <vscale x 16 x  i8>)
 declare <vscale x  8 x i16> @llvm.aarch64.sve.smulh.nxv8i16(<vscale x  8 x i1>, <vscale x  8 x i16>, <vscale x  8 x i16>)
 declare <vscale x  4 x i32> @llvm.aarch64.sve.smulh.nxv4i32(<vscale x  4 x i1>, <vscale x  4 x i32>, <vscale x  4 x i32>)
 declare <vscale x  2 x i64> @llvm.aarch64.sve.smulh.nxv2i64(<vscale x  2 x i1>, <vscale x  2 x i64>, <vscale x  2 x i64>)
 declare <vscale x 16 x  i8> @llvm.aarch64.sve.umulh.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x  i8>, <vscale x 16 x  i8>)
 declare <vscale x  8 x i16> @llvm.aarch64.sve.umulh.nxv8i16(<vscale x  8 x i1>, <vscale x  8 x i16>, <vscale x  8 x i16>)
 declare <vscale x  4 x i32> @llvm.aarch64.sve.umulh.nxv4i32(<vscale x  4 x i1>, <vscale x  4 x i32>, <vscale x  4 x i32>)
 declare <vscale x  2 x i64> @llvm.aarch64.sve.umulh.nxv2i64(<vscale x  2 x i1>, <vscale x  2 x i64>, <vscale x  2 x i64>)

llvm/test/CodeGen/AArch64/sve-neg-int-arith-imm-2.ll

-; RUN: not llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s
-
-; Numbers smaller than -127 and greater than or equal to 127 are not allowed.
-; This should get lowered to a regular vector multiply and these tests should
-; be updated when those patterns are added.
-
-define <vscale x 2 x i64> @mul_i64_neg_1(<vscale x 2 x i64> %a) {
-  %elt = insertelement <vscale x 2 x i64> undef, i64 255, i32 0
-  %splat = shufflevector <vscale x 2 x i64> %elt, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
-  %res = mul <vscale x 2 x i64> %a, %splat
-  ret <vscale x 2 x i64> %res
-}

llvm/test/CodeGen/AArch64/sve-neg-int-arith-imm.ll

-; RUN: not llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s
-
-; Numbers smaller than -127 and greater than or equal to 127 allowed for imm mul.
-; This should get lowered to a regular vector multiply and these tests should
-; be updated when those patterns are added.
-define <vscale x 2 x i64> @mul_i64_neg_1(<vscale x 2 x i64> %a) {
-  %elt = insertelement <vscale x 2 x i64> undef, i64 -130, i32 0
-  %splat = shufflevector <vscale x 2 x i64> %elt, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
-  %res = mul <vscale x 2 x i64> %a, %splat
-  ret <vscale x 2 x i64> %res
-}

llvm/test/CodeGen/AArch64/sve2-int-mul.ll

+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve2 < %s | FileCheck %s
+
+;
+; MUL with SPLAT
+;
+define <vscale x 8 x i16> @mul_i16_imm(<vscale x 8 x i16> %a) {
+; CHECK-LABEL: mul_i16_imm
+; CHECK: mov w[[W:[0-9]+]], #255
+; CHECK-NEXT: mov z1.h, w[[W]]
+; CHECK-NEXT: mul z0.h, z0.h, z1.h
+  %elt = insertelement <vscale x 8 x i16> undef, i16 255, i32 0
+  %splat = shufflevector <vscale x 8 x i16> %elt, <vscale x 8 x i16> undef, <vscale x 8 x i32> zeroinitializer
+  %res = mul <vscale x 8 x i16> %a, %splat
+  ret <vscale x 8 x i16> %res
+}
+
+define <vscale x 8 x i16> @mul_i16_imm_neg(<vscale x 8 x i16> %a) {
+; CHECK-LABEL: mul_i16_imm_neg
+; CHECK: mov w[[W:[0-9]+]], #-200
+; CHECK-NEXT: mov z1.h, w[[W]]
+; CHECK-NEXT: mul z0.h, z0.h, z1.h
+  %elt = insertelement <vscale x 8 x i16> undef, i16 -200, i32 0
+  %splat = shufflevector <vscale x 8 x i16> %elt, <vscale x 8 x i16> undef, <vscale x 8 x i32> zeroinitializer
+  %res = mul <vscale x 8 x i16> %a, %splat
+  ret <vscale x 8 x i16> %res
+}
+
+define <vscale x 4 x i32> @mul_i32_imm(<vscale x 4 x i32> %a) {
+; CHECK-LABEL: mul_i32_imm
+; CHECK: mov w[[W:[0-9]+]], #255
+; CHECK-NEXT: mov z1.s, w[[W]]
+; CHECK-NEXT: mul z0.s, z0.s, z1.s
+  %elt = insertelement <vscale x 4 x i32> undef, i32 255, i32 0
+  %splat = shufflevector <vscale x 4 x i32> %elt, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  %res = mul <vscale x 4 x i32> %a, %splat
+  ret <vscale x 4 x i32> %res
+}
+
+define <vscale x 4 x i32> @mul_i32_imm_neg(<vscale x 4 x i32> %a) {
+; CHECK-LABEL: mul_i32_imm_neg
+; CHECK: mov w[[W:[0-9]+]], #-200
+; CHECK-NEXT: mov z1.s, w[[W]]
+; CHECK-NEXT: mul z0.s, z0.s, z1.s
+  %elt = insertelement <vscale x 4 x i32> undef, i32 -200, i32 0
+  %splat = shufflevector <vscale x 4 x i32> %elt, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  %res = mul <vscale x 4 x i32> %a, %splat
+  ret <vscale x 4 x i32> %res
+}
+
+define <vscale x 2 x i64> @mul_i64_imm(<vscale x 2 x i64> %a) {
+; CHECK-LABEL: mul_i64_imm
+; CHECK: mov w[[X:[0-9]+]], #255
+; CHECK-NEXT: z1.d, x[[X]]
+; CHECK-NEXT: mul z0.d, z0.d, z1.d
+  %elt = insertelement <vscale x 2 x i64> undef, i64 255, i32 0
+  %splat = shufflevector <vscale x 2 x i64> %elt, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+  %res = mul <vscale x 2 x i64> %a, %splat
+  ret <vscale x 2 x i64> %res
+}
+
+define <vscale x 2 x i64> @mul_i64_imm_neg(<vscale x 2 x i64> %a) {
+; CHECK-LABEL: mul_i64_imm_neg
+; CHECK: mov x[[X:[0-9]+]], #-200
+; CHECK-NEXT: z1.d, x[[X]]
+; CHECK-NEXT: mul z0.d, z0.d, z1.d
+  %elt = insertelement <vscale x 2 x i64> undef, i64 -200, i32 0
+  %splat = shufflevector <vscale x 2 x i64> %elt, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+  %res = mul <vscale x 2 x i64> %a, %splat
+  ret <vscale x 2 x i64> %res
+}
+
+;
+; MUL (vector, unpredicated)
+;
+define <vscale x 16 x i8> @mul_i8(<vscale x 16 x i8> %a,
+                                  <vscale x 16 x i8> %b) {
+; CHECK-LABEL: mul_i8
+; CHECK: mul z0.b, z0.b, z1.b
+; CHECK-NEXT: ret
+  %res = mul <vscale x 16 x i8> %a, %b
+  ret <vscale x 16 x i8> %res
+}
+
+define <vscale x 8 x i16> @mul_i16(<vscale x 8 x i16> %a,
+                                  <vscale x 8 x i16> %b) {
+; CHECK-LABEL: mul_i16
+; CHECK: mul z0.h, z0.h, z1.h
+; CHECK-NEXT: ret
+  %res = mul <vscale x 8 x i16> %a, %b
+  ret <vscale x 8 x i16> %res
+}
+
+define <vscale x 4 x i32> @mul_i32(<vscale x 4 x i32> %a,
+                                  <vscale x 4 x i32> %b) {
+; CHECK-LABEL: mul_i32
+; CHECK: mul z0.s, z0.s, z1.s
+; CHECK-NEXT: ret
+  %res = mul <vscale x 4 x i32> %a, %b
+  ret <vscale x 4 x i32> %res
+}
+
+define <vscale x 2 x i64> @mul_i64(<vscale x 2 x i64> %a,
+                                  <vscale x 2 x i64> %b) {
+; CHECK-LABEL: mul_i64
+; CHECK: mul z0.d, z0.d, z1.d
+; CHECK-NEXT: ret
+  %res = mul <vscale x 2 x i64> %a, %b
+  ret <vscale x 2 x i64> %res
+}
+
+;
+; SMULH (vector, unpredicated)
+;
+define <vscale x 16 x i8> @smulh_i8(<vscale x 16 x i8> %a,
+                                    <vscale x 16 x i8> %b) {
+; CHECK-LABEL: smulh_i8
+; CHECK: smulh z0.b, z0.b, z1.b
+; CHECK-NEXT: ret
+  %sel = call <vscale x 16 x i1> @llvm.aarch64.sve.ptrue.nxv16i1(i32 31)
+  %res = call <vscale x 16 x i8> @llvm.aarch64.sve.smulh.nxv16i8(<vscale x 16 x i1> %sel, <vscale x 16 x i8> %a,
+                                                                 <vscale x 16 x i8> %b)
+  ret <vscale x 16 x i8> %res
+}
+
+define <vscale x 8 x i16> @smulh_i16(<vscale x 8 x i16> %a,
+                                     <vscale x 8 x i16> %b) {
+; CHECK-LABEL: smulh_i16
+; CHECK: smulh z0.h, z0.h, z1.h
+; CHECK-NEXT: ret
+  %sel = call <vscale x 8 x i1> @llvm.aarch64.sve.ptrue.nxv8i1(i32 31)
+  %res = call <vscale x 8 x i16> @llvm.aarch64.sve.smulh.nxv8i16(<vscale x 8 x i1> %sel, <vscale x 8 x i16> %a,
+                                                                 <vscale x 8 x i16> %b)
+  ret <vscale x 8 x i16> %res
+}
+
+define <vscale x 4 x i32> @smulh_i32(<vscale x 4 x i32> %a,
+                                     <vscale x 4 x i32> %b) {
+; CHECK-LABEL: smulh_i32
+; CHECK: smulh z0.s, z0.s, z1.s
+; CHECK-NEXT: ret
+  %sel = call <vscale x 4 x i1> @llvm.aarch64.sve.ptrue.nxv4i1(i32 31)
+  %res = call <vscale x 4 x i32> @llvm.aarch64.sve.smulh.nxv4i32(<vscale x 4 x i1> %sel, <vscale x 4 x i32> %a,
+                                                                 <vscale x 4 x i32> %b)
+  ret <vscale x 4 x i32> %res
+}
+
+define <vscale x 2 x i64> @smulh_i64(<vscale x 2 x i64> %a,
+                                     <vscale x 2 x i64> %b) {
+; CHECK-LABEL: smulh_i64
+; CHECK: smulh z0.d, z0.d, z1.d
+; CHECK-NEXT: ret
+  %sel = call <vscale x 2 x i1> @llvm.aarch64.sve.ptrue.nxv2i1(i32 31)
+  %res = call <vscale x 2 x i64> @llvm.aarch64.sve.smulh.nxv2i64(<vscale x 2 x i1> %sel, <vscale x 2 x i64> %a,
+                                                                 <vscale x 2 x i64> %b)
+  ret <vscale x 2 x i64> %res
+}
+
+;
+; UMULH (vector, unpredicated)
+;
+define <vscale x 16 x i8> @umulh_i8(<vscale x 16 x i8> %a,
+                                    <vscale x 16 x i8> %b) {
+; CHECK-LABEL: umulh_i8
+; CHECK: umulh z0.b, z0.b, z1.b
+; CHECK-NEXT: ret
+  %sel = call <vscale x 16 x i1> @llvm.aarch64.sve.ptrue.nxv16i1(i32 31)
+  %res = call <vscale x 16 x i8> @llvm.aarch64.sve.umulh.nxv16i8(<vscale x 16 x i1> %sel, <vscale x 16 x i8> %a,
+                                                                 <vscale x 16 x i8> %b)
+  ret <vscale x 16 x i8> %res
+}
+
+define <vscale x 8 x i16> @umulh_i16(<vscale x 8 x i16> %a,
+                                     <vscale x 8 x i16> %b) {
+; CHECK-LABEL: umulh_i16
+; CHECK: umulh z0.h, z0.h, z1.h
+; CHECK-NEXT: ret
+  %sel = call <vscale x 8 x i1> @llvm.aarch64.sve.ptrue.nxv8i1(i32 31)
+  %res = call <vscale x 8 x i16> @llvm.aarch64.sve.umulh.nxv8i16(<vscale x 8 x i1> %sel, <vscale x 8 x i16> %a,
+                                                                 <vscale x 8 x i16> %b)
+  ret <vscale x 8 x i16> %res
+}
+
+define <vscale x 4 x i32> @umulh_i32(<vscale x 4 x i32> %a,
+                                     <vscale x 4 x i32> %b) {
+; CHECK-LABEL: umulh_i32
+; CHECK: umulh z0.s, z0.s, z1.s
+; CHECK-NEXT: ret
+  %sel = call <vscale x 4 x i1> @llvm.aarch64.sve.ptrue.nxv4i1(i32 31)
+  %res = call <vscale x 4 x i32> @llvm.aarch64.sve.umulh.nxv4i32(<vscale x 4 x i1> %sel, <vscale x 4 x i32> %a,
+                                                                 <vscale x 4 x i32> %b)
+  ret <vscale x 4 x i32> %res
+}
+
+define <vscale x 2 x i64> @umulh_i64(<vscale x 2 x i64> %a,
+                                     <vscale x 2 x i64> %b) {
+; CHECK-LABEL: umulh_i64
+; CHECK: umulh z0.d, z0.d, z1.d
+; CHECK-NEXT: ret
+  %sel = call <vscale x 2 x i1> @llvm.aarch64.sve.ptrue.nxv2i1(i32 31)
+  %res = call <vscale x 2 x i64> @llvm.aarch64.sve.umulh.nxv2i64(<vscale x 2 x i1> %sel, <vscale x 2 x i64> %a,
+                                                                 <vscale x 2 x i64> %b)
+  ret <vscale x 2 x i64> %res
+}
+
+;
+; PMUL (vector, unpredicated)
+;
+define <vscale x 16 x i8> @pmul_i8(<vscale x 16 x i8> %a,
+                                   <vscale x 16 x i8> %b) {
+; CHECK-LABEL: pmul_i8
+; CHECK: pmul z0.b, z0.b, z1.b
+; CHECK-NEXT: ret
+  %res = call <vscale x 16 x i8> @llvm.aarch64.sve.pmul.nxv16i8(<vscale x 16 x i8> %a,
+                                                                <vscale x 16 x i8> %b)
+  ret <vscale x 16 x i8> %res
+}
+
+;
+; SQDMULH (vector, unpredicated)
+;
+define <vscale x 16 x i8> @sqdmulh_i8(<vscale x 16 x i8> %a,
+                                      <vscale x 16 x i8> %b) {
+; CHECK-LABEL: sqdmulh_i8
+; CHECK: sqdmulh z0.b, z0.b, z1.b
+; CHECK-NEXT: ret
+  %res = call <vscale x 16 x i8> @llvm.aarch64.sve.sqdmulh.nxv16i8(<vscale x 16 x i8> %a,
+                                                                   <vscale x 16 x i8> %b)
+  ret <vscale x 16 x i8> %res
+}
+
+define <vscale x 8 x i16> @sqdmulh_i16(<vscale x 8 x i16> %a,
+                                       <vscale x 8 x i16> %b) {
+; CHECK-LABEL: sqdmulh_i16
+; CHECK: sqdmulh z0.h, z0.h, z1.h
+; CHECK-NEXT: ret
+  %res = call <vscale x 8 x i16> @llvm.aarch64.sve.sqdmulh.nxv8i16(<vscale x 8 x i16> %a,
+                                                                   <vscale x 8 x i16> %b)
+  ret <vscale x 8 x i16> %res
+}
+
+define <vscale x 4 x i32> @sqdmulh_i32(<vscale x 4 x i32> %a,
+                                       <vscale x 4 x i32> %b) {
+; CHECK-LABEL: sqdmulh_i32
+; CHECK: sqdmulh z0.s, z0.s, z1.s
+; CHECK-NEXT: ret
+  %res = call <vscale x 4 x i32> @llvm.aarch64.sve.sqdmulh.nxv4i32(<vscale x 4 x i32> %a,
+                                                                   <vscale x 4 x i32> %b)
+  ret <vscale x 4 x i32> %res
+}
+
+define <vscale x 2 x i64> @sqdmulh_i64(<vscale x 2 x i64> %a,
+                                       <vscale x 2 x i64> %b) {
+; CHECK-LABEL: sqdmulh_i64
+; CHECK: sqdmulh z0.d, z0.d, z1.d
+; CHECK-NEXT: ret
+  %res = call <vscale x 2 x i64> @llvm.aarch64.sve.sqdmulh.nxv2i64(<vscale x 2 x i64> %a,
+                                                                   <vscale x 2 x i64> %b)
+  ret <vscale x 2 x i64> %res
+}
+
+;
+; SQRDMULH (vector, unpredicated)
+;
+define <vscale x 16 x i8> @sqrdmulh_i8(<vscale x 16 x i8> %a,
+                                       <vscale x 16 x i8> %b) {
+; CHECK-LABEL: sqrdmulh_i8
+; CHECK: sqrdmulh z0.b, z0.b, z1.b
+; CHECK-NEXT: ret
+  %res = call <vscale x 16 x i8> @llvm.aarch64.sve.sqrdmulh.nxv16i8(<vscale x 16 x i8> %a,
+                                                                    <vscale x 16 x i8> %b)
+  ret <vscale x 16 x i8> %res
+}
+
+define <vscale x 8 x i16> @sqrdmulh_i16(<vscale x 8 x i16> %a,
+                                        <vscale x 8 x i16> %b) {
+; CHECK-LABEL: sqrdmulh_i16
+; CHECK: sqrdmulh z0.h, z0.h, z1.h
+; CHECK-NEXT: ret
+  %res = call <vscale x 8 x i16> @llvm.aarch64.sve.sqrdmulh.nxv8i16(<vscale x 8 x i16> %a,
+                                                                    <vscale x 8 x i16> %b)
+  ret <vscale x 8 x i16> %res
+}
+
+define <vscale x 4 x i32> @sqrdmulh_i32(<vscale x 4 x i32> %a,
+                                        <vscale x 4 x i32> %b) {
+; CHECK-LABEL: sqrdmulh_i32
+; CHECK: sqrdmulh z0.s, z0.s, z1.s
+; CHECK-NEXT: ret
+  %res = call <vscale x 4 x i32> @llvm.aarch64.sve.sqrdmulh.nxv4i32(<vscale x 4 x i32> %a,
+                                                                    <vscale x 4 x i32> %b)
+  ret <vscale x 4 x i32> %res
+}
+
+define <vscale x 2 x i64> @sqrdmulh_i64(<vscale x 2 x i64> %a,
+                                        <vscale x 2 x i64> %b) {
+; CHECK-LABEL: sqrdmulh_i64
+; CHECK: sqrdmulh z0.d, z0.d, z1.d
+; CHECK-NEXT: ret
+  %res = call <vscale x 2 x i64> @llvm.aarch64.sve.sqrdmulh.nxv2i64(<vscale x 2 x i64> %a,
+                                                                    <vscale x 2 x i64> %b)
+  ret <vscale x 2 x i64> %res
+}
+
+declare <vscale x 16 x i1> @llvm.aarch64.sve.ptrue.nxv16i1(i32)
+declare <vscale x 8 x i1> @llvm.aarch64.sve.ptrue.nxv8i1(i32)
+declare <vscale x 4 x i1> @llvm.aarch64.sve.ptrue.nxv4i1(i32)
+declare <vscale x 2 x i1> @llvm.aarch64.sve.ptrue.nxv2i1(i32)
+declare <vscale x 16 x  i8> @llvm.aarch64.sve.smulh.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x  i8>, <vscale x 16 x  i8>)
+declare <vscale x  8 x i16> @llvm.aarch64.sve.smulh.nxv8i16(<vscale x  8 x i1>, <vscale x  8 x i16>, <vscale x  8 x i16>)
+declare <vscale x  4 x i32> @llvm.aarch64.sve.smulh.nxv4i32(<vscale x  4 x i1>, <vscale x  4 x i32>, <vscale x  4 x i32>)
+declare <vscale x  2 x i64> @llvm.aarch64.sve.smulh.nxv2i64(<vscale x  2 x i1>, <vscale x  2 x i64>, <vscale x  2 x i64>)
+declare <vscale x 16 x  i8> @llvm.aarch64.sve.umulh.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x  i8>, <vscale x 16 x  i8>)
+declare <vscale x  8 x i16> @llvm.aarch64.sve.umulh.nxv8i16(<vscale x  8 x i1>, <vscale x  8 x i16>, <vscale x  8 x i16>)
+declare <vscale x  4 x i32> @llvm.aarch64.sve.umulh.nxv4i32(<vscale x  4 x i1>, <vscale x  4 x i32>, <vscale x  4 x i32>)
+declare <vscale x  2 x i64> @llvm.aarch64.sve.umulh.nxv2i64(<vscale x  2 x i1>, <vscale x  2 x i64>, <vscale x  2 x i64>)
+declare <vscale x 16 x i8> @llvm.aarch64.sve.pmul.nxv16i8(<vscale x 16 x i8>, <vscale x 16 x i8>)
+declare <vscale x 16 x i8> @llvm.aarch64.sve.sqdmulh.nxv16i8(<vscale x 16 x i8>, <vscale x 16 x i8>)
+declare <vscale x 8 x i16> @llvm.aarch64.sve.sqdmulh.nxv8i16(<vscale x 8 x i16>, <vscale x 8 x i16>)
+declare <vscale x 4 x i32> @llvm.aarch64.sve.sqdmulh.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>)
+declare <vscale x 2 x i64> @llvm.aarch64.sve.sqdmulh.nxv2i64(<vscale x 2 x i64>, <vscale x 2 x i64>)
+declare <vscale x 16 x i8> @llvm.aarch64.sve.sqrdmulh.nxv16i8(<vscale x 16 x i8>, <vscale x 16 x i8>)
+declare <vscale x 8 x i16> @llvm.aarch64.sve.sqrdmulh.nxv8i16(<vscale x 8 x i16>, <vscale x 8 x i16>)
+declare <vscale x 4 x i32> @llvm.aarch64.sve.sqrdmulh.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>)
+declare <vscale x 2 x i64> @llvm.aarch64.sve.sqrdmulh.nxv2i64(<vscale x 2 x i64>, <vscale x 2 x i64>)