[RISCV] Add patterns for vector widening integer multiply

[jacquesguan]

Summary

Add patterns for vector widening integer multiply instructions

[eopXD]

Hi JianJian,

I see 12.14 Integer Widening Multiply-Add not added yet, may I give it a try?

Best regards,

Yueh-Ting (eop) Chen

Jianjian Guan via Phabricator <[e-mail]> 於 2022年1月15日 下午2:30 寫道：

jacquesguan created this revision.
jacquesguan added reviewers: craig.topper, asb, luismarques, frasercrmck, HsiangKai, khchen, benshi001.
Herald added subscribers: VincentWu, luke957, achieveartificialintelligence, vkmr, evandro, apazos, sameer.abuasal, s.egerton, Jim, benna, psnobl, jocewei, PkmX, the_o, brucehoult, MartinMosbeck, rogfer01, edward-jones, zzheng, jrtc27, kito-cheng, niosHD, sabuasal, simoncook, johnrusso, rbar, hiraditya.
jacquesguan requested review of this revision.
Herald added subscribers: llvm-commits, MaskRay.
Herald added a project: LLVM.

Add patterns for vector widening integer multiply instructions

Repository:
rG LLVM Github Monorepo

https://reviews.llvm.org/D117385

Files:
llvm/lib/Target/RISCV/RISCVInstrInfoVSDPatterns.td
llvm/test/CodeGen/RISCV/rvv/vwmul-sdnode.ll

<D117385.400241.patch>

[jacquesguan]

In D117385#3245748, @eopXD wrote:

Hi JianJian,

I see 12.14 Integer Widening Multiply-Add not added yet, may I give it a try?

Best regards,

Yueh-Ting (eop) Chen

Jianjian Guan via Phabricator <[e-mail]> 於 2022年1月15日 下午2:30 寫道：

jacquesguan created this revision.
jacquesguan added reviewers: craig.topper, asb, luismarques, frasercrmck, HsiangKai, khchen, benshi001.
Herald added subscribers: VincentWu, luke957, achieveartificialintelligence, vkmr, evandro, apazos, sameer.abuasal, s.egerton, Jim, benna, psnobl, jocewei, PkmX, the_o, brucehoult, MartinMosbeck, rogfer01, edward-jones, zzheng, jrtc27, kito-cheng, niosHD, sabuasal, simoncook, johnrusso, rbar, hiraditya.
jacquesguan requested review of this revision.
Herald added subscribers: llvm-commits, MaskRay.
Herald added a project: LLVM.

Add patterns for vector widening integer multiply instructions

Repository:
rG LLVM Github Monorepo

https://reviews.llvm.org/D117385

Files:
llvm/lib/Target/RISCV/RISCVInstrInfoVSDPatterns.td
llvm/test/CodeGen/RISCV/rvv/vwmul-sdnode.ll

<D117385.400241.patch>

Of course, you can, thanks.

[craig.topper]

What about vwmulsu?

I am curious what we should do for something like (nxvXi32 (mul (sext (nxvXi8 X)), (sext (nxvXi8 Y)). Should we leave the sexts alone or should be shrink them to (nxvXi16 (sext (nxvXi8 X)) and use a widening multiply to do the rest of the extend?

[jacquesguan]

Add patterns for VWMULSU

[jacquesguan]

In D117385#3246290, @craig.topper wrote:

What about vwmulsu?

I am curious what we should do for something like (nxvXi32 (mul (sext (nxvXi8 X)), (sext (nxvXi8 Y)). Should we leave the sexts alone or should be shrink them to (nxvXi16 (sext (nxvXi8 X)) and use a widening multiply to do the rest of the extend?

I think that maybe (nxvXi32 (mul (sext (nxvXi8 X)), (sext (nxvXi8 Y)) -> (nxvXi32 sext (mul (nxvXi16 (sext (nxvXi8 X))), (nxvXi16 (sext (nxvXi8 Y))))) is better? In this way, we could use less width vector to calaculate, it could be faster?

[craig.topper]

In D117385#3247401, @jacquesguan wrote:

In D117385#3246290, @craig.topper wrote:

What about vwmulsu?

I am curious what we should do for something like (nxvXi32 (mul (sext (nxvXi8 X)), (sext (nxvXi8 Y)). Should we leave the sexts alone or should be shrink them to (nxvXi16 (sext (nxvXi8 X)) and use a widening multiply to do the rest of the extend?

I think that maybe (nxvXi32 (mul (sext (nxvXi8 X)), (sext (nxvXi8 Y)) -> (nxvXi32 sext (mul (nxvXi16 (sext (nxvXi8 X))), (nxvXi16 (sext (nxvXi8 Y))))) is better? In this way, we could use less width vector to calaculate, it could be faster?

That's a good point. I guess my question better applies to the case where one type is 16 bits, the other is 8 bits and the result is 32 bits.

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfoVSDPatterns.td

651

I'm a bit skeptical that we would see an anyext before a multiply, but if we're going to handle anyext, then we should probably handle one operand being a zext and the other being an anyext.

[jacquesguan]

In D117385#3249194, @craig.topper wrote:

In D117385#3247401, @jacquesguan wrote:

In D117385#3246290, @craig.topper wrote:

What about vwmulsu?

I am curious what we should do for something like (nxvXi32 (mul (sext (nxvXi8 X)), (sext (nxvXi8 Y)). Should we leave the sexts alone or should be shrink them to (nxvXi16 (sext (nxvXi8 X)) and use a widening multiply to do the rest of the extend?

I think that maybe (nxvXi32 (mul (sext (nxvXi8 X)), (sext (nxvXi8 Y)) -> (nxvXi32 sext (mul (nxvXi16 (sext (nxvXi8 X))), (nxvXi16 (sext (nxvXi8 Y))))) is better? In this way, we could use less width vector to calaculate, it could be faster?

That's a good point. I guess my question better applies to the case where one type is 16 bits, the other is 8 bits and the result is 32 bits.

For this case, I think maybe extend the 8-bits vector to 16-bits vector firstly, and then use the widen instruction to get the 32-bits result.

[jacquesguan]

Address comment.

llvm/lib/Target/RISCV/RISCVInstrInfoVSDPatterns.td

651

Done.

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfoVSDPatterns.td

657

You don't need to handle both orders or zext/anyext. Tablegen knows mul is commutable and will autogenerate the swapped version. Sorry I suspected you might not know that and should have mentioned it.

[jacquesguan]

Address comment

llvm/lib/Target/RISCV/RISCVInstrInfoVSDPatterns.td

657

Done, thanks a lot.

[jacquesguan]

ping.

[jacquesguan]

ping.

[jacquesguan]

ping.

[jacquesguan]

ping.

[jacquesguan]

ping

[craig.topper]

LGTM

llvm/lib/Target/RISCV/RISCVInstrInfoVSDPatterns.td

foreach vtiToWti = AllWidenableIntToFloatVectors in {
     defvar vti = vtiToWti.Vti;
     defvar fwti = vtiToWti.Wti;
     def : Pat<(vti.Vector (vop (fwti.Vector fwti.RegClass:$rs1))),
               (!cast<Instruction>(instruction_name#"_"#vti.LMul.MX)
                   fwti.RegClass:$rs1, vti.AVL, vti.Log2SEW)>;
   }
 }
 
-multiclass VPatWidenBinarySDNode_VV_VX_WV_WX<SDNode op, PatFrags extop, string instruction_name> {
+multiclass VPatWidenBinarySDNode_VV_VX<SDNode op, PatFrags extop1, PatFrags extop2,
+                                       string instruction_name> {
   foreach vti = AllWidenableIntVectors in {
-    def : Pat<(op (vti.Wti.Vector (extop (vti.Vti.Vector vti.Vti.RegClass:$rs2))),
-                  (vti.Wti.Vector (extop (vti.Vti.Vector vti.Vti.RegClass:$rs1)))),
+    def : Pat<(op (vti.Wti.Vector (extop1 (vti.Vti.Vector vti.Vti.RegClass:$rs2))),
+                  (vti.Wti.Vector (extop2 (vti.Vti.Vector vti.Vti.RegClass:$rs1)))),
               (!cast<Instruction>(instruction_name#"_VV_"#vti.Vti.LMul.MX)
                  vti.Vti.RegClass:$rs2, vti.Vti.RegClass:$rs1,
                  vti.Vti.AVL, vti.Vti.Log2SEW)>;
-    def : Pat<(op (vti.Wti.Vector (extop (vti.Vti.Vector vti.Vti.RegClass:$rs2))),
-                  (vti.Wti.Vector (extop (vti.Vti.Vector (SplatPat GPR:$rs1))))),
+    def : Pat<(op (vti.Wti.Vector (extop1 (vti.Vti.Vector vti.Vti.RegClass:$rs2))),
+                  (vti.Wti.Vector (extop2 (vti.Vti.Vector (SplatPat GPR:$rs1))))),
               (!cast<Instruction>(instruction_name#"_VX_"#vti.Vti.LMul.MX)
                  vti.Vti.RegClass:$rs2, GPR:$rs1,
                  vti.Vti.AVL, vti.Vti.Log2SEW)>;
+  }
+}
+
+multiclass VPatWidenBinarySDNode_WV_WX<SDNode op, PatFrags extop,
+                                       string instruction_name> {
+  foreach vti = AllWidenableIntVectors in {
     def : Pat<(op (vti.Wti.Vector vti.Wti.RegClass:$rs2),
                   (vti.Wti.Vector (extop (vti.Vti.Vector vti.Vti.RegClass:$rs1)))),
               (!cast<Instruction>(instruction_name#"_WV_"#vti.Vti.LMul.MX)
                  vti.Wti.RegClass:$rs2, vti.Vti.RegClass:$rs1,
                  vti.Vti.AVL, vti.Vti.Log2SEW)>;
     def : Pat<(op (vti.Wti.Vector vti.Wti.RegClass:$rs2),
                   (vti.Wti.Vector (extop (vti.Vti.Vector (SplatPat GPR:$rs1))))),
               (!cast<Instruction>(instruction_name#"_WX_"#vti.Vti.LMul.MX)
                  vti.Wti.RegClass:$rs2, GPR:$rs1,
                  vti.Vti.AVL, vti.Vti.Log2SEW)>;
   }
 }
 
+multiclass VPatWidenBinarySDNode_VV_VX_WV_WX<SDNode op, PatFrags extop,
+                                             string instruction_name> {
+  defm : VPatWidenBinarySDNode_VV_VX<op, extop, extop, instruction_name>;
+  defm : VPatWidenBinarySDNode_WV_WX<op, extop, instruction_name>;
+}
+
 multiclass VPatWidenMulAddSDNode_VV<PatFrags extop1, PatFrags extop2, string instruction_name> {
   foreach vti = AllWidenableIntVectors in {
     def : Pat<
       (add (vti.Wti.Vector vti.Wti.RegClass:$rd),
         (mul_oneuse (vti.Wti.Vector (extop1 (vti.Vti.Vector vti.Vti.RegClass:$rs1))),
                     (vti.Wti.Vector (extop2 (vti.Vti.Vector vti.Vti.RegClass:$rs2))))),
       (!cast<Instruction>(instruction_name#"_VV_"#vti.Vti.LMul.MX)
         vti.Wti.RegClass:$rd, vti.Vti.RegClass:$rs1, vti.Vti.RegClass:$rs2,

 defm : VPatBinarySDNode_VV_VX<mulhu, "PseudoVMULHU">;
 
 // 12.11. Vector Integer Divide Instructions
 defm : VPatBinarySDNode_VV_VX<udiv, "PseudoVDIVU">;
 defm : VPatBinarySDNode_VV_VX<sdiv, "PseudoVDIV">;
 defm : VPatBinarySDNode_VV_VX<urem, "PseudoVREMU">;
 defm : VPatBinarySDNode_VV_VX<srem, "PseudoVREM">;
 
+// 12.12. Vector Widening Integer Multiply Instructions
+defm : VPatWidenBinarySDNode_VV_VX<mul, sext_oneuse, sext_oneuse,
+                                   "PseudoVWMUL">;
+defm : VPatWidenBinarySDNode_VV_VX<mul, zext_oneuse, zext_oneuse,
+                                   "PseudoVWMULU">;
+defm : VPatWidenBinarySDNode_VV_VX<mul, anyext_oneuse, anyext_oneuse,
+                                   "PseudoVWMULU">;
+defm : VPatWidenBinarySDNode_VV_VX<mul, zext_oneuse, anyext_oneuse,
+                                   "PseudoVWMULU">;
+defm : VPatWidenBinarySDNode_VV_VX<mul, sext_oneuse, zext_oneuse,
+                                   "PseudoVWMULSU">;
+defm : VPatWidenBinarySDNode_VV_VX<mul, sext_oneuse, anyext_oneuse,
+                                   "PseudoVWMULSU">;
+
 // 12.13 Vector Single-Width Integer Multiply-Add Instructions.
 foreach vti = AllIntegerVectors in {
   // NOTE: We choose VMADD because it has the most commuting freedom. So it
   // works best with how TwoAddressInstructionPass tries commuting.
   defvar suffix = vti.LMul.MX;
   def : Pat<(vti.Vector (add vti.RegClass:$rs2,
                               (mul_oneuse vti.RegClass:$rs1, vti.RegClass:$rd))),
             (!cast<Instruction>("PseudoVMADD_VV_"# suffix)

llvm/test/CodeGen/RISCV/rvv/vwmul-sdnode.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+v -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64 -mattr=+v -verify-machineinstrs < %s | FileCheck %s
+
+define <vscale x 1 x i64> @vwmul_vv_nxv1i64(<vscale x 1 x i32> %va, <vscale x 1 x i32> %vb) {
+; CHECK-LABEL: vwmul_vv_nxv1i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; CHECK-NEXT:    vwmul.vv v10, v8, v9
+; CHECK-NEXT:    vmv1r.v v8, v10
+; CHECK-NEXT:    ret
+  %vc = sext <vscale x 1 x i32> %va to <vscale x 1 x i64>
+  %vd = sext <vscale x 1 x i32> %vb to <vscale x 1 x i64>
+  %ve = mul <vscale x 1 x i64> %vc, %vd
+  ret <vscale x 1 x i64> %ve
+}
+
+define <vscale x 1 x i64> @vwmulu_vv_nxv1i64(<vscale x 1 x i32> %va, <vscale x 1 x i32> %vb) {
+; CHECK-LABEL: vwmulu_vv_nxv1i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; CHECK-NEXT:    vwmulu.vv v10, v8, v9
+; CHECK-NEXT:    vmv1r.v v8, v10
+; CHECK-NEXT:    ret
+  %vc = zext <vscale x 1 x i32> %va to <vscale x 1 x i64>
+  %vd = zext <vscale x 1 x i32> %vb to <vscale x 1 x i64>
+  %ve = mul <vscale x 1 x i64> %vc, %vd
+  ret <vscale x 1 x i64> %ve
+}
+
+define <vscale x 1 x i64> @vwmulsu_vv_nxv1i64(<vscale x 1 x i32> %va, <vscale x 1 x i32> %vb) {
+; CHECK-LABEL: vwmulsu_vv_nxv1i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; CHECK-NEXT:    vwmulsu.vv v10, v8, v9
+; CHECK-NEXT:    vmv1r.v v8, v10
+; CHECK-NEXT:    ret
+  %vc = sext <vscale x 1 x i32> %va to <vscale x 1 x i64>
+  %vd = zext <vscale x 1 x i32> %vb to <vscale x 1 x i64>
+  %ve = mul <vscale x 1 x i64> %vc, %vd
+  ret <vscale x 1 x i64> %ve
+}
+
+define <vscale x 1 x i64> @vwmul_vx_nxv1i64(<vscale x 1 x i32> %va, i32 %b) {
+; CHECK-LABEL: vwmul_vx_nxv1i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a1, zero, e32, mf2, ta, mu
+; CHECK-NEXT:    vwmul.vx v9, v8, a0
+; CHECK-NEXT:    vmv1r.v v8, v9
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x i32> undef, i32 %b, i32 0
+  %splat = shufflevector <vscale x 1 x i32> %head, <vscale x 1 x i32> undef, <vscale x 1 x i32> zeroinitializer
+  %vc = sext <vscale x 1 x i32> %va to <vscale x 1 x i64>
+  %vd = sext <vscale x 1 x i32> %splat to <vscale x 1 x i64>
+  %ve = mul <vscale x 1 x i64> %vc, %vd
+  ret <vscale x 1 x i64> %ve
+}
+
+define <vscale x 1 x i64> @vwmulu_vx_nxv1i64(<vscale x 1 x i32> %va, i32 %b) {
+; CHECK-LABEL: vwmulu_vx_nxv1i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a1, zero, e32, mf2, ta, mu
+; CHECK-NEXT:    vwmulu.vx v9, v8, a0
+; CHECK-NEXT:    vmv1r.v v8, v9
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x i32> undef, i32 %b, i32 0
+  %splat = shufflevector <vscale x 1 x i32> %head, <vscale x 1 x i32> undef, <vscale x 1 x i32> zeroinitializer
+  %vc = zext <vscale x 1 x i32> %va to <vscale x 1 x i64>
+  %vd = zext <vscale x 1 x i32> %splat to <vscale x 1 x i64>
+  %ve = mul <vscale x 1 x i64> %vc, %vd
+  ret <vscale x 1 x i64> %ve
+}
+
+define <vscale x 1 x i64> @vwmulsu_vx_nxv1i64(<vscale x 1 x i32> %va, i32 %b) {
+; CHECK-LABEL: vwmulsu_vx_nxv1i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a1, zero, e32, mf2, ta, mu
+; CHECK-NEXT:    vwmulsu.vx v9, v8, a0
+; CHECK-NEXT:    vmv1r.v v8, v9
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 1 x i32> undef, i32 %b, i32 0
+  %splat = shufflevector <vscale x 1 x i32> %head, <vscale x 1 x i32> undef, <vscale x 1 x i32> zeroinitializer
+  %vc = sext <vscale x 1 x i32> %va to <vscale x 1 x i64>
+  %vd = zext <vscale x 1 x i32> %splat to <vscale x 1 x i64>
+  %ve = mul <vscale x 1 x i64> %vc, %vd
+  ret <vscale x 1 x i64> %ve
+}
+
+define <vscale x 2 x i64> @vwmul_vv_nxv2i64(<vscale x 2 x i32> %va, <vscale x 2 x i32> %vb) {
+; CHECK-LABEL: vwmul_vv_nxv2i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; CHECK-NEXT:    vwmul.vv v10, v8, v9
+; CHECK-NEXT:    vmv2r.v v8, v10
+; CHECK-NEXT:    ret
+  %vc = sext <vscale x 2 x i32> %va to <vscale x 2 x i64>
+  %vd = sext <vscale x 2 x i32> %vb to <vscale x 2 x i64>
+  %ve = mul <vscale x 2 x i64> %vc, %vd
+  ret <vscale x 2 x i64> %ve
+}
+
+define <vscale x 2 x i64> @vwmulu_vv_nxv2i64(<vscale x 2 x i32> %va, <vscale x 2 x i32> %vb) {
+; CHECK-LABEL: vwmulu_vv_nxv2i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; CHECK-NEXT:    vwmulu.vv v10, v8, v9
+; CHECK-NEXT:    vmv2r.v v8, v10
+; CHECK-NEXT:    ret
+  %vc = zext <vscale x 2 x i32> %va to <vscale x 2 x i64>
+  %vd = zext <vscale x 2 x i32> %vb to <vscale x 2 x i64>
+  %ve = mul <vscale x 2 x i64> %vc, %vd
+  ret <vscale x 2 x i64> %ve
+}
+
+define <vscale x 2 x i64> @vwmulsu_vv_nxv2i64(<vscale x 2 x i32> %va, <vscale x 2 x i32> %vb) {
+; CHECK-LABEL: vwmulsu_vv_nxv2i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; CHECK-NEXT:    vwmulsu.vv v10, v8, v9
+; CHECK-NEXT:    vmv2r.v v8, v10
+; CHECK-NEXT:    ret
+  %vc = sext <vscale x 2 x i32> %va to <vscale x 2 x i64>
+  %vd = zext <vscale x 2 x i32> %vb to <vscale x 2 x i64>
+  %ve = mul <vscale x 2 x i64> %vc, %vd
+  ret <vscale x 2 x i64> %ve
+}
+
+define <vscale x 2 x i64> @vwmul_vx_nxv2i64(<vscale x 2 x i32> %va, i32 %b) {
+; CHECK-LABEL: vwmul_vx_nxv2i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a1, zero, e32, m1, ta, mu
+; CHECK-NEXT:    vwmul.vx v10, v8, a0
+; CHECK-NEXT:    vmv2r.v v8, v10
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x i32> undef, i32 %b, i32 0
+  %splat = shufflevector <vscale x 2 x i32> %head, <vscale x 2 x i32> undef, <vscale x 2 x i32> zeroinitializer
+  %vc = sext <vscale x 2 x i32> %va to <vscale x 2 x i64>
+  %vd = sext <vscale x 2 x i32> %splat to <vscale x 2 x i64>
+  %ve = mul <vscale x 2 x i64> %vc, %vd
+  ret <vscale x 2 x i64> %ve
+}
+
+define <vscale x 2 x i64> @vwmulu_vx_nxv2i64(<vscale x 2 x i32> %va, i32 %b) {
+; CHECK-LABEL: vwmulu_vx_nxv2i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a1, zero, e32, m1, ta, mu
+; CHECK-NEXT:    vwmulu.vx v10, v8, a0
+; CHECK-NEXT:    vmv2r.v v8, v10
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x i32> undef, i32 %b, i32 0
+  %splat = shufflevector <vscale x 2 x i32> %head, <vscale x 2 x i32> undef, <vscale x 2 x i32> zeroinitializer
+  %vc = zext <vscale x 2 x i32> %va to <vscale x 2 x i64>
+  %vd = zext <vscale x 2 x i32> %splat to <vscale x 2 x i64>
+  %ve = mul <vscale x 2 x i64> %vc, %vd
+  ret <vscale x 2 x i64> %ve
+}
+
+define <vscale x 2 x i64> @vwmulsu_vx_nxv2i64(<vscale x 2 x i32> %va, i32 %b) {
+; CHECK-LABEL: vwmulsu_vx_nxv2i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a1, zero, e32, m1, ta, mu
+; CHECK-NEXT:    vwmulsu.vx v10, v8, a0
+; CHECK-NEXT:    vmv2r.v v8, v10
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 2 x i32> undef, i32 %b, i32 0
+  %splat = shufflevector <vscale x 2 x i32> %head, <vscale x 2 x i32> undef, <vscale x 2 x i32> zeroinitializer
+  %vc = sext <vscale x 2 x i32> %va to <vscale x 2 x i64>
+  %vd = zext <vscale x 2 x i32> %splat to <vscale x 2 x i64>
+  %ve = mul <vscale x 2 x i64> %vc, %vd
+  ret <vscale x 2 x i64> %ve
+}
+
+define <vscale x 4 x i64> @vwmul_vv_nxv4i64(<vscale x 4 x i32> %va, <vscale x 4 x i32> %vb) {
+; CHECK-LABEL: vwmul_vv_nxv4i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; CHECK-NEXT:    vwmul.vv v12, v8, v10
+; CHECK-NEXT:    vmv4r.v v8, v12
+; CHECK-NEXT:    ret
+  %vc = sext <vscale x 4 x i32> %va to <vscale x 4 x i64>
+  %vd = sext <vscale x 4 x i32> %vb to <vscale x 4 x i64>
+  %ve = mul <vscale x 4 x i64> %vc, %vd
+  ret <vscale x 4 x i64> %ve
+}
+
+define <vscale x 4 x i64> @vwmulu_vv_nxv4i64(<vscale x 4 x i32> %va, <vscale x 4 x i32> %vb) {
+; CHECK-LABEL: vwmulu_vv_nxv4i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; CHECK-NEXT:    vwmulu.vv v12, v8, v10
+; CHECK-NEXT:    vmv4r.v v8, v12
+; CHECK-NEXT:    ret
+  %vc = zext <vscale x 4 x i32> %va to <vscale x 4 x i64>
+  %vd = zext <vscale x 4 x i32> %vb to <vscale x 4 x i64>
+  %ve = mul <vscale x 4 x i64> %vc, %vd
+  ret <vscale x 4 x i64> %ve
+}
+
+define <vscale x 4 x i64> @vwmulsu_vv_nxv4i64(<vscale x 4 x i32> %va, <vscale x 4 x i32> %vb) {
+; CHECK-LABEL: vwmulsu_vv_nxv4i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; CHECK-NEXT:    vwmulsu.vv v12, v8, v10
+; CHECK-NEXT:    vmv4r.v v8, v12
+; CHECK-NEXT:    ret
+  %vc = sext <vscale x 4 x i32> %va to <vscale x 4 x i64>
+  %vd = zext <vscale x 4 x i32> %vb to <vscale x 4 x i64>
+  %ve = mul <vscale x 4 x i64> %vc, %vd
+  ret <vscale x 4 x i64> %ve
+}
+
+define <vscale x 4 x i64> @vwmul_vx_nxv4i64(<vscale x 4 x i32> %va, i32 %b) {
+; CHECK-LABEL: vwmul_vx_nxv4i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a1, zero, e32, m2, ta, mu
+; CHECK-NEXT:    vwmul.vx v12, v8, a0
+; CHECK-NEXT:    vmv4r.v v8, v12
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x i32> undef, i32 %b, i32 0
+  %splat = shufflevector <vscale x 4 x i32> %head, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  %vc = sext <vscale x 4 x i32> %va to <vscale x 4 x i64>
+  %vd = sext <vscale x 4 x i32> %splat to <vscale x 4 x i64>
+  %ve = mul <vscale x 4 x i64> %vc, %vd
+  ret <vscale x 4 x i64> %ve
+}
+
+define <vscale x 4 x i64> @vwmulu_vx_nxv4i64(<vscale x 4 x i32> %va, i32 %b) {
+; CHECK-LABEL: vwmulu_vx_nxv4i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a1, zero, e32, m2, ta, mu
+; CHECK-NEXT:    vwmulu.vx v12, v8, a0
+; CHECK-NEXT:    vmv4r.v v8, v12
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x i32> undef, i32 %b, i32 0
+  %splat = shufflevector <vscale x 4 x i32> %head, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  %vc = zext <vscale x 4 x i32> %va to <vscale x 4 x i64>
+  %vd = zext <vscale x 4 x i32> %splat to <vscale x 4 x i64>
+  %ve = mul <vscale x 4 x i64> %vc, %vd
+  ret <vscale x 4 x i64> %ve
+}
+
+define <vscale x 4 x i64> @vwmulsu_vx_nxv4i64(<vscale x 4 x i32> %va, i32 %b) {
+; CHECK-LABEL: vwmulsu_vx_nxv4i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a1, zero, e32, m2, ta, mu
+; CHECK-NEXT:    vwmulsu.vx v12, v8, a0
+; CHECK-NEXT:    vmv4r.v v8, v12
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 4 x i32> undef, i32 %b, i32 0
+  %splat = shufflevector <vscale x 4 x i32> %head, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  %vc = sext <vscale x 4 x i32> %va to <vscale x 4 x i64>
+  %vd = zext <vscale x 4 x i32> %splat to <vscale x 4 x i64>
+  %ve = mul <vscale x 4 x i64> %vc, %vd
+  ret <vscale x 4 x i64> %ve
+}
+
+define <vscale x 8 x i64> @vwmul_vv_nxv8i64(<vscale x 8 x i32> %va, <vscale x 8 x i32> %vb) {
+; CHECK-LABEL: vwmul_vv_nxv8i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; CHECK-NEXT:    vwmul.vv v16, v8, v12
+; CHECK-NEXT:    vmv8r.v v8, v16
+; CHECK-NEXT:    ret
+  %vc = sext <vscale x 8 x i32> %va to <vscale x 8 x i64>
+  %vd = sext <vscale x 8 x i32> %vb to <vscale x 8 x i64>
+  %ve = mul <vscale x 8 x i64> %vc, %vd
+  ret <vscale x 8 x i64> %ve
+}
+
+define <vscale x 8 x i64> @vwmulu_vv_nxv8i64(<vscale x 8 x i32> %va, <vscale x 8 x i32> %vb) {
+; CHECK-LABEL: vwmulu_vv_nxv8i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; CHECK-NEXT:    vwmulu.vv v16, v8, v12
+; CHECK-NEXT:    vmv8r.v v8, v16
+; CHECK-NEXT:    ret
+  %vc = zext <vscale x 8 x i32> %va to <vscale x 8 x i64>
+  %vd = zext <vscale x 8 x i32> %vb to <vscale x 8 x i64>
+  %ve = mul <vscale x 8 x i64> %vc, %vd
+  ret <vscale x 8 x i64> %ve
+}
+
+define <vscale x 8 x i64> @vwmulsu_vv_nxv8i64(<vscale x 8 x i32> %va, <vscale x 8 x i32> %vb) {
+; CHECK-LABEL: vwmulsu_vv_nxv8i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; CHECK-NEXT:    vwmulsu.vv v16, v8, v12
+; CHECK-NEXT:    vmv8r.v v8, v16
+; CHECK-NEXT:    ret
+  %vc = sext <vscale x 8 x i32> %va to <vscale x 8 x i64>
+  %vd = zext <vscale x 8 x i32> %vb to <vscale x 8 x i64>
+  %ve = mul <vscale x 8 x i64> %vc, %vd
+  ret <vscale x 8 x i64> %ve
+}
+
+define <vscale x 8 x i64> @vwmul_vx_nxv8i64(<vscale x 8 x i32> %va, i32 %b) {
+; CHECK-LABEL: vwmul_vx_nxv8i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a1, zero, e32, m4, ta, mu
+; CHECK-NEXT:    vwmul.vx v16, v8, a0
+; CHECK-NEXT:    vmv8r.v v8, v16
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x i32> undef, i32 %b, i32 0
+  %splat = shufflevector <vscale x 8 x i32> %head, <vscale x 8 x i32> undef, <vscale x 8 x i32> zeroinitializer
+  %vc = sext <vscale x 8 x i32> %va to <vscale x 8 x i64>
+  %vd = sext <vscale x 8 x i32> %splat to <vscale x 8 x i64>
+  %ve = mul <vscale x 8 x i64> %vc, %vd
+  ret <vscale x 8 x i64> %ve
+}
+
+define <vscale x 8 x i64> @vwmulu_vx_nxv8i64(<vscale x 8 x i32> %va, i32 %b) {
+; CHECK-LABEL: vwmulu_vx_nxv8i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a1, zero, e32, m4, ta, mu
+; CHECK-NEXT:    vwmulu.vx v16, v8, a0
+; CHECK-NEXT:    vmv8r.v v8, v16
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x i32> undef, i32 %b, i32 0
+  %splat = shufflevector <vscale x 8 x i32> %head, <vscale x 8 x i32> undef, <vscale x 8 x i32> zeroinitializer
+  %vc = zext <vscale x 8 x i32> %va to <vscale x 8 x i64>
+  %vd = zext <vscale x 8 x i32> %splat to <vscale x 8 x i64>
+  %ve = mul <vscale x 8 x i64> %vc, %vd
+  ret <vscale x 8 x i64> %ve
+}
+
+define <vscale x 8 x i64> @vwmulsu_vx_nxv8i64(<vscale x 8 x i32> %va, i32 %b) {
+; CHECK-LABEL: vwmulsu_vx_nxv8i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a1, zero, e32, m4, ta, mu
+; CHECK-NEXT:    vwmulsu.vx v16, v8, a0
+; CHECK-NEXT:    vmv8r.v v8, v16
+; CHECK-NEXT:    ret
+  %head = insertelement <vscale x 8 x i32> undef, i32 %b, i32 0
+  %splat = shufflevector <vscale x 8 x i32> %head, <vscale x 8 x i32> undef, <vscale x 8 x i32> zeroinitializer
+  %vc = sext <vscale x 8 x i32> %va to <vscale x 8 x i64>
+  %vd = zext <vscale x 8 x i32> %splat to <vscale x 8 x i64>
+  %ve = mul <vscale x 8 x i64> %vc, %vd
+  ret <vscale x 8 x i64> %ve
+}