This is an archive of the discontinued LLVM Phabricator instance.

Differential D139721

[RISCV][VP] Support vp.reduce.mul by ExpandVectorPredication
ClosedPublic

Authored by inclyc on Dec 9 2022, 8:59 AM.

Details

Reviewers
frasercrmck
craig.topper
reames
Commits
rG0359c19e8f14: [RISCV][VP] support vp.reduce.mul by ExpandVectorPredication
Summary

Most of VP intrinsics are implemented in RISC-V backends, but vp.reduce.mul (element length > 1) does not yet. Legalizes vp.reduce.mul using ExpandVectorPredication Pass.

Diff Detail

Event Timeline

inclyc created this revision.Dec 9 2022, 8:59 AM
Herald added a project: Restricted Project. · View Herald TranscriptDec 9 2022, 8:59 AM
Herald added subscribers: sunshaoce, VincentWu, vkmr and 28 others. · View Herald Transcript
inclyc added reviewers: frasercrmck, craig.topper.Dec 9 2022, 9:00 AM
Herald added a subscriber: StephenFan. · View Herald TranscriptDec 9 2022, 9:00 AM
inclyc published this revision for review.Dec 9 2022, 9:00 AM
Herald added a project: Restricted Project. · View Herald TranscriptDec 9 2022, 9:00 AM
Herald added subscribers: llvm-commits, • pcwang-thead, eopXD, MaskRay. · View Herald Transcript
Harbormaster completed remote builds in B202247: Diff 481667.Dec 9 2022, 9:40 AM
craig.topper added a comment.Dec 13 2022, 8:53 AM

Does this work for scalable vectors?

inclyc added a comment.Dec 13 2022, 10:48 AM
In D139721#3992238, @craig.topper wrote:

Does this work for scalable vectors?

I'm afraid not.... expand vp pass seems not able to expand reduction (causes an assertion failure).

IR:

declare i8 @llvm.vp.reduce.mul.nxv2i8(i8, <vscale x 2 x i8>, <vscale x 2 x i1>, i32)

define signext i8 @vpreduce_mul_nxv2i8(i8 signext %s, <vscale x 2 x i8> %v, <vscale x 2 x i1> %m, i32 zeroext %evl) {
  %r = call i8 @llvm.vp.reduce.mul.nxv2i8(i8 %s, <vscale x 2 x i8> %v, <vscale x 2 x i1> %m, i32 %evl)
  ret i8 %r
}

After expand vp:

define signext i8 @vpreduce_mul_nxv2i8(i8 signext %s, <vscale x 2 x i8> %v, <vscale x 2 x i1> %m, i32 zeroext %evl) {
  %1 = call <vscale x 2 x i1> @llvm.get.active.lane.mask.nxv2i1.i32(i32 0, i32 %evl)
  %2 = and <vscale x 2 x i1> %1, %m
  %vscale = call i32 @llvm.vscale.i32()
  %scalable_size = mul nuw i32 %vscale, 2
  %3 = select <vscale x 2 x i1> %2, <vscale x 2 x i8> %v, <vscale x 2 x i8> shufflevector (<vscale x 2 x i8> insertelement (<vscale x 2 x i8> poison, i8 1, i32 0), <vscale x 2 x i8> poison, <vscale x 2 x i32> zeroinitializer)
  %4 = call i8 @llvm.vector.reduce.mul.nxv2i8(<vscale x 2 x i8> %3)
  %5 = mul i8 %4, %s
  ret i8 %5
}

Other targets may not able to handle this as well. https://godbolt.org/z/qevxT5r8Y

Perhaps some of our next patches should be to improve the related passes (like Expand reduction intrinsics)?

Stack dump:

llc: <llvm-project>/llvm/include/llvm/Support/Casting.h:578: decltype(auto) llvm::cast(From *) [To = llvm::FixedVectorType, From = llvm::Type]: Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: build/bin/llc -mtriple=riscv32 local/buddy/reduce/mul/scale.ll
1.      Running pass 'Function Pass Manager' on module 'local/buddy/reduce/mul/scale.ll'.
2.      Running pass 'Expand reduction intrinsics' on function '@vpreduce_mul_nxv2i8'
 #0 0x00007f5ebd0c31da llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) <llvm-project>/llvm/lib/Support/Unix/Signals.inc:567:11
 #1 0x00007f5ebd0c338b PrintStackTraceSignalHandler(void*) <llvm-project>/llvm/lib/Support/Unix/Signals.inc:641:1
 #2 0x00007f5ebd0c1a16 llvm::sys::RunSignalHandlers() <llvm-project>/llvm/lib/Support/Signals.cpp:104:5
 #3 0x00007f5ebd0c3a75 SignalHandler(int) <llvm-project>/llvm/lib/Support/Unix/Signals.inc:412:1
 #4 0x00007f5ebc7c2950 (/usr/lib64/libc.so.6+0x37950)
 #5 0x00007f5ebc80e42c (/usr/lib64/libc.so.6+0x8342c)
 #6 0x00007f5ebc7c28b2 raise (/usr/lib64/libc.so.6+0x378b2)
 #7 0x00007f5ebc7ad471 abort (/usr/lib64/libc.so.6+0x22471)
 #8 0x00007f5ebc7ad395 (/usr/lib64/libc.so.6+0x22395)
 #9 0x00007f5ebc7bbb92 (/usr/lib64/libc.so.6+0x30b92)
#10 0x00007f5ec07c8c88 decltype(auto) llvm::cast<llvm::FixedVectorType, llvm::Type>(llvm::Type*) <llvm-project>/llvm/include/llvm/Support/Casting.h:579:10
#11 0x00007f5ec080c6d6 (anonymous namespace)::expandReductions(llvm::Function&, llvm::TargetTransformInfo const*) <llvm-project>/llvm/lib/CodeGen/ExpandReductions.cpp:147:15
#12 0x00007f5ec080c976 (anonymous namespace)::ExpandReductions::runOnFunction(llvm::Function&) <llvm-project>/llvm/lib/CodeGen/ExpandReductions.cpp:183:5
#13 0x00007f5ebdcb9643 llvm::FPPassManager::runOnFunction(llvm::Function&) <llvm-project>/llvm/lib/IR/LegacyPassManager.cpp:1430:23
#14 0x00007f5ebdcbe462 llvm::FPPassManager::runOnModule(llvm::Module&) <llvm-project>/llvm/lib/IR/LegacyPassManager.cpp:1476:16
#15 0x00007f5ebdcb9f19 (anonymous namespace)::MPPassManager::runOnModule(llvm::Module&) <llvm-project>/llvm/lib/IR/LegacyPassManager.cpp:1545:23
#16 0x00007f5ebdcb9a8a llvm::legacy::PassManagerImpl::run(llvm::Module&) <llvm-project>/llvm/lib/IR/LegacyPassManager.cpp:535:16
#17 0x00007f5ebdcbe741 llvm::legacy::PassManager::run(llvm::Module&) <llvm-project>/llvm/lib/IR/LegacyPassManager.cpp:1672:3
#18 0x000055f10326689a compileModule(char**, llvm::LLVMContext&) <llvm-project>/llvm/tools/llc/llc.cpp:737:41
#19 0x000055f103264ca2 main <llvm-project>/llvm/tools/llc/llc.cpp:418:13
#20 0x00007f5ebc7ae34a (/usr/lib64/libc.so.6+0x2334a)
#21 0x00007f5ebc7ae3fc __libc_start_main (/usr/lib64/libc.so.6+0x233fc)
#22 0x000055f1032644b1 _start (build/bin/llc+0x284b1)
zsh: IOT instruction (core dumped)  build/bin/llc -mtriple=riscv32 local/buddy/reduce/mul/scale.ll
craig.topper added inline comments.Dec 13 2022, 1:17 PM
llvm/test/CodeGen/RISCV/rvv/fixed-vectors-reduction-mask-vp.ll
1010 ↗(On Diff #481667)

This test file has "mask" in its name because it is the test file for i1 vectors. These tests should be be in fixed-vectors-reduction-vp.ll

inclyc updated this revision to Diff 482718.Dec 13 2022, 10:19 PM

Hoist tests to fixed-vectors-reduction-int-vp.ll.

Harbormaster completed remote builds in B203020: Diff 482718.Dec 13 2022, 11:40 PM
reames accepted this revision.Dec 16 2022, 12:51 PM
reames added a subscriber: reames.

LGTM

This revision is now accepted and ready to land.Dec 16 2022, 12:51 PM
This revision was landed with ongoing or failed builds.Dec 16 2022, 6:50 PM
Closed by commit rG0359c19e8f14: [RISCV][VP] support vp.reduce.mul by ExpandVectorPredication (authored by inclyc). · Explain Why
This revision was automatically updated to reflect the committed changes.
inclyc added a commit: rG0359c19e8f14: [RISCV][VP] support vp.reduce.mul by ExpandVectorPredication.

Revision Contents

PathSize
llvm/
lib/
Target/
RISCV/
5 lines
test/
CodeGen/
RISCV/
rvv/
462 lines

Diff 483706

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h

Show First 20 Lines • Show All 236 Lines • ▼ Show 20 Lines bool forceScalarizeMaskedScatter(VectorType *VTy, Align Alignment) {
return ST->is64Bit() && !ST->hasVInstructionsI64(); return ST->is64Bit() && !ST->hasVInstructionsI64();
} }
/// \returns How the target needs this vector-predicated operation to be /// \returns How the target needs this vector-predicated operation to be
/// transformed. /// transformed.
TargetTransformInfo::VPLegalization TargetTransformInfo::VPLegalization
getVPLegalizationStrategy(const VPIntrinsic &PI) const { getVPLegalizationStrategy(const VPIntrinsic &PI) const {
using VPLegalization = TargetTransformInfo::VPLegalization; using VPLegalization = TargetTransformInfo::VPLegalization;
if (PI.getIntrinsicID() == Intrinsic::vp_reduce_mul &&
cast<VectorType>(PI.getArgOperand(1)->getType())
->getElementType()
->getIntegerBitWidth() != 1)
return VPLegalization(VPLegalization::Discard, VPLegalization::Convert);
return VPLegalization(VPLegalization::Legal, VPLegalization::Legal); return VPLegalization(VPLegalization::Legal, VPLegalization::Legal);
} }
bool isLegalToVectorizeReduction(const RecurrenceDescriptor &RdxDesc, bool isLegalToVectorizeReduction(const RecurrenceDescriptor &RdxDesc,
ElementCount VF) const { ElementCount VF) const {
if (!VF.isScalable()) if (!VF.isScalable())
return true; return true;
▲ Show 20 Lines • Show All 84 Lines • Show Last 20 Lines

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-reduction-int-vp.ll

Show First 20 Lines • Show All 1,410 Lines • ▼ Show 20 Lines
; RV64-NEXT: vmv.s.x v10, a0 ; RV64-NEXT: vmv.s.x v10, a0
; RV64-NEXT: vsetvli zero, a1, e64, m2, tu, ma ; RV64-NEXT: vsetvli zero, a1, e64, m2, tu, ma
; RV64-NEXT: vredxor.vs v10, v8, v10, v0.t ; RV64-NEXT: vredxor.vs v10, v8, v10, v0.t
; RV64-NEXT: vmv.x.s a0, v10 ; RV64-NEXT: vmv.x.s a0, v10
; RV64-NEXT: ret ; RV64-NEXT: ret
%r = call i64 @llvm.vp.reduce.xor.v4i64(i64 %s, <4 x i64> %v, <4 x i1> %m, i32 %evl) %r = call i64 @llvm.vp.reduce.xor.v4i64(i64 %s, <4 x i64> %v, <4 x i1> %m, i32 %evl)
ret i64 %r ret i64 %r
} }
declare i8 @llvm.vp.reduce.mul.v1i8(i8, <1 x i8>, <1 x i1>, i32)
define i8 @vpreduce_mul_v1i8(i8 %s, <1 x i8> %v, <1 x i1> %m, i32 zeroext %evl) {
; RV32-LABEL: vpreduce_mul_v1i8:
; RV32: # %bb.0:
; RV32-NEXT: addi sp, sp, -16
; RV32-NEXT: .cfi_def_cfa_offset 16
; RV32-NEXT: sw ra, 12(sp) # 4-byte Folded Spill
; RV32-NEXT: .cfi_offset ra, -4
; RV32-NEXT: mv a2, a0
; RV32-NEXT: vsetivli zero, 1, e32, mf2, ta, ma
; RV32-NEXT: vmv.v.x v9, a1
; RV32-NEXT: vmsne.vi v9, v9, 0
; RV32-NEXT: vmand.mm v0, v9, v0
; RV32-NEXT: vsetvli zero, zero, e8, mf8, ta, ma
; RV32-NEXT: vmv.v.i v9, 1
; RV32-NEXT: vmerge.vvm v8, v9, v8, v0
; RV32-NEXT: vmv.x.s a0, v8
; RV32-NEXT: mv a1, a2
; RV32-NEXT: call __mulsi3@plt
; RV32-NEXT: lw ra, 12(sp) # 4-byte Folded Reload
; RV32-NEXT: addi sp, sp, 16
; RV32-NEXT: ret
;
; RV64-LABEL: vpreduce_mul_v1i8:
; RV64: # %bb.0:
; RV64-NEXT: addi sp, sp, -16
; RV64-NEXT: .cfi_def_cfa_offset 16
; RV64-NEXT: sd ra, 8(sp) # 8-byte Folded Spill
; RV64-NEXT: .cfi_offset ra, -8
; RV64-NEXT: mv a2, a0
; RV64-NEXT: vsetivli zero, 1, e32, mf2, ta, ma
; RV64-NEXT: vmv.v.x v9, a1
; RV64-NEXT: vmsne.vi v9, v9, 0
; RV64-NEXT: vmand.mm v0, v9, v0
; RV64-NEXT: vsetvli zero, zero, e8, mf8, ta, ma
; RV64-NEXT: vmv.v.i v9, 1
; RV64-NEXT: vmerge.vvm v8, v9, v8, v0
; RV64-NEXT: vmv.x.s a0, v8
; RV64-NEXT: mv a1, a2
; RV64-NEXT: call __muldi3@plt
; RV64-NEXT: ld ra, 8(sp) # 8-byte Folded Reload
; RV64-NEXT: addi sp, sp, 16
; RV64-NEXT: ret
%r = call i8 @llvm.vp.reduce.mul.v1i8(i8 %s, <1 x i8> %v, <1 x i1> %m, i32 %evl)
ret i8 %r
}
declare i8 @llvm.vp.reduce.mul.v2i8(i8, <2 x i8>, <2 x i1>, i32)
define signext i8 @vpreduce_mul_v2i8(i8 signext %s, <2 x i8> %v, <2 x i1> %m, i32 zeroext %evl) {
; RV32-LABEL: vpreduce_mul_v2i8:
; RV32: # %bb.0:
; RV32-NEXT: addi sp, sp, -16
; RV32-NEXT: .cfi_def_cfa_offset 16
; RV32-NEXT: sw ra, 12(sp) # 4-byte Folded Spill
; RV32-NEXT: .cfi_offset ra, -4
; RV32-NEXT: mv a2, a0
; RV32-NEXT: vsetivli zero, 2, e32, mf2, ta, ma
; RV32-NEXT: vid.v v9
; RV32-NEXT: vmsltu.vx v9, v9, a1
; RV32-NEXT: vmand.mm v0, v9, v0
; RV32-NEXT: vsetvli zero, zero, e8, mf8, ta, ma
; RV32-NEXT: vmv.v.i v9, 1
; RV32-NEXT: vmerge.vvm v8, v9, v8, v0
; RV32-NEXT: vrgather.vi v9, v8, 1
; RV32-NEXT: vmul.vv v8, v8, v9
; RV32-NEXT: vmv.x.s a0, v8
; RV32-NEXT: mv a1, a2
; RV32-NEXT: call __mulsi3@plt
; RV32-NEXT: slli a0, a0, 24
; RV32-NEXT: srai a0, a0, 24
; RV32-NEXT: lw ra, 12(sp) # 4-byte Folded Reload
; RV32-NEXT: addi sp, sp, 16
; RV32-NEXT: ret
;
; RV64-LABEL: vpreduce_mul_v2i8:
; RV64: # %bb.0:
; RV64-NEXT: addi sp, sp, -16
; RV64-NEXT: .cfi_def_cfa_offset 16
; RV64-NEXT: sd ra, 8(sp) # 8-byte Folded Spill
; RV64-NEXT: .cfi_offset ra, -8
; RV64-NEXT: mv a2, a0
; RV64-NEXT: vsetivli zero, 2, e32, mf2, ta, ma
; RV64-NEXT: vid.v v9
; RV64-NEXT: vmsltu.vx v9, v9, a1
; RV64-NEXT: vmand.mm v0, v9, v0
; RV64-NEXT: vsetvli zero, zero, e8, mf8, ta, ma
; RV64-NEXT: vmv.v.i v9, 1
; RV64-NEXT: vmerge.vvm v8, v9, v8, v0
; RV64-NEXT: vrgather.vi v9, v8, 1
; RV64-NEXT: vmul.vv v8, v8, v9
; RV64-NEXT: vmv.x.s a0, v8
; RV64-NEXT: mv a1, a2
; RV64-NEXT: call __muldi3@plt
; RV64-NEXT: slli a0, a0, 56
; RV64-NEXT: srai a0, a0, 56
; RV64-NEXT: ld ra, 8(sp) # 8-byte Folded Reload
; RV64-NEXT: addi sp, sp, 16
; RV64-NEXT: ret
%r = call i8 @llvm.vp.reduce.mul.v2i8(i8 %s, <2 x i8> %v, <2 x i1> %m, i32 %evl)
ret i8 %r
}
declare i8 @llvm.vp.reduce.mul.v4i8(i8, <4 x i8>, <4 x i1>, i32)
define signext i8 @vpreduce_mul_v4i8(i8 signext %s, <4 x i8> %v, <4 x i1> %m, i32 zeroext %evl) {
; RV32-LABEL: vpreduce_mul_v4i8:
; RV32: # %bb.0:
; RV32-NEXT: addi sp, sp, -16
; RV32-NEXT: .cfi_def_cfa_offset 16
; RV32-NEXT: sw ra, 12(sp) # 4-byte Folded Spill
; RV32-NEXT: .cfi_offset ra, -4
; RV32-NEXT: mv a2, a0
; RV32-NEXT: vsetivli zero, 4, e32, m1, ta, ma
; RV32-NEXT: vid.v v9
; RV32-NEXT: vmsltu.vx v9, v9, a1
; RV32-NEXT: vmand.mm v0, v9, v0
; RV32-NEXT: vsetvli zero, zero, e8, mf4, ta, ma
; RV32-NEXT: vmv.v.i v9, 1
; RV32-NEXT: vmerge.vvm v8, v9, v8, v0
; RV32-NEXT: vslidedown.vi v9, v8, 2
; RV32-NEXT: vmul.vv v8, v8, v9
; RV32-NEXT: vrgather.vi v9, v8, 1
; RV32-NEXT: vmul.vv v8, v8, v9
; RV32-NEXT: vmv.x.s a0, v8
; RV32-NEXT: mv a1, a2
; RV32-NEXT: call __mulsi3@plt
; RV32-NEXT: slli a0, a0, 24
; RV32-NEXT: srai a0, a0, 24
; RV32-NEXT: lw ra, 12(sp) # 4-byte Folded Reload
; RV32-NEXT: addi sp, sp, 16
; RV32-NEXT: ret
;
; RV64-LABEL: vpreduce_mul_v4i8:
; RV64: # %bb.0:
; RV64-NEXT: addi sp, sp, -16
; RV64-NEXT: .cfi_def_cfa_offset 16
; RV64-NEXT: sd ra, 8(sp) # 8-byte Folded Spill
; RV64-NEXT: .cfi_offset ra, -8
; RV64-NEXT: mv a2, a0
; RV64-NEXT: vsetivli zero, 4, e32, m1, ta, ma
; RV64-NEXT: vid.v v9
; RV64-NEXT: vmsltu.vx v9, v9, a1
; RV64-NEXT: vmand.mm v0, v9, v0
; RV64-NEXT: vsetvli zero, zero, e8, mf4, ta, ma
; RV64-NEXT: vmv.v.i v9, 1
; RV64-NEXT: vmerge.vvm v8, v9, v8, v0
; RV64-NEXT: vslidedown.vi v9, v8, 2
; RV64-NEXT: vmul.vv v8, v8, v9
; RV64-NEXT: vrgather.vi v9, v8, 1
; RV64-NEXT: vmul.vv v8, v8, v9
; RV64-NEXT: vmv.x.s a0, v8
; RV64-NEXT: mv a1, a2
; RV64-NEXT: call __muldi3@plt
; RV64-NEXT: slli a0, a0, 56
; RV64-NEXT: srai a0, a0, 56
; RV64-NEXT: ld ra, 8(sp) # 8-byte Folded Reload
; RV64-NEXT: addi sp, sp, 16
; RV64-NEXT: ret
%r = call i8 @llvm.vp.reduce.mul.v4i8(i8 %s, <4 x i8> %v, <4 x i1> %m, i32 %evl)
ret i8 %r
}
declare i8 @llvm.vp.reduce.mul.v8i8(i8, <8 x i8>, <8 x i1>, i32)
define signext i8 @vpreduce_mul_v8i8(i8 signext %s, <8 x i8> %v, <8 x i1> %m, i32 zeroext %evl) {
; RV32-LABEL: vpreduce_mul_v8i8:
; RV32: # %bb.0:
; RV32-NEXT: addi sp, sp, -16
; RV32-NEXT: .cfi_def_cfa_offset 16
; RV32-NEXT: sw ra, 12(sp) # 4-byte Folded Spill
; RV32-NEXT: .cfi_offset ra, -4
; RV32-NEXT: mv a2, a0
; RV32-NEXT: vsetivli zero, 8, e32, m2, ta, ma
; RV32-NEXT: vid.v v10
; RV32-NEXT: vmsltu.vx v9, v10, a1
; RV32-NEXT: vmand.mm v0, v9, v0
; RV32-NEXT: vsetvli zero, zero, e8, mf2, ta, ma
; RV32-NEXT: vmv.v.i v9, 1
; RV32-NEXT: vmerge.vvm v8, v9, v8, v0
; RV32-NEXT: vslidedown.vi v9, v8, 4
; RV32-NEXT: vmul.vv v8, v8, v9
; RV32-NEXT: vslidedown.vi v9, v8, 2
; RV32-NEXT: vmul.vv v8, v8, v9
; RV32-NEXT: vrgather.vi v9, v8, 1
; RV32-NEXT: vmul.vv v8, v8, v9
; RV32-NEXT: vmv.x.s a0, v8
; RV32-NEXT: mv a1, a2
; RV32-NEXT: call __mulsi3@plt
; RV32-NEXT: slli a0, a0, 24
; RV32-NEXT: srai a0, a0, 24
; RV32-NEXT: lw ra, 12(sp) # 4-byte Folded Reload
; RV32-NEXT: addi sp, sp, 16
; RV32-NEXT: ret
;
; RV64-LABEL: vpreduce_mul_v8i8:
; RV64: # %bb.0:
; RV64-NEXT: addi sp, sp, -16
; RV64-NEXT: .cfi_def_cfa_offset 16
; RV64-NEXT: sd ra, 8(sp) # 8-byte Folded Spill
; RV64-NEXT: .cfi_offset ra, -8
; RV64-NEXT: mv a2, a0
; RV64-NEXT: vsetivli zero, 8, e32, m2, ta, ma
; RV64-NEXT: vid.v v10
; RV64-NEXT: vmsltu.vx v9, v10, a1
; RV64-NEXT: vmand.mm v0, v9, v0
; RV64-NEXT: vsetvli zero, zero, e8, mf2, ta, ma
; RV64-NEXT: vmv.v.i v9, 1
; RV64-NEXT: vmerge.vvm v8, v9, v8, v0
; RV64-NEXT: vslidedown.vi v9, v8, 4
; RV64-NEXT: vmul.vv v8, v8, v9
; RV64-NEXT: vslidedown.vi v9, v8, 2
; RV64-NEXT: vmul.vv v8, v8, v9
; RV64-NEXT: vrgather.vi v9, v8, 1
; RV64-NEXT: vmul.vv v8, v8, v9
; RV64-NEXT: vmv.x.s a0, v8
; RV64-NEXT: mv a1, a2
; RV64-NEXT: call __muldi3@plt
; RV64-NEXT: slli a0, a0, 56
; RV64-NEXT: srai a0, a0, 56
; RV64-NEXT: ld ra, 8(sp) # 8-byte Folded Reload
; RV64-NEXT: addi sp, sp, 16
; RV64-NEXT: ret
%r = call i8 @llvm.vp.reduce.mul.v8i8(i8 %s, <8 x i8> %v, <8 x i1> %m, i32 %evl)
ret i8 %r
}
declare i8 @llvm.vp.reduce.mul.v16i8(i8, <16 x i8>, <16 x i1>, i32)
define signext i8 @vpreduce_mul_v16i8(i8 signext %s, <16 x i8> %v, <16 x i1> %m, i32 zeroext %evl) {
; RV32-LABEL: vpreduce_mul_v16i8:
; RV32: # %bb.0:
; RV32-NEXT: addi sp, sp, -16
; RV32-NEXT: .cfi_def_cfa_offset 16
; RV32-NEXT: sw ra, 12(sp) # 4-byte Folded Spill
; RV32-NEXT: .cfi_offset ra, -4
; RV32-NEXT: mv a2, a0
; RV32-NEXT: vsetivli zero, 16, e32, m4, ta, ma
; RV32-NEXT: vid.v v12
; RV32-NEXT: vmsltu.vx v9, v12, a1
; RV32-NEXT: vmand.mm v0, v9, v0
; RV32-NEXT: vsetvli zero, zero, e8, m1, ta, ma
; RV32-NEXT: vmv.v.i v9, 1
; RV32-NEXT: vmerge.vvm v8, v9, v8, v0
; RV32-NEXT: vslidedown.vi v9, v8, 8
; RV32-NEXT: vmul.vv v8, v8, v9
; RV32-NEXT: vslidedown.vi v9, v8, 4
; RV32-NEXT: vmul.vv v8, v8, v9
; RV32-NEXT: vslidedown.vi v9, v8, 2
; RV32-NEXT: vmul.vv v8, v8, v9
; RV32-NEXT: vrgather.vi v9, v8, 1
; RV32-NEXT: vmul.vv v8, v8, v9
; RV32-NEXT: vmv.x.s a0, v8
; RV32-NEXT: mv a1, a2
; RV32-NEXT: call __mulsi3@plt
; RV32-NEXT: slli a0, a0, 24
; RV32-NEXT: srai a0, a0, 24
; RV32-NEXT: lw ra, 12(sp) # 4-byte Folded Reload
; RV32-NEXT: addi sp, sp, 16
; RV32-NEXT: ret
;
; RV64-LABEL: vpreduce_mul_v16i8:
; RV64: # %bb.0:
; RV64-NEXT: addi sp, sp, -16
; RV64-NEXT: .cfi_def_cfa_offset 16
; RV64-NEXT: sd ra, 8(sp) # 8-byte Folded Spill
; RV64-NEXT: .cfi_offset ra, -8
; RV64-NEXT: mv a2, a0
; RV64-NEXT: vsetivli zero, 16, e32, m4, ta, ma
; RV64-NEXT: vid.v v12
; RV64-NEXT: vmsltu.vx v9, v12, a1
; RV64-NEXT: vmand.mm v0, v9, v0
; RV64-NEXT: vsetvli zero, zero, e8, m1, ta, ma
; RV64-NEXT: vmv.v.i v9, 1
; RV64-NEXT: vmerge.vvm v8, v9, v8, v0
; RV64-NEXT: vslidedown.vi v9, v8, 8
; RV64-NEXT: vmul.vv v8, v8, v9
; RV64-NEXT: vslidedown.vi v9, v8, 4
; RV64-NEXT: vmul.vv v8, v8, v9
; RV64-NEXT: vslidedown.vi v9, v8, 2
; RV64-NEXT: vmul.vv v8, v8, v9
; RV64-NEXT: vrgather.vi v9, v8, 1
; RV64-NEXT: vmul.vv v8, v8, v9
; RV64-NEXT: vmv.x.s a0, v8
; RV64-NEXT: mv a1, a2
; RV64-NEXT: call __muldi3@plt
; RV64-NEXT: slli a0, a0, 56
; RV64-NEXT: srai a0, a0, 56
; RV64-NEXT: ld ra, 8(sp) # 8-byte Folded Reload
; RV64-NEXT: addi sp, sp, 16
; RV64-NEXT: ret
%r = call i8 @llvm.vp.reduce.mul.v16i8(i8 %s, <16 x i8> %v, <16 x i1> %m, i32 %evl)
ret i8 %r
}
declare i8 @llvm.vp.reduce.mul.v32i8(i8, <32 x i8>, <32 x i1>, i32)
define signext i8 @vpreduce_mul_v32i8(i8 signext %s, <32 x i8> %v, <32 x i1> %m, i32 zeroext %evl) {
; RV32-LABEL: vpreduce_mul_v32i8:
; RV32: # %bb.0:
; RV32-NEXT: addi sp, sp, -16
; RV32-NEXT: .cfi_def_cfa_offset 16
; RV32-NEXT: sw ra, 12(sp) # 4-byte Folded Spill
; RV32-NEXT: .cfi_offset ra, -4
; RV32-NEXT: mv a2, a0
; RV32-NEXT: li a0, 32
; RV32-NEXT: vsetvli zero, a0, e32, m8, ta, ma
; RV32-NEXT: vid.v v16
; RV32-NEXT: vmsltu.vx v10, v16, a1
; RV32-NEXT: vmand.mm v0, v10, v0
; RV32-NEXT: vsetvli zero, zero, e8, m2, ta, ma
; RV32-NEXT: vmv.v.i v10, 1
; RV32-NEXT: vmerge.vvm v8, v10, v8, v0
; RV32-NEXT: vslidedown.vi v10, v8, 16
; RV32-NEXT: vmul.vv v8, v8, v10
; RV32-NEXT: vslidedown.vi v10, v8, 8
; RV32-NEXT: vmul.vv v8, v8, v10
; RV32-NEXT: vslidedown.vi v10, v8, 4
; RV32-NEXT: vmul.vv v8, v8, v10
; RV32-NEXT: vslidedown.vi v10, v8, 2
; RV32-NEXT: vmul.vv v8, v8, v10
; RV32-NEXT: vrgather.vi v10, v8, 1
; RV32-NEXT: vmul.vv v8, v8, v10
; RV32-NEXT: vmv.x.s a0, v8
; RV32-NEXT: mv a1, a2
; RV32-NEXT: call __mulsi3@plt
; RV32-NEXT: slli a0, a0, 24
; RV32-NEXT: srai a0, a0, 24
; RV32-NEXT: lw ra, 12(sp) # 4-byte Folded Reload
; RV32-NEXT: addi sp, sp, 16
; RV32-NEXT: ret
;
; RV64-LABEL: vpreduce_mul_v32i8:
; RV64: # %bb.0:
; RV64-NEXT: addi sp, sp, -16
; RV64-NEXT: .cfi_def_cfa_offset 16
; RV64-NEXT: sd ra, 8(sp) # 8-byte Folded Spill
; RV64-NEXT: .cfi_offset ra, -8
; RV64-NEXT: mv a2, a0
; RV64-NEXT: li a0, 32
; RV64-NEXT: vsetvli zero, a0, e32, m8, ta, ma
; RV64-NEXT: vid.v v16
; RV64-NEXT: vmsltu.vx v10, v16, a1
; RV64-NEXT: vmand.mm v0, v10, v0
; RV64-NEXT: vsetvli zero, zero, e8, m2, ta, ma
; RV64-NEXT: vmv.v.i v10, 1
; RV64-NEXT: vmerge.vvm v8, v10, v8, v0
; RV64-NEXT: vslidedown.vi v10, v8, 16
; RV64-NEXT: vmul.vv v8, v8, v10
; RV64-NEXT: vslidedown.vi v10, v8, 8
; RV64-NEXT: vmul.vv v8, v8, v10
; RV64-NEXT: vslidedown.vi v10, v8, 4
; RV64-NEXT: vmul.vv v8, v8, v10
; RV64-NEXT: vslidedown.vi v10, v8, 2
; RV64-NEXT: vmul.vv v8, v8, v10
; RV64-NEXT: vrgather.vi v10, v8, 1
; RV64-NEXT: vmul.vv v8, v8, v10
; RV64-NEXT: vmv.x.s a0, v8
; RV64-NEXT: mv a1, a2
; RV64-NEXT: call __muldi3@plt
; RV64-NEXT: slli a0, a0, 56
; RV64-NEXT: srai a0, a0, 56
; RV64-NEXT: ld ra, 8(sp) # 8-byte Folded Reload
; RV64-NEXT: addi sp, sp, 16
; RV64-NEXT: ret
%r = call i8 @llvm.vp.reduce.mul.v32i8(i8 %s, <32 x i8> %v, <32 x i1> %m, i32 %evl)
ret i8 %r
}
declare i8 @llvm.vp.reduce.mul.v64i8(i8, <64 x i8>, <64 x i1>, i32)
define signext i8 @vpreduce_mul_v64i8(i8 signext %s, <64 x i8> %v, <64 x i1> %m, i32 zeroext %evl) {
; RV32-LABEL: vpreduce_mul_v64i8:
; RV32: # %bb.0:
; RV32-NEXT: addi sp, sp, -16
; RV32-NEXT: .cfi_def_cfa_offset 16
; RV32-NEXT: sw ra, 12(sp) # 4-byte Folded Spill
; RV32-NEXT: .cfi_offset ra, -4
; RV32-NEXT: li a3, 32
; RV32-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; RV32-NEXT: lui a2, %hi(.LCPI72_0)
; RV32-NEXT: addi a2, a2, %lo(.LCPI72_0)
; RV32-NEXT: vle32.v v16, (a2)
; RV32-NEXT: mv a2, a0
; RV32-NEXT: vid.v v24
; RV32-NEXT: vmsltu.vx v12, v24, a1
; RV32-NEXT: vmsltu.vx v13, v16, a1
; RV32-NEXT: vsetivli zero, 8, e8, mf2, tu, ma
; RV32-NEXT: vslideup.vi v12, v13, 4
; RV32-NEXT: li a0, 64
; RV32-NEXT: vsetvli zero, a0, e8, m4, ta, ma
; RV32-NEXT: vmand.mm v0, v12, v0
; RV32-NEXT: vmv.v.i v12, 1
; RV32-NEXT: vmerge.vvm v8, v12, v8, v0
; RV32-NEXT: vslidedown.vx v12, v8, a3
; RV32-NEXT: vmul.vv v8, v8, v12
; RV32-NEXT: vslidedown.vi v12, v8, 16
; RV32-NEXT: vmul.vv v8, v8, v12
; RV32-NEXT: vslidedown.vi v12, v8, 8
; RV32-NEXT: vmul.vv v8, v8, v12
; RV32-NEXT: vslidedown.vi v12, v8, 4
; RV32-NEXT: vmul.vv v8, v8, v12
; RV32-NEXT: vslidedown.vi v12, v8, 2
; RV32-NEXT: vmul.vv v8, v8, v12
; RV32-NEXT: vrgather.vi v12, v8, 1
; RV32-NEXT: vmul.vv v8, v8, v12
; RV32-NEXT: vmv.x.s a0, v8
; RV32-NEXT: mv a1, a2
; RV32-NEXT: call __mulsi3@plt
; RV32-NEXT: slli a0, a0, 24
; RV32-NEXT: srai a0, a0, 24
; RV32-NEXT: lw ra, 12(sp) # 4-byte Folded Reload
; RV32-NEXT: addi sp, sp, 16
; RV32-NEXT: ret
;
; RV64-LABEL: vpreduce_mul_v64i8:
; RV64: # %bb.0:
; RV64-NEXT: addi sp, sp, -16
; RV64-NEXT: .cfi_def_cfa_offset 16
; RV64-NEXT: sd ra, 8(sp) # 8-byte Folded Spill
; RV64-NEXT: .cfi_offset ra, -8
; RV64-NEXT: li a3, 32
; RV64-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; RV64-NEXT: lui a2, %hi(.LCPI72_0)
; RV64-NEXT: addi a2, a2, %lo(.LCPI72_0)
; RV64-NEXT: vle32.v v16, (a2)
; RV64-NEXT: mv a2, a0
; RV64-NEXT: vid.v v24
; RV64-NEXT: vmsltu.vx v12, v24, a1
; RV64-NEXT: vmsltu.vx v13, v16, a1
; RV64-NEXT: vsetivli zero, 8, e8, mf2, tu, ma
; RV64-NEXT: vslideup.vi v12, v13, 4
; RV64-NEXT: li a0, 64
; RV64-NEXT: vsetvli zero, a0, e8, m4, ta, ma
; RV64-NEXT: vmand.mm v0, v12, v0
; RV64-NEXT: vmv.v.i v12, 1
; RV64-NEXT: vmerge.vvm v8, v12, v8, v0
; RV64-NEXT: vslidedown.vx v12, v8, a3
; RV64-NEXT: vmul.vv v8, v8, v12
; RV64-NEXT: vslidedown.vi v12, v8, 16
; RV64-NEXT: vmul.vv v8, v8, v12
; RV64-NEXT: vslidedown.vi v12, v8, 8
; RV64-NEXT: vmul.vv v8, v8, v12
; RV64-NEXT: vslidedown.vi v12, v8, 4
; RV64-NEXT: vmul.vv v8, v8, v12
; RV64-NEXT: vslidedown.vi v12, v8, 2
; RV64-NEXT: vmul.vv v8, v8, v12
; RV64-NEXT: vrgather.vi v12, v8, 1
; RV64-NEXT: vmul.vv v8, v8, v12
; RV64-NEXT: vmv.x.s a0, v8
; RV64-NEXT: mv a1, a2
; RV64-NEXT: call __muldi3@plt
; RV64-NEXT: slli a0, a0, 56
; RV64-NEXT: srai a0, a0, 56
; RV64-NEXT: ld ra, 8(sp) # 8-byte Folded Reload
; RV64-NEXT: addi sp, sp, 16
; RV64-NEXT: ret
%r = call i8 @llvm.vp.reduce.mul.v64i8(i8 %s, <64 x i8> %v, <64 x i1> %m, i32 %evl)
ret i8 %r
}