This is an archive of the discontinued LLVM Phabricator instance.

Differential D128023

[RISCV] Add merge operand to RISCVISD::VRGATHER*_VL nodes.
ClosedPublic

Authored by craig.topper on Jun 16 2022, 7:38 PM.

Details

Reviewers
reames
frasercrmck
rogfer01
kito-cheng
Commits
rG16d3a82de53d: [RISCV] Add merge operand to RISCVISD::VRGATHER*_VL nodes.
Summary

Use it in place of VSELECT_VL+VRGATHER*_VL.

This simplifies the isel patterns.

Overall, I think trying to match select+op to create masked instructions
in isel doesn't scale. We either need to do it in DAG combine, pre-isel
peepole, or post-isel peephole. I don't yet know which is the right
answer, but for this case it seemed best to be able to request the
masked form directly from lowering.

Diff Detail

Event Timeline

craig.topper created this revision.Jun 16 2022, 7:38 PM
Herald added a project: Restricted Project. · View Herald TranscriptJun 16 2022, 7:38 PM
Herald added subscribers: sunshaoce, VincentWu, luke957 and 27 others. · View Herald Transcript
craig.topper requested review of this revision.Jun 16 2022, 7:38 PM
Herald added a project: Restricted Project. · View Herald TranscriptJun 16 2022, 7:38 PM
Herald added subscribers: • pcwang-thead, eopXD, MaskRay. · View Herald Transcript
Harbormaster completed remote builds in B170418: Diff 437774.Jun 16 2022, 8:29 PM
frasercrmck added a comment.Jun 16 2022, 10:37 PM

Seems reasonable to me, it is hard to see how select+op patterns could ever scale on large ISA like this. I've done a post-isel MIR peephole for this sort of thing in the past. We could perhaps also do a DAG post-process like we do for masked/unmasked RVV.

On the patch: we probably need to update the ISD opcode comment in RISCVISelLowering.h? We do comment passthru/merge operands though I don't know if our coverage is 100%.

llvm/lib/Target/RISCV/RISCVInstrInfoVVLPatterns.td
1841–1842

I'm a bit out of the loop, but could the DAG post-process turn masked vrgathers into unmasked ones? Could we remove the true_mask patterns while we're at it?

craig.topper updated this revision to Diff 437809.Jun 17 2022, 12:20 AM

Update description in RISCVISelLowering.h

Harbormaster completed remote builds in B170446: Diff 437809.Jun 17 2022, 1:08 AM
reames added a comment.Jun 17 2022, 12:05 PM

Looks reasonable to me, though the TD changes are a bit more involved than I'm comfortable signing off on just yet. Will defer to Fraser.

craig.topper added inline comments.Jun 17 2022, 12:35 PM
llvm/lib/Target/RISCV/RISCVInstrInfoVVLPatterns.td
1841

Fraser knows this but I'll mention it for @reames. srcvalue happens to be hole in the isel pattern system that causes the operand to be completely ignored. Since the mask is true_mask here we don't need the value. This allows the operand to be dropped even if it isn't undef. I think we get an error if we capture an operand and then don't use it in the output pattern.

frasercrmck accepted this revision.Jun 20 2022, 8:42 AM

LGTM

This revision is now accepted and ready to land.Jun 20 2022, 8:42 AM
This revision was landed with ongoing or failed builds.Jun 20 2022, 7:16 PM
Closed by commit rG16d3a82de53d: [RISCV] Add merge operand to RISCVISD::VRGATHER*_VL nodes. (authored by craig.topper). · Explain Why
This revision was automatically updated to reflect the committed changes.
craig.topper added a commit: rG16d3a82de53d: [RISCV] Add merge operand to RISCVISD::VRGATHER*_VL nodes..

Revision Contents

PathSize
llvm/
lib/
Target/
RISCV/
4 lines
45 lines
143 lines

Diff 437809

llvm/lib/Target/RISCV/RISCVISelLowering.h

//===-- RISCVISelLowering.h - RISCV DAG Lowering Interface ------*- C++ -*-===// //===-- RISCVISelLowering.h - RISCV DAG Lowering Interface ------*- C++ -*-===//
Lint: Lint
Inline Actions

clang-format not found in user’s local PATH; not linting file.

Lint: Lint: clang-format not found in user’s local PATH; not linting file.
// //
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information. // See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
// //
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// //
▲ Show 20 Lines • Show All 272 Lines • ▼ Show 20 Linesenum NodeType : unsigned {
VMAND_VL, VMAND_VL,
VMOR_VL, VMOR_VL,
VMXOR_VL, VMXOR_VL,
// Set mask vector to all zeros or ones. // Set mask vector to all zeros or ones.
VMCLR_VL, VMCLR_VL,
VMSET_VL, VMSET_VL,
// Matches the semantics of vrgather.vx and vrgather.vv with an extra operand // Matches the semantics of vrgather.vx and vrgather.vv with extra operands
// for VL. // for passthru and VL. First operand is the passthru operand.
VRGATHER_VX_VL, VRGATHER_VX_VL,
VRGATHER_VV_VL, VRGATHER_VV_VL,
VRGATHEREI16_VV_VL, VRGATHEREI16_VV_VL,
// Vector sign/zero extend with additional mask & VL operands. // Vector sign/zero extend with additional mask & VL operands.
VSEXT_VL, VSEXT_VL,
VZEXT_VL, VZEXT_VL,
▲ Show 20 Lines • Show All 441 Lines • Show Last 20 Lines

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

  • This file is larger than 256 KB, so syntax highlighting is disabled by default.
//===-- RISCVISelLowering.cpp - RISCV DAG Lowering Implementation --------===// //===-- RISCVISelLowering.cpp - RISCV DAG Lowering Implementation --------===//
Lint: Lint
Inline Actions

clang-format not found in user’s local PATH; not linting file.

Lint: Lint: clang-format not found in user’s local PATH; not linting file.
// //
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information. // See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
// //
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// //
▲ Show 20 Lines • Show All 1,940 Lines • ▼ Show 20 Linesstatic SDValue matchSplatAsGather(SDValue SplatVal, MVT VT, const SDLoc &DL,
if (VT.isFixedLengthVector()) { if (VT.isFixedLengthVector()) {
ContainerVT = getContainerForFixedLengthVector(DAG, VT, Subtarget); ContainerVT = getContainerForFixedLengthVector(DAG, VT, Subtarget);
Vec = convertToScalableVector(ContainerVT, Vec, DAG, Subtarget); Vec = convertToScalableVector(ContainerVT, Vec, DAG, Subtarget);
} }
SDValue Mask, VL; SDValue Mask, VL;
std::tie(Mask, VL) = getDefaultVLOps(VT, ContainerVT, DL, DAG, Subtarget); std::tie(Mask, VL) = getDefaultVLOps(VT, ContainerVT, DL, DAG, Subtarget);
SDValue Gather = DAG.getNode(RISCVISD::VRGATHER_VX_VL, DL, ContainerVT, Vec, SDValue Gather = DAG.getNode(RISCVISD::VRGATHER_VX_VL, DL, ContainerVT,
Idx, Mask, VL); DAG.getUNDEF(ContainerVT), Vec, Idx, Mask, VL);
if (!VT.isFixedLengthVector()) if (!VT.isFixedLengthVector())
return Gather; return Gather;
return convertFromScalableVector(VT, Gather, DAG, Subtarget); return convertFromScalableVector(VT, Gather, DAG, Subtarget);
} }
static SDValue lowerBUILD_VECTOR(SDValue Op, SelectionDAG &DAG, static SDValue lowerBUILD_VECTOR(SDValue Op, SelectionDAG &DAG,
▲ Show 20 Lines • Show All 606 Lines • ▼ Show 20 Lines if (Lane >= 0) {
VT.isFloatingPoint() ? RISCVISD::VFMV_V_F_VL : RISCVISD::VMV_V_X_VL; VT.isFloatingPoint() ? RISCVISD::VFMV_V_F_VL : RISCVISD::VMV_V_X_VL;
SDValue Splat = SDValue Splat =
DAG.getNode(Opc, DL, ContainerVT, DAG.getUNDEF(ContainerVT), V, VL); DAG.getNode(Opc, DL, ContainerVT, DAG.getUNDEF(ContainerVT), V, VL);
return convertFromScalableVector(VT, Splat, DAG, Subtarget); return convertFromScalableVector(VT, Splat, DAG, Subtarget);
} }
V1 = convertToScalableVector(ContainerVT, V1, DAG, Subtarget); V1 = convertToScalableVector(ContainerVT, V1, DAG, Subtarget);
assert(Lane < (int)NumElts && "Unexpected lane!"); assert(Lane < (int)NumElts && "Unexpected lane!");
SDValue Gather = SDValue Gather = DAG.getNode(
DAG.getNode(RISCVISD::VRGATHER_VX_VL, DL, ContainerVT, V1, RISCVISD::VRGATHER_VX_VL, DL, ContainerVT, DAG.getUNDEF(ContainerVT),
DAG.getConstant(Lane, DL, XLenVT), TrueMask, VL); V1, DAG.getConstant(Lane, DL, XLenVT), TrueMask, VL);
return convertFromScalableVector(VT, Gather, DAG, Subtarget); return convertFromScalableVector(VT, Gather, DAG, Subtarget);
} }
} }
ArrayRef<int> Mask = SVN->getMask(); ArrayRef<int> Mask = SVN->getMask();
// Lower rotations to a SLIDEDOWN and a SLIDEUP. One of the source vectors may // Lower rotations to a SLIDEDOWN and a SLIDEUP. One of the source vectors may
// be undef which can be handled with a single SLIDEDOWN/UP. // be undef which can be handled with a single SLIDEDOWN/UP.
▲ Show 20 Lines • Show All 193 Lines • ▼ Show 20 Lines Gather = lowerScalarSplat(SDValue(), SplatValue, VL, ContainerVT, DL, DAG,
Subtarget); Subtarget);
} else { } else {
V1 = convertToScalableVector(ContainerVT, V1, DAG, Subtarget); V1 = convertToScalableVector(ContainerVT, V1, DAG, Subtarget);
// If only one index is used, we can use a "splat" vrgather. // If only one index is used, we can use a "splat" vrgather.
// TODO: We can splat the most-common index and fix-up any stragglers, if // TODO: We can splat the most-common index and fix-up any stragglers, if
// that's beneficial. // that's beneficial.
if (LHSIndexCounts.size() == 1) { if (LHSIndexCounts.size() == 1) {
int SplatIndex = LHSIndexCounts.begin()->getFirst(); int SplatIndex = LHSIndexCounts.begin()->getFirst();
Gather = Gather = DAG.getNode(
DAG.getNode(GatherVXOpc, DL, ContainerVT, V1, GatherVXOpc, DL, ContainerVT, DAG.getUNDEF(ContainerVT), V1,
DAG.getConstant(SplatIndex, DL, XLenVT), TrueMask, VL); DAG.getConstant(SplatIndex, DL, XLenVT), TrueMask, VL);
} else { } else {
SDValue LHSIndices = DAG.getBuildVector(IndexVT, DL, GatherIndicesLHS); SDValue LHSIndices = DAG.getBuildVector(IndexVT, DL, GatherIndicesLHS);
LHSIndices = LHSIndices =
convertToScalableVector(IndexContainerVT, LHSIndices, DAG, Subtarget); convertToScalableVector(IndexContainerVT, LHSIndices, DAG, Subtarget);
Gather = DAG.getNode(GatherVVOpc, DL, ContainerVT, V1, LHSIndices, Gather =
TrueMask, VL); DAG.getNode(GatherVVOpc, DL, ContainerVT, DAG.getUNDEF(ContainerVT),
V1, LHSIndices, TrueMask, VL);
} }
} }
// If a second vector operand is used by this shuffle, blend it in with an // If a second vector operand is used by this shuffle, blend it in with an
// additional vrgather. // additional vrgather.
if (!V2.isUndef()) { if (!V2.isUndef()) {
V2 = convertToScalableVector(ContainerVT, V2, DAG, Subtarget); V2 = convertToScalableVector(ContainerVT, V2, DAG, Subtarget);
MVT MaskContainerVT = ContainerVT.changeVectorElementType(MVT::i1);
SelectMask =
convertToScalableVector(MaskContainerVT, SelectMask, DAG, Subtarget);
// If only one index is used, we can use a "splat" vrgather. // If only one index is used, we can use a "splat" vrgather.
// TODO: We can splat the most-common index and fix-up any stragglers, if // TODO: We can splat the most-common index and fix-up any stragglers, if
// that's beneficial. // that's beneficial.
if (RHSIndexCounts.size() == 1) { if (RHSIndexCounts.size() == 1) {
int SplatIndex = RHSIndexCounts.begin()->getFirst(); int SplatIndex = RHSIndexCounts.begin()->getFirst();
V2 = DAG.getNode(GatherVXOpc, DL, ContainerVT, V2, Gather =
DAG.getConstant(SplatIndex, DL, XLenVT), TrueMask, VL); DAG.getNode(GatherVXOpc, DL, ContainerVT, Gather, V2,
DAG.getConstant(SplatIndex, DL, XLenVT), SelectMask, VL);
} else { } else {
SDValue RHSIndices = DAG.getBuildVector(IndexVT, DL, GatherIndicesRHS); SDValue RHSIndices = DAG.getBuildVector(IndexVT, DL, GatherIndicesRHS);
RHSIndices = RHSIndices =
convertToScalableVector(IndexContainerVT, RHSIndices, DAG, Subtarget); convertToScalableVector(IndexContainerVT, RHSIndices, DAG, Subtarget);
V2 = DAG.getNode(GatherVVOpc, DL, ContainerVT, V2, RHSIndices, TrueMask, Gather = DAG.getNode(GatherVVOpc, DL, ContainerVT, Gather, V2, RHSIndices,
VL); SelectMask, VL);
} }
MVT MaskContainerVT = ContainerVT.changeVectorElementType(MVT::i1);
SelectMask =
convertToScalableVector(MaskContainerVT, SelectMask, DAG, Subtarget);
Gather = DAG.getNode(RISCVISD::VSELECT_VL, DL, ContainerVT, SelectMask, V2,
Gather, VL);
} }
return convertFromScalableVector(VT, Gather, DAG, Subtarget); return convertFromScalableVector(VT, Gather, DAG, Subtarget);
} }
bool RISCVTargetLowering::isShuffleMaskLegal(ArrayRef<int> M, EVT VT) const { bool RISCVTargetLowering::isShuffleMaskLegal(ArrayRef<int> M, EVT VT) const {
// Support splats for any type. These should type legalize well. // Support splats for any type. These should type legalize well.
if (ShuffleVectorSDNode::isSplatMask(M.data(), VT)) if (ShuffleVectorSDNode::isSplatMask(M.data(), VT))
▲ Show 20 Lines • Show All 2,844 Lines • ▼ Show 20 LinesSDValue RISCVTargetLowering::lowerVECTOR_REVERSE(SDValue Op,
else else
SplatVL = DAG.getNode(RISCVISD::VMV_V_X_VL, DL, IntVT, DAG.getUNDEF(IntVT), SplatVL = DAG.getNode(RISCVISD::VMV_V_X_VL, DL, IntVT, DAG.getUNDEF(IntVT),
VLMinus1, DAG.getRegister(RISCV::X0, XLenVT)); VLMinus1, DAG.getRegister(RISCV::X0, XLenVT));
SDValue VID = DAG.getNode(RISCVISD::VID_VL, DL, IntVT, Mask, VL); SDValue VID = DAG.getNode(RISCVISD::VID_VL, DL, IntVT, Mask, VL);
SDValue Indices = SDValue Indices =
DAG.getNode(RISCVISD::SUB_VL, DL, IntVT, SplatVL, VID, Mask, VL); DAG.getNode(RISCVISD::SUB_VL, DL, IntVT, SplatVL, VID, Mask, VL);
return DAG.getNode(GatherOpc, DL, VecVT, Op.getOperand(0), Indices, Mask, VL); return DAG.getNode(GatherOpc, DL, VecVT, DAG.getUNDEF(VecVT),
Op.getOperand(0), Indices, Mask, VL);
} }
SDValue RISCVTargetLowering::lowerVECTOR_SPLICE(SDValue Op, SDValue RISCVTargetLowering::lowerVECTOR_SPLICE(SDValue Op,
SelectionDAG &DAG) const { SelectionDAG &DAG) const {
SDLoc DL(Op); SDLoc DL(Op);
SDValue V1 = Op.getOperand(0); SDValue V1 = Op.getOperand(0);
SDValue V2 = Op.getOperand(1); SDValue V2 = Op.getOperand(1);
MVT XLenVT = Subtarget.getXLenVT(); MVT XLenVT = Subtarget.getXLenVT();
▲ Show 20 Lines • Show All 6,398 Lines • Show Last 20 Lines

llvm/lib/Target/RISCV/RISCVInstrInfoVVLPatterns.td

Show First 20 Lines • Show All 137 Lines • ▼ Show 20 Linesdef riscv_setcc_vl : SDNode<"RISCVISD::SETCC_VL",
SDTCisVec<1>, SDTCisVec<1>,
SDTCisSameNumEltsAs<0, 1>, SDTCisSameNumEltsAs<0, 1>,
SDTCisSameAs<1, 2>, SDTCisSameAs<1, 2>,
SDTCisVT<3, OtherVT>, SDTCisVT<3, OtherVT>,
SDTCisSameAs<0, 4>, SDTCisSameAs<0, 4>,
SDTCisVT<5, XLenVT>]>>; SDTCisVT<5, XLenVT>]>>;
def riscv_vrgather_vx_vl : SDNode<"RISCVISD::VRGATHER_VX_VL", def riscv_vrgather_vx_vl : SDNode<"RISCVISD::VRGATHER_VX_VL",
SDTypeProfile<1, 4, [SDTCisVec<0>, SDTypeProfile<1, 5, [SDTCisVec<0>,
SDTCisSameAs<0, 1>, SDTCisSameAs<0, 1>,
SDTCisVT<2, XLenVT>, SDTCisSameAs<0, 2>,
SDTCVecEltisVT<3, i1>, SDTCisVT<3, XLenVT>,
SDTCisSameNumEltsAs<0, 3>, SDTCVecEltisVT<4, i1>,
SDTCisVT<4, XLenVT>]>>; SDTCisSameNumEltsAs<0, 4>,
SDTCisVT<5, XLenVT>]>>;
def riscv_vrgather_vv_vl : SDNode<"RISCVISD::VRGATHER_VV_VL", def riscv_vrgather_vv_vl : SDNode<"RISCVISD::VRGATHER_VV_VL",
SDTypeProfile<1, 4, [SDTCisVec<0>, SDTypeProfile<1, 5, [SDTCisVec<0>,
SDTCisSameAs<0, 1>, SDTCisSameAs<0, 1>,
SDTCisInt<2>, SDTCisSameAs<0, 2>,
SDTCisSameNumEltsAs<0, 2>, SDTCisInt<3>,
SDTCisSameSizeAs<0, 2>,
SDTCVecEltisVT<3, i1>,
SDTCisSameNumEltsAs<0, 3>, SDTCisSameNumEltsAs<0, 3>,
SDTCisVT<4, XLenVT>]>>; SDTCisSameSizeAs<0, 3>,
SDTCVecEltisVT<4, i1>,
SDTCisSameNumEltsAs<0, 4>,
SDTCisVT<5, XLenVT>]>>;
def riscv_vrgatherei16_vv_vl : SDNode<"RISCVISD::VRGATHEREI16_VV_VL", def riscv_vrgatherei16_vv_vl : SDNode<"RISCVISD::VRGATHEREI16_VV_VL",
SDTypeProfile<1, 4, [SDTCisVec<0>, SDTypeProfile<1, 5, [SDTCisVec<0>,
SDTCisSameAs<0, 1>, SDTCisSameAs<0, 1>,
SDTCisInt<2>, SDTCisSameAs<0, 2>,
SDTCVecEltisVT<2, i16>, SDTCisInt<3>,
SDTCisSameNumEltsAs<0, 2>, SDTCVecEltisVT<3, i16>,
SDTCVecEltisVT<3, i1>,
SDTCisSameNumEltsAs<0, 3>, SDTCisSameNumEltsAs<0, 3>,
SDTCisVT<4, XLenVT>]>>; SDTCVecEltisVT<4, i1>,
SDTCisSameNumEltsAs<0, 4>,
SDTCisVT<5, XLenVT>]>>;
def SDT_RISCVSelect_VL : SDTypeProfile<1, 4, [ def SDT_RISCVSelect_VL : SDTypeProfile<1, 4, [
SDTCisVec<0>, SDTCisVec<1>, SDTCisSameNumEltsAs<0, 1>, SDTCVecEltisVT<1, i1>, SDTCisVec<0>, SDTCisVec<1>, SDTCisSameNumEltsAs<0, 1>, SDTCVecEltisVT<1, i1>,
SDTCisSameAs<0, 2>, SDTCisSameAs<2, 3>, SDTCisVT<4, XLenVT> SDTCisSameAs<0, 2>, SDTCisSameAs<2, 3>, SDTCisVT<4, XLenVT>
]>; ]>;
def riscv_vselect_vl : SDNode<"RISCVISD::VSELECT_VL", SDT_RISCVSelect_VL>; def riscv_vselect_vl : SDNode<"RISCVISD::VSELECT_VL", SDT_RISCVSelect_VL>;
def riscv_vp_merge_vl : SDNode<"RISCVISD::VP_MERGE_VL", SDT_RISCVSelect_VL>; def riscv_vp_merge_vl : SDNode<"RISCVISD::VP_MERGE_VL", SDT_RISCVSelect_VL>;
▲ Show 20 Lines • Show All 1,652 Lines • ▼ Show 20 Lines
// 17.4. Vector Register Gather Instruction // 17.4. Vector Register Gather Instruction
foreach vti = AllIntegerVectors in { foreach vti = AllIntegerVectors in {
def : Pat<(vti.Vector (riscv_vmv_s_x_vl (vti.Vector vti.RegClass:$merge), def : Pat<(vti.Vector (riscv_vmv_s_x_vl (vti.Vector vti.RegClass:$merge),
vti.ScalarRegClass:$rs1, vti.ScalarRegClass:$rs1,
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>("PseudoVMV_S_X_"#vti.LMul.MX) (!cast<Instruction>("PseudoVMV_S_X_"#vti.LMul.MX)
vti.RegClass:$merge, vti.RegClass:$merge,
(vti.Scalar vti.ScalarRegClass:$rs1), GPR:$vl, vti.Log2SEW)>; (vti.Scalar vti.ScalarRegClass:$rs1), GPR:$vl, vti.Log2SEW)>;
def : Pat<(vti.Vector (riscv_vrgather_vv_vl vti.RegClass:$rs2, def : Pat<(vti.Vector (riscv_vrgather_vv_vl (vti.Vector srcvalue),
craig.topperAuthorUnsubmitted
Done
ReplyInline Actions

Fraser knows this but I'll mention it for @reames. srcvalue happens to be hole in the isel pattern system that causes the operand to be completely ignored. Since the mask is true_mask here we don't need the value. This allows the operand to be dropped even if it isn't undef. I think we get an error if we capture an operand and then don't use it in the output pattern.

craig.topper: Fraser knows this but I'll mention it for @reames. `srcvalue` happens to be hole in the isel…
vti.RegClass:$rs2,
frasercrmckUnsubmitted
Not Done
ReplyInline Actions

I'm a bit out of the loop, but could the DAG post-process turn masked vrgathers into unmasked ones? Could we remove the true_mask patterns while we're at it?

frasercrmck: I'm a bit out of the loop, but could the DAG post-process turn masked vrgathers into unmasked…
(vti.Vector vti.RegClass:$rs1), (vti.Vector vti.RegClass:$rs1),
(vti.Mask true_mask), (vti.Mask true_mask),
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>("PseudoVRGATHER_VV_"# vti.LMul.MX) (!cast<Instruction>("PseudoVRGATHER_VV_"# vti.LMul.MX)
vti.RegClass:$rs2, vti.RegClass:$rs1, GPR:$vl, vti.Log2SEW)>; vti.RegClass:$rs2, vti.RegClass:$rs1, GPR:$vl, vti.Log2SEW)>;
def : Pat<(vti.Vector (riscv_vrgather_vx_vl vti.RegClass:$rs2, GPR:$rs1, def : Pat<(vti.Vector (riscv_vrgather_vx_vl (vti.Vector srcvalue),
vti.RegClass:$rs2, GPR:$rs1,
(vti.Mask true_mask), (vti.Mask true_mask),
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>("PseudoVRGATHER_VX_"# vti.LMul.MX) (!cast<Instruction>("PseudoVRGATHER_VX_"# vti.LMul.MX)
vti.RegClass:$rs2, GPR:$rs1, GPR:$vl, vti.Log2SEW)>; vti.RegClass:$rs2, GPR:$rs1, GPR:$vl, vti.Log2SEW)>;
def : Pat<(vti.Vector (riscv_vrgather_vx_vl vti.RegClass:$rs2, uimm5:$imm, def : Pat<(vti.Vector (riscv_vrgather_vx_vl (vti.Vector srcvalue),
vti.RegClass:$rs2, uimm5:$imm,
(vti.Mask true_mask), (vti.Mask true_mask),
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>("PseudoVRGATHER_VI_"# vti.LMul.MX) (!cast<Instruction>("PseudoVRGATHER_VI_"# vti.LMul.MX)
vti.RegClass:$rs2, uimm5:$imm, GPR:$vl, vti.Log2SEW)>; vti.RegClass:$rs2, uimm5:$imm, GPR:$vl, vti.Log2SEW)>;
def : Pat<(vti.Vector (riscv_vselect_vl (vti.Mask V0), def : Pat<(vti.Vector (riscv_vrgather_vv_vl vti.RegClass:$merge,
(riscv_vrgather_vv_vl
vti.RegClass:$rs2, vti.RegClass:$rs2,
vti.RegClass:$rs1, vti.RegClass:$rs1,
(vti.Mask true_mask), (vti.Mask V0),
VLOpFrag),
vti.RegClass:$merge,
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>("PseudoVRGATHER_VV_"# vti.LMul.MX#"_MASK") (!cast<Instruction>("PseudoVRGATHER_VV_"# vti.LMul.MX#"_MASK")
vti.RegClass:$merge, vti.RegClass:$rs2, vti.RegClass:$rs1, vti.RegClass:$merge, vti.RegClass:$rs2, vti.RegClass:$rs1,
(vti.Mask V0), GPR:$vl, vti.Log2SEW, TAIL_AGNOSTIC)>; (vti.Mask V0), GPR:$vl, vti.Log2SEW, TAIL_AGNOSTIC)>;
def : Pat<(vti.Vector (riscv_vselect_vl (vti.Mask V0), def : Pat<(vti.Vector (riscv_vrgather_vx_vl vti.RegClass:$merge,
(riscv_vrgather_vx_vl
vti.RegClass:$rs2, vti.RegClass:$rs2,
uimm5:$imm, uimm5:$imm,
(vti.Mask true_mask), (vti.Mask V0),
VLOpFrag),
vti.RegClass:$merge,
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>("PseudoVRGATHER_VI_"# vti.LMul.MX#"_MASK") (!cast<Instruction>("PseudoVRGATHER_VI_"# vti.LMul.MX#"_MASK")
vti.RegClass:$merge, vti.RegClass:$rs2, uimm5:$imm, vti.RegClass:$merge, vti.RegClass:$rs2, uimm5:$imm,
(vti.Mask V0), GPR:$vl, vti.Log2SEW, TAIL_AGNOSTIC)>; (vti.Mask V0), GPR:$vl, vti.Log2SEW, TAIL_AGNOSTIC)>;
// emul = lmul * 16 / sew // emul = lmul * 16 / sew
defvar vlmul = vti.LMul; defvar vlmul = vti.LMul;
defvar octuple_lmul = vlmul.octuple; defvar octuple_lmul = vlmul.octuple;
defvar octuple_emul = !srl(!mul(octuple_lmul, 16), vti.Log2SEW); defvar octuple_emul = !srl(!mul(octuple_lmul, 16), vti.Log2SEW);
if !and(!ge(octuple_emul, 1), !le(octuple_emul, 64)) then { if !and(!ge(octuple_emul, 1), !le(octuple_emul, 64)) then {
defvar emul_str = octuple_to_str<octuple_emul>.ret; defvar emul_str = octuple_to_str<octuple_emul>.ret;
defvar ivti = !cast<VTypeInfo>("VI16" # emul_str); defvar ivti = !cast<VTypeInfo>("VI16" # emul_str);
defvar inst = "PseudoVRGATHEREI16_VV_" # vti.LMul.MX # "_" # emul_str; defvar inst = "PseudoVRGATHEREI16_VV_" # vti.LMul.MX # "_" # emul_str;
def : Pat<(vti.Vector (riscv_vrgatherei16_vv_vl vti.RegClass:$rs2, def : Pat<(vti.Vector (riscv_vrgatherei16_vv_vl (vti.Vector srcvalue),
vti.RegClass:$rs2,
(ivti.Vector ivti.RegClass:$rs1), (ivti.Vector ivti.RegClass:$rs1),
(vti.Mask true_mask), (vti.Mask true_mask),
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>(inst) (!cast<Instruction>(inst)
vti.RegClass:$rs2, ivti.RegClass:$rs1, GPR:$vl, vti.Log2SEW)>; vti.RegClass:$rs2, ivti.RegClass:$rs1, GPR:$vl, vti.Log2SEW)>;
def : Pat<(vti.Vector (riscv_vselect_vl (vti.Mask V0), def : Pat<(vti.Vector
(riscv_vrgatherei16_vv_vl (riscv_vrgatherei16_vv_vl vti.RegClass:$merge,
vti.RegClass:$rs2, vti.RegClass:$rs2,
(ivti.Vector ivti.RegClass:$rs1), (ivti.Vector ivti.RegClass:$rs1),
(vti.Mask true_mask), (vti.Mask V0),
VLOpFrag),
vti.RegClass:$merge,
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>(inst#"_MASK") (!cast<Instruction>(inst#"_MASK")
vti.RegClass:$merge, vti.RegClass:$rs2, ivti.RegClass:$rs1, vti.RegClass:$merge, vti.RegClass:$rs2, ivti.RegClass:$rs1,
(vti.Mask V0), GPR:$vl, vti.Log2SEW, TAIL_AGNOSTIC)>; (vti.Mask V0), GPR:$vl, vti.Log2SEW, TAIL_AGNOSTIC)>;
} }
} }
} // Predicates = [HasVInstructions] } // Predicates = [HasVInstructions]
let Predicates = [HasVInstructionsAnyF] in { let Predicates = [HasVInstructionsAnyF] in {
// 17.2. Floating-Point Scalar Move Instructions // 17.2. Floating-Point Scalar Move Instructions
foreach vti = AllFloatVectors in { foreach vti = AllFloatVectors in {
def : Pat<(vti.Vector (riscv_vfmv_s_f_vl (vti.Vector vti.RegClass:$merge), def : Pat<(vti.Vector (riscv_vfmv_s_f_vl (vti.Vector vti.RegClass:$merge),
(vti.Scalar (fpimm0)), (vti.Scalar (fpimm0)),
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>("PseudoVMV_S_X_"#vti.LMul.MX) (!cast<Instruction>("PseudoVMV_S_X_"#vti.LMul.MX)
vti.RegClass:$merge, X0, GPR:$vl, vti.Log2SEW)>; vti.RegClass:$merge, X0, GPR:$vl, vti.Log2SEW)>;
def : Pat<(vti.Vector (riscv_vfmv_s_f_vl (vti.Vector vti.RegClass:$merge), def : Pat<(vti.Vector (riscv_vfmv_s_f_vl (vti.Vector vti.RegClass:$merge),
vti.ScalarRegClass:$rs1, vti.ScalarRegClass:$rs1,
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>("PseudoVFMV_S_"#vti.ScalarSuffix#"_"#vti.LMul.MX) (!cast<Instruction>("PseudoVFMV_S_"#vti.ScalarSuffix#"_"#vti.LMul.MX)
vti.RegClass:$merge, vti.RegClass:$merge,
(vti.Scalar vti.ScalarRegClass:$rs1), GPR:$vl, vti.Log2SEW)>; (vti.Scalar vti.ScalarRegClass:$rs1), GPR:$vl, vti.Log2SEW)>;
defvar ivti = GetIntVTypeInfo<vti>.Vti; defvar ivti = GetIntVTypeInfo<vti>.Vti;
def : Pat<(vti.Vector (riscv_vrgather_vv_vl vti.RegClass:$rs2, def : Pat<(vti.Vector (riscv_vrgather_vv_vl (vti.Vector srcvalue),
vti.RegClass:$rs2,
(ivti.Vector vti.RegClass:$rs1), (ivti.Vector vti.RegClass:$rs1),
(vti.Mask true_mask), (vti.Mask true_mask),
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>("PseudoVRGATHER_VV_"# vti.LMul.MX) (!cast<Instruction>("PseudoVRGATHER_VV_"# vti.LMul.MX)
vti.RegClass:$rs2, vti.RegClass:$rs1, GPR:$vl, vti.Log2SEW)>; vti.RegClass:$rs2, vti.RegClass:$rs1, GPR:$vl, vti.Log2SEW)>;
def : Pat<(vti.Vector (riscv_vrgather_vx_vl vti.RegClass:$rs2, GPR:$rs1, def : Pat<(vti.Vector (riscv_vrgather_vx_vl (vti.Vector srcvalue),
vti.RegClass:$rs2, GPR:$rs1,
(vti.Mask true_mask), (vti.Mask true_mask),
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>("PseudoVRGATHER_VX_"# vti.LMul.MX) (!cast<Instruction>("PseudoVRGATHER_VX_"# vti.LMul.MX)
vti.RegClass:$rs2, GPR:$rs1, GPR:$vl, vti.Log2SEW)>; vti.RegClass:$rs2, GPR:$rs1, GPR:$vl, vti.Log2SEW)>;
def : Pat<(vti.Vector (riscv_vrgather_vx_vl vti.RegClass:$rs2, uimm5:$imm, def : Pat<(vti.Vector (riscv_vrgather_vx_vl (vti.Vector srcvalue),
vti.RegClass:$rs2, uimm5:$imm,
(vti.Mask true_mask), (vti.Mask true_mask),
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>("PseudoVRGATHER_VI_"# vti.LMul.MX) (!cast<Instruction>("PseudoVRGATHER_VI_"# vti.LMul.MX)
vti.RegClass:$rs2, uimm5:$imm, GPR:$vl, vti.Log2SEW)>; vti.RegClass:$rs2, uimm5:$imm, GPR:$vl, vti.Log2SEW)>;
def : Pat<(vti.Vector (riscv_vselect_vl (vti.Mask V0), def : Pat<(vti.Vector
(riscv_vrgather_vv_vl (riscv_vrgather_vv_vl vti.RegClass:$merge,
vti.RegClass:$rs2, vti.RegClass:$rs2,
(ivti.Vector vti.RegClass:$rs1), (ivti.Vector vti.RegClass:$rs1),
(vti.Mask true_mask), (vti.Mask V0),
VLOpFrag),
vti.RegClass:$merge,
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>("PseudoVRGATHER_VV_"# vti.LMul.MX#"_MASK") (!cast<Instruction>("PseudoVRGATHER_VV_"# vti.LMul.MX#"_MASK")
vti.RegClass:$merge, vti.RegClass:$rs2, vti.RegClass:$rs1, vti.RegClass:$merge, vti.RegClass:$rs2, vti.RegClass:$rs1,
(vti.Mask V0), GPR:$vl, vti.Log2SEW, TAIL_AGNOSTIC)>; (vti.Mask V0), GPR:$vl, vti.Log2SEW, TAIL_AGNOSTIC)>;
def : Pat<(vti.Vector (riscv_vselect_vl (vti.Mask V0), def : Pat<(vti.Vector
(riscv_vrgather_vx_vl (riscv_vrgather_vx_vl vti.RegClass:$merge,
vti.RegClass:$rs2, vti.RegClass:$rs2,
uimm5:$imm, uimm5:$imm,
(vti.Mask true_mask), (vti.Mask V0),
VLOpFrag),
vti.RegClass:$merge,
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>("PseudoVRGATHER_VI_"# vti.LMul.MX#"_MASK") (!cast<Instruction>("PseudoVRGATHER_VI_"# vti.LMul.MX#"_MASK")
vti.RegClass:$merge, vti.RegClass:$rs2, uimm5:$imm, vti.RegClass:$merge, vti.RegClass:$rs2, uimm5:$imm,
(vti.Mask V0), GPR:$vl, vti.Log2SEW, TAIL_AGNOSTIC)>; (vti.Mask V0), GPR:$vl, vti.Log2SEW, TAIL_AGNOSTIC)>;
defvar vlmul = vti.LMul; defvar vlmul = vti.LMul;
defvar octuple_lmul = vlmul.octuple; defvar octuple_lmul = vlmul.octuple;
defvar octuple_emul = !srl(!mul(octuple_lmul, 16), vti.Log2SEW); defvar octuple_emul = !srl(!mul(octuple_lmul, 16), vti.Log2SEW);
if !and(!ge(octuple_emul, 1), !le(octuple_emul, 64)) then { if !and(!ge(octuple_emul, 1), !le(octuple_emul, 64)) then {
defvar emul_str = octuple_to_str<octuple_emul>.ret; defvar emul_str = octuple_to_str<octuple_emul>.ret;
defvar ivti = !cast<VTypeInfo>("VI16" # emul_str); defvar ivti = !cast<VTypeInfo>("VI16" # emul_str);
defvar inst = "PseudoVRGATHEREI16_VV_" # vti.LMul.MX # "_" # emul_str; defvar inst = "PseudoVRGATHEREI16_VV_" # vti.LMul.MX # "_" # emul_str;
def : Pat<(vti.Vector (riscv_vrgatherei16_vv_vl vti.RegClass:$rs2, def : Pat<(vti.Vector (riscv_vrgatherei16_vv_vl (vti.Vector srcvalue),
vti.RegClass:$rs2,
(ivti.Vector ivti.RegClass:$rs1), (ivti.Vector ivti.RegClass:$rs1),
(vti.Mask true_mask), (vti.Mask true_mask),
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>(inst) (!cast<Instruction>(inst)
vti.RegClass:$rs2, ivti.RegClass:$rs1, GPR:$vl, vti.Log2SEW)>; vti.RegClass:$rs2, ivti.RegClass:$rs1, GPR:$vl, vti.Log2SEW)>;
def : Pat<(vti.Vector (riscv_vselect_vl (vti.Mask V0), def : Pat<(vti.Vector
(riscv_vrgatherei16_vv_vl (riscv_vrgatherei16_vv_vl vti.RegClass:$merge,
vti.RegClass:$rs2, vti.RegClass:$rs2,
(ivti.Vector ivti.RegClass:$rs1), (ivti.Vector ivti.RegClass:$rs1),
(vti.Mask true_mask), (vti.Mask V0),
VLOpFrag),
vti.RegClass:$merge,
VLOpFrag)), VLOpFrag)),
(!cast<Instruction>(inst#"_MASK") (!cast<Instruction>(inst#"_MASK")
vti.RegClass:$merge, vti.RegClass:$rs2, ivti.RegClass:$rs1, vti.RegClass:$merge, vti.RegClass:$rs2, ivti.RegClass:$rs1,
(vti.Mask V0), GPR:$vl, vti.Log2SEW, TAIL_AGNOSTIC)>; (vti.Mask V0), GPR:$vl, vti.Log2SEW, TAIL_AGNOSTIC)>;
} }
} }
} // Predicates = [HasVInstructionsAnyF] } // Predicates = [HasVInstructionsAnyF]
▲ Show 20 Lines • Show All 105 Lines • Show Last 20 Lines