This is an archive of the discontinued LLVM Phabricator instance.

Differential D81850

[AArch64][SVE] NFCI: Choose consistent naming for predicated SDAG nodes
ClosedPublic

Authored by sdesmalen on Jun 15 2020, 9:33 AM.

Details

Reviewers
paulwalker-arm
cameron.mcinally
eli.friedman
dancgr
efriedma
Commits
rG39f6a36a24ce: [AArch64][SVE] NFCI: Choose consistent naming for predicated SDAG nodes
Summary

This patch proposes a naming convention for operations that take
a general predicate (and are thus predicated) that specifies
what happens to the false lanes.

Currently the _PRED suffix is used, which doesn't really say much other
than that it takes a predicate. In some instances this means it has
merging predication and in other cases it means zeroing-predication.

This patch also changes the order of operands to
AArch64ISD::DUP_MERGE_PASSTHRU, to pass the predicate as the first
operand, which is in line with all other predicates nodes. It takes the
passthru value as an explicit passthru value, which is always passed as
the last operand.

Diff Detail

Event Timeline

sdesmalen created this revision.Jun 15 2020, 9:33 AM
Herald added a reviewer: efriedma. · View Herald TranscriptJun 15 2020, 9:33 AM
Herald added a project: Restricted Project. · View Herald Transcript
Herald added subscribers: danielkiss, psnobl, rkruppe and 3 others. · View Herald Transcript
paulwalker-arm added inline comments.Jun 15 2020, 9:54 AM
llvm/lib/Target/AArch64/AArch64ISelLowering.h
32

Nothing really against the names, being short is certainly a benefit over my next suggestion.

Is is worth being explicit when it comes to merging. For example _MERGE_OP1 and _MERGE_ZERO. Here MERGE implies a predicated operation and it's clear from the name where the results for inactive lanes comes from. Here I'm referring to _PRED_PASSTHRU where such things are not clear. In essence I'm suggesting forcing people to clearly state where the passthrough value comes from in all cases.

38

Given the above I'm wondering if this needs a special name, and if _PRED can be used for these cases as well as you describe below. I don't think there's any overlap so it's meaning should be clear.

Harbormaster failed remote builds in B60315: Diff 270778!Jun 15 2020, 10:23 AM
sdesmalen updated this revision to Diff 273756.Jun 26 2020, 9:18 AM

Renamed nodes as follows:

// For predicated nodes where the result is a vector, the operation is
// controlled by a governing predicate and the inactive lanes are explicitly
// defined with a value, please stick the following naming convention:
//    _MERGE_OP0
//    _MERGE_OP1
//         :
//    _MERGE_OP<n>     The result value is a vector with false lanes equal to
//                     source operand OP<n>.
//
//    _MERGE_ZERO      The result value is a vector with false lanes actively
//                     zeroed.
//
// For other cases where no explicit action is needed to set the inactive lanes,
// or when the result is not a vector and it is needed or helpful to
// distinguish a node from similar unpredicated nodes, use:
//
//    _PRED
sdesmalen updated this revision to Diff 273832.Jun 26 2020, 2:08 PM
sdesmalen retitled this revision from [AArch64][SVE] NFC: Choose consistent naming for predicated SDAG nodes to [AArch64][SVE] NFCI: Choose consistent naming for predicated SDAG nodes.
sdesmalen edited the summary of this revision. (Show Details)

Following offline discussion with @paulwalker-arm, I've renamed DUP_MERGE_OP0 to DUP_MERGE_PASSTHRU and changed order of its operands to take the predicate as the first operand, which is more in line with the other predicated nodes. This consistency is useful for lowering to predidcated ops, like done in D82483. This means the patch isn't strictly NFC anymore though.

paulwalker-arm accepted this revision.Jun 27 2020, 1:26 AM
paulwalker-arm added inline comments.
llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
1300

Should these be aligned with the case statements? Although I guess this is just where clang-format wants to put them?

llvm/lib/Target/AArch64/AArch64ISelLowering.h
32

With the DUP change, can you remove _MERGE_OP0 when pushing the patch.

This revision is now accepted and ready to land.Jun 27 2020, 1:26 AM
Closed by commit rG39f6a36a24ce: [AArch64][SVE] NFCI: Choose consistent naming for predicated SDAG nodes (authored by sdesmalen). · Explain WhyJun 29 2020, 5:54 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathSize
llvm/
lib/
Target/
AArch64/
16 lines
150 lines
761 lines
572 lines
2 lines

Diff 270778

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

Show First 20 Lines • Show All 4,622 Lines • ▼ Show 20 Lines if (VT == MVT::v16i8 || VT == MVT::v8i8) {
return; return;
} else if (VT == MVT::v2i64 || VT == MVT::v1i64 || VT == MVT::v2f64 || } else if (VT == MVT::v2i64 || VT == MVT::v1i64 || VT == MVT::v2f64 ||
VT == MVT::v1f64) { VT == MVT::v1f64) {
SelectPostStoreLane(Node, 4, AArch64::ST4i64_POST); SelectPostStoreLane(Node, 4, AArch64::ST4i64_POST);
return; return;
} }
break; break;
} }
case AArch64ISD::SVE_LD2: { case AArch64ISD::SVE_LD2_PRED_Z: {
if (VT == MVT::nxv16i8) { if (VT == MVT::nxv16i8) {
SelectPredicatedLoad(Node, 2, AArch64::LD2B_IMM); SelectPredicatedLoad(Node, 2, AArch64::LD2B_IMM);
return; return;
} else if (VT == MVT::nxv8i16 || VT == MVT::nxv8f16) { } else if (VT == MVT::nxv8i16 || VT == MVT::nxv8f16) {
SelectPredicatedLoad(Node, 2, AArch64::LD2H_IMM); SelectPredicatedLoad(Node, 2, AArch64::LD2H_IMM);
return; return;
} else if (VT == MVT::nxv4i32 || VT == MVT::nxv4f32) { } else if (VT == MVT::nxv4i32 || VT == MVT::nxv4f32) {
SelectPredicatedLoad(Node, 2, AArch64::LD2W_IMM); SelectPredicatedLoad(Node, 2, AArch64::LD2W_IMM);
return; return;
} else if (VT == MVT::nxv2i64 || VT == MVT::nxv2f64) { } else if (VT == MVT::nxv2i64 || VT == MVT::nxv2f64) {
SelectPredicatedLoad(Node, 2, AArch64::LD2D_IMM); SelectPredicatedLoad(Node, 2, AArch64::LD2D_IMM);
return; return;
} }
break; break;
} }
case AArch64ISD::SVE_LD3: { case AArch64ISD::SVE_LD3_PRED_Z: {
if (VT == MVT::nxv16i8) { if (VT == MVT::nxv16i8) {
SelectPredicatedLoad(Node, 3, AArch64::LD3B_IMM); SelectPredicatedLoad(Node, 3, AArch64::LD3B_IMM);
return; return;
} else if (VT == MVT::nxv8i16 || VT == MVT::nxv8f16) { } else if (VT == MVT::nxv8i16 || VT == MVT::nxv8f16) {
SelectPredicatedLoad(Node, 3, AArch64::LD3H_IMM); SelectPredicatedLoad(Node, 3, AArch64::LD3H_IMM);
return; return;
} else if (VT == MVT::nxv4i32 || VT == MVT::nxv4f32) { } else if (VT == MVT::nxv4i32 || VT == MVT::nxv4f32) {
SelectPredicatedLoad(Node, 3, AArch64::LD3W_IMM); SelectPredicatedLoad(Node, 3, AArch64::LD3W_IMM);
return; return;
} else if (VT == MVT::nxv2i64 || VT == MVT::nxv2f64) { } else if (VT == MVT::nxv2i64 || VT == MVT::nxv2f64) {
SelectPredicatedLoad(Node, 3, AArch64::LD3D_IMM); SelectPredicatedLoad(Node, 3, AArch64::LD3D_IMM);
return; return;
} }
break; break;
} }
case AArch64ISD::SVE_LD4: { case AArch64ISD::SVE_LD4_PRED_Z: {
if (VT == MVT::nxv16i8) { if (VT == MVT::nxv16i8) {
SelectPredicatedLoad(Node, 4, AArch64::LD4B_IMM); SelectPredicatedLoad(Node, 4, AArch64::LD4B_IMM);
return; return;
} else if (VT == MVT::nxv8i16 || VT == MVT::nxv8f16) { } else if (VT == MVT::nxv8i16 || VT == MVT::nxv8f16) {
SelectPredicatedLoad(Node, 4, AArch64::LD4H_IMM); SelectPredicatedLoad(Node, 4, AArch64::LD4H_IMM);
return; return;
} else if (VT == MVT::nxv4i32 || VT == MVT::nxv4f32) { } else if (VT == MVT::nxv4i32 || VT == MVT::nxv4f32) {
SelectPredicatedLoad(Node, 4, AArch64::LD4W_IMM); SelectPredicatedLoad(Node, 4, AArch64::LD4W_IMM);
▲ Show 20 Lines • Show All 44 Lines • ▼ Show 20 Linesstatic EVT getMemVTFromNode(LLVMContext &Ctx, SDNode *Root) {
if (isa<MemIntrinsicSDNode>(Root)) if (isa<MemIntrinsicSDNode>(Root))
return cast<MemIntrinsicSDNode>(Root)->getMemoryVT(); return cast<MemIntrinsicSDNode>(Root)->getMemoryVT();
const unsigned Opcode = Root->getOpcode(); const unsigned Opcode = Root->getOpcode();
// For custom ISD nodes, we have to look at them individually to extract the // For custom ISD nodes, we have to look at them individually to extract the
// type of the data moved to/from memory. // type of the data moved to/from memory.
switch (Opcode) { switch (Opcode) {
case AArch64ISD::LD1: case AArch64ISD::LD1_PRED_Z:
case AArch64ISD::LD1S: case AArch64ISD::LD1S_PRED_Z:
case AArch64ISD::LDNF1: case AArch64ISD::LDNF1_PRED_Z:
case AArch64ISD::LDNF1S: case AArch64ISD::LDNF1S_PRED_Z:
return cast<VTSDNode>(Root->getOperand(3))->getVT(); return cast<VTSDNode>(Root->getOperand(3))->getVT();
case AArch64ISD::ST1: case AArch64ISD::ST1_PRED:
return cast<VTSDNode>(Root->getOperand(4))->getVT(); return cast<VTSDNode>(Root->getOperand(4))->getVT();
default: default:
break; break;
} }
if (Opcode != ISD::INTRINSIC_VOID) if (Opcode != ISD::INTRINSIC_VOID)
return EVT(); return EVT();
▲ Show 20 Lines • Show All 80 Lines • Show Last 20 Lines

llvm/lib/Target/AArch64/AArch64ISelLowering.h

Show All 19 Lines
#include "llvm/CodeGen/TargetLowering.h" #include "llvm/CodeGen/TargetLowering.h"
#include "llvm/IR/CallingConv.h" #include "llvm/IR/CallingConv.h"
#include "llvm/IR/Instruction.h" #include "llvm/IR/Instruction.h"
namespace llvm { namespace llvm {
namespace AArch64ISD { namespace AArch64ISD {
// For predicated nodes where the result is a vector and the operation is
// controlled by a governing predicate, please stick the following naming
// convention:
//
// _PRED_M The result value is a vector with false lanes equal to
paulwalker-armUnsubmitted
Not Done
ReplyInline Actions

Nothing really against the names, being short is certainly a benefit over my next suggestion.

Is is worth being explicit when it comes to merging. For example _MERGE_OP1 and _MERGE_ZERO. Here MERGE implies a predicated operation and it's clear from the name where the results for inactive lanes comes from. Here I'm referring to _PRED_PASSTHRU where such things are not clear. In essence I'm suggesting forcing people to clearly state where the passthrough value comes from in all cases.

paulwalker-arm: Nothing really against the names, being short is certainly a benefit over my next suggestion.
paulwalker-armUnsubmitted
Not Done
ReplyInline Actions

With the DUP change, can you remove _MERGE_OP0 when pushing the patch.

paulwalker-arm: With the DUP change, can you remove _MERGE_OP0 when pushing the patch.
// the first source operand.
//
// _PRED_Z The result value is a vector with false lanes actively
// zeroed.
//
// _PRED_X The result value is a vector where values of false lanes
paulwalker-armUnsubmitted
Not Done
ReplyInline Actions

Given the above I'm wondering if this needs a special name, and if _PRED can be used for these cases as well as you describe below. I don't think there's any overlap so it's meaning should be clear.

paulwalker-arm: Given the above I'm wondering if this needs a special name, and if _PRED can be used for these…
// are explicitly left unspecified.
//
// _PRED_PASSTHRU The result value is a vector with false lanes equal
// to a separate passthru operand (passed as the last
// operand).
//
// For other cases where no explicit action is needed on the false lanes
// or when the result is not a vector, and it is needed or helpful to
// distinguish a node from similar unpredicated nodes, use:
//
// _PRED
//
enum NodeType : unsigned { enum NodeType : unsigned {
FIRST_NUMBER = ISD::BUILTIN_OP_END, FIRST_NUMBER = ISD::BUILTIN_OP_END,
WrapperLarge, // 4-instruction MOVZ/MOVK sequence for 64-bit addresses. WrapperLarge, // 4-instruction MOVZ/MOVK sequence for 64-bit addresses.
CALL, // Function call. CALL, // Function call.
// Produces the full sequence of instructions for getting the thread pointer // Produces the full sequence of instructions for getting the thread pointer
// offset of a variable into X0, using the TLSDesc model. // offset of a variable into X0, using the TLSDesc model.
TLSDESC_CALLSEQ, TLSDESC_CALLSEQ,
Show All 12 Linesenum NodeType : unsigned {
// Pointer to the thread's local storage area. Materialised from TPIDR_EL0 on // Pointer to the thread's local storage area. Materialised from TPIDR_EL0 on
// ELF. // ELF.
THREAD_POINTER, THREAD_POINTER,
ADC, ADC,
SBC, // adc, sbc instructions SBC, // adc, sbc instructions
// Arithmetic instructions // Arithmetic instructions
SDIV_PRED, SDIV_PRED_M,
UDIV_PRED, UDIV_PRED_M,
SMIN_PRED, SMIN_PRED_M,
UMIN_PRED, UMIN_PRED_M,
SMAX_PRED, SMAX_PRED_M,
UMAX_PRED, UMAX_PRED_M,
SHL_PRED, SHL_PRED_M,
SRL_PRED, SRL_PRED_M,
SRA_PRED, SRA_PRED_M,
SETCC_PRED, SETCC_PRED_Z,
// Arithmetic instructions which write flags. // Arithmetic instructions which write flags.
ADDS, ADDS,
SUBS, SUBS,
ADCS, ADCS,
SBCS, SBCS,
ANDS, ANDS,
▲ Show 20 Lines • Show All 162 Lines • ▼ Show 20 Linesenum NodeType : unsigned {
FMAXNMV_PRED, FMAXNMV_PRED,
FMINV_PRED, FMINV_PRED,
FMINNMV_PRED, FMINNMV_PRED,
INSR, INSR,
PTEST, PTEST,
PTRUE, PTRUE,
DUP_PRED, DUP_PRED_M,
INDEX_VECTOR, INDEX_VECTOR,
REINTERPRET_CAST, REINTERPRET_CAST,
LD1, LD1_PRED_Z,
LD1S, LD1S_PRED_Z,
LDNF1, LDNF1_PRED_Z,
LDNF1S, LDNF1S_PRED_Z,
LDFF1, LDFF1_PRED_Z,
LDFF1S, LDFF1S_PRED_Z,
LD1RQ, LD1RQ_PRED_Z,
LD1RO, LD1RO_PRED_Z,
// Structured loads. // Structured loads.
SVE_LD2, SVE_LD2_PRED_Z,
SVE_LD3, SVE_LD3_PRED_Z,
SVE_LD4, SVE_LD4_PRED_Z,
// Unsigned gather loads. // Unsigned gather loads.
GLD1, GLD1_PRED_Z,
GLD1_SCALED, GLD1_SCALED_PRED_Z,
GLD1_UXTW, GLD1_UXTW_PRED_Z,
GLD1_SXTW, GLD1_SXTW_PRED_Z,
GLD1_UXTW_SCALED, GLD1_UXTW_SCALED_PRED_Z,
GLD1_SXTW_SCALED, GLD1_SXTW_SCALED_PRED_Z,
GLD1_IMM, GLD1_IMM_PRED_Z,
// Signed gather loads // Signed gather loads
GLD1S, GLD1S_PRED_Z,
GLD1S_SCALED, GLD1S_SCALED_PRED_Z,
GLD1S_UXTW, GLD1S_UXTW_PRED_Z,
GLD1S_SXTW, GLD1S_SXTW_PRED_Z,
GLD1S_UXTW_SCALED, GLD1S_UXTW_SCALED_PRED_Z,
GLD1S_SXTW_SCALED, GLD1S_SXTW_SCALED_PRED_Z,
GLD1S_IMM, GLD1S_IMM_PRED_Z,
// Unsigned gather loads. // Unsigned gather loads.
GLDFF1, GLDFF1_PRED_Z,
GLDFF1_SCALED, GLDFF1_SCALED_PRED_Z,
GLDFF1_UXTW, GLDFF1_UXTW_PRED_Z,
GLDFF1_SXTW, GLDFF1_SXTW_PRED_Z,
GLDFF1_UXTW_SCALED, GLDFF1_UXTW_SCALED_PRED_Z,
GLDFF1_SXTW_SCALED, GLDFF1_SXTW_SCALED_PRED_Z,
GLDFF1_IMM, GLDFF1_IMM_PRED_Z,
// Signed gather loads. // Signed gather loads.
GLDFF1S, GLDFF1S_PRED_Z,
GLDFF1S_SCALED, GLDFF1S_SCALED_PRED_Z,
GLDFF1S_UXTW, GLDFF1S_UXTW_PRED_Z,
GLDFF1S_SXTW, GLDFF1S_SXTW_PRED_Z,
GLDFF1S_UXTW_SCALED, GLDFF1S_UXTW_SCALED_PRED_Z,
GLDFF1S_SXTW_SCALED, GLDFF1S_SXTW_SCALED_PRED_Z,
GLDFF1S_IMM, GLDFF1S_IMM_PRED_Z,
// Non-temporal gather loads // Non-temporal gather loads
GLDNT1, GLDNT1_PRED_Z,
GLDNT1_INDEX, GLDNT1_INDEX_PRED_Z,
GLDNT1S, GLDNT1S_PRED_Z,
ST1, // Contiguous masked store.
ST1_PRED,
// Scatter store // Scatter store
SST1, SST1_PRED,
SST1_SCALED, SST1_SCALED_PRED,
SST1_UXTW, SST1_UXTW_PRED,
SST1_SXTW, SST1_SXTW_PRED,
SST1_UXTW_SCALED, SST1_UXTW_SCALED_PRED,
SST1_SXTW_SCALED, SST1_SXTW_SCALED_PRED,
SST1_IMM, SST1_IMM_PRED,
// Non-temporal scatter store // Non-temporal scatter store
SSTNT1, SSTNT1_PRED,
SSTNT1_INDEX, SSTNT1_INDEX_PRED,
// Strict (exception-raising) floating point comparison // Strict (exception-raising) floating point comparison
STRICT_FCMP = ISD::FIRST_TARGET_STRICTFP_OPCODE, STRICT_FCMP = ISD::FIRST_TARGET_STRICTFP_OPCODE,
STRICT_FCMPE, STRICT_FCMPE,
// NEON Load/Store with post-increment base updates // NEON Load/Store with post-increment base updates
LD2post = ISD::FIRST_TARGET_MEMORY_OPCODE, LD2post = ISD::FIRST_TARGET_MEMORY_OPCODE,
LD3post, LD3post,
▲ Show 20 Lines • Show All 594 Lines • Show Last 20 Lines

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

  • This file is larger than 256 KB, so syntax highlighting is disabled by default.
Show First 20 Lines • Show All 1,282 Lines • ▼ Show 20 Lines
FastISel * FastISel *
AArch64TargetLowering::createFastISel(FunctionLoweringInfo &funcInfo, AArch64TargetLowering::createFastISel(FunctionLoweringInfo &funcInfo,
const TargetLibraryInfo *libInfo) const { const TargetLibraryInfo *libInfo) const {
return AArch64::createFastISel(funcInfo, libInfo); return AArch64::createFastISel(funcInfo, libInfo);
} }
const char *AArch64TargetLowering::getTargetNodeName(unsigned Opcode) const { const char *AArch64TargetLowering::getTargetNodeName(unsigned Opcode) const {
#define STRINGIFY(V) #V
#define CONCAT(x, y) STRINGIFY(x##y)
#define MAKE_CASE(V) \
case V: \
return #V;
switch ((AArch64ISD::NodeType)Opcode) { switch ((AArch64ISD::NodeType)Opcode) {
case AArch64ISD::FIRST_NUMBER: break; case AArch64ISD::FIRST_NUMBER:
case AArch64ISD::CALL: return "AArch64ISD::CALL"; break;
case AArch64ISD::ADRP: return "AArch64ISD::ADRP"; MAKE_CASE(AArch64ISD::CALL)
paulwalker-armUnsubmitted
Not Done
ReplyInline Actions

Should these be aligned with the case statements? Although I guess this is just where clang-format wants to put them?

paulwalker-arm: Should these be aligned with the case statements? Although I guess this is just where clang…
case AArch64ISD::ADR: return "AArch64ISD::ADR"; MAKE_CASE(AArch64ISD::ADRP)
case AArch64ISD::ADDlow: return "AArch64ISD::ADDlow"; MAKE_CASE(AArch64ISD::ADR)
case AArch64ISD::LOADgot: return "AArch64ISD::LOADgot"; MAKE_CASE(AArch64ISD::ADDlow)
case AArch64ISD::RET_FLAG: return "AArch64ISD::RET_FLAG"; MAKE_CASE(AArch64ISD::LOADgot)
case AArch64ISD::BRCOND: return "AArch64ISD::BRCOND"; MAKE_CASE(AArch64ISD::RET_FLAG)
case AArch64ISD::CSEL: return "AArch64ISD::CSEL"; MAKE_CASE(AArch64ISD::BRCOND)
case AArch64ISD::FCSEL: return "AArch64ISD::FCSEL"; MAKE_CASE(AArch64ISD::CSEL)
case AArch64ISD::CSINV: return "AArch64ISD::CSINV"; MAKE_CASE(AArch64ISD::FCSEL)
case AArch64ISD::CSNEG: return "AArch64ISD::CSNEG"; MAKE_CASE(AArch64ISD::CSINV)
case AArch64ISD::CSINC: return "AArch64ISD::CSINC"; MAKE_CASE(AArch64ISD::CSNEG)
case AArch64ISD::THREAD_POINTER: return "AArch64ISD::THREAD_POINTER"; MAKE_CASE(AArch64ISD::CSINC)
case AArch64ISD::TLSDESC_CALLSEQ: return "AArch64ISD::TLSDESC_CALLSEQ"; MAKE_CASE(AArch64ISD::THREAD_POINTER)
case AArch64ISD::SDIV_PRED: return "AArch64ISD::SDIV_PRED"; MAKE_CASE(AArch64ISD::TLSDESC_CALLSEQ)
case AArch64ISD::UDIV_PRED: return "AArch64ISD::UDIV_PRED"; MAKE_CASE(AArch64ISD::SDIV_PRED_M)
case AArch64ISD::SMIN_PRED: return "AArch64ISD::SMIN_PRED"; MAKE_CASE(AArch64ISD::UDIV_PRED_M)
case AArch64ISD::UMIN_PRED: return "AArch64ISD::UMIN_PRED"; MAKE_CASE(AArch64ISD::SMIN_PRED_M)
case AArch64ISD::SMAX_PRED: return "AArch64ISD::SMAX_PRED"; MAKE_CASE(AArch64ISD::UMIN_PRED_M)
case AArch64ISD::UMAX_PRED: return "AArch64ISD::UMAX_PRED"; MAKE_CASE(AArch64ISD::SMAX_PRED_M)
case AArch64ISD::SHL_PRED: return "AArch64ISD::SHL_PRED"; MAKE_CASE(AArch64ISD::UMAX_PRED_M)
case AArch64ISD::SRL_PRED: return "AArch64ISD::SRL_PRED"; MAKE_CASE(AArch64ISD::SHL_PRED_M)
case AArch64ISD::SRA_PRED: return "AArch64ISD::SRA_PRED"; MAKE_CASE(AArch64ISD::SRL_PRED_M)
case AArch64ISD::SETCC_PRED: return "AArch64ISD::SETCC_PRED"; MAKE_CASE(AArch64ISD::SRA_PRED_M)
case AArch64ISD::ADC: return "AArch64ISD::ADC"; MAKE_CASE(AArch64ISD::SETCC_PRED_Z)
case AArch64ISD::SBC: return "AArch64ISD::SBC"; MAKE_CASE(AArch64ISD::ADC)
case AArch64ISD::ADDS: return "AArch64ISD::ADDS"; MAKE_CASE(AArch64ISD::SBC)
case AArch64ISD::SUBS: return "AArch64ISD::SUBS"; MAKE_CASE(AArch64ISD::ADDS)
case AArch64ISD::ADCS: return "AArch64ISD::ADCS"; MAKE_CASE(AArch64ISD::SUBS)
case AArch64ISD::SBCS: return "AArch64ISD::SBCS"; MAKE_CASE(AArch64ISD::ADCS)
case AArch64ISD::ANDS: return "AArch64ISD::ANDS"; MAKE_CASE(AArch64ISD::SBCS)
case AArch64ISD::CCMP: return "AArch64ISD::CCMP"; MAKE_CASE(AArch64ISD::ANDS)
case AArch64ISD::CCMN: return "AArch64ISD::CCMN"; MAKE_CASE(AArch64ISD::CCMP)
case AArch64ISD::FCCMP: return "AArch64ISD::FCCMP"; MAKE_CASE(AArch64ISD::CCMN)
case AArch64ISD::FCMP: return "AArch64ISD::FCMP"; MAKE_CASE(AArch64ISD::FCCMP)
case AArch64ISD::STRICT_FCMP: return "AArch64ISD::STRICT_FCMP"; MAKE_CASE(AArch64ISD::FCMP)
case AArch64ISD::STRICT_FCMPE: return "AArch64ISD::STRICT_FCMPE"; MAKE_CASE(AArch64ISD::STRICT_FCMP)
case AArch64ISD::DUP: return "AArch64ISD::DUP"; MAKE_CASE(AArch64ISD::STRICT_FCMPE)
case AArch64ISD::DUPLANE8: return "AArch64ISD::DUPLANE8"; MAKE_CASE(AArch64ISD::DUP)
case AArch64ISD::DUPLANE16: return "AArch64ISD::DUPLANE16"; MAKE_CASE(AArch64ISD::DUPLANE8)
case AArch64ISD::DUPLANE32: return "AArch64ISD::DUPLANE32"; MAKE_CASE(AArch64ISD::DUPLANE16)
case AArch64ISD::DUPLANE64: return "AArch64ISD::DUPLANE64"; MAKE_CASE(AArch64ISD::DUPLANE32)
case AArch64ISD::MOVI: return "AArch64ISD::MOVI"; MAKE_CASE(AArch64ISD::DUPLANE64)
case AArch64ISD::MOVIshift: return "AArch64ISD::MOVIshift"; MAKE_CASE(AArch64ISD::MOVI)
case AArch64ISD::MOVIedit: return "AArch64ISD::MOVIedit"; MAKE_CASE(AArch64ISD::MOVIshift)
case AArch64ISD::MOVImsl: return "AArch64ISD::MOVImsl"; MAKE_CASE(AArch64ISD::MOVIedit)
case AArch64ISD::FMOV: return "AArch64ISD::FMOV"; MAKE_CASE(AArch64ISD::MOVImsl)
case AArch64ISD::MVNIshift: return "AArch64ISD::MVNIshift"; MAKE_CASE(AArch64ISD::FMOV)
case AArch64ISD::MVNImsl: return "AArch64ISD::MVNImsl"; MAKE_CASE(AArch64ISD::MVNIshift)
case AArch64ISD::BICi: return "AArch64ISD::BICi"; MAKE_CASE(AArch64ISD::MVNImsl)
case AArch64ISD::ORRi: return "AArch64ISD::ORRi"; MAKE_CASE(AArch64ISD::BICi)
case AArch64ISD::BSP: return "AArch64ISD::BSP"; MAKE_CASE(AArch64ISD::ORRi)
case AArch64ISD::NEG: return "AArch64ISD::NEG"; MAKE_CASE(AArch64ISD::BSP)
case AArch64ISD::EXTR: return "AArch64ISD::EXTR"; MAKE_CASE(AArch64ISD::NEG)
case AArch64ISD::ZIP1: return "AArch64ISD::ZIP1"; MAKE_CASE(AArch64ISD::EXTR)
case AArch64ISD::ZIP2: return "AArch64ISD::ZIP2"; MAKE_CASE(AArch64ISD::ZIP1)
case AArch64ISD::UZP1: return "AArch64ISD::UZP1"; MAKE_CASE(AArch64ISD::ZIP2)
case AArch64ISD::UZP2: return "AArch64ISD::UZP2"; MAKE_CASE(AArch64ISD::UZP1)
case AArch64ISD::TRN1: return "AArch64ISD::TRN1"; MAKE_CASE(AArch64ISD::UZP2)
case AArch64ISD::TRN2: return "AArch64ISD::TRN2"; MAKE_CASE(AArch64ISD::TRN1)
case AArch64ISD::REV16: return "AArch64ISD::REV16"; MAKE_CASE(AArch64ISD::TRN2)
case AArch64ISD::REV32: return "AArch64ISD::REV32"; MAKE_CASE(AArch64ISD::REV16)
case AArch64ISD::REV64: return "AArch64ISD::REV64"; MAKE_CASE(AArch64ISD::REV32)
case AArch64ISD::EXT: return "AArch64ISD::EXT"; MAKE_CASE(AArch64ISD::REV64)
case AArch64ISD::VSHL: return "AArch64ISD::VSHL"; MAKE_CASE(AArch64ISD::EXT)
case AArch64ISD::VLSHR: return "AArch64ISD::VLSHR"; MAKE_CASE(AArch64ISD::VSHL)
case AArch64ISD::VASHR: return "AArch64ISD::VASHR"; MAKE_CASE(AArch64ISD::VLSHR)
case AArch64ISD::VSLI: return "AArch64ISD::VSLI"; MAKE_CASE(AArch64ISD::VASHR)
case AArch64ISD::VSRI: return "AArch64ISD::VSRI"; MAKE_CASE(AArch64ISD::VSLI)
case AArch64ISD::CMEQ: return "AArch64ISD::CMEQ"; MAKE_CASE(AArch64ISD::VSRI)
case AArch64ISD::CMGE: return "AArch64ISD::CMGE"; MAKE_CASE(AArch64ISD::CMEQ)
case AArch64ISD::CMGT: return "AArch64ISD::CMGT"; MAKE_CASE(AArch64ISD::CMGE)
case AArch64ISD::CMHI: return "AArch64ISD::CMHI"; MAKE_CASE(AArch64ISD::CMGT)
case AArch64ISD::CMHS: return "AArch64ISD::CMHS"; MAKE_CASE(AArch64ISD::CMHI)
case AArch64ISD::FCMEQ: return "AArch64ISD::FCMEQ"; MAKE_CASE(AArch64ISD::CMHS)
case AArch64ISD::FCMGE: return "AArch64ISD::FCMGE"; MAKE_CASE(AArch64ISD::FCMEQ)
case AArch64ISD::FCMGT: return "AArch64ISD::FCMGT"; MAKE_CASE(AArch64ISD::FCMGE)
case AArch64ISD::CMEQz: return "AArch64ISD::CMEQz"; MAKE_CASE(AArch64ISD::FCMGT)
case AArch64ISD::CMGEz: return "AArch64ISD::CMGEz"; MAKE_CASE(AArch64ISD::CMEQz)
case AArch64ISD::CMGTz: return "AArch64ISD::CMGTz"; MAKE_CASE(AArch64ISD::CMGEz)
case AArch64ISD::CMLEz: return "AArch64ISD::CMLEz"; MAKE_CASE(AArch64ISD::CMGTz)
case AArch64ISD::CMLTz: return "AArch64ISD::CMLTz"; MAKE_CASE(AArch64ISD::CMLEz)
case AArch64ISD::FCMEQz: return "AArch64ISD::FCMEQz"; MAKE_CASE(AArch64ISD::CMLTz)
case AArch64ISD::FCMGEz: return "AArch64ISD::FCMGEz"; MAKE_CASE(AArch64ISD::FCMEQz)
case AArch64ISD::FCMGTz: return "AArch64ISD::FCMGTz"; MAKE_CASE(AArch64ISD::FCMGEz)
case AArch64ISD::FCMLEz: return "AArch64ISD::FCMLEz"; MAKE_CASE(AArch64ISD::FCMGTz)
case AArch64ISD::FCMLTz: return "AArch64ISD::FCMLTz"; MAKE_CASE(AArch64ISD::FCMLEz)
case AArch64ISD::SADDV: return "AArch64ISD::SADDV"; MAKE_CASE(AArch64ISD::FCMLTz)
case AArch64ISD::UADDV: return "AArch64ISD::UADDV"; MAKE_CASE(AArch64ISD::SADDV)
case AArch64ISD::SMINV: return "AArch64ISD::SMINV"; MAKE_CASE(AArch64ISD::UADDV)
case AArch64ISD::UMINV: return "AArch64ISD::UMINV"; MAKE_CASE(AArch64ISD::SMINV)
case AArch64ISD::SMAXV: return "AArch64ISD::SMAXV"; MAKE_CASE(AArch64ISD::UMINV)
case AArch64ISD::UMAXV: return "AArch64ISD::UMAXV"; MAKE_CASE(AArch64ISD::SMAXV)
case AArch64ISD::SMAXV_PRED: return "AArch64ISD::SMAXV_PRED"; MAKE_CASE(AArch64ISD::UMAXV)
case AArch64ISD::UMAXV_PRED: return "AArch64ISD::UMAXV_PRED"; MAKE_CASE(AArch64ISD::SMAXV_PRED)
case AArch64ISD::SMINV_PRED: return "AArch64ISD::SMINV_PRED"; MAKE_CASE(AArch64ISD::UMAXV_PRED)
case AArch64ISD::UMINV_PRED: return "AArch64ISD::UMINV_PRED"; MAKE_CASE(AArch64ISD::SMINV_PRED)
case AArch64ISD::ORV_PRED: return "AArch64ISD::ORV_PRED"; MAKE_CASE(AArch64ISD::UMINV_PRED)
case AArch64ISD::EORV_PRED: return "AArch64ISD::EORV_PRED"; MAKE_CASE(AArch64ISD::ORV_PRED)
case AArch64ISD::ANDV_PRED: return "AArch64ISD::ANDV_PRED"; MAKE_CASE(AArch64ISD::EORV_PRED)
case AArch64ISD::CLASTA_N: return "AArch64ISD::CLASTA_N"; MAKE_CASE(AArch64ISD::ANDV_PRED)
case AArch64ISD::CLASTB_N: return "AArch64ISD::CLASTB_N"; MAKE_CASE(AArch64ISD::CLASTA_N)
case AArch64ISD::LASTA: return "AArch64ISD::LASTA"; MAKE_CASE(AArch64ISD::CLASTB_N)
case AArch64ISD::LASTB: return "AArch64ISD::LASTB"; MAKE_CASE(AArch64ISD::LASTA)
case AArch64ISD::REV: return "AArch64ISD::REV"; MAKE_CASE(AArch64ISD::LASTB)
case AArch64ISD::REINTERPRET_CAST: return "AArch64ISD::REINTERPRET_CAST"; MAKE_CASE(AArch64ISD::REV)
case AArch64ISD::TBL: return "AArch64ISD::TBL"; MAKE_CASE(AArch64ISD::REINTERPRET_CAST)
case AArch64ISD::FADDA_PRED: return "AArch64ISD::FADDA_PRED"; MAKE_CASE(AArch64ISD::TBL)
case AArch64ISD::FADDV_PRED: return "AArch64ISD::FADDV_PRED"; MAKE_CASE(AArch64ISD::FADDA_PRED)
case AArch64ISD::FMAXV_PRED: return "AArch64ISD::FMAXV_PRED"; MAKE_CASE(AArch64ISD::FADDV_PRED)
case AArch64ISD::FMAXNMV_PRED: return "AArch64ISD::FMAXNMV_PRED"; MAKE_CASE(AArch64ISD::FMAXV_PRED)
case AArch64ISD::FMINV_PRED: return "AArch64ISD::FMINV_PRED"; MAKE_CASE(AArch64ISD::FMAXNMV_PRED)
case AArch64ISD::FMINNMV_PRED: return "AArch64ISD::FMINNMV_PRED"; MAKE_CASE(AArch64ISD::FMINV_PRED)
case AArch64ISD::NOT: return "AArch64ISD::NOT"; MAKE_CASE(AArch64ISD::FMINNMV_PRED)
case AArch64ISD::BIT: return "AArch64ISD::BIT"; MAKE_CASE(AArch64ISD::NOT)
case AArch64ISD::CBZ: return "AArch64ISD::CBZ"; MAKE_CASE(AArch64ISD::BIT)
case AArch64ISD::CBNZ: return "AArch64ISD::CBNZ"; MAKE_CASE(AArch64ISD::CBZ)
case AArch64ISD::TBZ: return "AArch64ISD::TBZ"; MAKE_CASE(AArch64ISD::CBNZ)
case AArch64ISD::TBNZ: return "AArch64ISD::TBNZ"; MAKE_CASE(AArch64ISD::TBZ)
case AArch64ISD::TC_RETURN: return "AArch64ISD::TC_RETURN"; MAKE_CASE(AArch64ISD::TBNZ)
case AArch64ISD::PREFETCH: return "AArch64ISD::PREFETCH"; MAKE_CASE(AArch64ISD::TC_RETURN)
case AArch64ISD::SITOF: return "AArch64ISD::SITOF"; MAKE_CASE(AArch64ISD::PREFETCH)
case AArch64ISD::UITOF: return "AArch64ISD::UITOF"; MAKE_CASE(AArch64ISD::SITOF)
case AArch64ISD::NVCAST: return "AArch64ISD::NVCAST"; MAKE_CASE(AArch64ISD::UITOF)
case AArch64ISD::SQSHL_I: return "AArch64ISD::SQSHL_I"; MAKE_CASE(AArch64ISD::NVCAST)
case AArch64ISD::UQSHL_I: return "AArch64ISD::UQSHL_I"; MAKE_CASE(AArch64ISD::SQSHL_I)
case AArch64ISD::SRSHR_I: return "AArch64ISD::SRSHR_I"; MAKE_CASE(AArch64ISD::UQSHL_I)
case AArch64ISD::URSHR_I: return "AArch64ISD::URSHR_I"; MAKE_CASE(AArch64ISD::SRSHR_I)
case AArch64ISD::SQSHLU_I: return "AArch64ISD::SQSHLU_I"; MAKE_CASE(AArch64ISD::URSHR_I)
case AArch64ISD::WrapperLarge: return "AArch64ISD::WrapperLarge"; MAKE_CASE(AArch64ISD::SQSHLU_I)
case AArch64ISD::LD2post: return "AArch64ISD::LD2post"; MAKE_CASE(AArch64ISD::WrapperLarge)
case AArch64ISD::LD3post: return "AArch64ISD::LD3post"; MAKE_CASE(AArch64ISD::LD2post)
case AArch64ISD::LD4post: return "AArch64ISD::LD4post"; MAKE_CASE(AArch64ISD::LD3post)
case AArch64ISD::ST2post: return "AArch64ISD::ST2post"; MAKE_CASE(AArch64ISD::LD4post)
case AArch64ISD::ST3post: return "AArch64ISD::ST3post"; MAKE_CASE(AArch64ISD::ST2post)
case AArch64ISD::ST4post: return "AArch64ISD::ST4post"; MAKE_CASE(AArch64ISD::ST3post)
case AArch64ISD::LD1x2post: return "AArch64ISD::LD1x2post"; MAKE_CASE(AArch64ISD::ST4post)
case AArch64ISD::LD1x3post: return "AArch64ISD::LD1x3post"; MAKE_CASE(AArch64ISD::LD1x2post)
case AArch64ISD::LD1x4post: return "AArch64ISD::LD1x4post"; MAKE_CASE(AArch64ISD::LD1x3post)
case AArch64ISD::ST1x2post: return "AArch64ISD::ST1x2post"; MAKE_CASE(AArch64ISD::LD1x4post)
case AArch64ISD::ST1x3post: return "AArch64ISD::ST1x3post"; MAKE_CASE(AArch64ISD::ST1x2post)
case AArch64ISD::ST1x4post: return "AArch64ISD::ST1x4post"; MAKE_CASE(AArch64ISD::ST1x3post)
case AArch64ISD::LD1DUPpost: return "AArch64ISD::LD1DUPpost"; MAKE_CASE(AArch64ISD::ST1x4post)
case AArch64ISD::LD2DUPpost: return "AArch64ISD::LD2DUPpost"; MAKE_CASE(AArch64ISD::LD1DUPpost)
case AArch64ISD::LD3DUPpost: return "AArch64ISD::LD3DUPpost"; MAKE_CASE(AArch64ISD::LD2DUPpost)
case AArch64ISD::LD4DUPpost: return "AArch64ISD::LD4DUPpost"; MAKE_CASE(AArch64ISD::LD3DUPpost)
case AArch64ISD::LD1LANEpost: return "AArch64ISD::LD1LANEpost"; MAKE_CASE(AArch64ISD::LD4DUPpost)
case AArch64ISD::LD2LANEpost: return "AArch64ISD::LD2LANEpost"; MAKE_CASE(AArch64ISD::LD1LANEpost)
case AArch64ISD::LD3LANEpost: return "AArch64ISD::LD3LANEpost"; MAKE_CASE(AArch64ISD::LD2LANEpost)
case AArch64ISD::LD4LANEpost: return "AArch64ISD::LD4LANEpost"; MAKE_CASE(AArch64ISD::LD3LANEpost)
case AArch64ISD::ST2LANEpost: return "AArch64ISD::ST2LANEpost"; MAKE_CASE(AArch64ISD::LD4LANEpost)
case AArch64ISD::ST3LANEpost: return "AArch64ISD::ST3LANEpost"; MAKE_CASE(AArch64ISD::ST2LANEpost)
case AArch64ISD::ST4LANEpost: return "AArch64ISD::ST4LANEpost"; MAKE_CASE(AArch64ISD::ST3LANEpost)
case AArch64ISD::SMULL: return "AArch64ISD::SMULL"; MAKE_CASE(AArch64ISD::ST4LANEpost)
case AArch64ISD::UMULL: return "AArch64ISD::UMULL"; MAKE_CASE(AArch64ISD::SMULL)
case AArch64ISD::FRECPE: return "AArch64ISD::FRECPE"; MAKE_CASE(AArch64ISD::UMULL)
case AArch64ISD::FRECPS: return "AArch64ISD::FRECPS"; MAKE_CASE(AArch64ISD::FRECPE)
case AArch64ISD::FRSQRTE: return "AArch64ISD::FRSQRTE"; MAKE_CASE(AArch64ISD::FRECPS)
case AArch64ISD::FRSQRTS: return "AArch64ISD::FRSQRTS"; MAKE_CASE(AArch64ISD::FRSQRTE)
case AArch64ISD::STG: return "AArch64ISD::STG"; MAKE_CASE(AArch64ISD::FRSQRTS)
case AArch64ISD::STZG: return "AArch64ISD::STZG"; MAKE_CASE(AArch64ISD::STG)
case AArch64ISD::ST2G: return "AArch64ISD::ST2G"; MAKE_CASE(AArch64ISD::STZG)
case AArch64ISD::STZ2G: return "AArch64ISD::STZ2G"; MAKE_CASE(AArch64ISD::ST2G)
case AArch64ISD::SUNPKHI: return "AArch64ISD::SUNPKHI"; MAKE_CASE(AArch64ISD::STZ2G)
case AArch64ISD::SUNPKLO: return "AArch64ISD::SUNPKLO"; MAKE_CASE(AArch64ISD::SUNPKHI)
case AArch64ISD::UUNPKHI: return "AArch64ISD::UUNPKHI"; MAKE_CASE(AArch64ISD::SUNPKLO)
case AArch64ISD::UUNPKLO: return "AArch64ISD::UUNPKLO"; MAKE_CASE(AArch64ISD::UUNPKHI)
case AArch64ISD::INSR: return "AArch64ISD::INSR"; MAKE_CASE(AArch64ISD::UUNPKLO)
case AArch64ISD::PTEST: return "AArch64ISD::PTEST"; MAKE_CASE(AArch64ISD::INSR)
case AArch64ISD::PTRUE: return "AArch64ISD::PTRUE"; MAKE_CASE(AArch64ISD::PTEST)
case AArch64ISD::LD1: return "AArch64ISD::LD1"; MAKE_CASE(AArch64ISD::PTRUE)
case AArch64ISD::LD1S: return "AArch64ISD::LD1S"; MAKE_CASE(AArch64ISD::LD1_PRED_Z)
case AArch64ISD::LDNF1: return "AArch64ISD::LDNF1"; MAKE_CASE(AArch64ISD::LD1S_PRED_Z)
case AArch64ISD::LDNF1S: return "AArch64ISD::LDNF1S"; MAKE_CASE(AArch64ISD::LDNF1_PRED_Z)
case AArch64ISD::LDFF1: return "AArch64ISD::LDFF1"; MAKE_CASE(AArch64ISD::LDNF1S_PRED_Z)
case AArch64ISD::LDFF1S: return "AArch64ISD::LDFF1S"; MAKE_CASE(AArch64ISD::LDFF1_PRED_Z)
case AArch64ISD::LD1RQ: return "AArch64ISD::LD1RQ"; MAKE_CASE(AArch64ISD::LDFF1S_PRED_Z)
case AArch64ISD::LD1RO: return "AArch64ISD::LD1RO"; MAKE_CASE(AArch64ISD::LD1RQ_PRED_Z)
case AArch64ISD::SVE_LD2: return "AArch64ISD::SVE_LD2"; MAKE_CASE(AArch64ISD::LD1RO_PRED_Z)
case AArch64ISD::SVE_LD3: return "AArch64ISD::SVE_LD3"; MAKE_CASE(AArch64ISD::SVE_LD2_PRED_Z)
case AArch64ISD::SVE_LD4: return "AArch64ISD::SVE_LD4"; MAKE_CASE(AArch64ISD::SVE_LD3_PRED_Z)
case AArch64ISD::GLD1: return "AArch64ISD::GLD1"; MAKE_CASE(AArch64ISD::SVE_LD4_PRED_Z)
case AArch64ISD::GLD1_SCALED: return "AArch64ISD::GLD1_SCALED"; MAKE_CASE(AArch64ISD::GLD1_PRED_Z)
case AArch64ISD::GLD1_SXTW: return "AArch64ISD::GLD1_SXTW"; MAKE_CASE(AArch64ISD::GLD1_SCALED_PRED_Z)
case AArch64ISD::GLD1_UXTW: return "AArch64ISD::GLD1_UXTW"; MAKE_CASE(AArch64ISD::GLD1_SXTW_PRED_Z)
case AArch64ISD::GLD1_SXTW_SCALED: return "AArch64ISD::GLD1_SXTW_SCALED"; MAKE_CASE(AArch64ISD::GLD1_UXTW_PRED_Z)
case AArch64ISD::GLD1_UXTW_SCALED: return "AArch64ISD::GLD1_UXTW_SCALED"; MAKE_CASE(AArch64ISD::GLD1_SXTW_SCALED_PRED_Z)
case AArch64ISD::GLD1_IMM: return "AArch64ISD::GLD1_IMM"; MAKE_CASE(AArch64ISD::GLD1_UXTW_SCALED_PRED_Z)
case AArch64ISD::GLD1S: return "AArch64ISD::GLD1S"; MAKE_CASE(AArch64ISD::GLD1_IMM_PRED_Z)
case AArch64ISD::GLD1S_SCALED: return "AArch64ISD::GLD1S_SCALED"; MAKE_CASE(AArch64ISD::GLD1S_PRED_Z)
case AArch64ISD::GLD1S_SXTW: return "AArch64ISD::GLD1S_SXTW"; MAKE_CASE(AArch64ISD::GLD1S_SCALED_PRED_Z)
case AArch64ISD::GLD1S_UXTW: return "AArch64ISD::GLD1S_UXTW"; MAKE_CASE(AArch64ISD::GLD1S_SXTW_PRED_Z)
case AArch64ISD::GLD1S_SXTW_SCALED: return "AArch64ISD::GLD1S_SXTW_SCALED"; MAKE_CASE(AArch64ISD::GLD1S_UXTW_PRED_Z)
case AArch64ISD::GLD1S_UXTW_SCALED: return "AArch64ISD::GLD1S_UXTW_SCALED"; MAKE_CASE(AArch64ISD::GLD1S_SXTW_SCALED_PRED_Z)
case AArch64ISD::GLD1S_IMM: return "AArch64ISD::GLD1S_IMM"; MAKE_CASE(AArch64ISD::GLD1S_UXTW_SCALED_PRED_Z)
case AArch64ISD::GLDFF1: return "AArch64ISD::GLDFF1"; MAKE_CASE(AArch64ISD::GLD1S_IMM_PRED_Z)
case AArch64ISD::GLDFF1_SCALED: return "AArch64ISD::GLDFF1_SCALED"; MAKE_CASE(AArch64ISD::GLDFF1_PRED_Z)
case AArch64ISD::GLDFF1_SXTW: return "AArch64ISD::GLDFF1_SXTW"; MAKE_CASE(AArch64ISD::GLDFF1_SCALED_PRED_Z)
case AArch64ISD::GLDFF1_UXTW: return "AArch64ISD::GLDFF1_UXTW"; MAKE_CASE(AArch64ISD::GLDFF1_SXTW_PRED_Z)
case AArch64ISD::GLDFF1_SXTW_SCALED:return "AArch64ISD::GLDFF1_SXTW_SCALED"; MAKE_CASE(AArch64ISD::GLDFF1_UXTW_PRED_Z)
case AArch64ISD::GLDFF1_UXTW_SCALED:return "AArch64ISD::GLDFF1_UXTW_SCALED"; MAKE_CASE(AArch64ISD::GLDFF1_SXTW_SCALED_PRED_Z)
case AArch64ISD::GLDFF1_IMM: return "AArch64ISD::GLDFF1_IMM"; MAKE_CASE(AArch64ISD::GLDFF1_UXTW_SCALED_PRED_Z)
case AArch64ISD::GLDFF1S: return "AArch64ISD::GLDFF1S"; MAKE_CASE(AArch64ISD::GLDFF1_IMM_PRED_Z)
case AArch64ISD::GLDFF1S_SCALED: return "AArch64ISD::GLDFF1S_SCALED"; MAKE_CASE(AArch64ISD::GLDFF1S_PRED_Z)
case AArch64ISD::GLDFF1S_SXTW: return "AArch64ISD::GLDFF1S_SXTW"; MAKE_CASE(AArch64ISD::GLDFF1S_SCALED_PRED_Z)
case AArch64ISD::GLDFF1S_UXTW: return "AArch64ISD::GLDFF1S_UXTW"; MAKE_CASE(AArch64ISD::GLDFF1S_SXTW_PRED_Z)
case AArch64ISD::GLDFF1S_SXTW_SCALED: MAKE_CASE(AArch64ISD::GLDFF1S_UXTW_PRED_Z)
return "AArch64ISD::GLDFF1S_SXTW_SCALED"; MAKE_CASE(AArch64ISD::GLDFF1S_SXTW_SCALED_PRED_Z)
case AArch64ISD::GLDFF1S_UXTW_SCALED: MAKE_CASE(AArch64ISD::GLDFF1S_UXTW_SCALED_PRED_Z)
return "AArch64ISD::GLDFF1S_UXTW_SCALED"; MAKE_CASE(AArch64ISD::GLDFF1S_IMM_PRED_Z)
case AArch64ISD::GLDFF1S_IMM: return "AArch64ISD::GLDFF1S_IMM"; MAKE_CASE(AArch64ISD::GLDNT1_PRED_Z)
MAKE_CASE(AArch64ISD::GLDNT1_INDEX_PRED_Z)
case AArch64ISD::GLDNT1: return "AArch64ISD::GLDNT1"; MAKE_CASE(AArch64ISD::GLDNT1S_PRED_Z)
case AArch64ISD::GLDNT1_INDEX: return "AArch64ISD::GLDNT1_INDEX"; MAKE_CASE(AArch64ISD::ST1_PRED)
case AArch64ISD::GLDNT1S: return "AArch64ISD::GLDNT1S"; MAKE_CASE(AArch64ISD::SST1_PRED)
MAKE_CASE(AArch64ISD::SST1_SCALED_PRED)
case AArch64ISD::ST1: return "AArch64ISD::ST1"; MAKE_CASE(AArch64ISD::SST1_SXTW_PRED)
MAKE_CASE(AArch64ISD::SST1_UXTW_PRED)
case AArch64ISD::SST1: return "AArch64ISD::SST1"; MAKE_CASE(AArch64ISD::SST1_SXTW_SCALED_PRED)
case AArch64ISD::SST1_SCALED: return "AArch64ISD::SST1_SCALED"; MAKE_CASE(AArch64ISD::SST1_UXTW_SCALED_PRED)
case AArch64ISD::SST1_SXTW: return "AArch64ISD::SST1_SXTW"; MAKE_CASE(AArch64ISD::SST1_IMM_PRED)
case AArch64ISD::SST1_UXTW: return "AArch64ISD::SST1_UXTW"; MAKE_CASE(AArch64ISD::SSTNT1_PRED)
case AArch64ISD::SST1_SXTW_SCALED: return "AArch64ISD::SST1_SXTW_SCALED"; MAKE_CASE(AArch64ISD::SSTNT1_INDEX_PRED)
case AArch64ISD::SST1_UXTW_SCALED: return "AArch64ISD::SST1_UXTW_SCALED"; MAKE_CASE(AArch64ISD::LDP)
case AArch64ISD::SST1_IMM: return "AArch64ISD::SST1_IMM"; MAKE_CASE(AArch64ISD::STP)
MAKE_CASE(AArch64ISD::STNP)
case AArch64ISD::SSTNT1: return "AArch64ISD::SSTNT1"; MAKE_CASE(AArch64ISD::DUP_PRED_M)
case AArch64ISD::SSTNT1_INDEX: return "AArch64ISD::SSTNT1_INDEX"; MAKE_CASE(AArch64ISD::INDEX_VECTOR)
}
case AArch64ISD::LDP: return "AArch64ISD::LDP"; #undef STRINGIFY
case AArch64ISD::STP: return "AArch64ISD::STP"; #undef CONCAT
case AArch64ISD::STNP: return "AArch64ISD::STNP"; #undef MAKE_CASE
case AArch64ISD::DUP_PRED: return "AArch64ISD::DUP_PRED";
case AArch64ISD::INDEX_VECTOR: return "AArch64ISD::INDEX_VECTOR";
}
return nullptr; return nullptr;
} }
MachineBasicBlock * MachineBasicBlock *
AArch64TargetLowering::EmitF128CSEL(MachineInstr &MI, AArch64TargetLowering::EmitF128CSEL(MachineInstr &MI,
MachineBasicBlock *MBB) const { MachineBasicBlock *MBB) const {
// We materialise the F128CSEL pseudo-instruction as some control flow and a // We materialise the F128CSEL pseudo-instruction as some control flow and a
// phi node: // phi node:
▲ Show 20 Lines • Show All 1,864 Lines • ▼ Show 20 Lines case ISD::BUILD_VECTOR:
return LowerBUILD_VECTOR(Op, DAG); return LowerBUILD_VECTOR(Op, DAG);
case ISD::VECTOR_SHUFFLE: case ISD::VECTOR_SHUFFLE:
return LowerVECTOR_SHUFFLE(Op, DAG); return LowerVECTOR_SHUFFLE(Op, DAG);
case ISD::SPLAT_VECTOR: case ISD::SPLAT_VECTOR:
return LowerSPLAT_VECTOR(Op, DAG); return LowerSPLAT_VECTOR(Op, DAG);
case ISD::EXTRACT_SUBVECTOR: case ISD::EXTRACT_SUBVECTOR:
return LowerEXTRACT_SUBVECTOR(Op, DAG); return LowerEXTRACT_SUBVECTOR(Op, DAG);
case ISD::SDIV: case ISD::SDIV:
return LowerToPredicatedOp(Op, DAG, AArch64ISD::SDIV_PRED); return LowerToPredicatedOp(Op, DAG, AArch64ISD::SDIV_PRED_M);
case ISD::UDIV: case ISD::UDIV:
return LowerToPredicatedOp(Op, DAG, AArch64ISD::UDIV_PRED); return LowerToPredicatedOp(Op, DAG, AArch64ISD::UDIV_PRED_M);
case ISD::SMIN: case ISD::SMIN:
return LowerToPredicatedOp(Op, DAG, AArch64ISD::SMIN_PRED); return LowerToPredicatedOp(Op, DAG, AArch64ISD::SMIN_PRED_M);
case ISD::UMIN: case ISD::UMIN:
return LowerToPredicatedOp(Op, DAG, AArch64ISD::UMIN_PRED); return LowerToPredicatedOp(Op, DAG, AArch64ISD::UMIN_PRED_M);
case ISD::SMAX: case ISD::SMAX:
return LowerToPredicatedOp(Op, DAG, AArch64ISD::SMAX_PRED); return LowerToPredicatedOp(Op, DAG, AArch64ISD::SMAX_PRED_M);
case ISD::UMAX: case ISD::UMAX:
return LowerToPredicatedOp(Op, DAG, AArch64ISD::UMAX_PRED); return LowerToPredicatedOp(Op, DAG, AArch64ISD::UMAX_PRED_M);
case ISD::SRA: case ISD::SRA:
case ISD::SRL: case ISD::SRL:
case ISD::SHL: case ISD::SHL:
return LowerVectorSRA_SRL_SHL(Op, DAG); return LowerVectorSRA_SRL_SHL(Op, DAG);
case ISD::SHL_PARTS: case ISD::SHL_PARTS:
return LowerShiftLeftParts(Op, DAG); return LowerShiftLeftParts(Op, DAG);
case ISD::SRL_PARTS: case ISD::SRL_PARTS:
case ISD::SRA_PARTS: case ISD::SRA_PARTS:
▲ Show 20 Lines • Show All 5,268 Lines • ▼ Show 20 LinesSDValue AArch64TargetLowering::LowerVectorSRA_SRL_SHL(SDValue Op,
unsigned EltSize = VT.getScalarSizeInBits(); unsigned EltSize = VT.getScalarSizeInBits();
switch (Op.getOpcode()) { switch (Op.getOpcode()) {
default: default:
llvm_unreachable("unexpected shift opcode"); llvm_unreachable("unexpected shift opcode");
case ISD::SHL: case ISD::SHL:
if (VT.isScalableVector()) if (VT.isScalableVector())
return LowerToPredicatedOp(Op, DAG, AArch64ISD::SHL_PRED); return LowerToPredicatedOp(Op, DAG, AArch64ISD::SHL_PRED_M);
if (isVShiftLImm(Op.getOperand(1), VT, false, Cnt) && Cnt < EltSize) if (isVShiftLImm(Op.getOperand(1), VT, false, Cnt) && Cnt < EltSize)
return DAG.getNode(AArch64ISD::VSHL, DL, VT, Op.getOperand(0), return DAG.getNode(AArch64ISD::VSHL, DL, VT, Op.getOperand(0),
DAG.getConstant(Cnt, DL, MVT::i32)); DAG.getConstant(Cnt, DL, MVT::i32));
return DAG.getNode(ISD::INTRINSIC_WO_CHAIN, DL, VT, return DAG.getNode(ISD::INTRINSIC_WO_CHAIN, DL, VT,
DAG.getConstant(Intrinsic::aarch64_neon_ushl, DL, DAG.getConstant(Intrinsic::aarch64_neon_ushl, DL,
MVT::i32), MVT::i32),
Op.getOperand(0), Op.getOperand(1)); Op.getOperand(0), Op.getOperand(1));
case ISD::SRA: case ISD::SRA:
case ISD::SRL: case ISD::SRL:
if (VT.isScalableVector()) { if (VT.isScalableVector()) {
unsigned Opc = Op.getOpcode() == ISD::SRA ? AArch64ISD::SRA_PRED unsigned Opc = Op.getOpcode() == ISD::SRA ? AArch64ISD::SRA_PRED_M
: AArch64ISD::SRL_PRED; : AArch64ISD::SRL_PRED_M;
return LowerToPredicatedOp(Op, DAG, Opc); return LowerToPredicatedOp(Op, DAG, Opc);
} }
// Right shift immediate // Right shift immediate
if (isVShiftRImm(Op.getOperand(1), VT, false, Cnt) && Cnt < EltSize) { if (isVShiftRImm(Op.getOperand(1), VT, false, Cnt) && Cnt < EltSize) {
unsigned Opc = unsigned Opc =
(Op.getOpcode() == ISD::SRA) ? AArch64ISD::VASHR : AArch64ISD::VLSHR; (Op.getOpcode() == ISD::SRA) ? AArch64ISD::VASHR : AArch64ISD::VLSHR;
return DAG.getNode(Opc, DL, VT, Op.getOperand(0), return DAG.getNode(Opc, DL, VT, Op.getOperand(0),
▲ Show 20 Lines • Show All 112 Lines • ▼ Show 20 Linesstatic SDValue EmitVectorComparison(SDValue LHS, SDValue RHS,
} }
} }
SDValue AArch64TargetLowering::LowerVSETCC(SDValue Op, SDValue AArch64TargetLowering::LowerVSETCC(SDValue Op,
SelectionDAG &DAG) const { SelectionDAG &DAG) const {
if (Op.getValueType().isScalableVector()) { if (Op.getValueType().isScalableVector()) {
if (Op.getOperand(0).getValueType().isFloatingPoint()) if (Op.getOperand(0).getValueType().isFloatingPoint())
return Op; return Op;
return LowerToPredicatedOp(Op, DAG, AArch64ISD::SETCC_PRED); return LowerToPredicatedOp(Op, DAG, AArch64ISD::SETCC_PRED_Z);
} }
ISD::CondCode CC = cast<CondCodeSDNode>(Op.getOperand(2))->get(); ISD::CondCode CC = cast<CondCodeSDNode>(Op.getOperand(2))->get();
SDValue LHS = Op.getOperand(0); SDValue LHS = Op.getOperand(0);
SDValue RHS = Op.getOperand(1); SDValue RHS = Op.getOperand(1);
EVT CmpVT = LHS.getValueType().changeVectorElementTypeToInteger(); EVT CmpVT = LHS.getValueType().changeVectorElementTypeToInteger();
SDLoc dl(Op); SDLoc dl(Op);
▲ Show 20 Lines • Show All 978 Lines • ▼ Show 20 Lines
SDValue AArch64TargetLowering::LowerSVEStructLoad(unsigned Intrinsic, SDValue AArch64TargetLowering::LowerSVEStructLoad(unsigned Intrinsic,
ArrayRef<SDValue> LoadOps, ArrayRef<SDValue> LoadOps,
EVT VT, SelectionDAG &DAG, EVT VT, SelectionDAG &DAG,
const SDLoc &DL) const { const SDLoc &DL) const {
assert(VT.isScalableVector() && "Can only lower scalable vectors"); assert(VT.isScalableVector() && "Can only lower scalable vectors");
unsigned N, Opcode; unsigned N, Opcode;
static std::map<unsigned, std::pair<unsigned, unsigned>> IntrinsicMap = { static std::map<unsigned, std::pair<unsigned, unsigned>> IntrinsicMap = {
{Intrinsic::aarch64_sve_ld2, {2, AArch64ISD::SVE_LD2}}, {Intrinsic::aarch64_sve_ld2, {2, AArch64ISD::SVE_LD2_PRED_Z}},
{Intrinsic::aarch64_sve_ld3, {3, AArch64ISD::SVE_LD3}}, {Intrinsic::aarch64_sve_ld3, {3, AArch64ISD::SVE_LD3_PRED_Z}},
{Intrinsic::aarch64_sve_ld4, {4, AArch64ISD::SVE_LD4}}}; {Intrinsic::aarch64_sve_ld4, {4, AArch64ISD::SVE_LD4_PRED_Z}}};
std::tie(N, Opcode) = IntrinsicMap[Intrinsic]; std::tie(N, Opcode) = IntrinsicMap[Intrinsic];
assert(VT.getVectorElementCount().Min % N == 0 && assert(VT.getVectorElementCount().Min % N == 0 &&
"invalid tuple vector type!"); "invalid tuple vector type!");
EVT SplitVT = EVT::getVectorVT(*DAG.getContext(), VT.getVectorElementType(), EVT SplitVT = EVT::getVectorVT(*DAG.getContext(), VT.getVectorElementType(),
VT.getVectorElementCount() / N); VT.getVectorElementCount() / N);
assert(isTypeLegal(SplitVT)); assert(isTypeLegal(SplitVT));
▲ Show 20 Lines • Show All 924 Lines • ▼ Show 20 Linesstatic SDValue performSVEAndCombine(SDNode *N,
if (!Src.hasOneUse()) if (!Src.hasOneUse())
return SDValue(); return SDValue();
EVT MemVT; EVT MemVT;
// SVE load instructions perform an implicit zero-extend, which makes them // SVE load instructions perform an implicit zero-extend, which makes them
// perfect candidates for combining. // perfect candidates for combining.
switch (Opc) { switch (Opc) {
case AArch64ISD::LD1: case AArch64ISD::LD1_PRED_Z:
case AArch64ISD::LDNF1: case AArch64ISD::LDNF1_PRED_Z:
case AArch64ISD::LDFF1: case AArch64ISD::LDFF1_PRED_Z:
MemVT = cast<VTSDNode>(Src->getOperand(3))->getVT(); MemVT = cast<VTSDNode>(Src->getOperand(3))->getVT();
break; break;
case AArch64ISD::GLD1: case AArch64ISD::GLD1_PRED_Z:
case AArch64ISD::GLD1_SCALED: case AArch64ISD::GLD1_SCALED_PRED_Z:
case AArch64ISD::GLD1_SXTW: case AArch64ISD::GLD1_SXTW_PRED_Z:
case AArch64ISD::GLD1_SXTW_SCALED: case AArch64ISD::GLD1_SXTW_SCALED_PRED_Z:
case AArch64ISD::GLD1_UXTW: case AArch64ISD::GLD1_UXTW_PRED_Z:
case AArch64ISD::GLD1_UXTW_SCALED: case AArch64ISD::GLD1_UXTW_SCALED_PRED_Z:
case AArch64ISD::GLD1_IMM: case AArch64ISD::GLD1_IMM_PRED_Z:
case AArch64ISD::GLDFF1: case AArch64ISD::GLDFF1_PRED_Z:
case AArch64ISD::GLDFF1_SCALED: case AArch64ISD::GLDFF1_SCALED_PRED_Z:
case AArch64ISD::GLDFF1_SXTW: case AArch64ISD::GLDFF1_SXTW_PRED_Z:
case AArch64ISD::GLDFF1_SXTW_SCALED: case AArch64ISD::GLDFF1_SXTW_SCALED_PRED_Z:
case AArch64ISD::GLDFF1_UXTW: case AArch64ISD::GLDFF1_UXTW_PRED_Z:
case AArch64ISD::GLDFF1_UXTW_SCALED: case AArch64ISD::GLDFF1_UXTW_SCALED_PRED_Z:
case AArch64ISD::GLDFF1_IMM: case AArch64ISD::GLDFF1_IMM_PRED_Z:
case AArch64ISD::GLDNT1: case AArch64ISD::GLDNT1_PRED_Z:
MemVT = cast<VTSDNode>(Src->getOperand(4))->getVT(); MemVT = cast<VTSDNode>(Src->getOperand(4))->getVT();
break; break;
default: default:
return SDValue(); return SDValue();
} }
if (isConstantSplatVectorMaskForType(Mask.getNode(), MemVT)) if (isConstantSplatVectorMaskForType(Mask.getNode(), MemVT))
return Src; return Src;
▲ Show 20 Lines • Show All 617 Lines • ▼ Show 20 Lines
static SDValue LowerSVEIntrinsicDUP(SDNode *N, SelectionDAG &DAG) { static SDValue LowerSVEIntrinsicDUP(SDNode *N, SelectionDAG &DAG) {
SDLoc dl(N); SDLoc dl(N);
SDValue Scalar = N->getOperand(3); SDValue Scalar = N->getOperand(3);
EVT ScalarTy = Scalar.getValueType(); EVT ScalarTy = Scalar.getValueType();
if ((ScalarTy == MVT::i8) || (ScalarTy == MVT::i16)) if ((ScalarTy == MVT::i8) || (ScalarTy == MVT::i16))
Scalar = DAG.getNode(ISD::ANY_EXTEND, dl, MVT::i32, Scalar); Scalar = DAG.getNode(ISD::ANY_EXTEND, dl, MVT::i32, Scalar);
return DAG.getNode(AArch64ISD::DUP_PRED, dl, N->getValueType(0), return DAG.getNode(AArch64ISD::DUP_PRED_M, dl, N->getValueType(0),
N->getOperand(1), N->getOperand(2), Scalar); N->getOperand(1), N->getOperand(2), Scalar);
} }
static SDValue LowerSVEIntrinsicEXT(SDNode *N, SelectionDAG &DAG) { static SDValue LowerSVEIntrinsicEXT(SDNode *N, SelectionDAG &DAG) {
SDLoc dl(N); SDLoc dl(N);
LLVMContext &Ctx = *DAG.getContext(); LLVMContext &Ctx = *DAG.getContext();
EVT VT = N->getValueType(0); EVT VT = N->getValueType(0);
▲ Show 20 Lines • Show All 69 Lines • ▼ Show 20 Lines case Intrinsic::aarch64_sve_cmpls_wide: {
break; break;
} }
} }
if (!Imm) if (!Imm)
return SDValue(); return SDValue();
SDValue Splat = DAG.getNode(ISD::SPLAT_VECTOR, DL, CmpVT, Imm); SDValue Splat = DAG.getNode(ISD::SPLAT_VECTOR, DL, CmpVT, Imm);
return DAG.getNode(AArch64ISD::SETCC_PRED, DL, VT, Pred, N->getOperand(2), return DAG.getNode(AArch64ISD::SETCC_PRED_Z, DL, VT, Pred, N->getOperand(2),
Splat, DAG.getCondCode(CC)); Splat, DAG.getCondCode(CC));
} }
return SDValue(); return SDValue();
} }
static SDValue getPTest(SelectionDAG &DAG, EVT VT, SDValue Pg, SDValue Op, static SDValue getPTest(SelectionDAG &DAG, EVT VT, SDValue Pg, SDValue Op,
AArch64CC::CondCode Cond) { AArch64CC::CondCode Cond) {
▲ Show 20 Lines • Show All 132 Lines • ▼ Show 20 Linesstatic SDValue performIntrinsicCombine(SDNode *N,
case Intrinsic::aarch64_sve_dup: case Intrinsic::aarch64_sve_dup:
return LowerSVEIntrinsicDUP(N, DAG); return LowerSVEIntrinsicDUP(N, DAG);
case Intrinsic::aarch64_sve_dup_x: case Intrinsic::aarch64_sve_dup_x:
return DAG.getNode(ISD::SPLAT_VECTOR, SDLoc(N), N->getValueType(0), return DAG.getNode(ISD::SPLAT_VECTOR, SDLoc(N), N->getValueType(0),
N->getOperand(1)); N->getOperand(1));
case Intrinsic::aarch64_sve_ext: case Intrinsic::aarch64_sve_ext:
return LowerSVEIntrinsicEXT(N, DAG); return LowerSVEIntrinsicEXT(N, DAG);
case Intrinsic::aarch64_sve_sdiv: case Intrinsic::aarch64_sve_sdiv:
return DAG.getNode(AArch64ISD::SDIV_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::SDIV_PRED_M, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3)); N->getOperand(1), N->getOperand(2), N->getOperand(3));
case Intrinsic::aarch64_sve_udiv: case Intrinsic::aarch64_sve_udiv:
return DAG.getNode(AArch64ISD::UDIV_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::UDIV_PRED_M, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3)); N->getOperand(1), N->getOperand(2), N->getOperand(3));
case Intrinsic::aarch64_sve_smin: case Intrinsic::aarch64_sve_smin:
return DAG.getNode(AArch64ISD::SMIN_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::SMIN_PRED_M, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3)); N->getOperand(1), N->getOperand(2), N->getOperand(3));
case Intrinsic::aarch64_sve_umin: case Intrinsic::aarch64_sve_umin:
return DAG.getNode(AArch64ISD::UMIN_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::UMIN_PRED_M, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3)); N->getOperand(1), N->getOperand(2), N->getOperand(3));
case Intrinsic::aarch64_sve_smax: case Intrinsic::aarch64_sve_smax:
return DAG.getNode(AArch64ISD::SMAX_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::SMAX_PRED_M, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3)); N->getOperand(1), N->getOperand(2), N->getOperand(3));
case Intrinsic::aarch64_sve_umax: case Intrinsic::aarch64_sve_umax:
return DAG.getNode(AArch64ISD::UMAX_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::UMAX_PRED_M, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3)); N->getOperand(1), N->getOperand(2), N->getOperand(3));
case Intrinsic::aarch64_sve_lsl: case Intrinsic::aarch64_sve_lsl:
return DAG.getNode(AArch64ISD::SHL_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::SHL_PRED_M, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3)); N->getOperand(1), N->getOperand(2), N->getOperand(3));
case Intrinsic::aarch64_sve_lsr: case Intrinsic::aarch64_sve_lsr:
return DAG.getNode(AArch64ISD::SRL_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::SRL_PRED_M, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3)); N->getOperand(1), N->getOperand(2), N->getOperand(3));
case Intrinsic::aarch64_sve_asr: case Intrinsic::aarch64_sve_asr:
return DAG.getNode(AArch64ISD::SRA_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::SRA_PRED_M, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3)); N->getOperand(1), N->getOperand(2), N->getOperand(3));
case Intrinsic::aarch64_sve_cmphs: case Intrinsic::aarch64_sve_cmphs:
if (!N->getOperand(2).getValueType().isFloatingPoint()) if (!N->getOperand(2).getValueType().isFloatingPoint())
return DAG.getNode(AArch64ISD::SETCC_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::SETCC_PRED_Z, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3), N->getOperand(1), N->getOperand(2), N->getOperand(3),
DAG.getCondCode(ISD::SETUGE)); DAG.getCondCode(ISD::SETUGE));
break; break;
case Intrinsic::aarch64_sve_cmphi: case Intrinsic::aarch64_sve_cmphi:
if (!N->getOperand(2).getValueType().isFloatingPoint()) if (!N->getOperand(2).getValueType().isFloatingPoint())
return DAG.getNode(AArch64ISD::SETCC_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::SETCC_PRED_Z, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3), N->getOperand(1), N->getOperand(2), N->getOperand(3),
DAG.getCondCode(ISD::SETUGT)); DAG.getCondCode(ISD::SETUGT));
break; break;
case Intrinsic::aarch64_sve_cmpge: case Intrinsic::aarch64_sve_cmpge:
if (!N->getOperand(2).getValueType().isFloatingPoint()) if (!N->getOperand(2).getValueType().isFloatingPoint())
return DAG.getNode(AArch64ISD::SETCC_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::SETCC_PRED_Z, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3), N->getOperand(1), N->getOperand(2), N->getOperand(3),
DAG.getCondCode(ISD::SETGE)); DAG.getCondCode(ISD::SETGE));
break; break;
case Intrinsic::aarch64_sve_cmpgt: case Intrinsic::aarch64_sve_cmpgt:
if (!N->getOperand(2).getValueType().isFloatingPoint()) if (!N->getOperand(2).getValueType().isFloatingPoint())
return DAG.getNode(AArch64ISD::SETCC_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::SETCC_PRED_Z, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3), N->getOperand(1), N->getOperand(2), N->getOperand(3),
DAG.getCondCode(ISD::SETGT)); DAG.getCondCode(ISD::SETGT));
break; break;
case Intrinsic::aarch64_sve_cmpeq: case Intrinsic::aarch64_sve_cmpeq:
if (!N->getOperand(2).getValueType().isFloatingPoint()) if (!N->getOperand(2).getValueType().isFloatingPoint())
return DAG.getNode(AArch64ISD::SETCC_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::SETCC_PRED_Z, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3), N->getOperand(1), N->getOperand(2), N->getOperand(3),
DAG.getCondCode(ISD::SETEQ)); DAG.getCondCode(ISD::SETEQ));
break; break;
case Intrinsic::aarch64_sve_cmpne: case Intrinsic::aarch64_sve_cmpne:
if (!N->getOperand(2).getValueType().isFloatingPoint()) if (!N->getOperand(2).getValueType().isFloatingPoint())
return DAG.getNode(AArch64ISD::SETCC_PRED, SDLoc(N), N->getValueType(0), return DAG.getNode(AArch64ISD::SETCC_PRED_Z, SDLoc(N), N->getValueType(0),
N->getOperand(1), N->getOperand(2), N->getOperand(3), N->getOperand(1), N->getOperand(2), N->getOperand(3),
DAG.getCondCode(ISD::SETNE)); DAG.getCondCode(ISD::SETNE));
break; break;
case Intrinsic::aarch64_sve_fadda: case Intrinsic::aarch64_sve_fadda:
return combineSVEReductionOrderedFP(N, AArch64ISD::FADDA_PRED, DAG); return combineSVEReductionOrderedFP(N, AArch64ISD::FADDA_PRED, DAG);
case Intrinsic::aarch64_sve_faddv: case Intrinsic::aarch64_sve_faddv:
return combineSVEReductionFP(N, AArch64ISD::FADDV_PRED, DAG); return combineSVEReductionFP(N, AArch64ISD::FADDV_PRED, DAG);
case Intrinsic::aarch64_sve_fmaxnmv: case Intrinsic::aarch64_sve_fmaxnmv:
▲ Show 20 Lines • Show All 255 Lines • ▼ Show 20 Lines if (VT.isFloatingPoint()) {
return DAG.getMergeValues(Ops, DL); return DAG.getMergeValues(Ops, DL);
} }
return L; return L;
} }
template <unsigned Opcode> template <unsigned Opcode>
static SDValue performLD1ReplicateCombine(SDNode *N, SelectionDAG &DAG) { static SDValue performLD1ReplicateCombine(SDNode *N, SelectionDAG &DAG) {
static_assert(Opcode == AArch64ISD::LD1RQ || Opcode == AArch64ISD::LD1RO, static_assert(Opcode == AArch64ISD::LD1RQ_PRED_Z ||
Opcode == AArch64ISD::LD1RO_PRED_Z,
"Unsupported opcode."); "Unsupported opcode.");
SDLoc DL(N); SDLoc DL(N);
EVT VT = N->getValueType(0); EVT VT = N->getValueType(0);
EVT LoadVT = VT; EVT LoadVT = VT;
if (VT.isFloatingPoint()) if (VT.isFloatingPoint())
LoadVT = VT.changeTypeToInteger(); LoadVT = VT.changeTypeToInteger();
Show All 25 Linesstatic SDValue performST1Combine(SDNode *N, SelectionDAG &DAG) {
SDValue Ops[] = { N->getOperand(0), // Chain SDValue Ops[] = { N->getOperand(0), // Chain
SrcNew, SrcNew,
N->getOperand(4), // Base N->getOperand(4), // Base
N->getOperand(3), // Pg N->getOperand(3), // Pg
InputVT InputVT
}; };
return DAG.getNode(AArch64ISD::ST1, DL, N->getValueType(0), Ops); return DAG.getNode(AArch64ISD::ST1_PRED, DL, N->getValueType(0), Ops);
} }
static SDValue performSTNT1Combine(SDNode *N, SelectionDAG &DAG) { static SDValue performSTNT1Combine(SDNode *N, SelectionDAG &DAG) {
SDLoc DL(N); SDLoc DL(N);
SDValue Data = N->getOperand(2); SDValue Data = N->getOperand(2);
EVT DataVT = Data.getValueType(); EVT DataVT = Data.getValueType();
EVT PtrTy = N->getOperand(4).getValueType(); EVT PtrTy = N->getOperand(4).getValueType();
▲ Show 20 Lines • Show All 1,137 Lines • ▼ Show 20 Linesstatic SDValue performScatterStoreCombine(SDNode *N, SelectionDAG &DAG,
SDValue Base = N->getOperand(4); SDValue Base = N->getOperand(4);
// Depending on the addressing mode, this is either a single offset or a // Depending on the addressing mode, this is either a single offset or a
// vector of offsets (that fits into one register) // vector of offsets (that fits into one register)
SDValue Offset = N->getOperand(5); SDValue Offset = N->getOperand(5);
// For "scalar + vector of indices", just scale the indices. This only // For "scalar + vector of indices", just scale the indices. This only
// applies to non-temporal scatters because there's no instruction that takes // applies to non-temporal scatters because there's no instruction that takes
// indicies. // indicies.
if (Opcode == AArch64ISD::SSTNT1_INDEX) { if (Opcode == AArch64ISD::SSTNT1_INDEX_PRED) {
Offset = Offset =
getScaledOffsetForBitWidth(DAG, Offset, DL, SrcElVT.getSizeInBits()); getScaledOffsetForBitWidth(DAG, Offset, DL, SrcElVT.getSizeInBits());
Opcode = AArch64ISD::SSTNT1; Opcode = AArch64ISD::SSTNT1_PRED;
} }
// In the case of non-temporal gather loads there's only one SVE instruction // In the case of non-temporal gather loads there's only one SVE instruction
// per data-size: "scalar + vector", i.e. // per data-size: "scalar + vector", i.e.
// * stnt1{b|h|w|d} { z0.s }, p0/z, [z0.s, x0] // * stnt1{b|h|w|d} { z0.s }, p0/z, [z0.s, x0]
// Since we do have intrinsics that allow the arguments to be in a different // Since we do have intrinsics that allow the arguments to be in a different
// order, we may need to swap them to match the spec. // order, we may need to swap them to match the spec.
if (Opcode == AArch64ISD::SSTNT1 && Offset.getValueType().isVector()) if (Opcode == AArch64ISD::SSTNT1_PRED && Offset.getValueType().isVector())
std::swap(Base, Offset); std::swap(Base, Offset);
// SST1_IMM requires that the offset is an immediate that is: // SST1_IMM requires that the offset is an immediate that is:
// * a multiple of #SizeInBytes, // * a multiple of #SizeInBytes,
// * in the range [0, 31 x #SizeInBytes], // * in the range [0, 31 x #SizeInBytes],
// where #SizeInBytes is the size in bytes of the stored items. For // where #SizeInBytes is the size in bytes of the stored items. For
// immediates outside that range and non-immediate scalar offsets use SST1 or // immediates outside that range and non-immediate scalar offsets use SST1 or
// SST1_UXTW instead. // SST1_UXTW instead.
if (Opcode == AArch64ISD::SST1_IMM) { if (Opcode == AArch64ISD::SST1_IMM_PRED) {
if (!isValidImmForSVEVecImmAddrMode(Offset, if (!isValidImmForSVEVecImmAddrMode(Offset,
SrcVT.getScalarSizeInBits() / 8)) { SrcVT.getScalarSizeInBits() / 8)) {
if (MVT::nxv4i32 == Base.getValueType().getSimpleVT().SimpleTy) if (MVT::nxv4i32 == Base.getValueType().getSimpleVT().SimpleTy)
Opcode = AArch64ISD::SST1_UXTW; Opcode = AArch64ISD::SST1_UXTW_PRED;
else else
Opcode = AArch64ISD::SST1; Opcode = AArch64ISD::SST1_PRED;
std::swap(Base, Offset); std::swap(Base, Offset);
} }
} }
auto &TLI = DAG.getTargetLoweringInfo(); auto &TLI = DAG.getTargetLoweringInfo();
if (!TLI.isTypeLegal(Base.getValueType())) if (!TLI.isTypeLegal(Base.getValueType()))
return SDValue(); return SDValue();
▲ Show 20 Lines • Show All 54 Lines • ▼ Show 20 Linesstatic SDValue performGatherLoadCombine(SDNode *N, SelectionDAG &DAG,
SDValue Base = N->getOperand(3); SDValue Base = N->getOperand(3);
// Depending on the addressing mode, this is either a single offset or a // Depending on the addressing mode, this is either a single offset or a
// vector of offsets (that fits into one register) // vector of offsets (that fits into one register)
SDValue Offset = N->getOperand(4); SDValue Offset = N->getOperand(4);
// For "scalar + vector of indices", just scale the indices. This only // For "scalar + vector of indices", just scale the indices. This only
// applies to non-temporal gathers because there's no instruction that takes // applies to non-temporal gathers because there's no instruction that takes
// indicies. // indicies.
if (Opcode == AArch64ISD::GLDNT1_INDEX) { if (Opcode == AArch64ISD::GLDNT1_INDEX_PRED_Z) {
Offset = getScaledOffsetForBitWidth(DAG, Offset, DL, Offset = getScaledOffsetForBitWidth(DAG, Offset, DL,
RetVT.getScalarSizeInBits()); RetVT.getScalarSizeInBits());
Opcode = AArch64ISD::GLDNT1; Opcode = AArch64ISD::GLDNT1_PRED_Z;
} }
// In the case of non-temporal gather loads there's only one SVE instruction // In the case of non-temporal gather loads there's only one SVE instruction
// per data-size: "scalar + vector", i.e. // per data-size: "scalar + vector", i.e.
// * ldnt1{b|h|w|d} { z0.s }, p0/z, [z0.s, x0] // * ldnt1{b|h|w|d} { z0.s }, p0/z, [z0.s, x0]
// Since we do have intrinsics that allow the arguments to be in a different // Since we do have intrinsics that allow the arguments to be in a different
// order, we may need to swap them to match the spec. // order, we may need to swap them to match the spec.
if (Opcode == AArch64ISD::GLDNT1 && Offset.getValueType().isVector()) if (Opcode == AArch64ISD::GLDNT1_PRED_Z && Offset.getValueType().isVector())
std::swap(Base, Offset); std::swap(Base, Offset);
// GLD{FF}1_IMM requires that the offset is an immediate that is: // GLD{FF}1_IMM requires that the offset is an immediate that is:
// * a multiple of #SizeInBytes, // * a multiple of #SizeInBytes,
// * in the range [0, 31 x #SizeInBytes], // * in the range [0, 31 x #SizeInBytes],
// where #SizeInBytes is the size in bytes of the loaded items. For // where #SizeInBytes is the size in bytes of the loaded items. For
// immediates outside that range and non-immediate scalar offsets use GLD1 or // immediates outside that range and non-immediate scalar offsets use
// GLD1_UXTW instead. // GLD1_PRED_Z or GLD1_UXTW_PRED_Z instead.
if (Opcode == AArch64ISD::GLD1_IMM || Opcode == AArch64ISD::GLDFF1_IMM) { if (Opcode == AArch64ISD::GLD1_IMM_PRED_Z ||
Opcode == AArch64ISD::GLDFF1_IMM_PRED_Z) {
if (!isValidImmForSVEVecImmAddrMode(Offset, if (!isValidImmForSVEVecImmAddrMode(Offset,
RetVT.getScalarSizeInBits() / 8)) { RetVT.getScalarSizeInBits() / 8)) {
if (MVT::nxv4i32 == Base.getValueType().getSimpleVT().SimpleTy) if (MVT::nxv4i32 == Base.getValueType().getSimpleVT().SimpleTy)
Opcode = (Opcode == AArch64ISD::GLD1_IMM) ? AArch64ISD::GLD1_UXTW Opcode = (Opcode == AArch64ISD::GLD1_IMM_PRED_Z)
: AArch64ISD::GLDFF1_UXTW; ? AArch64ISD::GLD1_UXTW_PRED_Z
: AArch64ISD::GLDFF1_UXTW_PRED_Z;
else else
Opcode = (Opcode == AArch64ISD::GLD1_IMM) ? AArch64ISD::GLD1 Opcode = (Opcode == AArch64ISD::GLD1_IMM_PRED_Z)
: AArch64ISD::GLDFF1; ? AArch64ISD::GLD1_PRED_Z
: AArch64ISD::GLDFF1_PRED_Z;
std::swap(Base, Offset); std::swap(Base, Offset);
} }
} }
auto &TLI = DAG.getTargetLoweringInfo(); auto &TLI = DAG.getTargetLoweringInfo();
if (!TLI.isTypeLegal(Base.getValueType())) if (!TLI.isTypeLegal(Base.getValueType()))
return SDValue(); return SDValue();
▲ Show 20 Lines • Show All 76 Lines • ▼ Show 20 Lines if (Opc == AArch64ISD::UUNPKHI || Opc == AArch64ISD::UUNPKLO) {
return DAG.getNode(SOpc, DL, N->getValueType(0), Ext); return DAG.getNode(SOpc, DL, N->getValueType(0), Ext);
} }
// SVE load nodes (e.g. AArch64ISD::GLD1) are straightforward candidates // SVE load nodes (e.g. AArch64ISD::GLD1) are straightforward candidates
// for DAG Combine with SIGN_EXTEND_INREG. Bail out for all other nodes. // for DAG Combine with SIGN_EXTEND_INREG. Bail out for all other nodes.
unsigned NewOpc; unsigned NewOpc;
unsigned MemVTOpNum = 4; unsigned MemVTOpNum = 4;
switch (Opc) { switch (Opc) {
case AArch64ISD::LD1: case AArch64ISD::LD1_PRED_Z:
NewOpc = AArch64ISD::LD1S; NewOpc = AArch64ISD::LD1S_PRED_Z;
MemVTOpNum = 3; MemVTOpNum = 3;
break; break;
case AArch64ISD::LDNF1: case AArch64ISD::LDNF1_PRED_Z:
NewOpc = AArch64ISD::LDNF1S; NewOpc = AArch64ISD::LDNF1S_PRED_Z;
MemVTOpNum = 3; MemVTOpNum = 3;
break; break;
case AArch64ISD::LDFF1: case AArch64ISD::LDFF1_PRED_Z:
NewOpc = AArch64ISD::LDFF1S; NewOpc = AArch64ISD::LDFF1S_PRED_Z;
MemVTOpNum = 3; MemVTOpNum = 3;
break; break;
case AArch64ISD::GLD1: case AArch64ISD::GLD1_PRED_Z:
NewOpc = AArch64ISD::GLD1S; NewOpc = AArch64ISD::GLD1S_PRED_Z;
break; break;
case AArch64ISD::GLD1_SCALED: case AArch64ISD::GLD1_SCALED_PRED_Z:
NewOpc = AArch64ISD::GLD1S_SCALED; NewOpc = AArch64ISD::GLD1S_SCALED_PRED_Z;
break; break;
case AArch64ISD::GLD1_SXTW: case AArch64ISD::GLD1_SXTW_PRED_Z:
NewOpc = AArch64ISD::GLD1S_SXTW; NewOpc = AArch64ISD::GLD1S_SXTW_PRED_Z;
break; break;
case AArch64ISD::GLD1_SXTW_SCALED: case AArch64ISD::GLD1_SXTW_SCALED_PRED_Z:
NewOpc = AArch64ISD::GLD1S_SXTW_SCALED; NewOpc = AArch64ISD::GLD1S_SXTW_SCALED_PRED_Z;
break; break;
case AArch64ISD::GLD1_UXTW: case AArch64ISD::GLD1_UXTW_PRED_Z:
NewOpc = AArch64ISD::GLD1S_UXTW; NewOpc = AArch64ISD::GLD1S_UXTW_PRED_Z;
break; break;
case AArch64ISD::GLD1_UXTW_SCALED: case AArch64ISD::GLD1_UXTW_SCALED_PRED_Z:
NewOpc = AArch64ISD::GLD1S_UXTW_SCALED; NewOpc = AArch64ISD::GLD1S_UXTW_SCALED_PRED_Z;
break; break;
case AArch64ISD::GLD1_IMM: case AArch64ISD::GLD1_IMM_PRED_Z:
NewOpc = AArch64ISD::GLD1S_IMM; NewOpc = AArch64ISD::GLD1S_IMM_PRED_Z;
break; break;
case AArch64ISD::GLDFF1: case AArch64ISD::GLDFF1_PRED_Z:
NewOpc = AArch64ISD::GLDFF1S; NewOpc = AArch64ISD::GLDFF1S_PRED_Z;
break; break;
case AArch64ISD::GLDFF1_SCALED: case AArch64ISD::GLDFF1_SCALED_PRED_Z:
NewOpc = AArch64ISD::GLDFF1S_SCALED; NewOpc = AArch64ISD::GLDFF1S_SCALED_PRED_Z;
break; break;
case AArch64ISD::GLDFF1_SXTW: case AArch64ISD::GLDFF1_SXTW_PRED_Z:
NewOpc = AArch64ISD::GLDFF1S_SXTW; NewOpc = AArch64ISD::GLDFF1S_SXTW_PRED_Z;
break; break;
case AArch64ISD::GLDFF1_SXTW_SCALED: case AArch64ISD::GLDFF1_SXTW_SCALED_PRED_Z:
NewOpc = AArch64ISD::GLDFF1S_SXTW_SCALED; NewOpc = AArch64ISD::GLDFF1S_SXTW_SCALED_PRED_Z;
break; break;
case AArch64ISD::GLDFF1_UXTW: case AArch64ISD::GLDFF1_UXTW_PRED_Z:
NewOpc = AArch64ISD::GLDFF1S_UXTW; NewOpc = AArch64ISD::GLDFF1S_UXTW_PRED_Z;
break; break;
case AArch64ISD::GLDFF1_UXTW_SCALED: case AArch64ISD::GLDFF1_UXTW_SCALED_PRED_Z:
NewOpc = AArch64ISD::GLDFF1S_UXTW_SCALED; NewOpc = AArch64ISD::GLDFF1S_UXTW_SCALED_PRED_Z;
break; break;
case AArch64ISD::GLDFF1_IMM: case AArch64ISD::GLDFF1_IMM_PRED_Z:
NewOpc = AArch64ISD::GLDFF1S_IMM; NewOpc = AArch64ISD::GLDFF1S_IMM_PRED_Z;
break; break;
case AArch64ISD::GLDNT1: case AArch64ISD::GLDNT1_PRED_Z:
NewOpc = AArch64ISD::GLDNT1S; NewOpc = AArch64ISD::GLDNT1S_PRED_Z;
break; break;
default: default:
return SDValue(); return SDValue();
} }
EVT SignExtSrcVT = cast<VTSDNode>(N->getOperand(1))->getVT(); EVT SignExtSrcVT = cast<VTSDNode>(N->getOperand(1))->getVT();
EVT SrcMemVT = cast<VTSDNode>(Src->getOperand(MemVTOpNum))->getVT(); EVT SrcMemVT = cast<VTSDNode>(Src->getOperand(MemVTOpNum))->getVT();
▲ Show 20 Lines • Show All 161 Lines • ▼ Show 20 Lines case ISD::INTRINSIC_W_CHAIN:
case Intrinsic::aarch64_neon_st1x4: case Intrinsic::aarch64_neon_st1x4:
case Intrinsic::aarch64_neon_st2lane: case Intrinsic::aarch64_neon_st2lane:
case Intrinsic::aarch64_neon_st3lane: case Intrinsic::aarch64_neon_st3lane:
case Intrinsic::aarch64_neon_st4lane: case Intrinsic::aarch64_neon_st4lane:
return performNEONPostLDSTCombine(N, DCI, DAG); return performNEONPostLDSTCombine(N, DCI, DAG);
case Intrinsic::aarch64_sve_ldnt1: case Intrinsic::aarch64_sve_ldnt1:
return performLDNT1Combine(N, DAG); return performLDNT1Combine(N, DAG);
case Intrinsic::aarch64_sve_ld1rq: case Intrinsic::aarch64_sve_ld1rq:
return performLD1ReplicateCombine<AArch64ISD::LD1RQ>(N, DAG); return performLD1ReplicateCombine<AArch64ISD::LD1RQ_PRED_Z>(N, DAG);
case Intrinsic::aarch64_sve_ld1ro: case Intrinsic::aarch64_sve_ld1ro:
return performLD1ReplicateCombine<AArch64ISD::LD1RO>(N, DAG); return performLD1ReplicateCombine<AArch64ISD::LD1RO_PRED_Z>(N, DAG);
case Intrinsic::aarch64_sve_ldnt1_gather_scalar_offset: case Intrinsic::aarch64_sve_ldnt1_gather_scalar_offset:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLDNT1); return performGatherLoadCombine(N, DAG, AArch64ISD::GLDNT1_PRED_Z);
case Intrinsic::aarch64_sve_ldnt1_gather: case Intrinsic::aarch64_sve_ldnt1_gather:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLDNT1); return performGatherLoadCombine(N, DAG, AArch64ISD::GLDNT1_PRED_Z);
case Intrinsic::aarch64_sve_ldnt1_gather_index: case Intrinsic::aarch64_sve_ldnt1_gather_index:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLDNT1_INDEX); return performGatherLoadCombine(N, DAG, AArch64ISD::GLDNT1_INDEX_PRED_Z);
case Intrinsic::aarch64_sve_ldnt1_gather_uxtw: case Intrinsic::aarch64_sve_ldnt1_gather_uxtw:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLDNT1); return performGatherLoadCombine(N, DAG, AArch64ISD::GLDNT1_PRED_Z);
case Intrinsic::aarch64_sve_ld1: case Intrinsic::aarch64_sve_ld1:
return performLD1Combine(N, DAG, AArch64ISD::LD1); return performLD1Combine(N, DAG, AArch64ISD::LD1_PRED_Z);
case Intrinsic::aarch64_sve_ldnf1: case Intrinsic::aarch64_sve_ldnf1:
return performLD1Combine(N, DAG, AArch64ISD::LDNF1); return performLD1Combine(N, DAG, AArch64ISD::LDNF1_PRED_Z);
case Intrinsic::aarch64_sve_ldff1: case Intrinsic::aarch64_sve_ldff1:
return performLD1Combine(N, DAG, AArch64ISD::LDFF1); return performLD1Combine(N, DAG, AArch64ISD::LDFF1_PRED_Z);
case Intrinsic::aarch64_sve_st1: case Intrinsic::aarch64_sve_st1:
return performST1Combine(N, DAG); return performST1Combine(N, DAG);
case Intrinsic::aarch64_sve_stnt1: case Intrinsic::aarch64_sve_stnt1:
return performSTNT1Combine(N, DAG); return performSTNT1Combine(N, DAG);
case Intrinsic::aarch64_sve_stnt1_scatter_scalar_offset: case Intrinsic::aarch64_sve_stnt1_scatter_scalar_offset:
return performScatterStoreCombine(N, DAG, AArch64ISD::SSTNT1); return performScatterStoreCombine(N, DAG, AArch64ISD::SSTNT1_PRED);
case Intrinsic::aarch64_sve_stnt1_scatter_uxtw: case Intrinsic::aarch64_sve_stnt1_scatter_uxtw:
return performScatterStoreCombine(N, DAG, AArch64ISD::SSTNT1); return performScatterStoreCombine(N, DAG, AArch64ISD::SSTNT1_PRED);
case Intrinsic::aarch64_sve_stnt1_scatter: case Intrinsic::aarch64_sve_stnt1_scatter:
return performScatterStoreCombine(N, DAG, AArch64ISD::SSTNT1); return performScatterStoreCombine(N, DAG, AArch64ISD::SSTNT1_PRED);
case Intrinsic::aarch64_sve_stnt1_scatter_index: case Intrinsic::aarch64_sve_stnt1_scatter_index:
return performScatterStoreCombine(N, DAG, AArch64ISD::SSTNT1_INDEX); return performScatterStoreCombine(N, DAG, AArch64ISD::SSTNT1_INDEX_PRED);
case Intrinsic::aarch64_sve_ld1_gather: case Intrinsic::aarch64_sve_ld1_gather:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLD1); return performGatherLoadCombine(N, DAG, AArch64ISD::GLD1_PRED_Z);
case Intrinsic::aarch64_sve_ld1_gather_index: case Intrinsic::aarch64_sve_ld1_gather_index:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLD1_SCALED); return performGatherLoadCombine(N, DAG, AArch64ISD::GLD1_SCALED_PRED_Z);
case Intrinsic::aarch64_sve_ld1_gather_sxtw: case Intrinsic::aarch64_sve_ld1_gather_sxtw:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLD1_SXTW, return performGatherLoadCombine(N, DAG, AArch64ISD::GLD1_SXTW_PRED_Z,
/*OnlyPackedOffsets=*/false); /*OnlyPackedOffsets=*/false);
case Intrinsic::aarch64_sve_ld1_gather_uxtw: case Intrinsic::aarch64_sve_ld1_gather_uxtw:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLD1_UXTW, return performGatherLoadCombine(N, DAG, AArch64ISD::GLD1_UXTW_PRED_Z,
/*OnlyPackedOffsets=*/false); /*OnlyPackedOffsets=*/false);
case Intrinsic::aarch64_sve_ld1_gather_sxtw_index: case Intrinsic::aarch64_sve_ld1_gather_sxtw_index:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLD1_SXTW_SCALED, return performGatherLoadCombine(N, DAG,
AArch64ISD::GLD1_SXTW_SCALED_PRED_Z,
/*OnlyPackedOffsets=*/false); /*OnlyPackedOffsets=*/false);
case Intrinsic::aarch64_sve_ld1_gather_uxtw_index: case Intrinsic::aarch64_sve_ld1_gather_uxtw_index:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLD1_UXTW_SCALED, return performGatherLoadCombine(N, DAG,
AArch64ISD::GLD1_UXTW_SCALED_PRED_Z,
/*OnlyPackedOffsets=*/false); /*OnlyPackedOffsets=*/false);
case Intrinsic::aarch64_sve_ld1_gather_scalar_offset: case Intrinsic::aarch64_sve_ld1_gather_scalar_offset:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLD1_IMM); return performGatherLoadCombine(N, DAG, AArch64ISD::GLD1_IMM_PRED_Z);
case Intrinsic::aarch64_sve_ldff1_gather: case Intrinsic::aarch64_sve_ldff1_gather:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLDFF1); return performGatherLoadCombine(N, DAG, AArch64ISD::GLDFF1_PRED_Z);
case Intrinsic::aarch64_sve_ldff1_gather_index: case Intrinsic::aarch64_sve_ldff1_gather_index:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLDFF1_SCALED); return performGatherLoadCombine(N, DAG, AArch64ISD::GLDFF1_SCALED_PRED_Z);
case Intrinsic::aarch64_sve_ldff1_gather_sxtw: case Intrinsic::aarch64_sve_ldff1_gather_sxtw:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLDFF1_SXTW, return performGatherLoadCombine(N, DAG, AArch64ISD::GLDFF1_SXTW_PRED_Z,
/*OnlyPackedOffsets=*/false); /*OnlyPackedOffsets=*/false);
case Intrinsic::aarch64_sve_ldff1_gather_uxtw: case Intrinsic::aarch64_sve_ldff1_gather_uxtw:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLDFF1_UXTW, return performGatherLoadCombine(N, DAG, AArch64ISD::GLDFF1_UXTW_PRED_Z,
/*OnlyPackedOffsets=*/false); /*OnlyPackedOffsets=*/false);
case Intrinsic::aarch64_sve_ldff1_gather_sxtw_index: case Intrinsic::aarch64_sve_ldff1_gather_sxtw_index:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLDFF1_SXTW_SCALED, return performGatherLoadCombine(N, DAG,
AArch64ISD::GLDFF1_SXTW_SCALED_PRED_Z,
/*OnlyPackedOffsets=*/false); /*OnlyPackedOffsets=*/false);
case Intrinsic::aarch64_sve_ldff1_gather_uxtw_index: case Intrinsic::aarch64_sve_ldff1_gather_uxtw_index:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLDFF1_UXTW_SCALED, return performGatherLoadCombine(N, DAG,
AArch64ISD::GLDFF1_UXTW_SCALED_PRED_Z,
/*OnlyPackedOffsets=*/false); /*OnlyPackedOffsets=*/false);
case Intrinsic::aarch64_sve_ldff1_gather_scalar_offset: case Intrinsic::aarch64_sve_ldff1_gather_scalar_offset:
return performGatherLoadCombine(N, DAG, AArch64ISD::GLDFF1_IMM); return performGatherLoadCombine(N, DAG, AArch64ISD::GLDFF1_IMM_PRED_Z);
case Intrinsic::aarch64_sve_st1_scatter: case Intrinsic::aarch64_sve_st1_scatter:
return performScatterStoreCombine(N, DAG, AArch64ISD::SST1); return performScatterStoreCombine(N, DAG, AArch64ISD::SST1_PRED);
case Intrinsic::aarch64_sve_st1_scatter_index: case Intrinsic::aarch64_sve_st1_scatter_index:
return performScatterStoreCombine(N, DAG, AArch64ISD::SST1_SCALED); return performScatterStoreCombine(N, DAG, AArch64ISD::SST1_SCALED_PRED);
case Intrinsic::aarch64_sve_st1_scatter_sxtw: case Intrinsic::aarch64_sve_st1_scatter_sxtw:
return performScatterStoreCombine(N, DAG, AArch64ISD::SST1_SXTW, return performScatterStoreCombine(N, DAG, AArch64ISD::SST1_SXTW_PRED,
/*OnlyPackedOffsets=*/false); /*OnlyPackedOffsets=*/false);
case Intrinsic::aarch64_sve_st1_scatter_uxtw: case Intrinsic::aarch64_sve_st1_scatter_uxtw:
return performScatterStoreCombine(N, DAG, AArch64ISD::SST1_UXTW, return performScatterStoreCombine(N, DAG, AArch64ISD::SST1_UXTW_PRED,
/*OnlyPackedOffsets=*/false); /*OnlyPackedOffsets=*/false);
case Intrinsic::aarch64_sve_st1_scatter_sxtw_index: case Intrinsic::aarch64_sve_st1_scatter_sxtw_index:
return performScatterStoreCombine(N, DAG, AArch64ISD::SST1_SXTW_SCALED, return performScatterStoreCombine(N, DAG,
AArch64ISD::SST1_SXTW_SCALED_PRED,
/*OnlyPackedOffsets=*/false); /*OnlyPackedOffsets=*/false);
case Intrinsic::aarch64_sve_st1_scatter_uxtw_index: case Intrinsic::aarch64_sve_st1_scatter_uxtw_index:
return performScatterStoreCombine(N, DAG, AArch64ISD::SST1_UXTW_SCALED, return performScatterStoreCombine(N, DAG,
AArch64ISD::SST1_UXTW_SCALED_PRED,
/*OnlyPackedOffsets=*/false); /*OnlyPackedOffsets=*/false);
case Intrinsic::aarch64_sve_st1_scatter_scalar_offset: case Intrinsic::aarch64_sve_st1_scatter_scalar_offset:
return performScatterStoreCombine(N, DAG, AArch64ISD::SST1_IMM); return performScatterStoreCombine(N, DAG, AArch64ISD::SST1_IMM_PRED);
case Intrinsic::aarch64_sve_tuple_get: { case Intrinsic::aarch64_sve_tuple_get: {
SDLoc DL(N); SDLoc DL(N);
SDValue Chain = N->getOperand(0); SDValue Chain = N->getOperand(0);
SDValue Src1 = N->getOperand(2); SDValue Src1 = N->getOperand(2);
SDValue Idx = N->getOperand(3); SDValue Idx = N->getOperand(3);
uint64_t IdxConst = cast<ConstantSDNode>(Idx)->getZExtValue(); uint64_t IdxConst = cast<ConstantSDNode>(Idx)->getZExtValue();
if (IdxConst > Src1->getNumOperands() - 1) if (IdxConst > Src1->getNumOperands() - 1)
▲ Show 20 Lines • Show All 854 Lines • Show Last 20 Lines

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

//=- AArch64SVEInstrInfo.td - AArch64 SVE Instructions -*- tablegen -*-----=// //=- AArch64SVEInstrInfo.td - AArch64 SVE Instructions -*- tablegen -*-----=//
// //
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information. // See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
// //
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// //
// AArch64 Scalable Vector Extension (SVE) Instruction definitions. // AArch64 Scalable Vector Extension (SVE) Instruction definitions.
// //
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// For predicated nodes where the entire operation is controlled by a governing
// predicate, please stick to a similar naming convention as used for the
// ISD nodes.
// SDNode <=> AArch64ISD
// -------------------------------
// _m <=> _PRED_M
// _z <=> _PRED_Z
// _x <=> _PRED_X
// _passthru <=> _PRED_PASSTHRU
//
// Given the context of this file, it is not necessary to use _pred to
// distinguish predicated from unpredicated nodes given that most SVE
// instructions are predicated.
// Contiguous loads - node definitions // Contiguous loads - node definitions
// //
def SDT_AArch64_LD1 : SDTypeProfile<1, 3, [ def SDT_AArch64_LD1 : SDTypeProfile<1, 3, [
SDTCisVec<0>, SDTCisVec<1>, SDTCisPtrTy<2>, SDTCisVec<0>, SDTCisVec<1>, SDTCisPtrTy<2>,
SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1> SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1>
]>; ]>;
def AArch64ld1 : SDNode<"AArch64ISD::LD1", SDT_AArch64_LD1, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue]>; def AArch64ld1_z : SDNode<"AArch64ISD::LD1_PRED_Z", SDT_AArch64_LD1, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue]>;
def AArch64ld1s : SDNode<"AArch64ISD::LD1S", SDT_AArch64_LD1, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue]>; def AArch64ld1s_z : SDNode<"AArch64ISD::LD1S_PRED_Z", SDT_AArch64_LD1, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue]>;
// Non-faulting & first-faulting loads - node definitions // Non-faulting & first-faulting loads - node definitions
// //
def AArch64ldnf1 : SDNode<"AArch64ISD::LDNF1", SDT_AArch64_LD1, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldnf1_z : SDNode<"AArch64ISD::LDNF1_PRED_Z", SDT_AArch64_LD1, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1 : SDNode<"AArch64ISD::LDFF1", SDT_AArch64_LD1, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1_z : SDNode<"AArch64ISD::LDFF1_PRED_Z", SDT_AArch64_LD1, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldnf1s : SDNode<"AArch64ISD::LDNF1S", SDT_AArch64_LD1, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldnf1s_z : SDNode<"AArch64ISD::LDNF1S_PRED_Z", SDT_AArch64_LD1, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1s : SDNode<"AArch64ISD::LDFF1S", SDT_AArch64_LD1, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1s_z : SDNode<"AArch64ISD::LDFF1S_PRED_Z", SDT_AArch64_LD1, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
// Contiguous load and replicate - node definitions // Contiguous load and replicate - node definitions
// //
def SDT_AArch64_LD1Replicate : SDTypeProfile<1, 2, [ def SDT_AArch64_LD1Replicate : SDTypeProfile<1, 2, [
SDTCisVec<0>, SDTCisVec<1>, SDTCisPtrTy<2>, SDTCisVec<0>, SDTCisVec<1>, SDTCisPtrTy<2>,
SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1> SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1>
]>; ]>;
def AArch64ld1rq : SDNode<"AArch64ISD::LD1RQ", SDT_AArch64_LD1Replicate, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1rq_z : SDNode<"AArch64ISD::LD1RQ_PRED_Z", SDT_AArch64_LD1Replicate, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1ro : SDNode<"AArch64ISD::LD1RO", SDT_AArch64_LD1Replicate, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1ro_z : SDNode<"AArch64ISD::LD1RO_PRED_Z", SDT_AArch64_LD1Replicate, [SDNPHasChain, SDNPMayLoad]>;
// Gather loads - node definitions // Gather loads - node definitions
// //
def SDT_AArch64_GATHER_SV : SDTypeProfile<1, 4, [ def SDT_AArch64_GATHER_SV : SDTypeProfile<1, 4, [
SDTCisVec<0>, SDTCisVec<1>, SDTCisPtrTy<2>, SDTCisVec<3>, SDTCisVT<4, OtherVT>, SDTCisVec<0>, SDTCisVec<1>, SDTCisPtrTy<2>, SDTCisVec<3>, SDTCisVT<4, OtherVT>,
SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1> SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1>
]>; ]>;
def SDT_AArch64_GATHER_VS : SDTypeProfile<1, 4, [ def SDT_AArch64_GATHER_VS : SDTypeProfile<1, 4, [
SDTCisVec<0>, SDTCisVec<1>, SDTCisVec<2>, SDTCisInt<3>, SDTCisVT<4, OtherVT>, SDTCisVec<0>, SDTCisVec<1>, SDTCisVec<2>, SDTCisInt<3>, SDTCisVT<4, OtherVT>,
SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1> SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1>
]>; ]>;
def AArch64ld1_gather : SDNode<"AArch64ISD::GLD1", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1_gather_z : SDNode<"AArch64ISD::GLD1_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1_gather_scaled : SDNode<"AArch64ISD::GLD1_SCALED", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1_gather_scaled_z : SDNode<"AArch64ISD::GLD1_SCALED_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1_gather_uxtw : SDNode<"AArch64ISD::GLD1_UXTW", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1_gather_uxtw_z : SDNode<"AArch64ISD::GLD1_UXTW_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1_gather_sxtw : SDNode<"AArch64ISD::GLD1_SXTW", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1_gather_sxtw_z : SDNode<"AArch64ISD::GLD1_SXTW_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1_gather_uxtw_scaled : SDNode<"AArch64ISD::GLD1_UXTW_SCALED", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1_gather_uxtw_scaled_z : SDNode<"AArch64ISD::GLD1_UXTW_SCALED_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1_gather_sxtw_scaled : SDNode<"AArch64ISD::GLD1_SXTW_SCALED", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1_gather_sxtw_scaled_z : SDNode<"AArch64ISD::GLD1_SXTW_SCALED_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1_gather_imm : SDNode<"AArch64ISD::GLD1_IMM", SDT_AArch64_GATHER_VS, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1_gather_imm_z : SDNode<"AArch64ISD::GLD1_IMM_PRED_Z", SDT_AArch64_GATHER_VS, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1s_gather : SDNode<"AArch64ISD::GLD1S", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1s_gather_z : SDNode<"AArch64ISD::GLD1S_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1s_gather_scaled : SDNode<"AArch64ISD::GLD1S_SCALED", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1s_gather_scaled_z : SDNode<"AArch64ISD::GLD1S_SCALED_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1s_gather_uxtw : SDNode<"AArch64ISD::GLD1S_UXTW", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1s_gather_uxtw_z : SDNode<"AArch64ISD::GLD1S_UXTW_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1s_gather_sxtw : SDNode<"AArch64ISD::GLD1S_SXTW", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1s_gather_sxtw_z : SDNode<"AArch64ISD::GLD1S_SXTW_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1s_gather_uxtw_scaled : SDNode<"AArch64ISD::GLD1S_UXTW_SCALED", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1s_gather_uxtw_scaled_z : SDNode<"AArch64ISD::GLD1S_UXTW_SCALED_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1s_gather_sxtw_scaled : SDNode<"AArch64ISD::GLD1S_SXTW_SCALED", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1s_gather_sxtw_scaled_z : SDNode<"AArch64ISD::GLD1S_SXTW_SCALED_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ld1s_gather_imm : SDNode<"AArch64ISD::GLD1S_IMM", SDT_AArch64_GATHER_VS, [SDNPHasChain, SDNPMayLoad]>; def AArch64ld1s_gather_imm_z : SDNode<"AArch64ISD::GLD1S_IMM_PRED_Z", SDT_AArch64_GATHER_VS, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ldff1_gather : SDNode<"AArch64ISD::GLDFF1", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1_gather_z : SDNode<"AArch64ISD::GLDFF1_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1_gather_scaled : SDNode<"AArch64ISD::GLDFF1_SCALED", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1_gather_scaled_z : SDNode<"AArch64ISD::GLDFF1_SCALED_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1_gather_uxtw : SDNode<"AArch64ISD::GLDFF1_UXTW", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1_gather_uxtw_z : SDNode<"AArch64ISD::GLDFF1_UXTW_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1_gather_sxtw : SDNode<"AArch64ISD::GLDFF1_SXTW", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1_gather_sxtw_z : SDNode<"AArch64ISD::GLDFF1_SXTW_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1_gather_uxtw_scaled : SDNode<"AArch64ISD::GLDFF1_UXTW_SCALED", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1_gather_uxtw_scaled_z : SDNode<"AArch64ISD::GLDFF1_UXTW_SCALED_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1_gather_sxtw_scaled : SDNode<"AArch64ISD::GLDFF1_SXTW_SCALED", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1_gather_sxtw_scaled_z : SDNode<"AArch64ISD::GLDFF1_SXTW_SCALED_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1_gather_imm : SDNode<"AArch64ISD::GLDFF1_IMM", SDT_AArch64_GATHER_VS, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1_gather_imm_z : SDNode<"AArch64ISD::GLDFF1_IMM_PRED_Z", SDT_AArch64_GATHER_VS, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1s_gather : SDNode<"AArch64ISD::GLDFF1S", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1s_gather_z : SDNode<"AArch64ISD::GLDFF1S_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1s_gather_scaled : SDNode<"AArch64ISD::GLDFF1S_SCALED", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1s_gather_scaled_z : SDNode<"AArch64ISD::GLDFF1S_SCALED_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1s_gather_uxtw : SDNode<"AArch64ISD::GLDFF1S_UXTW", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1s_gather_uxtw_z : SDNode<"AArch64ISD::GLDFF1S_UXTW_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1s_gather_sxtw : SDNode<"AArch64ISD::GLDFF1S_SXTW", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1s_gather_sxtw_z : SDNode<"AArch64ISD::GLDFF1S_SXTW_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1s_gather_uxtw_scaled : SDNode<"AArch64ISD::GLDFF1S_UXTW_SCALED", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1s_gather_uxtw_scaled_z : SDNode<"AArch64ISD::GLDFF1S_UXTW_SCALED_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1s_gather_sxtw_scaled : SDNode<"AArch64ISD::GLDFF1S_SXTW_SCALED", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1s_gather_sxtw_scaled_z : SDNode<"AArch64ISD::GLDFF1S_SXTW_SCALED_PRED_Z", SDT_AArch64_GATHER_SV, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldff1s_gather_imm : SDNode<"AArch64ISD::GLDFF1S_IMM", SDT_AArch64_GATHER_VS, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>; def AArch64ldff1s_gather_imm_z : SDNode<"AArch64ISD::GLDFF1S_IMM_PRED_Z", SDT_AArch64_GATHER_VS, [SDNPHasChain, SDNPMayLoad, SDNPOptInGlue, SDNPOutGlue]>;
def AArch64ldnt1_gather : SDNode<"AArch64ISD::GLDNT1", SDT_AArch64_GATHER_VS, [SDNPHasChain, SDNPMayLoad]>; def AArch64ldnt1_gather_z : SDNode<"AArch64ISD::GLDNT1_PRED_Z", SDT_AArch64_GATHER_VS, [SDNPHasChain, SDNPMayLoad]>;
def AArch64ldnt1s_gather : SDNode<"AArch64ISD::GLDNT1S", SDT_AArch64_GATHER_VS, [SDNPHasChain, SDNPMayLoad]>; def AArch64ldnt1s_gather_z : SDNode<"AArch64ISD::GLDNT1S_PRED_Z", SDT_AArch64_GATHER_VS, [SDNPHasChain, SDNPMayLoad]>;
// Contiguous stores - node definitions // Contiguous stores - node definitions
// //
def SDT_AArch64_ST1 : SDTypeProfile<0, 4, [ def SDT_AArch64_ST1 : SDTypeProfile<0, 4, [
SDTCisVec<0>, SDTCisPtrTy<1>, SDTCisVec<2>, SDTCisVec<0>, SDTCisPtrTy<1>, SDTCisVec<2>,
SDTCVecEltisVT<2,i1>, SDTCisSameNumEltsAs<0,2> SDTCVecEltisVT<2,i1>, SDTCisSameNumEltsAs<0,2>
]>; ]>;
def AArch64st1 : SDNode<"AArch64ISD::ST1", SDT_AArch64_ST1, [SDNPHasChain, SDNPMayStore]>; def AArch64st1 : SDNode<"AArch64ISD::ST1_PRED", SDT_AArch64_ST1, [SDNPHasChain, SDNPMayStore]>;
// Scatter stores - node definitions // Scatter stores - node definitions
// //
def SDT_AArch64_SCATTER_SV : SDTypeProfile<0, 5, [ def SDT_AArch64_SCATTER_SV : SDTypeProfile<0, 5, [
SDTCisVec<0>, SDTCisVec<1>, SDTCisPtrTy<2>, SDTCisVec<3>, SDTCisVT<4, OtherVT>, SDTCisVec<0>, SDTCisVec<1>, SDTCisPtrTy<2>, SDTCisVec<3>, SDTCisVT<4, OtherVT>,
SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1> SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1>
]>; ]>;
def SDT_AArch64_SCATTER_VS : SDTypeProfile<0, 5, [ def SDT_AArch64_SCATTER_VS : SDTypeProfile<0, 5, [
SDTCisVec<0>, SDTCisVec<1>, SDTCisVec<2>, SDTCisInt<3>, SDTCisVT<4, OtherVT>, SDTCisVec<0>, SDTCisVec<1>, SDTCisVec<2>, SDTCisInt<3>, SDTCisVT<4, OtherVT>,
SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1> SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1>
]>; ]>;
def AArch64st1_scatter : SDNode<"AArch64ISD::SST1", SDT_AArch64_SCATTER_SV, [SDNPHasChain, SDNPMayStore]>; def AArch64st1_scatter : SDNode<"AArch64ISD::SST1_PRED", SDT_AArch64_SCATTER_SV, [SDNPHasChain, SDNPMayStore]>;
def AArch64st1_scatter_scaled : SDNode<"AArch64ISD::SST1_SCALED", SDT_AArch64_SCATTER_SV, [SDNPHasChain, SDNPMayStore]>; def AArch64st1_scatter_scaled : SDNode<"AArch64ISD::SST1_SCALED_PRED", SDT_AArch64_SCATTER_SV, [SDNPHasChain, SDNPMayStore]>;
def AArch64st1_scatter_uxtw : SDNode<"AArch64ISD::SST1_UXTW", SDT_AArch64_SCATTER_SV, [SDNPHasChain, SDNPMayStore]>; def AArch64st1_scatter_uxtw : SDNode<"AArch64ISD::SST1_UXTW_PRED", SDT_AArch64_SCATTER_SV, [SDNPHasChain, SDNPMayStore]>;
def AArch64st1_scatter_sxtw : SDNode<"AArch64ISD::SST1_SXTW", SDT_AArch64_SCATTER_SV, [SDNPHasChain, SDNPMayStore]>; def AArch64st1_scatter_sxtw : SDNode<"AArch64ISD::SST1_SXTW_PRED", SDT_AArch64_SCATTER_SV, [SDNPHasChain, SDNPMayStore]>;
def AArch64st1_scatter_uxtw_scaled : SDNode<"AArch64ISD::SST1_UXTW_SCALED", SDT_AArch64_SCATTER_SV, [SDNPHasChain, SDNPMayStore]>; def AArch64st1_scatter_uxtw_scaled : SDNode<"AArch64ISD::SST1_UXTW_SCALED_PRED", SDT_AArch64_SCATTER_SV, [SDNPHasChain, SDNPMayStore]>;
def AArch64st1_scatter_sxtw_scaled : SDNode<"AArch64ISD::SST1_SXTW_SCALED", SDT_AArch64_SCATTER_SV, [SDNPHasChain, SDNPMayStore]>; def AArch64st1_scatter_sxtw_scaled : SDNode<"AArch64ISD::SST1_SXTW_SCALED_PRED", SDT_AArch64_SCATTER_SV, [SDNPHasChain, SDNPMayStore]>;
def AArch64st1_scatter_imm : SDNode<"AArch64ISD::SST1_IMM", SDT_AArch64_SCATTER_VS, [SDNPHasChain, SDNPMayStore]>; def AArch64st1_scatter_imm : SDNode<"AArch64ISD::SST1_IMM_PRED", SDT_AArch64_SCATTER_VS, [SDNPHasChain, SDNPMayStore]>;
def AArch64stnt1_scatter : SDNode<"AArch64ISD::SSTNT1", SDT_AArch64_SCATTER_VS, [SDNPHasChain, SDNPMayStore]>; def AArch64stnt1_scatter : SDNode<"AArch64ISD::SSTNT1_PRED", SDT_AArch64_SCATTER_VS, [SDNPHasChain, SDNPMayStore]>;
// AArch64 SVE/SVE2 - the remaining node definitions // AArch64 SVE/SVE2 - the remaining node definitions
// //
// SVE CNT/INC/RDVL // SVE CNT/INC/RDVL
def sve_rdvl_imm : ComplexPattern<i32, 1, "SelectRDVLImm<-32, 31, 16>">; def sve_rdvl_imm : ComplexPattern<i32, 1, "SelectRDVLImm<-32, 31, 16>">;
def sve_cnth_imm : ComplexPattern<i32, 1, "SelectRDVLImm<1, 16, 8>">; def sve_cnth_imm : ComplexPattern<i32, 1, "SelectRDVLImm<1, 16, 8>">;
def sve_cntw_imm : ComplexPattern<i32, 1, "SelectRDVLImm<1, 16, 4>">; def sve_cntw_imm : ComplexPattern<i32, 1, "SelectRDVLImm<1, 16, 4>">;
def sve_cntd_imm : ComplexPattern<i32, 1, "SelectRDVLImm<1, 16, 2>">; def sve_cntd_imm : ComplexPattern<i32, 1, "SelectRDVLImm<1, 16, 2>">;
// SVE DEC // SVE DEC
def sve_cnth_imm_neg : ComplexPattern<i32, 1, "SelectRDVLImm<1, 16, -8>">; def sve_cnth_imm_neg : ComplexPattern<i32, 1, "SelectRDVLImm<1, 16, -8>">;
def sve_cntw_imm_neg : ComplexPattern<i32, 1, "SelectRDVLImm<1, 16, -4>">; def sve_cntw_imm_neg : ComplexPattern<i32, 1, "SelectRDVLImm<1, 16, -4>">;
def sve_cntd_imm_neg : ComplexPattern<i32, 1, "SelectRDVLImm<1, 16, -2>">; def sve_cntd_imm_neg : ComplexPattern<i32, 1, "SelectRDVLImm<1, 16, -2>">;
def SDT_AArch64Reduce : SDTypeProfile<1, 2, [SDTCisVec<1>, SDTCisVec<2>]>; def SDT_AArch64Reduce : SDTypeProfile<1, 2, [SDTCisVec<1>, SDTCisVec<2>]>;
def AArch64faddv_pred : SDNode<"AArch64ISD::FADDV_PRED", SDT_AArch64Reduce>; def AArch64faddv_m : SDNode<"AArch64ISD::FADDV_PRED", SDT_AArch64Reduce>;
def AArch64fmaxv_pred : SDNode<"AArch64ISD::FMAXV_PRED", SDT_AArch64Reduce>; def AArch64fmaxv_m : SDNode<"AArch64ISD::FMAXV_PRED", SDT_AArch64Reduce>;
def AArch64fmaxnmv_pred : SDNode<"AArch64ISD::FMAXNMV_PRED", SDT_AArch64Reduce>; def AArch64fmaxnmv_m : SDNode<"AArch64ISD::FMAXNMV_PRED", SDT_AArch64Reduce>;
def AArch64fminv_pred : SDNode<"AArch64ISD::FMINV_PRED", SDT_AArch64Reduce>; def AArch64fminv_m : SDNode<"AArch64ISD::FMINV_PRED", SDT_AArch64Reduce>;
def AArch64fminnmv_pred : SDNode<"AArch64ISD::FMINNMV_PRED", SDT_AArch64Reduce>; def AArch64fminnmv_m : SDNode<"AArch64ISD::FMINNMV_PRED", SDT_AArch64Reduce>;
def AArch64smaxv_pred : SDNode<"AArch64ISD::SMAXV_PRED", SDT_AArch64Reduce>; def AArch64smaxv_m : SDNode<"AArch64ISD::SMAXV_PRED", SDT_AArch64Reduce>;
def AArch64umaxv_pred : SDNode<"AArch64ISD::UMAXV_PRED", SDT_AArch64Reduce>; def AArch64umaxv_m : SDNode<"AArch64ISD::UMAXV_PRED", SDT_AArch64Reduce>;
def AArch64sminv_pred : SDNode<"AArch64ISD::SMINV_PRED", SDT_AArch64Reduce>; def AArch64sminv_m : SDNode<"AArch64ISD::SMINV_PRED", SDT_AArch64Reduce>;
def AArch64uminv_pred : SDNode<"AArch64ISD::UMINV_PRED", SDT_AArch64Reduce>; def AArch64uminv_m : SDNode<"AArch64ISD::UMINV_PRED", SDT_AArch64Reduce>;
def AArch64orv_pred : SDNode<"AArch64ISD::ORV_PRED", SDT_AArch64Reduce>; def AArch64orv_m : SDNode<"AArch64ISD::ORV_PRED", SDT_AArch64Reduce>;
def AArch64eorv_pred : SDNode<"AArch64ISD::EORV_PRED", SDT_AArch64Reduce>; def AArch64eorv_m : SDNode<"AArch64ISD::EORV_PRED", SDT_AArch64Reduce>;
def AArch64andv_pred : SDNode<"AArch64ISD::ANDV_PRED", SDT_AArch64Reduce>; def AArch64andv_m : SDNode<"AArch64ISD::ANDV_PRED", SDT_AArch64Reduce>;
def AArch64lasta : SDNode<"AArch64ISD::LASTA", SDT_AArch64Reduce>; def AArch64lasta : SDNode<"AArch64ISD::LASTA", SDT_AArch64Reduce>;
def AArch64lastb : SDNode<"AArch64ISD::LASTB", SDT_AArch64Reduce>; def AArch64lastb : SDNode<"AArch64ISD::LASTB", SDT_AArch64Reduce>;
def SDT_AArch64Arith : SDTypeProfile<1, 3, [ def SDT_AArch64Arith : SDTypeProfile<1, 3, [
SDTCisVec<0>, SDTCisVec<1>, SDTCisVec<2>, SDTCisVec<3>, SDTCisVec<0>, SDTCisVec<1>, SDTCisVec<2>, SDTCisVec<3>,
SDTCVecEltisVT<1,i1>, SDTCisSameAs<2,3> SDTCVecEltisVT<1,i1>, SDTCisSameAs<2,3>
]>; ]>;
def AArch64sdiv_pred : SDNode<"AArch64ISD::SDIV_PRED", SDT_AArch64Arith>; def AArch64sdiv_m : SDNode<"AArch64ISD::SDIV_PRED_M", SDT_AArch64Arith>;
def AArch64udiv_pred : SDNode<"AArch64ISD::UDIV_PRED", SDT_AArch64Arith>; def AArch64udiv_m : SDNode<"AArch64ISD::UDIV_PRED_M", SDT_AArch64Arith>;
def AArch64smin_pred : SDNode<"AArch64ISD::SMIN_PRED", SDT_AArch64Arith>; def AArch64smin_m : SDNode<"AArch64ISD::SMIN_PRED_M", SDT_AArch64Arith>;
def AArch64umin_pred : SDNode<"AArch64ISD::UMIN_PRED", SDT_AArch64Arith>; def AArch64umin_m : SDNode<"AArch64ISD::UMIN_PRED_M", SDT_AArch64Arith>;
def AArch64smax_pred : SDNode<"AArch64ISD::SMAX_PRED", SDT_AArch64Arith>; def AArch64smax_m : SDNode<"AArch64ISD::SMAX_PRED_M", SDT_AArch64Arith>;
def AArch64umax_pred : SDNode<"AArch64ISD::UMAX_PRED", SDT_AArch64Arith>; def AArch64umax_m : SDNode<"AArch64ISD::UMAX_PRED_M", SDT_AArch64Arith>;
def AArch64lsl_pred : SDNode<"AArch64ISD::SHL_PRED", SDT_AArch64Arith>; def AArch64lsl_m : SDNode<"AArch64ISD::SHL_PRED_M", SDT_AArch64Arith>;
def AArch64lsr_pred : SDNode<"AArch64ISD::SRL_PRED", SDT_AArch64Arith>; def AArch64lsr_m : SDNode<"AArch64ISD::SRL_PRED_M", SDT_AArch64Arith>;
def AArch64asr_pred : SDNode<"AArch64ISD::SRA_PRED", SDT_AArch64Arith>; def AArch64asr_m : SDNode<"AArch64ISD::SRA_PRED_M", SDT_AArch64Arith>;
def SDT_AArch64ReduceWithInit : SDTypeProfile<1, 3, [SDTCisVec<1>, SDTCisVec<3>]>; def SDT_AArch64ReduceWithInit : SDTypeProfile<1, 3, [SDTCisVec<1>, SDTCisVec<3>]>;
def AArch64clasta_n : SDNode<"AArch64ISD::CLASTA_N", SDT_AArch64ReduceWithInit>; def AArch64clasta_n : SDNode<"AArch64ISD::CLASTA_N", SDT_AArch64ReduceWithInit>;
def AArch64clastb_n : SDNode<"AArch64ISD::CLASTB_N", SDT_AArch64ReduceWithInit>; def AArch64clastb_n : SDNode<"AArch64ISD::CLASTB_N", SDT_AArch64ReduceWithInit>;
def AArch64fadda_pred : SDNode<"AArch64ISD::FADDA_PRED", SDT_AArch64ReduceWithInit>; def AArch64fadda_m : SDNode<"AArch64ISD::FADDA_PRED", SDT_AArch64ReduceWithInit>;
def SDT_AArch64Rev : SDTypeProfile<1, 1, [SDTCisVec<0>, SDTCisSameAs<0,1>]>; def SDT_AArch64Rev : SDTypeProfile<1, 1, [SDTCisVec<0>, SDTCisSameAs<0,1>]>;
def AArch64rev : SDNode<"AArch64ISD::REV", SDT_AArch64Rev>; def AArch64rev : SDNode<"AArch64ISD::REV", SDT_AArch64Rev>;
def SDT_AArch64PTest : SDTypeProfile<0, 2, [SDTCisVec<0>, SDTCisSameAs<0,1>]>; def SDT_AArch64PTest : SDTypeProfile<0, 2, [SDTCisVec<0>, SDTCisSameAs<0,1>]>;
def AArch64ptest : SDNode<"AArch64ISD::PTEST", SDT_AArch64PTest>; def AArch64ptest : SDNode<"AArch64ISD::PTEST", SDT_AArch64PTest>;
def SDT_AArch64DUP_PRED : SDTypeProfile<1, 3, [SDTCisVec<0>, SDTCisSameAs<0,1>, SDTCisVec<2>, SDTCVecEltisVT<2,i1>]>; def SDT_AArch64DUP_PRED : SDTypeProfile<1, 3, [SDTCisVec<0>, SDTCisSameAs<0,1>, SDTCisVec<2>, SDTCVecEltisVT<2,i1>]>;
def AArch64dup_pred : SDNode<"AArch64ISD::DUP_PRED", SDT_AArch64DUP_PRED>; def AArch64dup_m : SDNode<"AArch64ISD::DUP_PRED_M", SDT_AArch64DUP_PRED>;
def SDT_IndexVector : SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisSameAs<1, 2>, SDTCisInt<2>]>; def SDT_IndexVector : SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisSameAs<1, 2>, SDTCisInt<2>]>;
def index_vector : SDNode<"AArch64ISD::INDEX_VECTOR", SDT_IndexVector, []>; def index_vector : SDNode<"AArch64ISD::INDEX_VECTOR", SDT_IndexVector, []>;
def reinterpret_cast : SDNode<"AArch64ISD::REINTERPRET_CAST", SDTUnaryOp>; def reinterpret_cast : SDNode<"AArch64ISD::REINTERPRET_CAST", SDTUnaryOp>;
let Predicates = [HasSVE] in { let Predicates = [HasSVE] in {
defm RDFFR_PPz : sve_int_rdffr_pred<0b0, "rdffr", int_aarch64_sve_rdffr_z>; defm RDFFR_PPz : sve_int_rdffr_pred<0b0, "rdffr", int_aarch64_sve_rdffr_z>;
Show All 38 Lineslet Predicates = [HasSVE] in {
defm MAD_ZPmZZ : sve_int_mladdsub_vvv_pred<0b0, "mad", int_aarch64_sve_mad>; defm MAD_ZPmZZ : sve_int_mladdsub_vvv_pred<0b0, "mad", int_aarch64_sve_mad>;
defm MSB_ZPmZZ : sve_int_mladdsub_vvv_pred<0b1, "msb", int_aarch64_sve_msb>; defm MSB_ZPmZZ : sve_int_mladdsub_vvv_pred<0b1, "msb", int_aarch64_sve_msb>;
defm MLA_ZPmZZ : sve_int_mlas_vvv_pred<0b0, "mla", int_aarch64_sve_mla>; defm MLA_ZPmZZ : sve_int_mlas_vvv_pred<0b0, "mla", int_aarch64_sve_mla>;
defm MLS_ZPmZZ : sve_int_mlas_vvv_pred<0b1, "mls", int_aarch64_sve_mls>; defm MLS_ZPmZZ : sve_int_mlas_vvv_pred<0b1, "mls", int_aarch64_sve_mls>;
// SVE predicated integer reductions. // SVE predicated integer reductions.
defm SADDV_VPZ : sve_int_reduce_0_saddv<0b000, "saddv", int_aarch64_sve_saddv>; defm SADDV_VPZ : sve_int_reduce_0_saddv<0b000, "saddv", int_aarch64_sve_saddv>;
defm UADDV_VPZ : sve_int_reduce_0_uaddv<0b001, "uaddv", int_aarch64_sve_uaddv, int_aarch64_sve_saddv>; defm UADDV_VPZ : sve_int_reduce_0_uaddv<0b001, "uaddv", int_aarch64_sve_uaddv, int_aarch64_sve_saddv>;
defm SMAXV_VPZ : sve_int_reduce_1<0b000, "smaxv", AArch64smaxv_pred>; defm SMAXV_VPZ : sve_int_reduce_1<0b000, "smaxv", AArch64smaxv_m>;
defm UMAXV_VPZ : sve_int_reduce_1<0b001, "umaxv", AArch64umaxv_pred>; defm UMAXV_VPZ : sve_int_reduce_1<0b001, "umaxv", AArch64umaxv_m>;
defm SMINV_VPZ : sve_int_reduce_1<0b010, "sminv", AArch64sminv_pred>; defm SMINV_VPZ : sve_int_reduce_1<0b010, "sminv", AArch64sminv_m>;
defm UMINV_VPZ : sve_int_reduce_1<0b011, "uminv", AArch64uminv_pred>; defm UMINV_VPZ : sve_int_reduce_1<0b011, "uminv", AArch64uminv_m>;
defm ORV_VPZ : sve_int_reduce_2<0b000, "orv", AArch64orv_pred>; defm ORV_VPZ : sve_int_reduce_2<0b000, "orv", AArch64orv_m>;
defm EORV_VPZ : sve_int_reduce_2<0b001, "eorv", AArch64eorv_pred>; defm EORV_VPZ : sve_int_reduce_2<0b001, "eorv", AArch64eorv_m>;
defm ANDV_VPZ : sve_int_reduce_2<0b010, "andv", AArch64andv_pred>; defm ANDV_VPZ : sve_int_reduce_2<0b010, "andv", AArch64andv_m>;
defm ORR_ZI : sve_int_log_imm<0b00, "orr", "orn", or>; defm ORR_ZI : sve_int_log_imm<0b00, "orr", "orn", or>;
defm EOR_ZI : sve_int_log_imm<0b01, "eor", "eon", xor>; defm EOR_ZI : sve_int_log_imm<0b01, "eor", "eon", xor>;
defm AND_ZI : sve_int_log_imm<0b10, "and", "bic", and>; defm AND_ZI : sve_int_log_imm<0b10, "and", "bic", and>;
defm SMAX_ZI : sve_int_arith_imm1<0b00, "smax", AArch64smax_pred>; defm SMAX_ZI : sve_int_arith_imm1<0b00, "smax", AArch64smax_m>;
defm SMIN_ZI : sve_int_arith_imm1<0b10, "smin", AArch64smin_pred>; defm SMIN_ZI : sve_int_arith_imm1<0b10, "smin", AArch64smin_m>;
defm UMAX_ZI : sve_int_arith_imm1_unsigned<0b01, "umax", AArch64umax_pred>; defm UMAX_ZI : sve_int_arith_imm1_unsigned<0b01, "umax", AArch64umax_m>;
defm UMIN_ZI : sve_int_arith_imm1_unsigned<0b11, "umin", AArch64umin_pred>; defm UMIN_ZI : sve_int_arith_imm1_unsigned<0b11, "umin", AArch64umin_m>;
defm MUL_ZI : sve_int_arith_imm2<"mul", mul>; defm MUL_ZI : sve_int_arith_imm2<"mul", mul>;
defm MUL_ZPmZ : sve_int_bin_pred_arit_2<0b000, "mul", int_aarch64_sve_mul>; defm MUL_ZPmZ : sve_int_bin_pred_arit_2<0b000, "mul", int_aarch64_sve_mul>;
defm SMULH_ZPmZ : sve_int_bin_pred_arit_2<0b010, "smulh", int_aarch64_sve_smulh>; defm SMULH_ZPmZ : sve_int_bin_pred_arit_2<0b010, "smulh", int_aarch64_sve_smulh>;
defm UMULH_ZPmZ : sve_int_bin_pred_arit_2<0b011, "umulh", int_aarch64_sve_umulh>; defm UMULH_ZPmZ : sve_int_bin_pred_arit_2<0b011, "umulh", int_aarch64_sve_umulh>;
// Add unpredicated alternative for the mul instruction. // Add unpredicated alternative for the mul instruction.
def : Pat<(mul nxv16i8:$Op1, nxv16i8:$Op2), def : Pat<(mul nxv16i8:$Op1, nxv16i8:$Op2),
(MUL_ZPmZ_B (PTRUE_B 31), $Op1, $Op2)>; (MUL_ZPmZ_B (PTRUE_B 31), $Op1, $Op2)>;
def : Pat<(mul nxv8i16:$Op1, nxv8i16:$Op2), def : Pat<(mul nxv8i16:$Op1, nxv8i16:$Op2),
(MUL_ZPmZ_H (PTRUE_H 31), $Op1, $Op2)>; (MUL_ZPmZ_H (PTRUE_H 31), $Op1, $Op2)>;
def : Pat<(mul nxv4i32:$Op1, nxv4i32:$Op2), def : Pat<(mul nxv4i32:$Op1, nxv4i32:$Op2),
(MUL_ZPmZ_S (PTRUE_S 31), $Op1, $Op2)>; (MUL_ZPmZ_S (PTRUE_S 31), $Op1, $Op2)>;
def : Pat<(mul nxv2i64:$Op1, nxv2i64:$Op2), def : Pat<(mul nxv2i64:$Op1, nxv2i64:$Op2),
(MUL_ZPmZ_D (PTRUE_D 31), $Op1, $Op2)>; (MUL_ZPmZ_D (PTRUE_D 31), $Op1, $Op2)>;
defm SDIV_ZPmZ : sve_int_bin_pred_arit_2_div<0b100, "sdiv", AArch64sdiv_pred>; defm SDIV_ZPmZ : sve_int_bin_pred_arit_2_div<0b100, "sdiv", AArch64sdiv_m>;
defm UDIV_ZPmZ : sve_int_bin_pred_arit_2_div<0b101, "udiv", AArch64udiv_pred>; defm UDIV_ZPmZ : sve_int_bin_pred_arit_2_div<0b101, "udiv", AArch64udiv_m>;
defm SDIVR_ZPmZ : sve_int_bin_pred_arit_2_div<0b110, "sdivr", int_aarch64_sve_sdivr>; defm SDIVR_ZPmZ : sve_int_bin_pred_arit_2_div<0b110, "sdivr", int_aarch64_sve_sdivr>;
defm UDIVR_ZPmZ : sve_int_bin_pred_arit_2_div<0b111, "udivr", int_aarch64_sve_udivr>; defm UDIVR_ZPmZ : sve_int_bin_pred_arit_2_div<0b111, "udivr", int_aarch64_sve_udivr>;
defm SDOT_ZZZ : sve_intx_dot<0b0, "sdot", int_aarch64_sve_sdot>; defm SDOT_ZZZ : sve_intx_dot<0b0, "sdot", int_aarch64_sve_sdot>;
defm UDOT_ZZZ : sve_intx_dot<0b1, "udot", int_aarch64_sve_udot>; defm UDOT_ZZZ : sve_intx_dot<0b1, "udot", int_aarch64_sve_udot>;
defm SDOT_ZZZI : sve_intx_dot_by_indexed_elem<0b0, "sdot", int_aarch64_sve_sdot_lane>; defm SDOT_ZZZI : sve_intx_dot_by_indexed_elem<0b0, "sdot", int_aarch64_sve_sdot_lane>;
defm UDOT_ZZZI : sve_intx_dot_by_indexed_elem<0b1, "udot", int_aarch64_sve_udot_lane>; defm UDOT_ZZZI : sve_intx_dot_by_indexed_elem<0b1, "udot", int_aarch64_sve_udot_lane>;
Show All 10 Lineslet Predicates = [HasSVE] in {
defm CLS_ZPmZ : sve_int_un_pred_arit_1< 0b000, "cls", int_aarch64_sve_cls>; defm CLS_ZPmZ : sve_int_un_pred_arit_1< 0b000, "cls", int_aarch64_sve_cls>;
defm CLZ_ZPmZ : sve_int_un_pred_arit_1< 0b001, "clz", int_aarch64_sve_clz>; defm CLZ_ZPmZ : sve_int_un_pred_arit_1< 0b001, "clz", int_aarch64_sve_clz>;
defm CNT_ZPmZ : sve_int_un_pred_arit_1< 0b010, "cnt", int_aarch64_sve_cnt>; defm CNT_ZPmZ : sve_int_un_pred_arit_1< 0b010, "cnt", int_aarch64_sve_cnt>;
defm CNOT_ZPmZ : sve_int_un_pred_arit_1< 0b011, "cnot", int_aarch64_sve_cnot>; defm CNOT_ZPmZ : sve_int_un_pred_arit_1< 0b011, "cnot", int_aarch64_sve_cnot>;
defm NOT_ZPmZ : sve_int_un_pred_arit_1< 0b110, "not", int_aarch64_sve_not>; defm NOT_ZPmZ : sve_int_un_pred_arit_1< 0b110, "not", int_aarch64_sve_not>;
defm FABS_ZPmZ : sve_int_un_pred_arit_1_fp<0b100, "fabs", int_aarch64_sve_fabs>; defm FABS_ZPmZ : sve_int_un_pred_arit_1_fp<0b100, "fabs", int_aarch64_sve_fabs>;
defm FNEG_ZPmZ : sve_int_un_pred_arit_1_fp<0b101, "fneg", int_aarch64_sve_fneg>; defm FNEG_ZPmZ : sve_int_un_pred_arit_1_fp<0b101, "fneg", int_aarch64_sve_fneg>;
defm SMAX_ZPmZ : sve_int_bin_pred_arit_1<0b000, "smax", AArch64smax_pred>; defm SMAX_ZPmZ : sve_int_bin_pred_arit_1<0b000, "smax", AArch64smax_m>;
defm UMAX_ZPmZ : sve_int_bin_pred_arit_1<0b001, "umax", AArch64umax_pred>; defm UMAX_ZPmZ : sve_int_bin_pred_arit_1<0b001, "umax", AArch64umax_m>;
defm SMIN_ZPmZ : sve_int_bin_pred_arit_1<0b010, "smin", AArch64smin_pred>; defm SMIN_ZPmZ : sve_int_bin_pred_arit_1<0b010, "smin", AArch64smin_m>;
defm UMIN_ZPmZ : sve_int_bin_pred_arit_1<0b011, "umin", AArch64umin_pred>; defm UMIN_ZPmZ : sve_int_bin_pred_arit_1<0b011, "umin", AArch64umin_m>;
defm SABD_ZPmZ : sve_int_bin_pred_arit_1<0b100, "sabd", int_aarch64_sve_sabd>; defm SABD_ZPmZ : sve_int_bin_pred_arit_1<0b100, "sabd", int_aarch64_sve_sabd>;
defm UABD_ZPmZ : sve_int_bin_pred_arit_1<0b101, "uabd", int_aarch64_sve_uabd>; defm UABD_ZPmZ : sve_int_bin_pred_arit_1<0b101, "uabd", int_aarch64_sve_uabd>;
defm FRECPE_ZZ : sve_fp_2op_u_zd<0b110, "frecpe", int_aarch64_sve_frecpe_x>; defm FRECPE_ZZ : sve_fp_2op_u_zd<0b110, "frecpe", int_aarch64_sve_frecpe_x>;
defm FRSQRTE_ZZ : sve_fp_2op_u_zd<0b111, "frsqrte", int_aarch64_sve_frsqrte_x>; defm FRSQRTE_ZZ : sve_fp_2op_u_zd<0b111, "frsqrte", int_aarch64_sve_frsqrte_x>;
defm FADD_ZPmI : sve_fp_2op_i_p_zds<0b000, "fadd", sve_fpimm_half_one>; defm FADD_ZPmI : sve_fp_2op_i_p_zds<0b000, "fadd", sve_fpimm_half_one>;
defm FSUB_ZPmI : sve_fp_2op_i_p_zds<0b001, "fsub", sve_fpimm_half_one>; defm FSUB_ZPmI : sve_fp_2op_i_p_zds<0b001, "fsub", sve_fpimm_half_one>;
▲ Show 20 Lines • Show All 57 Lines • ▼ Show 20 Lineslet Predicates = [HasSVE] in {
defm FMLA_ZZZI : sve_fp_fma_by_indexed_elem<0b0, "fmla", int_aarch64_sve_fmla_lane>; defm FMLA_ZZZI : sve_fp_fma_by_indexed_elem<0b0, "fmla", int_aarch64_sve_fmla_lane>;
defm FMLS_ZZZI : sve_fp_fma_by_indexed_elem<0b1, "fmls", int_aarch64_sve_fmls_lane>; defm FMLS_ZZZI : sve_fp_fma_by_indexed_elem<0b1, "fmls", int_aarch64_sve_fmls_lane>;
defm FCMLA_ZZZI : sve_fp_fcmla_by_indexed_elem<"fcmla", int_aarch64_sve_fcmla_lane>; defm FCMLA_ZZZI : sve_fp_fcmla_by_indexed_elem<"fcmla", int_aarch64_sve_fcmla_lane>;
defm FMUL_ZZZI : sve_fp_fmul_by_indexed_elem<"fmul", int_aarch64_sve_fmul_lane>; defm FMUL_ZZZI : sve_fp_fmul_by_indexed_elem<"fmul", int_aarch64_sve_fmul_lane>;
// SVE floating point reductions. // SVE floating point reductions.
defm FADDA_VPZ : sve_fp_2op_p_vd<0b000, "fadda", AArch64fadda_pred>; defm FADDA_VPZ : sve_fp_2op_p_vd<0b000, "fadda", AArch64fadda_m>;
defm FADDV_VPZ : sve_fp_fast_red<0b000, "faddv", AArch64faddv_pred>; defm FADDV_VPZ : sve_fp_fast_red<0b000, "faddv", AArch64faddv_m>;
defm FMAXNMV_VPZ : sve_fp_fast_red<0b100, "fmaxnmv", AArch64fmaxnmv_pred>; defm FMAXNMV_VPZ : sve_fp_fast_red<0b100, "fmaxnmv", AArch64fmaxnmv_m>;
defm FMINNMV_VPZ : sve_fp_fast_red<0b101, "fminnmv", AArch64fminnmv_pred>; defm FMINNMV_VPZ : sve_fp_fast_red<0b101, "fminnmv", AArch64fminnmv_m>;
defm FMAXV_VPZ : sve_fp_fast_red<0b110, "fmaxv", AArch64fmaxv_pred>; defm FMAXV_VPZ : sve_fp_fast_red<0b110, "fmaxv", AArch64fmaxv_m>;
defm FMINV_VPZ : sve_fp_fast_red<0b111, "fminv", AArch64fminv_pred>; defm FMINV_VPZ : sve_fp_fast_red<0b111, "fminv", AArch64fminv_m>;
// Use more efficient NEON instructions to extract elements within the NEON // Use more efficient NEON instructions to extract elements within the NEON
// part (first 128bits) of an SVE register. // part (first 128bits) of an SVE register.
def : Pat<(vector_extract (nxv8f16 ZPR:$Zs), (i64 0)), def : Pat<(vector_extract (nxv8f16 ZPR:$Zs), (i64 0)),
(f16 (EXTRACT_SUBREG (v8f16 (EXTRACT_SUBREG ZPR:$Zs, zsub)), hsub))>; (f16 (EXTRACT_SUBREG (v8f16 (EXTRACT_SUBREG ZPR:$Zs, zsub)), hsub))>;
def : Pat<(vector_extract (nxv4f32 ZPR:$Zs), (i64 0)), def : Pat<(vector_extract (nxv4f32 ZPR:$Zs), (i64 0)),
(f32 (EXTRACT_SUBREG (v4f32 (EXTRACT_SUBREG ZPR:$Zs, zsub)), ssub))>; (f32 (EXTRACT_SUBREG (v4f32 (EXTRACT_SUBREG ZPR:$Zs, zsub)), ssub))>;
def : Pat<(vector_extract (nxv2f64 ZPR:$Zs), (i64 0)), def : Pat<(vector_extract (nxv2f64 ZPR:$Zs), (i64 0)),
Show All 9 Lineslet Predicates = [HasSVE] in {
defm CPY_ZPzI : sve_int_dup_imm_pred_zero<"cpy">; defm CPY_ZPzI : sve_int_dup_imm_pred_zero<"cpy">;
defm FCPY_ZPmI : sve_int_dup_fpimm_pred<"fcpy">; defm FCPY_ZPmI : sve_int_dup_fpimm_pred<"fcpy">;
// Splat scalar register (unpredicated, GPR or vector + element index) // Splat scalar register (unpredicated, GPR or vector + element index)
defm DUP_ZR : sve_int_perm_dup_r<"dup", AArch64dup>; defm DUP_ZR : sve_int_perm_dup_r<"dup", AArch64dup>;
defm DUP_ZZI : sve_int_perm_dup_i<"dup">; defm DUP_ZZI : sve_int_perm_dup_i<"dup">;
// Splat scalar register (predicated) // Splat scalar register (predicated)
defm CPY_ZPmR : sve_int_perm_cpy_r<"cpy", AArch64dup_pred>; defm CPY_ZPmR : sve_int_perm_cpy_r<"cpy", AArch64dup_m>;
defm CPY_ZPmV : sve_int_perm_cpy_v<"cpy", AArch64dup_pred>; defm CPY_ZPmV : sve_int_perm_cpy_v<"cpy", AArch64dup_m>;
// Duplicate FP scalar into all vector elements // Duplicate FP scalar into all vector elements
def : Pat<(nxv8f16 (AArch64dup (f16 FPR16:$src))), def : Pat<(nxv8f16 (AArch64dup (f16 FPR16:$src))),
(DUP_ZZI_H (INSERT_SUBREG (IMPLICIT_DEF), FPR16:$src, hsub), 0)>; (DUP_ZZI_H (INSERT_SUBREG (IMPLICIT_DEF), FPR16:$src, hsub), 0)>;
def : Pat<(nxv4f16 (AArch64dup (f16 FPR16:$src))), def : Pat<(nxv4f16 (AArch64dup (f16 FPR16:$src))),
(DUP_ZZI_H (INSERT_SUBREG (IMPLICIT_DEF), FPR16:$src, hsub), 0)>; (DUP_ZZI_H (INSERT_SUBREG (IMPLICIT_DEF), FPR16:$src, hsub), 0)>;
def : Pat<(nxv2f16 (AArch64dup (f16 FPR16:$src))), def : Pat<(nxv2f16 (AArch64dup (f16 FPR16:$src))),
(DUP_ZZI_H (INSERT_SUBREG (IMPLICIT_DEF), FPR16:$src, hsub), 0)>; (DUP_ZZI_H (INSERT_SUBREG (IMPLICIT_DEF), FPR16:$src, hsub), 0)>;
▲ Show 20 Lines • Show All 241 Lines • ▼ Show 20 Lineslet Predicates = [HasSVE] in {
def LD3W : sve_mem_eld_ss<0b10, 0b10, ZZZ_s, "ld3w", GPR64NoXZRshifted32>; def LD3W : sve_mem_eld_ss<0b10, 0b10, ZZZ_s, "ld3w", GPR64NoXZRshifted32>;
def LD4W : sve_mem_eld_ss<0b10, 0b11, ZZZZ_s, "ld4w", GPR64NoXZRshifted32>; def LD4W : sve_mem_eld_ss<0b10, 0b11, ZZZZ_s, "ld4w", GPR64NoXZRshifted32>;
def LD2D : sve_mem_eld_ss<0b11, 0b01, ZZ_d, "ld2d", GPR64NoXZRshifted64>; def LD2D : sve_mem_eld_ss<0b11, 0b01, ZZ_d, "ld2d", GPR64NoXZRshifted64>;
def LD3D : sve_mem_eld_ss<0b11, 0b10, ZZZ_d, "ld3d", GPR64NoXZRshifted64>; def LD3D : sve_mem_eld_ss<0b11, 0b10, ZZZ_d, "ld3d", GPR64NoXZRshifted64>;
def LD4D : sve_mem_eld_ss<0b11, 0b11, ZZZZ_d, "ld4d", GPR64NoXZRshifted64>; def LD4D : sve_mem_eld_ss<0b11, 0b11, ZZZZ_d, "ld4d", GPR64NoXZRshifted64>;
// Gathers using unscaled 32-bit offsets, e.g. // Gathers using unscaled 32-bit offsets, e.g.
// ld1h z0.s, p0/z, [x0, z0.s, uxtw] // ld1h z0.s, p0/z, [x0, z0.s, uxtw]
defm GLD1SB_S : sve_mem_32b_gld_vs_32_unscaled<0b0000, "ld1sb", AArch64ld1s_gather_sxtw, AArch64ld1s_gather_uxtw, ZPR32ExtSXTW8Only, ZPR32ExtUXTW8Only, nxv4i8>; defm GLD1SB_S : sve_mem_32b_gld_vs_32_unscaled<0b0000, "ld1sb", AArch64ld1s_gather_sxtw_z, AArch64ld1s_gather_uxtw_z, ZPR32ExtSXTW8Only, ZPR32ExtUXTW8Only, nxv4i8>;
defm GLDFF1SB_S : sve_mem_32b_gld_vs_32_unscaled<0b0001, "ldff1sb", AArch64ldff1s_gather_sxtw, AArch64ldff1s_gather_uxtw, ZPR32ExtSXTW8Only, ZPR32ExtUXTW8Only, nxv4i8>; defm GLDFF1SB_S : sve_mem_32b_gld_vs_32_unscaled<0b0001, "ldff1sb", AArch64ldff1s_gather_sxtw_z, AArch64ldff1s_gather_uxtw_z, ZPR32ExtSXTW8Only, ZPR32ExtUXTW8Only, nxv4i8>;
defm GLD1B_S : sve_mem_32b_gld_vs_32_unscaled<0b0010, "ld1b", AArch64ld1_gather_sxtw, AArch64ld1_gather_uxtw, ZPR32ExtSXTW8Only, ZPR32ExtUXTW8Only, nxv4i8>; defm GLD1B_S : sve_mem_32b_gld_vs_32_unscaled<0b0010, "ld1b", AArch64ld1_gather_sxtw_z, AArch64ld1_gather_uxtw_z, ZPR32ExtSXTW8Only, ZPR32ExtUXTW8Only, nxv4i8>;
defm GLDFF1B_S : sve_mem_32b_gld_vs_32_unscaled<0b0011, "ldff1b", AArch64ldff1_gather_sxtw, AArch64ldff1_gather_uxtw, ZPR32ExtSXTW8Only, ZPR32ExtUXTW8Only, nxv4i8>; defm GLDFF1B_S : sve_mem_32b_gld_vs_32_unscaled<0b0011, "ldff1b", AArch64ldff1_gather_sxtw_z, AArch64ldff1_gather_uxtw_z, ZPR32ExtSXTW8Only, ZPR32ExtUXTW8Only, nxv4i8>;
defm GLD1SH_S : sve_mem_32b_gld_vs_32_unscaled<0b0100, "ld1sh", AArch64ld1s_gather_sxtw, AArch64ld1s_gather_uxtw, ZPR32ExtSXTW8, ZPR32ExtUXTW8, nxv4i16>; defm GLD1SH_S : sve_mem_32b_gld_vs_32_unscaled<0b0100, "ld1sh", AArch64ld1s_gather_sxtw_z, AArch64ld1s_gather_uxtw_z, ZPR32ExtSXTW8, ZPR32ExtUXTW8, nxv4i16>;
defm GLDFF1SH_S : sve_mem_32b_gld_vs_32_unscaled<0b0101, "ldff1sh", AArch64ldff1s_gather_sxtw, AArch64ldff1s_gather_uxtw, ZPR32ExtSXTW8, ZPR32ExtUXTW8, nxv4i16>; defm GLDFF1SH_S : sve_mem_32b_gld_vs_32_unscaled<0b0101, "ldff1sh", AArch64ldff1s_gather_sxtw_z, AArch64ldff1s_gather_uxtw_z, ZPR32ExtSXTW8, ZPR32ExtUXTW8, nxv4i16>;
defm GLD1H_S : sve_mem_32b_gld_vs_32_unscaled<0b0110, "ld1h", AArch64ld1_gather_sxtw, AArch64ld1_gather_uxtw, ZPR32ExtSXTW8, ZPR32ExtUXTW8, nxv4i16>; defm GLD1H_S : sve_mem_32b_gld_vs_32_unscaled<0b0110, "ld1h", AArch64ld1_gather_sxtw_z, AArch64ld1_gather_uxtw_z, ZPR32ExtSXTW8, ZPR32ExtUXTW8, nxv4i16>;
defm GLDFF1H_S : sve_mem_32b_gld_vs_32_unscaled<0b0111, "ldff1h", AArch64ldff1_gather_sxtw, AArch64ldff1_gather_uxtw, ZPR32ExtSXTW8, ZPR32ExtUXTW8, nxv4i16>; defm GLDFF1H_S : sve_mem_32b_gld_vs_32_unscaled<0b0111, "ldff1h", AArch64ldff1_gather_sxtw_z, AArch64ldff1_gather_uxtw_z, ZPR32ExtSXTW8, ZPR32ExtUXTW8, nxv4i16>;
defm GLD1W : sve_mem_32b_gld_vs_32_unscaled<0b1010, "ld1w", AArch64ld1_gather_sxtw, AArch64ld1_gather_uxtw, ZPR32ExtSXTW8, ZPR32ExtUXTW8, nxv4i32>; defm GLD1W : sve_mem_32b_gld_vs_32_unscaled<0b1010, "ld1w", AArch64ld1_gather_sxtw_z, AArch64ld1_gather_uxtw_z, ZPR32ExtSXTW8, ZPR32ExtUXTW8, nxv4i32>;
defm GLDFF1W : sve_mem_32b_gld_vs_32_unscaled<0b1011, "ldff1w", AArch64ldff1_gather_sxtw, AArch64ldff1_gather_uxtw, ZPR32ExtSXTW8, ZPR32ExtUXTW8, nxv4i32>; defm GLDFF1W : sve_mem_32b_gld_vs_32_unscaled<0b1011, "ldff1w", AArch64ldff1_gather_sxtw_z, AArch64ldff1_gather_uxtw_z, ZPR32ExtSXTW8, ZPR32ExtUXTW8, nxv4i32>;
// Gathers using scaled 32-bit offsets, e.g. // Gathers using scaled 32-bit offsets, e.g.
// ld1h z0.s, p0/z, [x0, z0.s, uxtw #1] // ld1h z0.s, p0/z, [x0, z0.s, uxtw #1]
defm GLD1SH_S : sve_mem_32b_gld_sv_32_scaled<0b0100, "ld1sh", AArch64ld1s_gather_sxtw_scaled, AArch64ld1s_gather_uxtw_scaled, ZPR32ExtSXTW16, ZPR32ExtUXTW16, nxv4i16>; defm GLD1SH_S : sve_mem_32b_gld_sv_32_scaled<0b0100, "ld1sh", AArch64ld1s_gather_sxtw_scaled_z, AArch64ld1s_gather_uxtw_scaled_z, ZPR32ExtSXTW16, ZPR32ExtUXTW16, nxv4i16>;
defm GLDFF1SH_S : sve_mem_32b_gld_sv_32_scaled<0b0101, "ldff1sh", AArch64ldff1s_gather_sxtw_scaled, AArch64ldff1s_gather_uxtw_scaled, ZPR32ExtSXTW16, ZPR32ExtUXTW16, nxv4i16>; defm GLDFF1SH_S : sve_mem_32b_gld_sv_32_scaled<0b0101, "ldff1sh", AArch64ldff1s_gather_sxtw_scaled_z, AArch64ldff1s_gather_uxtw_scaled_z, ZPR32ExtSXTW16, ZPR32ExtUXTW16, nxv4i16>;
defm GLD1H_S : sve_mem_32b_gld_sv_32_scaled<0b0110, "ld1h", AArch64ld1_gather_sxtw_scaled, AArch64ld1_gather_uxtw_scaled, ZPR32ExtSXTW16, ZPR32ExtUXTW16, nxv4i16>; defm GLD1H_S : sve_mem_32b_gld_sv_32_scaled<0b0110, "ld1h", AArch64ld1_gather_sxtw_scaled_z, AArch64ld1_gather_uxtw_scaled_z, ZPR32ExtSXTW16, ZPR32ExtUXTW16, nxv4i16>;
defm GLDFF1H_S : sve_mem_32b_gld_sv_32_scaled<0b0111, "ldff1h", AArch64ldff1_gather_sxtw_scaled, AArch64ldff1_gather_uxtw_scaled, ZPR32ExtSXTW16, ZPR32ExtUXTW16, nxv4i16>; defm GLDFF1H_S : sve_mem_32b_gld_sv_32_scaled<0b0111, "ldff1h", AArch64ldff1_gather_sxtw_scaled_z, AArch64ldff1_gather_uxtw_scaled_z, ZPR32ExtSXTW16, ZPR32ExtUXTW16, nxv4i16>;
defm GLD1W : sve_mem_32b_gld_sv_32_scaled<0b1010, "ld1w", AArch64ld1_gather_sxtw_scaled, AArch64ld1_gather_uxtw_scaled, ZPR32ExtSXTW32, ZPR32ExtUXTW32, nxv4i32>; defm GLD1W : sve_mem_32b_gld_sv_32_scaled<0b1010, "ld1w", AArch64ld1_gather_sxtw_scaled_z, AArch64ld1_gather_uxtw_scaled_z, ZPR32ExtSXTW32, ZPR32ExtUXTW32, nxv4i32>;
defm GLDFF1W : sve_mem_32b_gld_sv_32_scaled<0b1011, "ldff1w", AArch64ldff1_gather_sxtw_scaled, AArch64ldff1_gather_uxtw_scaled, ZPR32ExtSXTW32, ZPR32ExtUXTW32, nxv4i32>; defm GLDFF1W : sve_mem_32b_gld_sv_32_scaled<0b1011, "ldff1w", AArch64ldff1_gather_sxtw_scaled_z, AArch64ldff1_gather_uxtw_scaled_z, ZPR32ExtSXTW32, ZPR32ExtUXTW32, nxv4i32>;
// Gathers using 32-bit pointers with scaled offset, e.g. // Gathers using 32-bit pointers with scaled offset, e.g.
// ld1h z0.s, p0/z, [z0.s, #16] // ld1h z0.s, p0/z, [z0.s, #16]
defm GLD1SB_S : sve_mem_32b_gld_vi_32_ptrs<0b0000, "ld1sb", imm0_31, AArch64ld1s_gather_imm, nxv4i8>; defm GLD1SB_S : sve_mem_32b_gld_vi_32_ptrs<0b0000, "ld1sb", imm0_31, AArch64ld1s_gather_imm_z, nxv4i8>;
defm GLDFF1SB_S : sve_mem_32b_gld_vi_32_ptrs<0b0001, "ldff1sb", imm0_31, AArch64ldff1s_gather_imm, nxv4i8>; defm GLDFF1SB_S : sve_mem_32b_gld_vi_32_ptrs<0b0001, "ldff1sb", imm0_31, AArch64ldff1s_gather_imm_z, nxv4i8>;
defm GLD1B_S : sve_mem_32b_gld_vi_32_ptrs<0b0010, "ld1b", imm0_31, AArch64ld1_gather_imm, nxv4i8>; defm GLD1B_S : sve_mem_32b_gld_vi_32_ptrs<0b0010, "ld1b", imm0_31, AArch64ld1_gather_imm_z, nxv4i8>;
defm GLDFF1B_S : sve_mem_32b_gld_vi_32_ptrs<0b0011, "ldff1b", imm0_31, AArch64ldff1_gather_imm, nxv4i8>; defm GLDFF1B_S : sve_mem_32b_gld_vi_32_ptrs<0b0011, "ldff1b", imm0_31, AArch64ldff1_gather_imm_z, nxv4i8>;
defm GLD1SH_S : sve_mem_32b_gld_vi_32_ptrs<0b0100, "ld1sh", uimm5s2, AArch64ld1s_gather_imm, nxv4i16>; defm GLD1SH_S : sve_mem_32b_gld_vi_32_ptrs<0b0100, "ld1sh", uimm5s2, AArch64ld1s_gather_imm_z, nxv4i16>;
defm GLDFF1SH_S : sve_mem_32b_gld_vi_32_ptrs<0b0101, "ldff1sh", uimm5s2, AArch64ldff1s_gather_imm, nxv4i16>; defm GLDFF1SH_S : sve_mem_32b_gld_vi_32_ptrs<0b0101, "ldff1sh", uimm5s2, AArch64ldff1s_gather_imm_z, nxv4i16>;
defm GLD1H_S : sve_mem_32b_gld_vi_32_ptrs<0b0110, "ld1h", uimm5s2, AArch64ld1_gather_imm, nxv4i16>; defm GLD1H_S : sve_mem_32b_gld_vi_32_ptrs<0b0110, "ld1h", uimm5s2, AArch64ld1_gather_imm_z, nxv4i16>;
defm GLDFF1H_S : sve_mem_32b_gld_vi_32_ptrs<0b0111, "ldff1h", uimm5s2, AArch64ldff1_gather_imm, nxv4i16>; defm GLDFF1H_S : sve_mem_32b_gld_vi_32_ptrs<0b0111, "ldff1h", uimm5s2, AArch64ldff1_gather_imm_z, nxv4i16>;
defm GLD1W : sve_mem_32b_gld_vi_32_ptrs<0b1010, "ld1w", uimm5s4, AArch64ld1_gather_imm, nxv4i32>; defm GLD1W : sve_mem_32b_gld_vi_32_ptrs<0b1010, "ld1w", uimm5s4, AArch64ld1_gather_imm_z, nxv4i32>;
defm GLDFF1W : sve_mem_32b_gld_vi_32_ptrs<0b1011, "ldff1w", uimm5s4, AArch64ldff1_gather_imm, nxv4i32>; defm GLDFF1W : sve_mem_32b_gld_vi_32_ptrs<0b1011, "ldff1w", uimm5s4, AArch64ldff1_gather_imm_z, nxv4i32>;
// Gathers using 64-bit pointers with scaled offset, e.g. // Gathers using 64-bit pointers with scaled offset, e.g.
// ld1h z0.d, p0/z, [z0.d, #16] // ld1h z0.d, p0/z, [z0.d, #16]
defm GLD1SB_D : sve_mem_64b_gld_vi_64_ptrs<0b0000, "ld1sb", imm0_31, AArch64ld1s_gather_imm, nxv2i8>; defm GLD1SB_D : sve_mem_64b_gld_vi_64_ptrs<0b0000, "ld1sb", imm0_31, AArch64ld1s_gather_imm_z, nxv2i8>;
defm GLDFF1SB_D : sve_mem_64b_gld_vi_64_ptrs<0b0001, "ldff1sb", imm0_31, AArch64ldff1s_gather_imm, nxv2i8>; defm GLDFF1SB_D : sve_mem_64b_gld_vi_64_ptrs<0b0001, "ldff1sb", imm0_31, AArch64ldff1s_gather_imm_z, nxv2i8>;
defm GLD1B_D : sve_mem_64b_gld_vi_64_ptrs<0b0010, "ld1b", imm0_31, AArch64ld1_gather_imm, nxv2i8>; defm GLD1B_D : sve_mem_64b_gld_vi_64_ptrs<0b0010, "ld1b", imm0_31, AArch64ld1_gather_imm_z, nxv2i8>;
defm GLDFF1B_D : sve_mem_64b_gld_vi_64_ptrs<0b0011, "ldff1b", imm0_31, AArch64ldff1_gather_imm, nxv2i8>; defm GLDFF1B_D : sve_mem_64b_gld_vi_64_ptrs<0b0011, "ldff1b", imm0_31, AArch64ldff1_gather_imm_z, nxv2i8>;
defm GLD1SH_D : sve_mem_64b_gld_vi_64_ptrs<0b0100, "ld1sh", uimm5s2, AArch64ld1s_gather_imm, nxv2i16>; defm GLD1SH_D : sve_mem_64b_gld_vi_64_ptrs<0b0100, "ld1sh", uimm5s2, AArch64ld1s_gather_imm_z, nxv2i16>;
defm GLDFF1SH_D : sve_mem_64b_gld_vi_64_ptrs<0b0101, "ldff1sh", uimm5s2, AArch64ldff1s_gather_imm, nxv2i16>; defm GLDFF1SH_D : sve_mem_64b_gld_vi_64_ptrs<0b0101, "ldff1sh", uimm5s2, AArch64ldff1s_gather_imm_z, nxv2i16>;
defm GLD1H_D : sve_mem_64b_gld_vi_64_ptrs<0b0110, "ld1h", uimm5s2, AArch64ld1_gather_imm, nxv2i16>; defm GLD1H_D : sve_mem_64b_gld_vi_64_ptrs<0b0110, "ld1h", uimm5s2, AArch64ld1_gather_imm_z, nxv2i16>;
defm GLDFF1H_D : sve_mem_64b_gld_vi_64_ptrs<0b0111, "ldff1h", uimm5s2, AArch64ldff1_gather_imm, nxv2i16>; defm GLDFF1H_D : sve_mem_64b_gld_vi_64_ptrs<0b0111, "ldff1h", uimm5s2, AArch64ldff1_gather_imm_z, nxv2i16>;
defm GLD1SW_D : sve_mem_64b_gld_vi_64_ptrs<0b1000, "ld1sw", uimm5s4, AArch64ld1s_gather_imm, nxv2i32>; defm GLD1SW_D : sve_mem_64b_gld_vi_64_ptrs<0b1000, "ld1sw", uimm5s4, AArch64ld1s_gather_imm_z, nxv2i32>;
defm GLDFF1SW_D : sve_mem_64b_gld_vi_64_ptrs<0b1001, "ldff1sw", uimm5s4, AArch64ldff1s_gather_imm, nxv2i32>; defm GLDFF1SW_D : sve_mem_64b_gld_vi_64_ptrs<0b1001, "ldff1sw", uimm5s4, AArch64ldff1s_gather_imm_z, nxv2i32>;
defm GLD1W_D : sve_mem_64b_gld_vi_64_ptrs<0b1010, "ld1w", uimm5s4, AArch64ld1_gather_imm, nxv2i32>; defm GLD1W_D : sve_mem_64b_gld_vi_64_ptrs<0b1010, "ld1w", uimm5s4, AArch64ld1_gather_imm_z, nxv2i32>;
defm GLDFF1W_D : sve_mem_64b_gld_vi_64_ptrs<0b1011, "ldff1w", uimm5s4, AArch64ldff1_gather_imm, nxv2i32>; defm GLDFF1W_D : sve_mem_64b_gld_vi_64_ptrs<0b1011, "ldff1w", uimm5s4, AArch64ldff1_gather_imm_z, nxv2i32>;
defm GLD1D : sve_mem_64b_gld_vi_64_ptrs<0b1110, "ld1d", uimm5s8, AArch64ld1_gather_imm, nxv2i64>; defm GLD1D : sve_mem_64b_gld_vi_64_ptrs<0b1110, "ld1d", uimm5s8, AArch64ld1_gather_imm_z, nxv2i64>;
defm GLDFF1D : sve_mem_64b_gld_vi_64_ptrs<0b1111, "ldff1d", uimm5s8, AArch64ldff1_gather_imm, nxv2i64>; defm GLDFF1D : sve_mem_64b_gld_vi_64_ptrs<0b1111, "ldff1d", uimm5s8, AArch64ldff1_gather_imm_z, nxv2i64>;
// Gathers using unscaled 64-bit offsets, e.g. // Gathers using unscaled 64-bit offsets, e.g.
// ld1h z0.d, p0/z, [x0, z0.d] // ld1h z0.d, p0/z, [x0, z0.d]
defm GLD1SB_D : sve_mem_64b_gld_vs2_64_unscaled<0b0000, "ld1sb", AArch64ld1s_gather, nxv2i8>; defm GLD1SB_D : sve_mem_64b_gld_vs2_64_unscaled<0b0000, "ld1sb", AArch64ld1s_gather_z, nxv2i8>;
defm GLDFF1SB_D : sve_mem_64b_gld_vs2_64_unscaled<0b0001, "ldff1sb", AArch64ldff1s_gather, nxv2i8>; defm GLDFF1SB_D : sve_mem_64b_gld_vs2_64_unscaled<0b0001, "ldff1sb", AArch64ldff1s_gather_z, nxv2i8>;
defm GLD1B_D : sve_mem_64b_gld_vs2_64_unscaled<0b0010, "ld1b", AArch64ld1_gather, nxv2i8>; defm GLD1B_D : sve_mem_64b_gld_vs2_64_unscaled<0b0010, "ld1b", AArch64ld1_gather_z, nxv2i8>;
defm GLDFF1B_D : sve_mem_64b_gld_vs2_64_unscaled<0b0011, "ldff1b", AArch64ldff1_gather, nxv2i8>; defm GLDFF1B_D : sve_mem_64b_gld_vs2_64_unscaled<0b0011, "ldff1b", AArch64ldff1_gather_z, nxv2i8>;
defm GLD1SH_D : sve_mem_64b_gld_vs2_64_unscaled<0b0100, "ld1sh", AArch64ld1s_gather, nxv2i16>; defm GLD1SH_D : sve_mem_64b_gld_vs2_64_unscaled<0b0100, "ld1sh", AArch64ld1s_gather_z, nxv2i16>;
defm GLDFF1SH_D : sve_mem_64b_gld_vs2_64_unscaled<0b0101, "ldff1sh", AArch64ldff1s_gather, nxv2i16>; defm GLDFF1SH_D : sve_mem_64b_gld_vs2_64_unscaled<0b0101, "ldff1sh", AArch64ldff1s_gather_z, nxv2i16>;
defm GLD1H_D : sve_mem_64b_gld_vs2_64_unscaled<0b0110, "ld1h", AArch64ld1_gather, nxv2i16>; defm GLD1H_D : sve_mem_64b_gld_vs2_64_unscaled<0b0110, "ld1h", AArch64ld1_gather_z, nxv2i16>;
defm GLDFF1H_D : sve_mem_64b_gld_vs2_64_unscaled<0b0111, "ldff1h", AArch64ldff1_gather, nxv2i16>; defm GLDFF1H_D : sve_mem_64b_gld_vs2_64_unscaled<0b0111, "ldff1h", AArch64ldff1_gather_z, nxv2i16>;
defm GLD1SW_D : sve_mem_64b_gld_vs2_64_unscaled<0b1000, "ld1sw", AArch64ld1s_gather, nxv2i32>; defm GLD1SW_D : sve_mem_64b_gld_vs2_64_unscaled<0b1000, "ld1sw", AArch64ld1s_gather_z, nxv2i32>;
defm GLDFF1SW_D : sve_mem_64b_gld_vs2_64_unscaled<0b1001, "ldff1sw", AArch64ldff1s_gather, nxv2i32>; defm GLDFF1SW_D : sve_mem_64b_gld_vs2_64_unscaled<0b1001, "ldff1sw", AArch64ldff1s_gather_z, nxv2i32>;
defm GLD1W_D : sve_mem_64b_gld_vs2_64_unscaled<0b1010, "ld1w", AArch64ld1_gather, nxv2i32>; defm GLD1W_D : sve_mem_64b_gld_vs2_64_unscaled<0b1010, "ld1w", AArch64ld1_gather_z, nxv2i32>;
defm GLDFF1W_D : sve_mem_64b_gld_vs2_64_unscaled<0b1011, "ldff1w", AArch64ldff1_gather, nxv2i32>; defm GLDFF1W_D : sve_mem_64b_gld_vs2_64_unscaled<0b1011, "ldff1w", AArch64ldff1_gather_z, nxv2i32>;
defm GLD1D : sve_mem_64b_gld_vs2_64_unscaled<0b1110, "ld1d", AArch64ld1_gather, nxv2i64>; defm GLD1D : sve_mem_64b_gld_vs2_64_unscaled<0b1110, "ld1d", AArch64ld1_gather_z, nxv2i64>;
defm GLDFF1D : sve_mem_64b_gld_vs2_64_unscaled<0b1111, "ldff1d", AArch64ldff1_gather, nxv2i64>; defm GLDFF1D : sve_mem_64b_gld_vs2_64_unscaled<0b1111, "ldff1d", AArch64ldff1_gather_z, nxv2i64>;
// Gathers using scaled 64-bit offsets, e.g. // Gathers using scaled 64-bit offsets, e.g.
// ld1h z0.d, p0/z, [x0, z0.d, lsl #1] // ld1h z0.d, p0/z, [x0, z0.d, lsl #1]
defm GLD1SH_D : sve_mem_64b_gld_sv2_64_scaled<0b0100, "ld1sh", AArch64ld1s_gather_scaled, ZPR64ExtLSL16, nxv2i16>; defm GLD1SH_D : sve_mem_64b_gld_sv2_64_scaled<0b0100, "ld1sh", AArch64ld1s_gather_scaled_z, ZPR64ExtLSL16, nxv2i16>;
defm GLDFF1SH_D : sve_mem_64b_gld_sv2_64_scaled<0b0101, "ldff1sh", AArch64ldff1s_gather_scaled, ZPR64ExtLSL16, nxv2i16>; defm GLDFF1SH_D : sve_mem_64b_gld_sv2_64_scaled<0b0101, "ldff1sh", AArch64ldff1s_gather_scaled_z, ZPR64ExtLSL16, nxv2i16>;
defm GLD1H_D : sve_mem_64b_gld_sv2_64_scaled<0b0110, "ld1h", AArch64ld1_gather_scaled, ZPR64ExtLSL16, nxv2i16>; defm GLD1H_D : sve_mem_64b_gld_sv2_64_scaled<0b0110, "ld1h", AArch64ld1_gather_scaled_z, ZPR64ExtLSL16, nxv2i16>;
defm GLDFF1H_D : sve_mem_64b_gld_sv2_64_scaled<0b0111, "ldff1h", AArch64ldff1_gather_scaled, ZPR64ExtLSL16, nxv2i16>; defm GLDFF1H_D : sve_mem_64b_gld_sv2_64_scaled<0b0111, "ldff1h", AArch64ldff1_gather_scaled_z, ZPR64ExtLSL16, nxv2i16>;
defm GLD1SW_D : sve_mem_64b_gld_sv2_64_scaled<0b1000, "ld1sw", AArch64ld1s_gather_scaled, ZPR64ExtLSL32, nxv2i32>; defm GLD1SW_D : sve_mem_64b_gld_sv2_64_scaled<0b1000, "ld1sw", AArch64ld1s_gather_scaled_z, ZPR64ExtLSL32, nxv2i32>;
defm GLDFF1SW_D : sve_mem_64b_gld_sv2_64_scaled<0b1001, "ldff1sw", AArch64ldff1s_gather_scaled, ZPR64ExtLSL32, nxv2i32>; defm GLDFF1SW_D : sve_mem_64b_gld_sv2_64_scaled<0b1001, "ldff1sw", AArch64ldff1s_gather_scaled_z, ZPR64ExtLSL32, nxv2i32>;
defm GLD1W_D : sve_mem_64b_gld_sv2_64_scaled<0b1010, "ld1w", AArch64ld1_gather_scaled, ZPR64ExtLSL32, nxv2i32>; defm GLD1W_D : sve_mem_64b_gld_sv2_64_scaled<0b1010, "ld1w", AArch64ld1_gather_scaled_z, ZPR64ExtLSL32, nxv2i32>;
defm GLDFF1W_D : sve_mem_64b_gld_sv2_64_scaled<0b1011, "ldff1w", AArch64ldff1_gather_scaled, ZPR64ExtLSL32, nxv2i32>; defm GLDFF1W_D : sve_mem_64b_gld_sv2_64_scaled<0b1011, "ldff1w", AArch64ldff1_gather_scaled_z, ZPR64ExtLSL32, nxv2i32>;
defm GLD1D : sve_mem_64b_gld_sv2_64_scaled<0b1110, "ld1d", AArch64ld1_gather_scaled, ZPR64ExtLSL64, nxv2i64>; defm GLD1D : sve_mem_64b_gld_sv2_64_scaled<0b1110, "ld1d", AArch64ld1_gather_scaled_z, ZPR64ExtLSL64, nxv2i64>;
defm GLDFF1D : sve_mem_64b_gld_sv2_64_scaled<0b1111, "ldff1d", AArch64ldff1_gather_scaled, ZPR64ExtLSL64, nxv2i64>; defm GLDFF1D : sve_mem_64b_gld_sv2_64_scaled<0b1111, "ldff1d", AArch64ldff1_gather_scaled_z, ZPR64ExtLSL64, nxv2i64>;
// Gathers using unscaled 32-bit offsets unpacked in 64-bits elements, e.g. // Gathers using unscaled 32-bit offsets unpacked in 64-bits elements, e.g.
// ld1h z0.d, p0/z, [x0, z0.d, uxtw] // ld1h z0.d, p0/z, [x0, z0.d, uxtw]
defm GLD1SB_D : sve_mem_64b_gld_vs_32_unscaled<0b0000, "ld1sb", AArch64ld1s_gather_sxtw, AArch64ld1s_gather_uxtw, ZPR64ExtSXTW8Only, ZPR64ExtUXTW8Only, nxv2i8>; defm GLD1SB_D : sve_mem_64b_gld_vs_32_unscaled<0b0000, "ld1sb", AArch64ld1s_gather_sxtw_z, AArch64ld1s_gather_uxtw_z, ZPR64ExtSXTW8Only, ZPR64ExtUXTW8Only, nxv2i8>;
defm GLDFF1SB_D : sve_mem_64b_gld_vs_32_unscaled<0b0001, "ldff1sb", AArch64ldff1s_gather_sxtw, AArch64ldff1s_gather_uxtw, ZPR64ExtSXTW8Only, ZPR64ExtUXTW8Only, nxv2i8>; defm GLDFF1SB_D : sve_mem_64b_gld_vs_32_unscaled<0b0001, "ldff1sb", AArch64ldff1s_gather_sxtw_z, AArch64ldff1s_gather_uxtw_z, ZPR64ExtSXTW8Only, ZPR64ExtUXTW8Only, nxv2i8>;
defm GLD1B_D : sve_mem_64b_gld_vs_32_unscaled<0b0010, "ld1b", AArch64ld1_gather_sxtw, AArch64ld1_gather_uxtw, ZPR64ExtSXTW8Only, ZPR64ExtUXTW8Only, nxv2i8>; defm GLD1B_D : sve_mem_64b_gld_vs_32_unscaled<0b0010, "ld1b", AArch64ld1_gather_sxtw_z, AArch64ld1_gather_uxtw_z, ZPR64ExtSXTW8Only, ZPR64ExtUXTW8Only, nxv2i8>;
defm GLDFF1B_D : sve_mem_64b_gld_vs_32_unscaled<0b0011, "ldff1b", AArch64ldff1_gather_sxtw, AArch64ldff1_gather_uxtw, ZPR64ExtSXTW8Only, ZPR64ExtUXTW8Only, nxv2i8>; defm GLDFF1B_D : sve_mem_64b_gld_vs_32_unscaled<0b0011, "ldff1b", AArch64ldff1_gather_sxtw_z, AArch64ldff1_gather_uxtw_z, ZPR64ExtSXTW8Only, ZPR64ExtUXTW8Only, nxv2i8>;
defm GLD1SH_D : sve_mem_64b_gld_vs_32_unscaled<0b0100, "ld1sh", AArch64ld1s_gather_sxtw, AArch64ld1s_gather_uxtw, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i16>; defm GLD1SH_D : sve_mem_64b_gld_vs_32_unscaled<0b0100, "ld1sh", AArch64ld1s_gather_sxtw_z, AArch64ld1s_gather_uxtw_z, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i16>;
defm GLDFF1SH_D : sve_mem_64b_gld_vs_32_unscaled<0b0101, "ldff1sh", AArch64ldff1s_gather_sxtw, AArch64ldff1s_gather_uxtw, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i16>; defm GLDFF1SH_D : sve_mem_64b_gld_vs_32_unscaled<0b0101, "ldff1sh", AArch64ldff1s_gather_sxtw_z, AArch64ldff1s_gather_uxtw_z, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i16>;
defm GLD1H_D : sve_mem_64b_gld_vs_32_unscaled<0b0110, "ld1h", AArch64ld1_gather_sxtw, AArch64ld1_gather_uxtw, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i16>; defm GLD1H_D : sve_mem_64b_gld_vs_32_unscaled<0b0110, "ld1h", AArch64ld1_gather_sxtw_z, AArch64ld1_gather_uxtw_z, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i16>;
defm GLDFF1H_D : sve_mem_64b_gld_vs_32_unscaled<0b0111, "ldff1h", AArch64ldff1_gather_sxtw, AArch64ldff1_gather_uxtw, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i16>; defm GLDFF1H_D : sve_mem_64b_gld_vs_32_unscaled<0b0111, "ldff1h", AArch64ldff1_gather_sxtw_z, AArch64ldff1_gather_uxtw_z, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i16>;
defm GLD1SW_D : sve_mem_64b_gld_vs_32_unscaled<0b1000, "ld1sw", AArch64ld1s_gather_sxtw, AArch64ld1s_gather_uxtw, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i32>; defm GLD1SW_D : sve_mem_64b_gld_vs_32_unscaled<0b1000, "ld1sw", AArch64ld1s_gather_sxtw_z, AArch64ld1s_gather_uxtw_z, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i32>;
defm GLDFF1SW_D : sve_mem_64b_gld_vs_32_unscaled<0b1001, "ldff1sw", AArch64ldff1s_gather_sxtw, AArch64ldff1s_gather_uxtw, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i32>; defm GLDFF1SW_D : sve_mem_64b_gld_vs_32_unscaled<0b1001, "ldff1sw", AArch64ldff1s_gather_sxtw_z, AArch64ldff1s_gather_uxtw_z, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i32>;
defm GLD1W_D : sve_mem_64b_gld_vs_32_unscaled<0b1010, "ld1w", AArch64ld1_gather_sxtw, AArch64ld1_gather_uxtw, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i32>; defm GLD1W_D : sve_mem_64b_gld_vs_32_unscaled<0b1010, "ld1w", AArch64ld1_gather_sxtw_z, AArch64ld1_gather_uxtw_z, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i32>;
defm GLDFF1W_D : sve_mem_64b_gld_vs_32_unscaled<0b1011, "ldff1w", AArch64ldff1_gather_sxtw, AArch64ldff1_gather_uxtw, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i32>; defm GLDFF1W_D : sve_mem_64b_gld_vs_32_unscaled<0b1011, "ldff1w", AArch64ldff1_gather_sxtw_z, AArch64ldff1_gather_uxtw_z, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i32>;
defm GLD1D : sve_mem_64b_gld_vs_32_unscaled<0b1110, "ld1d", AArch64ld1_gather_sxtw, AArch64ld1_gather_uxtw, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i64>; defm GLD1D : sve_mem_64b_gld_vs_32_unscaled<0b1110, "ld1d", AArch64ld1_gather_sxtw_z, AArch64ld1_gather_uxtw_z, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i64>;
defm GLDFF1D : sve_mem_64b_gld_vs_32_unscaled<0b1111, "ldff1d", AArch64ldff1_gather_sxtw, AArch64ldff1_gather_uxtw, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i64>; defm GLDFF1D : sve_mem_64b_gld_vs_32_unscaled<0b1111, "ldff1d", AArch64ldff1_gather_sxtw_z, AArch64ldff1_gather_uxtw_z, ZPR64ExtSXTW8, ZPR64ExtUXTW8, nxv2i64>;
// Gathers using scaled 32-bit offsets unpacked in 64-bits elements, e.g. // Gathers using scaled 32-bit offsets unpacked in 64-bits elements, e.g.
// ld1h z0.d, p0/z, [x0, z0.d, uxtw #1] // ld1h z0.d, p0/z, [x0, z0.d, uxtw #1]
defm GLD1SH_D : sve_mem_64b_gld_sv_32_scaled<0b0100, "ld1sh", AArch64ld1s_gather_sxtw_scaled, AArch64ld1s_gather_uxtw_scaled, ZPR64ExtSXTW16, ZPR64ExtUXTW16, nxv2i16>; defm GLD1SH_D : sve_mem_64b_gld_sv_32_scaled<0b0100, "ld1sh", AArch64ld1s_gather_sxtw_scaled_z, AArch64ld1s_gather_uxtw_scaled_z, ZPR64ExtSXTW16, ZPR64ExtUXTW16, nxv2i16>;
defm GLDFF1SH_D : sve_mem_64b_gld_sv_32_scaled<0b0101, "ldff1sh", AArch64ldff1s_gather_sxtw_scaled, AArch64ldff1s_gather_uxtw_scaled, ZPR64ExtSXTW16, ZPR64ExtUXTW16, nxv2i16>; defm GLDFF1SH_D : sve_mem_64b_gld_sv_32_scaled<0b0101, "ldff1sh", AArch64ldff1s_gather_sxtw_scaled_z, AArch64ldff1s_gather_uxtw_scaled_z, ZPR64ExtSXTW16, ZPR64ExtUXTW16, nxv2i16>;
defm GLD1H_D : sve_mem_64b_gld_sv_32_scaled<0b0110, "ld1h", AArch64ld1_gather_sxtw_scaled, AArch64ld1_gather_uxtw_scaled, ZPR64ExtSXTW16, ZPR64ExtUXTW16, nxv2i16>; defm GLD1H_D : sve_mem_64b_gld_sv_32_scaled<0b0110, "ld1h", AArch64ld1_gather_sxtw_scaled_z, AArch64ld1_gather_uxtw_scaled_z, ZPR64ExtSXTW16, ZPR64ExtUXTW16, nxv2i16>;
defm GLDFF1H_D : sve_mem_64b_gld_sv_32_scaled<0b0111, "ldff1h", AArch64ldff1_gather_sxtw_scaled, AArch64ldff1_gather_uxtw_scaled, ZPR64ExtSXTW16, ZPR64ExtUXTW16, nxv2i16>; defm GLDFF1H_D : sve_mem_64b_gld_sv_32_scaled<0b0111, "ldff1h", AArch64ldff1_gather_sxtw_scaled_z, AArch64ldff1_gather_uxtw_scaled_z, ZPR64ExtSXTW16, ZPR64ExtUXTW16, nxv2i16>;
defm GLD1SW_D : sve_mem_64b_gld_sv_32_scaled<0b1000, "ld1sw", AArch64ld1s_gather_sxtw_scaled, AArch64ld1s_gather_uxtw_scaled, ZPR64ExtSXTW32, ZPR64ExtUXTW32, nxv2i32>; defm GLD1SW_D : sve_mem_64b_gld_sv_32_scaled<0b1000, "ld1sw", AArch64ld1s_gather_sxtw_scaled_z, AArch64ld1s_gather_uxtw_scaled_z, ZPR64ExtSXTW32, ZPR64ExtUXTW32, nxv2i32>;
defm GLDFF1SW_D : sve_mem_64b_gld_sv_32_scaled<0b1001, "ldff1sw", AArch64ldff1s_gather_sxtw_scaled, AArch64ldff1s_gather_uxtw_scaled, ZPR64ExtSXTW32, ZPR64ExtUXTW32, nxv2i32>; defm GLDFF1SW_D : sve_mem_64b_gld_sv_32_scaled<0b1001, "ldff1sw", AArch64ldff1s_gather_sxtw_scaled_z, AArch64ldff1s_gather_uxtw_scaled_z, ZPR64ExtSXTW32, ZPR64ExtUXTW32, nxv2i32>;
defm GLD1W_D : sve_mem_64b_gld_sv_32_scaled<0b1010, "ld1w", AArch64ld1_gather_sxtw_scaled, AArch64ld1_gather_uxtw_scaled, ZPR64ExtSXTW32, ZPR64ExtUXTW32, nxv2i32>; defm GLD1W_D : sve_mem_64b_gld_sv_32_scaled<0b1010, "ld1w", AArch64ld1_gather_sxtw_scaled_z, AArch64ld1_gather_uxtw_scaled_z, ZPR64ExtSXTW32, ZPR64ExtUXTW32, nxv2i32>;
defm GLDFF1W_D : sve_mem_64b_gld_sv_32_scaled<0b1011, "ldff1w", AArch64ldff1_gather_sxtw_scaled, AArch64ldff1_gather_uxtw_scaled, ZPR64ExtSXTW32, ZPR64ExtUXTW32, nxv2i32>; defm GLDFF1W_D : sve_mem_64b_gld_sv_32_scaled<0b1011, "ldff1w", AArch64ldff1_gather_sxtw_scaled_z, AArch64ldff1_gather_uxtw_scaled_z, ZPR64ExtSXTW32, ZPR64ExtUXTW32, nxv2i32>;
defm GLD1D : sve_mem_64b_gld_sv_32_scaled<0b1110, "ld1d", AArch64ld1_gather_sxtw_scaled, AArch64ld1_gather_uxtw_scaled, ZPR64ExtSXTW64, ZPR64ExtUXTW64, nxv2i64>; defm GLD1D : sve_mem_64b_gld_sv_32_scaled<0b1110, "ld1d", AArch64ld1_gather_sxtw_scaled_z, AArch64ld1_gather_uxtw_scaled_z, ZPR64ExtSXTW64, ZPR64ExtUXTW64, nxv2i64>;
defm GLDFF1D : sve_mem_64b_gld_sv_32_scaled<0b1111, "ldff1d", AArch64ldff1_gather_sxtw_scaled, AArch64ldff1_gather_uxtw_scaled, ZPR64ExtSXTW64, ZPR64ExtUXTW64, nxv2i64>; defm GLDFF1D : sve_mem_64b_gld_sv_32_scaled<0b1111, "ldff1d", AArch64ldff1_gather_sxtw_scaled_z, AArch64ldff1_gather_uxtw_scaled_z, ZPR64ExtSXTW64, ZPR64ExtUXTW64, nxv2i64>;
// Non-temporal contiguous loads (register + immediate) // Non-temporal contiguous loads (register + immediate)
defm LDNT1B_ZRI : sve_mem_cldnt_si<0b00, "ldnt1b", Z_b, ZPR8>; defm LDNT1B_ZRI : sve_mem_cldnt_si<0b00, "ldnt1b", Z_b, ZPR8>;
defm LDNT1H_ZRI : sve_mem_cldnt_si<0b01, "ldnt1h", Z_h, ZPR16>; defm LDNT1H_ZRI : sve_mem_cldnt_si<0b01, "ldnt1h", Z_h, ZPR16>;
defm LDNT1W_ZRI : sve_mem_cldnt_si<0b10, "ldnt1w", Z_s, ZPR32>; defm LDNT1W_ZRI : sve_mem_cldnt_si<0b10, "ldnt1w", Z_s, ZPR32>;
defm LDNT1D_ZRI : sve_mem_cldnt_si<0b11, "ldnt1d", Z_d, ZPR64>; defm LDNT1D_ZRI : sve_mem_cldnt_si<0b11, "ldnt1d", Z_d, ZPR64>;
// Non-temporal contiguous loads (register + register) // Non-temporal contiguous loads (register + register)
▲ Show 20 Lines • Show All 382 Lines • ▼ Show 20 Linesmulticlass sve_prefetch<SDPatternOperator prefetch, ValueType PredTy, Instruction RegImmInst, Instruction RegRegInst, int scale, ComplexPattern AddrCP> {
defm DECP_ZP : sve_int_count_v<0b10100, "decp">; defm DECP_ZP : sve_int_count_v<0b10100, "decp">;
defm INDEX_RR : sve_int_index_rr<"index", index_vector>; defm INDEX_RR : sve_int_index_rr<"index", index_vector>;
defm INDEX_IR : sve_int_index_ir<"index", index_vector>; defm INDEX_IR : sve_int_index_ir<"index", index_vector>;
defm INDEX_RI : sve_int_index_ri<"index", index_vector>; defm INDEX_RI : sve_int_index_ri<"index", index_vector>;
defm INDEX_II : sve_int_index_ii<"index", index_vector>; defm INDEX_II : sve_int_index_ii<"index", index_vector>;
// Unpredicated shifts // Unpredicated shifts
defm ASR_ZZI : sve_int_bin_cons_shift_imm_right<0b00, "asr", AArch64asr_pred>; defm ASR_ZZI : sve_int_bin_cons_shift_imm_right<0b00, "asr", AArch64asr_m>;
defm LSR_ZZI : sve_int_bin_cons_shift_imm_right<0b01, "lsr", AArch64lsr_pred>; defm LSR_ZZI : sve_int_bin_cons_shift_imm_right<0b01, "lsr", AArch64lsr_m>;
defm LSL_ZZI : sve_int_bin_cons_shift_imm_left< 0b11, "lsl", AArch64lsl_pred>; defm LSL_ZZI : sve_int_bin_cons_shift_imm_left< 0b11, "lsl", AArch64lsl_m>;
defm ASR_WIDE_ZZZ : sve_int_bin_cons_shift_wide<0b00, "asr">; defm ASR_WIDE_ZZZ : sve_int_bin_cons_shift_wide<0b00, "asr">;
defm LSR_WIDE_ZZZ : sve_int_bin_cons_shift_wide<0b01, "lsr">; defm LSR_WIDE_ZZZ : sve_int_bin_cons_shift_wide<0b01, "lsr">;
defm LSL_WIDE_ZZZ : sve_int_bin_cons_shift_wide<0b11, "lsl">; defm LSL_WIDE_ZZZ : sve_int_bin_cons_shift_wide<0b11, "lsl">;
// Predicated shifts // Predicated shifts
defm ASR_ZPmI : sve_int_bin_pred_shift_imm_right<0b0000, "asr", "ASR_ZPZI">; defm ASR_ZPmI : sve_int_bin_pred_shift_imm_right<0b0000, "asr", "ASR_ZPZI">;
defm LSR_ZPmI : sve_int_bin_pred_shift_imm_right<0b0001, "lsr", "LSR_ZPZI">; defm LSR_ZPmI : sve_int_bin_pred_shift_imm_right<0b0001, "lsr", "LSR_ZPZI">;
defm LSL_ZPmI : sve_int_bin_pred_shift_imm_left< 0b0011, "lsl">; defm LSL_ZPmI : sve_int_bin_pred_shift_imm_left< 0b0011, "lsl">;
defm ASRD_ZPmI : sve_int_bin_pred_shift_imm_right<0b0100, "asrd", "ASRD_ZPZI", int_aarch64_sve_asrd>; defm ASRD_ZPmI : sve_int_bin_pred_shift_imm_right<0b0100, "asrd", "ASRD_ZPZI", int_aarch64_sve_asrd>;
defm ASR_ZPZZ : sve_int_bin_pred_zx<AArch64asr_pred>; defm ASR_ZPZZ : sve_int_bin_pred_zx<AArch64asr_m>;
defm LSR_ZPZZ : sve_int_bin_pred_zx<AArch64lsr_pred>; defm LSR_ZPZZ : sve_int_bin_pred_zx<AArch64lsr_m>;
defm LSL_ZPZZ : sve_int_bin_pred_zx<AArch64lsl_pred>; defm LSL_ZPZZ : sve_int_bin_pred_zx<AArch64lsl_m>;
defm ASRD_ZPZI : sve_int_bin_pred_shift_0_right_zx<int_aarch64_sve_asrd>; defm ASRD_ZPZI : sve_int_bin_pred_shift_0_right_zx<int_aarch64_sve_asrd>;
defm ASR_ZPmZ : sve_int_bin_pred_shift<0b000, "asr", "ASR_ZPZZ", AArch64asr_pred, "ASRR_ZPmZ", 1>; defm ASR_ZPmZ : sve_int_bin_pred_shift<0b000, "asr", "ASR_ZPZZ", AArch64asr_m, "ASRR_ZPmZ", 1>;
defm LSR_ZPmZ : sve_int_bin_pred_shift<0b001, "lsr", "LSR_ZPZZ", AArch64lsr_pred, "LSRR_ZPmZ", 1>; defm LSR_ZPmZ : sve_int_bin_pred_shift<0b001, "lsr", "LSR_ZPZZ", AArch64lsr_m, "LSRR_ZPmZ", 1>;
defm LSL_ZPmZ : sve_int_bin_pred_shift<0b011, "lsl", "LSL_ZPZZ", AArch64lsl_pred, "LSLR_ZPmZ", 1>; defm LSL_ZPmZ : sve_int_bin_pred_shift<0b011, "lsl", "LSL_ZPZZ", AArch64lsl_m, "LSLR_ZPmZ", 1>;
defm ASRR_ZPmZ : sve_int_bin_pred_shift<0b100, "asrr", "ASRR_ZPZZ", null_frag, "ASR_ZPmZ", 0>; defm ASRR_ZPmZ : sve_int_bin_pred_shift<0b100, "asrr", "ASRR_ZPZZ", null_frag, "ASR_ZPmZ", 0>;
defm LSRR_ZPmZ : sve_int_bin_pred_shift<0b101, "lsrr", "LSRR_ZPZZ", null_frag, "LSR_ZPmZ", 0>; defm LSRR_ZPmZ : sve_int_bin_pred_shift<0b101, "lsrr", "LSRR_ZPZZ", null_frag, "LSR_ZPmZ", 0>;
defm LSLR_ZPmZ : sve_int_bin_pred_shift<0b111, "lslr", "LSLR_ZPZZ", null_frag, "LSL_ZPmZ", 0>; defm LSLR_ZPmZ : sve_int_bin_pred_shift<0b111, "lslr", "LSLR_ZPZZ", null_frag, "LSL_ZPmZ", 0>;
defm ASR_WIDE_ZPmZ : sve_int_bin_pred_shift_wide<0b000, "asr", int_aarch64_sve_asr_wide>; defm ASR_WIDE_ZPmZ : sve_int_bin_pred_shift_wide<0b000, "asr", int_aarch64_sve_asr_wide>;
defm LSR_WIDE_ZPmZ : sve_int_bin_pred_shift_wide<0b001, "lsr", int_aarch64_sve_lsr_wide>; defm LSR_WIDE_ZPmZ : sve_int_bin_pred_shift_wide<0b001, "lsr", int_aarch64_sve_lsr_wide>;
defm LSL_WIDE_ZPmZ : sve_int_bin_pred_shift_wide<0b011, "lsl", int_aarch64_sve_lsl_wide>; defm LSL_WIDE_ZPmZ : sve_int_bin_pred_shift_wide<0b011, "lsl", int_aarch64_sve_lsl_wide>;
▲ Show 20 Lines • Show All 158 Lines • ▼ Show 20 Linesmulticlass sve_prefetch<SDPatternOperator prefetch, ValueType PredTy, Instruction RegImmInst, Instruction RegRegInst, int scale, ComplexPattern AddrCP> {
def : Pat<(AArch64ptest (nxv8i1 PPR:$pg), (nxv8i1 PPR:$src)), def : Pat<(AArch64ptest (nxv8i1 PPR:$pg), (nxv8i1 PPR:$src)),
(PTEST_PP PPR:$pg, PPR:$src)>; (PTEST_PP PPR:$pg, PPR:$src)>;
def : Pat<(AArch64ptest (nxv4i1 PPR:$pg), (nxv4i1 PPR:$src)), def : Pat<(AArch64ptest (nxv4i1 PPR:$pg), (nxv4i1 PPR:$src)),
(PTEST_PP PPR:$pg, PPR:$src)>; (PTEST_PP PPR:$pg, PPR:$src)>;
def : Pat<(AArch64ptest (nxv2i1 PPR:$pg), (nxv2i1 PPR:$src)), def : Pat<(AArch64ptest (nxv2i1 PPR:$pg), (nxv2i1 PPR:$src)),
(PTEST_PP PPR:$pg, PPR:$src)>; (PTEST_PP PPR:$pg, PPR:$src)>;
// LD1R of 128-bit masked data // LD1R of 128-bit masked data
def : Pat<(nxv16i8 (AArch64ld1rq PPR:$gp, GPR64:$base)), def : Pat<(nxv16i8 (AArch64ld1rq_z PPR:$gp, GPR64:$base)),
(LD1RQ_B_IMM $gp, $base, (i64 0))>; (LD1RQ_B_IMM $gp, $base, (i64 0))>;
def : Pat<(nxv8i16 (AArch64ld1rq PPR:$gp, GPR64:$base)), def : Pat<(nxv8i16 (AArch64ld1rq_z PPR:$gp, GPR64:$base)),
(LD1RQ_H_IMM $gp, $base, (i64 0))>; (LD1RQ_H_IMM $gp, $base, (i64 0))>;
def : Pat<(nxv4i32 (AArch64ld1rq PPR:$gp, GPR64:$base)), def : Pat<(nxv4i32 (AArch64ld1rq_z PPR:$gp, GPR64:$base)),
(LD1RQ_W_IMM $gp, $base, (i64 0))>; (LD1RQ_W_IMM $gp, $base, (i64 0))>;
def : Pat<(nxv2i64 (AArch64ld1rq PPR:$gp, GPR64:$base)), def : Pat<(nxv2i64 (AArch64ld1rq_z PPR:$gp, GPR64:$base)),
(LD1RQ_D_IMM $gp, $base, (i64 0))>; (LD1RQ_D_IMM $gp, $base, (i64 0))>;
def : Pat<(nxv16i8 (AArch64ld1rq PPR:$gp, (add GPR64:$base, (i64 simm4s16:$imm)))), def : Pat<(nxv16i8 (AArch64ld1rq_z PPR:$gp, (add GPR64:$base, (i64 simm4s16:$imm)))),
(LD1RQ_B_IMM $gp, $base, simm4s16:$imm)>; (LD1RQ_B_IMM $gp, $base, simm4s16:$imm)>;
def : Pat<(nxv8i16 (AArch64ld1rq PPR:$gp, (add GPR64:$base, (i64 simm4s16:$imm)))), def : Pat<(nxv8i16 (AArch64ld1rq_z PPR:$gp, (add GPR64:$base, (i64 simm4s16:$imm)))),
(LD1RQ_H_IMM $gp, $base, simm4s16:$imm)>; (LD1RQ_H_IMM $gp, $base, simm4s16:$imm)>;
def : Pat<(nxv4i32 (AArch64ld1rq PPR:$gp, (add GPR64:$base, (i64 simm4s16:$imm)))), def : Pat<(nxv4i32 (AArch64ld1rq_z PPR:$gp, (add GPR64:$base, (i64 simm4s16:$imm)))),
(LD1RQ_W_IMM $gp, $base, simm4s16:$imm)>; (LD1RQ_W_IMM $gp, $base, simm4s16:$imm)>;
def : Pat<(nxv2i64 (AArch64ld1rq PPR:$gp, (add GPR64:$base, (i64 simm4s16:$imm)))), def : Pat<(nxv2i64 (AArch64ld1rq_z PPR:$gp, (add GPR64:$base, (i64 simm4s16:$imm)))),
(LD1RQ_D_IMM $gp, $base, simm4s16:$imm)>; (LD1RQ_D_IMM $gp, $base, simm4s16:$imm)>;
def : Pat<(sext_inreg (nxv2i64 ZPR:$Zs), nxv2i32), (SXTW_ZPmZ_D (IMPLICIT_DEF), (PTRUE_D 31), ZPR:$Zs)>; def : Pat<(sext_inreg (nxv2i64 ZPR:$Zs), nxv2i32), (SXTW_ZPmZ_D (IMPLICIT_DEF), (PTRUE_D 31), ZPR:$Zs)>;
def : Pat<(sext_inreg (nxv2i64 ZPR:$Zs), nxv2i16), (SXTH_ZPmZ_D (IMPLICIT_DEF), (PTRUE_D 31), ZPR:$Zs)>; def : Pat<(sext_inreg (nxv2i64 ZPR:$Zs), nxv2i16), (SXTH_ZPmZ_D (IMPLICIT_DEF), (PTRUE_D 31), ZPR:$Zs)>;
def : Pat<(sext_inreg (nxv2i64 ZPR:$Zs), nxv2i8), (SXTB_ZPmZ_D (IMPLICIT_DEF), (PTRUE_D 31), ZPR:$Zs)>; def : Pat<(sext_inreg (nxv2i64 ZPR:$Zs), nxv2i8), (SXTB_ZPmZ_D (IMPLICIT_DEF), (PTRUE_D 31), ZPR:$Zs)>;
def : Pat<(sext_inreg (nxv4i32 ZPR:$Zs), nxv4i16), (SXTH_ZPmZ_S (IMPLICIT_DEF), (PTRUE_S 31), ZPR:$Zs)>; def : Pat<(sext_inreg (nxv4i32 ZPR:$Zs), nxv4i16), (SXTH_ZPmZ_S (IMPLICIT_DEF), (PTRUE_S 31), ZPR:$Zs)>;
def : Pat<(sext_inreg (nxv4i32 ZPR:$Zs), nxv4i8), (SXTB_ZPmZ_S (IMPLICIT_DEF), (PTRUE_S 31), ZPR:$Zs)>; def : Pat<(sext_inreg (nxv4i32 ZPR:$Zs), nxv4i8), (SXTB_ZPmZ_S (IMPLICIT_DEF), (PTRUE_S 31), ZPR:$Zs)>;
def : Pat<(sext_inreg (nxv8i16 ZPR:$Zs), nxv8i8), (SXTB_ZPmZ_H (IMPLICIT_DEF), (PTRUE_H 31), ZPR:$Zs)>; def : Pat<(sext_inreg (nxv8i16 ZPR:$Zs), nxv8i8), (SXTB_ZPmZ_H (IMPLICIT_DEF), (PTRUE_H 31), ZPR:$Zs)>;
▲ Show 20 Lines • Show All 295 Lines • ▼ Show 20 Lines multiclass ld1<Instruction RegRegInst, Instruction RegImmInst, ValueType Ty,
} }
// base // base
def : Pat<(Ty (Load (PredTy PPR:$gp), GPR64:$base, MemVT)), def : Pat<(Ty (Load (PredTy PPR:$gp), GPR64:$base, MemVT)),
(RegImmInst PPR:$gp, GPR64sp:$base, (i64 0))>; (RegImmInst PPR:$gp, GPR64sp:$base, (i64 0))>;
} }
// 2-element contiguous loads // 2-element contiguous loads
defm : ld1<LD1B_D, LD1B_D_IMM, nxv2i64, AArch64ld1, nxv2i1, nxv2i8, am_sve_regreg_lsl0>; defm : ld1<LD1B_D, LD1B_D_IMM, nxv2i64, AArch64ld1_z, nxv2i1, nxv2i8, am_sve_regreg_lsl0>;
defm : ld1<LD1SB_D, LD1SB_D_IMM, nxv2i64, AArch64ld1s, nxv2i1, nxv2i8, am_sve_regreg_lsl0>; defm : ld1<LD1SB_D, LD1SB_D_IMM, nxv2i64, AArch64ld1s_z, nxv2i1, nxv2i8, am_sve_regreg_lsl0>;
defm : ld1<LD1H_D, LD1H_D_IMM, nxv2i64, AArch64ld1, nxv2i1, nxv2i16, am_sve_regreg_lsl1>; defm : ld1<LD1H_D, LD1H_D_IMM, nxv2i64, AArch64ld1_z, nxv2i1, nxv2i16, am_sve_regreg_lsl1>;
defm : ld1<LD1SH_D, LD1SH_D_IMM, nxv2i64, AArch64ld1s, nxv2i1, nxv2i16, am_sve_regreg_lsl1>; defm : ld1<LD1SH_D, LD1SH_D_IMM, nxv2i64, AArch64ld1s_z, nxv2i1, nxv2i16, am_sve_regreg_lsl1>;
defm : ld1<LD1W_D, LD1W_D_IMM, nxv2i64, AArch64ld1, nxv2i1, nxv2i32, am_sve_regreg_lsl2>; defm : ld1<LD1W_D, LD1W_D_IMM, nxv2i64, AArch64ld1_z, nxv2i1, nxv2i32, am_sve_regreg_lsl2>;
defm : ld1<LD1SW_D, LD1SW_D_IMM, nxv2i64, AArch64ld1s, nxv2i1, nxv2i32, am_sve_regreg_lsl2>; defm : ld1<LD1SW_D, LD1SW_D_IMM, nxv2i64, AArch64ld1s_z, nxv2i1, nxv2i32, am_sve_regreg_lsl2>;
defm : ld1<LD1D, LD1D_IMM, nxv2i64, AArch64ld1, nxv2i1, nxv2i64, am_sve_regreg_lsl3>; defm : ld1<LD1D, LD1D_IMM, nxv2i64, AArch64ld1_z, nxv2i1, nxv2i64, am_sve_regreg_lsl3>;
defm : ld1<LD1D, LD1D_IMM, nxv2f64, AArch64ld1, nxv2i1, nxv2f64, am_sve_regreg_lsl3>; defm : ld1<LD1D, LD1D_IMM, nxv2f64, AArch64ld1_z, nxv2i1, nxv2f64, am_sve_regreg_lsl3>;
// 4-element contiguous loads // 4-element contiguous loads
defm : ld1<LD1B_S, LD1B_S_IMM, nxv4i32, AArch64ld1, nxv4i1, nxv4i8, am_sve_regreg_lsl0>; defm : ld1<LD1B_S, LD1B_S_IMM, nxv4i32, AArch64ld1_z, nxv4i1, nxv4i8, am_sve_regreg_lsl0>;
defm : ld1<LD1SB_S, LD1SB_S_IMM, nxv4i32, AArch64ld1s, nxv4i1, nxv4i8, am_sve_regreg_lsl0>; defm : ld1<LD1SB_S, LD1SB_S_IMM, nxv4i32, AArch64ld1s_z, nxv4i1, nxv4i8, am_sve_regreg_lsl0>;
defm : ld1<LD1H_S, LD1H_S_IMM, nxv4i32, AArch64ld1, nxv4i1, nxv4i16, am_sve_regreg_lsl1>; defm : ld1<LD1H_S, LD1H_S_IMM, nxv4i32, AArch64ld1_z, nxv4i1, nxv4i16, am_sve_regreg_lsl1>;
defm : ld1<LD1SH_S, LD1SH_S_IMM, nxv4i32, AArch64ld1s, nxv4i1, nxv4i16, am_sve_regreg_lsl1>; defm : ld1<LD1SH_S, LD1SH_S_IMM, nxv4i32, AArch64ld1s_z, nxv4i1, nxv4i16, am_sve_regreg_lsl1>;
defm : ld1<LD1W, LD1W_IMM, nxv4i32, AArch64ld1, nxv4i1, nxv4i32, am_sve_regreg_lsl2>; defm : ld1<LD1W, LD1W_IMM, nxv4i32, AArch64ld1_z, nxv4i1, nxv4i32, am_sve_regreg_lsl2>;
defm : ld1<LD1W, LD1W_IMM, nxv4f32, AArch64ld1, nxv4i1, nxv4f32, am_sve_regreg_lsl2>; defm : ld1<LD1W, LD1W_IMM, nxv4f32, AArch64ld1_z, nxv4i1, nxv4f32, am_sve_regreg_lsl2>;
// 8-element contiguous loads // 8-element contiguous loads
defm : ld1<LD1B_H, LD1B_H_IMM, nxv8i16, AArch64ld1, nxv8i1, nxv8i8, am_sve_regreg_lsl0>; defm : ld1<LD1B_H, LD1B_H_IMM, nxv8i16, AArch64ld1_z, nxv8i1, nxv8i8, am_sve_regreg_lsl0>;
defm : ld1<LD1SB_H, LD1SB_H_IMM, nxv8i16, AArch64ld1s, nxv8i1, nxv8i8, am_sve_regreg_lsl0>; defm : ld1<LD1SB_H, LD1SB_H_IMM, nxv8i16, AArch64ld1s_z, nxv8i1, nxv8i8, am_sve_regreg_lsl0>;
defm : ld1<LD1H, LD1H_IMM, nxv8i16, AArch64ld1, nxv8i1, nxv8i16, am_sve_regreg_lsl1>; defm : ld1<LD1H, LD1H_IMM, nxv8i16, AArch64ld1_z, nxv8i1, nxv8i16, am_sve_regreg_lsl1>;
defm : ld1<LD1H, LD1H_IMM, nxv8f16, AArch64ld1, nxv8i1, nxv8f16, am_sve_regreg_lsl1>; defm : ld1<LD1H, LD1H_IMM, nxv8f16, AArch64ld1_z, nxv8i1, nxv8f16, am_sve_regreg_lsl1>;
// 16-element contiguous loads // 16-element contiguous loads
defm : ld1<LD1B, LD1B_IMM, nxv16i8, AArch64ld1, nxv16i1, nxv16i8, am_sve_regreg_lsl0>; defm : ld1<LD1B, LD1B_IMM, nxv16i8, AArch64ld1_z, nxv16i1, nxv16i8, am_sve_regreg_lsl0>;
multiclass ldnf1<Instruction I, ValueType Ty, SDPatternOperator Load, ValueType PredTy, ValueType MemVT> { multiclass ldnf1<Instruction I, ValueType Ty, SDPatternOperator Load, ValueType PredTy, ValueType MemVT> {
// scalar + immediate (mul vl) // scalar + immediate (mul vl)
let AddedComplexity = 1 in { let AddedComplexity = 1 in {
def : Pat<(Ty (Load (PredTy PPR:$gp), (am_sve_indexed_s4 GPR64sp:$base, simm4s1:$offset), MemVT)), def : Pat<(Ty (Load (PredTy PPR:$gp), (am_sve_indexed_s4 GPR64sp:$base, simm4s1:$offset), MemVT)),
(I PPR:$gp, GPR64sp:$base, simm4s1:$offset)>; (I PPR:$gp, GPR64sp:$base, simm4s1:$offset)>;
} }
// base // base
def : Pat<(Ty (Load (PredTy PPR:$gp), GPR64:$base, MemVT)), def : Pat<(Ty (Load (PredTy PPR:$gp), GPR64:$base, MemVT)),
(I PPR:$gp, GPR64sp:$base, (i64 0))>; (I PPR:$gp, GPR64sp:$base, (i64 0))>;
} }
// 2-element contiguous non-faulting loads // 2-element contiguous non-faulting loads
defm : ldnf1<LDNF1B_D_IMM, nxv2i64, AArch64ldnf1, nxv2i1, nxv2i8>; defm : ldnf1<LDNF1B_D_IMM, nxv2i64, AArch64ldnf1_z, nxv2i1, nxv2i8>;
defm : ldnf1<LDNF1SB_D_IMM, nxv2i64, AArch64ldnf1s, nxv2i1, nxv2i8>; defm : ldnf1<LDNF1SB_D_IMM, nxv2i64, AArch64ldnf1s_z, nxv2i1, nxv2i8>;
defm : ldnf1<LDNF1H_D_IMM, nxv2i64, AArch64ldnf1, nxv2i1, nxv2i16>; defm : ldnf1<LDNF1H_D_IMM, nxv2i64, AArch64ldnf1_z, nxv2i1, nxv2i16>;
defm : ldnf1<LDNF1SH_D_IMM, nxv2i64, AArch64ldnf1s, nxv2i1, nxv2i16>; defm : ldnf1<LDNF1SH_D_IMM, nxv2i64, AArch64ldnf1s_z, nxv2i1, nxv2i16>;
defm : ldnf1<LDNF1W_D_IMM, nxv2i64, AArch64ldnf1, nxv2i1, nxv2i32>; defm : ldnf1<LDNF1W_D_IMM, nxv2i64, AArch64ldnf1_z, nxv2i1, nxv2i32>;
defm : ldnf1<LDNF1SW_D_IMM, nxv2i64, AArch64ldnf1s, nxv2i1, nxv2i32>; defm : ldnf1<LDNF1SW_D_IMM, nxv2i64, AArch64ldnf1s_z, nxv2i1, nxv2i32>;
defm : ldnf1<LDNF1D_IMM, nxv2i64, AArch64ldnf1, nxv2i1, nxv2i64>; defm : ldnf1<LDNF1D_IMM, nxv2i64, AArch64ldnf1_z, nxv2i1, nxv2i64>;
defm : ldnf1<LDNF1D_IMM, nxv2f64, AArch64ldnf1, nxv2i1, nxv2f64>; defm : ldnf1<LDNF1D_IMM, nxv2f64, AArch64ldnf1_z, nxv2i1, nxv2f64>;
// 4-element contiguous non-faulting loads // 4-element contiguous non-faulting loads
defm : ldnf1<LDNF1B_S_IMM, nxv4i32, AArch64ldnf1, nxv4i1, nxv4i8>; defm : ldnf1<LDNF1B_S_IMM, nxv4i32, AArch64ldnf1_z, nxv4i1, nxv4i8>;
defm : ldnf1<LDNF1SB_S_IMM, nxv4i32, AArch64ldnf1s, nxv4i1, nxv4i8>; defm : ldnf1<LDNF1SB_S_IMM, nxv4i32, AArch64ldnf1s_z, nxv4i1, nxv4i8>;
defm : ldnf1<LDNF1H_S_IMM, nxv4i32, AArch64ldnf1, nxv4i1, nxv4i16>; defm : ldnf1<LDNF1H_S_IMM, nxv4i32, AArch64ldnf1_z, nxv4i1, nxv4i16>;
defm : ldnf1<LDNF1SH_S_IMM, nxv4i32, AArch64ldnf1s, nxv4i1, nxv4i16>; defm : ldnf1<LDNF1SH_S_IMM, nxv4i32, AArch64ldnf1s_z, nxv4i1, nxv4i16>;
defm : ldnf1<LDNF1W_IMM, nxv4i32, AArch64ldnf1, nxv4i1, nxv4i32>; defm : ldnf1<LDNF1W_IMM, nxv4i32, AArch64ldnf1_z, nxv4i1, nxv4i32>;
defm : ldnf1<LDNF1W_IMM, nxv4f32, AArch64ldnf1, nxv4i1, nxv4f32>; defm : ldnf1<LDNF1W_IMM, nxv4f32, AArch64ldnf1_z, nxv4i1, nxv4f32>;
// 8-element contiguous non-faulting loads // 8-element contiguous non-faulting loads
defm : ldnf1<LDNF1B_H_IMM, nxv8i16, AArch64ldnf1, nxv8i1, nxv8i8>; defm : ldnf1<LDNF1B_H_IMM, nxv8i16, AArch64ldnf1_z, nxv8i1, nxv8i8>;
defm : ldnf1<LDNF1SB_H_IMM, nxv8i16, AArch64ldnf1s, nxv8i1, nxv8i8>; defm : ldnf1<LDNF1SB_H_IMM, nxv8i16, AArch64ldnf1s_z, nxv8i1, nxv8i8>;
defm : ldnf1<LDNF1H_IMM, nxv8i16, AArch64ldnf1, nxv8i1, nxv8i16>; defm : ldnf1<LDNF1H_IMM, nxv8i16, AArch64ldnf1_z, nxv8i1, nxv8i16>;
defm : ldnf1<LDNF1H_IMM, nxv8f16, AArch64ldnf1, nxv8i1, nxv8f16>; defm : ldnf1<LDNF1H_IMM, nxv8f16, AArch64ldnf1_z, nxv8i1, nxv8f16>;
// 16-element contiguous non-faulting loads // 16-element contiguous non-faulting loads
defm : ldnf1<LDNF1B_IMM, nxv16i8, AArch64ldnf1, nxv16i1, nxv16i8>; defm : ldnf1<LDNF1B_IMM, nxv16i8, AArch64ldnf1_z, nxv16i1, nxv16i8>;
multiclass ldff1<Instruction I, ValueType Ty, SDPatternOperator Load, ValueType PredTy, ValueType MemVT, ComplexPattern AddrCP> { multiclass ldff1<Instruction I, ValueType Ty, SDPatternOperator Load, ValueType PredTy, ValueType MemVT, ComplexPattern AddrCP> {
// reg + reg // reg + reg
let AddedComplexity = 1 in { let AddedComplexity = 1 in {
def : Pat<(Ty (Load (PredTy PPR:$gp), (AddrCP GPR64:$base, GPR64:$offset), MemVT)), def : Pat<(Ty (Load (PredTy PPR:$gp), (AddrCP GPR64:$base, GPR64:$offset), MemVT)),
(I PPR:$gp, GPR64sp:$base, GPR64:$offset)>; (I PPR:$gp, GPR64sp:$base, GPR64:$offset)>;
} }
// Base // Base
def : Pat<(Ty (Load (PredTy PPR:$gp), GPR64:$base, MemVT)), def : Pat<(Ty (Load (PredTy PPR:$gp), GPR64:$base, MemVT)),
(I PPR:$gp, GPR64sp:$base, XZR)>; (I PPR:$gp, GPR64sp:$base, XZR)>;
} }
// 2-element contiguous first faulting loads // 2-element contiguous first faulting loads
defm : ldff1<LDFF1B_D, nxv2i64, AArch64ldff1, nxv2i1, nxv2i8, am_sve_regreg_lsl0>; defm : ldff1<LDFF1B_D, nxv2i64, AArch64ldff1_z, nxv2i1, nxv2i8, am_sve_regreg_lsl0>;
defm : ldff1<LDFF1SB_D, nxv2i64, AArch64ldff1s, nxv2i1, nxv2i8, am_sve_regreg_lsl0>; defm : ldff1<LDFF1SB_D, nxv2i64, AArch64ldff1s_z, nxv2i1, nxv2i8, am_sve_regreg_lsl0>;
defm : ldff1<LDFF1H_D, nxv2i64, AArch64ldff1, nxv2i1, nxv2i16, am_sve_regreg_lsl1>; defm : ldff1<LDFF1H_D, nxv2i64, AArch64ldff1_z, nxv2i1, nxv2i16, am_sve_regreg_lsl1>;
defm : ldff1<LDFF1SH_D, nxv2i64, AArch64ldff1s, nxv2i1, nxv2i16, am_sve_regreg_lsl1>; defm : ldff1<LDFF1SH_D, nxv2i64, AArch64ldff1s_z, nxv2i1, nxv2i16, am_sve_regreg_lsl1>;
defm : ldff1<LDFF1W_D, nxv2i64, AArch64ldff1, nxv2i1, nxv2i32, am_sve_regreg_lsl2>; defm : ldff1<LDFF1W_D, nxv2i64, AArch64ldff1_z, nxv2i1, nxv2i32, am_sve_regreg_lsl2>;
defm : ldff1<LDFF1SW_D, nxv2i64, AArch64ldff1s, nxv2i1, nxv2i32, am_sve_regreg_lsl2>; defm : ldff1<LDFF1SW_D, nxv2i64, AArch64ldff1s_z, nxv2i1, nxv2i32, am_sve_regreg_lsl2>;
defm : ldff1<LDFF1D, nxv2i64, AArch64ldff1, nxv2i1, nxv2i64, am_sve_regreg_lsl3>; defm : ldff1<LDFF1D, nxv2i64, AArch64ldff1_z, nxv2i1, nxv2i64, am_sve_regreg_lsl3>;
defm : ldff1<LDFF1W_D, nxv2f32, AArch64ldff1, nxv2i1, nxv2f32, am_sve_regreg_lsl2>; defm : ldff1<LDFF1W_D, nxv2f32, AArch64ldff1_z, nxv2i1, nxv2f32, am_sve_regreg_lsl2>;
defm : ldff1<LDFF1D, nxv2f64, AArch64ldff1, nxv2i1, nxv2f64, am_sve_regreg_lsl3>; defm : ldff1<LDFF1D, nxv2f64, AArch64ldff1_z, nxv2i1, nxv2f64, am_sve_regreg_lsl3>;
// 4-element contiguous first faulting loads // 4-element contiguous first faulting loads
defm : ldff1<LDFF1B_S, nxv4i32, AArch64ldff1, nxv4i1, nxv4i8, am_sve_regreg_lsl0>; defm : ldff1<LDFF1B_S, nxv4i32, AArch64ldff1_z, nxv4i1, nxv4i8, am_sve_regreg_lsl0>;
defm : ldff1<LDFF1SB_S, nxv4i32, AArch64ldff1s, nxv4i1, nxv4i8, am_sve_regreg_lsl0>; defm : ldff1<LDFF1SB_S, nxv4i32, AArch64ldff1s_z, nxv4i1, nxv4i8, am_sve_regreg_lsl0>;
defm : ldff1<LDFF1H_S, nxv4i32, AArch64ldff1, nxv4i1, nxv4i16, am_sve_regreg_lsl1>; defm : ldff1<LDFF1H_S, nxv4i32, AArch64ldff1_z, nxv4i1, nxv4i16, am_sve_regreg_lsl1>;
defm : ldff1<LDFF1SH_S, nxv4i32, AArch64ldff1s, nxv4i1, nxv4i16, am_sve_regreg_lsl1>; defm : ldff1<LDFF1SH_S, nxv4i32, AArch64ldff1s_z, nxv4i1, nxv4i16, am_sve_regreg_lsl1>;
defm : ldff1<LDFF1W, nxv4i32, AArch64ldff1, nxv4i1, nxv4i32, am_sve_regreg_lsl2>; defm : ldff1<LDFF1W, nxv4i32, AArch64ldff1_z, nxv4i1, nxv4i32, am_sve_regreg_lsl2>;
defm : ldff1<LDFF1W, nxv4f32, AArch64ldff1, nxv4i1, nxv4f32, am_sve_regreg_lsl2>; defm : ldff1<LDFF1W, nxv4f32, AArch64ldff1_z, nxv4i1, nxv4f32, am_sve_regreg_lsl2>;
// 8-element contiguous first faulting loads // 8-element contiguous first faulting loads
defm : ldff1<LDFF1B_H, nxv8i16, AArch64ldff1, nxv8i1, nxv8i8, am_sve_regreg_lsl0>; defm : ldff1<LDFF1B_H, nxv8i16, AArch64ldff1_z, nxv8i1, nxv8i8, am_sve_regreg_lsl0>;
defm : ldff1<LDFF1SB_H, nxv8i16, AArch64ldff1s, nxv8i1, nxv8i8, am_sve_regreg_lsl0>; defm : ldff1<LDFF1SB_H, nxv8i16, AArch64ldff1s_z, nxv8i1, nxv8i8, am_sve_regreg_lsl0>;
defm : ldff1<LDFF1H, nxv8i16, AArch64ldff1, nxv8i1, nxv8i16, am_sve_regreg_lsl1>; defm : ldff1<LDFF1H, nxv8i16, AArch64ldff1_z, nxv8i1, nxv8i16, am_sve_regreg_lsl1>;
defm : ldff1<LDFF1H, nxv8f16, AArch64ldff1, nxv8i1, nxv8f16, am_sve_regreg_lsl1>; defm : ldff1<LDFF1H, nxv8f16, AArch64ldff1_z, nxv8i1, nxv8f16, am_sve_regreg_lsl1>;
// 16-element contiguous first faulting loads // 16-element contiguous first faulting loads
defm : ldff1<LDFF1B, nxv16i8, AArch64ldff1, nxv16i1, nxv16i8, am_sve_regreg_lsl0>; defm : ldff1<LDFF1B, nxv16i8, AArch64ldff1_z, nxv16i1, nxv16i8, am_sve_regreg_lsl0>;
multiclass st1<Instruction RegRegInst, Instruction RegImmInst, ValueType Ty, multiclass st1<Instruction RegRegInst, Instruction RegImmInst, ValueType Ty,
SDPatternOperator Store, ValueType PredTy, ValueType MemVT, ComplexPattern AddrCP> { SDPatternOperator Store, ValueType PredTy, ValueType MemVT, ComplexPattern AddrCP> {
// reg + reg // reg + reg
let AddedComplexity = 1 in { let AddedComplexity = 1 in {
def : Pat<(Store (Ty ZPR:$vec), (AddrCP GPR64:$base, GPR64:$offset), (PredTy PPR:$gp), MemVT), def : Pat<(Store (Ty ZPR:$vec), (AddrCP GPR64:$base, GPR64:$offset), (PredTy PPR:$gp), MemVT),
(RegRegInst ZPR:$vec, PPR:$gp, GPR64sp:$base, GPR64:$offset)>; (RegRegInst ZPR:$vec, PPR:$gp, GPR64sp:$base, GPR64:$offset)>;
} }
▲ Show 20 Lines • Show All 150 Lines • ▼ Show 20 Lines
} }
let Predicates = [HasSVE, HasMatMulFP32] in { let Predicates = [HasSVE, HasMatMulFP32] in {
defm FMMLA_ZZZ_S : sve_fp_matrix_mla<0, "fmmla", ZPR32, int_aarch64_sve_fmmla, nxv4f32>; defm FMMLA_ZZZ_S : sve_fp_matrix_mla<0, "fmmla", ZPR32, int_aarch64_sve_fmmla, nxv4f32>;
} }
let Predicates = [HasSVE, HasMatMulFP64] in { let Predicates = [HasSVE, HasMatMulFP64] in {
defm FMMLA_ZZZ_D : sve_fp_matrix_mla<1, "fmmla", ZPR64, int_aarch64_sve_fmmla, nxv2f64>; defm FMMLA_ZZZ_D : sve_fp_matrix_mla<1, "fmmla", ZPR64, int_aarch64_sve_fmmla, nxv2f64>;
defm LD1RO_B_IMM : sve_mem_ldor_si<0b00, "ld1rob", Z_b, ZPR8, nxv16i8, nxv16i1, AArch64ld1ro>; defm LD1RO_B_IMM : sve_mem_ldor_si<0b00, "ld1rob", Z_b, ZPR8, nxv16i8, nxv16i1, AArch64ld1ro_z>;
defm LD1RO_H_IMM : sve_mem_ldor_si<0b01, "ld1roh", Z_h, ZPR16, nxv8i16, nxv8i1, AArch64ld1ro>; defm LD1RO_H_IMM : sve_mem_ldor_si<0b01, "ld1roh", Z_h, ZPR16, nxv8i16, nxv8i1, AArch64ld1ro_z>;
defm LD1RO_W_IMM : sve_mem_ldor_si<0b10, "ld1row", Z_s, ZPR32, nxv4i32, nxv4i1, AArch64ld1ro>; defm LD1RO_W_IMM : sve_mem_ldor_si<0b10, "ld1row", Z_s, ZPR32, nxv4i32, nxv4i1, AArch64ld1ro_z>;
defm LD1RO_D_IMM : sve_mem_ldor_si<0b11, "ld1rod", Z_d, ZPR64, nxv2i64, nxv2i1, AArch64ld1ro>; defm LD1RO_D_IMM : sve_mem_ldor_si<0b11, "ld1rod", Z_d, ZPR64, nxv2i64, nxv2i1, AArch64ld1ro_z>;
defm LD1RO_B : sve_mem_ldor_ss<0b00, "ld1rob", Z_b, ZPR8, GPR64NoXZRshifted8>; defm LD1RO_B : sve_mem_ldor_ss<0b00, "ld1rob", Z_b, ZPR8, GPR64NoXZRshifted8>;
defm LD1RO_H : sve_mem_ldor_ss<0b01, "ld1roh", Z_h, ZPR16, GPR64NoXZRshifted16>; defm LD1RO_H : sve_mem_ldor_ss<0b01, "ld1roh", Z_h, ZPR16, GPR64NoXZRshifted16>;
defm LD1RO_W : sve_mem_ldor_ss<0b10, "ld1row", Z_s, ZPR32, GPR64NoXZRshifted32>; defm LD1RO_W : sve_mem_ldor_ss<0b10, "ld1row", Z_s, ZPR32, GPR64NoXZRshifted32>;
defm LD1RO_D : sve_mem_ldor_ss<0b11, "ld1rod", Z_d, ZPR64, GPR64NoXZRshifted64>; defm LD1RO_D : sve_mem_ldor_ss<0b11, "ld1rod", Z_d, ZPR64, GPR64NoXZRshifted64>;
def ZIP1_ZZZ_128 : sve_int_perm_bin_perm_128_zz<0b00, 0, "zip1">; def ZIP1_ZZZ_128 : sve_int_perm_bin_perm_128_zz<0b00, 0, "zip1">;
def ZIP2_ZZZ_128 : sve_int_perm_bin_perm_128_zz<0b00, 1, "zip2">; def ZIP2_ZZZ_128 : sve_int_perm_bin_perm_128_zz<0b00, 1, "zip2">;
def UZP1_ZZZ_128 : sve_int_perm_bin_perm_128_zz<0b01, 0, "uzp1">; def UZP1_ZZZ_128 : sve_int_perm_bin_perm_128_zz<0b01, 0, "uzp1">;
def UZP2_ZZZ_128 : sve_int_perm_bin_perm_128_zz<0b01, 1, "uzp2">; def UZP2_ZZZ_128 : sve_int_perm_bin_perm_128_zz<0b01, 1, "uzp2">;
▲ Show 20 Lines • Show All 336 Lines • ▼ Show 20 Lineslet Predicates = [HasSVE2] in {
// SVE2 bitwise xor and rotate right by immediate // SVE2 bitwise xor and rotate right by immediate
defm XAR_ZZZI : sve2_int_rotate_right_imm<"xar", int_aarch64_sve_xar>; defm XAR_ZZZI : sve2_int_rotate_right_imm<"xar", int_aarch64_sve_xar>;
// SVE2 extract vector (immediate offset, constructive) // SVE2 extract vector (immediate offset, constructive)
def EXT_ZZI_B : sve2_int_perm_extract_i_cons<"ext">; def EXT_ZZI_B : sve2_int_perm_extract_i_cons<"ext">;
// SVE2 non-temporal gather loads // SVE2 non-temporal gather loads
defm LDNT1SB_ZZR_S : sve2_mem_gldnt_vs_32_ptrs<0b00000, "ldnt1sb", AArch64ldnt1s_gather, nxv4i8>; defm LDNT1SB_ZZR_S : sve2_mem_gldnt_vs_32_ptrs<0b00000, "ldnt1sb", AArch64ldnt1s_gather_z, nxv4i8>;
defm LDNT1B_ZZR_S : sve2_mem_gldnt_vs_32_ptrs<0b00001, "ldnt1b", AArch64ldnt1_gather, nxv4i8>; defm LDNT1B_ZZR_S : sve2_mem_gldnt_vs_32_ptrs<0b00001, "ldnt1b", AArch64ldnt1_gather_z, nxv4i8>;
defm LDNT1SH_ZZR_S : sve2_mem_gldnt_vs_32_ptrs<0b00100, "ldnt1sh", AArch64ldnt1s_gather, nxv4i16>; defm LDNT1SH_ZZR_S : sve2_mem_gldnt_vs_32_ptrs<0b00100, "ldnt1sh", AArch64ldnt1s_gather_z, nxv4i16>;
defm LDNT1H_ZZR_S : sve2_mem_gldnt_vs_32_ptrs<0b00101, "ldnt1h", AArch64ldnt1_gather, nxv4i16>; defm LDNT1H_ZZR_S : sve2_mem_gldnt_vs_32_ptrs<0b00101, "ldnt1h", AArch64ldnt1_gather_z, nxv4i16>;
defm LDNT1W_ZZR_S : sve2_mem_gldnt_vs_32_ptrs<0b01001, "ldnt1w", AArch64ldnt1_gather, nxv4i32>; defm LDNT1W_ZZR_S : sve2_mem_gldnt_vs_32_ptrs<0b01001, "ldnt1w", AArch64ldnt1_gather_z, nxv4i32>;
defm LDNT1SB_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b10000, "ldnt1sb", AArch64ldnt1s_gather, nxv2i8>; defm LDNT1SB_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b10000, "ldnt1sb", AArch64ldnt1s_gather_z, nxv2i8>;
defm LDNT1B_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b10010, "ldnt1b", AArch64ldnt1_gather, nxv2i8>; defm LDNT1B_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b10010, "ldnt1b", AArch64ldnt1_gather_z, nxv2i8>;
defm LDNT1SH_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b10100, "ldnt1sh", AArch64ldnt1s_gather, nxv2i16>; defm LDNT1SH_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b10100, "ldnt1sh", AArch64ldnt1s_gather_z, nxv2i16>;
defm LDNT1H_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b10110, "ldnt1h", AArch64ldnt1_gather, nxv2i16>; defm LDNT1H_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b10110, "ldnt1h", AArch64ldnt1_gather_z, nxv2i16>;
defm LDNT1SW_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b11000, "ldnt1sw", AArch64ldnt1s_gather, nxv2i32>; defm LDNT1SW_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b11000, "ldnt1sw", AArch64ldnt1s_gather_z, nxv2i32>;
defm LDNT1W_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b11010, "ldnt1w", AArch64ldnt1_gather, nxv2i32>; defm LDNT1W_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b11010, "ldnt1w", AArch64ldnt1_gather_z, nxv2i32>;
defm LDNT1D_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b11110, "ldnt1d", AArch64ldnt1_gather, nxv2i64>; defm LDNT1D_ZZR_D : sve2_mem_gldnt_vs_64_ptrs<0b11110, "ldnt1d", AArch64ldnt1_gather_z, nxv2i64>;
// SVE2 vector splice (constructive) // SVE2 vector splice (constructive)
defm SPLICE_ZPZZ : sve2_int_perm_splice_cons<"splice">; defm SPLICE_ZPZZ : sve2_int_perm_splice_cons<"splice">;
// SVE2 non-temporal scatter stores // SVE2 non-temporal scatter stores
defm STNT1B_ZZR_S : sve2_mem_sstnt_vs_32_ptrs<0b001, "stnt1b", AArch64stnt1_scatter, nxv4i8>; defm STNT1B_ZZR_S : sve2_mem_sstnt_vs_32_ptrs<0b001, "stnt1b", AArch64stnt1_scatter, nxv4i8>;
defm STNT1H_ZZR_S : sve2_mem_sstnt_vs_32_ptrs<0b011, "stnt1h", AArch64stnt1_scatter, nxv4i16>; defm STNT1H_ZZR_S : sve2_mem_sstnt_vs_32_ptrs<0b011, "stnt1h", AArch64stnt1_scatter, nxv4i16>;
defm STNT1W_ZZR_S : sve2_mem_sstnt_vs_32_ptrs<0b101, "stnt1w", AArch64stnt1_scatter, nxv4i32>; defm STNT1W_ZZR_S : sve2_mem_sstnt_vs_32_ptrs<0b101, "stnt1w", AArch64stnt1_scatter, nxv4i32>;
▲ Show 20 Lines • Show All 60 Lines • Show Last 20 Lines

llvm/lib/Target/AArch64/SVEInstrFormats.td

  • This file is larger than 256 KB, so syntax highlighting is disabled by default.
Show All 10 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
def SDT_AArch64Setcc : SDTypeProfile<1, 4, [ def SDT_AArch64Setcc : SDTypeProfile<1, 4, [
SDTCisVec<0>, SDTCisVec<1>, SDTCisVec<2>, SDTCisVec<3>, SDTCisVec<0>, SDTCisVec<1>, SDTCisVec<2>, SDTCisVec<3>,
SDTCVecEltisVT<0, i1>, SDTCVecEltisVT<1, i1>, SDTCisSameAs<2, 3>, SDTCVecEltisVT<0, i1>, SDTCVecEltisVT<1, i1>, SDTCisSameAs<2, 3>,
SDTCisVT<4, OtherVT> SDTCisVT<4, OtherVT>
]>; ]>;
def AArch64setcc_pred : SDNode<"AArch64ISD::SETCC_PRED", SDT_AArch64Setcc>; def AArch64setcc_pred : SDNode<"AArch64ISD::SETCC_PRED_Z", SDT_AArch64Setcc>;
def SVEPatternOperand : AsmOperandClass { def SVEPatternOperand : AsmOperandClass {
let Name = "SVEPattern"; let Name = "SVEPattern";
let ParserMethod = "tryParseSVEPattern"; let ParserMethod = "tryParseSVEPattern";
let PredicateMethod = "isSVEPattern"; let PredicateMethod = "isSVEPattern";
let RenderMethod = "addImmOperands"; let RenderMethod = "addImmOperands";
let DiagnosticType = "InvalidSVEPattern"; let DiagnosticType = "InvalidSVEPattern";
} }
▲ Show 20 Lines • Show All 7,711 Lines • Show Last 20 Lines