This is an archive of the discontinued LLVM Phabricator instance.

Differential D63204

[AMDGPU] gfx1010 core wave32 changes
ClosedPublic

Authored by rampitec on Jun 12 2019, 7:42 AM.

Details

Reviewers
kzhuravl
msearles
Commits
rL363934: [AMDGPU] gfx1010 core wave32 changes
rG0846c125f98b: [AMDGPU] gfx1010 core wave32 changes

Diff Detail

Event Timeline

rampitec created this revision.Jun 12 2019, 7:42 AM
Herald added subscribers: t-tye, tpr, dstuttard and 5 others. · View Herald TranscriptJun 12 2019, 7:42 AM
rampitec added a parent revision: D63203: [AMDGPU] gfx1010 dpp16 and dpp8.Jun 12 2019, 7:42 AM
rampitec mentioned this in D63205: Virtualize TargetInstrInfo::getRegClass().Jun 12 2019, 7:45 AM
rampitec added a parent revision: D63205: Virtualize TargetInstrInfo::getRegClass().
rampitec added a child revision: D63206: [AMDGPU] gfx1010 wavefrontsize intrinsic folding.Jun 12 2019, 7:48 AM
rampitec added a child revision: D63207: [AMDGPU] gfx1010 wave32 metadata.
rampitec added a child revision: D63208: [AMDGPU] Pass to propagate ABI attributes from kernels to the functions.Jun 12 2019, 7:50 AM
rampitec added a child revision: D63209: [AMDGPU] gfx1010 wave32 clang support.
rampitec updated this revision to Diff 204338.Jun 12 2019, 11:51 AM

Rebased. Some of the tests already in the trunk.

rampitec mentioned this in D63293: [AMDGPU] gfx1010 base changes for wave32.Jun 13 2019, 11:51 AM
rampitec updated this revision to Diff 204605.Jun 13 2019, 12:23 PM

Rebased.

Herald added a subscriber: jsji. · View Herald TranscriptJun 13 2019, 12:23 PM
jsji removed a subscriber: jsji.Jun 13 2019, 12:26 PM
rampitec mentioned this in D63301: [AMDGPU] gfx1010 wave32 icmp/fcmp intrinsic changes for wave32.Jun 13 2019, 3:06 PM
rampitec updated this revision to Diff 204662.Jun 13 2019, 4:55 PM

Rebased.

kzhuravl accepted this revision.Jun 13 2019, 5:24 PM

LGTM

This revision is now accepted and ready to land.Jun 13 2019, 5:24 PM
uabelho added a subscriber: uabelho.Jun 14 2019, 5:00 AM
rampitec mentioned this in rL363416: [AMDGPU] gfx1010 BoolReg definition. NFC..Jun 14 2019, 9:23 AM
rampitec mentioned this in rGcdf339266b43: [AMDGPU] gfx1010 BoolReg definition. NFC..
rampitec updated this revision to Diff 204790.Jun 14 2019, 9:28 AM

Rebased.

rampitec mentioned this in D63351: [AMDGPU] gfx10 conditional registers handling.Jun 14 2019, 11:57 AM
rampitec updated this revision to Diff 204954.Jun 16 2019, 9:47 AM

Rebased.

rampitec updated this revision to Diff 204959.Jun 16 2019, 10:16 AM

Rebased.

rampitec updated this revision to Diff 205091.Jun 17 2019, 8:48 AM

Rebased.

rampitec removed a child revision: D63207: [AMDGPU] gfx1010 wave32 metadata.Jun 17 2019, 9:44 AM
rampitec updated this revision to Diff 205106.Jun 17 2019, 10:02 AM

Rebased. Reenabled HSA metadata test disabled with previous commit due to dependency.

rampitec removed a child revision: D63208: [AMDGPU] Pass to propagate ABI attributes from kernels to the functions.Jun 17 2019, 10:43 AM
rampitec removed a child revision: D63206: [AMDGPU] gfx1010 wavefrontsize intrinsic folding.Jun 17 2019, 10:53 AM
rampitec mentioned this in rL363625: [AMDGPU] Use custom inserter for gfx10 VOP2b.Jun 17 2019, 3:35 PM
rampitec mentioned this in rG121956108f27: [AMDGPU] Use custom inserter for gfx10 VOP2b.
rampitec updated this revision to Diff 205201.Jun 17 2019, 3:42 PM

Rebased and removed unused S_ADDC_U64_PSEUDO/S_SUBC_U64_PSEUDO.

rampitec mentioned this in D63506: [AMDGPU] gfx1010 disassembler changes for wave32.Jun 18 2019, 11:28 AM
rampitec updated this revision to Diff 205412.Jun 18 2019, 12:10 PM

Rebased.

rampitec mentioned this in D63511: [AMDGPU] gfx10 wave32 patterns.Jun 18 2019, 12:40 PM
rampitec updated this revision to Diff 205421.Jun 18 2019, 12:59 PM

Rebased.

Closed by commit rG0846c125f98b: [AMDGPU] gfx1010 core wave32 changes (authored by rampitec). · Explain WhyJun 20 2019, 8:08 AM
This revision was automatically updated to reflect the committed changes.
Herald added a project: Restricted Project. · View Herald TranscriptJun 20 2019, 8:08 AM
Herald added a subscriber: hiraditya. · View Herald Transcript

Revision Contents

PathSize
include/
llvm/
IR/
22 lines
lib/
Target/
AMDGPU/
2 lines
3 lines
20 lines
10 lines
4 lines
8 lines
10 lines
AsmParser/
58 lines
Disassembler/
2 lines
20 lines
MCTargetDesc/
7 lines
2 lines
44 lines
22 lines
134 lines
20 lines
6 lines
11 lines
216 lines
46 lines
136 lines
7 lines
81 lines
68 lines
87 lines
51 lines
5 lines
15 lines
29 lines
24 lines
25 lines
50 lines
50 lines
Utils/
18 lines
22 lines
43 lines
4 lines
25 lines
Transforms/
InstCombine/
4 lines
test/
CodeGen/
AMDGPU/
9 lines
12 lines
6 lines
6 lines
1 line
4 lines
2 lines
2 lines
27 lines
20 lines
20 lines
168 lines
36 lines
286 lines
66 lines
8 lines
32 lines
5 lines
18 lines
5 lines
8 lines
4 lines
12 lines
2 lines
12 lines
2 lines
2 lines
1154 lines
8 lines
5 lines
MC/
AMDGPU/
2 lines
10 lines
412 lines
Disassembler/
AMDGPU/
6 lines
164 lines
Transforms/
InstCombine/
AMDGPU/
288 lines
Verifier/
AMDGPU/
12 lines

Diff 204287

include/llvm/IR/IntrinsicsAMDGPU.td

Show First 20 Lines • Show All 181 Lines • ▼ Show 20 Lines
// Set EXEC according to a thread count packed in an SGPR input: // Set EXEC according to a thread count packed in an SGPR input:
// thread_count = (input >> bitoffset) & 0x7f; // thread_count = (input >> bitoffset) & 0x7f;
// This is always moved to the beginning of the basic block. // This is always moved to the beginning of the basic block.
def int_amdgcn_init_exec_from_input : Intrinsic<[], def int_amdgcn_init_exec_from_input : Intrinsic<[],
[llvm_i32_ty, // 32-bit SGPR input [llvm_i32_ty, // 32-bit SGPR input
llvm_i32_ty], // bit offset of the thread count llvm_i32_ty], // bit offset of the thread count
[IntrConvergent]>; [IntrConvergent]>;
def int_amdgcn_wavefrontsize :
GCCBuiltin<"__builtin_amdgcn_wavefrontsize">,
Intrinsic<[llvm_i32_ty], [], [IntrNoMem, IntrSpeculatable]>;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Instruction Intrinsics // Instruction Intrinsics
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// The first parameter is s_sendmsg immediate (i16), // The first parameter is s_sendmsg immediate (i16),
// the second one is copied to m0 // the second one is copied to m0
def int_amdgcn_s_sendmsg : GCCBuiltin<"__builtin_amdgcn_s_sendmsg">, def int_amdgcn_s_sendmsg : GCCBuiltin<"__builtin_amdgcn_s_sendmsg">,
▲ Show 20 Lines • Show All 1,099 Lines • ▼ Show 20 Lines
def int_amdgcn_cvt_pk_u8_f32 : def int_amdgcn_cvt_pk_u8_f32 :
GCCBuiltin<"__builtin_amdgcn_cvt_pk_u8_f32">, GCCBuiltin<"__builtin_amdgcn_cvt_pk_u8_f32">,
Intrinsic<[llvm_i32_ty], [llvm_float_ty, llvm_i32_ty, llvm_i32_ty], Intrinsic<[llvm_i32_ty], [llvm_float_ty, llvm_i32_ty, llvm_i32_ty],
[IntrNoMem, IntrSpeculatable] [IntrNoMem, IntrSpeculatable]
>; >;
def int_amdgcn_icmp : def int_amdgcn_icmp :
Intrinsic<[llvm_i64_ty], [llvm_anyint_ty, LLVMMatchType<0>, llvm_i32_ty], Intrinsic<[llvm_anyint_ty], [llvm_anyint_ty, LLVMMatchType<1>, llvm_i32_ty],
[IntrNoMem, IntrConvergent, ImmArg<2>]>; [IntrNoMem, IntrConvergent, ImmArg<2>]>;
def int_amdgcn_fcmp : def int_amdgcn_fcmp :
Intrinsic<[llvm_i64_ty], [llvm_anyfloat_ty, LLVMMatchType<0>, llvm_i32_ty], Intrinsic<[llvm_anyint_ty], [llvm_anyfloat_ty, LLVMMatchType<1>, llvm_i32_ty],
[IntrNoMem, IntrConvergent, ImmArg<2>]>; [IntrNoMem, IntrConvergent, ImmArg<2>]>;
def int_amdgcn_readfirstlane : def int_amdgcn_readfirstlane :
GCCBuiltin<"__builtin_amdgcn_readfirstlane">, GCCBuiltin<"__builtin_amdgcn_readfirstlane">,
Intrinsic<[llvm_i32_ty], [llvm_i32_ty], [IntrNoMem, IntrConvergent]>; Intrinsic<[llvm_i32_ty], [llvm_i32_ty], [IntrNoMem, IntrConvergent]>;
// The lane argument must be uniform across the currently active threads of the // The lane argument must be uniform across the currently active threads of the
// current wave. Otherwise, the result is undefined. // current wave. Otherwise, the result is undefined.
▲ Show 20 Lines • Show All 253 Lines • ▼ Show 20 Lines Intrinsic<
], ],
[IntrNoMem, IntrSpeculatable, ImmArg<3>] [IntrNoMem, IntrSpeculatable, ImmArg<3>]
>; >;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Special Intrinsics for backend internal use only. No frontend // Special Intrinsics for backend internal use only. No frontend
// should emit calls to these. // should emit calls to these.
// ===----------------------------------------------------------------------===// // ===----------------------------------------------------------------------===//
def int_amdgcn_if : Intrinsic<[llvm_i1_ty, llvm_i64_ty], def int_amdgcn_if : Intrinsic<[llvm_i1_ty, llvm_anyint_ty],
[llvm_i1_ty], [IntrConvergent] [llvm_i1_ty], [IntrConvergent]
>; >;
def int_amdgcn_else : Intrinsic<[llvm_i1_ty, llvm_i64_ty], def int_amdgcn_else : Intrinsic<[llvm_i1_ty, llvm_anyint_ty],
[llvm_i64_ty], [IntrConvergent] [llvm_anyint_ty], [IntrConvergent]
>; >;
def int_amdgcn_if_break : Intrinsic<[llvm_i64_ty], def int_amdgcn_if_break : Intrinsic<[llvm_anyint_ty],
[llvm_i1_ty, llvm_i64_ty], [IntrNoMem, IntrConvergent] [llvm_i1_ty, llvm_anyint_ty], [IntrNoMem, IntrConvergent]
>; >;
def int_amdgcn_loop : Intrinsic<[llvm_i1_ty], def int_amdgcn_loop : Intrinsic<[llvm_i1_ty],
[llvm_i64_ty], [IntrConvergent] [llvm_anyint_ty], [IntrConvergent]
>; >;
def int_amdgcn_end_cf : Intrinsic<[], [llvm_i64_ty], [IntrConvergent]>; def int_amdgcn_end_cf : Intrinsic<[], [llvm_anyint_ty], [IntrConvergent]>;
// Represent unreachable in a divergent region. // Represent unreachable in a divergent region.
def int_amdgcn_unreachable : Intrinsic<[], [], [IntrConvergent]>; def int_amdgcn_unreachable : Intrinsic<[], [], [IntrConvergent]>;
// Emit 2.5 ulp, no denormal division. Should only be inserted by // Emit 2.5 ulp, no denormal division. Should only be inserted by
// pass based on !fpmath metadata. // pass based on !fpmath metadata.
def int_amdgcn_fdiv_fast : Intrinsic< def int_amdgcn_fdiv_fast : Intrinsic<
[llvm_float_ty], [llvm_float_ty, llvm_float_ty], [llvm_float_ty], [llvm_float_ty, llvm_float_ty],
[IntrNoMem, IntrSpeculatable] [IntrNoMem, IntrSpeculatable]
>; >;
} }

lib/Target/AMDGPU/AMDGPU.td

Show First 20 Lines • Show All 759 Lines • ▼ Show 20 Lines
} }
def FeatureISAVersion10_1_0 : FeatureSet< def FeatureISAVersion10_1_0 : FeatureSet<
!listconcat(FeatureGroup.GFX10_1_Bugs, !listconcat(FeatureGroup.GFX10_1_Bugs,
[FeatureGFX10, [FeatureGFX10,
FeatureLDSBankCount32, FeatureLDSBankCount32,
FeatureDLInsts, FeatureDLInsts,
FeatureNSAEncoding, FeatureNSAEncoding,
FeatureWavefrontSize64, FeatureWavefrontSize32,
FeatureScalarStores, FeatureScalarStores,
FeatureScalarAtomics, FeatureScalarAtomics,
FeatureScalarFlatScratchInsts, FeatureScalarFlatScratchInsts,
FeatureLdsMisalignedBug, FeatureLdsMisalignedBug,
FeatureDoesNotSupportXNACK, FeatureDoesNotSupportXNACK,
FeatureCodeObjectV3])>; FeatureCodeObjectV3])>;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 255 Lines • Show Last 20 Lines

lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp

Show First 20 Lines • Show All 243 Lines • ▼ Show 20 Linesvoid AMDGPUAtomicOptimizer::optimizeAtomic(Instruction &I,
// This is the value in the atomic operation we need to combine in order to // This is the value in the atomic operation we need to combine in order to
// reduce the number of atomic operations. // reduce the number of atomic operations.
Value *const V = I.getOperand(ValIdx); Value *const V = I.getOperand(ValIdx);
// We need to know how many lanes are active within the wavefront, and we do // We need to know how many lanes are active within the wavefront, and we do
// this by doing a ballot of active lanes. // this by doing a ballot of active lanes.
CallInst *const Ballot = CallInst *const Ballot =
B.CreateIntrinsic(Intrinsic::amdgcn_icmp, {B.getInt32Ty()}, B.CreateIntrinsic(Intrinsic::amdgcn_icmp,
{B.getInt64Ty(), B.getInt32Ty()},
{B.getInt32(1), B.getInt32(0), B.getInt32(33)}); {B.getInt32(1), B.getInt32(0), B.getInt32(33)});
// We need to know how many lanes are active within the wavefront that are // We need to know how many lanes are active within the wavefront that are
// below us. If we counted each lane linearly starting from 0, a lane is // below us. If we counted each lane linearly starting from 0, a lane is
// below us only if its associated index was less than ours. We do this by // below us only if its associated index was less than ours. We do this by
// using the mbcnt intrinsic. // using the mbcnt intrinsic.
Value *const BitCast = B.CreateBitCast(Ballot, VecTy); Value *const BitCast = B.CreateBitCast(Ballot, VecTy);
Value *const ExtractLo = B.CreateExtractElement(BitCast, B.getInt32(0)); Value *const ExtractLo = B.CreateExtractElement(BitCast, B.getInt32(0));
▲ Show 20 Lines • Show All 178 Lines • Show Last 20 Lines

lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp

Show First 20 Lines • Show All 1,004 Lines • ▼ Show 20 Linesvoid AMDGPUDAGToDAGISel::SelectDIV_SCALE(SDNode *N) {
unsigned Opc unsigned Opc
= (VT == MVT::f64) ? AMDGPU::V_DIV_SCALE_F64 : AMDGPU::V_DIV_SCALE_F32; = (VT == MVT::f64) ? AMDGPU::V_DIV_SCALE_F64 : AMDGPU::V_DIV_SCALE_F32;
SDValue Ops[] = { N->getOperand(0), N->getOperand(1), N->getOperand(2) }; SDValue Ops[] = { N->getOperand(0), N->getOperand(1), N->getOperand(2) };
CurDAG->SelectNodeTo(N, Opc, N->getVTList(), Ops); CurDAG->SelectNodeTo(N, Opc, N->getVTList(), Ops);
} }
void AMDGPUDAGToDAGISel::SelectDIV_FMAS(SDNode *N) { void AMDGPUDAGToDAGISel::SelectDIV_FMAS(SDNode *N) {
const GCNSubtarget *ST = static_cast<const GCNSubtarget *>(Subtarget);
const SIRegisterInfo *TRI = ST->getRegisterInfo();
SDLoc SL(N); SDLoc SL(N);
EVT VT = N->getValueType(0); EVT VT = N->getValueType(0);
assert(VT == MVT::f32 || VT == MVT::f64); assert(VT == MVT::f32 || VT == MVT::f64);
unsigned Opc unsigned Opc
= (VT == MVT::f64) ? AMDGPU::V_DIV_FMAS_F64 : AMDGPU::V_DIV_FMAS_F32; = (VT == MVT::f64) ? AMDGPU::V_DIV_FMAS_F64 : AMDGPU::V_DIV_FMAS_F32;
SDValue CarryIn = N->getOperand(3); SDValue CarryIn = N->getOperand(3);
// V_DIV_FMAS implicitly reads VCC. // V_DIV_FMAS implicitly reads VCC.
SDValue VCC = CurDAG->getCopyToReg(CurDAG->getEntryNode(), SL, SDValue VCC = CurDAG->getCopyToReg(CurDAG->getEntryNode(), SL,
AMDGPU::VCC, CarryIn, SDValue()); TRI->getVCC(), CarryIn, SDValue());
SDValue Ops[10]; SDValue Ops[10];
SelectVOP3Mods0(N->getOperand(0), Ops[1], Ops[0], Ops[6], Ops[7]); SelectVOP3Mods0(N->getOperand(0), Ops[1], Ops[0], Ops[6], Ops[7]);
SelectVOP3Mods(N->getOperand(1), Ops[3], Ops[2]); SelectVOP3Mods(N->getOperand(1), Ops[3], Ops[2]);
SelectVOP3Mods(N->getOperand(2), Ops[5], Ops[4]); SelectVOP3Mods(N->getOperand(2), Ops[5], Ops[4]);
Ops[8] = VCC; Ops[8] = VCC;
▲ Show 20 Lines • Show All 800 Lines • ▼ Show 20 Linesvoid AMDGPUDAGToDAGISel::SelectBRCOND(SDNode *N) {
SDValue Cond = N->getOperand(1); SDValue Cond = N->getOperand(1);
if (Cond.isUndef()) { if (Cond.isUndef()) {
CurDAG->SelectNodeTo(N, AMDGPU::SI_BR_UNDEF, MVT::Other, CurDAG->SelectNodeTo(N, AMDGPU::SI_BR_UNDEF, MVT::Other,
N->getOperand(2), N->getOperand(0)); N->getOperand(2), N->getOperand(0));
return; return;
} }
const GCNSubtarget *ST = static_cast<const GCNSubtarget *>(Subtarget);
const SIRegisterInfo *TRI = ST->getRegisterInfo();
bool UseSCCBr = isCBranchSCC(N) && isUniformBr(N); bool UseSCCBr = isCBranchSCC(N) && isUniformBr(N);
unsigned BrOp = UseSCCBr ? AMDGPU::S_CBRANCH_SCC1 : AMDGPU::S_CBRANCH_VCCNZ; unsigned BrOp = UseSCCBr ? AMDGPU::S_CBRANCH_SCC1 : AMDGPU::S_CBRANCH_VCCNZ;
unsigned CondReg = UseSCCBr ? AMDGPU::SCC : AMDGPU::VCC; unsigned CondReg = UseSCCBr ? (unsigned)AMDGPU::SCC : TRI->getVCC();
SDLoc SL(N); SDLoc SL(N);
if (!UseSCCBr) { if (!UseSCCBr) {
// This is the case that we are selecting to S_CBRANCH_VCCNZ. We have not // This is the case that we are selecting to S_CBRANCH_VCCNZ. We have not
// analyzed what generates the vcc value, so we do not know whether vcc // analyzed what generates the vcc value, so we do not know whether vcc
// bits for disabled lanes are 0. Thus we need to mask out bits for // bits for disabled lanes are 0. Thus we need to mask out bits for
// disabled lanes. // disabled lanes.
// //
// For the case that we select S_CBRANCH_SCC1 and it gets // For the case that we select S_CBRANCH_SCC1 and it gets
// changed to S_CBRANCH_VCCNZ in SIFixSGPRCopies, SIFixSGPRCopies calls // changed to S_CBRANCH_VCCNZ in SIFixSGPRCopies, SIFixSGPRCopies calls
// SIInstrInfo::moveToVALU which inserts the S_AND). // SIInstrInfo::moveToVALU which inserts the S_AND).
// //
// We could add an analysis of what generates the vcc value here and omit // We could add an analysis of what generates the vcc value here and omit
// the S_AND when is unnecessary. But it would be better to add a separate // the S_AND when is unnecessary. But it would be better to add a separate
// pass after SIFixSGPRCopies to do the unnecessary S_AND removal, so it // pass after SIFixSGPRCopies to do the unnecessary S_AND removal, so it
// catches both cases. // catches both cases.
Cond = SDValue(CurDAG->getMachineNode(AMDGPU::S_AND_B64, SL, MVT::i1, Cond = SDValue(CurDAG->getMachineNode(ST->isWave32() ? AMDGPU::S_AND_B32
CurDAG->getRegister(AMDGPU::EXEC, MVT::i1), : AMDGPU::S_AND_B64,
SL, MVT::i1,
CurDAG->getRegister(ST->isWave32() ? AMDGPU::EXEC_LO
: AMDGPU::EXEC,
MVT::i1),
Cond), Cond),
0); 0);
} }
SDValue VCC = CurDAG->getCopyToReg(N->getOperand(0), SL, CondReg, Cond); SDValue VCC = CurDAG->getCopyToReg(N->getOperand(0), SL, CondReg, Cond);
CurDAG->SelectNodeTo(N, BrOp, MVT::Other, CurDAG->SelectNodeTo(N, BrOp, MVT::Other,
N->getOperand(2), // Basic Block N->getOperand(2), // Basic Block
VCC.getValue(0)); VCC.getValue(0));
} }
▲ Show 20 Lines • Show All 628 Lines • Show Last 20 Lines

lib/Target/AMDGPU/AMDGPUInstrInfo.td

Show First 20 Lines • Show All 44 Lines • ▼ Show 20 Lines
// float, float, float, vcc // float, float, float, vcc
def AMDGPUFmasOp : SDTypeProfile<1, 4, def AMDGPUFmasOp : SDTypeProfile<1, 4,
[SDTCisFP<0>, SDTCisSameAs<0, 1>, SDTCisSameAs<0, 2>, SDTCisSameAs<0, 3>, SDTCisInt<4>] [SDTCisFP<0>, SDTCisSameAs<0, 1>, SDTCisSameAs<0, 2>, SDTCisSameAs<0, 3>, SDTCisInt<4>]
>; >;
def AMDGPUKillSDT : SDTypeProfile<0, 1, [SDTCisInt<0>]>; def AMDGPUKillSDT : SDTypeProfile<0, 1, [SDTCisInt<0>]>;
def AMDGPUIfOp : SDTypeProfile<1, 2, def AMDGPUIfOp : SDTypeProfile<1, 2,
[SDTCisVT<0, i64>, SDTCisVT<1, i1>, SDTCisVT<2, OtherVT>] [SDTCisVT<0, i1>, SDTCisVT<1, i1>, SDTCisVT<2, OtherVT>]
>; >;
def AMDGPUElseOp : SDTypeProfile<1, 2, def AMDGPUElseOp : SDTypeProfile<1, 2,
[SDTCisVT<0, i64>, SDTCisVT<1, i64>, SDTCisVT<2, OtherVT>] [SDTCisVT<0, i1>, SDTCisVT<1, i1>, SDTCisVT<2, OtherVT>]
>; >;
def AMDGPULoopOp : SDTypeProfile<0, 2, def AMDGPULoopOp : SDTypeProfile<0, 2,
[SDTCisVT<0, i64>, SDTCisVT<1, OtherVT>] [SDTCisVT<0, i1>, SDTCisVT<1, OtherVT>]
>; >;
def AMDGPUIfBreakOp : SDTypeProfile<1, 2, def AMDGPUIfBreakOp : SDTypeProfile<1, 2,
[SDTCisVT<0, i64>, SDTCisVT<1, i1>, SDTCisVT<2, i64>] [SDTCisVT<0, i1>, SDTCisVT<1, i1>, SDTCisVT<2, i1>]
>; >;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// AMDGPU DAG Nodes // AMDGPU DAG Nodes
// //
def AMDGPUif : SDNode<"AMDGPUISD::IF", AMDGPUIfOp, [SDNPHasChain]>; def AMDGPUif : SDNode<"AMDGPUISD::IF", AMDGPUIfOp, [SDNPHasChain]>;
def AMDGPUelse : SDNode<"AMDGPUISD::ELSE", AMDGPUElseOp, [SDNPHasChain]>; def AMDGPUelse : SDNode<"AMDGPUISD::ELSE", AMDGPUElseOp, [SDNPHasChain]>;
▲ Show 20 Lines • Show All 121 Lines • ▼ Show 20 Lines
// out = (src0 + src1 > 0xFFFFFFFF) ? 1 : 0 // out = (src0 + src1 > 0xFFFFFFFF) ? 1 : 0
def AMDGPUcarry : SDNode<"AMDGPUISD::CARRY", SDTIntBinOp, []>; def AMDGPUcarry : SDNode<"AMDGPUISD::CARRY", SDTIntBinOp, []>;
// out = (src1 > src0) ? 1 : 0 // out = (src1 > src0) ? 1 : 0
def AMDGPUborrow : SDNode<"AMDGPUISD::BORROW", SDTIntBinOp, []>; def AMDGPUborrow : SDNode<"AMDGPUISD::BORROW", SDTIntBinOp, []>;
def AMDGPUSetCCOp : SDTypeProfile<1, 3, [ // setcc def AMDGPUSetCCOp : SDTypeProfile<1, 3, [ // setcc
SDTCisVT<0, i64>, SDTCisSameAs<1, 2>, SDTCisVT<3, OtherVT> SDTCisInt<0>, SDTCisSameAs<1, 2>, SDTCisVT<3, OtherVT>
]>; ]>;
def AMDGPUsetcc : SDNode<"AMDGPUISD::SETCC", AMDGPUSetCCOp>; def AMDGPUsetcc : SDNode<"AMDGPUISD::SETCC", AMDGPUSetCCOp>;
def AMDGPUSetRegOp : SDTypeProfile<0, 2, [ def AMDGPUSetRegOp : SDTypeProfile<0, 2, [
SDTCisInt<0>, SDTCisInt<1> SDTCisInt<0>, SDTCisInt<1>
]>; ]>;
▲ Show 20 Lines • Show All 216 Lines • Show Last 20 Lines

lib/Target/AMDGPU/AMDGPUInstructions.td

Show First 20 Lines • Show All 63 Lines • ▼ Show 20 Lines
} }
def TruePredicate : Predicate<"true">; def TruePredicate : Predicate<"true">;
class PredicateControl { class PredicateControl {
Predicate SubtargetPredicate = TruePredicate; Predicate SubtargetPredicate = TruePredicate;
list<Predicate> AssemblerPredicates = []; list<Predicate> AssemblerPredicates = [];
Predicate AssemblerPredicate = TruePredicate; Predicate AssemblerPredicate = TruePredicate;
Predicate WaveSizePredicate = TruePredicate;
list<Predicate> OtherPredicates = []; list<Predicate> OtherPredicates = [];
list<Predicate> Predicates = !listconcat([SubtargetPredicate, list<Predicate> Predicates = !listconcat([SubtargetPredicate,
AssemblerPredicate], AssemblerPredicate,
WaveSizePredicate],
AssemblerPredicates, AssemblerPredicates,
OtherPredicates); OtherPredicates);
} }
class AMDGPUPat<dag pattern, dag result> : Pat<pattern, result>, class AMDGPUPat<dag pattern, dag result> : Pat<pattern, result>,
PredicateControl; PredicateControl;
def FP16Denormals : Predicate<"Subtarget->hasFP16Denormals()">; def FP16Denormals : Predicate<"Subtarget->hasFP16Denormals()">;
def FP32Denormals : Predicate<"Subtarget->hasFP32Denormals()">; def FP32Denormals : Predicate<"Subtarget->hasFP32Denormals()">;
▲ Show 20 Lines • Show All 797 Lines • Show Last 20 Lines

lib/Target/AMDGPU/AMDGPUSubtarget.h

Show First 20 Lines • Show All 1,022 Lines • ▼ Show 20 Linespublic:
/// subtarget's specifications, or does not meet number of waves per execution /// subtarget's specifications, or does not meet number of waves per execution
/// unit requirement. /// unit requirement.
unsigned getMaxNumVGPRs(const MachineFunction &MF) const; unsigned getMaxNumVGPRs(const MachineFunction &MF) const;
void getPostRAMutations( void getPostRAMutations(
std::vector<std::unique_ptr<ScheduleDAGMutation>> &Mutations) std::vector<std::unique_ptr<ScheduleDAGMutation>> &Mutations)
const override; const override;
bool isWave32() const {
return WavefrontSize == 32;
}
const TargetRegisterClass *getBoolRC() const {
return getRegisterInfo()->getBoolRC();
}
/// \returns Maximum number of work groups per compute unit supported by the /// \returns Maximum number of work groups per compute unit supported by the
/// subtarget and limited by given \p FlatWorkGroupSize. /// subtarget and limited by given \p FlatWorkGroupSize.
unsigned getMaxWorkGroupsPerCU(unsigned FlatWorkGroupSize) const override { unsigned getMaxWorkGroupsPerCU(unsigned FlatWorkGroupSize) const override {
return AMDGPU::IsaInfo::getMaxWorkGroupsPerCU(this, FlatWorkGroupSize); return AMDGPU::IsaInfo::getMaxWorkGroupsPerCU(this, FlatWorkGroupSize);
} }
/// \returns Minimum flat work group size supported by the subtarget. /// \returns Minimum flat work group size supported by the subtarget.
unsigned getMinFlatWorkGroupSize() const override { unsigned getMinFlatWorkGroupSize() const override {
▲ Show 20 Lines • Show All 161 Lines • Show Last 20 Lines

lib/Target/AMDGPU/AMDGPUSubtarget.cpp

Show First 20 Lines • Show All 88 Lines • ▼ Show 20 LinesGCNSubtarget::initializeSubtargetDependencies(const Triple &TT,
if (getGeneration() >= AMDGPUSubtarget::SOUTHERN_ISLANDS) { if (getGeneration() >= AMDGPUSubtarget::SOUTHERN_ISLANDS) {
FullFS += "+fp64-fp16-denormals,"; FullFS += "+fp64-fp16-denormals,";
} else { } else {
FullFS += "-fp32-denormals,"; FullFS += "-fp32-denormals,";
} }
FullFS += "+enable-prt-strict-null,"; // This is overridden by a disable in FS FullFS += "+enable-prt-strict-null,"; // This is overridden by a disable in FS
// Disable mutually exclusive bits.
if (FS.find_lower("+wavefrontsize") != StringRef::npos) {
if (FS.find_lower("wavefrontsize16") == StringRef::npos)
FullFS += "-wavefrontsize16,";
if (FS.find_lower("wavefrontsize32") == StringRef::npos)
FullFS += "-wavefrontsize32,";
if (FS.find_lower("wavefrontsize64") == StringRef::npos)
FullFS += "-wavefrontsize64,";
}
FullFS += FS; FullFS += FS;
ParseSubtargetFeatures(GPU, FullFS); ParseSubtargetFeatures(GPU, FullFS);
// We don't support FP64 for EG/NI atm. // We don't support FP64 for EG/NI atm.
assert(!hasFP64() || (getGeneration() >= AMDGPUSubtarget::SOUTHERN_ISLANDS)); assert(!hasFP64() || (getGeneration() >= AMDGPUSubtarget::SOUTHERN_ISLANDS));
// Unless +-flat-for-global is specified, turn on FlatForGlobal for all OS-es // Unless +-flat-for-global is specified, turn on FlatForGlobal for all OS-es
▲ Show 20 Lines • Show All 651 Lines • Show Last 20 Lines

lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

Show First 20 Lines • Show All 369 Lines • ▼ Show 20 Linespublic:
bool isSCSrcB32() const { bool isSCSrcB32() const {
return isRegOrInlineNoMods(AMDGPU::SReg_32RegClassID, MVT::i32); return isRegOrInlineNoMods(AMDGPU::SReg_32RegClassID, MVT::i32);
} }
bool isSCSrcB64() const { bool isSCSrcB64() const {
return isRegOrInlineNoMods(AMDGPU::SReg_64RegClassID, MVT::i64); return isRegOrInlineNoMods(AMDGPU::SReg_64RegClassID, MVT::i64);
} }
bool isBoolReg() const;
bool isSCSrcF16() const { bool isSCSrcF16() const {
return isRegOrInlineNoMods(AMDGPU::SReg_32RegClassID, MVT::f16); return isRegOrInlineNoMods(AMDGPU::SReg_32RegClassID, MVT::f16);
} }
bool isSCSrcV2F16() const { bool isSCSrcV2F16() const {
return isSCSrcF16(); return isSCSrcF16();
} }
▲ Show 20 Lines • Show All 225 Lines • ▼ Show 20 Linespublic:
} }
void addKImmFP32Operands(MCInst &Inst, unsigned N) const { void addKImmFP32Operands(MCInst &Inst, unsigned N) const {
addKImmFPOperands<32>(Inst, N); addKImmFPOperands<32>(Inst, N);
} }
void addRegOperands(MCInst &Inst, unsigned N) const; void addRegOperands(MCInst &Inst, unsigned N) const;
void addBoolRegOperands(MCInst &Inst, unsigned N) const {
addRegOperands(Inst, N);
}
void addRegOrImmOperands(MCInst &Inst, unsigned N) const { void addRegOrImmOperands(MCInst &Inst, unsigned N) const {
if (isRegKind()) if (isRegKind())
addRegOperands(Inst, N); addRegOperands(Inst, N);
else if (isExpr()) else if (isExpr())
Inst.addOperand(MCOperand::createExpr(Expr)); Inst.addOperand(MCOperand::createExpr(Expr));
else else
addImmOperands(Inst, N); addImmOperands(Inst, N);
} }
▲ Show 20 Lines • Show All 249 Lines • ▼ Show 20 Linesprivate:
bool OutOfRangeError(SMRange Range); bool OutOfRangeError(SMRange Range);
/// Calculate VGPR/SGPR blocks required for given target, reserved /// Calculate VGPR/SGPR blocks required for given target, reserved
/// registers, and user-specified NextFreeXGPR values. /// registers, and user-specified NextFreeXGPR values.
/// ///
/// \param Features [in] Target features, used for bug corrections. /// \param Features [in] Target features, used for bug corrections.
/// \param VCCUsed [in] Whether VCC special SGPR is reserved. /// \param VCCUsed [in] Whether VCC special SGPR is reserved.
/// \param FlatScrUsed [in] Whether FLAT_SCRATCH special SGPR is reserved. /// \param FlatScrUsed [in] Whether FLAT_SCRATCH special SGPR is reserved.
/// \param XNACKUsed [in] Whether XNACK_MASK special SGPR is reserved. /// \param XNACKUsed [in] Whether XNACK_MASK special SGPR is reserved.
/// \param EnableWavefrontSize32 [in] Value of ENABLE_WAVEFRONT_SIZE32 kernel
/// descriptor field, if valid.
/// \param NextFreeVGPR [in] Max VGPR number referenced, plus one. /// \param NextFreeVGPR [in] Max VGPR number referenced, plus one.
/// \param VGPRRange [in] Token range, used for VGPR diagnostics. /// \param VGPRRange [in] Token range, used for VGPR diagnostics.
/// \param NextFreeSGPR [in] Max SGPR number referenced, plus one. /// \param NextFreeSGPR [in] Max SGPR number referenced, plus one.
/// \param SGPRRange [in] Token range, used for SGPR diagnostics. /// \param SGPRRange [in] Token range, used for SGPR diagnostics.
/// \param VGPRBlocks [out] Result VGPR block count. /// \param VGPRBlocks [out] Result VGPR block count.
/// \param SGPRBlocks [out] Result SGPR block count. /// \param SGPRBlocks [out] Result SGPR block count.
bool calculateGPRBlocks(const FeatureBitset &Features, bool VCCUsed, bool calculateGPRBlocks(const FeatureBitset &Features, bool VCCUsed,
bool FlatScrUsed, bool XNACKUsed, bool FlatScrUsed, bool XNACKUsed,
unsigned NextFreeVGPR, SMRange VGPRRange, Optional<bool> EnableWavefrontSize32, unsigned NextFreeVGPR,
unsigned NextFreeSGPR, SMRange SGPRRange, SMRange VGPRRange, unsigned NextFreeSGPR,
unsigned &VGPRBlocks, unsigned &SGPRBlocks); SMRange SGPRRange, unsigned &VGPRBlocks,
unsigned &SGPRBlocks);
bool ParseDirectiveAMDGCNTarget(); bool ParseDirectiveAMDGCNTarget();
bool ParseDirectiveAMDHSAKernel(); bool ParseDirectiveAMDHSAKernel();
bool ParseDirectiveMajorMinor(uint32_t &Major, uint32_t &Minor); bool ParseDirectiveMajorMinor(uint32_t &Major, uint32_t &Minor);
bool ParseDirectiveHSACodeObjectVersion(); bool ParseDirectiveHSACodeObjectVersion();
bool ParseDirectiveHSACodeObjectISA(); bool ParseDirectiveHSACodeObjectISA();
bool ParseAMDKernelCodeTValue(StringRef ID, amd_kernel_code_t &Header); bool ParseAMDKernelCodeTValue(StringRef ID, amd_kernel_code_t &Header);
bool ParseDirectiveAMDKernelCodeT(); bool ParseDirectiveAMDKernelCodeT();
bool subtargetHasRegister(const MCRegisterInfo &MRI, unsigned RegNo) const; bool subtargetHasRegister(const MCRegisterInfo &MRI, unsigned RegNo) const;
▲ Show 20 Lines • Show All 247 Lines • ▼ Show 20 Linesprivate:
bool validateMIMGAtomicDMask(const MCInst &Inst); bool validateMIMGAtomicDMask(const MCInst &Inst);
bool validateMIMGGatherDMask(const MCInst &Inst); bool validateMIMGGatherDMask(const MCInst &Inst);
bool validateMIMGDataSize(const MCInst &Inst); bool validateMIMGDataSize(const MCInst &Inst);
bool validateMIMGAddrSize(const MCInst &Inst); bool validateMIMGAddrSize(const MCInst &Inst);
bool validateMIMGD16(const MCInst &Inst); bool validateMIMGD16(const MCInst &Inst);
bool validateMIMGDim(const MCInst &Inst); bool validateMIMGDim(const MCInst &Inst);
bool validateLdsDirect(const MCInst &Inst); bool validateLdsDirect(const MCInst &Inst);
bool validateOpSel(const MCInst &Inst); bool validateOpSel(const MCInst &Inst);
bool validateVccOperand(unsigned Reg) const;
bool validateVOP3Literal(const MCInst &Inst) const; bool validateVOP3Literal(const MCInst &Inst) const;
bool usesConstantBus(const MCInst &Inst, unsigned OpIdx); bool usesConstantBus(const MCInst &Inst, unsigned OpIdx);
bool isInlineConstant(const MCInst &Inst, unsigned OpIdx) const; bool isInlineConstant(const MCInst &Inst, unsigned OpIdx) const;
unsigned findImplicitSGPRReadInVOP(const MCInst &Inst) const; unsigned findImplicitSGPRReadInVOP(const MCInst &Inst) const;
bool isId(const StringRef Id) const; bool isId(const StringRef Id) const;
bool isId(const AsmToken &Token, const StringRef Id) const; bool isId(const AsmToken &Token, const StringRef Id) const;
bool isToken(const AsmToken::TokenKind Kind) const; bool isToken(const AsmToken::TokenKind Kind) const;
Show All 15 Linespublic:
OperandMatchResultTy parseOptionalOperand(OperandVector &Operands); OperandMatchResultTy parseOptionalOperand(OperandVector &Operands);
OperandMatchResultTy parseOptionalOpr(OperandVector &Operands); OperandMatchResultTy parseOptionalOpr(OperandVector &Operands);
OperandMatchResultTy parseExpTgt(OperandVector &Operands); OperandMatchResultTy parseExpTgt(OperandVector &Operands);
OperandMatchResultTy parseSendMsgOp(OperandVector &Operands); OperandMatchResultTy parseSendMsgOp(OperandVector &Operands);
OperandMatchResultTy parseInterpSlot(OperandVector &Operands); OperandMatchResultTy parseInterpSlot(OperandVector &Operands);
OperandMatchResultTy parseInterpAttr(OperandVector &Operands); OperandMatchResultTy parseInterpAttr(OperandVector &Operands);
OperandMatchResultTy parseSOppBrTarget(OperandVector &Operands); OperandMatchResultTy parseSOppBrTarget(OperandVector &Operands);
OperandMatchResultTy parseBoolReg(OperandVector &Operands);
bool parseSwizzleOperands(const unsigned OpNum, int64_t* Op, bool parseSwizzleOperands(const unsigned OpNum, int64_t* Op,
const unsigned MinVal, const unsigned MinVal,
const unsigned MaxVal, const unsigned MaxVal,
const StringRef ErrMsg); const StringRef ErrMsg);
OperandMatchResultTy parseSwizzleOp(OperandVector &Operands); OperandMatchResultTy parseSwizzleOp(OperandVector &Operands);
bool parseSwizzleOffset(int64_t &Imm); bool parseSwizzleOffset(int64_t &Imm);
bool parseSwizzleMacro(int64_t &Imm); bool parseSwizzleMacro(int64_t &Imm);
▲ Show 20 Lines • Show All 273 Lines • ▼ Show 20 Lines
bool AMDGPUOperand::isSDWAInt16Operand() const { bool AMDGPUOperand::isSDWAInt16Operand() const {
return isSDWAOperand(MVT::i16); return isSDWAOperand(MVT::i16);
} }
bool AMDGPUOperand::isSDWAInt32Operand() const { bool AMDGPUOperand::isSDWAInt32Operand() const {
return isSDWAOperand(MVT::i32); return isSDWAOperand(MVT::i32);
} }
bool AMDGPUOperand::isBoolReg() const {
return AsmParser->getFeatureBits()[AMDGPU::FeatureWavefrontSize64] ?
isSCSrcB64() : isSCSrcB32();
}
uint64_t AMDGPUOperand::applyInputFPModifiers(uint64_t Val, unsigned Size) const uint64_t AMDGPUOperand::applyInputFPModifiers(uint64_t Val, unsigned Size) const
{ {
assert(isImmTy(ImmTyNone) && Imm.Mods.hasFPModifiers()); assert(isImmTy(ImmTyNone) && Imm.Mods.hasFPModifiers());
assert(Size == 2 || Size == 4 || Size == 8); assert(Size == 2 || Size == 4 || Size == 8);
const uint64_t FpSignMask = (1ULL << (Size * 8 - 1)); const uint64_t FpSignMask = (1ULL << (Size * 8 - 1));
if (Imm.Mods.Abs) { if (Imm.Mods.Abs) {
▲ Show 20 Lines • Show All 1,535 Lines • ▼ Show 20 Lines if (Opc == AMDGPU::V_PERMLANE16_B32_gfx10 ||
unsigned OpSel = Inst.getOperand(OpSelIdx).getImm(); unsigned OpSel = Inst.getOperand(OpSelIdx).getImm();
if (OpSel & ~3) if (OpSel & ~3)
return false; return false;
} }
return true; return true;
} }
// Check if VCC register matches wavefront size
bool AMDGPUAsmParser::validateVccOperand(unsigned Reg) const {
auto FB = getFeatureBits();
return (FB[AMDGPU::FeatureWavefrontSize64] && Reg == AMDGPU::VCC) ||
(FB[AMDGPU::FeatureWavefrontSize32] && Reg == AMDGPU::VCC_LO);
}
// VOP3 literal is only allowed in GFX10+ and only one can be used // VOP3 literal is only allowed in GFX10+ and only one can be used
bool AMDGPUAsmParser::validateVOP3Literal(const MCInst &Inst) const { bool AMDGPUAsmParser::validateVOP3Literal(const MCInst &Inst) const {
unsigned Opcode = Inst.getOpcode(); unsigned Opcode = Inst.getOpcode();
const MCInstrDesc &Desc = MII.get(Opcode); const MCInstrDesc &Desc = MII.get(Opcode);
if (!(Desc.TSFlags & (SIInstrFlags::VOP3 | SIInstrFlags::VOP3P))) if (!(Desc.TSFlags & (SIInstrFlags::VOP3 | SIInstrFlags::VOP3P)))
return true; return true;
const int Src0Idx = AMDGPU::getNamedOperandIdx(Opcode, AMDGPU::OpName::src0); const int Src0Idx = AMDGPU::getNamedOperandIdx(Opcode, AMDGPU::OpName::src0);
▲ Show 20 Lines • Show All 221 Lines • ▼ Show 20 Lines
} }
bool AMDGPUAsmParser::OutOfRangeError(SMRange Range) { bool AMDGPUAsmParser::OutOfRangeError(SMRange Range) {
return getParser().Error(Range.Start, "value out of range", Range); return getParser().Error(Range.Start, "value out of range", Range);
} }
bool AMDGPUAsmParser::calculateGPRBlocks( bool AMDGPUAsmParser::calculateGPRBlocks(
const FeatureBitset &Features, bool VCCUsed, bool FlatScrUsed, const FeatureBitset &Features, bool VCCUsed, bool FlatScrUsed,
bool XNACKUsed, unsigned NextFreeVGPR, SMRange VGPRRange, bool XNACKUsed, Optional<bool> EnableWavefrontSize32, unsigned NextFreeVGPR,
unsigned NextFreeSGPR, SMRange SGPRRange, unsigned &VGPRBlocks, SMRange VGPRRange, unsigned NextFreeSGPR, SMRange SGPRRange,
unsigned &SGPRBlocks) { unsigned &VGPRBlocks, unsigned &SGPRBlocks) {
// TODO(scott.linder): These calculations are duplicated from // TODO(scott.linder): These calculations are duplicated from
// AMDGPUAsmPrinter::getSIProgramInfo and could be unified. // AMDGPUAsmPrinter::getSIProgramInfo and could be unified.
IsaVersion Version = getIsaVersion(getSTI().getCPU()); IsaVersion Version = getIsaVersion(getSTI().getCPU());
unsigned NumVGPRs = NextFreeVGPR; unsigned NumVGPRs = NextFreeVGPR;
unsigned NumSGPRs = NextFreeSGPR; unsigned NumSGPRs = NextFreeSGPR;
if (Version.Major >= 10) if (Version.Major >= 10)
Show All 12 Lines else {
if ((Version.Major <= 7 || Features.test(FeatureSGPRInitBug)) && if ((Version.Major <= 7 || Features.test(FeatureSGPRInitBug)) &&
NumSGPRs > MaxAddressableNumSGPRs) NumSGPRs > MaxAddressableNumSGPRs)
return OutOfRangeError(SGPRRange); return OutOfRangeError(SGPRRange);
if (Features.test(FeatureSGPRInitBug)) if (Features.test(FeatureSGPRInitBug))
NumSGPRs = IsaInfo::FIXED_NUM_SGPRS_FOR_INIT_BUG; NumSGPRs = IsaInfo::FIXED_NUM_SGPRS_FOR_INIT_BUG;
} }
VGPRBlocks = IsaInfo::getNumVGPRBlocks(&getSTI(), NumVGPRs); VGPRBlocks =
IsaInfo::getNumVGPRBlocks(&getSTI(), NumVGPRs, EnableWavefrontSize32);
SGPRBlocks = IsaInfo::getNumSGPRBlocks(&getSTI(), NumSGPRs); SGPRBlocks = IsaInfo::getNumSGPRBlocks(&getSTI(), NumSGPRs);
return false; return false;
} }
bool AMDGPUAsmParser::ParseDirectiveAMDHSAKernel() { bool AMDGPUAsmParser::ParseDirectiveAMDHSAKernel() {
if (getSTI().getTargetTriple().getArch() != Triple::amdgcn) if (getSTI().getTargetTriple().getArch() != Triple::amdgcn)
return TokError("directive only supported for amdgcn architecture"); return TokError("directive only supported for amdgcn architecture");
Show All 14 Linesbool AMDGPUAsmParser::ParseDirectiveAMDHSAKernel() {
SMRange VGPRRange; SMRange VGPRRange;
uint64_t NextFreeVGPR = 0; uint64_t NextFreeVGPR = 0;
SMRange SGPRRange; SMRange SGPRRange;
uint64_t NextFreeSGPR = 0; uint64_t NextFreeSGPR = 0;
unsigned UserSGPRCount = 0; unsigned UserSGPRCount = 0;
bool ReserveVCC = true; bool ReserveVCC = true;
bool ReserveFlatScr = true; bool ReserveFlatScr = true;
bool ReserveXNACK = hasXNACK(); bool ReserveXNACK = hasXNACK();
Optional<bool> EnableWavefrontSize32;
while (true) { while (true) {
while (getLexer().is(AsmToken::EndOfStatement)) while (getLexer().is(AsmToken::EndOfStatement))
Lex(); Lex();
if (getLexer().isNot(AsmToken::Identifier)) if (getLexer().isNot(AsmToken::Identifier))
return TokError("expected .amdhsa_ directive or .end_amdhsa_kernel"); return TokError("expected .amdhsa_ directive or .end_amdhsa_kernel");
▲ Show 20 Lines • Show All 202 Lines • ▼ Show 20 Lines if (Seen.find(".amdhsa_next_free_vgpr") == Seen.end())
return TokError(".amdhsa_next_free_vgpr directive is required"); return TokError(".amdhsa_next_free_vgpr directive is required");
if (Seen.find(".amdhsa_next_free_sgpr") == Seen.end()) if (Seen.find(".amdhsa_next_free_sgpr") == Seen.end())
return TokError(".amdhsa_next_free_sgpr directive is required"); return TokError(".amdhsa_next_free_sgpr directive is required");
unsigned VGPRBlocks; unsigned VGPRBlocks;
unsigned SGPRBlocks; unsigned SGPRBlocks;
if (calculateGPRBlocks(getFeatureBits(), ReserveVCC, ReserveFlatScr, if (calculateGPRBlocks(getFeatureBits(), ReserveVCC, ReserveFlatScr,
ReserveXNACK, NextFreeVGPR, VGPRRange, NextFreeSGPR, ReserveXNACK, EnableWavefrontSize32, NextFreeVGPR,
SGPRRange, VGPRBlocks, SGPRBlocks)) VGPRRange, NextFreeSGPR, SGPRRange, VGPRBlocks,
SGPRBlocks))
return true; return true;
if (!isUInt<COMPUTE_PGM_RSRC1_GRANULATED_WORKITEM_VGPR_COUNT_WIDTH>( if (!isUInt<COMPUTE_PGM_RSRC1_GRANULATED_WORKITEM_VGPR_COUNT_WIDTH>(
VGPRBlocks)) VGPRBlocks))
return OutOfRangeError(VGPRRange); return OutOfRangeError(VGPRRange);
AMDHSA_BITS_SET(KD.compute_pgm_rsrc1, AMDHSA_BITS_SET(KD.compute_pgm_rsrc1,
COMPUTE_PGM_RSRC1_GRANULATED_WORKITEM_VGPR_COUNT, VGPRBlocks); COMPUTE_PGM_RSRC1_GRANULATED_WORKITEM_VGPR_COUNT, VGPRBlocks);
▲ Show 20 Lines • Show All 1,848 Lines • ▼ Show 20 Lines case AsmToken::Identifier:
MCSymbolRefExpr::create(getContext().getOrCreateSymbol( MCSymbolRefExpr::create(getContext().getOrCreateSymbol(
Parser.getTok().getString()), getContext()), S)); Parser.getTok().getString()), getContext()), S));
Parser.Lex(); Parser.Lex();
return MatchOperand_Success; return MatchOperand_Success;
} }
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Boolean holding registers
//===----------------------------------------------------------------------===//
OperandMatchResultTy
AMDGPUAsmParser::parseBoolReg(OperandVector &Operands) {
return parseReg(Operands);
}
//===----------------------------------------------------------------------===//
// mubuf // mubuf
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
AMDGPUOperand::Ptr AMDGPUAsmParser::defaultDLC() const { AMDGPUOperand::Ptr AMDGPUAsmParser::defaultDLC() const {
return AMDGPUOperand::CreateImm(this, 0, SMLoc(), AMDGPUOperand::ImmTyDLC); return AMDGPUOperand::CreateImm(this, 0, SMLoc(), AMDGPUOperand::ImmTyDLC);
} }
AMDGPUOperand::Ptr AMDGPUAsmParser::defaultGLC() const { AMDGPUOperand::Ptr AMDGPUAsmParser::defaultGLC() const {
▲ Show 20 Lines • Show All 894 Lines • ▼ Show 20 Lines auto TiedTo = Desc.getOperandConstraint(Inst.getNumOperands(),
MCOI::TIED_TO); MCOI::TIED_TO);
if (TiedTo != -1) { if (TiedTo != -1) {
assert((unsigned)TiedTo < Inst.getNumOperands()); assert((unsigned)TiedTo < Inst.getNumOperands());
// handle tied old or src2 for MAC instructions // handle tied old or src2 for MAC instructions
Inst.addOperand(Inst.getOperand(TiedTo)); Inst.addOperand(Inst.getOperand(TiedTo));
} }
AMDGPUOperand &Op = ((AMDGPUOperand &)*Operands[I]); AMDGPUOperand &Op = ((AMDGPUOperand &)*Operands[I]);
// Add the register arguments // Add the register arguments
if (Op.isReg() && Op.getReg() == AMDGPU::VCC) { if (Op.isReg() && validateVccOperand(Op.getReg())) {
// VOP2b (v_add_u32, v_sub_u32 ...) dpp use "vcc" token. // VOP2b (v_add_u32, v_sub_u32 ...) dpp use "vcc" token.
// Skip it. // Skip it.
continue; continue;
} }
if (IsDPP8) { if (IsDPP8) {
if (Op.isDPP8()) { if (Op.isDPP8()) {
Op.addImmOperands(Inst, 1); Op.addImmOperands(Inst, 1);
▲ Show 20 Lines • Show All 126 Lines • ▼ Show 20 Linesvoid AMDGPUAsmParser::cvtSDWA(MCInst &Inst, const OperandVector &Operands,
unsigned I = 1; unsigned I = 1;
const MCInstrDesc &Desc = MII.get(Inst.getOpcode()); const MCInstrDesc &Desc = MII.get(Inst.getOpcode());
for (unsigned J = 0; J < Desc.getNumDefs(); ++J) { for (unsigned J = 0; J < Desc.getNumDefs(); ++J) {
((AMDGPUOperand &)*Operands[I++]).addRegOperands(Inst, 1); ((AMDGPUOperand &)*Operands[I++]).addRegOperands(Inst, 1);
} }
for (unsigned E = Operands.size(); I != E; ++I) { for (unsigned E = Operands.size(); I != E; ++I) {
AMDGPUOperand &Op = ((AMDGPUOperand &)*Operands[I]); AMDGPUOperand &Op = ((AMDGPUOperand &)*Operands[I]);
if (skipVcc && !skippedVcc && Op.isReg() && Op.getReg() == AMDGPU::VCC) { if (skipVcc && !skippedVcc && Op.isReg() &&
(Op.getReg() == AMDGPU::VCC || Op.getReg() == AMDGPU::VCC_LO)) {
// VOP2b (v_add_u32, v_sub_u32 ...) sdwa use "vcc" token as dst. // VOP2b (v_add_u32, v_sub_u32 ...) sdwa use "vcc" token as dst.
// Skip it if it's 2nd (e.g. v_add_i32_sdwa v1, vcc, v2, v3) // Skip it if it's 2nd (e.g. v_add_i32_sdwa v1, vcc, v2, v3)
// or 4th (v_addc_u32_sdwa v1, vcc, v2, v3, vcc) operand. // or 4th (v_addc_u32_sdwa v1, vcc, v2, v3, vcc) operand.
// Skip VCC only if we didn't skip it on previous iteration. // Skip VCC only if we didn't skip it on previous iteration.
if (BasicInstType == SIInstrFlags::VOP2 && if (BasicInstType == SIInstrFlags::VOP2 &&
(Inst.getNumOperands() == 1 || Inst.getNumOperands() == 5)) { (Inst.getNumOperands() == 1 || Inst.getNumOperands() == 5)) {
skippedVcc = true; skippedVcc = true;
continue; continue;
▲ Show 20 Lines • Show All 148 Lines • Show Last 20 Lines

lib/Target/AMDGPU/Disassembler/AMDGPUDisassembler.h

Show First 20 Lines • Show All 117 Lines • ▼ Show 20 Linespublic:
MCOperand decodeSpecialReg32(unsigned Val) const; MCOperand decodeSpecialReg32(unsigned Val) const;
MCOperand decodeSpecialReg64(unsigned Val) const; MCOperand decodeSpecialReg64(unsigned Val) const;
MCOperand decodeSDWASrc(const OpWidthTy Width, unsigned Val) const; MCOperand decodeSDWASrc(const OpWidthTy Width, unsigned Val) const;
MCOperand decodeSDWASrc16(unsigned Val) const; MCOperand decodeSDWASrc16(unsigned Val) const;
MCOperand decodeSDWASrc32(unsigned Val) const; MCOperand decodeSDWASrc32(unsigned Val) const;
MCOperand decodeSDWAVopcDst(unsigned Val) const; MCOperand decodeSDWAVopcDst(unsigned Val) const;
MCOperand decodeBoolReg(unsigned Val) const;
int getTTmpIdx(unsigned Val) const; int getTTmpIdx(unsigned Val) const;
bool isVI() const; bool isVI() const;
bool isGFX9() const; bool isGFX9() const;
bool isGFX10() const; bool isGFX10() const;
}; };
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
Show All 25 Lines

lib/Target/AMDGPU/Disassembler/AMDGPUDisassembler.cpp

Show First 20 Lines • Show All 95 Lines • ▼ Show 20 Linesstatic DecodeStatus decodeSoppBrTarget(MCInst &Inst, unsigned Imm,
APInt SignedOffset(18, Imm * 4, true); APInt SignedOffset(18, Imm * 4, true);
int64_t Offset = (SignedOffset.sext(64) + 4 + Addr).getSExtValue(); int64_t Offset = (SignedOffset.sext(64) + 4 + Addr).getSExtValue();
if (DAsm->tryAddingSymbolicOperand(Inst, Offset, Addr, true, 2, 2)) if (DAsm->tryAddingSymbolicOperand(Inst, Offset, Addr, true, 2, 2))
return MCDisassembler::Success; return MCDisassembler::Success;
return addOperand(Inst, MCOperand::createImm(Imm)); return addOperand(Inst, MCOperand::createImm(Imm));
} }
static DecodeStatus decodeBoolReg(MCInst &Inst, unsigned Val,
uint64_t Addr, const void *Decoder) {
auto DAsm = static_cast<const AMDGPUDisassembler*>(Decoder);
return addOperand(Inst, DAsm->decodeBoolReg(Val));
}
#define DECODE_OPERAND(StaticDecoderName, DecoderName) \ #define DECODE_OPERAND(StaticDecoderName, DecoderName) \
static DecodeStatus StaticDecoderName(MCInst &Inst, \ static DecodeStatus StaticDecoderName(MCInst &Inst, \
unsigned Imm, \ unsigned Imm, \
uint64_t /*Addr*/, \ uint64_t /*Addr*/, \
const void *Decoder) { \ const void *Decoder) { \
auto DAsm = static_cast<const AMDGPUDisassembler*>(Decoder); \ auto DAsm = static_cast<const AMDGPUDisassembler*>(Decoder); \
return addOperand(Inst, DAsm->DecoderName(Imm)); \ return addOperand(Inst, DAsm->DecoderName(Imm)); \
} }
▲ Show 20 Lines • Show All 922 Lines • ▼ Show 20 Lines
MCOperand AMDGPUDisassembler::decodeSDWAVopcDst(unsigned Val) const { MCOperand AMDGPUDisassembler::decodeSDWAVopcDst(unsigned Val) const {
using namespace AMDGPU::SDWA; using namespace AMDGPU::SDWA;
assert((STI.getFeatureBits()[AMDGPU::FeatureGFX9] || assert((STI.getFeatureBits()[AMDGPU::FeatureGFX9] ||
STI.getFeatureBits()[AMDGPU::FeatureGFX10]) && STI.getFeatureBits()[AMDGPU::FeatureGFX10]) &&
"SDWAVopcDst should be present only on GFX9+"); "SDWAVopcDst should be present only on GFX9+");
bool IsWave64 = STI.getFeatureBits()[AMDGPU::FeatureWavefrontSize64];
if (Val & SDWA9EncValues::VOPC_DST_VCC_MASK) { if (Val & SDWA9EncValues::VOPC_DST_VCC_MASK) {
Val &= SDWA9EncValues::VOPC_DST_SGPR_MASK; Val &= SDWA9EncValues::VOPC_DST_SGPR_MASK;
int TTmpIdx = getTTmpIdx(Val); int TTmpIdx = getTTmpIdx(Val);
if (TTmpIdx >= 0) { if (TTmpIdx >= 0) {
return createSRegOperand(getTtmpClassId(OPW64), TTmpIdx); return createSRegOperand(getTtmpClassId(OPW64), TTmpIdx);
} else if (Val > SGPR_MAX) { } else if (Val > SGPR_MAX) {
return decodeSpecialReg64(Val); return IsWave64 ? decodeSpecialReg64(Val)
: decodeSpecialReg32(Val);
} else { } else {
return createSRegOperand(getSgprClassId(OPW64), Val); return createSRegOperand(getSgprClassId(IsWave64 ? OPW64 : OPW32), Val);
} }
} else { } else {
return createRegOperand(AMDGPU::VCC); return createRegOperand(IsWave64 ? AMDGPU::VCC : AMDGPU::VCC_LO);
} }
} }
MCOperand AMDGPUDisassembler::decodeBoolReg(unsigned Val) const {
return STI.getFeatureBits()[AMDGPU::FeatureWavefrontSize64] ?
decodeOperand_SReg_64(Val) : decodeOperand_SReg_32(Val);
}
bool AMDGPUDisassembler::isVI() const { bool AMDGPUDisassembler::isVI() const {
return STI.getFeatureBits()[AMDGPU::FeatureVolcanicIslands]; return STI.getFeatureBits()[AMDGPU::FeatureVolcanicIslands];
} }
bool AMDGPUDisassembler::isGFX9() const { bool AMDGPUDisassembler::isGFX9() const {
return STI.getFeatureBits()[AMDGPU::FeatureGFX9]; return STI.getFeatureBits()[AMDGPU::FeatureGFX9];
} }
▲ Show 20 Lines • Show All 69 Lines • Show Last 20 Lines

lib/Target/AMDGPU/MCTargetDesc/AMDGPUInstPrinter.cpp

Show First 20 Lines • Show All 436 Lines • ▼ Show 20 Lines if (OpNo == 0) {
else if (MII.get(MI->getOpcode()).TSFlags & SIInstrFlags::SDWA) else if (MII.get(MI->getOpcode()).TSFlags & SIInstrFlags::SDWA)
O << "_sdwa "; O << "_sdwa ";
else else
O << "_e32 "; O << "_e32 ";
} }
printOperand(MI, OpNo, STI, O); printOperand(MI, OpNo, STI, O);
// Print default vcc/vcc_lo operand.
switch (MI->getOpcode()) { switch (MI->getOpcode()) {
default: break; default: break;
case AMDGPU::V_ADD_CO_CI_U32_e32_gfx10: case AMDGPU::V_ADD_CO_CI_U32_e32_gfx10:
case AMDGPU::V_SUB_CO_CI_U32_e32_gfx10: case AMDGPU::V_SUB_CO_CI_U32_e32_gfx10:
case AMDGPU::V_SUBREV_CO_CI_U32_e32_gfx10: case AMDGPU::V_SUBREV_CO_CI_U32_e32_gfx10:
case AMDGPU::V_ADD_CO_CI_U32_sdwa_gfx10: case AMDGPU::V_ADD_CO_CI_U32_sdwa_gfx10:
case AMDGPU::V_SUB_CO_CI_U32_sdwa_gfx10: case AMDGPU::V_SUB_CO_CI_U32_sdwa_gfx10:
▲ Show 20 Lines • Show All 131 Lines • ▼ Show 20 Linesvoid AMDGPUInstPrinter::printImmediate64(uint64_t Imm,
} }
} }
void AMDGPUInstPrinter::printDefaultVccOperand(unsigned OpNo, void AMDGPUInstPrinter::printDefaultVccOperand(unsigned OpNo,
const MCSubtargetInfo &STI, const MCSubtargetInfo &STI,
raw_ostream &O) { raw_ostream &O) {
if (OpNo > 0) if (OpNo > 0)
O << ", "; O << ", ";
printRegOperand(AMDGPU::VCC, O, MRI); printRegOperand(STI.getFeatureBits()[AMDGPU::FeatureWavefrontSize64] ?
AMDGPU::VCC : AMDGPU::VCC_LO, O, MRI);
if (OpNo == 0) if (OpNo == 0)
O << ", "; O << ", ";
} }
void AMDGPUInstPrinter::printOperand(const MCInst *MI, unsigned OpNo, void AMDGPUInstPrinter::printOperand(const MCInst *MI, unsigned OpNo,
const MCSubtargetInfo &STI, const MCSubtargetInfo &STI,
raw_ostream &O) { raw_ostream &O) {
// Print default vcc/vcc_lo operand of VOPC.
const MCInstrDesc &Desc = MII.get(MI->getOpcode()); const MCInstrDesc &Desc = MII.get(MI->getOpcode());
if (OpNo == 0 && (Desc.TSFlags & SIInstrFlags::VOPC) && if (OpNo == 0 && (Desc.TSFlags & SIInstrFlags::VOPC) &&
(Desc.hasImplicitDefOfPhysReg(AMDGPU::VCC) || (Desc.hasImplicitDefOfPhysReg(AMDGPU::VCC) ||
Desc.hasImplicitDefOfPhysReg(AMDGPU::VCC_LO))) Desc.hasImplicitDefOfPhysReg(AMDGPU::VCC_LO)))
printDefaultVccOperand(OpNo, STI, O); printDefaultVccOperand(OpNo, STI, O);
if (OpNo >= MI->getNumOperands()) { if (OpNo >= MI->getNumOperands()) {
O << "/*Missing OP" << OpNo << "*/"; O << "/*Missing OP" << OpNo << "*/";
▲ Show 20 Lines • Show All 67 Lines • ▼ Show 20 Lines if (Op.isReg()) {
} }
} else if (Op.isExpr()) { } else if (Op.isExpr()) {
const MCExpr *Exp = Op.getExpr(); const MCExpr *Exp = Op.getExpr();
Exp->print(O, &MAI); Exp->print(O, &MAI);
} else { } else {
O << "/*INV_OP*/"; O << "/*INV_OP*/";
} }
// Print default vcc/vcc_lo operand of v_cndmask_b32_e32.
switch (MI->getOpcode()) { switch (MI->getOpcode()) {
default: break; default: break;
case AMDGPU::V_CNDMASK_B32_e32_gfx10: case AMDGPU::V_CNDMASK_B32_e32_gfx10:
case AMDGPU::V_ADD_CO_CI_U32_e32_gfx10: case AMDGPU::V_ADD_CO_CI_U32_e32_gfx10:
case AMDGPU::V_SUB_CO_CI_U32_e32_gfx10: case AMDGPU::V_SUB_CO_CI_U32_e32_gfx10:
case AMDGPU::V_SUBREV_CO_CI_U32_e32_gfx10: case AMDGPU::V_SUBREV_CO_CI_U32_e32_gfx10:
case AMDGPU::V_ADD_CO_CI_U32_dpp_gfx10: case AMDGPU::V_ADD_CO_CI_U32_dpp_gfx10:
▲ Show 20 Lines • Show All 53 Lines • ▼ Show 20 Linesvoid AMDGPUInstPrinter::printOperandAndIntInputMods(const MCInst *MI,
raw_ostream &O) { raw_ostream &O) {
unsigned InputModifiers = MI->getOperand(OpNo).getImm(); unsigned InputModifiers = MI->getOperand(OpNo).getImm();
if (InputModifiers & SISrcMods::SEXT) if (InputModifiers & SISrcMods::SEXT)
O << "sext("; O << "sext(";
printOperand(MI, OpNo + 1, STI, O); printOperand(MI, OpNo + 1, STI, O);
if (InputModifiers & SISrcMods::SEXT) if (InputModifiers & SISrcMods::SEXT)
O << ')'; O << ')';
// Print default vcc/vcc_lo operand of VOP2b.
switch (MI->getOpcode()) { switch (MI->getOpcode()) {
default: break; default: break;
case AMDGPU::V_ADD_CO_CI_U32_sdwa_gfx10: case AMDGPU::V_ADD_CO_CI_U32_sdwa_gfx10:
case AMDGPU::V_SUB_CO_CI_U32_sdwa_gfx10: case AMDGPU::V_SUB_CO_CI_U32_sdwa_gfx10:
case AMDGPU::V_SUBREV_CO_CI_U32_sdwa_gfx10: case AMDGPU::V_SUBREV_CO_CI_U32_sdwa_gfx10:
if ((int)OpNo + 1 == AMDGPU::getNamedOperandIdx(MI->getOpcode(), if ((int)OpNo + 1 == AMDGPU::getNamedOperandIdx(MI->getOpcode(),
AMDGPU::OpName::src1)) AMDGPU::OpName::src1))
▲ Show 20 Lines • Show All 888 Lines • Show Last 20 Lines

lib/Target/AMDGPU/MCTargetDesc/SIMCCodeEmitter.cpp

Show First 20 Lines • Show All 383 Lines • ▼ Show 20 LinesSIMCCodeEmitter::getSDWAVopcDstEncoding(const MCInst &MI, unsigned OpNo,
const MCSubtargetInfo &STI) const { const MCSubtargetInfo &STI) const {
using namespace AMDGPU::SDWA; using namespace AMDGPU::SDWA;
uint64_t RegEnc = 0; uint64_t RegEnc = 0;
const MCOperand &MO = MI.getOperand(OpNo); const MCOperand &MO = MI.getOperand(OpNo);
unsigned Reg = MO.getReg(); unsigned Reg = MO.getReg();
if (Reg != AMDGPU::VCC) { if (Reg != AMDGPU::VCC && Reg != AMDGPU::VCC_LO) {
RegEnc |= MRI.getEncodingValue(Reg); RegEnc |= MRI.getEncodingValue(Reg);
RegEnc &= SDWA9EncValues::VOPC_DST_SGPR_MASK; RegEnc &= SDWA9EncValues::VOPC_DST_SGPR_MASK;
RegEnc |= SDWA9EncValues::VOPC_DST_VCC_MASK; RegEnc |= SDWA9EncValues::VOPC_DST_VCC_MASK;
} }
return RegEnc; return RegEnc;
} }
static bool needsPCRel(const MCExpr *Expr) { static bool needsPCRel(const MCExpr *Expr) {
▲ Show 20 Lines • Show All 73 Lines • Show Last 20 Lines

lib/Target/AMDGPU/SIAnnotateControlFlow.cpp

//===- SIAnnotateControlFlow.cpp ------------------------------------------===// //===- SIAnnotateControlFlow.cpp ------------------------------------------===//
// //
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information. // See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
// //
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// //
/// \file /// \file
/// Annotates the control flow with hardware specific intrinsics. /// Annotates the control flow with hardware specific intrinsics.
// //
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
#include "AMDGPU.h" #include "AMDGPU.h"
#include "AMDGPUSubtarget.h"
#include "llvm/ADT/DepthFirstIterator.h" #include "llvm/ADT/DepthFirstIterator.h"
#include "llvm/ADT/STLExtras.h" #include "llvm/ADT/STLExtras.h"
#include "llvm/ADT/SmallVector.h" #include "llvm/ADT/SmallVector.h"
#include "llvm/Analysis/LegacyDivergenceAnalysis.h" #include "llvm/Analysis/LegacyDivergenceAnalysis.h"
#include "llvm/Analysis/LoopInfo.h" #include "llvm/Analysis/LoopInfo.h"
#include "llvm/CodeGen/TargetPassConfig.h"
#include "llvm/IR/BasicBlock.h" #include "llvm/IR/BasicBlock.h"
#include "llvm/IR/CFG.h" #include "llvm/IR/CFG.h"
#include "llvm/IR/Constant.h" #include "llvm/IR/Constant.h"
#include "llvm/IR/Constants.h" #include "llvm/IR/Constants.h"
#include "llvm/IR/DerivedTypes.h" #include "llvm/IR/DerivedTypes.h"
#include "llvm/IR/Dominators.h" #include "llvm/IR/Dominators.h"
#include "llvm/IR/Function.h" #include "llvm/IR/Function.h"
#include "llvm/IR/Instruction.h" #include "llvm/IR/Instruction.h"
Show All 22 Lines
using StackEntry = std::pair<BasicBlock *, Value *>; using StackEntry = std::pair<BasicBlock *, Value *>;
using StackVector = SmallVector<StackEntry, 16>; using StackVector = SmallVector<StackEntry, 16>;
class SIAnnotateControlFlow : public FunctionPass { class SIAnnotateControlFlow : public FunctionPass {
LegacyDivergenceAnalysis *DA; LegacyDivergenceAnalysis *DA;
Type *Boolean; Type *Boolean;
Type *Void; Type *Void;
Type *Int64; Type *IntMask;
Type *ReturnStruct; Type *ReturnStruct;
ConstantInt *BoolTrue; ConstantInt *BoolTrue;
ConstantInt *BoolFalse; ConstantInt *BoolFalse;
UndefValue *BoolUndef; UndefValue *BoolUndef;
Constant *Int64Zero; Constant *IntMaskZero;
Function *If; Function *If;
Function *Else; Function *Else;
Function *IfBreak; Function *IfBreak;
Function *Loop; Function *Loop;
Function *EndCf; Function *EndCf;
DominatorTree *DT; DominatorTree *DT;
StackVector Stack; StackVector Stack;
LoopInfo *LI; LoopInfo *LI;
void initialize(Module &M, const GCNSubtarget &ST);
bool isUniform(BranchInst *T); bool isUniform(BranchInst *T);
bool isTopOfStack(BasicBlock *BB); bool isTopOfStack(BasicBlock *BB);
Value *popSaved(); Value *popSaved();
void push(BasicBlock *BB, Value *Saved); void push(BasicBlock *BB, Value *Saved);
Show All 13 Linesclass SIAnnotateControlFlow : public FunctionPass {
void closeControlFlow(BasicBlock *BB); void closeControlFlow(BasicBlock *BB);
public: public:
static char ID; static char ID;
SIAnnotateControlFlow() : FunctionPass(ID) {} SIAnnotateControlFlow() : FunctionPass(ID) {}
bool doInitialization(Module &M) override;
bool runOnFunction(Function &F) override; bool runOnFunction(Function &F) override;
StringRef getPassName() const override { return "SI annotate control flow"; } StringRef getPassName() const override { return "SI annotate control flow"; }
void getAnalysisUsage(AnalysisUsage &AU) const override { void getAnalysisUsage(AnalysisUsage &AU) const override {
AU.addRequired<LoopInfoWrapperPass>(); AU.addRequired<LoopInfoWrapperPass>();
AU.addRequired<DominatorTreeWrapperPass>(); AU.addRequired<DominatorTreeWrapperPass>();
AU.addRequired<LegacyDivergenceAnalysis>(); AU.addRequired<LegacyDivergenceAnalysis>();
AU.addPreserved<DominatorTreeWrapperPass>(); AU.addPreserved<DominatorTreeWrapperPass>();
AU.addRequired<TargetPassConfig>();
FunctionPass::getAnalysisUsage(AU); FunctionPass::getAnalysisUsage(AU);
} }
}; };
} // end anonymous namespace } // end anonymous namespace
INITIALIZE_PASS_BEGIN(SIAnnotateControlFlow, DEBUG_TYPE, INITIALIZE_PASS_BEGIN(SIAnnotateControlFlow, DEBUG_TYPE,
"Annotate SI Control Flow", false, false) "Annotate SI Control Flow", false, false)
INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass) INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
INITIALIZE_PASS_DEPENDENCY(LegacyDivergenceAnalysis) INITIALIZE_PASS_DEPENDENCY(LegacyDivergenceAnalysis)
INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
INITIALIZE_PASS_END(SIAnnotateControlFlow, DEBUG_TYPE, INITIALIZE_PASS_END(SIAnnotateControlFlow, DEBUG_TYPE,
"Annotate SI Control Flow", false, false) "Annotate SI Control Flow", false, false)
char SIAnnotateControlFlow::ID = 0; char SIAnnotateControlFlow::ID = 0;
/// Initialize all the types and constants used in the pass /// Initialize all the types and constants used in the pass
bool SIAnnotateControlFlow::doInitialization(Module &M) { void SIAnnotateControlFlow::initialize(Module &M, const GCNSubtarget &ST) {
LLVMContext &Context = M.getContext(); LLVMContext &Context = M.getContext();
Void = Type::getVoidTy(Context); Void = Type::getVoidTy(Context);
Boolean = Type::getInt1Ty(Context); Boolean = Type::getInt1Ty(Context);
Int64 = Type::getInt64Ty(Context); IntMask = ST.isWave32() ? Type::getInt32Ty(Context)
ReturnStruct = StructType::get(Boolean, Int64); : Type::getInt64Ty(Context);
ReturnStruct = StructType::get(Boolean, IntMask);
BoolTrue = ConstantInt::getTrue(Context); BoolTrue = ConstantInt::getTrue(Context);
BoolFalse = ConstantInt::getFalse(Context); BoolFalse = ConstantInt::getFalse(Context);
BoolUndef = UndefValue::get(Boolean); BoolUndef = UndefValue::get(Boolean);
Int64Zero = ConstantInt::get(Int64, 0); IntMaskZero = ConstantInt::get(IntMask, 0);
If = Intrinsic::getDeclaration(&M, Intrinsic::amdgcn_if); If = Intrinsic::getDeclaration(&M, Intrinsic::amdgcn_if, { IntMask });
Else = Intrinsic::getDeclaration(&M, Intrinsic::amdgcn_else); Else = Intrinsic::getDeclaration(&M, Intrinsic::amdgcn_else,
IfBreak = Intrinsic::getDeclaration(&M, Intrinsic::amdgcn_if_break); { IntMask, IntMask });
Loop = Intrinsic::getDeclaration(&M, Intrinsic::amdgcn_loop); IfBreak = Intrinsic::getDeclaration(&M, Intrinsic::amdgcn_if_break,
EndCf = Intrinsic::getDeclaration(&M, Intrinsic::amdgcn_end_cf); { IntMask, IntMask });
return false; Loop = Intrinsic::getDeclaration(&M, Intrinsic::amdgcn_loop, { IntMask });
EndCf = Intrinsic::getDeclaration(&M, Intrinsic::amdgcn_end_cf, { IntMask });
} }
/// Is the branch condition uniform or did the StructurizeCFG pass /// Is the branch condition uniform or did the StructurizeCFG pass
/// consider it as such? /// consider it as such?
bool SIAnnotateControlFlow::isUniform(BranchInst *T) { bool SIAnnotateControlFlow::isUniform(BranchInst *T) {
return DA->isUniform(T) || return DA->isUniform(T) ||
T->getMetadata("structurizecfg.uniform") != nullptr; T->getMetadata("structurizecfg.uniform") != nullptr;
} }
▲ Show 20 Lines • Show All 93 Lines • ▼ Show 20 Lines if (isUniform(Term))
return; return;
BasicBlock *BB = Term->getParent(); BasicBlock *BB = Term->getParent();
llvm::Loop *L = LI->getLoopFor(BB); llvm::Loop *L = LI->getLoopFor(BB);
if (!L) if (!L)
return; return;
BasicBlock *Target = Term->getSuccessor(1); BasicBlock *Target = Term->getSuccessor(1);
PHINode *Broken = PHINode::Create(Int64, 0, "phi.broken", &Target->front()); PHINode *Broken = PHINode::Create(IntMask, 0, "phi.broken", &Target->front());
Value *Cond = Term->getCondition(); Value *Cond = Term->getCondition();
Term->setCondition(BoolTrue); Term->setCondition(BoolTrue);
Value *Arg = handleLoopCondition(Cond, Broken, L, Term); Value *Arg = handleLoopCondition(Cond, Broken, L, Term);
for (BasicBlock *Pred : predecessors(Target)) { for (BasicBlock *Pred : predecessors(Target)) {
Value *PHIValue = Int64Zero; Value *PHIValue = IntMaskZero;
if (Pred == BB) // Remember the value of the previous iteration. if (Pred == BB) // Remember the value of the previous iteration.
PHIValue = Arg; PHIValue = Arg;
// If the backedge from Pred to Target could be executed before the exit // If the backedge from Pred to Target could be executed before the exit
// of the loop at BB, it should not reset or change "Broken", which keeps // of the loop at BB, it should not reset or change "Broken", which keeps
// track of the number of threads exited the loop at BB. // track of the number of threads exited the loop at BB.
else if (L->contains(Pred) && DT->dominates(Pred, BB)) else if (L->contains(Pred) && DT->dominates(Pred, BB))
PHIValue = Broken; PHIValue = Broken;
Broken->addIncoming(PHIValue, Pred); Broken->addIncoming(PHIValue, Pred);
Show All 34 Lines
} }
/// Annotate the control flow with intrinsics so the backend can /// Annotate the control flow with intrinsics so the backend can
/// recognize if/then/else and loops. /// recognize if/then/else and loops.
bool SIAnnotateControlFlow::runOnFunction(Function &F) { bool SIAnnotateControlFlow::runOnFunction(Function &F) {
DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree(); DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
LI = &getAnalysis<LoopInfoWrapperPass>().getLoopInfo(); LI = &getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
DA = &getAnalysis<LegacyDivergenceAnalysis>(); DA = &getAnalysis<LegacyDivergenceAnalysis>();
TargetPassConfig &TPC = getAnalysis<TargetPassConfig>();
const TargetMachine &TM = TPC.getTM<TargetMachine>();
initialize(*F.getParent(), TM.getSubtarget<GCNSubtarget>(F));
for (df_iterator<BasicBlock *> I = df_begin(&F.getEntryBlock()), for (df_iterator<BasicBlock *> I = df_begin(&F.getEntryBlock()),
E = df_end(&F.getEntryBlock()); I != E; ++I) { E = df_end(&F.getEntryBlock()); I != E; ++I) {
BasicBlock *BB = *I; BasicBlock *BB = *I;
BranchInst *Term = dyn_cast<BranchInst>(BB->getTerminator()); BranchInst *Term = dyn_cast<BranchInst>(BB->getTerminator());
if (!Term || Term->isUnconditional()) { if (!Term || Term->isUnconditional()) {
if (isTopOfStack(BB)) if (isTopOfStack(BB))
Show All 39 Lines

lib/Target/AMDGPU/SIFrameLowering.cpp

Show First 20 Lines • Show All 621 Lines • ▼ Show 20 Lines for (const SIMachineFunctionInfo::SGPRSpillVGPRCSR &Reg
if (ScratchExecCopy == AMDGPU::NoRegister) { if (ScratchExecCopy == AMDGPU::NoRegister) {
if (LiveRegs.empty()) { if (LiveRegs.empty()) {
LiveRegs.init(TRI); LiveRegs.init(TRI);
LiveRegs.addLiveIns(MBB); LiveRegs.addLiveIns(MBB);
} }
ScratchExecCopy ScratchExecCopy
= findScratchNonCalleeSaveRegister(MF, LiveRegs, = findScratchNonCalleeSaveRegister(MF, LiveRegs,
AMDGPU::SReg_64_XEXECRegClass); *TRI.getWaveMaskRegClass());
BuildMI(MBB, MBBI, DL, TII->get(AMDGPU::S_OR_SAVEEXEC_B64), const unsigned OrSaveExec = ST.isWave32() ?
AMDGPU::S_OR_SAVEEXEC_B32 : AMDGPU::S_OR_SAVEEXEC_B64;
BuildMI(MBB, MBBI, DL, TII->get(OrSaveExec),
ScratchExecCopy) ScratchExecCopy)
.addImm(-1); .addImm(-1);
} }
TII->storeRegToStackSlot(MBB, MBBI, Reg.VGPR, true, TII->storeRegToStackSlot(MBB, MBBI, Reg.VGPR, true,
Reg.FI.getValue(), &AMDGPU::VGPR_32RegClass, Reg.FI.getValue(), &AMDGPU::VGPR_32RegClass,
&TII->getRegisterInfo()); &TII->getRegisterInfo());
} }
if (ScratchExecCopy != AMDGPU::NoRegister) { if (ScratchExecCopy != AMDGPU::NoRegister) {
// FIXME: Split block and make terminator. // FIXME: Split block and make terminator.
BuildMI(MBB, MBBI, DL, TII->get(AMDGPU::S_MOV_B64), AMDGPU::EXEC) unsigned ExecMov = ST.isWave32() ? AMDGPU::S_MOV_B32 : AMDGPU::S_MOV_B64;
unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
BuildMI(MBB, MBBI, DL, TII->get(ExecMov), Exec)
.addReg(ScratchExecCopy); .addReg(ScratchExecCopy);
} }
} }
void SIFrameLowering::emitEpilogue(MachineFunction &MF, void SIFrameLowering::emitEpilogue(MachineFunction &MF,
MachineBasicBlock &MBB) const { MachineBasicBlock &MBB) const {
const SIMachineFunctionInfo *FuncInfo = MF.getInfo<SIMachineFunctionInfo>(); const SIMachineFunctionInfo *FuncInfo = MF.getInfo<SIMachineFunctionInfo>();
if (FuncInfo->isEntryFunction()) if (FuncInfo->isEntryFunction())
return; return;
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>(); const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
const SIInstrInfo *TII = ST.getInstrInfo(); const SIInstrInfo *TII = ST.getInstrInfo();
MachineBasicBlock::iterator MBBI = MBB.getFirstTerminator(); MachineBasicBlock::iterator MBBI = MBB.getFirstTerminator();
DebugLoc DL; DebugLoc DL;
unsigned ScratchExecCopy = AMDGPU::NoRegister; unsigned ScratchExecCopy = AMDGPU::NoRegister;
for (const SIMachineFunctionInfo::SGPRSpillVGPRCSR &Reg for (const SIMachineFunctionInfo::SGPRSpillVGPRCSR &Reg
: FuncInfo->getSGPRSpillVGPRs()) { : FuncInfo->getSGPRSpillVGPRs()) {
if (!Reg.FI.hasValue()) if (!Reg.FI.hasValue())
continue; continue;
const SIRegisterInfo &TRI = TII->getRegisterInfo();
if (ScratchExecCopy == AMDGPU::NoRegister) { if (ScratchExecCopy == AMDGPU::NoRegister) {
// See emitPrologue // See emitPrologue
LivePhysRegs LiveRegs(*ST.getRegisterInfo()); LivePhysRegs LiveRegs(*ST.getRegisterInfo());
LiveRegs.addLiveIns(MBB); LiveRegs.addLiveIns(MBB);
ScratchExecCopy ScratchExecCopy
= findScratchNonCalleeSaveRegister(MF, LiveRegs, = findScratchNonCalleeSaveRegister(MF, LiveRegs,
AMDGPU::SReg_64_XEXECRegClass); *TRI.getWaveMaskRegClass());
const unsigned OrSaveExec = ST.isWave32() ?
AMDGPU::S_OR_SAVEEXEC_B32 : AMDGPU::S_OR_SAVEEXEC_B64;
BuildMI(MBB, MBBI, DL, TII->get(AMDGPU::S_OR_SAVEEXEC_B64), ScratchExecCopy) BuildMI(MBB, MBBI, DL, TII->get(OrSaveExec), ScratchExecCopy)
.addImm(-1); .addImm(-1);
} }
TII->loadRegFromStackSlot(MBB, MBBI, Reg.VGPR, TII->loadRegFromStackSlot(MBB, MBBI, Reg.VGPR,
Reg.FI.getValue(), &AMDGPU::VGPR_32RegClass, Reg.FI.getValue(), &AMDGPU::VGPR_32RegClass,
&TII->getRegisterInfo()); &TII->getRegisterInfo());
} }
if (ScratchExecCopy != AMDGPU::NoRegister) { if (ScratchExecCopy != AMDGPU::NoRegister) {
// FIXME: Split block and make terminator. // FIXME: Split block and make terminator.
BuildMI(MBB, MBBI, DL, TII->get(AMDGPU::S_MOV_B64), AMDGPU::EXEC) unsigned ExecMov = ST.isWave32() ? AMDGPU::S_MOV_B32 : AMDGPU::S_MOV_B64;
unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
BuildMI(MBB, MBBI, DL, TII->get(ExecMov), Exec)
.addReg(ScratchExecCopy); .addReg(ScratchExecCopy);
} }
if (hasFP(MF)) { if (hasFP(MF)) {
const MachineFrameInfo &MFI = MF.getFrameInfo(); const MachineFrameInfo &MFI = MF.getFrameInfo();
uint32_t NumBytes = MFI.getStackSize(); uint32_t NumBytes = MFI.getStackSize();
uint32_t RoundedSize = FuncInfo->isStackRealigned() ? uint32_t RoundedSize = FuncInfo->isStackRealigned() ?
NumBytes + MFI.getMaxAlignment() : NumBytes; NumBytes + MFI.getMaxAlignment() : NumBytes;
▲ Show 20 Lines • Show All 162 Lines • Show Last 20 Lines

lib/Target/AMDGPU/SIISelLowering.cpp

  • This file is larger than 256 KB, so syntax highlighting is disabled by default.
Show First 20 Lines • Show All 2,919 Lines • ▼ Show 20 Linesstatic MachineBasicBlock::iterator emitLoadM0FromVGPRLoop(
const MachineOperand &IdxReg, const MachineOperand &IdxReg,
unsigned InitReg, unsigned InitReg,
unsigned ResultReg, unsigned ResultReg,
unsigned PhiReg, unsigned PhiReg,
unsigned InitSaveExecReg, unsigned InitSaveExecReg,
int Offset, int Offset,
bool UseGPRIdxMode, bool UseGPRIdxMode,
bool IsIndirectSrc) { bool IsIndirectSrc) {
MachineFunction *MF = OrigBB.getParent();
const GCNSubtarget &ST = MF->getSubtarget<GCNSubtarget>();
const SIRegisterInfo *TRI = ST.getRegisterInfo();
MachineBasicBlock::iterator I = LoopBB.begin(); MachineBasicBlock::iterator I = LoopBB.begin();
unsigned PhiExec = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); const TargetRegisterClass *BoolRC = TRI->getBoolRC();
unsigned NewExec = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned PhiExec = MRI.createVirtualRegister(BoolRC);
unsigned NewExec = MRI.createVirtualRegister(BoolRC);
unsigned CurrentIdxReg = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass); unsigned CurrentIdxReg = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
unsigned CondReg = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned CondReg = MRI.createVirtualRegister(BoolRC);
BuildMI(LoopBB, I, DL, TII->get(TargetOpcode::PHI), PhiReg) BuildMI(LoopBB, I, DL, TII->get(TargetOpcode::PHI), PhiReg)
.addReg(InitReg) .addReg(InitReg)
.addMBB(&OrigBB) .addMBB(&OrigBB)
.addReg(ResultReg) .addReg(ResultReg)
.addMBB(&LoopBB); .addMBB(&LoopBB);
BuildMI(LoopBB, I, DL, TII->get(TargetOpcode::PHI), PhiExec) BuildMI(LoopBB, I, DL, TII->get(TargetOpcode::PHI), PhiExec)
.addReg(InitSaveExecReg) .addReg(InitSaveExecReg)
.addMBB(&OrigBB) .addMBB(&OrigBB)
.addReg(NewExec) .addReg(NewExec)
.addMBB(&LoopBB); .addMBB(&LoopBB);
// Read the next variant <- also loop target. // Read the next variant <- also loop target.
BuildMI(LoopBB, I, DL, TII->get(AMDGPU::V_READFIRSTLANE_B32), CurrentIdxReg) BuildMI(LoopBB, I, DL, TII->get(AMDGPU::V_READFIRSTLANE_B32), CurrentIdxReg)
.addReg(IdxReg.getReg(), getUndefRegState(IdxReg.isUndef())); .addReg(IdxReg.getReg(), getUndefRegState(IdxReg.isUndef()));
// Compare the just read M0 value to all possible Idx values. // Compare the just read M0 value to all possible Idx values.
BuildMI(LoopBB, I, DL, TII->get(AMDGPU::V_CMP_EQ_U32_e64), CondReg) BuildMI(LoopBB, I, DL, TII->get(AMDGPU::V_CMP_EQ_U32_e64), CondReg)
.addReg(CurrentIdxReg) .addReg(CurrentIdxReg)
.addReg(IdxReg.getReg(), 0, IdxReg.getSubReg()); .addReg(IdxReg.getReg(), 0, IdxReg.getSubReg());
// Update EXEC, save the original EXEC value to VCC. // Update EXEC, save the original EXEC value to VCC.
BuildMI(LoopBB, I, DL, TII->get(AMDGPU::S_AND_SAVEEXEC_B64), NewExec) BuildMI(LoopBB, I, DL, TII->get(ST.isWave32() ? AMDGPU::S_AND_SAVEEXEC_B32
: AMDGPU::S_AND_SAVEEXEC_B64),
NewExec)
.addReg(CondReg, RegState::Kill); .addReg(CondReg, RegState::Kill);
MRI.setSimpleHint(NewExec, CondReg); MRI.setSimpleHint(NewExec, CondReg);
if (UseGPRIdxMode) { if (UseGPRIdxMode) {
unsigned IdxReg; unsigned IdxReg;
if (Offset == 0) { if (Offset == 0) {
IdxReg = CurrentIdxReg; IdxReg = CurrentIdxReg;
Show All 18 Lines if (UseGPRIdxMode) {
} else { } else {
BuildMI(LoopBB, I, DL, TII->get(AMDGPU::S_ADD_I32), AMDGPU::M0) BuildMI(LoopBB, I, DL, TII->get(AMDGPU::S_ADD_I32), AMDGPU::M0)
.addReg(CurrentIdxReg, RegState::Kill) .addReg(CurrentIdxReg, RegState::Kill)
.addImm(Offset); .addImm(Offset);
} }
} }
// Update EXEC, switch all done bits to 0 and all todo bits to 1. // Update EXEC, switch all done bits to 0 and all todo bits to 1.
unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
MachineInstr *InsertPt = MachineInstr *InsertPt =
BuildMI(LoopBB, I, DL, TII->get(AMDGPU::S_XOR_B64_term), AMDGPU::EXEC) BuildMI(LoopBB, I, DL, TII->get(ST.isWave32() ? AMDGPU::S_XOR_B32_term
.addReg(AMDGPU::EXEC) : AMDGPU::S_XOR_B64_term), Exec)
.addReg(Exec)
.addReg(NewExec); .addReg(NewExec);
// XXX - s_xor_b64 sets scc to 1 if the result is nonzero, so can we use // XXX - s_xor_b64 sets scc to 1 if the result is nonzero, so can we use
// s_cbranch_scc0? // s_cbranch_scc0?
// Loop back to V_READFIRSTLANE_B32 if there are still variants to cover. // Loop back to V_READFIRSTLANE_B32 if there are still variants to cover.
BuildMI(LoopBB, I, DL, TII->get(AMDGPU::S_CBRANCH_EXECNZ)) BuildMI(LoopBB, I, DL, TII->get(AMDGPU::S_CBRANCH_EXECNZ))
.addMBB(&LoopBB); .addMBB(&LoopBB);
Show All 9 Linesstatic MachineBasicBlock::iterator loadM0FromVGPR(const SIInstrInfo *TII,
MachineBasicBlock &MBB, MachineBasicBlock &MBB,
MachineInstr &MI, MachineInstr &MI,
unsigned InitResultReg, unsigned InitResultReg,
unsigned PhiReg, unsigned PhiReg,
int Offset, int Offset,
bool UseGPRIdxMode, bool UseGPRIdxMode,
bool IsIndirectSrc) { bool IsIndirectSrc) {
MachineFunction *MF = MBB.getParent(); MachineFunction *MF = MBB.getParent();
const GCNSubtarget &ST = MF->getSubtarget<GCNSubtarget>();
const SIRegisterInfo *TRI = ST.getRegisterInfo();
MachineRegisterInfo &MRI = MF->getRegInfo(); MachineRegisterInfo &MRI = MF->getRegInfo();
const DebugLoc &DL = MI.getDebugLoc(); const DebugLoc &DL = MI.getDebugLoc();
MachineBasicBlock::iterator I(&MI); MachineBasicBlock::iterator I(&MI);
const auto *BoolXExecRC = TRI->getRegClass(AMDGPU::SReg_1_XEXECRegClassID);
unsigned DstReg = MI.getOperand(0).getReg(); unsigned DstReg = MI.getOperand(0).getReg();
unsigned SaveExec = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); unsigned SaveExec = MRI.createVirtualRegister(BoolXExecRC);
unsigned TmpExec = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); unsigned TmpExec = MRI.createVirtualRegister(BoolXExecRC);
unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
unsigned MovExecOpc = ST.isWave32() ? AMDGPU::S_MOV_B32 : AMDGPU::S_MOV_B64;
BuildMI(MBB, I, DL, TII->get(TargetOpcode::IMPLICIT_DEF), TmpExec); BuildMI(MBB, I, DL, TII->get(TargetOpcode::IMPLICIT_DEF), TmpExec);
// Save the EXEC mask // Save the EXEC mask
BuildMI(MBB, I, DL, TII->get(AMDGPU::S_MOV_B64), SaveExec) BuildMI(MBB, I, DL, TII->get(MovExecOpc), SaveExec)
.addReg(AMDGPU::EXEC); .addReg(Exec);
// To insert the loop we need to split the block. Move everything after this // To insert the loop we need to split the block. Move everything after this
// point to a new block, and insert a new empty block between the two. // point to a new block, and insert a new empty block between the two.
MachineBasicBlock *LoopBB = MF->CreateMachineBasicBlock(); MachineBasicBlock *LoopBB = MF->CreateMachineBasicBlock();
MachineBasicBlock *RemainderBB = MF->CreateMachineBasicBlock(); MachineBasicBlock *RemainderBB = MF->CreateMachineBasicBlock();
MachineFunction::iterator MBBI(MBB); MachineFunction::iterator MBBI(MBB);
++MBBI; ++MBBI;
Show All 11 Linesstatic MachineBasicBlock::iterator loadM0FromVGPR(const SIInstrInfo *TII,
const MachineOperand *Idx = TII->getNamedOperand(MI, AMDGPU::OpName::idx); const MachineOperand *Idx = TII->getNamedOperand(MI, AMDGPU::OpName::idx);
auto InsPt = emitLoadM0FromVGPRLoop(TII, MRI, MBB, *LoopBB, DL, *Idx, auto InsPt = emitLoadM0FromVGPRLoop(TII, MRI, MBB, *LoopBB, DL, *Idx,
InitResultReg, DstReg, PhiReg, TmpExec, InitResultReg, DstReg, PhiReg, TmpExec,
Offset, UseGPRIdxMode, IsIndirectSrc); Offset, UseGPRIdxMode, IsIndirectSrc);
MachineBasicBlock::iterator First = RemainderBB->begin(); MachineBasicBlock::iterator First = RemainderBB->begin();
BuildMI(*RemainderBB, First, DL, TII->get(AMDGPU::S_MOV_B64), AMDGPU::EXEC) BuildMI(*RemainderBB, First, DL, TII->get(MovExecOpc), Exec)
.addReg(SaveExec); .addReg(SaveExec);
return InsPt; return InsPt;
} }
// Returns subreg index, offset // Returns subreg index, offset
static std::pair<unsigned, int> static std::pair<unsigned, int>
computeIndirectRegAndOffset(const SIRegisterInfo &TRI, computeIndirectRegAndOffset(const SIRegisterInfo &TRI,
▲ Show 20 Lines • Show All 275 Lines • ▼ Show 20 Lines if (TII->isMIMG(MI)) {
return BB; return BB;
} }
switch (MI.getOpcode()) { switch (MI.getOpcode()) {
case AMDGPU::S_ADD_U64_PSEUDO: case AMDGPU::S_ADD_U64_PSEUDO:
case AMDGPU::S_SUB_U64_PSEUDO: { case AMDGPU::S_SUB_U64_PSEUDO: {
MachineRegisterInfo &MRI = BB->getParent()->getRegInfo(); MachineRegisterInfo &MRI = BB->getParent()->getRegInfo();
const GCNSubtarget &ST = MF->getSubtarget<GCNSubtarget>();
const SIRegisterInfo *TRI = ST.getRegisterInfo();
const TargetRegisterClass *BoolRC = TRI->getBoolRC();
const DebugLoc &DL = MI.getDebugLoc(); const DebugLoc &DL = MI.getDebugLoc();
MachineOperand &Dest = MI.getOperand(0); MachineOperand &Dest = MI.getOperand(0);
MachineOperand &Src0 = MI.getOperand(1); MachineOperand &Src0 = MI.getOperand(1);
MachineOperand &Src1 = MI.getOperand(2); MachineOperand &Src1 = MI.getOperand(2);
unsigned DestSub0 = MRI.createVirtualRegister(&AMDGPU::SReg_32_XM0RegClass); unsigned DestSub0 = MRI.createVirtualRegister(&AMDGPU::SReg_32_XM0RegClass);
unsigned DestSub1 = MRI.createVirtualRegister(&AMDGPU::SReg_32_XM0RegClass); unsigned DestSub1 = MRI.createVirtualRegister(&AMDGPU::SReg_32_XM0RegClass);
MachineOperand Src0Sub0 = TII->buildExtractSubRegOrImm(MI, MRI, MachineOperand Src0Sub0 = TII->buildExtractSubRegOrImm(MI, MRI,
Src0, &AMDGPU::SReg_64RegClass, AMDGPU::sub0, Src0, BoolRC, AMDGPU::sub0,
&AMDGPU::SReg_32_XM0RegClass); &AMDGPU::SReg_32_XM0RegClass);
MachineOperand Src0Sub1 = TII->buildExtractSubRegOrImm(MI, MRI, MachineOperand Src0Sub1 = TII->buildExtractSubRegOrImm(MI, MRI,
Src0, &AMDGPU::SReg_64RegClass, AMDGPU::sub1, Src0, BoolRC, AMDGPU::sub1,
&AMDGPU::SReg_32_XM0RegClass); &AMDGPU::SReg_32_XM0RegClass);
MachineOperand Src1Sub0 = TII->buildExtractSubRegOrImm(MI, MRI, MachineOperand Src1Sub0 = TII->buildExtractSubRegOrImm(MI, MRI,
Src1, &AMDGPU::SReg_64RegClass, AMDGPU::sub0, Src1, BoolRC, AMDGPU::sub0,
&AMDGPU::SReg_32_XM0RegClass); &AMDGPU::SReg_32_XM0RegClass);
MachineOperand Src1Sub1 = TII->buildExtractSubRegOrImm(MI, MRI, MachineOperand Src1Sub1 = TII->buildExtractSubRegOrImm(MI, MRI,
Src1, &AMDGPU::SReg_64RegClass, AMDGPU::sub1, Src1, BoolRC, AMDGPU::sub1,
&AMDGPU::SReg_32_XM0RegClass); &AMDGPU::SReg_32_XM0RegClass);
bool IsAdd = (MI.getOpcode() == AMDGPU::S_ADD_U64_PSEUDO); bool IsAdd = (MI.getOpcode() == AMDGPU::S_ADD_U64_PSEUDO);
unsigned LoOpc = IsAdd ? AMDGPU::S_ADD_U32 : AMDGPU::S_SUB_U32; unsigned LoOpc = IsAdd ? AMDGPU::S_ADD_U32 : AMDGPU::S_SUB_U32;
unsigned HiOpc = IsAdd ? AMDGPU::S_ADDC_U32 : AMDGPU::S_SUBB_U32; unsigned HiOpc = IsAdd ? AMDGPU::S_ADDC_U32 : AMDGPU::S_SUBB_U32;
BuildMI(*BB, MI, DL, TII->get(LoOpc), DestSub0) BuildMI(*BB, MI, DL, TII->get(LoOpc), DestSub0)
.add(Src0Sub0) .add(Src0Sub0)
Show All 19 LinesMachineBasicBlock *SITargetLowering::EmitInstrWithCustomInserter(
case AMDGPU::SI_INIT_EXEC: case AMDGPU::SI_INIT_EXEC:
// This should be before all vector instructions. // This should be before all vector instructions.
BuildMI(*BB, &*BB->begin(), MI.getDebugLoc(), TII->get(AMDGPU::S_MOV_B64), BuildMI(*BB, &*BB->begin(), MI.getDebugLoc(), TII->get(AMDGPU::S_MOV_B64),
AMDGPU::EXEC) AMDGPU::EXEC)
.addImm(MI.getOperand(0).getImm()); .addImm(MI.getOperand(0).getImm());
MI.eraseFromParent(); MI.eraseFromParent();
return BB; return BB;
case AMDGPU::SI_INIT_EXEC_LO:
// This should be before all vector instructions.
BuildMI(*BB, &*BB->begin(), MI.getDebugLoc(), TII->get(AMDGPU::S_MOV_B32),
AMDGPU::EXEC_LO)
.addImm(MI.getOperand(0).getImm());
MI.eraseFromParent();
return BB;
case AMDGPU::SI_INIT_EXEC_FROM_INPUT: { case AMDGPU::SI_INIT_EXEC_FROM_INPUT: {
// Extract the thread count from an SGPR input and set EXEC accordingly. // Extract the thread count from an SGPR input and set EXEC accordingly.
// Since BFM can't shift by 64, handle that case with CMP + CMOV. // Since BFM can't shift by 64, handle that case with CMP + CMOV.
// //
// S_BFE_U32 count, input, {shift, 7} // S_BFE_U32 count, input, {shift, 7}
// S_BFM_B64 exec, count, 0 // S_BFM_B64 exec, count, 0
// S_CMP_EQ_U32 count, 64 // S_CMP_EQ_U32 count, 64
// S_CMOV_B64 exec, -1 // S_CMOV_B64 exec, -1
Show All 17 Lines for (auto I = BB->begin(); I != &MI; I++) {
} }
Found = true; Found = true;
break; break;
} }
assert(Found); assert(Found);
(void)Found; (void)Found;
// This should be before all vector instructions. // This should be before all vector instructions.
unsigned Mask = (getSubtarget()->getWavefrontSize() << 1) - 1;
bool isWave32 = getSubtarget()->isWave32();
unsigned Exec = isWave32 ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
BuildMI(*BB, FirstMI, DebugLoc(), TII->get(AMDGPU::S_BFE_U32), CountReg) BuildMI(*BB, FirstMI, DebugLoc(), TII->get(AMDGPU::S_BFE_U32), CountReg)
.addReg(InputReg) .addReg(InputReg)
.addImm((MI.getOperand(1).getImm() & 0x7f) | 0x70000); .addImm((MI.getOperand(1).getImm() & Mask) | 0x70000);
BuildMI(*BB, FirstMI, DebugLoc(), TII->get(AMDGPU::S_BFM_B64), BuildMI(*BB, FirstMI, DebugLoc(),
AMDGPU::EXEC) TII->get(isWave32 ? AMDGPU::S_BFM_B32 : AMDGPU::S_BFM_B64),
Exec)
.addReg(CountReg) .addReg(CountReg)
.addImm(0); .addImm(0);
BuildMI(*BB, FirstMI, DebugLoc(), TII->get(AMDGPU::S_CMP_EQ_U32)) BuildMI(*BB, FirstMI, DebugLoc(), TII->get(AMDGPU::S_CMP_EQ_U32))
.addReg(CountReg, RegState::Kill) .addReg(CountReg, RegState::Kill)
.addImm(64); .addImm(getSubtarget()->getWavefrontSize());
BuildMI(*BB, FirstMI, DebugLoc(), TII->get(AMDGPU::S_CMOV_B64), BuildMI(*BB, FirstMI, DebugLoc(),
AMDGPU::EXEC) TII->get(isWave32 ? AMDGPU::S_CMOV_B32 : AMDGPU::S_CMOV_B64),
Exec)
.addImm(-1); .addImm(-1);
MI.eraseFromParent(); MI.eraseFromParent();
return BB; return BB;
} }
case AMDGPU::GET_GROUPSTATICSIZE: { case AMDGPU::GET_GROUPSTATICSIZE: {
DebugLoc DL = MI.getDebugLoc(); DebugLoc DL = MI.getDebugLoc();
BuildMI(*BB, MI, DL, TII->get(AMDGPU::S_MOV_B32)) BuildMI(*BB, MI, DL, TII->get(AMDGPU::S_MOV_B32))
Show All 14 LinesMachineBasicBlock *SITargetLowering::EmitInstrWithCustomInserter(
case AMDGPU::SI_INDIRECT_DST_V8: case AMDGPU::SI_INDIRECT_DST_V8:
case AMDGPU::SI_INDIRECT_DST_V16: case AMDGPU::SI_INDIRECT_DST_V16:
return emitIndirectDst(MI, *BB, *getSubtarget()); return emitIndirectDst(MI, *BB, *getSubtarget());
case AMDGPU::SI_KILL_F32_COND_IMM_PSEUDO: case AMDGPU::SI_KILL_F32_COND_IMM_PSEUDO:
case AMDGPU::SI_KILL_I1_PSEUDO: case AMDGPU::SI_KILL_I1_PSEUDO:
return splitKillBlock(MI, BB); return splitKillBlock(MI, BB);
case AMDGPU::V_CNDMASK_B64_PSEUDO: { case AMDGPU::V_CNDMASK_B64_PSEUDO: {
MachineRegisterInfo &MRI = BB->getParent()->getRegInfo(); MachineRegisterInfo &MRI = BB->getParent()->getRegInfo();
const GCNSubtarget &ST = MF->getSubtarget<GCNSubtarget>();
const SIRegisterInfo *TRI = ST.getRegisterInfo();
unsigned Dst = MI.getOperand(0).getReg(); unsigned Dst = MI.getOperand(0).getReg();
unsigned Src0 = MI.getOperand(1).getReg(); unsigned Src0 = MI.getOperand(1).getReg();
unsigned Src1 = MI.getOperand(2).getReg(); unsigned Src1 = MI.getOperand(2).getReg();
const DebugLoc &DL = MI.getDebugLoc(); const DebugLoc &DL = MI.getDebugLoc();
unsigned SrcCond = MI.getOperand(3).getReg(); unsigned SrcCond = MI.getOperand(3).getReg();
unsigned DstLo = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass); unsigned DstLo = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass);
unsigned DstHi = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass); unsigned DstHi = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass);
unsigned SrcCondCopy = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); const auto *CondRC = TRI->getRegClass(AMDGPU::SReg_1_XEXECRegClassID);
unsigned SrcCondCopy = MRI.createVirtualRegister(CondRC);
BuildMI(*BB, MI, DL, TII->get(AMDGPU::COPY), SrcCondCopy) BuildMI(*BB, MI, DL, TII->get(AMDGPU::COPY), SrcCondCopy)
.addReg(SrcCond); .addReg(SrcCond);
BuildMI(*BB, MI, DL, TII->get(AMDGPU::V_CNDMASK_B32_e64), DstLo) BuildMI(*BB, MI, DL, TII->get(AMDGPU::V_CNDMASK_B32_e64), DstLo)
.addImm(0) .addImm(0)
.addReg(Src0, 0, AMDGPU::sub0) .addReg(Src0, 0, AMDGPU::sub0)
.addImm(0) .addImm(0)
.addReg(Src1, 0, AMDGPU::sub0) .addReg(Src1, 0, AMDGPU::sub0)
▲ Show 20 Lines • Show All 61 Lines • ▼ Show 20 Lines case AMDGPU::V_SUBREV_I32_e32: {
bool NeedClampOperand = false; bool NeedClampOperand = false;
if (TII->pseudoToMCOpcode(Opc) == -1) { if (TII->pseudoToMCOpcode(Opc) == -1) {
Opc = AMDGPU::getVOPe64(Opc); Opc = AMDGPU::getVOPe64(Opc);
NeedClampOperand = true; NeedClampOperand = true;
} }
auto I = BuildMI(*BB, MI, DL, TII->get(Opc), MI.getOperand(0).getReg()); auto I = BuildMI(*BB, MI, DL, TII->get(Opc), MI.getOperand(0).getReg());
if (TII->isVOP3(*I)) { if (TII->isVOP3(*I)) {
I.addReg(AMDGPU::VCC, RegState::Define); const GCNSubtarget &ST = MF->getSubtarget<GCNSubtarget>();
const SIRegisterInfo *TRI = ST.getRegisterInfo();
I.addReg(TRI->getVCC(), RegState::Define);
} }
I.add(MI.getOperand(1)) I.add(MI.getOperand(1))
.add(MI.getOperand(2)); .add(MI.getOperand(2));
if (NeedClampOperand) if (NeedClampOperand)
I.addImm(0); // clamp bit for e64 encoding I.addImm(0); // clamp bit for e64 encoding
TII->legalizeOperands(*I); TII->legalizeOperands(*I);
▲ Show 20 Lines • Show All 255 Lines • ▼ Show 20 Linesstatic SDValue lowerICMPIntrinsic(const SITargetLowering &TLI,
const auto *CD = cast<ConstantSDNode>(N->getOperand(3)); const auto *CD = cast<ConstantSDNode>(N->getOperand(3));
int CondCode = CD->getSExtValue(); int CondCode = CD->getSExtValue();
if (CondCode < ICmpInst::Predicate::FIRST_ICMP_PREDICATE || if (CondCode < ICmpInst::Predicate::FIRST_ICMP_PREDICATE ||
CondCode > ICmpInst::Predicate::LAST_ICMP_PREDICATE) CondCode > ICmpInst::Predicate::LAST_ICMP_PREDICATE)
return DAG.getUNDEF(VT); return DAG.getUNDEF(VT);
ICmpInst::Predicate IcInput = static_cast<ICmpInst::Predicate>(CondCode); ICmpInst::Predicate IcInput = static_cast<ICmpInst::Predicate>(CondCode);
SDValue LHS = N->getOperand(1); SDValue LHS = N->getOperand(1);
SDValue RHS = N->getOperand(2); SDValue RHS = N->getOperand(2);
SDLoc DL(N); SDLoc DL(N);
EVT CmpVT = LHS.getValueType(); EVT CmpVT = LHS.getValueType();
if (CmpVT == MVT::i16 && !TLI.isTypeLegal(MVT::i16)) { if (CmpVT == MVT::i16 && !TLI.isTypeLegal(MVT::i16)) {
unsigned PromoteOp = ICmpInst::isSigned(IcInput) ? unsigned PromoteOp = ICmpInst::isSigned(IcInput) ?
ISD::SIGN_EXTEND : ISD::ZERO_EXTEND; ISD::SIGN_EXTEND : ISD::ZERO_EXTEND;
LHS = DAG.getNode(PromoteOp, DL, MVT::i32, LHS); LHS = DAG.getNode(PromoteOp, DL, MVT::i32, LHS);
RHS = DAG.getNode(PromoteOp, DL, MVT::i32, RHS); RHS = DAG.getNode(PromoteOp, DL, MVT::i32, RHS);
} }
ISD::CondCode CCOpcode = getICmpCondCode(IcInput); ISD::CondCode CCOpcode = getICmpCondCode(IcInput);
return DAG.getNode(AMDGPUISD::SETCC, DL, VT, LHS, RHS, unsigned WavefrontSize = TLI.getSubtarget()->getWavefrontSize();
EVT CCVT = EVT::getIntegerVT(*DAG.getContext(), WavefrontSize);
SDValue SetCC = DAG.getNode(AMDGPUISD::SETCC, DL, CCVT, LHS, RHS,
DAG.getCondCode(CCOpcode)); DAG.getCondCode(CCOpcode));
if (VT.bitsEq(CCVT))
return SetCC;
return DAG.getZExtOrTrunc(SetCC, DL, VT);
} }
static SDValue lowerFCMPIntrinsic(const SITargetLowering &TLI, static SDValue lowerFCMPIntrinsic(const SITargetLowering &TLI,
SDNode *N, SelectionDAG &DAG) { SDNode *N, SelectionDAG &DAG) {
EVT VT = N->getValueType(0); EVT VT = N->getValueType(0);
const auto *CD = cast<ConstantSDNode>(N->getOperand(3)); const auto *CD = cast<ConstantSDNode>(N->getOperand(3));
int CondCode = CD->getSExtValue(); int CondCode = CD->getSExtValue();
Show All 9 Linesstatic SDValue lowerFCMPIntrinsic(const SITargetLowering &TLI,
if (CmpVT == MVT::f16 && !TLI.isTypeLegal(CmpVT)) { if (CmpVT == MVT::f16 && !TLI.isTypeLegal(CmpVT)) {
Src0 = DAG.getNode(ISD::FP_EXTEND, SL, MVT::f32, Src0); Src0 = DAG.getNode(ISD::FP_EXTEND, SL, MVT::f32, Src0);
Src1 = DAG.getNode(ISD::FP_EXTEND, SL, MVT::f32, Src1); Src1 = DAG.getNode(ISD::FP_EXTEND, SL, MVT::f32, Src1);
} }
FCmpInst::Predicate IcInput = static_cast<FCmpInst::Predicate>(CondCode); FCmpInst::Predicate IcInput = static_cast<FCmpInst::Predicate>(CondCode);
ISD::CondCode CCOpcode = getFCmpCondCode(IcInput); ISD::CondCode CCOpcode = getFCmpCondCode(IcInput);
return DAG.getNode(AMDGPUISD::SETCC, SL, VT, Src0, unsigned WavefrontSize = TLI.getSubtarget()->getWavefrontSize();
EVT CCVT = EVT::getIntegerVT(*DAG.getContext(), WavefrontSize);
SDValue SetCC = DAG.getNode(AMDGPUISD::SETCC, SL, CCVT, Src0,
Src1, DAG.getCondCode(CCOpcode)); Src1, DAG.getCondCode(CCOpcode));
if (VT.bitsEq(CCVT))
return SetCC;
return DAG.getZExtOrTrunc(SetCC, SL, VT);
} }
void SITargetLowering::ReplaceNodeResults(SDNode *N, void SITargetLowering::ReplaceNodeResults(SDNode *N,
SmallVectorImpl<SDValue> &Results, SmallVectorImpl<SDValue> &Results,
SelectionDAG &DAG) const { SelectionDAG &DAG) const {
switch (N->getOpcode()) { switch (N->getOpcode()) {
case ISD::INSERT_VECTOR_ELT: { case ISD::INSERT_VECTOR_ELT: {
if (SDValue Res = lowerINSERT_VECTOR_ELT(SDValue(N, 0), DAG)) if (SDValue Res = lowerINSERT_VECTOR_ELT(SDValue(N, 0), DAG))
▲ Show 20 Lines • Show All 1,481 Lines • ▼ Show 20 Lines case Intrinsic::r600_read_tidig_y:
return loadInputValue(DAG, &AMDGPU::VGPR_32RegClass, MVT::i32, return loadInputValue(DAG, &AMDGPU::VGPR_32RegClass, MVT::i32,
SDLoc(DAG.getEntryNode()), SDLoc(DAG.getEntryNode()),
MFI->getArgInfo().WorkItemIDY); MFI->getArgInfo().WorkItemIDY);
case Intrinsic::amdgcn_workitem_id_z: case Intrinsic::amdgcn_workitem_id_z:
case Intrinsic::r600_read_tidig_z: case Intrinsic::r600_read_tidig_z:
return loadInputValue(DAG, &AMDGPU::VGPR_32RegClass, MVT::i32, return loadInputValue(DAG, &AMDGPU::VGPR_32RegClass, MVT::i32,
SDLoc(DAG.getEntryNode()), SDLoc(DAG.getEntryNode()),
MFI->getArgInfo().WorkItemIDZ); MFI->getArgInfo().WorkItemIDZ);
case Intrinsic::amdgcn_wavefrontsize:
return DAG.getConstant(MF.getSubtarget<GCNSubtarget>().getWavefrontSize(),
SDLoc(Op), MVT::i32);
case Intrinsic::amdgcn_s_buffer_load: { case Intrinsic::amdgcn_s_buffer_load: {
unsigned Cache = cast<ConstantSDNode>(Op.getOperand(3))->getZExtValue(); unsigned Cache = cast<ConstantSDNode>(Op.getOperand(3))->getZExtValue();
return lowerSBuffer(VT, DL, Op.getOperand(1), Op.getOperand(2), return lowerSBuffer(VT, DL, Op.getOperand(1), Op.getOperand(2),
DAG.getTargetConstant(Cache & 1, DL, MVT::i1), DAG); DAG.getTargetConstant(Cache & 1, DL, MVT::i1), DAG);
} }
case Intrinsic::amdgcn_fdiv_fast: case Intrinsic::amdgcn_fdiv_fast:
return lowerFDIV_FAST(Op, DAG); return lowerFDIV_FAST(Op, DAG);
case Intrinsic::amdgcn_interp_mov: { case Intrinsic::amdgcn_interp_mov: {
▲ Show 20 Lines • Show All 188 Lines • ▼ Show 20 LinesSDValue SITargetLowering::LowerINTRINSIC_WO_CHAIN(SDValue Op,
case Intrinsic::amdgcn_wwm: { case Intrinsic::amdgcn_wwm: {
SDValue Src = Op.getOperand(1); SDValue Src = Op.getOperand(1);
return SDValue(DAG.getMachineNode(AMDGPU::WWM, DL, Src.getValueType(), Src), return SDValue(DAG.getMachineNode(AMDGPU::WWM, DL, Src.getValueType(), Src),
0); 0);
} }
case Intrinsic::amdgcn_fmad_ftz: case Intrinsic::amdgcn_fmad_ftz:
return DAG.getNode(AMDGPUISD::FMAD_FTZ, DL, VT, Op.getOperand(1), return DAG.getNode(AMDGPUISD::FMAD_FTZ, DL, VT, Op.getOperand(1),
Op.getOperand(2), Op.getOperand(3)); Op.getOperand(2), Op.getOperand(3));
case Intrinsic::amdgcn_if_break:
return SDValue(DAG.getMachineNode(AMDGPU::SI_IF_BREAK, DL, VT,
Op->getOperand(1), Op->getOperand(2)), 0);
default: default:
if (const AMDGPU::ImageDimIntrinsicInfo *ImageDimIntr = if (const AMDGPU::ImageDimIntrinsicInfo *ImageDimIntr =
AMDGPU::getImageDimIntrinsicInfo(IntrinsicID)) AMDGPU::getImageDimIntrinsicInfo(IntrinsicID))
return lowerImage(Op, ImageDimIntr, DAG); return lowerImage(Op, ImageDimIntr, DAG);
return Op; return Op;
} }
} }
▲ Show 20 Lines • Show All 881 Lines • ▼ Show 20 Lines case Intrinsic::amdgcn_struct_buffer_store_format: {
EVT VDataType = VData.getValueType().getScalarType(); EVT VDataType = VData.getValueType().getScalarType();
if (VDataType == MVT::i8 || VDataType == MVT::i16) if (VDataType == MVT::i8 || VDataType == MVT::i16)
return handleByteShortBufferStores(DAG, VDataType, DL, Ops, M); return handleByteShortBufferStores(DAG, VDataType, DL, Ops, M);
return DAG.getMemIntrinsicNode(Opc, DL, Op->getVTList(), Ops, return DAG.getMemIntrinsicNode(Opc, DL, Op->getVTList(), Ops,
M->getMemoryVT(), M->getMemOperand()); M->getMemoryVT(), M->getMemOperand());
} }
case Intrinsic::amdgcn_end_cf:
return SDValue(DAG.getMachineNode(AMDGPU::SI_END_CF, DL, MVT::Other,
Op->getOperand(2), Chain), 0);
default: { default: {
if (const AMDGPU::ImageDimIntrinsicInfo *ImageDimIntr = if (const AMDGPU::ImageDimIntrinsicInfo *ImageDimIntr =
AMDGPU::getImageDimIntrinsicInfo(IntrinsicID)) AMDGPU::getImageDimIntrinsicInfo(IntrinsicID))
return lowerImage(Op, ImageDimIntr, DAG); return lowerImage(Op, ImageDimIntr, DAG);
return Op; return Op;
} }
} }
▲ Show 20 Lines • Show All 3,469 Lines • ▼ Show 20 Lines
} }
// Figure out which registers should be reserved for stack access. Only after // Figure out which registers should be reserved for stack access. Only after
// the function is legalized do we know all of the non-spill stack objects or if // the function is legalized do we know all of the non-spill stack objects or if
// calls are present. // calls are present.
void SITargetLowering::finalizeLowering(MachineFunction &MF) const { void SITargetLowering::finalizeLowering(MachineFunction &MF) const {
MachineRegisterInfo &MRI = MF.getRegInfo(); MachineRegisterInfo &MRI = MF.getRegInfo();
SIMachineFunctionInfo *Info = MF.getInfo<SIMachineFunctionInfo>(); SIMachineFunctionInfo *Info = MF.getInfo<SIMachineFunctionInfo>();
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
const SIRegisterInfo *TRI = Subtarget->getRegisterInfo(); const SIRegisterInfo *TRI = Subtarget->getRegisterInfo();
if (Info->isEntryFunction()) { if (Info->isEntryFunction()) {
// Callable functions have fixed registers used for stack access. // Callable functions have fixed registers used for stack access.
reservePrivateMemoryRegs(getTargetMachine(), MF, *TRI, *Info); reservePrivateMemoryRegs(getTargetMachine(), MF, *TRI, *Info);
} }
assert(!TRI->isSubRegister(Info->getScratchRSrcReg(), assert(!TRI->isSubRegister(Info->getScratchRSrcReg(),
Show All 11 Linesvoid SITargetLowering::finalizeLowering(MachineFunction &MF) const {
if (Info->getScratchWaveOffsetReg() != AMDGPU::SCRATCH_WAVE_OFFSET_REG) { if (Info->getScratchWaveOffsetReg() != AMDGPU::SCRATCH_WAVE_OFFSET_REG) {
MRI.replaceRegWith(AMDGPU::SCRATCH_WAVE_OFFSET_REG, MRI.replaceRegWith(AMDGPU::SCRATCH_WAVE_OFFSET_REG,
Info->getScratchWaveOffsetReg()); Info->getScratchWaveOffsetReg());
} }
Info->limitOccupancy(MF); Info->limitOccupancy(MF);
if (ST.isWave32() && !MF.empty()) {
// Add VCC_HI def because many instructions marked as imp-use VCC where
// we may only define VCC_LO. If nothing defines VCC_HI we may end up
// having a use of undef.
const SIInstrInfo *TII = ST.getInstrInfo();
DebugLoc DL;
MachineBasicBlock &MBB = MF.front();
MachineBasicBlock::iterator I = MBB.getFirstNonDebugInstr();
BuildMI(MBB, I, DL, TII->get(TargetOpcode::IMPLICIT_DEF), AMDGPU::VCC_HI);
for (auto &MBB : MF) {
for (auto &MI : MBB) {
TII->fixImplicitOperands(MI);
}
}
}
TargetLoweringBase::finalizeLowering(MF); TargetLoweringBase::finalizeLowering(MF);
} }
void SITargetLowering::computeKnownBitsForFrameIndex(const SDValue Op, void SITargetLowering::computeKnownBitsForFrameIndex(const SDValue Op,
KnownBits &Known, KnownBits &Known,
const APInt &DemandedElts, const APInt &DemandedElts,
const SelectionDAG &DAG, const SelectionDAG &DAG,
unsigned Depth) const { unsigned Depth) const {
▲ Show 20 Lines • Show All 209 Lines • Show Last 20 Lines

lib/Target/AMDGPU/SIInsertSkips.cpp

Show First 20 Lines • Show All 265 Lines • ▼ Show 20 Lines if (TRI->isVGPR(MBB.getParent()->getRegInfo(),
.addImm(0) // src1 modifiers .addImm(0) // src1 modifiers
.add(MI.getOperand(0)); .add(MI.getOperand(0));
I.addImm(0); // omod I.addImm(0); // omod
} }
break; break;
} }
case AMDGPU::SI_KILL_I1_TERMINATOR: { case AMDGPU::SI_KILL_I1_TERMINATOR: {
const MachineFunction *MF = MI.getParent()->getParent();
const GCNSubtarget &ST = MF->getSubtarget<GCNSubtarget>();
unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
const MachineOperand &Op = MI.getOperand(0); const MachineOperand &Op = MI.getOperand(0);
int64_t KillVal = MI.getOperand(1).getImm(); int64_t KillVal = MI.getOperand(1).getImm();
assert(KillVal == 0 || KillVal == -1); assert(KillVal == 0 || KillVal == -1);
// Kill all threads if Op0 is an immediate and equal to the Kill value. // Kill all threads if Op0 is an immediate and equal to the Kill value.
if (Op.isImm()) { if (Op.isImm()) {
int64_t Imm = Op.getImm(); int64_t Imm = Op.getImm();
assert(Imm == 0 || Imm == -1); assert(Imm == 0 || Imm == -1);
if (Imm == KillVal) if (Imm == KillVal)
BuildMI(MBB, &MI, DL, TII->get(AMDGPU::S_MOV_B64), AMDGPU::EXEC) BuildMI(MBB, &MI, DL, TII->get(ST.isWave32() ? AMDGPU::S_MOV_B32
: AMDGPU::S_MOV_B64), Exec)
.addImm(0); .addImm(0);
break; break;
} }
unsigned Opcode = KillVal ? AMDGPU::S_ANDN2_B64 : AMDGPU::S_AND_B64; unsigned Opcode = KillVal ? AMDGPU::S_ANDN2_B64 : AMDGPU::S_AND_B64;
BuildMI(MBB, &MI, DL, TII->get(Opcode), AMDGPU::EXEC) if (ST.isWave32())
.addReg(AMDGPU::EXEC) Opcode = KillVal ? AMDGPU::S_ANDN2_B32 : AMDGPU::S_AND_B32;
BuildMI(MBB, &MI, DL, TII->get(Opcode), Exec)
.addReg(Exec)
.add(Op); .add(Op);
break; break;
} }
default: default:
llvm_unreachable("invalid opcode, expected SI_KILL_*_TERMINATOR"); llvm_unreachable("invalid opcode, expected SI_KILL_*_TERMINATOR");
} }
} }
Show All 32 Linesbool SIInsertSkips::optimizeVccBranch(MachineInstr &MI) const {
// Match: // Match:
// sreg = -1 // sreg = -1
// vcc = S_AND_B64 exec, sreg // vcc = S_AND_B64 exec, sreg
// S_CBRANCH_VCC[N]Z // S_CBRANCH_VCC[N]Z
// => // =>
// S_CBRANCH_EXEC[N]Z // S_CBRANCH_EXEC[N]Z
bool Changed = false; bool Changed = false;
MachineBasicBlock &MBB = *MI.getParent(); MachineBasicBlock &MBB = *MI.getParent();
const unsigned CondReg = AMDGPU::VCC; const GCNSubtarget &ST = MBB.getParent()->getSubtarget<GCNSubtarget>();
const unsigned ExecReg = AMDGPU::EXEC; const bool IsWave32 = ST.isWave32();
const unsigned And = AMDGPU::S_AND_B64; const unsigned CondReg = TRI->getVCC();
const unsigned ExecReg = IsWave32 ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
const unsigned And = IsWave32 ? AMDGPU::S_AND_B32 : AMDGPU::S_AND_B64;
MachineBasicBlock::reverse_iterator A = MI.getReverseIterator(), MachineBasicBlock::reverse_iterator A = MI.getReverseIterator(),
E = MBB.rend(); E = MBB.rend();
bool ReadsCond = false; bool ReadsCond = false;
unsigned Threshold = 5; unsigned Threshold = 5;
for (++A ; A != E ; ++A) { for (++A ; A != E ; ++A) {
if (!--Threshold) if (!--Threshold)
return false; return false;
▲ Show 20 Lines • Show All 180 Lines • Show Last 20 Lines

lib/Target/AMDGPU/SIInsertWaitcnts.cpp

Show First 20 Lines • Show All 1,396 Lines • ▼ Show 20 Lines#endif
// TODO: Remove this work-around after fixing the scheduler and enable the // TODO: Remove this work-around after fixing the scheduler and enable the
// assert above. // assert above.
if (VCCZBugWorkAround) { if (VCCZBugWorkAround) {
// Restore the vccz bit. Any time a value is written to vcc, the vcc // Restore the vccz bit. Any time a value is written to vcc, the vcc
// bit is updated, so we can restore the bit by reading the value of // bit is updated, so we can restore the bit by reading the value of
// vcc and then writing it back to the register. // vcc and then writing it back to the register.
BuildMI(Block, Inst, Inst.getDebugLoc(), BuildMI(Block, Inst, Inst.getDebugLoc(),
TII->get(AMDGPU::S_MOV_B64), TII->get(ST->isWave32() ? AMDGPU::S_MOV_B32 : AMDGPU::S_MOV_B64),
AMDGPU::VCC) TRI->getVCC())
.addReg(AMDGPU::VCC); .addReg(TRI->getVCC());
VCCZBugHandledSet.insert(&Inst); VCCZBugHandledSet.insert(&Inst);
Modified = true; Modified = true;
} }
++Iter; ++Iter;
} }
return Modified; return Modified;
▲ Show 20 Lines • Show All 166 Lines • Show Last 20 Lines

lib/Target/AMDGPU/SIInstrInfo.h

Show First 20 Lines • Show All 936 Lines • ▼ Show 20 Linespublic:
static bool isLegalMUBUFImmOffset(unsigned Imm) { static bool isLegalMUBUFImmOffset(unsigned Imm) {
return isUInt<12>(Imm); return isUInt<12>(Imm);
} }
/// \brief Return a target-specific opcode if Opcode is a pseudo instruction. /// \brief Return a target-specific opcode if Opcode is a pseudo instruction.
/// Return -1 if the target-specific opcode for the pseudo instruction does /// Return -1 if the target-specific opcode for the pseudo instruction does
/// not exist. If Opcode is not a pseudo instruction, this is identity. /// not exist. If Opcode is not a pseudo instruction, this is identity.
int pseudoToMCOpcode(int Opcode) const; int pseudoToMCOpcode(int Opcode) const;
const TargetRegisterClass *getRegClass(const MCInstrDesc &TID, unsigned OpNum,
const TargetRegisterInfo *TRI,
const MachineFunction &MF)
const override {
if (OpNum >= TID.getNumOperands())
return nullptr;
return RI.getRegClass(TID.OpInfo[OpNum].RegClass);
}
void fixImplicitOperands(MachineInstr &MI) const;
}; };
/// \brief Returns true if a reg:subreg pair P has a TRC class /// \brief Returns true if a reg:subreg pair P has a TRC class
inline bool isOfRegClass(const TargetInstrInfo::RegSubRegPair &P, inline bool isOfRegClass(const TargetInstrInfo::RegSubRegPair &P,
const TargetRegisterClass &TRC, const TargetRegisterClass &TRC,
MachineRegisterInfo &MRI) { MachineRegisterInfo &MRI) {
auto *RC = MRI.getRegClass(P.Reg); auto *RC = MRI.getRegClass(P.Reg);
if (!P.SubReg) if (!P.SubReg)
▲ Show 20 Lines • Show All 107 Lines • Show Last 20 Lines

lib/Target/AMDGPU/SIInstrInfo.cpp

Show First 20 Lines • Show All 521 Lines • ▼ Show 20 Lines if (RC == &AMDGPU::SReg_32_XM0RegClass ||
RC == &AMDGPU::SReg_32RegClass) { RC == &AMDGPU::SReg_32RegClass) {
if (SrcReg == AMDGPU::SCC) { if (SrcReg == AMDGPU::SCC) {
BuildMI(MBB, MI, DL, get(AMDGPU::S_CSELECT_B32), DestReg) BuildMI(MBB, MI, DL, get(AMDGPU::S_CSELECT_B32), DestReg)
.addImm(-1) .addImm(-1)
.addImm(0); .addImm(0);
return; return;
} }
if (DestReg == AMDGPU::VCC_LO) {
if (AMDGPU::SReg_32RegClass.contains(SrcReg)) {
BuildMI(MBB, MI, DL, get(AMDGPU::S_MOV_B32), AMDGPU::VCC_LO)
.addReg(SrcReg, getKillRegState(KillSrc));
} else {
// FIXME: Hack until VReg_1 removed.
assert(AMDGPU::VGPR_32RegClass.contains(SrcReg));
BuildMI(MBB, MI, DL, get(AMDGPU::V_CMP_NE_U32_e32))
.addImm(0)
.addReg(SrcReg, getKillRegState(KillSrc));
}
return;
}
if (!AMDGPU::SReg_32RegClass.contains(SrcReg)) { if (!AMDGPU::SReg_32RegClass.contains(SrcReg)) {
reportIllegalCopy(this, MBB, MI, DL, DestReg, SrcReg, KillSrc); reportIllegalCopy(this, MBB, MI, DL, DestReg, SrcReg, KillSrc);
return; return;
} }
BuildMI(MBB, MI, DL, get(AMDGPU::S_MOV_B32), DestReg) BuildMI(MBB, MI, DL, get(AMDGPU::S_MOV_B32), DestReg)
.addReg(SrcReg, getKillRegState(KillSrc)); .addReg(SrcReg, getKillRegState(KillSrc));
return; return;
▲ Show 20 Lines • Show All 155 Lines • ▼ Show 20 Lines
void SIInstrInfo::insertVectorSelect(MachineBasicBlock &MBB, void SIInstrInfo::insertVectorSelect(MachineBasicBlock &MBB,
MachineBasicBlock::iterator I, MachineBasicBlock::iterator I,
const DebugLoc &DL, unsigned DstReg, const DebugLoc &DL, unsigned DstReg,
ArrayRef<MachineOperand> Cond, ArrayRef<MachineOperand> Cond,
unsigned TrueReg, unsigned TrueReg,
unsigned FalseReg) const { unsigned FalseReg) const {
MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo(); MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
MachineFunction *MF = MBB.getParent();
const GCNSubtarget &ST = MF->getSubtarget<GCNSubtarget>();
const TargetRegisterClass *BoolXExecRC =
RI.getRegClass(AMDGPU::SReg_1_XEXECRegClassID);
assert(MRI.getRegClass(DstReg) == &AMDGPU::VGPR_32RegClass && assert(MRI.getRegClass(DstReg) == &AMDGPU::VGPR_32RegClass &&
"Not a VGPR32 reg"); "Not a VGPR32 reg");
if (Cond.size() == 1) { if (Cond.size() == 1) {
unsigned SReg = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); unsigned SReg = MRI.createVirtualRegister(BoolXExecRC);
BuildMI(MBB, I, DL, get(AMDGPU::COPY), SReg) BuildMI(MBB, I, DL, get(AMDGPU::COPY), SReg)
.add(Cond[0]); .add(Cond[0]);
BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg) BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg)
.addImm(0) .addImm(0)
.addReg(FalseReg) .addReg(FalseReg)
.addImm(0) .addImm(0)
.addReg(TrueReg) .addReg(TrueReg)
.addReg(SReg); .addReg(SReg);
} else if (Cond.size() == 2) { } else if (Cond.size() == 2) {
assert(Cond[0].isImm() && "Cond[0] is not an immediate"); assert(Cond[0].isImm() && "Cond[0] is not an immediate");
switch (Cond[0].getImm()) { switch (Cond[0].getImm()) {
case SIInstrInfo::SCC_TRUE: { case SIInstrInfo::SCC_TRUE: {
unsigned SReg = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); unsigned SReg = MRI.createVirtualRegister(BoolXExecRC);
BuildMI(MBB, I, DL, get(AMDGPU::S_CSELECT_B64), SReg) BuildMI(MBB, I, DL, get(ST.isWave32() ? AMDGPU::S_CSELECT_B32
: AMDGPU::S_CSELECT_B64), SReg)
.addImm(-1) .addImm(-1)
.addImm(0); .addImm(0);
BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg) BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg)
.addImm(0) .addImm(0)
.addReg(FalseReg) .addReg(FalseReg)
.addImm(0) .addImm(0)
.addReg(TrueReg) .addReg(TrueReg)
.addReg(SReg); .addReg(SReg);
break; break;
} }
case SIInstrInfo::SCC_FALSE: { case SIInstrInfo::SCC_FALSE: {
unsigned SReg = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); unsigned SReg = MRI.createVirtualRegister(BoolXExecRC);
BuildMI(MBB, I, DL, get(AMDGPU::S_CSELECT_B64), SReg) BuildMI(MBB, I, DL, get(ST.isWave32() ? AMDGPU::S_CSELECT_B32
: AMDGPU::S_CSELECT_B64), SReg)
.addImm(0) .addImm(0)
.addImm(-1); .addImm(-1);
BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg) BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg)
.addImm(0) .addImm(0)
.addReg(FalseReg) .addReg(FalseReg)
.addImm(0) .addImm(0)
.addReg(TrueReg) .addReg(TrueReg)
.addReg(SReg); .addReg(SReg);
break; break;
} }
case SIInstrInfo::VCCNZ: { case SIInstrInfo::VCCNZ: {
MachineOperand RegOp = Cond[1]; MachineOperand RegOp = Cond[1];
RegOp.setImplicit(false); RegOp.setImplicit(false);
unsigned SReg = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); unsigned SReg = MRI.createVirtualRegister(BoolXExecRC);
BuildMI(MBB, I, DL, get(AMDGPU::COPY), SReg) BuildMI(MBB, I, DL, get(AMDGPU::COPY), SReg)
.add(RegOp); .add(RegOp);
BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg) BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg)
.addImm(0) .addImm(0)
.addReg(FalseReg) .addReg(FalseReg)
.addImm(0) .addImm(0)
.addReg(TrueReg) .addReg(TrueReg)
.addReg(SReg); .addReg(SReg);
break; break;
} }
case SIInstrInfo::VCCZ: { case SIInstrInfo::VCCZ: {
MachineOperand RegOp = Cond[1]; MachineOperand RegOp = Cond[1];
RegOp.setImplicit(false); RegOp.setImplicit(false);
unsigned SReg = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); unsigned SReg = MRI.createVirtualRegister(BoolXExecRC);
BuildMI(MBB, I, DL, get(AMDGPU::COPY), SReg) BuildMI(MBB, I, DL, get(AMDGPU::COPY), SReg)
.add(RegOp); .add(RegOp);
BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg) BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg)
.addImm(0) .addImm(0)
.addReg(TrueReg) .addReg(TrueReg)
.addImm(0) .addImm(0)
.addReg(FalseReg) .addReg(FalseReg)
.addReg(SReg); .addReg(SReg);
break; break;
} }
case SIInstrInfo::EXECNZ: { case SIInstrInfo::EXECNZ: {
unsigned SReg = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); unsigned SReg = MRI.createVirtualRegister(BoolXExecRC);
unsigned SReg2 = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned SReg2 = MRI.createVirtualRegister(RI.getBoolRC());
BuildMI(MBB, I, DL, get(AMDGPU::S_OR_SAVEEXEC_B64), SReg2) BuildMI(MBB, I, DL, get(ST.isWave32() ? AMDGPU::S_OR_SAVEEXEC_B32
: AMDGPU::S_OR_SAVEEXEC_B64), SReg2)
.addImm(0); .addImm(0);
BuildMI(MBB, I, DL, get(AMDGPU::S_CSELECT_B64), SReg) BuildMI(MBB, I, DL, get(ST.isWave32() ? AMDGPU::S_CSELECT_B32
: AMDGPU::S_CSELECT_B64), SReg)
.addImm(-1) .addImm(-1)
.addImm(0); .addImm(0);
BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg) BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg)
.addImm(0) .addImm(0)
.addReg(FalseReg) .addReg(FalseReg)
.addImm(0) .addImm(0)
.addReg(TrueReg) .addReg(TrueReg)
.addReg(SReg); .addReg(SReg);
break; break;
} }
case SIInstrInfo::EXECZ: { case SIInstrInfo::EXECZ: {
unsigned SReg = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); unsigned SReg = MRI.createVirtualRegister(BoolXExecRC);
unsigned SReg2 = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned SReg2 = MRI.createVirtualRegister(RI.getBoolRC());
BuildMI(MBB, I, DL, get(AMDGPU::S_OR_SAVEEXEC_B64), SReg2) BuildMI(MBB, I, DL, get(ST.isWave32() ? AMDGPU::S_OR_SAVEEXEC_B32
: AMDGPU::S_OR_SAVEEXEC_B64), SReg2)
.addImm(0); .addImm(0);
BuildMI(MBB, I, DL, get(AMDGPU::S_CSELECT_B64), SReg) BuildMI(MBB, I, DL, get(ST.isWave32() ? AMDGPU::S_CSELECT_B32
: AMDGPU::S_CSELECT_B64), SReg)
.addImm(0) .addImm(0)
.addImm(-1); .addImm(-1);
BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg) BuildMI(MBB, I, DL, get(AMDGPU::V_CNDMASK_B32_e64), DstReg)
.addImm(0) .addImm(0)
.addReg(FalseReg) .addReg(FalseReg)
.addImm(0) .addImm(0)
.addReg(TrueReg) .addReg(TrueReg)
.addReg(SReg); .addReg(SReg);
llvm_unreachable("Unhandled branch predicate EXECZ"); llvm_unreachable("Unhandled branch predicate EXECZ");
break; break;
} }
default: default:
llvm_unreachable("invalid branch predicate"); llvm_unreachable("invalid branch predicate");
} }
} else { } else {
llvm_unreachable("Can only handle Cond size 1 or 2"); llvm_unreachable("Can only handle Cond size 1 or 2");
} }
} }
unsigned SIInstrInfo::insertEQ(MachineBasicBlock *MBB, unsigned SIInstrInfo::insertEQ(MachineBasicBlock *MBB,
MachineBasicBlock::iterator I, MachineBasicBlock::iterator I,
const DebugLoc &DL, const DebugLoc &DL,
unsigned SrcReg, int Value) const { unsigned SrcReg, int Value) const {
MachineRegisterInfo &MRI = MBB->getParent()->getRegInfo(); MachineRegisterInfo &MRI = MBB->getParent()->getRegInfo();
unsigned Reg = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned Reg = MRI.createVirtualRegister(RI.getBoolRC());
BuildMI(*MBB, I, DL, get(AMDGPU::V_CMP_EQ_I32_e64), Reg) BuildMI(*MBB, I, DL, get(AMDGPU::V_CMP_EQ_I32_e64), Reg)
.addImm(Value) .addImm(Value)
.addReg(SrcReg); .addReg(SrcReg);
return Reg; return Reg;
} }
unsigned SIInstrInfo::insertNE(MachineBasicBlock *MBB, unsigned SIInstrInfo::insertNE(MachineBasicBlock *MBB,
MachineBasicBlock::iterator I, MachineBasicBlock::iterator I,
const DebugLoc &DL, const DebugLoc &DL,
unsigned SrcReg, int Value) const { unsigned SrcReg, int Value) const {
MachineRegisterInfo &MRI = MBB->getParent()->getRegInfo(); MachineRegisterInfo &MRI = MBB->getParent()->getRegInfo();
unsigned Reg = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned Reg = MRI.createVirtualRegister(RI.getBoolRC());
BuildMI(*MBB, I, DL, get(AMDGPU::V_CMP_NE_I32_e64), Reg) BuildMI(*MBB, I, DL, get(AMDGPU::V_CMP_NE_I32_e64), Reg)
.addImm(Value) .addImm(Value)
.addReg(SrcReg); .addReg(SrcReg);
return Reg; return Reg;
} }
unsigned SIInstrInfo::getMovOpcode(const TargetRegisterClass *DstRC) const { unsigned SIInstrInfo::getMovOpcode(const TargetRegisterClass *DstRC) const {
▲ Show 20 Lines • Show All 364 Lines • ▼ Show 20 Linesbool SIInstrInfo::expandPostRAPseudo(MachineInstr &MI) const {
switch (MI.getOpcode()) { switch (MI.getOpcode()) {
default: return TargetInstrInfo::expandPostRAPseudo(MI); default: return TargetInstrInfo::expandPostRAPseudo(MI);
case AMDGPU::S_MOV_B64_term: case AMDGPU::S_MOV_B64_term:
// This is only a terminator to get the correct spill code placement during // This is only a terminator to get the correct spill code placement during
// register allocation. // register allocation.
MI.setDesc(get(AMDGPU::S_MOV_B64)); MI.setDesc(get(AMDGPU::S_MOV_B64));
break; break;
case AMDGPU::S_MOV_B32_term:
// This is only a terminator to get the correct spill code placement during
// register allocation.
MI.setDesc(get(AMDGPU::S_MOV_B32));
break;
case AMDGPU::S_XOR_B64_term: case AMDGPU::S_XOR_B64_term:
// This is only a terminator to get the correct spill code placement during // This is only a terminator to get the correct spill code placement during
// register allocation. // register allocation.
MI.setDesc(get(AMDGPU::S_XOR_B64)); MI.setDesc(get(AMDGPU::S_XOR_B64));
break; break;
case AMDGPU::S_XOR_B32_term:
// This is only a terminator to get the correct spill code placement during
// register allocation.
MI.setDesc(get(AMDGPU::S_XOR_B32));
break;
case AMDGPU::S_OR_B32_term:
// This is only a terminator to get the correct spill code placement during
// register allocation.
MI.setDesc(get(AMDGPU::S_OR_B32));
break;
case AMDGPU::S_ANDN2_B64_term: case AMDGPU::S_ANDN2_B64_term:
// This is only a terminator to get the correct spill code placement during // This is only a terminator to get the correct spill code placement during
// register allocation. // register allocation.
MI.setDesc(get(AMDGPU::S_ANDN2_B64)); MI.setDesc(get(AMDGPU::S_ANDN2_B64));
break; break;
case AMDGPU::S_ANDN2_B32_term:
// This is only a terminator to get the correct spill code placement during
// register allocation.
MI.setDesc(get(AMDGPU::S_ANDN2_B32));
break;
case AMDGPU::V_MOV_B64_PSEUDO: { case AMDGPU::V_MOV_B64_PSEUDO: {
unsigned Dst = MI.getOperand(0).getReg(); unsigned Dst = MI.getOperand(0).getReg();
unsigned DstLo = RI.getSubReg(Dst, AMDGPU::sub0); unsigned DstLo = RI.getSubReg(Dst, AMDGPU::sub0);
unsigned DstHi = RI.getSubReg(Dst, AMDGPU::sub1); unsigned DstHi = RI.getSubReg(Dst, AMDGPU::sub1);
const MachineOperand &SrcOp = MI.getOperand(1); const MachineOperand &SrcOp = MI.getOperand(1);
// FIXME: Will this work for 64-bit floating point immediates? // FIXME: Will this work for 64-bit floating point immediates?
assert(!SrcOp.isFPImm()); assert(!SrcOp.isFPImm());
Show All 13 Lines if (SrcOp.isImm()) {
BuildMI(MBB, MI, DL, get(AMDGPU::V_MOV_B32_e32), DstHi) BuildMI(MBB, MI, DL, get(AMDGPU::V_MOV_B32_e32), DstHi)
.addReg(RI.getSubReg(SrcOp.getReg(), AMDGPU::sub1)) .addReg(RI.getSubReg(SrcOp.getReg(), AMDGPU::sub1))
.addReg(Dst, RegState::Implicit | RegState::Define); .addReg(Dst, RegState::Implicit | RegState::Define);
} }
MI.eraseFromParent(); MI.eraseFromParent();
break; break;
} }
case AMDGPU::V_SET_INACTIVE_B32: { case AMDGPU::V_SET_INACTIVE_B32: {
BuildMI(MBB, MI, DL, get(AMDGPU::S_NOT_B64), AMDGPU::EXEC) unsigned NotOpc = ST.isWave32() ? AMDGPU::S_NOT_B32 : AMDGPU::S_NOT_B64;
.addReg(AMDGPU::EXEC); unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
BuildMI(MBB, MI, DL, get(NotOpc), Exec)
.addReg(Exec);
BuildMI(MBB, MI, DL, get(AMDGPU::V_MOV_B32_e32), MI.getOperand(0).getReg()) BuildMI(MBB, MI, DL, get(AMDGPU::V_MOV_B32_e32), MI.getOperand(0).getReg())
.add(MI.getOperand(2)); .add(MI.getOperand(2));
BuildMI(MBB, MI, DL, get(AMDGPU::S_NOT_B64), AMDGPU::EXEC) BuildMI(MBB, MI, DL, get(NotOpc), Exec)
.addReg(AMDGPU::EXEC); .addReg(Exec);
MI.eraseFromParent(); MI.eraseFromParent();
break; break;
} }
case AMDGPU::V_SET_INACTIVE_B64: { case AMDGPU::V_SET_INACTIVE_B64: {
BuildMI(MBB, MI, DL, get(AMDGPU::S_NOT_B64), AMDGPU::EXEC) unsigned NotOpc = ST.isWave32() ? AMDGPU::S_NOT_B32 : AMDGPU::S_NOT_B64;
.addReg(AMDGPU::EXEC); unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
BuildMI(MBB, MI, DL, get(NotOpc), Exec)
.addReg(Exec);
MachineInstr *Copy = BuildMI(MBB, MI, DL, get(AMDGPU::V_MOV_B64_PSEUDO), MachineInstr *Copy = BuildMI(MBB, MI, DL, get(AMDGPU::V_MOV_B64_PSEUDO),
MI.getOperand(0).getReg()) MI.getOperand(0).getReg())
.add(MI.getOperand(2)); .add(MI.getOperand(2));
expandPostRAPseudo(*Copy); expandPostRAPseudo(*Copy);
BuildMI(MBB, MI, DL, get(AMDGPU::S_NOT_B64), AMDGPU::EXEC) BuildMI(MBB, MI, DL, get(NotOpc), Exec)
.addReg(AMDGPU::EXEC); .addReg(Exec);
MI.eraseFromParent(); MI.eraseFromParent();
break; break;
} }
case AMDGPU::V_MOVRELD_B32_V1: case AMDGPU::V_MOVRELD_B32_V1:
case AMDGPU::V_MOVRELD_B32_V2: case AMDGPU::V_MOVRELD_B32_V2:
case AMDGPU::V_MOVRELD_B32_V4: case AMDGPU::V_MOVRELD_B32_V4:
case AMDGPU::V_MOVRELD_B32_V8: case AMDGPU::V_MOVRELD_B32_V8:
case AMDGPU::V_MOVRELD_B32_V16: { case AMDGPU::V_MOVRELD_B32_V16: {
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 Lines case AMDGPU::SI_PC_ADD_REL_OFFSET: {
finalizeBundle(MBB, Bundler.begin()); finalizeBundle(MBB, Bundler.begin());
MI.eraseFromParent(); MI.eraseFromParent();
break; break;
} }
case AMDGPU::ENTER_WWM: { case AMDGPU::ENTER_WWM: {
// This only gets its own opcode so that SIPreAllocateWWMRegs can tell when // This only gets its own opcode so that SIPreAllocateWWMRegs can tell when
// WWM is entered. // WWM is entered.
MI.setDesc(get(AMDGPU::S_OR_SAVEEXEC_B64)); MI.setDesc(get(ST.isWave32() ? AMDGPU::S_OR_SAVEEXEC_B32
: AMDGPU::S_OR_SAVEEXEC_B64));
break; break;
} }
case AMDGPU::EXIT_WWM: { case AMDGPU::EXIT_WWM: {
// This only gets its own opcode so that SIPreAllocateWWMRegs can tell when // This only gets its own opcode so that SIPreAllocateWWMRegs can tell when
// WWM is exited. // WWM is exited.
MI.setDesc(get(AMDGPU::S_MOV_B64)); MI.setDesc(get(ST.isWave32() ? AMDGPU::S_MOV_B32 : AMDGPU::S_MOV_B64));
break; break;
} }
case TargetOpcode::BUNDLE: { case TargetOpcode::BUNDLE: {
if (!MI.mayLoad()) if (!MI.mayLoad())
return false; return false;
// If it is a load it must be a memory clause // If it is a load it must be a memory clause
for (MachineBasicBlock::instr_iterator I = MI.getIterator(); for (MachineBasicBlock::instr_iterator I = MI.getIterator();
▲ Show 20 Lines • Show All 346 Lines • ▼ Show 20 Linesbool SIInstrInfo::analyzeBranch(MachineBasicBlock &MBB, MachineBasicBlock *&TBB,
// exec management. // exec management.
while (I != E && !I->isBranch() && !I->isReturn() && while (I != E && !I->isBranch() && !I->isReturn() &&
I->getOpcode() != AMDGPU::SI_MASK_BRANCH) { I->getOpcode() != AMDGPU::SI_MASK_BRANCH) {
switch (I->getOpcode()) { switch (I->getOpcode()) {
case AMDGPU::SI_MASK_BRANCH: case AMDGPU::SI_MASK_BRANCH:
case AMDGPU::S_MOV_B64_term: case AMDGPU::S_MOV_B64_term:
case AMDGPU::S_XOR_B64_term: case AMDGPU::S_XOR_B64_term:
case AMDGPU::S_ANDN2_B64_term: case AMDGPU::S_ANDN2_B64_term:
case AMDGPU::S_MOV_B32_term:
case AMDGPU::S_XOR_B32_term:
case AMDGPU::S_OR_B32_term:
case AMDGPU::S_ANDN2_B32_term:
break; break;
case AMDGPU::SI_IF: case AMDGPU::SI_IF:
case AMDGPU::SI_ELSE: case AMDGPU::SI_ELSE:
case AMDGPU::SI_KILL_I1_TERMINATOR: case AMDGPU::SI_KILL_I1_TERMINATOR:
case AMDGPU::SI_KILL_F32_COND_IMM_TERMINATOR: case AMDGPU::SI_KILL_F32_COND_IMM_TERMINATOR:
// FIXME: It's messy that these need to be considered here at all. // FIXME: It's messy that these need to be considered here at all.
return true; return true;
default: default:
▲ Show 20 Lines • Show All 263 Lines • ▼ Show 20 Lines for (int Idx = 0; Idx != NElts; ++Idx) {
unsigned SubIdx = SubIndices[Idx]; unsigned SubIdx = SubIndices[Idx];
MachineInstr *Select = MachineInstr *Select =
BuildMI(MBB, I, DL, get(SelOp), DstElt) BuildMI(MBB, I, DL, get(SelOp), DstElt)
.addReg(FalseReg, 0, SubIdx) .addReg(FalseReg, 0, SubIdx)
.addReg(TrueReg, 0, SubIdx); .addReg(TrueReg, 0, SubIdx);
preserveCondRegFlags(Select->getOperand(3), Cond[1]); preserveCondRegFlags(Select->getOperand(3), Cond[1]);
fixImplicitOperands(*Select);
MIB.addReg(DstElt) MIB.addReg(DstElt)
.addImm(SubIdx); .addImm(SubIdx);
} }
} }
bool SIInstrInfo::isFoldableCopy(const MachineInstr &MI) const { bool SIInstrInfo::isFoldableCopy(const MachineInstr &MI) const {
switch (MI.getOpcode()) { switch (MI.getOpcode()) {
▲ Show 20 Lines • Show All 788 Lines • ▼ Show 20 LinesMachineInstr *SIInstrInfo::buildShrunkInst(MachineInstr &MI,
// Add the dst operand if the 32-bit encoding also has an explicit $vdst. // Add the dst operand if the 32-bit encoding also has an explicit $vdst.
// For VOPC instructions, this is replaced by an implicit def of vcc. // For VOPC instructions, this is replaced by an implicit def of vcc.
int Op32DstIdx = AMDGPU::getNamedOperandIdx(Op32, AMDGPU::OpName::vdst); int Op32DstIdx = AMDGPU::getNamedOperandIdx(Op32, AMDGPU::OpName::vdst);
if (Op32DstIdx != -1) { if (Op32DstIdx != -1) {
// dst // dst
Inst32.add(MI.getOperand(0)); Inst32.add(MI.getOperand(0));
} else { } else {
assert(MI.getOperand(0).getReg() == AMDGPU::VCC && assert(((MI.getOperand(0).getReg() == AMDGPU::VCC) ||
(MI.getOperand(0).getReg() == AMDGPU::VCC_LO)) &&
"Unexpected case"); "Unexpected case");
} }
Inst32.add(*getNamedOperand(MI, AMDGPU::OpName::src0)); Inst32.add(*getNamedOperand(MI, AMDGPU::OpName::src0));
const MachineOperand *Src1 = getNamedOperand(MI, AMDGPU::OpName::src1); const MachineOperand *Src1 = getNamedOperand(MI, AMDGPU::OpName::src1);
if (Src1) if (Src1)
Inst32.add(*Src1); Inst32.add(*Src1);
▲ Show 20 Lines • Show All 51 Lines • ▼ Show 20 Lines
static unsigned findImplicitSGPRRead(const MachineInstr &MI) { static unsigned findImplicitSGPRRead(const MachineInstr &MI) {
for (const MachineOperand &MO : MI.implicit_operands()) { for (const MachineOperand &MO : MI.implicit_operands()) {
// We only care about reads. // We only care about reads.
if (MO.isDef()) if (MO.isDef())
continue; continue;
switch (MO.getReg()) { switch (MO.getReg()) {
case AMDGPU::VCC: case AMDGPU::VCC:
case AMDGPU::VCC_LO:
case AMDGPU::VCC_HI:
case AMDGPU::M0: case AMDGPU::M0:
case AMDGPU::FLAT_SCR: case AMDGPU::FLAT_SCR:
return MO.getReg(); return MO.getReg();
default: default:
break; break;
} }
} }
▲ Show 20 Lines • Show All 929 Lines • ▼ Show 20 Linesvoid SIInstrInfo::legalizeOperandsVOP2(MachineRegisterInfo &MRI,
else if (Src1.isReg()) { else if (Src1.isReg()) {
Src0.ChangeToRegister(Src1.getReg(), false, false, Src1.isKill()); Src0.ChangeToRegister(Src1.getReg(), false, false, Src1.isKill());
Src0.setSubReg(Src1.getSubReg()); Src0.setSubReg(Src1.getSubReg());
} else } else
llvm_unreachable("Should only have register or immediate operands"); llvm_unreachable("Should only have register or immediate operands");
Src1.ChangeToRegister(Src0Reg, false, false, Src0Kill); Src1.ChangeToRegister(Src0Reg, false, false, Src0Kill);
Src1.setSubReg(Src0SubReg); Src1.setSubReg(Src0SubReg);
fixImplicitOperands(MI);
} }
// Legalize VOP3 operands. All operand types are supported for any operand // Legalize VOP3 operands. All operand types are supported for any operand
// but only one literal constant and only starting from GFX10. // but only one literal constant and only starting from GFX10.
void SIInstrInfo::legalizeOperandsVOP3(MachineRegisterInfo &MRI, void SIInstrInfo::legalizeOperandsVOP3(MachineRegisterInfo &MRI,
MachineInstr &MI) const { MachineInstr &MI) const {
unsigned Opc = MI.getOpcode(); unsigned Opc = MI.getOpcode();
▲ Show 20 Lines • Show All 160 Lines • ▼ Show 20 Lines
// Emit the actual waterfall loop, executing the wrapped instruction for each // Emit the actual waterfall loop, executing the wrapped instruction for each
// unique value of \p Rsrc across all lanes. In the best case we execute 1 // unique value of \p Rsrc across all lanes. In the best case we execute 1
// iteration, in the worst case we execute 64 (once per lane). // iteration, in the worst case we execute 64 (once per lane).
static void static void
emitLoadSRsrcFromVGPRLoop(const SIInstrInfo &TII, MachineRegisterInfo &MRI, emitLoadSRsrcFromVGPRLoop(const SIInstrInfo &TII, MachineRegisterInfo &MRI,
MachineBasicBlock &OrigBB, MachineBasicBlock &LoopBB, MachineBasicBlock &OrigBB, MachineBasicBlock &LoopBB,
const DebugLoc &DL, MachineOperand &Rsrc) { const DebugLoc &DL, MachineOperand &Rsrc) {
MachineFunction &MF = *OrigBB.getParent();
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
const SIRegisterInfo *TRI = ST.getRegisterInfo();
unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
unsigned SaveExecOpc =
ST.isWave32() ? AMDGPU::S_AND_SAVEEXEC_B32 : AMDGPU::S_AND_SAVEEXEC_B64;
unsigned XorTermOpc =
ST.isWave32() ? AMDGPU::S_XOR_B32_term : AMDGPU::S_XOR_B64_term;
unsigned AndOpc =
ST.isWave32() ? AMDGPU::S_AND_B32 : AMDGPU::S_AND_B64;
const auto *BoolXExecRC = TRI->getRegClass(AMDGPU::SReg_1_XEXECRegClassID);
MachineBasicBlock::iterator I = LoopBB.begin(); MachineBasicBlock::iterator I = LoopBB.begin();
unsigned VRsrc = Rsrc.getReg(); unsigned VRsrc = Rsrc.getReg();
unsigned VRsrcUndef = getUndefRegState(Rsrc.isUndef()); unsigned VRsrcUndef = getUndefRegState(Rsrc.isUndef());
unsigned SaveExec = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned SaveExec = MRI.createVirtualRegister(BoolXExecRC);
unsigned CondReg0 = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned CondReg0 = MRI.createVirtualRegister(BoolXExecRC);
unsigned CondReg1 = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned CondReg1 = MRI.createVirtualRegister(BoolXExecRC);
unsigned AndCond = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned AndCond = MRI.createVirtualRegister(BoolXExecRC);
unsigned SRsrcSub0 = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass); unsigned SRsrcSub0 = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
unsigned SRsrcSub1 = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass); unsigned SRsrcSub1 = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
unsigned SRsrcSub2 = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass); unsigned SRsrcSub2 = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
unsigned SRsrcSub3 = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass); unsigned SRsrcSub3 = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
unsigned SRsrc = MRI.createVirtualRegister(&AMDGPU::SReg_128RegClass); unsigned SRsrc = MRI.createVirtualRegister(&AMDGPU::SReg_128RegClass);
// Beginning of the loop, read the next Rsrc variant. // Beginning of the loop, read the next Rsrc variant.
BuildMI(LoopBB, I, DL, TII.get(AMDGPU::V_READFIRSTLANE_B32), SRsrcSub0) BuildMI(LoopBB, I, DL, TII.get(AMDGPU::V_READFIRSTLANE_B32), SRsrcSub0)
Show All 21 LinesemitLoadSRsrcFromVGPRLoop(const SIInstrInfo &TII, MachineRegisterInfo &MRI,
// Identify all lanes with identical Rsrc operands in their VGPRs. // Identify all lanes with identical Rsrc operands in their VGPRs.
BuildMI(LoopBB, I, DL, TII.get(AMDGPU::V_CMP_EQ_U64_e64), CondReg0) BuildMI(LoopBB, I, DL, TII.get(AMDGPU::V_CMP_EQ_U64_e64), CondReg0)
.addReg(SRsrc, 0, AMDGPU::sub0_sub1) .addReg(SRsrc, 0, AMDGPU::sub0_sub1)
.addReg(VRsrc, 0, AMDGPU::sub0_sub1); .addReg(VRsrc, 0, AMDGPU::sub0_sub1);
BuildMI(LoopBB, I, DL, TII.get(AMDGPU::V_CMP_EQ_U64_e64), CondReg1) BuildMI(LoopBB, I, DL, TII.get(AMDGPU::V_CMP_EQ_U64_e64), CondReg1)
.addReg(SRsrc, 0, AMDGPU::sub2_sub3) .addReg(SRsrc, 0, AMDGPU::sub2_sub3)
.addReg(VRsrc, 0, AMDGPU::sub2_sub3); .addReg(VRsrc, 0, AMDGPU::sub2_sub3);
BuildMI(LoopBB, I, DL, TII.get(AMDGPU::S_AND_B64), AndCond) BuildMI(LoopBB, I, DL, TII.get(AndOpc), AndCond)
.addReg(CondReg0) .addReg(CondReg0)
.addReg(CondReg1); .addReg(CondReg1);
MRI.setSimpleHint(SaveExec, AndCond); MRI.setSimpleHint(SaveExec, AndCond);
// Update EXEC to matching lanes, saving original to SaveExec. // Update EXEC to matching lanes, saving original to SaveExec.
BuildMI(LoopBB, I, DL, TII.get(AMDGPU::S_AND_SAVEEXEC_B64), SaveExec) BuildMI(LoopBB, I, DL, TII.get(SaveExecOpc), SaveExec)
.addReg(AndCond, RegState::Kill); .addReg(AndCond, RegState::Kill);
// The original instruction is here; we insert the terminators after it. // The original instruction is here; we insert the terminators after it.
I = LoopBB.end(); I = LoopBB.end();
// Update EXEC, switch all done bits to 0 and all todo bits to 1. // Update EXEC, switch all done bits to 0 and all todo bits to 1.
BuildMI(LoopBB, I, DL, TII.get(AMDGPU::S_XOR_B64_term), AMDGPU::EXEC) BuildMI(LoopBB, I, DL, TII.get(XorTermOpc), Exec)
.addReg(AMDGPU::EXEC) .addReg(Exec)
.addReg(SaveExec); .addReg(SaveExec);
BuildMI(LoopBB, I, DL, TII.get(AMDGPU::S_CBRANCH_EXECNZ)).addMBB(&LoopBB); BuildMI(LoopBB, I, DL, TII.get(AMDGPU::S_CBRANCH_EXECNZ)).addMBB(&LoopBB);
} }
// Build a waterfall loop around \p MI, replacing the VGPR \p Rsrc register // Build a waterfall loop around \p MI, replacing the VGPR \p Rsrc register
// with SGPRs by iterating over all unique values across all lanes. // with SGPRs by iterating over all unique values across all lanes.
static void loadSRsrcFromVGPR(const SIInstrInfo &TII, MachineInstr &MI, static void loadSRsrcFromVGPR(const SIInstrInfo &TII, MachineInstr &MI,
MachineOperand &Rsrc, MachineDominatorTree *MDT) { MachineOperand &Rsrc, MachineDominatorTree *MDT) {
MachineBasicBlock &MBB = *MI.getParent(); MachineBasicBlock &MBB = *MI.getParent();
MachineFunction &MF = *MBB.getParent(); MachineFunction &MF = *MBB.getParent();
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
const SIRegisterInfo *TRI = ST.getRegisterInfo();
MachineRegisterInfo &MRI = MF.getRegInfo(); MachineRegisterInfo &MRI = MF.getRegInfo();
MachineBasicBlock::iterator I(&MI); MachineBasicBlock::iterator I(&MI);
const DebugLoc &DL = MI.getDebugLoc(); const DebugLoc &DL = MI.getDebugLoc();
unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
unsigned MovExecOpc = ST.isWave32() ? AMDGPU::S_MOV_B32 : AMDGPU::S_MOV_B64;
const auto *BoolXExecRC = TRI->getRegClass(AMDGPU::SReg_1_XEXECRegClassID);
unsigned SaveExec = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); unsigned SaveExec = MRI.createVirtualRegister(BoolXExecRC);
// Save the EXEC mask // Save the EXEC mask
BuildMI(MBB, I, DL, TII.get(AMDGPU::S_MOV_B64), SaveExec) BuildMI(MBB, I, DL, TII.get(MovExecOpc), SaveExec).addReg(Exec);
.addReg(AMDGPU::EXEC);
// Killed uses in the instruction we are waterfalling around will be // Killed uses in the instruction we are waterfalling around will be
// incorrect due to the added control-flow. // incorrect due to the added control-flow.
for (auto &MO : MI.uses()) { for (auto &MO : MI.uses()) {
if (MO.isReg() && MO.isUse()) { if (MO.isReg() && MO.isUse()) {
MRI.clearKillFlags(MO.getReg()); MRI.clearKillFlags(MO.getReg());
} }
} }
Show All 32 Lines for (auto &Succ : RemainderBB->successors()) {
} }
} }
} }
emitLoadSRsrcFromVGPRLoop(TII, MRI, MBB, *LoopBB, DL, Rsrc); emitLoadSRsrcFromVGPRLoop(TII, MRI, MBB, *LoopBB, DL, Rsrc);
// Restore the EXEC mask // Restore the EXEC mask
MachineBasicBlock::iterator First = RemainderBB->begin(); MachineBasicBlock::iterator First = RemainderBB->begin();
BuildMI(*RemainderBB, First, DL, TII.get(AMDGPU::S_MOV_B64), AMDGPU::EXEC) BuildMI(*RemainderBB, First, DL, TII.get(MovExecOpc), Exec).addReg(SaveExec);
.addReg(SaveExec);
} }
// Extract pointer from Rsrc and return a zero-value Rsrc replacement. // Extract pointer from Rsrc and return a zero-value Rsrc replacement.
static std::tuple<unsigned, unsigned> static std::tuple<unsigned, unsigned>
extractRsrcPtr(const SIInstrInfo &TII, MachineInstr &MI, MachineOperand &Rsrc) { extractRsrcPtr(const SIInstrInfo &TII, MachineInstr &MI, MachineOperand &Rsrc) {
MachineBasicBlock &MBB = *MI.getParent(); MachineBasicBlock &MBB = *MI.getParent();
MachineFunction &MF = *MBB.getParent(); MachineFunction &MF = *MBB.getParent();
MachineRegisterInfo &MRI = MF.getRegInfo(); MachineRegisterInfo &MRI = MF.getRegInfo();
▲ Show 20 Lines • Show All 214 Lines • ▼ Show 20 Lines if (VAddr && AMDGPU::getIfAddr64Inst(MI.getOpcode()) != -1) {
unsigned NewVAddrHi = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass); unsigned NewVAddrHi = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass);
unsigned NewVAddr = MRI.createVirtualRegister(&AMDGPU::VReg_64RegClass); unsigned NewVAddr = MRI.createVirtualRegister(&AMDGPU::VReg_64RegClass);
unsigned RsrcPtr, NewSRsrc; unsigned RsrcPtr, NewSRsrc;
std::tie(RsrcPtr, NewSRsrc) = extractRsrcPtr(*this, MI, *Rsrc); std::tie(RsrcPtr, NewSRsrc) = extractRsrcPtr(*this, MI, *Rsrc);
// NewVaddrLo = RsrcPtr:sub0 + VAddr:sub0 // NewVaddrLo = RsrcPtr:sub0 + VAddr:sub0
DebugLoc DL = MI.getDebugLoc(); DebugLoc DL = MI.getDebugLoc();
fixImplicitOperands(*
BuildMI(MBB, MI, DL, get(AMDGPU::V_ADD_I32_e32), NewVAddrLo) BuildMI(MBB, MI, DL, get(AMDGPU::V_ADD_I32_e32), NewVAddrLo)
.addReg(RsrcPtr, 0, AMDGPU::sub0) .addReg(RsrcPtr, 0, AMDGPU::sub0)
.addReg(VAddr->getReg(), 0, AMDGPU::sub0); .addReg(VAddr->getReg(), 0, AMDGPU::sub0));
// NewVaddrHi = RsrcPtr:sub1 + VAddr:sub1 // NewVaddrHi = RsrcPtr:sub1 + VAddr:sub1
fixImplicitOperands(*
BuildMI(MBB, MI, DL, get(AMDGPU::V_ADDC_U32_e32), NewVAddrHi) BuildMI(MBB, MI, DL, get(AMDGPU::V_ADDC_U32_e32), NewVAddrHi)
.addReg(RsrcPtr, 0, AMDGPU::sub1) .addReg(RsrcPtr, 0, AMDGPU::sub1)
.addReg(VAddr->getReg(), 0, AMDGPU::sub1); .addReg(VAddr->getReg(), 0, AMDGPU::sub1));
// NewVaddr = {NewVaddrHi, NewVaddrLo} // NewVaddr = {NewVaddrHi, NewVaddrLo}
BuildMI(MBB, MI, MI.getDebugLoc(), get(AMDGPU::REG_SEQUENCE), NewVAddr) BuildMI(MBB, MI, MI.getDebugLoc(), get(AMDGPU::REG_SEQUENCE), NewVAddr)
.addReg(NewVAddrLo) .addReg(NewVAddrLo)
.addImm(AMDGPU::sub0) .addImm(AMDGPU::sub0)
.addReg(NewVAddrHi) .addReg(NewVAddrHi)
.addImm(AMDGPU::sub1); .addImm(AMDGPU::sub1);
▲ Show 20 Lines • Show All 207 Lines • ▼ Show 20 Lines while (!Worklist.empty()) {
case AMDGPU::S_ABS_I32: case AMDGPU::S_ABS_I32:
lowerScalarAbs(Worklist, Inst); lowerScalarAbs(Worklist, Inst);
Inst.eraseFromParent(); Inst.eraseFromParent();
continue; continue;
case AMDGPU::S_CBRANCH_SCC0: case AMDGPU::S_CBRANCH_SCC0:
case AMDGPU::S_CBRANCH_SCC1: case AMDGPU::S_CBRANCH_SCC1:
// Clear unused bits of vcc // Clear unused bits of vcc
if (ST.isWave32())
BuildMI(*MBB, Inst, Inst.getDebugLoc(), get(AMDGPU::S_AND_B32),
AMDGPU::VCC_LO)
.addReg(AMDGPU::EXEC_LO)
.addReg(AMDGPU::VCC_LO);
else
BuildMI(*MBB, Inst, Inst.getDebugLoc(), get(AMDGPU::S_AND_B64), BuildMI(*MBB, Inst, Inst.getDebugLoc(), get(AMDGPU::S_AND_B64),
AMDGPU::VCC) AMDGPU::VCC)
.addReg(AMDGPU::EXEC) .addReg(AMDGPU::EXEC)
.addReg(AMDGPU::VCC); .addReg(AMDGPU::VCC);
break; break;
case AMDGPU::S_BFE_U64: case AMDGPU::S_BFE_U64:
case AMDGPU::S_BFM_B64: case AMDGPU::S_BFM_B64:
llvm_unreachable("Moving this op to VALU not implemented"); llvm_unreachable("Moving this op to VALU not implemented");
case AMDGPU::S_PACK_LL_B32_B16: case AMDGPU::S_PACK_LL_B32_B16:
case AMDGPU::S_PACK_LH_B32_B16: case AMDGPU::S_PACK_LH_B32_B16:
▲ Show 20 Lines • Show All 61 Lines • ▼ Show 20 Lines while (!Worklist.empty()) {
} else if (Opcode == AMDGPU::S_BCNT1_I32_B32) { } else if (Opcode == AMDGPU::S_BCNT1_I32_B32) {
// The VALU version adds the second operand to the result, so insert an // The VALU version adds the second operand to the result, so insert an
// extra 0 operand. // extra 0 operand.
Inst.addOperand(MachineOperand::CreateImm(0)); Inst.addOperand(MachineOperand::CreateImm(0));
} }
Inst.addImplicitDefUseOperands(*Inst.getParent()->getParent()); Inst.addImplicitDefUseOperands(*Inst.getParent()->getParent());
fixImplicitOperands(Inst);
if (Opcode == AMDGPU::S_BFE_I32 || Opcode == AMDGPU::S_BFE_U32) { if (Opcode == AMDGPU::S_BFE_I32 || Opcode == AMDGPU::S_BFE_U32) {
const MachineOperand &OffsetWidthOp = Inst.getOperand(2); const MachineOperand &OffsetWidthOp = Inst.getOperand(2);
// If we need to move this to VGPRs, we need to unpack the second operand // If we need to move this to VGPRs, we need to unpack the second operand
// back into the 2 separate ones for bit offset and width. // back into the 2 separate ones for bit offset and width.
assert(OffsetWidthOp.isImm() && assert(OffsetWidthOp.isImm() &&
"Scalar BFE is only implemented for constant width and offset"); "Scalar BFE is only implemented for constant width and offset");
uint32_t Imm = OffsetWidthOp.getImm(); uint32_t Imm = OffsetWidthOp.getImm();
▲ Show 20 Lines • Show All 297 Lines • ▼ Show 20 Lines
void SIInstrInfo::splitScalar64BitAddSub(SetVectorType &Worklist, void SIInstrInfo::splitScalar64BitAddSub(SetVectorType &Worklist,
MachineInstr &Inst, MachineInstr &Inst,
MachineDominatorTree *MDT) const { MachineDominatorTree *MDT) const {
bool IsAdd = (Inst.getOpcode() == AMDGPU::S_ADD_U64_PSEUDO); bool IsAdd = (Inst.getOpcode() == AMDGPU::S_ADD_U64_PSEUDO);
MachineBasicBlock &MBB = *Inst.getParent(); MachineBasicBlock &MBB = *Inst.getParent();
MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo(); MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
const auto *CarryRC = RI.getRegClass(AMDGPU::SReg_1_XEXECRegClassID);
unsigned FullDestReg = MRI.createVirtualRegister(&AMDGPU::VReg_64RegClass); unsigned FullDestReg = MRI.createVirtualRegister(&AMDGPU::VReg_64RegClass);
unsigned DestSub0 = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass); unsigned DestSub0 = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass);
unsigned DestSub1 = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass); unsigned DestSub1 = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass);
unsigned CarryReg = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); unsigned CarryReg = MRI.createVirtualRegister(CarryRC);
unsigned DeadCarryReg = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); unsigned DeadCarryReg = MRI.createVirtualRegister(CarryRC);
MachineOperand &Dest = Inst.getOperand(0); MachineOperand &Dest = Inst.getOperand(0);
MachineOperand &Src0 = Inst.getOperand(1); MachineOperand &Src0 = Inst.getOperand(1);
MachineOperand &Src1 = Inst.getOperand(2); MachineOperand &Src1 = Inst.getOperand(2);
const DebugLoc &DL = Inst.getDebugLoc(); const DebugLoc &DL = Inst.getDebugLoc();
MachineBasicBlock::iterator MII = Inst; MachineBasicBlock::iterator MII = Inst;
const TargetRegisterClass *Src0RC = MRI.getRegClass(Src0.getReg()); const TargetRegisterClass *Src0RC = MRI.getRegClass(Src0.getReg());
▲ Show 20 Lines • Show All 681 Lines • ▼ Show 20 Linesvoid SIInstrInfo::convertNonUniformIfRegion(MachineBasicBlock *IfEntry,
MachineBasicBlock::iterator TI = IfEntry->getFirstTerminator(); MachineBasicBlock::iterator TI = IfEntry->getFirstTerminator();
assert(TI != IfEntry->end()); assert(TI != IfEntry->end());
MachineInstr *Branch = &(*TI); MachineInstr *Branch = &(*TI);
MachineFunction *MF = IfEntry->getParent(); MachineFunction *MF = IfEntry->getParent();
MachineRegisterInfo &MRI = IfEntry->getParent()->getRegInfo(); MachineRegisterInfo &MRI = IfEntry->getParent()->getRegInfo();
if (Branch->getOpcode() == AMDGPU::SI_NON_UNIFORM_BRCOND_PSEUDO) { if (Branch->getOpcode() == AMDGPU::SI_NON_UNIFORM_BRCOND_PSEUDO) {
unsigned DstReg = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned DstReg = MRI.createVirtualRegister(RI.getBoolRC());
MachineInstr *SIIF = MachineInstr *SIIF =
BuildMI(*MF, Branch->getDebugLoc(), get(AMDGPU::SI_IF), DstReg) BuildMI(*MF, Branch->getDebugLoc(), get(AMDGPU::SI_IF), DstReg)
.add(Branch->getOperand(0)) .add(Branch->getOperand(0))
.add(Branch->getOperand(1)); .add(Branch->getOperand(1));
MachineInstr *SIEND = MachineInstr *SIEND =
BuildMI(*MF, Branch->getDebugLoc(), get(AMDGPU::SI_END_CF)) BuildMI(*MF, Branch->getDebugLoc(), get(AMDGPU::SI_END_CF))
.addReg(DstReg); .addReg(DstReg);
Show All 10 Linesvoid SIInstrInfo::convertNonUniformLoopRegion(
assert(TI != LoopEnd->end()); assert(TI != LoopEnd->end());
MachineInstr *Branch = &(*TI); MachineInstr *Branch = &(*TI);
MachineFunction *MF = LoopEnd->getParent(); MachineFunction *MF = LoopEnd->getParent();
MachineRegisterInfo &MRI = LoopEnd->getParent()->getRegInfo(); MachineRegisterInfo &MRI = LoopEnd->getParent()->getRegInfo();
if (Branch->getOpcode() == AMDGPU::SI_NON_UNIFORM_BRCOND_PSEUDO) { if (Branch->getOpcode() == AMDGPU::SI_NON_UNIFORM_BRCOND_PSEUDO) {
unsigned DstReg = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned DstReg = MRI.createVirtualRegister(RI.getBoolRC());
unsigned BackEdgeReg = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned BackEdgeReg = MRI.createVirtualRegister(RI.getBoolRC());
MachineInstrBuilder HeaderPHIBuilder = MachineInstrBuilder HeaderPHIBuilder =
BuildMI(*(MF), Branch->getDebugLoc(), get(TargetOpcode::PHI), DstReg); BuildMI(*(MF), Branch->getDebugLoc(), get(TargetOpcode::PHI), DstReg);
for (MachineBasicBlock::pred_iterator PI = LoopEntry->pred_begin(), for (MachineBasicBlock::pred_iterator PI = LoopEntry->pred_begin(),
E = LoopEntry->pred_end(); E = LoopEntry->pred_end();
PI != E; ++PI) { PI != E; ++PI) {
if (*PI == LoopEnd) { if (*PI == LoopEnd) {
HeaderPHIBuilder.addReg(BackEdgeReg); HeaderPHIBuilder.addReg(BackEdgeReg);
} else { } else {
MachineBasicBlock *PMBB = *PI; MachineBasicBlock *PMBB = *PI;
unsigned ZeroReg = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned ZeroReg = MRI.createVirtualRegister(RI.getBoolRC());
materializeImmediate(*PMBB, PMBB->getFirstTerminator(), DebugLoc(), materializeImmediate(*PMBB, PMBB->getFirstTerminator(), DebugLoc(),
ZeroReg, 0); ZeroReg, 0);
HeaderPHIBuilder.addReg(ZeroReg); HeaderPHIBuilder.addReg(ZeroReg);
} }
HeaderPHIBuilder.addMBB(*PI); HeaderPHIBuilder.addMBB(*PI);
} }
MachineInstr *HeaderPhi = HeaderPHIBuilder; MachineInstr *HeaderPhi = HeaderPHIBuilder;
MachineInstr *SIIFBREAK = BuildMI(*(MF), Branch->getDebugLoc(), MachineInstr *SIIFBREAK = BuildMI(*(MF), Branch->getDebugLoc(),
▲ Show 20 Lines • Show All 65 Lines • ▼ Show 20 Lines
SIInstrInfo::getAddNoCarry(MachineBasicBlock &MBB, SIInstrInfo::getAddNoCarry(MachineBasicBlock &MBB,
MachineBasicBlock::iterator I, MachineBasicBlock::iterator I,
const DebugLoc &DL, const DebugLoc &DL,
unsigned DestReg) const { unsigned DestReg) const {
if (ST.hasAddNoCarry()) if (ST.hasAddNoCarry())
return BuildMI(MBB, I, DL, get(AMDGPU::V_ADD_U32_e64), DestReg); return BuildMI(MBB, I, DL, get(AMDGPU::V_ADD_U32_e64), DestReg);
MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo(); MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
unsigned UnusedCarry = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned UnusedCarry = MRI.createVirtualRegister(RI.getBoolRC());
MRI.setRegAllocationHint(UnusedCarry, 0, AMDGPU::VCC); MRI.setRegAllocationHint(UnusedCarry, 0, RI.getVCC());
return BuildMI(MBB, I, DL, get(AMDGPU::V_ADD_I32_e64), DestReg) return BuildMI(MBB, I, DL, get(AMDGPU::V_ADD_I32_e64), DestReg)
.addReg(UnusedCarry, RegState::Define | RegState::Dead); .addReg(UnusedCarry, RegState::Define | RegState::Dead);
} }
bool SIInstrInfo::isKillTerminator(unsigned Opcode) { bool SIInstrInfo::isKillTerminator(unsigned Opcode) {
switch (Opcode) { switch (Opcode) {
case AMDGPU::SI_KILL_F32_COND_IMM_TERMINATOR: case AMDGPU::SI_KILL_F32_COND_IMM_TERMINATOR:
Show All 10 Lines case AMDGPU::SI_KILL_F32_COND_IMM_PSEUDO:
return get(AMDGPU::SI_KILL_F32_COND_IMM_TERMINATOR); return get(AMDGPU::SI_KILL_F32_COND_IMM_TERMINATOR);
case AMDGPU::SI_KILL_I1_PSEUDO: case AMDGPU::SI_KILL_I1_PSEUDO:
return get(AMDGPU::SI_KILL_I1_TERMINATOR); return get(AMDGPU::SI_KILL_I1_TERMINATOR);
default: default:
llvm_unreachable("invalid opcode, expected SI_KILL_*_PSEUDO"); llvm_unreachable("invalid opcode, expected SI_KILL_*_PSEUDO");
} }
} }
void SIInstrInfo::fixImplicitOperands(MachineInstr &MI) const {
MachineBasicBlock *MBB = MI.getParent();
MachineFunction *MF = MBB->getParent();
const GCNSubtarget &ST = MF->getSubtarget<GCNSubtarget>();
if (!ST.isWave32())
return;
for (auto &Op : MI.implicit_operands()) {
if (Op.isReg() && Op.getReg() == AMDGPU::VCC)
Op.setReg(AMDGPU::VCC_LO);
}
}
bool SIInstrInfo::isBufferSMRD(const MachineInstr &MI) const { bool SIInstrInfo::isBufferSMRD(const MachineInstr &MI) const {
if (!isSMRD(MI)) if (!isSMRD(MI))
return false; return false;
// Check that it is using a buffer resource. // Check that it is using a buffer resource.
int Idx = AMDGPU::getNamedOperandIdx(MI.getOpcode(), AMDGPU::OpName::sbase); int Idx = AMDGPU::getNamedOperandIdx(MI.getOpcode(), AMDGPU::OpName::sbase);
if (Idx == -1) // e.g. s_memtime if (Idx == -1) // e.g. s_memtime
return false; return false;
▲ Show 20 Lines • Show All 180 Lines • Show Last 20 Lines

lib/Target/AMDGPU/SIInstrInfo.td

//===-- SIInstrInfo.td - SI Instruction Infos -------------*- tablegen -*--===// //===-- SIInstrInfo.td - SI Instruction Infos -------------*- tablegen -*--===//
// //
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information. // See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
// //
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
def isWave32 : Predicate<"Subtarget->getWavefrontSize() == 32">,
AssemblerPredicate <"FeatureWavefrontSize32">;
def isWave64 : Predicate<"Subtarget->getWavefrontSize() == 64">,
AssemblerPredicate <"FeatureWavefrontSize64">;
def DisableInst : Predicate <"false">, AssemblerPredicate<"FeatureDisable">; def DisableInst : Predicate <"false">, AssemblerPredicate<"FeatureDisable">;
class GCNPredicateControl : PredicateControl { class GCNPredicateControl : PredicateControl {
Predicate SIAssemblerPredicate = isGFX6GFX7; Predicate SIAssemblerPredicate = isGFX6GFX7;
Predicate VIAssemblerPredicate = isGFX8GFX9; Predicate VIAssemblerPredicate = isGFX8GFX9;
} }
// Execpt for the NONE field, this must be kept in sync with the // Execpt for the NONE field, this must be kept in sync with the
▲ Show 20 Lines • Show All 700 Lines • ▼ Show 20 Lines
def WAIT_FLAG : Operand <i32> { def WAIT_FLAG : Operand <i32> {
let ParserMatchClass = SWaitMatchClass; let ParserMatchClass = SWaitMatchClass;
let PrintMethod = "printWaitFlag"; let PrintMethod = "printWaitFlag";
} }
include "SIInstrFormats.td" include "SIInstrFormats.td"
include "VIInstrFormats.td" include "VIInstrFormats.td"
def BoolReg : AsmOperandClass {
let Name = "BoolReg";
let ParserMethod = "parseBoolReg";
let RenderMethod = "addRegOperands";
}
class BoolRC : RegisterOperand<SReg_1> {
let ParserMatchClass = BoolReg;
let DecoderMethod = "decodeBoolReg";
}
def SSrc_i1 : RegisterOperand<SReg_1_XEXEC> {
let ParserMatchClass = BoolReg;
let DecoderMethod = "decodeBoolReg";
}
def VOPDstS64orS32 : BoolRC {
let PrintMethod = "printVOPDst";
}
// SCSrc_i1 is the operand for pseudo instructions only.
// Boolean immeadiates shall not be exposed to codegen instructions.
def SCSrc_i1 : RegisterOperand<SReg_1_XEXEC> {
let OperandNamespace = "AMDGPU";
let OperandType = "OPERAND_REG_IMM_INT32";
let ParserMatchClass = BoolReg;
let DecoderMethod = "decodeBoolReg";
}
// ===----------------------------------------------------------------------===// // ===----------------------------------------------------------------------===//
// ExpSrc* Special cases for exp src operands which are printed as // ExpSrc* Special cases for exp src operands which are printed as
// "off" depending on en operand. // "off" depending on en operand.
// ===----------------------------------------------------------------------===// // ===----------------------------------------------------------------------===//
def ExpSrc0 : RegisterOperand<VGPR_32> { def ExpSrc0 : RegisterOperand<VGPR_32> {
let PrintMethod = "printExpSrc0"; let PrintMethod = "printExpSrc0";
let ParserMatchClass = VReg32OrOffClass; let ParserMatchClass = VReg32OrOffClass;
Show All 22 Linesclass SDWASrc<ValueType vt> : RegisterOperand<VS_32> {
let EncoderMethod = "getSDWASrcEncoding"; let EncoderMethod = "getSDWASrcEncoding";
} }
def SDWASrc_i32 : SDWASrc<i32>; def SDWASrc_i32 : SDWASrc<i32>;
def SDWASrc_i16 : SDWASrc<i16>; def SDWASrc_i16 : SDWASrc<i16>;
def SDWASrc_f32 : SDWASrc<f32>; def SDWASrc_f32 : SDWASrc<f32>;
def SDWASrc_f16 : SDWASrc<f16>; def SDWASrc_f16 : SDWASrc<f16>;
def SDWAVopcDst : VOPDstOperand<SReg_64> { def SDWAVopcDst : BoolRC {
let OperandNamespace = "AMDGPU"; let OperandNamespace = "AMDGPU";
let OperandType = "OPERAND_SDWA_VOPC_DST"; let OperandType = "OPERAND_SDWA_VOPC_DST";
let EncoderMethod = "getSDWAVopcDstEncoding"; let EncoderMethod = "getSDWAVopcDstEncoding";
let DecoderMethod = "decodeSDWAVopcDst"; let DecoderMethod = "decodeSDWAVopcDst";
let PrintMethod = "printVOPDst";
} }
class NamedMatchClass<string CName, bit Optional = 1> : AsmOperandClass { class NamedMatchClass<string CName, bit Optional = 1> : AsmOperandClass {
let Name = "Imm"#CName; let Name = "Imm"#CName;
let PredicateMethod = "is"#CName; let PredicateMethod = "is"#CName;
let ParserMethod = !if(Optional, "parseOptionalOperand", "parse"#CName); let ParserMethod = !if(Optional, "parseOptionalOperand", "parse"#CName);
let RenderMethod = "addImmOperands"; let RenderMethod = "addImmOperands";
let IsOptional = Optional; let IsOptional = Optional;
▲ Show 20 Lines • Show All 115 Lines • ▼ Show 20 Lines
def KImmFP32MatchClass : KImmMatchClass<32>; def KImmFP32MatchClass : KImmMatchClass<32>;
def f32kimm : kimmOperand<i32>; def f32kimm : kimmOperand<i32>;
// 32-bit VALU immediate operand with a 16-bit value that uses the // 32-bit VALU immediate operand with a 16-bit value that uses the
// constant bus. // constant bus.
def KImmFP16MatchClass : KImmMatchClass<16>; def KImmFP16MatchClass : KImmMatchClass<16>;
def f16kimm : kimmOperand<i16>; def f16kimm : kimmOperand<i16>;
def VOPDstS64 : VOPDstOperand <SReg_64> {
let PrintMethod = "printVOPDst";
}
class FPInputModsMatchClass <int opSize> : AsmOperandClass { class FPInputModsMatchClass <int opSize> : AsmOperandClass {
let Name = "RegOrImmWithFP"#opSize#"InputMods"; let Name = "RegOrImmWithFP"#opSize#"InputMods";
let ParserMethod = "parseRegOrImmWithFPInputMods"; let ParserMethod = "parseRegOrImmWithFPInputMods";
let PredicateMethod = "isRegOrImmWithFP"#opSize#"InputMods"; let PredicateMethod = "isRegOrImmWithFP"#opSize#"InputMods";
} }
def FP16InputModsMatchClass : FPInputModsMatchClass<16>; def FP16InputModsMatchClass : FPInputModsMatchClass<16>;
def FP32InputModsMatchClass : FPInputModsMatchClass<32>; def FP32InputModsMatchClass : FPInputModsMatchClass<32>;
▲ Show 20 Lines • Show All 276 Lines • ▼ Show 20 Lines
// Returns the register class to use for the destination of VOP[123C] // Returns the register class to use for the destination of VOP[123C]
// instructions for the given VT. // instructions for the given VT.
class getVALUDstForVT<ValueType VT> { class getVALUDstForVT<ValueType VT> {
RegisterOperand ret = !if(!eq(VT.Size, 32), VOPDstOperand<VGPR_32>, RegisterOperand ret = !if(!eq(VT.Size, 32), VOPDstOperand<VGPR_32>,
!if(!eq(VT.Size, 128), VOPDstOperand<VReg_128>, !if(!eq(VT.Size, 128), VOPDstOperand<VReg_128>,
!if(!eq(VT.Size, 64), VOPDstOperand<VReg_64>, !if(!eq(VT.Size, 64), VOPDstOperand<VReg_64>,
!if(!eq(VT.Size, 16), VOPDstOperand<VGPR_32>, !if(!eq(VT.Size, 16), VOPDstOperand<VGPR_32>,
VOPDstOperand<SReg_64>)))); // else VT == i1 VOPDstS64orS32)))); // else VT == i1
} }
// Returns the register class to use for the destination of VOP[12C] // Returns the register class to use for the destination of VOP[12C]
// instructions with SDWA extension // instructions with SDWA extension
class getSDWADstForVT<ValueType VT> { class getSDWADstForVT<ValueType VT> {
RegisterOperand ret = !if(!eq(VT.Size, 1), RegisterOperand ret = !if(!eq(VT.Size, 1),
SDWAVopcDst, // VOPC SDWAVopcDst, // VOPC
VOPDstOperand<VGPR_32>); // VOP1/2 32-bit dst VOPDstOperand<VGPR_32>); // VOP1/2 32-bit dst
▲ Show 20 Lines • Show All 59 Lines • ▼ Show 20 Linesclass getVOP3SrcForVT<ValueType VT> {
RegisterOperand ret = RegisterOperand ret =
!if(!eq(VT.Size, 128), !if(!eq(VT.Size, 128),
VSrc_128, VSrc_128,
!if(!eq(VT.Size, 64), !if(!eq(VT.Size, 64),
!if(isFP, !if(isFP,
VSrc_f64, VSrc_f64,
VSrc_b64), VSrc_b64),
!if(!eq(VT.Value, i1.Value), !if(!eq(VT.Value, i1.Value),
SCSrc_i1, SSrc_i1,
!if(isFP, !if(isFP,
!if(!eq(VT.Value, f16.Value), !if(!eq(VT.Value, f16.Value),
VSrc_f16, VSrc_f16,
!if(!eq(VT.Value, v2f16.Value), !if(!eq(VT.Value, v2f16.Value),
VSrc_v2f16, VSrc_v2f16,
VSrc_f32 VSrc_f32
) )
), ),
▲ Show 20 Lines • Show All 989 Lines • Show Last 20 Lines

lib/Target/AMDGPU/SIInstructions.td

Show First 20 Lines • Show All 115 Lines • ▼ Show 20 Lines
// the instruction that defines $src0 (which is run in WWM) doesn't // the instruction that defines $src0 (which is run in WWM) doesn't
// accidentally clobber inactive channels of $vdst. // accidentally clobber inactive channels of $vdst.
let Constraints = "@earlyclobber $vdst" in { let Constraints = "@earlyclobber $vdst" in {
def WWM : PseudoInstSI <(outs unknown:$vdst), (ins unknown:$src0)>; def WWM : PseudoInstSI <(outs unknown:$vdst), (ins unknown:$src0)>;
} }
} // End let hasSideEffects = 0, mayLoad = 0, mayStore = 0, Uses = [EXEC] } // End let hasSideEffects = 0, mayLoad = 0, mayStore = 0, Uses = [EXEC]
def ENTER_WWM : SPseudoInstSI <(outs SReg_64:$sdst), (ins i64imm:$src0)> { def ENTER_WWM : SPseudoInstSI <(outs SReg_1:$sdst), (ins i64imm:$src0)> {
let Defs = [EXEC]; let Defs = [EXEC];
let hasSideEffects = 0; let hasSideEffects = 0;
let mayLoad = 0; let mayLoad = 0;
let mayStore = 0; let mayStore = 0;
} }
def EXIT_WWM : SPseudoInstSI <(outs SReg_64:$sdst), (ins SReg_64:$src0)> { def EXIT_WWM : SPseudoInstSI <(outs SReg_1:$sdst), (ins SReg_1:$src0)> {
let hasSideEffects = 0; let hasSideEffects = 0;
let mayLoad = 0; let mayLoad = 0;
let mayStore = 0; let mayStore = 0;
} }
// Invert the exec mask and overwrite the inactive lanes of dst with inactive, // Invert the exec mask and overwrite the inactive lanes of dst with inactive,
// restoring it after we're done. // restoring it after we're done.
def V_SET_INACTIVE_B32 : VPseudoInstSI <(outs VGPR_32:$vdst), def V_SET_INACTIVE_B32 : VPseudoInstSI <(outs VGPR_32:$vdst),
Show All 16 Lines
>; >;
def S_SUB_U64_PSEUDO : SPseudoInstSI < def S_SUB_U64_PSEUDO : SPseudoInstSI <
(outs SReg_64:$vdst), (ins SSrc_b64:$src0, SSrc_b64:$src1), (outs SReg_64:$vdst), (ins SSrc_b64:$src0, SSrc_b64:$src1),
[(set SReg_64:$vdst, (sub i64:$src0, i64:$src1))] [(set SReg_64:$vdst, (sub i64:$src0, i64:$src1))]
>; >;
def S_ADD_U64_CO_PSEUDO : SPseudoInstSI < def S_ADD_U64_CO_PSEUDO : SPseudoInstSI <
(outs SReg_64:$vdst, VOPDstS64:$sdst), (ins SSrc_b64:$src0, SSrc_b64:$src1) (outs SReg_64:$vdst, VOPDstS64orS32:$sdst), (ins SSrc_b64:$src0, SSrc_b64:$src1)
>; >;
def S_SUB_U64_CO_PSEUDO : SPseudoInstSI < def S_SUB_U64_CO_PSEUDO : SPseudoInstSI <
(outs SReg_64:$vdst, VOPDstS64:$sdst), (ins SSrc_b64:$src0, SSrc_b64:$src1) (outs SReg_64:$vdst, VOPDstS64orS32:$sdst), (ins SSrc_b64:$src0, SSrc_b64:$src1)
>; >;
def S_ADDC_U64_PSEUDO : SPseudoInstSI <(outs SReg_64:$vdst, SReg_1:$sdst),
(ins SSrc_b64:$src0, SSrc_b64:$src1)>;
def S_SUBC_U64_PSEUDO : SPseudoInstSI <(outs SReg_64:$vdst, SReg_1:$sdst),
(ins SSrc_b64:$src0, SSrc_b64:$src1)>;
} // End usesCustomInserter = 1, Defs = [SCC] } // End usesCustomInserter = 1, Defs = [SCC]
let usesCustomInserter = 1 in { let usesCustomInserter = 1 in {
def GET_GROUPSTATICSIZE : SPseudoInstSI <(outs SReg_32:$sdst), (ins), def GET_GROUPSTATICSIZE : SPseudoInstSI <(outs SReg_32:$sdst), (ins),
[(set SReg_32:$sdst, (int_amdgcn_groupstaticsize))]>; [(set SReg_32:$sdst, (int_amdgcn_groupstaticsize))]>;
} // End let usesCustomInserter = 1, SALU = 1 } // End let usesCustomInserter = 1, SALU = 1
// Wrap an instruction by duplicating it, except for setting isTerminator. // Wrap an instruction by duplicating it, except for setting isTerminator.
class WrapTerminatorInst<SOP_Pseudo base_inst> : SPseudoInstSI< class WrapTerminatorInst<SOP_Pseudo base_inst> : SPseudoInstSI<
base_inst.OutOperandList, base_inst.OutOperandList,
base_inst.InOperandList> { base_inst.InOperandList> {
let Uses = base_inst.Uses; let Uses = base_inst.Uses;
let Defs = base_inst.Defs; let Defs = base_inst.Defs;
let isTerminator = 1; let isTerminator = 1;
let isAsCheapAsAMove = base_inst.isAsCheapAsAMove; let isAsCheapAsAMove = base_inst.isAsCheapAsAMove;
let hasSideEffects = base_inst.hasSideEffects; let hasSideEffects = base_inst.hasSideEffects;
let UseNamedOperandTable = base_inst.UseNamedOperandTable; let UseNamedOperandTable = base_inst.UseNamedOperandTable;
let CodeSize = base_inst.CodeSize; let CodeSize = base_inst.CodeSize;
} }
let WaveSizePredicate = isWave64 in {
def S_MOV_B64_term : WrapTerminatorInst<S_MOV_B64>; def S_MOV_B64_term : WrapTerminatorInst<S_MOV_B64>;
def S_XOR_B64_term : WrapTerminatorInst<S_XOR_B64>; def S_XOR_B64_term : WrapTerminatorInst<S_XOR_B64>;
def S_ANDN2_B64_term : WrapTerminatorInst<S_ANDN2_B64>; def S_ANDN2_B64_term : WrapTerminatorInst<S_ANDN2_B64>;
}
let WaveSizePredicate = isWave32 in {
def S_MOV_B32_term : WrapTerminatorInst<S_MOV_B32>;
def S_XOR_B32_term : WrapTerminatorInst<S_XOR_B32>;
def S_OR_B32_term : WrapTerminatorInst<S_OR_B32>;
def S_ANDN2_B32_term : WrapTerminatorInst<S_ANDN2_B32>;
}
def WAVE_BARRIER : SPseudoInstSI<(outs), (ins), def WAVE_BARRIER : SPseudoInstSI<(outs), (ins),
[(int_amdgcn_wave_barrier)]> { [(int_amdgcn_wave_barrier)]> {
let SchedRW = []; let SchedRW = [];
let hasNoSchedulingInfo = 1; let hasNoSchedulingInfo = 1;
let hasSideEffects = 1; let hasSideEffects = 1;
let mayLoad = 1; let mayLoad = 1;
let mayStore = 1; let mayStore = 1;
Show All 18 Linesdef SI_MASK_BRANCH : VPseudoInstSI <
let Size = 0; let Size = 0;
} }
let isTerminator = 1 in { let isTerminator = 1 in {
let OtherPredicates = [EnableLateCFGStructurize] in { let OtherPredicates = [EnableLateCFGStructurize] in {
def SI_NON_UNIFORM_BRCOND_PSEUDO : CFPseudoInstSI < def SI_NON_UNIFORM_BRCOND_PSEUDO : CFPseudoInstSI <
(outs), (outs),
(ins SReg_64:$vcc, brtarget:$target), (ins SReg_1:$vcc, brtarget:$target),
[(brcond i1:$vcc, bb:$target)]> { [(brcond i1:$vcc, bb:$target)]> {
let Size = 12; let Size = 12;
} }
} }
def SI_IF: CFPseudoInstSI < def SI_IF: CFPseudoInstSI <
(outs SReg_64:$dst), (ins SReg_64:$vcc, brtarget:$target), (outs SReg_1:$dst), (ins SReg_1:$vcc, brtarget:$target),
[(set i64:$dst, (AMDGPUif i1:$vcc, bb:$target))], 1, 1> { [(set i1:$dst, (AMDGPUif i1:$vcc, bb:$target))], 1, 1> {
let Constraints = ""; let Constraints = "";
let Size = 12; let Size = 12;
let hasSideEffects = 1; let hasSideEffects = 1;
} }
def SI_ELSE : CFPseudoInstSI < def SI_ELSE : CFPseudoInstSI <
(outs SReg_64:$dst), (outs SReg_1:$dst),
(ins SReg_64:$src, brtarget:$target, i1imm:$execfix), [], 1, 1> { (ins SReg_1:$src, brtarget:$target, i1imm:$execfix), [], 1, 1> {
let Size = 12; let Size = 12;
let hasSideEffects = 1; let hasSideEffects = 1;
} }
def SI_LOOP : CFPseudoInstSI < def SI_LOOP : CFPseudoInstSI <
(outs), (ins SReg_64:$saved, brtarget:$target), (outs), (ins SReg_1:$saved, brtarget:$target),
[(AMDGPUloop i64:$saved, bb:$target)], 1, 1> { [(AMDGPUloop i1:$saved, bb:$target)], 1, 1> {
let Size = 8; let Size = 8;
let isBranch = 1; let isBranch = 1;
let hasSideEffects = 1; let hasSideEffects = 1;
} }
} // End isTerminator = 1 } // End isTerminator = 1
def SI_END_CF : CFPseudoInstSI < def SI_END_CF : CFPseudoInstSI <
(outs), (ins SReg_64:$saved), (outs), (ins SReg_1:$saved), [], 1, 1> {
[(int_amdgcn_end_cf i64:$saved)], 1, 1> {
let Size = 4; let Size = 4;
let isAsCheapAsAMove = 1; let isAsCheapAsAMove = 1;
let isReMaterializable = 1; let isReMaterializable = 1;
let hasSideEffects = 1; let hasSideEffects = 1;
let mayLoad = 1; // FIXME: Should not need memory flags let mayLoad = 1; // FIXME: Should not need memory flags
let mayStore = 1; let mayStore = 1;
} }
def SI_IF_BREAK : CFPseudoInstSI < def SI_IF_BREAK : CFPseudoInstSI <
(outs SReg_64:$dst), (ins SReg_64:$vcc, SReg_64:$src), (outs SReg_1:$dst), (ins SReg_1:$vcc, SReg_1:$src), []> {
[(set i64:$dst, (int_amdgcn_if_break i1:$vcc, i64:$src))]> {
let Size = 4; let Size = 4;
let isAsCheapAsAMove = 1; let isAsCheapAsAMove = 1;
let isReMaterializable = 1; let isReMaterializable = 1;
} }
let Uses = [EXEC] in { let Uses = [EXEC] in {
multiclass PseudoInstKill <dag ins> { multiclass PseudoInstKill <dag ins> {
Show All 9 Linesmulticlass PseudoInstKill <dag ins> {
} }
let Defs = [EXEC,VCC,SCC] in let Defs = [EXEC,VCC,SCC] in
def _TERMINATOR : SPseudoInstSI <(outs), ins> { def _TERMINATOR : SPseudoInstSI <(outs), ins> {
let isTerminator = 1; let isTerminator = 1;
} }
} }
defm SI_KILL_I1 : PseudoInstKill <(ins SSrc_b64:$src, i1imm:$killvalue)>; defm SI_KILL_I1 : PseudoInstKill <(ins SCSrc_i1:$src, i1imm:$killvalue)>;
defm SI_KILL_F32_COND_IMM : PseudoInstKill <(ins VSrc_b32:$src0, i32imm:$src1, i32imm:$cond)>; defm SI_KILL_F32_COND_IMM : PseudoInstKill <(ins VSrc_b32:$src0, i32imm:$src1, i32imm:$cond)>;
let Defs = [EXEC,VCC] in let Defs = [EXEC,VCC] in
def SI_ILLEGAL_COPY : SPseudoInstSI < def SI_ILLEGAL_COPY : SPseudoInstSI <
(outs unknown:$dst), (ins unknown:$src), (outs unknown:$dst), (ins unknown:$src),
[], " ; illegal copy $src to $dst">; [], " ; illegal copy $src to $dst">;
} // End Uses = [EXEC], Defs = [EXEC,VCC] } // End Uses = [EXEC], Defs = [EXEC,VCC]
// Branch on undef scc. Used to avoid intermediate copy from // Branch on undef scc. Used to avoid intermediate copy from
// IMPLICIT_DEF to SCC. // IMPLICIT_DEF to SCC.
def SI_BR_UNDEF : SPseudoInstSI <(outs), (ins sopp_brtarget:$simm16)> { def SI_BR_UNDEF : SPseudoInstSI <(outs), (ins sopp_brtarget:$simm16)> {
let isTerminator = 1; let isTerminator = 1;
let usesCustomInserter = 1; let usesCustomInserter = 1;
let isBranch = 1; let isBranch = 1;
} }
def SI_PS_LIVE : PseudoInstSI < def SI_PS_LIVE : PseudoInstSI <
(outs SReg_64:$dst), (ins), (outs SReg_1:$dst), (ins),
[(set i1:$dst, (int_amdgcn_ps_live))]> { [(set i1:$dst, (int_amdgcn_ps_live))]> {
let SALU = 1; let SALU = 1;
} }
def SI_MASKED_UNREACHABLE : SPseudoInstSI <(outs), (ins), def SI_MASKED_UNREACHABLE : SPseudoInstSI <(outs), (ins),
[(int_amdgcn_unreachable)], [(int_amdgcn_unreachable)],
"; divergent unreachable"> { "; divergent unreachable"> {
let Size = 0; let Size = 0;
Show All 12 Linesdef SI_INIT_M0 : SPseudoInstSI <(outs), (ins SSrc_b32:$src)> {
let isReMaterializable = 1; let isReMaterializable = 1;
} }
def SI_INIT_EXEC : SPseudoInstSI < def SI_INIT_EXEC : SPseudoInstSI <
(outs), (ins i64imm:$src), []> { (outs), (ins i64imm:$src), []> {
let Defs = [EXEC]; let Defs = [EXEC];
let usesCustomInserter = 1; let usesCustomInserter = 1;
let isAsCheapAsAMove = 1; let isAsCheapAsAMove = 1;
let WaveSizePredicate = isWave64;
}
def SI_INIT_EXEC_LO : SPseudoInstSI <
(outs), (ins i32imm:$src), []> {
let Defs = [EXEC_LO];
let usesCustomInserter = 1;
let isAsCheapAsAMove = 1;
let WaveSizePredicate = isWave32;
} }
def SI_INIT_EXEC_FROM_INPUT : SPseudoInstSI < def SI_INIT_EXEC_FROM_INPUT : SPseudoInstSI <
(outs), (ins SSrc_b32:$input, i32imm:$shift), []> { (outs), (ins SSrc_b32:$input, i32imm:$shift), []> {
let Defs = [EXEC]; let Defs = [EXEC];
let usesCustomInserter = 1; let usesCustomInserter = 1;
} }
▲ Show 20 Lines • Show All 181 Lines • ▼ Show 20 Linesdef SI_PC_ADD_REL_OFFSET : SPseudoInstSI <
[(set SReg_64:$dst, [(set SReg_64:$dst,
(i64 (SIpc_add_rel_offset (tglobaladdr:$ptr_lo), (tglobaladdr:$ptr_hi))))]> { (i64 (SIpc_add_rel_offset (tglobaladdr:$ptr_lo), (tglobaladdr:$ptr_hi))))]> {
let Defs = [SCC]; let Defs = [SCC];
} }
def : GCNPat < def : GCNPat <
(AMDGPUinit_exec i64:$src), (AMDGPUinit_exec i64:$src),
(SI_INIT_EXEC (as_i64imm $src)) (SI_INIT_EXEC (as_i64imm $src))
>; > {
let WaveSizePredicate = isWave64;
}
def : GCNPat <
(AMDGPUinit_exec i64:$src),
(SI_INIT_EXEC_LO (as_i32imm $src))
> {
let WaveSizePredicate = isWave32;
}
def : GCNPat < def : GCNPat <
(AMDGPUinit_exec_from_input i32:$input, i32:$shift), (AMDGPUinit_exec_from_input i32:$input, i32:$shift),
(SI_INIT_EXEC_FROM_INPUT (i32 $input), (as_i32imm $shift)) (SI_INIT_EXEC_FROM_INPUT (i32 $input), (as_i32imm $shift))
>; >;
def : GCNPat< def : GCNPat<
(AMDGPUtrap timm:$trapid), (AMDGPUtrap timm:$trapid),
(S_TRAP $trapid) (S_TRAP $trapid)
>; >;
def : GCNPat< def : GCNPat<
(AMDGPUelse i64:$src, bb:$target), (AMDGPUelse i1:$src, bb:$target),
(SI_ELSE $src, $target, 0) (SI_ELSE $src, $target, 0)
>; >;
def : Pat < def : Pat <
// -1.0 as i32 (LowerINTRINSIC_VOID converts all other constants to -1.0) // -1.0 as i32 (LowerINTRINSIC_VOID converts all other constants to -1.0)
(AMDGPUkill (i32 -1082130432)), (AMDGPUkill (i32 -1082130432)),
(SI_KILL_I1_PSEUDO (i1 0), 0) (SI_KILL_I1_PSEUDO (i1 0), 0)
>; >;
Show All 16 Lines
def : Pat < def : Pat <
(int_amdgcn_kill (i1 (setcc f32:$src, InlineFPImm<f32>:$imm, cond:$cond))), (int_amdgcn_kill (i1 (setcc f32:$src, InlineFPImm<f32>:$imm, cond:$cond))),
(SI_KILL_F32_COND_IMM_PSEUDO $src, (bitcast_fpimm_to_i32 $imm), (cond_as_i32imm $cond)) (SI_KILL_F32_COND_IMM_PSEUDO $src, (bitcast_fpimm_to_i32 $imm), (cond_as_i32imm $cond))
>; >;
// TODO: we could add more variants for other types of conditionals // TODO: we could add more variants for other types of conditionals
def : Pat < def : Pat <
(int_amdgcn_icmp i1:$src, (i1 0), (i32 33)), (i64 (int_amdgcn_icmp i1:$src, (i1 0), (i32 33))),
(COPY $src) // Return the SGPRs representing i1 src
>;
def : Pat <
(i32 (int_amdgcn_icmp i1:$src, (i1 0), (i32 33))),
(COPY $src) // Return the SGPRs representing i1 src (COPY $src) // Return the SGPRs representing i1 src
>; >;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// VOP1 Patterns // VOP1 Patterns
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
let OtherPredicates = [UnsafeFPMath] in { let OtherPredicates = [UnsafeFPMath] in {
▲ Show 20 Lines • Show All 539 Lines • ▼ Show 20 Lines
>; >;
// XXX - Should this use a s_cmp to set SCC? // XXX - Should this use a s_cmp to set SCC?
// Set to sign-extended 64-bit value (true = -1, false = 0) // Set to sign-extended 64-bit value (true = -1, false = 0)
def : GCNPat < def : GCNPat <
(i1 imm:$imm), (i1 imm:$imm),
(S_MOV_B64 (i64 (as_i64imm $imm))) (S_MOV_B64 (i64 (as_i64imm $imm)))
>; > {
let WaveSizePredicate = isWave64;
}
def : GCNPat <
(i1 imm:$imm),
(S_MOV_B32 (i32 (as_i32imm $imm)))
> {
let WaveSizePredicate = isWave32;
}
def : GCNPat < def : GCNPat <
(f64 InlineFPImm<f64>:$imm), (f64 InlineFPImm<f64>:$imm),
(S_MOV_B64 (f64 (bitcast_fpimm_to_i64 InlineFPImm<f64>:$imm))) (S_MOV_B64 (f64 (bitcast_fpimm_to_i64 InlineFPImm<f64>:$imm)))
>; >;
/********** ================== **********/ /********** ================== **********/
/********** Intrinsic Patterns **********/ /********** Intrinsic Patterns **********/
▲ Show 20 Lines • Show All 174 Lines • ▼ Show 20 Lines
def : FPToI1Pat<V_CMP_EQ_F32_e64, CONST.FP32_ONE, i32, f32, fp_to_uint>; def : FPToI1Pat<V_CMP_EQ_F32_e64, CONST.FP32_ONE, i32, f32, fp_to_uint>;
def : FPToI1Pat<V_CMP_EQ_F32_e64, CONST.FP32_NEG_ONE, i32, f32, fp_to_sint>; def : FPToI1Pat<V_CMP_EQ_F32_e64, CONST.FP32_NEG_ONE, i32, f32, fp_to_sint>;
def : FPToI1Pat<V_CMP_EQ_F64_e64, CONST.FP64_ONE, i64, f64, fp_to_uint>; def : FPToI1Pat<V_CMP_EQ_F64_e64, CONST.FP64_ONE, i64, f64, fp_to_uint>;
def : FPToI1Pat<V_CMP_EQ_F64_e64, CONST.FP64_NEG_ONE, i64, f64, fp_to_sint>; def : FPToI1Pat<V_CMP_EQ_F64_e64, CONST.FP64_NEG_ONE, i64, f64, fp_to_sint>;
// If we need to perform a logical operation on i1 values, we need to // If we need to perform a logical operation on i1 values, we need to
// use vector comparisons since there is only one SCC register. Vector // use vector comparisons since there is only one SCC register. Vector
// comparisons still write to a pair of SGPRs, so treat these as // comparisons may write to a pair of SGPRs or a single SGPR, so treat
// 64-bit comparisons. When legalizing SGPR copies, instructions // these as 32 or 64-bit comparisons. When legalizing SGPR copies,
// resulting in the copies from SCC to these instructions will be // instructions resulting in the copies from SCC to these instructions
// moved to the VALU. // will be moved to the VALU.
let WaveSizePredicate = isWave64 in {
def : GCNPat < def : GCNPat <
(i1 (and i1:$src0, i1:$src1)), (i1 (and i1:$src0, i1:$src1)),
(S_AND_B64 $src0, $src1) (S_AND_B64 $src0, $src1)
>; >;
def : GCNPat < def : GCNPat <
(i1 (or i1:$src0, i1:$src1)), (i1 (or i1:$src0, i1:$src1)),
(S_OR_B64 $src0, $src1) (S_OR_B64 $src0, $src1)
Show All 20 Linesdef : GCNPat <
(S_NOT_B64 $src0) (S_NOT_B64 $src0)
>; >;
def : GCNPat < def : GCNPat <
(i1 (sub i1:$src0, (i1 -1))), (i1 (sub i1:$src0, (i1 -1))),
(S_NOT_B64 $src0) (S_NOT_B64 $src0)
>; >;
} }
} // end isWave64
let WaveSizePredicate = isWave32 in {
def : GCNPat <
(i1 (and i1:$src0, i1:$src1)),
(S_AND_B32 $src0, $src1)
>;
def : GCNPat <
(i1 (or i1:$src0, i1:$src1)),
(S_OR_B32 $src0, $src1)
>;
def : GCNPat <
(i1 (xor i1:$src0, i1:$src1)),
(S_XOR_B32 $src0, $src1)
>;
def : GCNPat <
(i1 (add i1:$src0, i1:$src1)),
(S_XOR_B32 $src0, $src1)
>;
def : GCNPat <
(i1 (sub i1:$src0, i1:$src1)),
(S_XOR_B32 $src0, $src1)
>;
let AddedComplexity = 1 in {
def : GCNPat <
(i1 (add i1:$src0, (i1 -1))),
(S_NOT_B32 $src0)
>;
def : GCNPat <
(i1 (sub i1:$src0, (i1 -1))),
(S_NOT_B32 $src0)
>;
}
} // end isWave32
def : GCNPat < def : GCNPat <
(f16 (sint_to_fp i1:$src)), (f16 (sint_to_fp i1:$src)),
(V_CVT_F16_F32_e32 ( (V_CVT_F16_F32_e32 (
V_CNDMASK_B32_e64 /*src0mod*/(i32 0), /*src0*/(i32 0), V_CNDMASK_B32_e64 /*src0mod*/(i32 0), /*src0*/(i32 0),
/*src1mod*/(i32 0), /*src1*/(i32 CONST.FP32_NEG_ONE), /*src1mod*/(i32 0), /*src1*/(i32 CONST.FP32_NEG_ONE),
$src)) $src))
>; >;
▲ Show 20 Lines • Show All 362 Lines • Show Last 20 Lines

lib/Target/AMDGPU/SILoadStoreOptimizer.cpp

Show First 20 Lines • Show All 1,138 Lines • ▼ Show 20 Linesunsigned SILoadStoreOptimizer::computeBase(MachineInstr &MI,
assert((TRI->getRegSizeInBits(Addr.Base.HiReg, *MRI) == 32 || assert((TRI->getRegSizeInBits(Addr.Base.HiReg, *MRI) == 32 ||
Addr.Base.HiSubReg) && Addr.Base.HiSubReg) &&
"Expected 32-bit Base-Register-Hi!!"); "Expected 32-bit Base-Register-Hi!!");
LLVM_DEBUG(dbgs() << " Re-Computed Anchor-Base:\n"); LLVM_DEBUG(dbgs() << " Re-Computed Anchor-Base:\n");
MachineOperand OffsetLo = createRegOrImm(static_cast<int32_t>(Addr.Offset), MI); MachineOperand OffsetLo = createRegOrImm(static_cast<int32_t>(Addr.Offset), MI);
MachineOperand OffsetHi = MachineOperand OffsetHi =
createRegOrImm(static_cast<int32_t>(Addr.Offset >> 32), MI); createRegOrImm(static_cast<int32_t>(Addr.Offset >> 32), MI);
unsigned CarryReg = MRI->createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass);
unsigned DeadCarryReg = const auto *CarryRC = TRI->getRegClass(AMDGPU::SReg_1_XEXECRegClassID);
MRI->createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass); unsigned CarryReg = MRI->createVirtualRegister(CarryRC);
unsigned DeadCarryReg = MRI->createVirtualRegister(CarryRC);
unsigned DestSub0 = MRI->createVirtualRegister(&AMDGPU::VGPR_32RegClass); unsigned DestSub0 = MRI->createVirtualRegister(&AMDGPU::VGPR_32RegClass);
unsigned DestSub1 = MRI->createVirtualRegister(&AMDGPU::VGPR_32RegClass); unsigned DestSub1 = MRI->createVirtualRegister(&AMDGPU::VGPR_32RegClass);
MachineInstr *LoHalf = MachineInstr *LoHalf =
BuildMI(*MBB, MBBI, DL, TII->get(AMDGPU::V_ADD_I32_e64), DestSub0) BuildMI(*MBB, MBBI, DL, TII->get(AMDGPU::V_ADD_I32_e64), DestSub0)
.addReg(CarryReg, RegState::Define) .addReg(CarryReg, RegState::Define)
.addReg(Addr.Base.LoReg, 0, Addr.Base.LoSubReg) .addReg(Addr.Base.LoReg, 0, Addr.Base.LoSubReg)
.add(OffsetLo) .add(OffsetLo)
▲ Show 20 Lines • Show All 390 Lines • Show Last 20 Lines

lib/Target/AMDGPU/SILowerControlFlow.cpp

Show First 20 Lines • Show All 76 Lines • ▼ Show 20 Lines
class SILowerControlFlow : public MachineFunctionPass { class SILowerControlFlow : public MachineFunctionPass {
private: private:
const SIRegisterInfo *TRI = nullptr; const SIRegisterInfo *TRI = nullptr;
const SIInstrInfo *TII = nullptr; const SIInstrInfo *TII = nullptr;
LiveIntervals *LIS = nullptr; LiveIntervals *LIS = nullptr;
MachineRegisterInfo *MRI = nullptr; MachineRegisterInfo *MRI = nullptr;
const TargetRegisterClass *BoolRC = nullptr;
unsigned AndOpc;
unsigned OrOpc;
unsigned XorOpc;
unsigned MovTermOpc;
unsigned Andn2TermOpc;
unsigned XorTermrOpc;
unsigned OrSaveExecOpc;
unsigned Exec;
void emitIf(MachineInstr &MI); void emitIf(MachineInstr &MI);
void emitElse(MachineInstr &MI); void emitElse(MachineInstr &MI);
void emitIfBreak(MachineInstr &MI); void emitIfBreak(MachineInstr &MI);
void emitLoop(MachineInstr &MI); void emitLoop(MachineInstr &MI);
void emitEndCf(MachineInstr &MI); void emitEndCf(MachineInstr &MI);
void findMaskOperands(MachineInstr &MI, unsigned OpNo, void findMaskOperands(MachineInstr &MI, unsigned OpNo,
SmallVectorImpl<MachineOperand> &Src) const; SmallVectorImpl<MachineOperand> &Src) const;
▲ Show 20 Lines • Show All 90 Lines • ▼ Show 20 Linesvoid SILowerControlFlow::emitIf(MachineInstr &MI) {
// If there is only one use of save exec register and that use is SI_END_CF, // If there is only one use of save exec register and that use is SI_END_CF,
// we can optimize SI_IF by returning the full saved exec mask instead of // we can optimize SI_IF by returning the full saved exec mask instead of
// just cleared bits. // just cleared bits.
bool SimpleIf = isSimpleIf(MI, MRI, TII); bool SimpleIf = isSimpleIf(MI, MRI, TII);
// Add an implicit def of exec to discourage scheduling VALU after this which // Add an implicit def of exec to discourage scheduling VALU after this which
// will interfere with trying to form s_and_saveexec_b64 later. // will interfere with trying to form s_and_saveexec_b64 later.
unsigned CopyReg = SimpleIf ? SaveExecReg unsigned CopyReg = SimpleIf ? SaveExecReg
: MRI->createVirtualRegister(&AMDGPU::SReg_64RegClass); : MRI->createVirtualRegister(BoolRC);
MachineInstr *CopyExec = MachineInstr *CopyExec =
BuildMI(MBB, I, DL, TII->get(AMDGPU::COPY), CopyReg) BuildMI(MBB, I, DL, TII->get(AMDGPU::COPY), CopyReg)
.addReg(AMDGPU::EXEC) .addReg(Exec)
.addReg(AMDGPU::EXEC, RegState::ImplicitDefine); .addReg(Exec, RegState::ImplicitDefine);
unsigned Tmp = MRI->createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned Tmp = MRI->createVirtualRegister(BoolRC);
MachineInstr *And = MachineInstr *And =
BuildMI(MBB, I, DL, TII->get(AMDGPU::S_AND_B64), Tmp) BuildMI(MBB, I, DL, TII->get(AndOpc), Tmp)
.addReg(CopyReg) .addReg(CopyReg)
.add(Cond); .add(Cond);
setImpSCCDefDead(*And, true); setImpSCCDefDead(*And, true);
MachineInstr *Xor = nullptr; MachineInstr *Xor = nullptr;
if (!SimpleIf) { if (!SimpleIf) {
Xor = Xor =
BuildMI(MBB, I, DL, TII->get(AMDGPU::S_XOR_B64), SaveExecReg) BuildMI(MBB, I, DL, TII->get(XorOpc), SaveExecReg)
.addReg(Tmp) .addReg(Tmp)
.addReg(CopyReg); .addReg(CopyReg);
setImpSCCDefDead(*Xor, ImpDefSCC.isDead()); setImpSCCDefDead(*Xor, ImpDefSCC.isDead());
} }
// Use a copy that is a terminator to get correct spill code placement it with // Use a copy that is a terminator to get correct spill code placement it with
// fast regalloc. // fast regalloc.
MachineInstr *SetExec = MachineInstr *SetExec =
BuildMI(MBB, I, DL, TII->get(AMDGPU::S_MOV_B64_term), AMDGPU::EXEC) BuildMI(MBB, I, DL, TII->get(MovTermOpc), Exec)
.addReg(Tmp, RegState::Kill); .addReg(Tmp, RegState::Kill);
// Insert a pseudo terminator to help keep the verifier happy. This will also // Insert a pseudo terminator to help keep the verifier happy. This will also
// be used later when inserting skips. // be used later when inserting skips.
MachineInstr *NewBr = BuildMI(MBB, I, DL, TII->get(AMDGPU::SI_MASK_BRANCH)) MachineInstr *NewBr = BuildMI(MBB, I, DL, TII->get(AMDGPU::SI_MASK_BRANCH))
.add(MI.getOperand(2)); .add(MI.getOperand(2));
if (!LIS) { if (!LIS) {
Show All 33 Linesvoid SILowerControlFlow::emitElse(MachineInstr &MI) {
assert(MI.getOperand(0).getSubReg() == AMDGPU::NoSubRegister); assert(MI.getOperand(0).getSubReg() == AMDGPU::NoSubRegister);
bool ExecModified = MI.getOperand(3).getImm() != 0; bool ExecModified = MI.getOperand(3).getImm() != 0;
MachineBasicBlock::iterator Start = MBB.begin(); MachineBasicBlock::iterator Start = MBB.begin();
// We are running before TwoAddressInstructions, and si_else's operands are // We are running before TwoAddressInstructions, and si_else's operands are
// tied. In order to correctly tie the registers, split this into a copy of // tied. In order to correctly tie the registers, split this into a copy of
// the src like it does. // the src like it does.
unsigned CopyReg = MRI->createVirtualRegister(&AMDGPU::SReg_64RegClass); unsigned CopyReg = MRI->createVirtualRegister(BoolRC);
MachineInstr *CopyExec = MachineInstr *CopyExec =
BuildMI(MBB, Start, DL, TII->get(AMDGPU::COPY), CopyReg) BuildMI(MBB, Start, DL, TII->get(AMDGPU::COPY), CopyReg)
.add(MI.getOperand(1)); // Saved EXEC .add(MI.getOperand(1)); // Saved EXEC
// This must be inserted before phis and any spill code inserted before the // This must be inserted before phis and any spill code inserted before the
// else. // else.
unsigned SaveReg = ExecModified ? unsigned SaveReg = ExecModified ?
MRI->createVirtualRegister(&AMDGPU::SReg_64RegClass) : DstReg; MRI->createVirtualRegister(BoolRC) : DstReg;
MachineInstr *OrSaveExec = MachineInstr *OrSaveExec =
BuildMI(MBB, Start, DL, TII->get(AMDGPU::S_OR_SAVEEXEC_B64), SaveReg) BuildMI(MBB, Start, DL, TII->get(OrSaveExecOpc), SaveReg)
.addReg(CopyReg); .addReg(CopyReg);
MachineBasicBlock *DestBB = MI.getOperand(2).getMBB(); MachineBasicBlock *DestBB = MI.getOperand(2).getMBB();
MachineBasicBlock::iterator ElsePt(MI); MachineBasicBlock::iterator ElsePt(MI);
if (ExecModified) { if (ExecModified) {
MachineInstr *And = MachineInstr *And =
BuildMI(MBB, ElsePt, DL, TII->get(AMDGPU::S_AND_B64), DstReg) BuildMI(MBB, ElsePt, DL, TII->get(AndOpc), DstReg)
.addReg(AMDGPU::EXEC) .addReg(Exec)
.addReg(SaveReg); .addReg(SaveReg);
if (LIS) if (LIS)
LIS->InsertMachineInstrInMaps(*And); LIS->InsertMachineInstrInMaps(*And);
} }
MachineInstr *Xor = MachineInstr *Xor =
BuildMI(MBB, ElsePt, DL, TII->get(AMDGPU::S_XOR_B64_term), AMDGPU::EXEC) BuildMI(MBB, ElsePt, DL, TII->get(XorTermrOpc), Exec)
.addReg(AMDGPU::EXEC) .addReg(Exec)
.addReg(DstReg); .addReg(DstReg);
MachineInstr *Branch = MachineInstr *Branch =
BuildMI(MBB, ElsePt, DL, TII->get(AMDGPU::SI_MASK_BRANCH)) BuildMI(MBB, ElsePt, DL, TII->get(AMDGPU::SI_MASK_BRANCH))
.addMBB(DestBB); .addMBB(DestBB);
if (!LIS) { if (!LIS) {
MI.eraseFromParent(); MI.eraseFromParent();
Show All 36 Lines if (MachineInstr *Def = MRI->getUniqueVRegDef(MI.getOperand(1).getReg())) {
&& SIInstrInfo::isVALU(*Def); && SIInstrInfo::isVALU(*Def);
} }
} }
// AND the break condition operand with exec, then OR that into the "loop // AND the break condition operand with exec, then OR that into the "loop
// exit" mask. // exit" mask.
MachineInstr *And = nullptr, *Or = nullptr; MachineInstr *And = nullptr, *Or = nullptr;
if (!SkipAnding) { if (!SkipAnding) {
And = BuildMI(MBB, &MI, DL, TII->get(AMDGPU::S_AND_B64), Dst) And = BuildMI(MBB, &MI, DL, TII->get(AndOpc), Dst)
.addReg(AMDGPU::EXEC) .addReg(Exec)
.add(MI.getOperand(1)); .add(MI.getOperand(1));
Or = BuildMI(MBB, &MI, DL, TII->get(AMDGPU::S_OR_B64), Dst) Or = BuildMI(MBB, &MI, DL, TII->get(OrOpc), Dst)
.addReg(Dst) .addReg(Dst)
.add(MI.getOperand(2)); .add(MI.getOperand(2));
} else } else
Or = BuildMI(MBB, &MI, DL, TII->get(AMDGPU::S_OR_B64), Dst) Or = BuildMI(MBB, &MI, DL, TII->get(OrOpc), Dst)
.add(MI.getOperand(1)) .add(MI.getOperand(1))
.add(MI.getOperand(2)); .add(MI.getOperand(2));
if (LIS) { if (LIS) {
if (And) if (And)
LIS->InsertMachineInstrInMaps(*And); LIS->InsertMachineInstrInMaps(*And);
LIS->ReplaceMachineInstrInMaps(MI, *Or); LIS->ReplaceMachineInstrInMaps(MI, *Or);
} }
MI.eraseFromParent(); MI.eraseFromParent();
} }
void SILowerControlFlow::emitLoop(MachineInstr &MI) { void SILowerControlFlow::emitLoop(MachineInstr &MI) {
MachineBasicBlock &MBB = *MI.getParent(); MachineBasicBlock &MBB = *MI.getParent();
const DebugLoc &DL = MI.getDebugLoc(); const DebugLoc &DL = MI.getDebugLoc();
MachineInstr *AndN2 = MachineInstr *AndN2 =
BuildMI(MBB, &MI, DL, TII->get(AMDGPU::S_ANDN2_B64_term), AMDGPU::EXEC) BuildMI(MBB, &MI, DL, TII->get(Andn2TermOpc), Exec)
.addReg(AMDGPU::EXEC) .addReg(Exec)
.add(MI.getOperand(0)); .add(MI.getOperand(0));
MachineInstr *Branch = MachineInstr *Branch =
BuildMI(MBB, &MI, DL, TII->get(AMDGPU::S_CBRANCH_EXECNZ)) BuildMI(MBB, &MI, DL, TII->get(AMDGPU::S_CBRANCH_EXECNZ))
.add(MI.getOperand(1)); .add(MI.getOperand(1));
if (LIS) { if (LIS) {
LIS->ReplaceMachineInstrInMaps(MI, *AndN2); LIS->ReplaceMachineInstrInMaps(MI, *AndN2);
LIS->InsertMachineInstrInMaps(*Branch); LIS->InsertMachineInstrInMaps(*Branch);
} }
MI.eraseFromParent(); MI.eraseFromParent();
} }
void SILowerControlFlow::emitEndCf(MachineInstr &MI) { void SILowerControlFlow::emitEndCf(MachineInstr &MI) {
MachineBasicBlock &MBB = *MI.getParent(); MachineBasicBlock &MBB = *MI.getParent();
const DebugLoc &DL = MI.getDebugLoc(); const DebugLoc &DL = MI.getDebugLoc();
MachineBasicBlock::iterator InsPt = MBB.begin(); MachineBasicBlock::iterator InsPt = MBB.begin();
MachineInstr *NewMI = MachineInstr *NewMI =
BuildMI(MBB, InsPt, DL, TII->get(AMDGPU::S_OR_B64), AMDGPU::EXEC) BuildMI(MBB, InsPt, DL, TII->get(OrOpc), Exec)
.addReg(AMDGPU::EXEC) .addReg(Exec)
.add(MI.getOperand(0)); .add(MI.getOperand(0));
if (LIS) if (LIS)
LIS->ReplaceMachineInstrInMaps(MI, *NewMI); LIS->ReplaceMachineInstrInMaps(MI, *NewMI);
MI.eraseFromParent(); MI.eraseFromParent();
if (LIS) if (LIS)
Show All 15 Lines if (!Def || Def->getParent() != MI.getParent() ||
!(Def->isFullCopy() || (Def->getOpcode() == MI.getOpcode()))) !(Def->isFullCopy() || (Def->getOpcode() == MI.getOpcode())))
return; return;
// Make sure we do not modify exec between def and use. // Make sure we do not modify exec between def and use.
// A copy with implcitly defined exec inserted earlier is an exclusion, it // A copy with implcitly defined exec inserted earlier is an exclusion, it
// does not really modify exec. // does not really modify exec.
for (auto I = Def->getIterator(); I != MI.getIterator(); ++I) for (auto I = Def->getIterator(); I != MI.getIterator(); ++I)
if (I->modifiesRegister(AMDGPU::EXEC, TRI) && if (I->modifiesRegister(AMDGPU::EXEC, TRI) &&
!(I->isCopy() && I->getOperand(0).getReg() != AMDGPU::EXEC)) !(I->isCopy() && I->getOperand(0).getReg() != Exec))
return; return;
for (const auto &SrcOp : Def->explicit_operands()) for (const auto &SrcOp : Def->explicit_operands())
if (SrcOp.isReg() && SrcOp.isUse() && if (SrcOp.isReg() && SrcOp.isUse() &&
(TargetRegisterInfo::isVirtualRegister(SrcOp.getReg()) || (TargetRegisterInfo::isVirtualRegister(SrcOp.getReg()) ||
SrcOp.getReg() == AMDGPU::EXEC)) SrcOp.getReg() == Exec))
Src.push_back(SrcOp); Src.push_back(SrcOp);
} }
// Search and combine pairs of equivalent instructions, like // Search and combine pairs of equivalent instructions, like
// S_AND_B64 x, (S_AND_B64 x, y) => S_AND_B64 x, y // S_AND_B64 x, (S_AND_B64 x, y) => S_AND_B64 x, y
// S_OR_B64 x, (S_OR_B64 x, y) => S_OR_B64 x, y // S_OR_B64 x, (S_OR_B64 x, y) => S_OR_B64 x, y
// One of the operands is exec mask. // One of the operands is exec mask.
void SILowerControlFlow::combineMasks(MachineInstr &MI) { void SILowerControlFlow::combineMasks(MachineInstr &MI) {
Show All 21 Lines
bool SILowerControlFlow::runOnMachineFunction(MachineFunction &MF) { bool SILowerControlFlow::runOnMachineFunction(MachineFunction &MF) {
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>(); const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
TII = ST.getInstrInfo(); TII = ST.getInstrInfo();
TRI = &TII->getRegisterInfo(); TRI = &TII->getRegisterInfo();
// This doesn't actually need LiveIntervals, but we can preserve them. // This doesn't actually need LiveIntervals, but we can preserve them.
LIS = getAnalysisIfAvailable<LiveIntervals>(); LIS = getAnalysisIfAvailable<LiveIntervals>();
MRI = &MF.getRegInfo(); MRI = &MF.getRegInfo();
BoolRC = TRI->getBoolRC();
if (ST.isWave32()) {
AndOpc = AMDGPU::S_AND_B32;
OrOpc = AMDGPU::S_OR_B32;
XorOpc = AMDGPU::S_XOR_B32;
MovTermOpc = AMDGPU::S_MOV_B32_term;
Andn2TermOpc = AMDGPU::S_ANDN2_B32_term;
XorTermrOpc = AMDGPU::S_XOR_B32_term;
OrSaveExecOpc = AMDGPU::S_OR_SAVEEXEC_B32;
Exec = AMDGPU::EXEC_LO;
} else {
AndOpc = AMDGPU::S_AND_B64;
OrOpc = AMDGPU::S_OR_B64;
XorOpc = AMDGPU::S_XOR_B64;
MovTermOpc = AMDGPU::S_MOV_B64_term;
Andn2TermOpc = AMDGPU::S_ANDN2_B64_term;
XorTermrOpc = AMDGPU::S_XOR_B64_term;
OrSaveExecOpc = AMDGPU::S_OR_SAVEEXEC_B64;
Exec = AMDGPU::EXEC;
}
MachineFunction::iterator NextBB; MachineFunction::iterator NextBB;
for (MachineFunction::iterator BI = MF.begin(), BE = MF.end(); for (MachineFunction::iterator BI = MF.begin(), BE = MF.end();
BI != BE; BI = NextBB) { BI != BE; BI = NextBB) {
NextBB = std::next(BI); NextBB = std::next(BI);
MachineBasicBlock &MBB = *BI; MachineBasicBlock &MBB = *BI;
MachineBasicBlock::iterator I, Next, Last; MachineBasicBlock::iterator I, Next, Last;
Show All 20 Lines for (I = MBB.begin(), Last = MBB.end(); I != MBB.end(); I = Next) {
break; break;
case AMDGPU::SI_END_CF: case AMDGPU::SI_END_CF:
emitEndCf(MI); emitEndCf(MI);
break; break;
case AMDGPU::S_AND_B64: case AMDGPU::S_AND_B64:
case AMDGPU::S_OR_B64: case AMDGPU::S_OR_B64:
case AMDGPU::S_AND_B32:
case AMDGPU::S_OR_B32:
// Cleanup bit manipulations on exec mask // Cleanup bit manipulations on exec mask
combineMasks(MI); combineMasks(MI);
Last = I; Last = I;
continue; continue;
default: default:
Last = I; Last = I;
continue; continue;
Show All 9 Lines

lib/Target/AMDGPU/SILowerI1Copies.cpp

//===-- SILowerI1Copies.cpp - Lower I1 Copies -----------------------------===// //===-- SILowerI1Copies.cpp - Lower I1 Copies -----------------------------===//
// //
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information. // See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
// //
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// //
// This pass lowers all occurrences of i1 values (with a vreg_1 register class) // This pass lowers all occurrences of i1 values (with a vreg_1 register class)
// to lane masks (64-bit scalar registers). The pass assumes machine SSA form // to lane masks (32 / 64-bit scalar registers). The pass assumes machine SSA
// and a wave-level control flow graph. // form and a wave-level control flow graph.
// //
// Before this pass, values that are semantically i1 and are defined and used // Before this pass, values that are semantically i1 and are defined and used
// within the same basic block are already represented as lane masks in scalar // within the same basic block are already represented as lane masks in scalar
// registers. However, values that cross basic blocks are always transferred // registers. However, values that cross basic blocks are always transferred
// between basic blocks in vreg_1 virtual registers and are lowered by this // between basic blocks in vreg_1 virtual registers and are lowered by this
// pass. // pass.
// //
// The only instructions that use or define vreg_1 virtual registers are COPY, // The only instructions that use or define vreg_1 virtual registers are COPY,
Show All 25 Lines
namespace { namespace {
class SILowerI1Copies : public MachineFunctionPass { class SILowerI1Copies : public MachineFunctionPass {
public: public:
static char ID; static char ID;
private: private:
bool IsWave32 = false;
MachineFunction *MF = nullptr; MachineFunction *MF = nullptr;
MachineDominatorTree *DT = nullptr; MachineDominatorTree *DT = nullptr;
MachinePostDominatorTree *PDT = nullptr; MachinePostDominatorTree *PDT = nullptr;
MachineRegisterInfo *MRI = nullptr; MachineRegisterInfo *MRI = nullptr;
const GCNSubtarget *ST = nullptr; const GCNSubtarget *ST = nullptr;
const SIInstrInfo *TII = nullptr; const SIInstrInfo *TII = nullptr;
unsigned ExecReg;
unsigned MovOp;
unsigned AndOp;
unsigned OrOp;
unsigned XorOp;
unsigned AndN2Op;
unsigned OrN2Op;
DenseSet<unsigned> ConstrainRegs; DenseSet<unsigned> ConstrainRegs;
public: public:
SILowerI1Copies() : MachineFunctionPass(ID) { SILowerI1Copies() : MachineFunctionPass(ID) {
initializeSILowerI1CopiesPass(*PassRegistry::getPassRegistry()); initializeSILowerI1CopiesPass(*PassRegistry::getPassRegistry());
} }
bool runOnMachineFunction(MachineFunction &MF) override; bool runOnMachineFunction(MachineFunction &MF) override;
▲ Show 20 Lines • Show All 338 Lines • ▼ Show 20 Lines
char &llvm::SILowerI1CopiesID = SILowerI1Copies::ID; char &llvm::SILowerI1CopiesID = SILowerI1Copies::ID;
FunctionPass *llvm::createSILowerI1CopiesPass() { FunctionPass *llvm::createSILowerI1CopiesPass() {
return new SILowerI1Copies(); return new SILowerI1Copies();
} }
static unsigned createLaneMaskReg(MachineFunction &MF) { static unsigned createLaneMaskReg(MachineFunction &MF) {
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
MachineRegisterInfo &MRI = MF.getRegInfo(); MachineRegisterInfo &MRI = MF.getRegInfo();
return MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass); return MRI.createVirtualRegister(ST.isWave32() ? &AMDGPU::SReg_32RegClass
: &AMDGPU::SReg_64RegClass);
} }
static unsigned insertUndefLaneMask(MachineBasicBlock &MBB) { static unsigned insertUndefLaneMask(MachineBasicBlock &MBB) {
MachineFunction &MF = *MBB.getParent(); MachineFunction &MF = *MBB.getParent();
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>(); const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
const SIInstrInfo *TII = ST.getInstrInfo(); const SIInstrInfo *TII = ST.getInstrInfo();
unsigned UndefReg = createLaneMaskReg(MF); unsigned UndefReg = createLaneMaskReg(MF);
BuildMI(MBB, MBB.getFirstTerminator(), {}, TII->get(AMDGPU::IMPLICIT_DEF), BuildMI(MBB, MBB.getFirstTerminator(), {}, TII->get(AMDGPU::IMPLICIT_DEF),
Show All 13 Lines
bool SILowerI1Copies::runOnMachineFunction(MachineFunction &TheMF) { bool SILowerI1Copies::runOnMachineFunction(MachineFunction &TheMF) {
MF = &TheMF; MF = &TheMF;
MRI = &MF->getRegInfo(); MRI = &MF->getRegInfo();
DT = &getAnalysis<MachineDominatorTree>(); DT = &getAnalysis<MachineDominatorTree>();
PDT = &getAnalysis<MachinePostDominatorTree>(); PDT = &getAnalysis<MachinePostDominatorTree>();
ST = &MF->getSubtarget<GCNSubtarget>(); ST = &MF->getSubtarget<GCNSubtarget>();
TII = ST->getInstrInfo(); TII = ST->getInstrInfo();
IsWave32 = ST->isWave32();
if (IsWave32) {
ExecReg = AMDGPU::EXEC_LO;
MovOp = AMDGPU::S_MOV_B32;
AndOp = AMDGPU::S_AND_B32;
OrOp = AMDGPU::S_OR_B32;
XorOp = AMDGPU::S_XOR_B32;
AndN2Op = AMDGPU::S_ANDN2_B32;
OrN2Op = AMDGPU::S_ORN2_B32;
} else {
ExecReg = AMDGPU::EXEC;
MovOp = AMDGPU::S_MOV_B64;
AndOp = AMDGPU::S_AND_B64;
OrOp = AMDGPU::S_OR_B64;
XorOp = AMDGPU::S_XOR_B64;
AndN2Op = AMDGPU::S_ANDN2_B64;
OrN2Op = AMDGPU::S_ORN2_B64;
}
lowerCopiesFromI1(); lowerCopiesFromI1();
lowerPhis(); lowerPhis();
lowerCopiesToI1(); lowerCopiesToI1();
for (unsigned Reg : ConstrainRegs) for (unsigned Reg : ConstrainRegs)
MRI->constrainRegClass(Reg, &AMDGPU::SReg_64_XEXECRegClass); MRI->constrainRegClass(Reg, &AMDGPU::SReg_1_XEXECRegClass);
ConstrainRegs.clear(); ConstrainRegs.clear();
return true; return true;
} }
void SILowerI1Copies::lowerCopiesFromI1() { void SILowerI1Copies::lowerCopiesFromI1() {
SmallVector<MachineInstr *, 4> DeadCopies; SmallVector<MachineInstr *, 4> DeadCopies;
▲ Show 20 Lines • Show All 53 Lines • ▼ Show 20 Lines for (MachineBasicBlock &MBB : *MF) {
for (MachineInstr &MI : MBB.phis()) { for (MachineInstr &MI : MBB.phis()) {
unsigned DstReg = MI.getOperand(0).getReg(); unsigned DstReg = MI.getOperand(0).getReg();
if (MRI->getRegClass(DstReg) != &AMDGPU::VReg_1RegClass) if (MRI->getRegClass(DstReg) != &AMDGPU::VReg_1RegClass)
continue; continue;
LLVM_DEBUG(dbgs() << "Lower PHI: " << MI); LLVM_DEBUG(dbgs() << "Lower PHI: " << MI);
MRI->setRegClass(DstReg, &AMDGPU::SReg_64RegClass); MRI->setRegClass(DstReg, IsWave32 ? &AMDGPU::SReg_32RegClass
: &AMDGPU::SReg_64RegClass);
// Collect incoming values. // Collect incoming values.
for (unsigned i = 1; i < MI.getNumOperands(); i += 2) { for (unsigned i = 1; i < MI.getNumOperands(); i += 2) {
assert(i + 1 < MI.getNumOperands()); assert(i + 1 < MI.getNumOperands());
unsigned IncomingReg = MI.getOperand(i).getReg(); unsigned IncomingReg = MI.getOperand(i).getReg();
MachineBasicBlock *IncomingMBB = MI.getOperand(i + 1).getMBB(); MachineBasicBlock *IncomingMBB = MI.getOperand(i + 1).getMBB();
MachineInstr *IncomingDef = MRI->getUniqueVRegDef(IncomingReg); MachineInstr *IncomingDef = MRI->getUniqueVRegDef(IncomingReg);
▲ Show 20 Lines • Show All 113 Lines • ▼ Show 20 Lines for (MachineInstr &MI : MBB) {
if (MRI->use_empty(DstReg)) { if (MRI->use_empty(DstReg)) {
DeadCopies.push_back(&MI); DeadCopies.push_back(&MI);
continue; continue;
} }
LLVM_DEBUG(dbgs() << "Lower Other: " << MI); LLVM_DEBUG(dbgs() << "Lower Other: " << MI);
MRI->setRegClass(DstReg, &AMDGPU::SReg_64RegClass); MRI->setRegClass(DstReg, IsWave32 ? &AMDGPU::SReg_32RegClass
: &AMDGPU::SReg_64RegClass);
if (MI.getOpcode() == AMDGPU::IMPLICIT_DEF) if (MI.getOpcode() == AMDGPU::IMPLICIT_DEF)
continue; continue;
DebugLoc DL = MI.getDebugLoc(); DebugLoc DL = MI.getDebugLoc();
unsigned SrcReg = MI.getOperand(1).getReg(); unsigned SrcReg = MI.getOperand(1).getReg();
assert(!MI.getOperand(1).getSubReg()); assert(!MI.getOperand(1).getSubReg());
if (!TargetRegisterInfo::isVirtualRegister(SrcReg) || if (!TargetRegisterInfo::isVirtualRegister(SrcReg) ||
▲ Show 20 Lines • Show All 42 Lines • ▼ Show 20 Lines for (;;) {
Reg = MI->getOperand(1).getReg(); Reg = MI->getOperand(1).getReg();
if (!TargetRegisterInfo::isVirtualRegister(Reg)) if (!TargetRegisterInfo::isVirtualRegister(Reg))
return false; return false;
if (!isLaneMaskReg(Reg)) if (!isLaneMaskReg(Reg))
return false; return false;
} }
if (MI->getOpcode() != AMDGPU::S_MOV_B64) if (MI->getOpcode() != MovOp)
return false; return false;
if (!MI->getOperand(1).isImm()) if (!MI->getOperand(1).isImm())
return false; return false;
int64_t Imm = MI->getOperand(1).getImm(); int64_t Imm = MI->getOperand(1).getImm();
if (Imm == 0) { if (Imm == 0) {
Val = false; Val = false;
▲ Show 20 Lines • Show All 58 Lines • ▼ Show 20 Linesvoid SILowerI1Copies::buildMergeLaneMasks(MachineBasicBlock &MBB,
bool PrevConstant = isConstantLaneMask(PrevReg, PrevVal); bool PrevConstant = isConstantLaneMask(PrevReg, PrevVal);
bool CurVal; bool CurVal;
bool CurConstant = isConstantLaneMask(CurReg, CurVal); bool CurConstant = isConstantLaneMask(CurReg, CurVal);
if (PrevConstant && CurConstant) { if (PrevConstant && CurConstant) {
if (PrevVal == CurVal) { if (PrevVal == CurVal) {
BuildMI(MBB, I, DL, TII->get(AMDGPU::COPY), DstReg).addReg(CurReg); BuildMI(MBB, I, DL, TII->get(AMDGPU::COPY), DstReg).addReg(CurReg);
} else if (CurVal) { } else if (CurVal) {
BuildMI(MBB, I, DL, TII->get(AMDGPU::COPY), DstReg).addReg(AMDGPU::EXEC); BuildMI(MBB, I, DL, TII->get(AMDGPU::COPY), DstReg).addReg(ExecReg);
} else { } else {
BuildMI(MBB, I, DL, TII->get(AMDGPU::S_XOR_B64), DstReg) BuildMI(MBB, I, DL, TII->get(XorOp), DstReg)
.addReg(AMDGPU::EXEC) .addReg(ExecReg)
.addImm(-1); .addImm(-1);
} }
return; return;
} }
unsigned PrevMaskedReg = 0; unsigned PrevMaskedReg = 0;
unsigned CurMaskedReg = 0; unsigned CurMaskedReg = 0;
if (!PrevConstant) { if (!PrevConstant) {
if (CurConstant && CurVal) { if (CurConstant && CurVal) {
PrevMaskedReg = PrevReg; PrevMaskedReg = PrevReg;
} else { } else {
PrevMaskedReg = createLaneMaskReg(*MF); PrevMaskedReg = createLaneMaskReg(*MF);
BuildMI(MBB, I, DL, TII->get(AMDGPU::S_ANDN2_B64), PrevMaskedReg) BuildMI(MBB, I, DL, TII->get(AndN2Op), PrevMaskedReg)
.addReg(PrevReg) .addReg(PrevReg)
.addReg(AMDGPU::EXEC); .addReg(ExecReg);
} }
} }
if (!CurConstant) { if (!CurConstant) {
// TODO: check whether CurReg is already masked by EXEC // TODO: check whether CurReg is already masked by EXEC
if (PrevConstant && PrevVal) { if (PrevConstant && PrevVal) {
CurMaskedReg = CurReg; CurMaskedReg = CurReg;
} else { } else {
CurMaskedReg = createLaneMaskReg(*MF); CurMaskedReg = createLaneMaskReg(*MF);
BuildMI(MBB, I, DL, TII->get(AMDGPU::S_AND_B64), CurMaskedReg) BuildMI(MBB, I, DL, TII->get(AndOp), CurMaskedReg)
.addReg(CurReg) .addReg(CurReg)
.addReg(AMDGPU::EXEC); .addReg(ExecReg);
} }
} }
if (PrevConstant && !PrevVal) { if (PrevConstant && !PrevVal) {
BuildMI(MBB, I, DL, TII->get(AMDGPU::COPY), DstReg) BuildMI(MBB, I, DL, TII->get(AMDGPU::COPY), DstReg)
.addReg(CurMaskedReg); .addReg(CurMaskedReg);
} else if (CurConstant && !CurVal) { } else if (CurConstant && !CurVal) {
BuildMI(MBB, I, DL, TII->get(AMDGPU::COPY), DstReg) BuildMI(MBB, I, DL, TII->get(AMDGPU::COPY), DstReg)
.addReg(PrevMaskedReg); .addReg(PrevMaskedReg);
} else if (PrevConstant && PrevVal) { } else if (PrevConstant && PrevVal) {
BuildMI(MBB, I, DL, TII->get(AMDGPU::S_ORN2_B64), DstReg) BuildMI(MBB, I, DL, TII->get(OrN2Op), DstReg)
.addReg(CurMaskedReg) .addReg(CurMaskedReg)
.addReg(AMDGPU::EXEC); .addReg(ExecReg);
} else { } else {
BuildMI(MBB, I, DL, TII->get(AMDGPU::S_OR_B64), DstReg) BuildMI(MBB, I, DL, TII->get(OrOp), DstReg)
.addReg(PrevMaskedReg) .addReg(PrevMaskedReg)
.addReg(CurMaskedReg ? CurMaskedReg : (unsigned)AMDGPU::EXEC); .addReg(CurMaskedReg ? CurMaskedReg : ExecReg);
} }
} }

lib/Target/AMDGPU/SIOptimizeExecMasking.cpp

Show First 20 Lines • Show All 50 Lines • ▼ Show 20 Lines
INITIALIZE_PASS_END(SIOptimizeExecMasking, DEBUG_TYPE, INITIALIZE_PASS_END(SIOptimizeExecMasking, DEBUG_TYPE,
"SI optimize exec mask operations", false, false) "SI optimize exec mask operations", false, false)
char SIOptimizeExecMasking::ID = 0; char SIOptimizeExecMasking::ID = 0;
char &llvm::SIOptimizeExecMaskingID = SIOptimizeExecMasking::ID; char &llvm::SIOptimizeExecMaskingID = SIOptimizeExecMasking::ID;
/// If \p MI is a copy from exec, return the register copied to. /// If \p MI is a copy from exec, return the register copied to.
static unsigned isCopyFromExec(const MachineInstr &MI) { static unsigned isCopyFromExec(const MachineInstr &MI, const GCNSubtarget &ST) {
switch (MI.getOpcode()) { switch (MI.getOpcode()) {
case AMDGPU::COPY: case AMDGPU::COPY:
case AMDGPU::S_MOV_B64: case AMDGPU::S_MOV_B64:
case AMDGPU::S_MOV_B64_term: { case AMDGPU::S_MOV_B64_term:
case AMDGPU::S_MOV_B32:
case AMDGPU::S_MOV_B32_term: {
const MachineOperand &Src = MI.getOperand(1); const MachineOperand &Src = MI.getOperand(1);
if (Src.isReg() && Src.getReg() == AMDGPU::EXEC) if (Src.isReg() &&
Src.getReg() == (ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC))
return MI.getOperand(0).getReg(); return MI.getOperand(0).getReg();
} }
} }
return AMDGPU::NoRegister; return AMDGPU::NoRegister;
} }
/// If \p MI is a copy to exec, return the register copied from. /// If \p MI is a copy to exec, return the register copied from.
static unsigned isCopyToExec(const MachineInstr &MI) { static unsigned isCopyToExec(const MachineInstr &MI, const GCNSubtarget &ST) {
switch (MI.getOpcode()) { switch (MI.getOpcode()) {
case AMDGPU::COPY: case AMDGPU::COPY:
case AMDGPU::S_MOV_B64: { case AMDGPU::S_MOV_B64:
case AMDGPU::S_MOV_B32: {
const MachineOperand &Dst = MI.getOperand(0); const MachineOperand &Dst = MI.getOperand(0);
if (Dst.isReg() && Dst.getReg() == AMDGPU::EXEC && MI.getOperand(1).isReg()) if (Dst.isReg() &&
Dst.getReg() == (ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC) &&
MI.getOperand(1).isReg())
return MI.getOperand(1).getReg(); return MI.getOperand(1).getReg();
break; break;
} }
case AMDGPU::S_MOV_B64_term: case AMDGPU::S_MOV_B64_term:
case AMDGPU::S_MOV_B32_term:
llvm_unreachable("should have been replaced"); llvm_unreachable("should have been replaced");
} }
return AMDGPU::NoRegister; return AMDGPU::NoRegister;
} }
/// If \p MI is a logical operation on an exec value, /// If \p MI is a logical operation on an exec value,
/// return the register copied to. /// return the register copied to.
static unsigned isLogicalOpOnExec(const MachineInstr &MI) { static unsigned isLogicalOpOnExec(const MachineInstr &MI) {
switch (MI.getOpcode()) { switch (MI.getOpcode()) {
case AMDGPU::S_AND_B64: case AMDGPU::S_AND_B64:
case AMDGPU::S_OR_B64: case AMDGPU::S_OR_B64:
case AMDGPU::S_XOR_B64: case AMDGPU::S_XOR_B64:
case AMDGPU::S_ANDN2_B64: case AMDGPU::S_ANDN2_B64:
case AMDGPU::S_ORN2_B64: case AMDGPU::S_ORN2_B64:
case AMDGPU::S_NAND_B64: case AMDGPU::S_NAND_B64:
case AMDGPU::S_NOR_B64: case AMDGPU::S_NOR_B64:
case AMDGPU::S_XNOR_B64: { case AMDGPU::S_XNOR_B64: {
const MachineOperand &Src1 = MI.getOperand(1); const MachineOperand &Src1 = MI.getOperand(1);
if (Src1.isReg() && Src1.getReg() == AMDGPU::EXEC) if (Src1.isReg() && Src1.getReg() == AMDGPU::EXEC)
return MI.getOperand(0).getReg(); return MI.getOperand(0).getReg();
const MachineOperand &Src2 = MI.getOperand(2); const MachineOperand &Src2 = MI.getOperand(2);
if (Src2.isReg() && Src2.getReg() == AMDGPU::EXEC) if (Src2.isReg() && Src2.getReg() == AMDGPU::EXEC)
return MI.getOperand(0).getReg(); return MI.getOperand(0).getReg();
break;
}
case AMDGPU::S_AND_B32:
case AMDGPU::S_OR_B32:
case AMDGPU::S_XOR_B32:
case AMDGPU::S_ANDN2_B32:
case AMDGPU::S_ORN2_B32:
case AMDGPU::S_NAND_B32:
case AMDGPU::S_NOR_B32:
case AMDGPU::S_XNOR_B32: {
const MachineOperand &Src1 = MI.getOperand(1);
if (Src1.isReg() && Src1.getReg() == AMDGPU::EXEC_LO)
return MI.getOperand(0).getReg();
const MachineOperand &Src2 = MI.getOperand(2);
if (Src2.isReg() && Src2.getReg() == AMDGPU::EXEC_LO)
return MI.getOperand(0).getReg();
break;
} }
} }
return AMDGPU::NoRegister; return AMDGPU::NoRegister;
} }
static unsigned getSaveExecOp(unsigned Opc) { static unsigned getSaveExecOp(unsigned Opc) {
switch (Opc) { switch (Opc) {
case AMDGPU::S_AND_B64: case AMDGPU::S_AND_B64:
return AMDGPU::S_AND_SAVEEXEC_B64; return AMDGPU::S_AND_SAVEEXEC_B64;
case AMDGPU::S_OR_B64: case AMDGPU::S_OR_B64:
return AMDGPU::S_OR_SAVEEXEC_B64; return AMDGPU::S_OR_SAVEEXEC_B64;
case AMDGPU::S_XOR_B64: case AMDGPU::S_XOR_B64:
return AMDGPU::S_XOR_SAVEEXEC_B64; return AMDGPU::S_XOR_SAVEEXEC_B64;
case AMDGPU::S_ANDN2_B64: case AMDGPU::S_ANDN2_B64:
return AMDGPU::S_ANDN2_SAVEEXEC_B64; return AMDGPU::S_ANDN2_SAVEEXEC_B64;
case AMDGPU::S_ORN2_B64: case AMDGPU::S_ORN2_B64:
return AMDGPU::S_ORN2_SAVEEXEC_B64; return AMDGPU::S_ORN2_SAVEEXEC_B64;
case AMDGPU::S_NAND_B64: case AMDGPU::S_NAND_B64:
return AMDGPU::S_NAND_SAVEEXEC_B64; return AMDGPU::S_NAND_SAVEEXEC_B64;
case AMDGPU::S_NOR_B64: case AMDGPU::S_NOR_B64:
return AMDGPU::S_NOR_SAVEEXEC_B64; return AMDGPU::S_NOR_SAVEEXEC_B64;
case AMDGPU::S_XNOR_B64: case AMDGPU::S_XNOR_B64:
return AMDGPU::S_XNOR_SAVEEXEC_B64; return AMDGPU::S_XNOR_SAVEEXEC_B64;
case AMDGPU::S_AND_B32:
return AMDGPU::S_AND_SAVEEXEC_B32;
case AMDGPU::S_OR_B32:
return AMDGPU::S_OR_SAVEEXEC_B32;
case AMDGPU::S_XOR_B32:
return AMDGPU::S_XOR_SAVEEXEC_B32;
case AMDGPU::S_ANDN2_B32:
return AMDGPU::S_ANDN2_SAVEEXEC_B32;
case AMDGPU::S_ORN2_B32:
return AMDGPU::S_ORN2_SAVEEXEC_B32;
case AMDGPU::S_NAND_B32:
return AMDGPU::S_NAND_SAVEEXEC_B32;
case AMDGPU::S_NOR_B32:
return AMDGPU::S_NOR_SAVEEXEC_B32;
case AMDGPU::S_XNOR_B32:
return AMDGPU::S_XNOR_SAVEEXEC_B32;
default: default:
return AMDGPU::INSTRUCTION_LIST_END; return AMDGPU::INSTRUCTION_LIST_END;
} }
} }
// These are only terminators to get correct spill code placement during // These are only terminators to get correct spill code placement during
// register allocation, so turn them back into normal instructions. Only one of // register allocation, so turn them back into normal instructions. Only one of
// these is expected per block. // these is expected per block.
static bool removeTerminatorBit(const SIInstrInfo &TII, MachineInstr &MI) { static bool removeTerminatorBit(const SIInstrInfo &TII, MachineInstr &MI) {
switch (MI.getOpcode()) { switch (MI.getOpcode()) {
case AMDGPU::S_MOV_B64_term: { case AMDGPU::S_MOV_B64_term:
case AMDGPU::S_MOV_B32_term: {
MI.setDesc(TII.get(AMDGPU::COPY)); MI.setDesc(TII.get(AMDGPU::COPY));
return true; return true;
} }
case AMDGPU::S_XOR_B64_term: { case AMDGPU::S_XOR_B64_term: {
// This is only a terminator to get the correct spill code placement during // This is only a terminator to get the correct spill code placement during
// register allocation. // register allocation.
MI.setDesc(TII.get(AMDGPU::S_XOR_B64)); MI.setDesc(TII.get(AMDGPU::S_XOR_B64));
return true; return true;
} }
case AMDGPU::S_XOR_B32_term: {
// This is only a terminator to get the correct spill code placement during
// register allocation.
MI.setDesc(TII.get(AMDGPU::S_XOR_B32));
return true;
}
case AMDGPU::S_OR_B32_term: {
// This is only a terminator to get the correct spill code placement during
// register allocation.
MI.setDesc(TII.get(AMDGPU::S_OR_B32));
return true;
}
case AMDGPU::S_ANDN2_B64_term: { case AMDGPU::S_ANDN2_B64_term: {
// This is only a terminator to get the correct spill code placement during // This is only a terminator to get the correct spill code placement during
// register allocation. // register allocation.
MI.setDesc(TII.get(AMDGPU::S_ANDN2_B64)); MI.setDesc(TII.get(AMDGPU::S_ANDN2_B64));
return true; return true;
} }
case AMDGPU::S_ANDN2_B32_term: {
// This is only a terminator to get the correct spill code placement during
// register allocation.
MI.setDesc(TII.get(AMDGPU::S_ANDN2_B32));
return true;
}
default: default:
return false; return false;
} }
} }
static MachineBasicBlock::reverse_iterator fixTerminators( static MachineBasicBlock::reverse_iterator fixTerminators(
const SIInstrInfo &TII, const SIInstrInfo &TII,
MachineBasicBlock &MBB) { MachineBasicBlock &MBB) {
MachineBasicBlock::reverse_iterator I = MBB.rbegin(), E = MBB.rend(); MachineBasicBlock::reverse_iterator I = MBB.rbegin(), E = MBB.rend();
for (; I != E; ++I) { for (; I != E; ++I) {
if (!I->isTerminator()) if (!I->isTerminator())
return I; return I;
if (removeTerminatorBit(TII, *I)) if (removeTerminatorBit(TII, *I))
return I; return I;
} }
return E; return E;
} }
static MachineBasicBlock::reverse_iterator findExecCopy( static MachineBasicBlock::reverse_iterator findExecCopy(
const SIInstrInfo &TII, const SIInstrInfo &TII,
const GCNSubtarget &ST,
MachineBasicBlock &MBB, MachineBasicBlock &MBB,
MachineBasicBlock::reverse_iterator I, MachineBasicBlock::reverse_iterator I,
unsigned CopyToExec) { unsigned CopyToExec) {
const unsigned InstLimit = 25; const unsigned InstLimit = 25;
auto E = MBB.rend(); auto E = MBB.rend();
for (unsigned N = 0; N <= InstLimit && I != E; ++I, ++N) { for (unsigned N = 0; N <= InstLimit && I != E; ++I, ++N) {
unsigned CopyFromExec = isCopyFromExec(*I); unsigned CopyFromExec = isCopyFromExec(*I, ST);
if (CopyFromExec != AMDGPU::NoRegister) if (CopyFromExec != AMDGPU::NoRegister)
return I; return I;
} }
return E; return E;
} }
// XXX - Seems LivePhysRegs doesn't work correctly since it will incorrectly // XXX - Seems LivePhysRegs doesn't work correctly since it will incorrectly
Show All 10 Lines
bool SIOptimizeExecMasking::runOnMachineFunction(MachineFunction &MF) { bool SIOptimizeExecMasking::runOnMachineFunction(MachineFunction &MF) {
if (skipFunction(MF.getFunction())) if (skipFunction(MF.getFunction()))
return false; return false;
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>(); const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
const SIRegisterInfo *TRI = ST.getRegisterInfo(); const SIRegisterInfo *TRI = ST.getRegisterInfo();
const SIInstrInfo *TII = ST.getInstrInfo(); const SIInstrInfo *TII = ST.getInstrInfo();
unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
// Optimize sequences emitted for control flow lowering. They are originally // Optimize sequences emitted for control flow lowering. They are originally
// emitted as the separate operations because spill code may need to be // emitted as the separate operations because spill code may need to be
// inserted for the saved copy of exec. // inserted for the saved copy of exec.
// //
// x = copy exec // x = copy exec
// z = s_<op>_b64 x, y // z = s_<op>_b64 x, y
// exec = copy z // exec = copy z
// => // =>
// x = s_<op>_saveexec_b64 y // x = s_<op>_saveexec_b64 y
// //
for (MachineBasicBlock &MBB : MF) { for (MachineBasicBlock &MBB : MF) {
MachineBasicBlock::reverse_iterator I = fixTerminators(*TII, MBB); MachineBasicBlock::reverse_iterator I = fixTerminators(*TII, MBB);
MachineBasicBlock::reverse_iterator E = MBB.rend(); MachineBasicBlock::reverse_iterator E = MBB.rend();
if (I == E) if (I == E)
continue; continue;
unsigned CopyToExec = isCopyToExec(*I); unsigned CopyToExec = isCopyToExec(*I, ST);
if (CopyToExec == AMDGPU::NoRegister) if (CopyToExec == AMDGPU::NoRegister)
continue; continue;
// Scan backwards to find the def. // Scan backwards to find the def.
auto CopyToExecInst = &*I; auto CopyToExecInst = &*I;
auto CopyFromExecInst = findExecCopy(*TII, MBB, I, CopyToExec); auto CopyFromExecInst = findExecCopy(*TII, ST, MBB, I, CopyToExec);
if (CopyFromExecInst == E) { if (CopyFromExecInst == E) {
auto PrepareExecInst = std::next(I); auto PrepareExecInst = std::next(I);
if (PrepareExecInst == E) if (PrepareExecInst == E)
continue; continue;
// Fold exec = COPY (S_AND_B64 reg, exec) -> exec = S_AND_B64 reg, exec // Fold exec = COPY (S_AND_B64 reg, exec) -> exec = S_AND_B64 reg, exec
if (CopyToExecInst->getOperand(1).isKill() && if (CopyToExecInst->getOperand(1).isKill() &&
isLogicalOpOnExec(*PrepareExecInst) == CopyToExec) { isLogicalOpOnExec(*PrepareExecInst) == CopyToExec) {
LLVM_DEBUG(dbgs() << "Fold exec copy: " << *PrepareExecInst); LLVM_DEBUG(dbgs() << "Fold exec copy: " << *PrepareExecInst);
PrepareExecInst->getOperand(0).setReg(AMDGPU::EXEC); PrepareExecInst->getOperand(0).setReg(Exec);
LLVM_DEBUG(dbgs() << "into: " << *PrepareExecInst << '\n'); LLVM_DEBUG(dbgs() << "into: " << *PrepareExecInst << '\n');
CopyToExecInst->eraseFromParent(); CopyToExecInst->eraseFromParent();
} }
continue; continue;
} }
if (isLiveOut(MBB, CopyToExec)) { if (isLiveOut(MBB, CopyToExec)) {
// The copied register is live out and has a second use in another block. // The copied register is live out and has a second use in another block.
LLVM_DEBUG(dbgs() << "Exec copy source register is live out\n"); LLVM_DEBUG(dbgs() << "Exec copy source register is live out\n");
continue; continue;
} }
unsigned CopyFromExec = CopyFromExecInst->getOperand(0).getReg(); unsigned CopyFromExec = CopyFromExecInst->getOperand(0).getReg();
MachineInstr *SaveExecInst = nullptr; MachineInstr *SaveExecInst = nullptr;
SmallVector<MachineInstr *, 4> OtherUseInsts; SmallVector<MachineInstr *, 4> OtherUseInsts;
for (MachineBasicBlock::iterator J for (MachineBasicBlock::iterator J
= std::next(CopyFromExecInst->getIterator()), JE = I->getIterator(); = std::next(CopyFromExecInst->getIterator()), JE = I->getIterator();
J != JE; ++J) { J != JE; ++J) {
if (SaveExecInst && J->readsRegister(AMDGPU::EXEC, TRI)) { if (SaveExecInst && J->readsRegister(Exec, TRI)) {
LLVM_DEBUG(dbgs() << "exec read prevents saveexec: " << *J << '\n'); LLVM_DEBUG(dbgs() << "exec read prevents saveexec: " << *J << '\n');
// Make sure this is inserted after any VALU ops that may have been // Make sure this is inserted after any VALU ops that may have been
// scheduled in between. // scheduled in between.
SaveExecInst = nullptr; SaveExecInst = nullptr;
break; break;
} }
bool ReadsCopyFromExec = J->readsRegister(CopyFromExec, TRI); bool ReadsCopyFromExec = J->readsRegister(CopyFromExec, TRI);
▲ Show 20 Lines • Show All 67 Lines • ▼ Show 20 Lines for (MachineBasicBlock &MBB : MF) {
BuildMI(MBB, InsPt, DL, TII->get(getSaveExecOp(SaveExecInst->getOpcode())), BuildMI(MBB, InsPt, DL, TII->get(getSaveExecOp(SaveExecInst->getOpcode())),
CopyFromExec) CopyFromExec)
.addReg(OtherOp->getReg()); .addReg(OtherOp->getReg());
SaveExecInst->eraseFromParent(); SaveExecInst->eraseFromParent();
CopyToExecInst->eraseFromParent(); CopyToExecInst->eraseFromParent();
for (MachineInstr *OtherInst : OtherUseInsts) { for (MachineInstr *OtherInst : OtherUseInsts) {
OtherInst->substituteRegister(CopyToExec, AMDGPU::EXEC, OtherInst->substituteRegister(CopyToExec, Exec,
AMDGPU::NoSubRegister, *TRI); AMDGPU::NoSubRegister, *TRI);
} }
} }
return true; return true;
} }

lib/Target/AMDGPU/SIOptimizeExecMaskingPreRA.cpp

Show First 20 Lines • Show All 76 Lines • ▼ Show 20 Lines
char SIOptimizeExecMaskingPreRA::ID = 0; char SIOptimizeExecMaskingPreRA::ID = 0;
char &llvm::SIOptimizeExecMaskingPreRAID = SIOptimizeExecMaskingPreRA::ID; char &llvm::SIOptimizeExecMaskingPreRAID = SIOptimizeExecMaskingPreRA::ID;
FunctionPass *llvm::createSIOptimizeExecMaskingPreRAPass() { FunctionPass *llvm::createSIOptimizeExecMaskingPreRAPass() {
return new SIOptimizeExecMaskingPreRA(); return new SIOptimizeExecMaskingPreRA();
} }
static bool isEndCF(const MachineInstr& MI, const SIRegisterInfo* TRI) { static bool isEndCF(const MachineInstr &MI, const SIRegisterInfo *TRI,
const GCNSubtarget &ST) {
if (ST.isWave32()) {
return MI.getOpcode() == AMDGPU::S_OR_B32 &&
MI.modifiesRegister(AMDGPU::EXEC_LO, TRI);
}
return MI.getOpcode() == AMDGPU::S_OR_B64 && return MI.getOpcode() == AMDGPU::S_OR_B64 &&
MI.modifiesRegister(AMDGPU::EXEC, TRI); MI.modifiesRegister(AMDGPU::EXEC, TRI);
} }
static bool isFullExecCopy(const MachineInstr& MI) { static bool isFullExecCopy(const MachineInstr& MI, const GCNSubtarget& ST) {
if (MI.isCopy() && MI.getOperand(1).getReg() == AMDGPU::EXEC) { unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
if (MI.isCopy() && MI.getOperand(1).getReg() == Exec) {
assert(MI.isFullCopy()); assert(MI.isFullCopy());
return true; return true;
} }
return false; return false;
} }
static unsigned getOrNonExecReg(const MachineInstr &MI, static unsigned getOrNonExecReg(const MachineInstr &MI,
const SIInstrInfo &TII) { const SIInstrInfo &TII,
const GCNSubtarget& ST) {
unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
auto Op = TII.getNamedOperand(MI, AMDGPU::OpName::src1); auto Op = TII.getNamedOperand(MI, AMDGPU::OpName::src1);
if (Op->isReg() && Op->getReg() != AMDGPU::EXEC) if (Op->isReg() && Op->getReg() != Exec)
return Op->getReg(); return Op->getReg();
Op = TII.getNamedOperand(MI, AMDGPU::OpName::src0); Op = TII.getNamedOperand(MI, AMDGPU::OpName::src0);
if (Op->isReg() && Op->getReg() != AMDGPU::EXEC) if (Op->isReg() && Op->getReg() != Exec)
return Op->getReg(); return Op->getReg();
return AMDGPU::NoRegister; return AMDGPU::NoRegister;
} }
static MachineInstr* getOrExecSource(const MachineInstr &MI, static MachineInstr* getOrExecSource(const MachineInstr &MI,
const SIInstrInfo &TII, const SIInstrInfo &TII,
const MachineRegisterInfo &MRI) { const MachineRegisterInfo &MRI,
auto SavedExec = getOrNonExecReg(MI, TII); const GCNSubtarget& ST) {
auto SavedExec = getOrNonExecReg(MI, TII, ST);
if (SavedExec == AMDGPU::NoRegister) if (SavedExec == AMDGPU::NoRegister)
return nullptr; return nullptr;
auto SaveExecInst = MRI.getUniqueVRegDef(SavedExec); auto SaveExecInst = MRI.getUniqueVRegDef(SavedExec);
if (!SaveExecInst || !isFullExecCopy(*SaveExecInst)) if (!SaveExecInst || !isFullExecCopy(*SaveExecInst, ST))
return nullptr; return nullptr;
return SaveExecInst; return SaveExecInst;
} }
/// Skip over instructions that don't care about the exec mask. /// Skip over instructions that don't care about the exec mask.
MachineBasicBlock::iterator SIOptimizeExecMaskingPreRA::skipIgnoreExecInsts( MachineBasicBlock::iterator SIOptimizeExecMaskingPreRA::skipIgnoreExecInsts(
MachineBasicBlock::iterator I, MachineBasicBlock::iterator E) const { MachineBasicBlock::iterator I, MachineBasicBlock::iterator E) const {
for ( ; I != E; ++I) { for ( ; I != E; ++I) {
▲ Show 20 Lines • Show All 49 Lines • ▼ Show 20 Lines
// //
// Returns %cc register on success. // Returns %cc register on success.
static unsigned optimizeVcndVcmpPair(MachineBasicBlock &MBB, static unsigned optimizeVcndVcmpPair(MachineBasicBlock &MBB,
const GCNSubtarget &ST, const GCNSubtarget &ST,
MachineRegisterInfo &MRI, MachineRegisterInfo &MRI,
LiveIntervals *LIS) { LiveIntervals *LIS) {
const SIRegisterInfo *TRI = ST.getRegisterInfo(); const SIRegisterInfo *TRI = ST.getRegisterInfo();
const SIInstrInfo *TII = ST.getInstrInfo(); const SIInstrInfo *TII = ST.getInstrInfo();
const unsigned AndOpc = AMDGPU::S_AND_B64; bool Wave32 = ST.isWave32();
const unsigned Andn2Opc = AMDGPU::S_ANDN2_B64; const unsigned AndOpc = Wave32 ? AMDGPU::S_AND_B32 : AMDGPU::S_AND_B64;
const unsigned CondReg = AMDGPU::VCC; const unsigned Andn2Opc = Wave32 ? AMDGPU::S_ANDN2_B32 : AMDGPU::S_ANDN2_B64;
const unsigned ExecReg = AMDGPU::EXEC; const unsigned CondReg = Wave32 ? AMDGPU::VCC_LO : AMDGPU::VCC;
const unsigned ExecReg = Wave32 ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
auto I = llvm::find_if(MBB.terminators(), [](const MachineInstr &MI) { auto I = llvm::find_if(MBB.terminators(), [](const MachineInstr &MI) {
unsigned Opc = MI.getOpcode(); unsigned Opc = MI.getOpcode();
return Opc == AMDGPU::S_CBRANCH_VCCZ || return Opc == AMDGPU::S_CBRANCH_VCCZ ||
Opc == AMDGPU::S_CBRANCH_VCCNZ; }); Opc == AMDGPU::S_CBRANCH_VCCNZ; });
if (I == MBB.terminators().end()) if (I == MBB.terminators().end())
return AMDGPU::NoRegister; return AMDGPU::NoRegister;
▲ Show 20 Lines • Show All 90 Lines • ▼ Show 20 Linesbool SIOptimizeExecMaskingPreRA::runOnMachineFunction(MachineFunction &MF) {
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>(); const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
TRI = ST.getRegisterInfo(); TRI = ST.getRegisterInfo();
TII = ST.getInstrInfo(); TII = ST.getInstrInfo();
MRI = &MF.getRegInfo(); MRI = &MF.getRegInfo();
MachineRegisterInfo &MRI = MF.getRegInfo(); MachineRegisterInfo &MRI = MF.getRegInfo();
LiveIntervals *LIS = &getAnalysis<LiveIntervals>(); LiveIntervals *LIS = &getAnalysis<LiveIntervals>();
DenseSet<unsigned> RecalcRegs({AMDGPU::EXEC_LO, AMDGPU::EXEC_HI}); DenseSet<unsigned> RecalcRegs({AMDGPU::EXEC_LO, AMDGPU::EXEC_HI});
unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
bool Changed = false; bool Changed = false;
for (MachineBasicBlock &MBB : MF) { for (MachineBasicBlock &MBB : MF) {
if (unsigned Reg = optimizeVcndVcmpPair(MBB, ST, MRI, LIS)) { if (unsigned Reg = optimizeVcndVcmpPair(MBB, ST, MRI, LIS)) {
RecalcRegs.insert(Reg); RecalcRegs.insert(Reg);
RecalcRegs.insert(AMDGPU::VCC_LO); RecalcRegs.insert(AMDGPU::VCC_LO);
RecalcRegs.insert(AMDGPU::VCC_HI); RecalcRegs.insert(AMDGPU::VCC_HI);
▲ Show 20 Lines • Show All 62 Lines • ▼ Show 20 Lines if (MBB.succ_empty()) {
} }
} }
continue; continue;
} }
// Try to collapse adjacent endifs. // Try to collapse adjacent endifs.
auto E = MBB.end(); auto E = MBB.end();
auto Lead = skipDebugInstructionsForward(MBB.begin(), E); auto Lead = skipDebugInstructionsForward(MBB.begin(), E);
if (MBB.succ_size() != 1 || Lead == E || !isEndCF(*Lead, TRI)) if (MBB.succ_size() != 1 || Lead == E || !isEndCF(*Lead, TRI, ST))
continue; continue;
MachineBasicBlock *TmpMBB = &MBB; MachineBasicBlock *TmpMBB = &MBB;
auto NextLead = skipIgnoreExecInstsTrivialSucc(TmpMBB, std::next(Lead)); auto NextLead = skipIgnoreExecInstsTrivialSucc(TmpMBB, std::next(Lead));
if (NextLead == TmpMBB->end() || !isEndCF(*NextLead, TRI) || if (NextLead == TmpMBB->end() || !isEndCF(*NextLead, TRI, ST) ||
!getOrExecSource(*NextLead, *TII, MRI)) !getOrExecSource(*NextLead, *TII, MRI, ST))
continue; continue;
LLVM_DEBUG(dbgs() << "Redundant EXEC = S_OR_B64 found: " << *Lead << '\n'); LLVM_DEBUG(dbgs() << "Redundant EXEC = S_OR_B64 found: " << *Lead << '\n');
auto SaveExec = getOrExecSource(*Lead, *TII, MRI); auto SaveExec = getOrExecSource(*Lead, *TII, MRI, ST);
unsigned SaveExecReg = getOrNonExecReg(*Lead, *TII); unsigned SaveExecReg = getOrNonExecReg(*Lead, *TII, ST);
for (auto &Op : Lead->operands()) { for (auto &Op : Lead->operands()) {
if (Op.isReg()) if (Op.isReg())
RecalcRegs.insert(Op.getReg()); RecalcRegs.insert(Op.getReg());
} }
LIS->RemoveMachineInstrFromMaps(*Lead); LIS->RemoveMachineInstrFromMaps(*Lead);
Lead->eraseFromParent(); Lead->eraseFromParent();
if (SaveExecReg) { if (SaveExecReg) {
Show All 17 Lines for (auto& U : MRI.use_nodbg_instructions(SavedExec)) {
} }
LLVM_DEBUG(dbgs() << "Redundant EXEC COPY: " << *SaveExec << '\n'); LLVM_DEBUG(dbgs() << "Redundant EXEC COPY: " << *SaveExec << '\n');
} }
if (SafeToReplace) { if (SafeToReplace) {
LIS->RemoveMachineInstrFromMaps(*SaveExec); LIS->RemoveMachineInstrFromMaps(*SaveExec);
SaveExec->eraseFromParent(); SaveExec->eraseFromParent();
MRI.replaceRegWith(SavedExec, AMDGPU::EXEC); MRI.replaceRegWith(SavedExec, Exec);
LIS->removeInterval(SavedExec); LIS->removeInterval(SavedExec);
} }
} }
if (Changed) { if (Changed) {
for (auto Reg : RecalcRegs) { for (auto Reg : RecalcRegs) {
if (TargetRegisterInfo::isVirtualRegister(Reg)) { if (TargetRegisterInfo::isVirtualRegister(Reg)) {
LIS->removeInterval(Reg); LIS->removeInterval(Reg);
Show All 10 Lines

lib/Target/AMDGPU/SIPeepholeSDWA.cpp

Show First 20 Lines • Show All 948 Lines • ▼ Show 20 Lines if (AMDGPU::getSDWAOp(Opc) == -1)
return false; return false;
if (!ST.hasSDWAOmod() && TII->hasModifiersSet(MI, AMDGPU::OpName::omod)) if (!ST.hasSDWAOmod() && TII->hasModifiersSet(MI, AMDGPU::OpName::omod))
return false; return false;
if (TII->isVOPC(Opc)) { if (TII->isVOPC(Opc)) {
if (!ST.hasSDWASdst()) { if (!ST.hasSDWASdst()) {
const MachineOperand *SDst = TII->getNamedOperand(MI, AMDGPU::OpName::sdst); const MachineOperand *SDst = TII->getNamedOperand(MI, AMDGPU::OpName::sdst);
if (SDst && SDst->getReg() != AMDGPU::VCC) if (SDst && (SDst->getReg() != AMDGPU::VCC &&
SDst->getReg() != AMDGPU::VCC_LO))
return false; return false;
} }
if (!ST.hasSDWAOutModsVOPC() && if (!ST.hasSDWAOutModsVOPC() &&
(TII->hasModifiersSet(MI, AMDGPU::OpName::clamp) || (TII->hasModifiersSet(MI, AMDGPU::OpName::clamp) ||
TII->hasModifiersSet(MI, AMDGPU::OpName::omod))) TII->hasModifiersSet(MI, AMDGPU::OpName::omod)))
return false; return false;
▲ Show 20 Lines • Show All 48 Lines • ▼ Show 20 Lines if (Dst) {
assert(AMDGPU::getNamedOperandIdx(SDWAOpcode, AMDGPU::OpName::vdst) != -1); assert(AMDGPU::getNamedOperandIdx(SDWAOpcode, AMDGPU::OpName::vdst) != -1);
SDWAInst.add(*Dst); SDWAInst.add(*Dst);
} else if ((Dst = TII->getNamedOperand(MI, AMDGPU::OpName::sdst))) { } else if ((Dst = TII->getNamedOperand(MI, AMDGPU::OpName::sdst))) {
assert(Dst && assert(Dst &&
AMDGPU::getNamedOperandIdx(SDWAOpcode, AMDGPU::OpName::sdst) != -1); AMDGPU::getNamedOperandIdx(SDWAOpcode, AMDGPU::OpName::sdst) != -1);
SDWAInst.add(*Dst); SDWAInst.add(*Dst);
} else { } else {
assert(AMDGPU::getNamedOperandIdx(SDWAOpcode, AMDGPU::OpName::sdst) != -1); assert(AMDGPU::getNamedOperandIdx(SDWAOpcode, AMDGPU::OpName::sdst) != -1);
SDWAInst.addReg(AMDGPU::VCC, RegState::Define); SDWAInst.addReg(TRI->getVCC(), RegState::Define);
} }
// Copy src0, initialize src0_modifiers. All sdwa instructions has src0 and // Copy src0, initialize src0_modifiers. All sdwa instructions has src0 and
// src0_modifiers (except for v_nop_sdwa, but it can't get here) // src0_modifiers (except for v_nop_sdwa, but it can't get here)
MachineOperand *Src0 = TII->getNamedOperand(MI, AMDGPU::OpName::src0); MachineOperand *Src0 = TII->getNamedOperand(MI, AMDGPU::OpName::src0);
assert( assert(
Src0 && Src0 &&
AMDGPU::getNamedOperandIdx(SDWAOpcode, AMDGPU::OpName::src0) != -1 && AMDGPU::getNamedOperandIdx(SDWAOpcode, AMDGPU::OpName::src0) != -1 &&
▲ Show 20 Lines • Show All 232 Lines • Show Last 20 Lines

lib/Target/AMDGPU/SIRegisterInfo.h

Show All 27 Lines
class SIRegisterInfo final : public AMDGPURegisterInfo { class SIRegisterInfo final : public AMDGPURegisterInfo {
private: private:
unsigned SGPRSetID; unsigned SGPRSetID;
unsigned VGPRSetID; unsigned VGPRSetID;
BitVector SGPRPressureSets; BitVector SGPRPressureSets;
BitVector VGPRPressureSets; BitVector VGPRPressureSets;
bool SpillSGPRToVGPR; bool SpillSGPRToVGPR;
bool SpillSGPRToSMEM; bool SpillSGPRToSMEM;
bool isWave32;
void classifyPressureSet(unsigned PSetID, unsigned Reg, void classifyPressureSet(unsigned PSetID, unsigned Reg,
BitVector &PressureSets) const; BitVector &PressureSets) const;
public: public:
SIRegisterInfo(const GCNSubtarget &ST); SIRegisterInfo(const GCNSubtarget &ST);
bool spillSGPRToVGPR() const { bool spillSGPRToVGPR() const {
return SpillSGPRToVGPR; return SpillSGPRToVGPR;
▲ Show 20 Lines • Show All 182 Lines • ▼ Show 20 Linespublic:
const int *getRegUnitPressureSets(unsigned RegUnit) const override; const int *getRegUnitPressureSets(unsigned RegUnit) const override;
unsigned getReturnAddressReg(const MachineFunction &MF) const; unsigned getReturnAddressReg(const MachineFunction &MF) const;
const TargetRegisterClass * const TargetRegisterClass *
getConstrainedRegClassForOperand(const MachineOperand &MO, getConstrainedRegClassForOperand(const MachineOperand &MO,
const MachineRegisterInfo &MRI) const override; const MachineRegisterInfo &MRI) const override;
const TargetRegisterClass *getBoolRC() const {
return isWave32 ? &AMDGPU::SReg_32_XM0RegClass
: &AMDGPU::SReg_64RegClass;
}
const TargetRegisterClass *getWaveMaskRegClass() const {
return isWave32 ? &AMDGPU::SReg_32_XM0_XEXECRegClass
: &AMDGPU::SReg_64_XEXECRegClass;
}
unsigned getVCC() const;
const TargetRegisterClass *getRegClass(unsigned RCID) const;
// Find reaching register definition // Find reaching register definition
MachineInstr *findReachingDef(unsigned Reg, unsigned SubReg, MachineInstr *findReachingDef(unsigned Reg, unsigned SubReg,
MachineInstr &Use, MachineInstr &Use,
MachineRegisterInfo &MRI, MachineRegisterInfo &MRI,
LiveIntervals *LIS) const; LiveIntervals *LIS) const;
private: private:
void buildSpillLoadStore(MachineBasicBlock::iterator MI, void buildSpillLoadStore(MachineBasicBlock::iterator MI,
Show All 14 Lines

lib/Target/AMDGPU/SIRegisterInfo.cpp

Show First 20 Lines • Show All 57 Lines • ▼ Show 20 Linesstatic cl::opt<bool> EnableSpillSGPRToVGPR(
cl::ReallyHidden, cl::ReallyHidden,
cl::init(true)); cl::init(true));
SIRegisterInfo::SIRegisterInfo(const GCNSubtarget &ST) : SIRegisterInfo::SIRegisterInfo(const GCNSubtarget &ST) :
AMDGPURegisterInfo(), AMDGPURegisterInfo(),
SGPRPressureSets(getNumRegPressureSets()), SGPRPressureSets(getNumRegPressureSets()),
VGPRPressureSets(getNumRegPressureSets()), VGPRPressureSets(getNumRegPressureSets()),
SpillSGPRToVGPR(false), SpillSGPRToVGPR(false),
SpillSGPRToSMEM(false) { SpillSGPRToSMEM(false),
isWave32(ST.isWave32()) {
if (EnableSpillSGPRToSMEM && ST.hasScalarStores()) if (EnableSpillSGPRToSMEM && ST.hasScalarStores())
SpillSGPRToSMEM = true; SpillSGPRToSMEM = true;
else if (EnableSpillSGPRToVGPR) else if (EnableSpillSGPRToVGPR)
SpillSGPRToVGPR = true; SpillSGPRToVGPR = true;
unsigned NumRegPressureSets = getNumRegPressureSets(); unsigned NumRegPressureSets = getNumRegPressureSets();
SGPRSetID = NumRegPressureSets; SGPRSetID = NumRegPressureSets;
▲ Show 20 Lines • Show All 104 Lines • ▼ Show 20 LinesBitVector SIRegisterInfo::getReservedRegs(const MachineFunction &MF) const {
reserveRegisterTuples(Reserved, AMDGPU::TTMP8_TTMP9); reserveRegisterTuples(Reserved, AMDGPU::TTMP8_TTMP9);
reserveRegisterTuples(Reserved, AMDGPU::TTMP10_TTMP11); reserveRegisterTuples(Reserved, AMDGPU::TTMP10_TTMP11);
reserveRegisterTuples(Reserved, AMDGPU::TTMP12_TTMP13); reserveRegisterTuples(Reserved, AMDGPU::TTMP12_TTMP13);
reserveRegisterTuples(Reserved, AMDGPU::TTMP14_TTMP15); reserveRegisterTuples(Reserved, AMDGPU::TTMP14_TTMP15);
// Reserve null register - it shall never be allocated // Reserve null register - it shall never be allocated
reserveRegisterTuples(Reserved, AMDGPU::SGPR_NULL); reserveRegisterTuples(Reserved, AMDGPU::SGPR_NULL);
// Disallow vcc_hi allocation in wave32. It may be allocated but most likely
// will result in bugs.
if (isWave32) {
Reserved.set(AMDGPU::VCC);
Reserved.set(AMDGPU::VCC_HI);
}
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>(); const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
unsigned MaxNumSGPRs = ST.getMaxNumSGPRs(MF); unsigned MaxNumSGPRs = ST.getMaxNumSGPRs(MF);
unsigned TotalNumSGPRs = AMDGPU::SGPR_32RegClass.getNumRegs(); unsigned TotalNumSGPRs = AMDGPU::SGPR_32RegClass.getNumRegs();
for (unsigned i = MaxNumSGPRs; i < TotalNumSGPRs; ++i) { for (unsigned i = MaxNumSGPRs; i < TotalNumSGPRs; ++i) {
unsigned Reg = AMDGPU::SGPR_32RegClass.getRegister(i); unsigned Reg = AMDGPU::SGPR_32RegClass.getRegister(i);
reserveRegisterTuples(Reserved, Reg); reserveRegisterTuples(Reserved, Reg);
} }
▲ Show 20 Lines • Show All 1,506 Lines • ▼ Show 20 LinesSIRegisterInfo::getConstrainedRegClassForOperand(const MachineOperand &MO,
case 512: case 512:
return RB->getID() == AMDGPU::VGPRRegBankID ? &AMDGPU::VReg_512RegClass : return RB->getID() == AMDGPU::VGPRRegBankID ? &AMDGPU::VReg_512RegClass :
&AMDGPU::SReg_512RegClass; &AMDGPU::SReg_512RegClass;
default: default:
llvm_unreachable("not implemented"); llvm_unreachable("not implemented");
} }
} }
unsigned SIRegisterInfo::getVCC() const {
return isWave32 ? AMDGPU::VCC_LO : AMDGPU::VCC;
}
const TargetRegisterClass *
SIRegisterInfo::getRegClass(unsigned RCID) const {
switch ((int)RCID) {
case AMDGPU::SReg_1RegClassID:
return getBoolRC();
case AMDGPU::SReg_1_XEXECRegClassID:
return isWave32 ? &AMDGPU::SReg_32_XM0_XEXECRegClass
: &AMDGPU::SReg_64_XEXECRegClass;
case -1:
return nullptr;
default:
return AMDGPURegisterInfo::getRegClass(RCID);
}
}
// Find reaching register definition // Find reaching register definition
MachineInstr *SIRegisterInfo::findReachingDef(unsigned Reg, unsigned SubReg, MachineInstr *SIRegisterInfo::findReachingDef(unsigned Reg, unsigned SubReg,
MachineInstr &Use, MachineInstr &Use,
MachineRegisterInfo &MRI, MachineRegisterInfo &MRI,
LiveIntervals *LIS) const { LiveIntervals *LIS) const {
auto &MDT = LIS->getAnalysis<MachineDominatorTree>(); auto &MDT = LIS->getAnalysis<MachineDominatorTree>();
SlotIndex UseIdx = LIS->getInstructionIndex(Use); SlotIndex UseIdx = LIS->getInstructionIndex(Use);
SlotIndex DefIdx; SlotIndex DefIdx;
▲ Show 20 Lines • Show All 45 Lines • Show Last 20 Lines

lib/Target/AMDGPU/SIRegisterInfo.td

Show First 20 Lines • Show All 448 Lines • ▼ Show 20 Lines
def LDS_DIRECT_CLASS : RegisterClass<"AMDGPU", [i32, f32, i16, f16, v2i16, v2f16], 32, def LDS_DIRECT_CLASS : RegisterClass<"AMDGPU", [i32, f32, i16, f16, v2i16, v2f16], 32,
(add LDS_DIRECT)> { (add LDS_DIRECT)> {
let isAllocatable = 0; let isAllocatable = 0;
let CopyCost = -1; let CopyCost = -1;
} }
// Subset of SReg_32 without M0 for SMRD instructions and alike. // Subset of SReg_32 without M0 for SMRD instructions and alike.
// See comments in SIInstructions.td for more info. // See comments in SIInstructions.td for more info.
def SReg_32_XM0_XEXEC : RegisterClass<"AMDGPU", [i32, f32, i16, f16, v2i16, v2f16], 32, def SReg_32_XM0_XEXEC : RegisterClass<"AMDGPU", [i32, f32, i16, f16, v2i16, v2f16, i1], 32,
(add SGPR_32, VCC_LO, VCC_HI, FLAT_SCR_LO, FLAT_SCR_HI, XNACK_MASK_LO, XNACK_MASK_HI, (add SGPR_32, VCC_LO, VCC_HI, FLAT_SCR_LO, FLAT_SCR_HI, XNACK_MASK_LO, XNACK_MASK_HI,
SGPR_NULL, TTMP_32, TMA_LO, TMA_HI, TBA_LO, TBA_HI, SRC_SHARED_BASE, SRC_SHARED_LIMIT, SGPR_NULL, TTMP_32, TMA_LO, TMA_HI, TBA_LO, TBA_HI, SRC_SHARED_BASE, SRC_SHARED_LIMIT,
SRC_PRIVATE_BASE, SRC_PRIVATE_LIMIT, SRC_POPS_EXITING_WAVE_ID, SRC_PRIVATE_BASE, SRC_PRIVATE_LIMIT, SRC_POPS_EXITING_WAVE_ID,
SRC_VCCZ, SRC_EXECZ, SRC_SCC)> { SRC_VCCZ, SRC_EXECZ, SRC_SCC)> {
let AllocationPriority = 8; let AllocationPriority = 8;
} }
def SReg_32_XEXEC_HI : RegisterClass<"AMDGPU", [i32, f32, i16, f16, v2i16, v2f16], 32, def SReg_32_XEXEC_HI : RegisterClass<"AMDGPU", [i32, f32, i16, f16, v2i16, v2f16, i1], 32,
(add SReg_32_XM0_XEXEC, EXEC_LO, M0_CLASS)> { (add SReg_32_XM0_XEXEC, EXEC_LO, M0_CLASS)> {
let AllocationPriority = 8; let AllocationPriority = 8;
} }
def SReg_32_XM0 : RegisterClass<"AMDGPU", [i32, f32, i16, f16, v2i16, v2f16], 32, def SReg_32_XM0 : RegisterClass<"AMDGPU", [i32, f32, i16, f16, v2i16, v2f16, i1], 32,
(add SReg_32_XM0_XEXEC, EXEC_LO, EXEC_HI)> { (add SReg_32_XM0_XEXEC, EXEC_LO, EXEC_HI)> {
let AllocationPriority = 8; let AllocationPriority = 8;
} }
// Register class for all scalar registers (SGPRs + Special Registers) // Register class for all scalar registers (SGPRs + Special Registers)
def SReg_32 : RegisterClass<"AMDGPU", [i32, f32, i16, f16, v2i16, v2f16], 32, def SReg_32 : RegisterClass<"AMDGPU", [i32, f32, i16, f16, v2i16, v2f16, i1], 32,
(add SReg_32_XM0, M0_CLASS, EXEC_LO, EXEC_HI, SReg_32_XEXEC_HI)> { (add SReg_32_XM0, M0_CLASS, EXEC_LO, EXEC_HI, SReg_32_XEXEC_HI)> {
let AllocationPriority = 8; let AllocationPriority = 8;
} }
def SRegOrLds_32 : RegisterClass<"AMDGPU", [i32, f32, i16, f16, v2i16, v2f16], 32, def SRegOrLds_32 : RegisterClass<"AMDGPU", [i32, f32, i16, f16, v2i16, v2f16, i1], 32,
(add SReg_32_XM0, M0_CLASS, EXEC_LO, EXEC_HI, SReg_32_XEXEC_HI, LDS_DIRECT_CLASS)> { (add SReg_32_XM0, M0_CLASS, EXEC_LO, EXEC_HI, SReg_32_XEXEC_HI, LDS_DIRECT_CLASS)> {
let isAllocatable = 0; let isAllocatable = 0;
} }
def SGPR_64 : RegisterClass<"AMDGPU", [v2i32, i64, v2f32, f64, v4i16, v4f16], 32, (add SGPR_64Regs)> { def SGPR_64 : RegisterClass<"AMDGPU", [v2i32, i64, v2f32, f64, v4i16, v4f16], 32, (add SGPR_64Regs)> {
let CopyCost = 1; let CopyCost = 1;
let AllocationPriority = 9; let AllocationPriority = 9;
} }
Show All 9 Lines
} }
def SReg_64 : RegisterClass<"AMDGPU", [v2i32, i64, v2f32, f64, i1, v4i16, v4f16], 32, def SReg_64 : RegisterClass<"AMDGPU", [v2i32, i64, v2f32, f64, i1, v4i16, v4f16], 32,
(add SReg_64_XEXEC, EXEC)> { (add SReg_64_XEXEC, EXEC)> {
let CopyCost = 1; let CopyCost = 1;
let AllocationPriority = 9; let AllocationPriority = 9;
} }
def SReg_1_XEXEC : RegisterClass<"AMDGPU", [i1], 32,
(add SReg_64_XEXEC, SReg_32_XM0_XEXEC)> {
let CopyCost = 1;
let isAllocatable = 0;
}
def SReg_1 : RegisterClass<"AMDGPU", [i1], 32,
(add SReg_1_XEXEC, EXEC, EXEC_LO)> {
let CopyCost = 1;
let isAllocatable = 0;
}
// Requires 2 s_mov_b64 to copy // Requires 2 s_mov_b64 to copy
let CopyCost = 2 in { let CopyCost = 2 in {
// There are no 3-component scalar instructions, but this is needed // There are no 3-component scalar instructions, but this is needed
// for symmetry with VGPRs. // for symmetry with VGPRs.
def SGPR_96 : RegisterClass<"AMDGPU", [v3i32, v3f32], 32, def SGPR_96 : RegisterClass<"AMDGPU", [v3i32, v3f32], 32,
(add SGPR_96Regs)> { (add SGPR_96Regs)> {
let AllocationPriority = 10; let AllocationPriority = 10;
▲ Show 20 Lines • Show All 201 Lines • ▼ Show 20 Lines
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SCSrc_* Operands with an SGPR or a inline constant // SCSrc_* Operands with an SGPR or a inline constant
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
defm SCSrc : RegInlineOperand<"SReg", "SCSrc"> ; defm SCSrc : RegInlineOperand<"SReg", "SCSrc"> ;
def SCSrc_i1 : RegisterOperand<SReg_64_XEXEC>;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// VSrc_* Operands with an SGPR, VGPR or a 32-bit immediate // VSrc_* Operands with an SGPR, VGPR or a 32-bit immediate
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
defm VSrc : RegImmOperand<"VS", "VSrc">; defm VSrc : RegImmOperand<"VS", "VSrc">;
def VSrc_128 : RegisterOperand<VReg_128> { def VSrc_128 : RegisterOperand<VReg_128> {
let DecoderMethod = "DecodeVS_128RegisterClass"; let DecoderMethod = "DecodeVS_128RegisterClass";
Show All 18 Lines

lib/Target/AMDGPU/SIShrinkInstructions.cpp

Show First 20 Lines • Show All 545 Lines • ▼ Show 20 Lines
bool SIShrinkInstructions::runOnMachineFunction(MachineFunction &MF) { bool SIShrinkInstructions::runOnMachineFunction(MachineFunction &MF) {
if (skipFunction(MF.getFunction())) if (skipFunction(MF.getFunction()))
return false; return false;
MachineRegisterInfo &MRI = MF.getRegInfo(); MachineRegisterInfo &MRI = MF.getRegInfo();
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>(); const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
const SIInstrInfo *TII = ST.getInstrInfo(); const SIInstrInfo *TII = ST.getInstrInfo();
unsigned VCCReg = ST.isWave32() ? AMDGPU::VCC_LO : AMDGPU::VCC;
std::vector<unsigned> I1Defs; std::vector<unsigned> I1Defs;
for (MachineFunction::iterator BI = MF.begin(), BE = MF.end(); for (MachineFunction::iterator BI = MF.begin(), BE = MF.end();
BI != BE; ++BI) { BI != BE; ++BI) {
MachineBasicBlock &MBB = *BI; MachineBasicBlock &MBB = *BI;
MachineBasicBlock::iterator I, Next; MachineBasicBlock::iterator I, Next;
▲ Show 20 Lines • Show All 159 Lines • ▼ Show 20 Lines for (I = MBB.begin(); I != MBB.end(); I = Next) {
// VOPC instructions can only write to the VCC register. We can't // VOPC instructions can only write to the VCC register. We can't
// force them to use VCC here, because this is only one register and // force them to use VCC here, because this is only one register and
// cannot deal with sequences which would require multiple copies of // cannot deal with sequences which would require multiple copies of
// VCC, e.g. S_AND_B64 (vcc = V_CMP_...), (vcc = V_CMP_...) // VCC, e.g. S_AND_B64 (vcc = V_CMP_...), (vcc = V_CMP_...)
// //
// So, instead of forcing the instruction to write to VCC, we provide // So, instead of forcing the instruction to write to VCC, we provide
// a hint to the register allocator to use VCC and then we will run // a hint to the register allocator to use VCC and then we will run
// this pass again after RA and shrink it if it outputs to VCC. // this pass again after RA and shrink it if it outputs to VCC.
MRI.setRegAllocationHint(MI.getOperand(0).getReg(), 0, AMDGPU::VCC); MRI.setRegAllocationHint(MI.getOperand(0).getReg(), 0, VCCReg);
continue; continue;
} }
if (DstReg != AMDGPU::VCC) if (DstReg != VCCReg)
continue; continue;
} }
if (Op32 == AMDGPU::V_CNDMASK_B32_e32) { if (Op32 == AMDGPU::V_CNDMASK_B32_e32) {
// We shrink V_CNDMASK_B32_e64 using regalloc hints like we do for VOPC // We shrink V_CNDMASK_B32_e64 using regalloc hints like we do for VOPC
// instructions. // instructions.
const MachineOperand *Src2 = const MachineOperand *Src2 =
TII->getNamedOperand(MI, AMDGPU::OpName::src2); TII->getNamedOperand(MI, AMDGPU::OpName::src2);
if (!Src2->isReg()) if (!Src2->isReg())
continue; continue;
unsigned SReg = Src2->getReg(); unsigned SReg = Src2->getReg();
if (TargetRegisterInfo::isVirtualRegister(SReg)) { if (TargetRegisterInfo::isVirtualRegister(SReg)) {
MRI.setRegAllocationHint(SReg, 0, AMDGPU::VCC); MRI.setRegAllocationHint(SReg, 0, VCCReg);
continue; continue;
} }
if (SReg != AMDGPU::VCC) if (SReg != VCCReg)
continue; continue;
} }
// Check for the bool flag output for instructions like V_ADD_I32_e64. // Check for the bool flag output for instructions like V_ADD_I32_e64.
const MachineOperand *SDst = TII->getNamedOperand(MI, const MachineOperand *SDst = TII->getNamedOperand(MI,
AMDGPU::OpName::sdst); AMDGPU::OpName::sdst);
// Check the carry-in operand for v_addc_u32_e64. // Check the carry-in operand for v_addc_u32_e64.
const MachineOperand *Src2 = TII->getNamedOperand(MI, const MachineOperand *Src2 = TII->getNamedOperand(MI,
AMDGPU::OpName::src2); AMDGPU::OpName::src2);
if (SDst) { if (SDst) {
if (SDst->getReg() != AMDGPU::VCC) { bool Next = false;
if (SDst->getReg() != VCCReg) {
if (TargetRegisterInfo::isVirtualRegister(SDst->getReg())) if (TargetRegisterInfo::isVirtualRegister(SDst->getReg()))
MRI.setRegAllocationHint(SDst->getReg(), 0, AMDGPU::VCC); MRI.setRegAllocationHint(SDst->getReg(), 0, VCCReg);
continue; Next = true;
} }
// All of the instructions with carry outs also have an SGPR input in // All of the instructions with carry outs also have an SGPR input in
// src2. // src2.
if (Src2 && Src2->getReg() != AMDGPU::VCC) { if (Src2 && Src2->getReg() != VCCReg) {
if (TargetRegisterInfo::isVirtualRegister(Src2->getReg())) if (TargetRegisterInfo::isVirtualRegister(Src2->getReg()))
MRI.setRegAllocationHint(Src2->getReg(), 0, AMDGPU::VCC); MRI.setRegAllocationHint(Src2->getReg(), 0, VCCReg);
Next = true;
}
if (Next)
continue; continue;
} }
}
// We can shrink this instruction // We can shrink this instruction
LLVM_DEBUG(dbgs() << "Shrinking " << MI); LLVM_DEBUG(dbgs() << "Shrinking " << MI);
MachineInstr *Inst32 = TII->buildShrunkInst(MI, Op32); MachineInstr *Inst32 = TII->buildShrunkInst(MI, Op32);
++NumInstructionsShrunk; ++NumInstructionsShrunk;
// Copy extra operands not present in the instruction definition. // Copy extra operands not present in the instruction definition.
Show All 10 Lines

lib/Target/AMDGPU/SIWholeQuadMode.cpp

Show First 20 Lines • Show All 142 Lines • ▼ Show 20 Linesstruct WorkItem {
WorkItem(MachineInstr *MI) : MI(MI) {} WorkItem(MachineInstr *MI) : MI(MI) {}
}; };
class SIWholeQuadMode : public MachineFunctionPass { class SIWholeQuadMode : public MachineFunctionPass {
private: private:
CallingConv::ID CallingConv; CallingConv::ID CallingConv;
const SIInstrInfo *TII; const SIInstrInfo *TII;
const SIRegisterInfo *TRI; const SIRegisterInfo *TRI;
const GCNSubtarget *ST;
MachineRegisterInfo *MRI; MachineRegisterInfo *MRI;
LiveIntervals *LIS; LiveIntervals *LIS;
DenseMap<const MachineInstr *, InstrInfo> Instructions; DenseMap<const MachineInstr *, InstrInfo> Instructions;
DenseMap<MachineBasicBlock *, BlockInfo> Blocks; DenseMap<MachineBasicBlock *, BlockInfo> Blocks;
SmallVector<MachineInstr *, 1> LiveMaskQueries; SmallVector<MachineInstr *, 1> LiveMaskQueries;
SmallVector<MachineInstr *, 4> LowerToCopyInstrs; SmallVector<MachineInstr *, 4> LowerToCopyInstrs;
▲ Show 20 Lines • Show All 114 Lines • ▼ Show 20 Lines if (!Use.isReg() || !Use.isUse())
continue; continue;
unsigned Reg = Use.getReg(); unsigned Reg = Use.getReg();
// Handle physical registers that we need to track; this is mostly relevant // Handle physical registers that we need to track; this is mostly relevant
// for VCC, which can appear as the (implicit) input of a uniform branch, // for VCC, which can appear as the (implicit) input of a uniform branch,
// e.g. when a loop counter is stored in a VGPR. // e.g. when a loop counter is stored in a VGPR.
if (!TargetRegisterInfo::isVirtualRegister(Reg)) { if (!TargetRegisterInfo::isVirtualRegister(Reg)) {
if (Reg == AMDGPU::EXEC) if (Reg == AMDGPU::EXEC || Reg == AMDGPU::EXEC_LO)
continue; continue;
for (MCRegUnitIterator RegUnit(Reg, TRI); RegUnit.isValid(); ++RegUnit) { for (MCRegUnitIterator RegUnit(Reg, TRI); RegUnit.isValid(); ++RegUnit) {
LiveRange &LR = LIS->getRegUnit(*RegUnit); LiveRange &LR = LIS->getRegUnit(*RegUnit);
const VNInfo *Value = LR.Query(LIS->getInstructionIndex(MI)).valueIn(); const VNInfo *Value = LR.Query(LIS->getInstructionIndex(MI)).valueIn();
if (!Value) if (!Value)
continue; continue;
▲ Show 20 Lines • Show All 325 Lines • ▼ Show 20 Lines
} }
void SIWholeQuadMode::toExact(MachineBasicBlock &MBB, void SIWholeQuadMode::toExact(MachineBasicBlock &MBB,
MachineBasicBlock::iterator Before, MachineBasicBlock::iterator Before,
unsigned SaveWQM, unsigned LiveMaskReg) { unsigned SaveWQM, unsigned LiveMaskReg) {
MachineInstr *MI; MachineInstr *MI;
if (SaveWQM) { if (SaveWQM) {
MI = BuildMI(MBB, Before, DebugLoc(), TII->get(AMDGPU::S_AND_SAVEEXEC_B64), MI = BuildMI(MBB, Before, DebugLoc(), TII->get(ST->isWave32() ?
AMDGPU::S_AND_SAVEEXEC_B32 : AMDGPU::S_AND_SAVEEXEC_B64),
SaveWQM) SaveWQM)
.addReg(LiveMaskReg); .addReg(LiveMaskReg);
} else { } else {
MI = BuildMI(MBB, Before, DebugLoc(), TII->get(AMDGPU::S_AND_B64), unsigned Exec = ST->isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
AMDGPU::EXEC) MI = BuildMI(MBB, Before, DebugLoc(), TII->get(ST->isWave32() ?
.addReg(AMDGPU::EXEC) AMDGPU::S_AND_B32 : AMDGPU::S_AND_B64),
Exec)
.addReg(Exec)
.addReg(LiveMaskReg); .addReg(LiveMaskReg);
} }
LIS->InsertMachineInstrInMaps(*MI); LIS->InsertMachineInstrInMaps(*MI);
} }
void SIWholeQuadMode::toWQM(MachineBasicBlock &MBB, void SIWholeQuadMode::toWQM(MachineBasicBlock &MBB,
MachineBasicBlock::iterator Before, MachineBasicBlock::iterator Before,
unsigned SavedWQM) { unsigned SavedWQM) {
MachineInstr *MI; MachineInstr *MI;
unsigned Exec = ST->isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
if (SavedWQM) { if (SavedWQM) {
MI = BuildMI(MBB, Before, DebugLoc(), TII->get(AMDGPU::COPY), AMDGPU::EXEC) MI = BuildMI(MBB, Before, DebugLoc(), TII->get(AMDGPU::COPY), Exec)
.addReg(SavedWQM); .addReg(SavedWQM);
} else { } else {
MI = BuildMI(MBB, Before, DebugLoc(), TII->get(AMDGPU::S_WQM_B64), MI = BuildMI(MBB, Before, DebugLoc(), TII->get(ST->isWave32() ?
AMDGPU::EXEC) AMDGPU::S_WQM_B32 : AMDGPU::S_WQM_B64),
.addReg(AMDGPU::EXEC); Exec)
.addReg(Exec);
} }
LIS->InsertMachineInstrInMaps(*MI); LIS->InsertMachineInstrInMaps(*MI);
} }
void SIWholeQuadMode::toWWM(MachineBasicBlock &MBB, void SIWholeQuadMode::toWWM(MachineBasicBlock &MBB,
MachineBasicBlock::iterator Before, MachineBasicBlock::iterator Before,
unsigned SaveOrig) { unsigned SaveOrig) {
MachineInstr *MI; MachineInstr *MI;
assert(SaveOrig); assert(SaveOrig);
MI = BuildMI(MBB, Before, DebugLoc(), TII->get(AMDGPU::ENTER_WWM), SaveOrig) MI = BuildMI(MBB, Before, DebugLoc(), TII->get(AMDGPU::ENTER_WWM), SaveOrig)
.addImm(-1); .addImm(-1);
LIS->InsertMachineInstrInMaps(*MI); LIS->InsertMachineInstrInMaps(*MI);
} }
void SIWholeQuadMode::fromWWM(MachineBasicBlock &MBB, void SIWholeQuadMode::fromWWM(MachineBasicBlock &MBB,
MachineBasicBlock::iterator Before, MachineBasicBlock::iterator Before,
unsigned SavedOrig) { unsigned SavedOrig) {
MachineInstr *MI; MachineInstr *MI;
assert(SavedOrig); assert(SavedOrig);
MI = BuildMI(MBB, Before, DebugLoc(), TII->get(AMDGPU::EXIT_WWM), AMDGPU::EXEC) MI = BuildMI(MBB, Before, DebugLoc(), TII->get(AMDGPU::EXIT_WWM),
ST->isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC)
.addReg(SavedOrig); .addReg(SavedOrig);
LIS->InsertMachineInstrInMaps(*MI); LIS->InsertMachineInstrInMaps(*MI);
} }
void SIWholeQuadMode::processBlock(MachineBasicBlock &MBB, unsigned LiveMaskReg, void SIWholeQuadMode::processBlock(MachineBasicBlock &MBB, unsigned LiveMaskReg,
bool isEntry) { bool isEntry) {
auto BII = Blocks.find(&MBB); auto BII = Blocks.find(&MBB);
if (BII == Blocks.end()) if (BII == Blocks.end())
Show All 9 Linesvoid SIWholeQuadMode::processBlock(MachineBasicBlock &MBB, unsigned LiveMaskReg,
LLVM_DEBUG(dbgs() << "\nProcessing block " << printMBBReference(MBB) LLVM_DEBUG(dbgs() << "\nProcessing block " << printMBBReference(MBB)
<< ":\n"); << ":\n");
unsigned SavedWQMReg = 0; unsigned SavedWQMReg = 0;
unsigned SavedNonWWMReg = 0; unsigned SavedNonWWMReg = 0;
bool WQMFromExec = isEntry; bool WQMFromExec = isEntry;
char State = (isEntry || !(BI.InNeeds & StateWQM)) ? StateExact : StateWQM; char State = (isEntry || !(BI.InNeeds & StateWQM)) ? StateExact : StateWQM;
char NonWWMState = 0; char NonWWMState = 0;
const TargetRegisterClass *BoolRC = TRI->getBoolRC();
auto II = MBB.getFirstNonPHI(), IE = MBB.end(); auto II = MBB.getFirstNonPHI(), IE = MBB.end();
if (isEntry) if (isEntry)
++II; // Skip the instruction that saves LiveMask ++II; // Skip the instruction that saves LiveMask
// This stores the first instruction where it's safe to switch from WQM to // This stores the first instruction where it's safe to switch from WQM to
// Exact or vice versa. // Exact or vice versa.
MachineBasicBlock::iterator FirstWQM = IE; MachineBasicBlock::iterator FirstWQM = IE;
▲ Show 20 Lines • Show All 71 Lines • ▼ Show 20 Lines if (!(Needs & State)) {
if (State == StateWWM) { if (State == StateWWM) {
assert(SavedNonWWMReg); assert(SavedNonWWMReg);
fromWWM(MBB, Before, SavedNonWWMReg); fromWWM(MBB, Before, SavedNonWWMReg);
State = NonWWMState; State = NonWWMState;
} }
if (Needs == StateWWM) { if (Needs == StateWWM) {
NonWWMState = State; NonWWMState = State;
SavedNonWWMReg = MRI->createVirtualRegister(&AMDGPU::SReg_64RegClass); SavedNonWWMReg = MRI->createVirtualRegister(BoolRC);
toWWM(MBB, Before, SavedNonWWMReg); toWWM(MBB, Before, SavedNonWWMReg);
State = StateWWM; State = StateWWM;
} else { } else {
if (State == StateWQM && (Needs & StateExact) && !(Needs & StateWQM)) { if (State == StateWQM && (Needs & StateExact) && !(Needs & StateWQM)) {
if (!WQMFromExec && (OutNeeds & StateWQM)) if (!WQMFromExec && (OutNeeds & StateWQM))
SavedWQMReg = MRI->createVirtualRegister(&AMDGPU::SReg_64RegClass); SavedWQMReg = MRI->createVirtualRegister(BoolRC);
toExact(MBB, Before, SavedWQMReg, LiveMaskReg); toExact(MBB, Before, SavedWQMReg, LiveMaskReg);
State = StateExact; State = StateExact;
} else if (State == StateExact && (Needs & StateWQM) && } else if (State == StateExact && (Needs & StateWQM) &&
!(Needs & StateExact)) { !(Needs & StateExact)) {
assert(WQMFromExec == (SavedWQMReg == 0)); assert(WQMFromExec == (SavedWQMReg == 0));
toWQM(MBB, Before, SavedWQMReg); toWQM(MBB, Before, SavedWQMReg);
▲ Show 20 Lines • Show All 62 Lines • ▼ Show 20 Lines
bool SIWholeQuadMode::runOnMachineFunction(MachineFunction &MF) { bool SIWholeQuadMode::runOnMachineFunction(MachineFunction &MF) {
Instructions.clear(); Instructions.clear();
Blocks.clear(); Blocks.clear();
LiveMaskQueries.clear(); LiveMaskQueries.clear();
LowerToCopyInstrs.clear(); LowerToCopyInstrs.clear();
CallingConv = MF.getFunction().getCallingConv(); CallingConv = MF.getFunction().getCallingConv();
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>(); ST = &MF.getSubtarget<GCNSubtarget>();
TII = ST.getInstrInfo(); TII = ST->getInstrInfo();
TRI = &TII->getRegisterInfo(); TRI = &TII->getRegisterInfo();
MRI = &MF.getRegInfo(); MRI = &MF.getRegInfo();
LIS = &getAnalysis<LiveIntervals>(); LIS = &getAnalysis<LiveIntervals>();
char GlobalFlags = analyzeFunction(MF); char GlobalFlags = analyzeFunction(MF);
unsigned LiveMaskReg = 0; unsigned LiveMaskReg = 0;
unsigned Exec = ST->isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
if (!(GlobalFlags & StateWQM)) { if (!(GlobalFlags & StateWQM)) {
lowerLiveMaskQueries(AMDGPU::EXEC); lowerLiveMaskQueries(Exec);
if (!(GlobalFlags & StateWWM)) if (!(GlobalFlags & StateWWM))
return !LiveMaskQueries.empty(); return !LiveMaskQueries.empty();
} else { } else {
// Store a copy of the original live mask when required // Store a copy of the original live mask when required
MachineBasicBlock &Entry = MF.front(); MachineBasicBlock &Entry = MF.front();
MachineBasicBlock::iterator EntryMI = Entry.getFirstNonPHI(); MachineBasicBlock::iterator EntryMI = Entry.getFirstNonPHI();
if (GlobalFlags & StateExact || !LiveMaskQueries.empty()) { if (GlobalFlags & StateExact || !LiveMaskQueries.empty()) {
LiveMaskReg = MRI->createVirtualRegister(&AMDGPU::SReg_64RegClass); LiveMaskReg = MRI->createVirtualRegister(TRI->getBoolRC());
MachineInstr *MI = BuildMI(Entry, EntryMI, DebugLoc(), MachineInstr *MI = BuildMI(Entry, EntryMI, DebugLoc(),
TII->get(AMDGPU::COPY), LiveMaskReg) TII->get(AMDGPU::COPY), LiveMaskReg)
.addReg(AMDGPU::EXEC); .addReg(Exec);
LIS->InsertMachineInstrInMaps(*MI); LIS->InsertMachineInstrInMaps(*MI);
} }
lowerLiveMaskQueries(LiveMaskReg); lowerLiveMaskQueries(LiveMaskReg);
if (GlobalFlags == StateWQM) { if (GlobalFlags == StateWQM) {
// For a shader that needs only WQM, we can just set it once. // For a shader that needs only WQM, we can just set it once.
BuildMI(Entry, EntryMI, DebugLoc(), TII->get(AMDGPU::S_WQM_B64), BuildMI(Entry, EntryMI, DebugLoc(), TII->get(ST->isWave32() ?
AMDGPU::EXEC) AMDGPU::S_WQM_B32 : AMDGPU::S_WQM_B64),
.addReg(AMDGPU::EXEC); Exec)
.addReg(Exec);
lowerCopyInstrs(); lowerCopyInstrs();
// EntryMI may become invalid here // EntryMI may become invalid here
return true; return true;
} }
} }
LLVM_DEBUG(printInfo()); LLVM_DEBUG(printInfo());
Show All 14 Lines

lib/Target/AMDGPU/SOPInstructions.td

Show First 20 Lines • Show All 146 Lines • ▼ Show 20 Lineslet Defs = [SCC] in {
def S_NOT_B32 : SOP1_32 <"s_not_b32", def S_NOT_B32 : SOP1_32 <"s_not_b32",
[(set i32:$sdst, (not i32:$src0))] [(set i32:$sdst, (not i32:$src0))]
>; >;
def S_NOT_B64 : SOP1_64 <"s_not_b64", def S_NOT_B64 : SOP1_64 <"s_not_b64",
[(set i64:$sdst, (not i64:$src0))] [(set i64:$sdst, (not i64:$src0))]
>; >;
def S_WQM_B32 : SOP1_32 <"s_wqm_b32">; def S_WQM_B32 : SOP1_32 <"s_wqm_b32">;
def S_WQM_B64 : SOP1_64 <"s_wqm_b64", def S_WQM_B64 : SOP1_64 <"s_wqm_b64">;
[(set i1:$sdst, (int_amdgcn_wqm_vote i1:$src0))]
>;
} // End Defs = [SCC] } // End Defs = [SCC]
let WaveSizePredicate = isWave32 in {
def : GCNPat <
(int_amdgcn_wqm_vote i1:$src0),
(S_WQM_B32 $src0)
>;
}
let WaveSizePredicate = isWave64 in {
def : GCNPat <
(int_amdgcn_wqm_vote i1:$src0),
(S_WQM_B64 $src0)
>;
}
def S_BREV_B32 : SOP1_32 <"s_brev_b32", def S_BREV_B32 : SOP1_32 <"s_brev_b32",
[(set i32:$sdst, (bitreverse i32:$src0))] [(set i32:$sdst, (bitreverse i32:$src0))]
>; >;
def S_BREV_B64 : SOP1_64 <"s_brev_b64">; def S_BREV_B64 : SOP1_64 <"s_brev_b64">;
let Defs = [SCC] in { let Defs = [SCC] in {
def S_BCNT0_I32_B32 : SOP1_32 <"s_bcnt0_i32_b32">; def S_BCNT0_I32_B32 : SOP1_32 <"s_bcnt0_i32_b32">;
def S_BCNT0_I32_B64 : SOP1_32_64 <"s_bcnt0_i32_b64">; def S_BCNT0_I32_B64 : SOP1_32_64 <"s_bcnt0_i32_b64">;
▲ Show 20 Lines • Show All 101 Lines • ▼ Show 20 Lines let hasSideEffects = 1, Defs = [EXEC, SCC], Uses = [EXEC] in {
def S_ANDN1_WREXEC_B64 : SOP1_64<"s_andn1_wrexec_b64">; def S_ANDN1_WREXEC_B64 : SOP1_64<"s_andn1_wrexec_b64">;
def S_ANDN2_WREXEC_B64 : SOP1_64<"s_andn2_wrexec_b64">; def S_ANDN2_WREXEC_B64 : SOP1_64<"s_andn2_wrexec_b64">;
} // End hasSideEffects = 1, Defs = [EXEC, SCC], Uses = [EXEC] } // End hasSideEffects = 1, Defs = [EXEC, SCC], Uses = [EXEC]
def S_BITREPLICATE_B64_B32 : SOP1_64_32<"s_bitreplicate_b64_b32">; def S_BITREPLICATE_B64_B32 : SOP1_64_32<"s_bitreplicate_b64_b32">;
} // End SubtargetPredicate = isGFX9Plus } // End SubtargetPredicate = isGFX9Plus
let SubtargetPredicate = isGFX10Plus in { let SubtargetPredicate = isGFX10Plus in {
let hasSideEffects = 1, Defs = [EXEC, SCC], Uses = [EXEC] in {
def S_AND_SAVEEXEC_B32 : SOP1_32<"s_and_saveexec_b32">;
def S_OR_SAVEEXEC_B32 : SOP1_32<"s_or_saveexec_b32">;
def S_XOR_SAVEEXEC_B32 : SOP1_32<"s_xor_saveexec_b32">;
def S_ANDN2_SAVEEXEC_B32 : SOP1_32<"s_andn2_saveexec_b32">;
def S_ORN2_SAVEEXEC_B32 : SOP1_32<"s_orn2_saveexec_b32">;
def S_NAND_SAVEEXEC_B32 : SOP1_32<"s_nand_saveexec_b32">;
def S_NOR_SAVEEXEC_B32 : SOP1_32<"s_nor_saveexec_b32">;
def S_XNOR_SAVEEXEC_B32 : SOP1_32<"s_xnor_saveexec_b32">;
def S_ANDN1_SAVEEXEC_B32 : SOP1_32<"s_andn1_saveexec_b32">;
def S_ORN1_SAVEEXEC_B32 : SOP1_32<"s_orn1_saveexec_b32">;
def S_ANDN1_WREXEC_B32 : SOP1_32<"s_andn1_wrexec_b32">;
def S_ANDN2_WREXEC_B32 : SOP1_32<"s_andn2_wrexec_b32">;
} // End hasSideEffects = 1, Defs = [EXEC, SCC], Uses = [EXEC]
let Uses = [M0] in { let Uses = [M0] in {
def S_MOVRELSD_2_B32 : SOP1_32<"s_movrelsd_2_b32">; def S_MOVRELSD_2_B32 : SOP1_32<"s_movrelsd_2_b32">;
} // End Uses = [M0] } // End Uses = [M0]
} // End SubtargetPredicate = isGFX10Plus } // End SubtargetPredicate = isGFX10Plus
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SOP2 Instructions // SOP2 Instructions
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
▲ Show 20 Lines • Show All 491 Lines • ▼ Show 20 Lineslet SubtargetPredicate = isGFX10Plus in {
def S_VERSION : SOPK_Pseudo< def S_VERSION : SOPK_Pseudo<
"s_version", "s_version",
(outs), (outs),
(ins s16imm:$simm16), (ins s16imm:$simm16),
"$simm16"> { "$simm16"> {
let has_sdst = 0; let has_sdst = 0;
} }
def S_SUBVECTOR_LOOP_BEGIN : SOPK_32_BR<"s_subvector_loop_begin">;
def S_SUBVECTOR_LOOP_END : SOPK_32_BR<"s_subvector_loop_end">;
def S_WAITCNT_VSCNT : SOPK_WAITCNT<"s_waitcnt_vscnt">; def S_WAITCNT_VSCNT : SOPK_WAITCNT<"s_waitcnt_vscnt">;
def S_WAITCNT_VMCNT : SOPK_WAITCNT<"s_waitcnt_vmcnt">; def S_WAITCNT_VMCNT : SOPK_WAITCNT<"s_waitcnt_vmcnt">;
def S_WAITCNT_EXPCNT : SOPK_WAITCNT<"s_waitcnt_expcnt">; def S_WAITCNT_EXPCNT : SOPK_WAITCNT<"s_waitcnt_expcnt">;
def S_WAITCNT_LGKMCNT : SOPK_WAITCNT<"s_waitcnt_lgkmcnt">; def S_WAITCNT_LGKMCNT : SOPK_WAITCNT<"s_waitcnt_lgkmcnt">;
} // End SubtargetPredicate = isGFX10Plus } // End SubtargetPredicate = isGFX10Plus
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SOPC Instructions // SOPC Instructions
▲ Show 20 Lines • Show All 417 Lines • ▼ Show 20 Lines def _gfx10 : SOP1_Real<op, !cast<SOP1_Pseudo>(NAME)>,
Select_gfx10<!cast<SOP1_Pseudo>(NAME).Mnemonic>; Select_gfx10<!cast<SOP1_Pseudo>(NAME).Mnemonic>;
} }
defm S_ANDN1_SAVEEXEC_B64 : SOP1_Real_gfx10<0x037>; defm S_ANDN1_SAVEEXEC_B64 : SOP1_Real_gfx10<0x037>;
defm S_ORN1_SAVEEXEC_B64 : SOP1_Real_gfx10<0x038>; defm S_ORN1_SAVEEXEC_B64 : SOP1_Real_gfx10<0x038>;
defm S_ANDN1_WREXEC_B64 : SOP1_Real_gfx10<0x039>; defm S_ANDN1_WREXEC_B64 : SOP1_Real_gfx10<0x039>;
defm S_ANDN2_WREXEC_B64 : SOP1_Real_gfx10<0x03a>; defm S_ANDN2_WREXEC_B64 : SOP1_Real_gfx10<0x03a>;
defm S_BITREPLICATE_B64_B32 : SOP1_Real_gfx10<0x03b>; defm S_BITREPLICATE_B64_B32 : SOP1_Real_gfx10<0x03b>;
defm S_AND_SAVEEXEC_B32 : SOP1_Real_gfx10<0x03c>;
defm S_OR_SAVEEXEC_B32 : SOP1_Real_gfx10<0x03d>;
defm S_XOR_SAVEEXEC_B32 : SOP1_Real_gfx10<0x03e>;
defm S_ANDN2_SAVEEXEC_B32 : SOP1_Real_gfx10<0x03f>;
defm S_ORN2_SAVEEXEC_B32 : SOP1_Real_gfx10<0x040>;
defm S_NAND_SAVEEXEC_B32 : SOP1_Real_gfx10<0x041>;
defm S_NOR_SAVEEXEC_B32 : SOP1_Real_gfx10<0x042>;
defm S_XNOR_SAVEEXEC_B32 : SOP1_Real_gfx10<0x043>;
defm S_ANDN1_SAVEEXEC_B32 : SOP1_Real_gfx10<0x044>;
defm S_ORN1_SAVEEXEC_B32 : SOP1_Real_gfx10<0x045>;
defm S_ANDN1_WREXEC_B32 : SOP1_Real_gfx10<0x046>;
defm S_ANDN2_WREXEC_B32 : SOP1_Real_gfx10<0x047>;
defm S_MOVRELSD_2_B32 : SOP1_Real_gfx10<0x049>; defm S_MOVRELSD_2_B32 : SOP1_Real_gfx10<0x049>;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SOP1 - GFX6, GFX7. // SOP1 - GFX6, GFX7.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
class Select_gfx6_gfx7<string opName> : SIMCInstr<opName, SIEncodingFamily.SI> { class Select_gfx6_gfx7<string opName> : SIMCInstr<opName, SIEncodingFamily.SI> {
Predicate AssemblerPredicate = isGFX6GFX7; Predicate AssemblerPredicate = isGFX6GFX7;
▲ Show 20 Lines • Show All 151 Lines • ▼ Show 20 Lines
} }
defm S_VERSION : SOPK_Real32_gfx10<0x001>; defm S_VERSION : SOPK_Real32_gfx10<0x001>;
defm S_CALL_B64 : SOPK_Real32_gfx10<0x016>; defm S_CALL_B64 : SOPK_Real32_gfx10<0x016>;
defm S_WAITCNT_VSCNT : SOPK_Real32_gfx10<0x017>; defm S_WAITCNT_VSCNT : SOPK_Real32_gfx10<0x017>;
defm S_WAITCNT_VMCNT : SOPK_Real32_gfx10<0x018>; defm S_WAITCNT_VMCNT : SOPK_Real32_gfx10<0x018>;
defm S_WAITCNT_EXPCNT : SOPK_Real32_gfx10<0x019>; defm S_WAITCNT_EXPCNT : SOPK_Real32_gfx10<0x019>;
defm S_WAITCNT_LGKMCNT : SOPK_Real32_gfx10<0x01a>; defm S_WAITCNT_LGKMCNT : SOPK_Real32_gfx10<0x01a>;
defm S_SUBVECTOR_LOOP_BEGIN : SOPK_Real32_gfx10<0x01b>;
defm S_SUBVECTOR_LOOP_END : SOPK_Real32_gfx10<0x01c>;
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// SOPK - GFX6, GFX7. // SOPK - GFX6, GFX7.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
multiclass SOPK_Real32_gfx6_gfx7<bits<5> op> { multiclass SOPK_Real32_gfx6_gfx7<bits<5> op> {
def _gfx6_gfx7 : SOPK_Real32<op, !cast<SOPK_Pseudo>(NAME)>, def _gfx6_gfx7 : SOPK_Real32<op, !cast<SOPK_Pseudo>(NAME)>,
Select_gfx6_gfx7<!cast<SOPK_Pseudo>(NAME).Mnemonic>; Select_gfx6_gfx7<!cast<SOPK_Pseudo>(NAME).Mnemonic>;
▲ Show 20 Lines • Show All 203 Lines • Show Last 20 Lines

lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h

Show First 20 Lines • Show All 143 Lines • ▼ Show 20 Linesunsigned getNumExtraSGPRs(const MCSubtargetInfo *STI, bool VCCUsed,
bool FlatScrUsed); bool FlatScrUsed);
/// \returns Number of SGPR blocks needed for given subtarget \p STI when /// \returns Number of SGPR blocks needed for given subtarget \p STI when
/// \p NumSGPRs are used. \p NumSGPRs should already include any special /// \p NumSGPRs are used. \p NumSGPRs should already include any special
/// register counts. /// register counts.
unsigned getNumSGPRBlocks(const MCSubtargetInfo *STI, unsigned NumSGPRs); unsigned getNumSGPRBlocks(const MCSubtargetInfo *STI, unsigned NumSGPRs);
/// \returns VGPR allocation granularity for given subtarget \p STI. /// \returns VGPR allocation granularity for given subtarget \p STI.
unsigned getVGPRAllocGranule(const MCSubtargetInfo *STI); ///
/// For subtargets which support it, \p EnableWavefrontSize32 should match
/// the ENABLE_WAVEFRONT_SIZE32 kernel descriptor field.
unsigned getVGPRAllocGranule(const MCSubtargetInfo *STI,
Optional<bool> EnableWavefrontSize32 = None);
/// \returns VGPR encoding granularity for given subtarget \p STI. /// \returns VGPR encoding granularity for given subtarget \p STI.
unsigned getVGPREncodingGranule(const MCSubtargetInfo *STI); ///
/// For subtargets which support it, \p EnableWavefrontSize32 should match
/// the ENABLE_WAVEFRONT_SIZE32 kernel descriptor field.
unsigned getVGPREncodingGranule(const MCSubtargetInfo *STI,
Optional<bool> EnableWavefrontSize32 = None);
/// \returns Total number of VGPRs for given subtarget \p STI. /// \returns Total number of VGPRs for given subtarget \p STI.
unsigned getTotalNumVGPRs(const MCSubtargetInfo *STI); unsigned getTotalNumVGPRs(const MCSubtargetInfo *STI);
/// \returns Addressable number of VGPRs for given subtarget \p STI. /// \returns Addressable number of VGPRs for given subtarget \p STI.
unsigned getAddressableNumVGPRs(const MCSubtargetInfo *STI); unsigned getAddressableNumVGPRs(const MCSubtargetInfo *STI);
/// \returns Minimum number of VGPRs that meets given number of waves per /// \returns Minimum number of VGPRs that meets given number of waves per
/// execution unit requirement for given subtarget \p STI. /// execution unit requirement for given subtarget \p STI.
unsigned getMinNumVGPRs(const MCSubtargetInfo *STI, unsigned WavesPerEU); unsigned getMinNumVGPRs(const MCSubtargetInfo *STI, unsigned WavesPerEU);
/// \returns Maximum number of VGPRs that meets given number of waves per /// \returns Maximum number of VGPRs that meets given number of waves per
/// execution unit requirement for given subtarget \p STI. /// execution unit requirement for given subtarget \p STI.
unsigned getMaxNumVGPRs(const MCSubtargetInfo *STI, unsigned WavesPerEU); unsigned getMaxNumVGPRs(const MCSubtargetInfo *STI, unsigned WavesPerEU);
/// \returns Number of VGPR blocks needed for given subtarget \p STI when /// \returns Number of VGPR blocks needed for given subtarget \p STI when
/// \p NumVGPRs are used. /// \p NumVGPRs are used.
unsigned getNumVGPRBlocks(const MCSubtargetInfo *STI, unsigned NumSGPRs); ///
/// For subtargets which support it, \p EnableWavefrontSize32 should match the
/// ENABLE_WAVEFRONT_SIZE32 kernel descriptor field.
unsigned getNumVGPRBlocks(const MCSubtargetInfo *STI, unsigned NumSGPRs,
Optional<bool> EnableWavefrontSize32 = None);
} // end namespace IsaInfo } // end namespace IsaInfo
LLVM_READONLY LLVM_READONLY
int16_t getNamedOperandIdx(uint16_t Opcode, uint16_t NamedIdx); int16_t getNamedOperandIdx(uint16_t Opcode, uint16_t NamedIdx);
struct MIMGBaseOpcodeInfo { struct MIMGBaseOpcodeInfo {
MIMGBaseOpcode BaseOpcode; MIMGBaseOpcode BaseOpcode;
▲ Show 20 Lines • Show All 393 Lines • Show Last 20 Lines

lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp

Show First 20 Lines • Show All 372 Lines • ▼ Show 20 Lines
} }
unsigned getNumSGPRBlocks(const MCSubtargetInfo *STI, unsigned NumSGPRs) { unsigned getNumSGPRBlocks(const MCSubtargetInfo *STI, unsigned NumSGPRs) {
NumSGPRs = alignTo(std::max(1u, NumSGPRs), getSGPREncodingGranule(STI)); NumSGPRs = alignTo(std::max(1u, NumSGPRs), getSGPREncodingGranule(STI));
// SGPRBlocks is actual number of SGPR blocks minus 1. // SGPRBlocks is actual number of SGPR blocks minus 1.
return NumSGPRs / getSGPREncodingGranule(STI) - 1; return NumSGPRs / getSGPREncodingGranule(STI) - 1;
} }
unsigned getVGPRAllocGranule(const MCSubtargetInfo *STI) { unsigned getVGPRAllocGranule(const MCSubtargetInfo *STI,
return 4; Optional<bool> EnableWavefrontSize32) {
bool IsWave32 = EnableWavefrontSize32 ?
*EnableWavefrontSize32 :
STI->getFeatureBits().test(FeatureWavefrontSize32);
return IsWave32 ? 8 : 4;
} }
unsigned getVGPREncodingGranule(const MCSubtargetInfo *STI) { unsigned getVGPREncodingGranule(const MCSubtargetInfo *STI,
return getVGPRAllocGranule(STI); Optional<bool> EnableWavefrontSize32) {
return getVGPRAllocGranule(STI, EnableWavefrontSize32);
} }
unsigned getTotalNumVGPRs(const MCSubtargetInfo *STI) { unsigned getTotalNumVGPRs(const MCSubtargetInfo *STI) {
return 256; return 256;
} }
unsigned getAddressableNumVGPRs(const MCSubtargetInfo *STI) { unsigned getAddressableNumVGPRs(const MCSubtargetInfo *STI) {
return getTotalNumVGPRs(STI); return getTotalNumVGPRs(STI);
Show All 14 Linesunsigned getMaxNumVGPRs(const MCSubtargetInfo *STI, unsigned WavesPerEU) {
assert(WavesPerEU != 0); assert(WavesPerEU != 0);
unsigned MaxNumVGPRs = alignDown(getTotalNumVGPRs(STI) / WavesPerEU, unsigned MaxNumVGPRs = alignDown(getTotalNumVGPRs(STI) / WavesPerEU,
getVGPRAllocGranule(STI)); getVGPRAllocGranule(STI));
unsigned AddressableNumVGPRs = getAddressableNumVGPRs(STI); unsigned AddressableNumVGPRs = getAddressableNumVGPRs(STI);
return std::min(MaxNumVGPRs, AddressableNumVGPRs); return std::min(MaxNumVGPRs, AddressableNumVGPRs);
} }
unsigned getNumVGPRBlocks(const MCSubtargetInfo *STI, unsigned NumVGPRs) { unsigned getNumVGPRBlocks(const MCSubtargetInfo *STI, unsigned NumVGPRs,
NumVGPRs = alignTo(std::max(1u, NumVGPRs), getVGPREncodingGranule(STI)); Optional<bool> EnableWavefrontSize32) {
NumVGPRs = alignTo(std::max(1u, NumVGPRs),
getVGPREncodingGranule(STI, EnableWavefrontSize32));
// VGPRBlocks is actual number of VGPR blocks minus 1. // VGPRBlocks is actual number of VGPR blocks minus 1.
return NumVGPRs / getVGPREncodingGranule(STI) - 1; return NumVGPRs / getVGPREncodingGranule(STI, EnableWavefrontSize32) - 1;
} }
} // end namespace IsaInfo } // end namespace IsaInfo
void initDefaultAMDKernelCodeT(amd_kernel_code_t &Header, void initDefaultAMDKernelCodeT(amd_kernel_code_t &Header,
const MCSubtargetInfo *STI) { const MCSubtargetInfo *STI) {
IsaVersion Version = getIsaVersion(STI->getCPU()); IsaVersion Version = getIsaVersion(STI->getCPU());
memset(&Header, 0, sizeof(Header)); memset(&Header, 0, sizeof(Header));
Header.amd_kernel_code_version_major = 1; Header.amd_kernel_code_version_major = 1;
Header.amd_kernel_code_version_minor = 2; Header.amd_kernel_code_version_minor = 2;
Header.amd_machine_kind = 1; // AMD_MACHINE_KIND_AMDGPU Header.amd_machine_kind = 1; // AMD_MACHINE_KIND_AMDGPU
Header.amd_machine_version_major = Version.Major; Header.amd_machine_version_major = Version.Major;
Header.amd_machine_version_minor = Version.Minor; Header.amd_machine_version_minor = Version.Minor;
Header.amd_machine_version_stepping = Version.Stepping; Header.amd_machine_version_stepping = Version.Stepping;
Header.kernel_code_entry_byte_offset = sizeof(Header); Header.kernel_code_entry_byte_offset = sizeof(Header);
// wavefront_size is specified as a power of 2: 2^6 = 64 threads.
Header.wavefront_size = 6; Header.wavefront_size = 6;
// If the code object does not support indirect functions, then the value must // If the code object does not support indirect functions, then the value must
// be 0xffffffff. // be 0xffffffff.
Header.call_convention = -1; Header.call_convention = -1;
// These alignment values are specified in powers of two, so alignment = // These alignment values are specified in powers of two, so alignment =
// 2^n. The minimum alignment is 2^4 = 16. // 2^n. The minimum alignment is 2^4 = 16.
▲ Show 20 Lines • Show All 639 Lines • Show Last 20 Lines

lib/Target/AMDGPU/VOP2Instructions.td

Show First 20 Lines • Show All 169 Lines • ▼ Show 20 Linesmulticlass VOP2bInst <string opName,
SDPatternOperator node = null_frag, SDPatternOperator node = null_frag,
string revOp = opName, string revOp = opName,
bit GFX9Renamed = 0, bit GFX9Renamed = 0,
bit useSGPRInput = !eq(P.NumSrcArgs, 3)> { bit useSGPRInput = !eq(P.NumSrcArgs, 3)> {
let renamedInGFX9 = GFX9Renamed in { let renamedInGFX9 = GFX9Renamed in {
let SchedRW = [Write32Bit, WriteSALU] in { let SchedRW = [Write32Bit, WriteSALU] in {
let Uses = !if(useSGPRInput, [VCC, EXEC], [EXEC]), Defs = [VCC] in { let Uses = !if(useSGPRInput, [VCC, EXEC], [EXEC]), Defs = [VCC] in {
def _e32 : VOP2_Pseudo <opName, P, VOPPatOrNull<node,P>.ret>, def _e32 : VOP2_Pseudo <opName, P, VOPPatOrNull<node,P>.ret>,
Commutable_REV<revOp#"_e32", !eq(revOp, opName)>; Commutable_REV<revOp#"_e32", !eq(revOp, opName)> {
let usesCustomInserter = !eq(P.NumSrcArgs, 2);
}
def _sdwa : VOP2_SDWA_Pseudo <opName, P> { def _sdwa : VOP2_SDWA_Pseudo <opName, P> {
let AsmMatchConverter = "cvtSdwaVOP2b"; let AsmMatchConverter = "cvtSdwaVOP2b";
} }
foreach _ = BoolToList<P.HasExtDPP>.ret in foreach _ = BoolToList<P.HasExtDPP>.ret in
def _dpp : VOP2_DPP_Pseudo <opName, P>; def _dpp : VOP2_DPP_Pseudo <opName, P>;
} }
def _e64 : VOP3_Pseudo <opName, P, getVOP2Pat64<node, P>.ret>, def _e64 : VOP3_Pseudo <opName, P, getVOP2Pat64<node, P>.ret>,
Commutable_REV<revOp#"_e64", !eq(revOp, opName)>; Commutable_REV<revOp#"_e64", !eq(revOp, opName)>;
} }
} }
} }
class VOP2bInstAlias <VOP2_Pseudo ps, Instruction inst, class VOP2bInstAlias <VOP2_Pseudo ps, Instruction inst,
string OpName, string opnd> : string OpName, string opnd> :
InstAlias <OpName#" "#!subst("vcc", opnd, ps.Pfl.Asm32), InstAlias <OpName#" "#!subst("vcc", opnd, ps.Pfl.Asm32),
(inst ps.Pfl.DstRC:$vdst, ps.Pfl.Src0RC32:$src0, (inst ps.Pfl.DstRC:$vdst, ps.Pfl.Src0RC32:$src0,
ps.Pfl.Src1RC32:$src1)>, ps.Pfl.Src1RC32:$src1)>,
PredicateControl { PredicateControl {
} }
multiclass VOP2bInstAliases<VOP2_Pseudo ps, VOP2_Real inst, string OpName> { multiclass VOP2bInstAliases<VOP2_Pseudo ps, VOP2_Real inst, string OpName> {
let WaveSizePredicate = isWave32 in {
def : VOP2bInstAlias<ps, inst, OpName, "vcc_lo">;
}
let WaveSizePredicate = isWave64 in {
def : VOP2bInstAlias<ps, inst, OpName, "vcc">; def : VOP2bInstAlias<ps, inst, OpName, "vcc">;
} }
}
multiclass VOP2eInst <string opName, multiclass VOP2eInst <string opName,
VOPProfile P, VOPProfile P,
SDPatternOperator node = null_frag, SDPatternOperator node = null_frag,
string revOp = opName, string revOp = opName,
bit useSGPRInput = !eq(P.NumSrcArgs, 3)> { bit useSGPRInput = !eq(P.NumSrcArgs, 3)> {
let SchedRW = [Write32Bit] in { let SchedRW = [Write32Bit] in {
Show All 17 Lines
class VOP2eInstAlias <VOP2_Pseudo ps, Instruction inst, string opnd> : class VOP2eInstAlias <VOP2_Pseudo ps, Instruction inst, string opnd> :
InstAlias <ps.OpName#" "#ps.Pfl.Asm32#", "#opnd, InstAlias <ps.OpName#" "#ps.Pfl.Asm32#", "#opnd,
(inst ps.Pfl.DstRC:$vdst, ps.Pfl.Src0RC32:$src0, (inst ps.Pfl.DstRC:$vdst, ps.Pfl.Src0RC32:$src0,
ps.Pfl.Src1RC32:$src1)>, ps.Pfl.Src1RC32:$src1)>,
PredicateControl { PredicateControl {
} }
multiclass VOP2eInstAliases<VOP2_Pseudo ps, VOP2_Real inst> { multiclass VOP2eInstAliases<VOP2_Pseudo ps, VOP2_Real inst> {
let WaveSizePredicate = isWave32 in {
def : VOP2eInstAlias<ps, inst, "vcc_lo">;
}
let WaveSizePredicate = isWave64 in {
def : VOP2eInstAlias<ps, inst, "vcc">; def : VOP2eInstAlias<ps, inst, "vcc">;
} }
}
class VOP_MADAK <ValueType vt> : VOPProfile <[vt, vt, vt, vt]> { class VOP_MADAK <ValueType vt> : VOPProfile <[vt, vt, vt, vt]> {
field Operand ImmOpType = !if(!eq(vt.Size, 32), f32kimm, f16kimm); field Operand ImmOpType = !if(!eq(vt.Size, 32), f32kimm, f16kimm);
field dag Ins32 = (ins VCSrc_f32:$src0, VGPR_32:$src1, ImmOpType:$imm); field dag Ins32 = (ins VCSrc_f32:$src0, VGPR_32:$src1, ImmOpType:$imm);
field bit HasExt = 0; field bit HasExt = 0;
// Hack to stop printing _e64 // Hack to stop printing _e64
let DstRC = RegisterOperand<VGPR_32>; let DstRC = RegisterOperand<VGPR_32>;
▲ Show 20 Lines • Show All 66 Lines • ▼ Show 20 Linesdef VOP2b_I32_I1_I32_I32 : VOPProfile<[i32, i32, i32, untyped], 0, /*EnableClamp=*/1> {
let Asm32 = "$vdst, vcc, $src0, $src1"; let Asm32 = "$vdst, vcc, $src0, $src1";
let Asm64 = "$vdst, $sdst, $src0, $src1$clamp"; let Asm64 = "$vdst, $sdst, $src0, $src1$clamp";
let AsmSDWA = "$vdst, vcc, $src0_modifiers, $src1_modifiers$clamp $dst_sel $dst_unused $src0_sel $src1_sel"; let AsmSDWA = "$vdst, vcc, $src0_modifiers, $src1_modifiers$clamp $dst_sel $dst_unused $src0_sel $src1_sel";
let AsmSDWA9 = "$vdst, vcc, $src0_modifiers, $src1_modifiers$clamp $dst_sel $dst_unused $src0_sel $src1_sel"; let AsmSDWA9 = "$vdst, vcc, $src0_modifiers, $src1_modifiers$clamp $dst_sel $dst_unused $src0_sel $src1_sel";
let AsmDPP = "$vdst, vcc, $src0, $src1 $dpp_ctrl$row_mask$bank_mask$bound_ctrl"; let AsmDPP = "$vdst, vcc, $src0, $src1 $dpp_ctrl$row_mask$bank_mask$bound_ctrl";
let AsmDPP8 = "$vdst, vcc, $src0, $src1 $dpp8$fi"; let AsmDPP8 = "$vdst, vcc, $src0, $src1 $dpp8$fi";
let AsmDPP16 = AsmDPP#"$fi"; let AsmDPP16 = AsmDPP#"$fi";
let Outs32 = (outs DstRC:$vdst); let Outs32 = (outs DstRC:$vdst);
let Outs64 = (outs DstRC:$vdst, SReg_64:$sdst); let Outs64 = (outs DstRC:$vdst, VOPDstS64orS32:$sdst);
} }
// Write out to vcc or arbitrary SGPR and read in from vcc or // Write out to vcc or arbitrary SGPR and read in from vcc or
// arbitrary SGPR. // arbitrary SGPR.
def VOP2b_I32_I1_I32_I32_I1 : VOPProfile<[i32, i32, i32, i1], 0, /*EnableClamp=*/1> { def VOP2b_I32_I1_I32_I32_I1 : VOPProfile<[i32, i32, i32, i1], 0, /*EnableClamp=*/1> {
let Asm32 = "$vdst, vcc, $src0, $src1, vcc"; let Asm32 = "$vdst, vcc, $src0, $src1, vcc";
let Asm64 = "$vdst, $sdst, $src0, $src1, $src2$clamp"; let Asm64 = "$vdst, $sdst, $src0, $src1, $src2$clamp";
let AsmSDWA = "$vdst, vcc, $src0_modifiers, $src1_modifiers, vcc $clamp $dst_sel $dst_unused $src0_sel $src1_sel"; let AsmSDWA = "$vdst, vcc, $src0_modifiers, $src1_modifiers, vcc $clamp $dst_sel $dst_unused $src0_sel $src1_sel";
let AsmSDWA9 = "$vdst, vcc, $src0_modifiers, $src1_modifiers, vcc $clamp $dst_sel $dst_unused $src0_sel $src1_sel"; let AsmSDWA9 = "$vdst, vcc, $src0_modifiers, $src1_modifiers, vcc $clamp $dst_sel $dst_unused $src0_sel $src1_sel";
let AsmDPP = "$vdst, vcc, $src0, $src1, vcc $dpp_ctrl$row_mask$bank_mask$bound_ctrl"; let AsmDPP = "$vdst, vcc, $src0, $src1, vcc $dpp_ctrl$row_mask$bank_mask$bound_ctrl";
let AsmDPP8 = "$vdst, vcc, $src0, $src1, vcc $dpp8$fi"; let AsmDPP8 = "$vdst, vcc, $src0, $src1, vcc $dpp8$fi";
let AsmDPP16 = AsmDPP#"$fi"; let AsmDPP16 = AsmDPP#"$fi";
let Outs32 = (outs DstRC:$vdst); let Outs32 = (outs DstRC:$vdst);
let Outs64 = (outs DstRC:$vdst, SReg_64:$sdst); let Outs64 = (outs DstRC:$vdst, VOPDstS64orS32:$sdst);
// Suppress src2 implied by type since the 32-bit encoding uses an // Suppress src2 implied by type since the 32-bit encoding uses an
// implicit VCC use. // implicit VCC use.
let Ins32 = (ins Src0RC32:$src0, Src1RC32:$src1); let Ins32 = (ins Src0RC32:$src0, Src1RC32:$src1);
let InsSDWA = (ins Src0ModSDWA:$src0_modifiers, Src0SDWA:$src0, let InsSDWA = (ins Src0ModSDWA:$src0_modifiers, Src0SDWA:$src0,
Src1ModSDWA:$src1_modifiers, Src1SDWA:$src1, Src1ModSDWA:$src1_modifiers, Src1SDWA:$src1,
clampmod:$clamp, clampmod:$clamp,
▲ Show 20 Lines • Show All 604 Lines • ▼ Show 20 Lines def _dpp_gfx10 :
} }
def _dpp8_gfx10 : def _dpp8_gfx10 :
VOP2_DPP8<op, !cast<VOP2_Pseudo>(opName#"_e32"), asmName> { VOP2_DPP8<op, !cast<VOP2_Pseudo>(opName#"_e32"), asmName> {
string AsmDPP8 = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP8; string AsmDPP8 = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP8;
let AsmString = asmName # !subst(", vcc", "", AsmDPP8); let AsmString = asmName # !subst(", vcc", "", AsmDPP8);
let DecoderNamespace = "DPP8"; let DecoderNamespace = "DPP8";
} }
let WaveSizePredicate = isWave32 in {
def _sdwa_w32_gfx10 :
Base_VOP_SDWA10_Real<!cast<VOP2_SDWA_Pseudo>(opName#"_sdwa")>,
VOP2_SDWA9Ae<op{5-0}, !cast<VOP2_SDWA_Pseudo>(opName#"_sdwa").Pfl> {
VOP2_SDWA_Pseudo Ps = !cast<VOP2_SDWA_Pseudo>(opName#"_sdwa");
let AsmString = asmName # !subst("vcc", "vcc_lo", Ps.AsmOperands);
let isAsmParserOnly = 1;
let DecoderNamespace = "SDWA10";
}
def _dpp_w32_gfx10 :
VOP2_DPP16<op, !cast<VOP2_Pseudo>(opName#"_e32"), asmName> {
string AsmDPP = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP16;
let AsmString = asmName # !subst("vcc", "vcc_lo", AsmDPP);
let isAsmParserOnly = 1;
}
def _dpp8_w32_gfx10 :
VOP2_DPP8<op, !cast<VOP2_Pseudo>(opName#"_e32"), asmName> {
string AsmDPP8 = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP8;
let AsmString = asmName # !subst("vcc", "vcc_lo", AsmDPP8);
let isAsmParserOnly = 1;
}
} // End WaveSizePredicate = isWave32
let WaveSizePredicate = isWave64 in {
def _sdwa_w64_gfx10 : def _sdwa_w64_gfx10 :
Base_VOP_SDWA10_Real<!cast<VOP2_SDWA_Pseudo>(opName#"_sdwa")>, Base_VOP_SDWA10_Real<!cast<VOP2_SDWA_Pseudo>(opName#"_sdwa")>,
VOP2_SDWA9Ae<op{5-0}, !cast<VOP2_SDWA_Pseudo>(opName#"_sdwa").Pfl> { VOP2_SDWA9Ae<op{5-0}, !cast<VOP2_SDWA_Pseudo>(opName#"_sdwa").Pfl> {
VOP2_SDWA_Pseudo Ps = !cast<VOP2_SDWA_Pseudo>(opName#"_sdwa"); VOP2_SDWA_Pseudo Ps = !cast<VOP2_SDWA_Pseudo>(opName#"_sdwa");
let AsmString = asmName # Ps.AsmOperands; let AsmString = asmName # Ps.AsmOperands;
let isAsmParserOnly = 1; let isAsmParserOnly = 1;
let DecoderNamespace = "SDWA10"; let DecoderNamespace = "SDWA10";
} }
def _dpp_w64_gfx10 : def _dpp_w64_gfx10 :
VOP2_DPP16<op, !cast<VOP2_Pseudo>(opName#"_e32"), asmName> { VOP2_DPP16<op, !cast<VOP2_Pseudo>(opName#"_e32"), asmName> {
string AsmDPP = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP16; string AsmDPP = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP16;
let AsmString = asmName # AsmDPP; let AsmString = asmName # AsmDPP;
let isAsmParserOnly = 1; let isAsmParserOnly = 1;
} }
def _dpp8_w64_gfx10 : def _dpp8_w64_gfx10 :
VOP2_DPP8<op, !cast<VOP2_Pseudo>(opName#"_e32"), asmName> { VOP2_DPP8<op, !cast<VOP2_Pseudo>(opName#"_e32"), asmName> {
string AsmDPP8 = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP8; string AsmDPP8 = !cast<VOP2_Pseudo>(opName#"_e32").Pfl.AsmDPP8;
let AsmString = asmName # AsmDPP8; let AsmString = asmName # AsmDPP8;
let isAsmParserOnly = 1; let isAsmParserOnly = 1;
} }
} // End WaveSizePredicate = isWave64
} }
//===----------------------------- VOP3Only -----------------------------===// //===----------------------------- VOP3Only -----------------------------===//
multiclass VOP3Only_Real_gfx10<bits<10> op> { multiclass VOP3Only_Real_gfx10<bits<10> op> {
def _e64_gfx10 : def _e64_gfx10 :
VOP3_Real<!cast<VOP3_Pseudo>(NAME#"_e64"), SIEncodingFamily.GFX10>, VOP3_Real<!cast<VOP3_Pseudo>(NAME#"_e64"), SIEncodingFamily.GFX10>,
VOP3e_gfx10<op, !cast<VOP3_Pseudo>(NAME#"_e64").Pfl>; VOP3e_gfx10<op, !cast<VOP3_Pseudo>(NAME#"_e64").Pfl>;
} }
▲ Show 20 Lines • Show All 476 Lines • Show Last 20 Lines

lib/Target/AMDGPU/VOP3Instructions.td

Show First 20 Lines • Show All 177 Lines • ▼ Show 20 Linesclass VOP3_Profile<VOPProfile P, VOP3Features Features = VOP3_REGULAR> : VOPProfile<P.ArgVT> {
let NeedPatGen = P.NeedPatGen; let NeedPatGen = P.NeedPatGen;
} }
class VOP3b_Profile<ValueType vt> : VOPProfile<[vt, vt, vt, vt]> { class VOP3b_Profile<ValueType vt> : VOPProfile<[vt, vt, vt, vt]> {
// v_div_scale_{f32|f64} do not support input modifiers. // v_div_scale_{f32|f64} do not support input modifiers.
let HasModifiers = 0; let HasModifiers = 0;
let HasClamp = 0; let HasClamp = 0;
let HasOMod = 0; let HasOMod = 0;
let Outs64 = (outs DstRC:$vdst, SReg_64:$sdst); let Outs64 = (outs DstRC:$vdst, VOPDstS64orS32:$sdst);
let Asm64 = " $vdst, $sdst, $src0, $src1, $src2"; let Asm64 = " $vdst, $sdst, $src0, $src1, $src2";
} }
def VOP3b_F32_I1_F32_F32_F32 : VOP3b_Profile<f32> { def VOP3b_F32_I1_F32_F32_F32 : VOP3b_Profile<f32> {
// FIXME: Hack to stop printing _e64 // FIXME: Hack to stop printing _e64
let DstRC = RegisterOperand<VGPR_32>; let DstRC = RegisterOperand<VGPR_32>;
} }
def VOP3b_F64_I1_F64_F64_F64 : VOP3b_Profile<f64> { def VOP3b_F64_I1_F64_F64_F64 : VOP3b_Profile<f64> {
// FIXME: Hack to stop printing _e64 // FIXME: Hack to stop printing _e64
let DstRC = RegisterOperand<VReg_64>; let DstRC = RegisterOperand<VReg_64>;
} }
def VOP3b_I64_I1_I32_I32_I64 : VOPProfile<[i64, i32, i32, i64]> { def VOP3b_I64_I1_I32_I32_I64 : VOPProfile<[i64, i32, i32, i64]> {
let HasClamp = 1; let HasClamp = 1;
// FIXME: Hack to stop printing _e64 // FIXME: Hack to stop printing _e64
let DstRC = RegisterOperand<VReg_64>; let DstRC = RegisterOperand<VReg_64>;
let Outs64 = (outs DstRC:$vdst, SReg_64:$sdst); let Outs64 = (outs DstRC:$vdst, VOPDstS64orS32:$sdst);
let Asm64 = " $vdst, $sdst, $src0, $src1, $src2$clamp"; let Asm64 = " $vdst, $sdst, $src0, $src1, $src2$clamp";
} }
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// VOP3 INTERP // VOP3 INTERP
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
class VOP3Interp<string OpName, VOPProfile P, list<dag> pattern = []> : class VOP3Interp<string OpName, VOPProfile P, list<dag> pattern = []> :
▲ Show 20 Lines • Show All 922 Lines • Show Last 20 Lines

lib/Target/AMDGPU/VOPCInstructions.td

Show First 20 Lines • Show All 50 Lines • ▼ Show 20 Lines
// VOPC instructions are a special case because for the 32-bit // VOPC instructions are a special case because for the 32-bit
// encoding, we want to display the implicit vcc write as if it were // encoding, we want to display the implicit vcc write as if it were
// an explicit $dst. // an explicit $dst.
class VOPC_Profile<list<SchedReadWrite> sched, ValueType vt0, ValueType vt1 = vt0> : class VOPC_Profile<list<SchedReadWrite> sched, ValueType vt0, ValueType vt1 = vt0> :
VOPProfile <[i1, vt0, vt1, untyped]> { VOPProfile <[i1, vt0, vt1, untyped]> {
let Asm32 = "$src0, $src1"; let Asm32 = "$src0, $src1";
// The destination for 32-bit encoding is implicit. // The destination for 32-bit encoding is implicit.
let HasDst32 = 0; let HasDst32 = 0;
let Outs64 = (outs VOPDstS64:$sdst); let Outs64 = (outs VOPDstS64orS32:$sdst);
list<SchedReadWrite> Schedule = sched; list<SchedReadWrite> Schedule = sched;
} }
class VOPC_NoSdst_Profile<list<SchedReadWrite> sched, ValueType vt0, class VOPC_NoSdst_Profile<list<SchedReadWrite> sched, ValueType vt0,
ValueType vt1 = vt0> : ValueType vt1 = vt0> :
VOPC_Profile<sched, vt0, vt1> { VOPC_Profile<sched, vt0, vt1> {
let Outs64 = (outs ); let Outs64 = (outs );
let OutsSDWA = (outs ); let OutsSDWA = (outs );
▲ Show 20 Lines • Show All 92 Lines • ▼ Show 20 Linesclass VOPCInstAlias <VOP3_Pseudo ps, Instruction inst,
let AsmVariantName = AMDGPUAsmVariants.Default; let AsmVariantName = AMDGPUAsmVariants.Default;
let SubtargetPredicate = AssemblerPredicate; let SubtargetPredicate = AssemblerPredicate;
} }
multiclass VOPCInstAliases <string OpName, string Arch> { multiclass VOPCInstAliases <string OpName, string Arch> {
def : VOPCInstAlias <!cast<VOP3_Pseudo>(OpName#"_e64"), def : VOPCInstAlias <!cast<VOP3_Pseudo>(OpName#"_e64"),
!cast<Instruction>(OpName#"_e32_"#Arch)>; !cast<Instruction>(OpName#"_e32_"#Arch)>;
let WaveSizePredicate = isWave32 in {
def : VOPCInstAlias <!cast<VOP3_Pseudo>(OpName#"_e64"),
!cast<Instruction>(OpName#"_e32_"#Arch),
"vcc_lo, "#!cast<VOP3_Pseudo>(OpName#"_e64").Pfl.Asm32>;
}
let WaveSizePredicate = isWave64 in {
def : VOPCInstAlias <!cast<VOP3_Pseudo>(OpName#"_e64"), def : VOPCInstAlias <!cast<VOP3_Pseudo>(OpName#"_e64"),
!cast<Instruction>(OpName#"_e32_"#Arch), !cast<Instruction>(OpName#"_e32_"#Arch),
"vcc, "#!cast<VOP3_Pseudo>(OpName#"_e64").Pfl.Asm32>; "vcc, "#!cast<VOP3_Pseudo>(OpName#"_e64").Pfl.Asm32>;
} }
}
multiclass VOPCXInstAliases <string OpName, string Arch> { multiclass VOPCXInstAliases <string OpName, string Arch> {
def : VOPCInstAlias <!cast<VOP3_Pseudo>(OpName#"_e64"), def : VOPCInstAlias <!cast<VOP3_Pseudo>(OpName#"_e64"),
!cast<Instruction>(OpName#"_e32_"#Arch)>; !cast<Instruction>(OpName#"_e32_"#Arch)>;
} }
class getVOPCPat64 <PatLeaf cond, VOPProfile P> : LetDummies { class getVOPCPat64 <PatLeaf cond, VOPProfile P> : LetDummies {
▲ Show 20 Lines • Show All 555 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// V_ICMPIntrinsic Pattern. // V_ICMPIntrinsic Pattern.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// We need to use COPY_TO_REGCLASS to w/a the problem when ReplaceAllUsesWith() // We need to use COPY_TO_REGCLASS to w/a the problem when ReplaceAllUsesWith()
// complaints it cannot replace i1 <-> i64/i32 if node was not morphed in place. // complaints it cannot replace i1 <-> i64/i32 if node was not morphed in place.
multiclass ICMP_Pattern <PatLeaf cond, Instruction inst, ValueType vt> { multiclass ICMP_Pattern <PatLeaf cond, Instruction inst, ValueType vt> {
let WaveSizePredicate = isWave64 in
def : GCNPat < def : GCNPat <
(i64 (AMDGPUsetcc vt:$src0, vt:$src1, cond)), (i64 (AMDGPUsetcc vt:$src0, vt:$src1, cond)),
(i64 (COPY_TO_REGCLASS (inst $src0, $src1), SReg_64)) (i64 (COPY_TO_REGCLASS (inst $src0, $src1), SReg_64))
>; >;
let WaveSizePredicate = isWave32 in
def : GCNPat <
(i32 (AMDGPUsetcc vt:$src0, vt:$src1, cond)),
(i32 (COPY_TO_REGCLASS (inst $src0, $src1), SReg_32))
>;
} }
defm : ICMP_Pattern <COND_EQ, V_CMP_EQ_U32_e64, i32>; defm : ICMP_Pattern <COND_EQ, V_CMP_EQ_U32_e64, i32>;
defm : ICMP_Pattern <COND_NE, V_CMP_NE_U32_e64, i32>; defm : ICMP_Pattern <COND_NE, V_CMP_NE_U32_e64, i32>;
defm : ICMP_Pattern <COND_UGT, V_CMP_GT_U32_e64, i32>; defm : ICMP_Pattern <COND_UGT, V_CMP_GT_U32_e64, i32>;
defm : ICMP_Pattern <COND_UGE, V_CMP_GE_U32_e64, i32>; defm : ICMP_Pattern <COND_UGE, V_CMP_GE_U32_e64, i32>;
defm : ICMP_Pattern <COND_ULT, V_CMP_LT_U32_e64, i32>; defm : ICMP_Pattern <COND_ULT, V_CMP_LT_U32_e64, i32>;
defm : ICMP_Pattern <COND_ULE, V_CMP_LE_U32_e64, i32>; defm : ICMP_Pattern <COND_ULE, V_CMP_LE_U32_e64, i32>;
Show All 20 Lines
defm : ICMP_Pattern <COND_ULT, V_CMP_LT_U16_e64, i16>; defm : ICMP_Pattern <COND_ULT, V_CMP_LT_U16_e64, i16>;
defm : ICMP_Pattern <COND_ULE, V_CMP_LE_U16_e64, i16>; defm : ICMP_Pattern <COND_ULE, V_CMP_LE_U16_e64, i16>;
defm : ICMP_Pattern <COND_SGT, V_CMP_GT_I16_e64, i16>; defm : ICMP_Pattern <COND_SGT, V_CMP_GT_I16_e64, i16>;
defm : ICMP_Pattern <COND_SGE, V_CMP_GE_I16_e64, i16>; defm : ICMP_Pattern <COND_SGE, V_CMP_GE_I16_e64, i16>;
defm : ICMP_Pattern <COND_SLT, V_CMP_LT_I16_e64, i16>; defm : ICMP_Pattern <COND_SLT, V_CMP_LT_I16_e64, i16>;
defm : ICMP_Pattern <COND_SLE, V_CMP_LE_I16_e64, i16>; defm : ICMP_Pattern <COND_SLE, V_CMP_LE_I16_e64, i16>;
multiclass FCMP_Pattern <PatLeaf cond, Instruction inst, ValueType vt> { multiclass FCMP_Pattern <PatLeaf cond, Instruction inst, ValueType vt> {
let WaveSizePredicate = isWave64 in
def : GCNPat < def : GCNPat <
(i64 (AMDGPUsetcc (vt (VOP3Mods vt:$src0, i32:$src0_modifiers)), (i64 (AMDGPUsetcc (vt (VOP3Mods vt:$src0, i32:$src0_modifiers)),
(vt (VOP3Mods vt:$src1, i32:$src1_modifiers)), cond)), (vt (VOP3Mods vt:$src1, i32:$src1_modifiers)), cond)),
(i64 (COPY_TO_REGCLASS (inst $src0_modifiers, $src0, $src1_modifiers, $src1, (i64 (COPY_TO_REGCLASS (inst $src0_modifiers, $src0, $src1_modifiers, $src1,
DSTCLAMP.NONE), SReg_64)) DSTCLAMP.NONE), SReg_64))
>; >;
let WaveSizePredicate = isWave32 in
def : GCNPat <
(i32 (AMDGPUsetcc (vt (VOP3Mods vt:$src0, i32:$src0_modifiers)),
(vt (VOP3Mods vt:$src1, i32:$src1_modifiers)), cond)),
(i32 (COPY_TO_REGCLASS (inst $src0_modifiers, $src0, $src1_modifiers, $src1,
DSTCLAMP.NONE), SReg_32))
>;
} }
defm : FCMP_Pattern <COND_OEQ, V_CMP_EQ_F32_e64, f32>; defm : FCMP_Pattern <COND_OEQ, V_CMP_EQ_F32_e64, f32>;
defm : FCMP_Pattern <COND_ONE, V_CMP_NEQ_F32_e64, f32>; defm : FCMP_Pattern <COND_ONE, V_CMP_NEQ_F32_e64, f32>;
defm : FCMP_Pattern <COND_OGT, V_CMP_GT_F32_e64, f32>; defm : FCMP_Pattern <COND_OGT, V_CMP_GT_F32_e64, f32>;
defm : FCMP_Pattern <COND_OGE, V_CMP_GE_F32_e64, f32>; defm : FCMP_Pattern <COND_OGE, V_CMP_GE_F32_e64, f32>;
defm : FCMP_Pattern <COND_OLT, V_CMP_LT_F32_e64, f32>; defm : FCMP_Pattern <COND_OLT, V_CMP_LT_F32_e64, f32>;
defm : FCMP_Pattern <COND_OLE, V_CMP_LE_F32_e64, f32>; defm : FCMP_Pattern <COND_OLE, V_CMP_LE_F32_e64, f32>;
▲ Show 20 Lines • Show All 635 Lines • Show Last 20 Lines

lib/Transforms/InstCombine/InstCombineCalls.cpp

Show First 20 Lines • Show All 3,727 Lines • ▼ Show 20 Lines if (match(Src1, m_Zero()) &&
SrcLHS = Builder.CreateZExt(SrcLHS, CmpTy); SrcLHS = Builder.CreateZExt(SrcLHS, CmpTy);
SrcRHS = Builder.CreateZExt(SrcRHS, CmpTy); SrcRHS = Builder.CreateZExt(SrcRHS, CmpTy);
} }
} }
} else if (!Ty->isFloatTy() && !Ty->isDoubleTy() && !Ty->isHalfTy()) } else if (!Ty->isFloatTy() && !Ty->isDoubleTy() && !Ty->isHalfTy())
break; break;
Function *NewF = Function *NewF =
Intrinsic::getDeclaration(II->getModule(), NewIID, SrcLHS->getType()); Intrinsic::getDeclaration(II->getModule(), NewIID,
{ II->getType(),
SrcLHS->getType() });
Value *Args[] = { SrcLHS, SrcRHS, Value *Args[] = { SrcLHS, SrcRHS,
ConstantInt::get(CC->getType(), SrcPred) }; ConstantInt::get(CC->getType(), SrcPred) };
CallInst *NewCall = Builder.CreateCall(NewF, Args); CallInst *NewCall = Builder.CreateCall(NewF, Args);
NewCall->takeName(II); NewCall->takeName(II);
return replaceInstUsesWith(*II, NewCall); return replaceInstUsesWith(*II, NewCall);
} }
break; break;
▲ Show 20 Lines • Show All 950 Lines • Show Last 20 Lines

test/CodeGen/AMDGPU/add3.ll

Show All 16 Lines
; GFX9-LABEL: add3: ; GFX9-LABEL: add3:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_add3_u32 v0, v0, v1, v2 ; GFX9-NEXT: v_add3_u32 v0, v0, v1, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add3: ; GFX10-LABEL: add3:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_add3_u32 v0, v0, v1, v2 ; GFX10-NEXT: v_add3_u32 v0, v0, v1, v2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = add i32 %a, %b %x = add i32 %a, %b
%result = add i32 %x, %c %result = add i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
; V_MAD_U32_U24 is given higher priority. ; V_MAD_U32_U24 is given higher priority.
define amdgpu_ps float @mad_no_add3(i32 %a, i32 %b, i32 %c, i32 %d, i32 %e) { define amdgpu_ps float @mad_no_add3(i32 %a, i32 %b, i32 %c, i32 %d, i32 %e) {
; VI-LABEL: mad_no_add3: ; VI-LABEL: mad_no_add3:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_mad_u32_u24 v0, v0, v1, v4 ; VI-NEXT: v_mad_u32_u24 v0, v0, v1, v4
; VI-NEXT: v_mad_u32_u24 v0, v2, v3, v0 ; VI-NEXT: v_mad_u32_u24 v0, v2, v3, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: mad_no_add3: ; GFX9-LABEL: mad_no_add3:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_mad_u32_u24 v0, v0, v1, v4 ; GFX9-NEXT: v_mad_u32_u24 v0, v0, v1, v4
; GFX9-NEXT: v_mad_u32_u24 v0, v2, v3, v0 ; GFX9-NEXT: v_mad_u32_u24 v0, v2, v3, v0
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: mad_no_add3: ; GFX10-LABEL: mad_no_add3:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_mad_u32_u24 v0, v0, v1, v4 ; GFX10-NEXT: v_mad_u32_u24 v0, v0, v1, v4
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: v_mad_u32_u24 v0, v2, v3, v0 ; GFX10-NEXT: v_mad_u32_u24 v0, v2, v3, v0
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%a0 = shl i32 %a, 8 %a0 = shl i32 %a, 8
%a1 = lshr i32 %a0, 8 %a1 = lshr i32 %a0, 8
%b0 = shl i32 %b, 8 %b0 = shl i32 %b, 8
%b1 = lshr i32 %b0, 8 %b1 = lshr i32 %b0, 8
%mul1 = mul i32 %a1, %b1 %mul1 = mul i32 %a1, %b1
Show All 23 Lines
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: s_add_i32 s3, s3, s2 ; GFX9-NEXT: s_add_i32 s3, s3, s2
; GFX9-NEXT: v_add_u32_e32 v0, s3, v0 ; GFX9-NEXT: v_add_u32_e32 v0, s3, v0
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add3_vgpr_b: ; GFX10-LABEL: add3_vgpr_b:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_add3_u32 v0, s3, s2, v0 ; GFX10-NEXT: v_add3_u32 v0, s3, s2, v0
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = add i32 %a, %b %x = add i32 %a, %b
%result = add i32 %x, %c %result = add i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @add3_vgpr_all2(i32 %a, i32 %b, i32 %c) { define amdgpu_ps float @add3_vgpr_all2(i32 %a, i32 %b, i32 %c) {
; VI-LABEL: add3_vgpr_all2: ; VI-LABEL: add3_vgpr_all2:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_add_u32_e32 v1, vcc, v1, v2 ; VI-NEXT: v_add_u32_e32 v1, vcc, v1, v2
; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1 ; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: add3_vgpr_all2: ; GFX9-LABEL: add3_vgpr_all2:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_add3_u32 v0, v1, v2, v0 ; GFX9-NEXT: v_add3_u32 v0, v1, v2, v0
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add3_vgpr_all2: ; GFX10-LABEL: add3_vgpr_all2:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_add3_u32 v0, v1, v2, v0 ; GFX10-NEXT: v_add3_u32 v0, v1, v2, v0
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = add i32 %b, %c %x = add i32 %b, %c
%result = add i32 %a, %x %result = add i32 %a, %x
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @add3_vgpr_bc(i32 inreg %a, i32 %b, i32 %c) { define amdgpu_ps float @add3_vgpr_bc(i32 inreg %a, i32 %b, i32 %c) {
; VI-LABEL: add3_vgpr_bc: ; VI-LABEL: add3_vgpr_bc:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v0 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v0
; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1 ; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: add3_vgpr_bc: ; GFX9-LABEL: add3_vgpr_bc:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_add3_u32 v0, s2, v0, v1 ; GFX9-NEXT: v_add3_u32 v0, s2, v0, v1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add3_vgpr_bc: ; GFX10-LABEL: add3_vgpr_bc:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_add3_u32 v0, s2, v0, v1 ; GFX10-NEXT: v_add3_u32 v0, s2, v0, v1
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = add i32 %a, %b %x = add i32 %a, %b
%result = add i32 %x, %c %result = add i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @add3_vgpr_const(i32 %a, i32 %b) { define amdgpu_ps float @add3_vgpr_const(i32 %a, i32 %b) {
; VI-LABEL: add3_vgpr_const: ; VI-LABEL: add3_vgpr_const:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1 ; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1
; VI-NEXT: v_add_u32_e32 v0, vcc, 16, v0 ; VI-NEXT: v_add_u32_e32 v0, vcc, 16, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: add3_vgpr_const: ; GFX9-LABEL: add3_vgpr_const:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_add3_u32 v0, v0, v1, 16 ; GFX9-NEXT: v_add3_u32 v0, v0, v1, 16
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add3_vgpr_const: ; GFX10-LABEL: add3_vgpr_const:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_add3_u32 v0, v0, v1, 16 ; GFX10-NEXT: v_add3_u32 v0, v0, v1, 16
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = add i32 %a, %b %x = add i32 %a, %b
%result = add i32 %x, 16 %result = add i32 %x, 16
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps <2 x float> @add3_multiuse_outer(i32 %a, i32 %b, i32 %c, i32 %x) { define amdgpu_ps <2 x float> @add3_multiuse_outer(i32 %a, i32 %b, i32 %c, i32 %x) {
; VI-LABEL: add3_multiuse_outer: ; VI-LABEL: add3_multiuse_outer:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1 ; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1
; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v2 ; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v2
; VI-NEXT: v_mul_lo_u32 v1, v0, v3 ; VI-NEXT: v_mul_lo_u32 v1, v0, v3
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: add3_multiuse_outer: ; GFX9-LABEL: add3_multiuse_outer:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_add3_u32 v0, v0, v1, v2 ; GFX9-NEXT: v_add3_u32 v0, v0, v1, v2
; GFX9-NEXT: v_mul_lo_u32 v1, v0, v3 ; GFX9-NEXT: v_mul_lo_u32 v1, v0, v3
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add3_multiuse_outer: ; GFX10-LABEL: add3_multiuse_outer:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_add3_u32 v0, v0, v1, v2 ; GFX10-NEXT: v_add3_u32 v0, v0, v1, v2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: v_mul_lo_u32 v1, v0, v3 ; GFX10-NEXT: v_mul_lo_u32 v1, v0, v3
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%inner = add i32 %a, %b %inner = add i32 %a, %b
%outer = add i32 %inner, %c %outer = add i32 %inner, %c
%x1 = mul i32 %outer, %x %x1 = mul i32 %outer, %x
%r1 = insertelement <2 x i32> undef, i32 %outer, i32 0 %r1 = insertelement <2 x i32> undef, i32 %outer, i32 0
%r0 = insertelement <2 x i32> %r1, i32 %x1, i32 1 %r0 = insertelement <2 x i32> %r1, i32 %x1, i32 1
%bc = bitcast <2 x i32> %r0 to <2 x float> %bc = bitcast <2 x i32> %r0 to <2 x float>
Show All 11 Lines
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_add_u32_e32 v0, v0, v1 ; GFX9-NEXT: v_add_u32_e32 v0, v0, v1
; GFX9-NEXT: v_add_u32_e32 v1, v0, v2 ; GFX9-NEXT: v_add_u32_e32 v1, v0, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add3_multiuse_inner: ; GFX10-LABEL: add3_multiuse_inner:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_add_nc_u32_e32 v0, v0, v1 ; GFX10-NEXT: v_add_nc_u32_e32 v0, v0, v1
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: v_add_nc_u32_e32 v1, v0, v2 ; GFX10-NEXT: v_add_nc_u32_e32 v1, v0, v2
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%inner = add i32 %a, %b %inner = add i32 %a, %b
%outer = add i32 %inner, %c %outer = add i32 %inner, %c
%r1 = insertelement <2 x i32> undef, i32 %inner, i32 0 %r1 = insertelement <2 x i32> undef, i32 %inner, i32 0
%r0 = insertelement <2 x i32> %r1, i32 %outer, i32 1 %r0 = insertelement <2 x i32> %r1, i32 %outer, i32 1
%bc = bitcast <2 x i32> %r0 to <2 x float> %bc = bitcast <2 x i32> %r0 to <2 x float>
ret <2 x float> %bc ret <2 x float> %bc
Show All 22 Lines
; GFX9-NEXT: v_add_u32_e32 v0, v0, v2 ; GFX9-NEXT: v_add_u32_e32 v0, v0, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add3_uniform_vgpr: ; GFX10-LABEL: add3_uniform_vgpr:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_add_f32_e64 v1, s3, 2.0 ; GFX10-NEXT: v_add_f32_e64 v1, s3, 2.0
; GFX10-NEXT: v_add_f32_e64 v2, s2, 1.0 ; GFX10-NEXT: v_add_f32_e64 v2, s2, 1.0
; GFX10-NEXT: v_add_f32_e64 v0, 0x40400000, s4 ; GFX10-NEXT: v_add_f32_e64 v0, 0x40400000, s4
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: v_add_nc_u32_e32 v1, v2, v1 ; GFX10-NEXT: v_add_nc_u32_e32 v1, v2, v1
; GFX10-NEXT: v_add_nc_u32_e32 v0, v1, v0 ; GFX10-NEXT: v_add_nc_u32_e32 v0, v1, v0
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%a1 = fadd float %a, 1.0 %a1 = fadd float %a, 1.0
%b2 = fadd float %b, 2.0 %b2 = fadd float %b, 2.0
%c3 = fadd float %c, 3.0 %c3 = fadd float %c, 3.0
%bc.a = bitcast float %a1 to i32 %bc.a = bitcast float %a1 to i32
%bc.b = bitcast float %b2 to i32 %bc.b = bitcast float %b2 to i32
%bc.c = bitcast float %c3 to i32 %bc.c = bitcast float %c3 to i32
%x = add i32 %bc.a, %bc.b %x = add i32 %bc.a, %bc.b
%result = add i32 %x, %bc.c %result = add i32 %x, %bc.c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }

test/CodeGen/AMDGPU/add_i1.ll

; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s ; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX9 %s
; RUN: llc -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX10 %s
; GCN-LABEL: {{^}}add_var_var_i1: ; GCN-LABEL: {{^}}add_var_var_i1:
; GCN: s_xor_b64 ; GFX9: s_xor_b64
; GFX10: s_xor_b32
define amdgpu_kernel void @add_var_var_i1(i1 addrspace(1)* %out, i1 addrspace(1)* %in0, i1 addrspace(1)* %in1) { define amdgpu_kernel void @add_var_var_i1(i1 addrspace(1)* %out, i1 addrspace(1)* %in0, i1 addrspace(1)* %in1) {
%a = load volatile i1, i1 addrspace(1)* %in0 %a = load volatile i1, i1 addrspace(1)* %in0
%b = load volatile i1, i1 addrspace(1)* %in1 %b = load volatile i1, i1 addrspace(1)* %in1
%add = add i1 %a, %b %add = add i1 %a, %b
store i1 %add, i1 addrspace(1)* %out store i1 %add, i1 addrspace(1)* %out
ret void ret void
} }
; GCN-LABEL: {{^}}add_var_imm_i1: ; GCN-LABEL: {{^}}add_var_imm_i1:
; GCN: s_not_b64 ; GFX9: s_not_b64
; GFX10: s_not_b32
define amdgpu_kernel void @add_var_imm_i1(i1 addrspace(1)* %out, i1 addrspace(1)* %in) { define amdgpu_kernel void @add_var_imm_i1(i1 addrspace(1)* %out, i1 addrspace(1)* %in) {
%a = load volatile i1, i1 addrspace(1)* %in %a = load volatile i1, i1 addrspace(1)* %in
%add = add i1 %a, 1 %add = add i1 %a, 1
store i1 %add, i1 addrspace(1)* %out store i1 %add, i1 addrspace(1)* %out
ret void ret void
} }
; GCN-LABEL: {{^}}add_i1_cf: ; GCN-LABEL: {{^}}add_i1_cf:
; GCN: ; %endif ; GCN: ; %endif
; GCN: s_not_b64 ; GFX9: s_not_b64
; GFX10: s_not_b32
define amdgpu_kernel void @add_i1_cf(i1 addrspace(1)* %out, i1 addrspace(1)* %a, i1 addrspace(1)* %b) { define amdgpu_kernel void @add_i1_cf(i1 addrspace(1)* %out, i1 addrspace(1)* %a, i1 addrspace(1)* %b) {
entry: entry:
%tid = call i32 @llvm.amdgcn.workitem.id.x() %tid = call i32 @llvm.amdgcn.workitem.id.x()
%d_cmp = icmp ult i32 %tid, 16 %d_cmp = icmp ult i32 %tid, 16
br i1 %d_cmp, label %if, label %else br i1 %d_cmp, label %if, label %else
if: if:
%0 = load volatile i1, i1 addrspace(1)* %a %0 = load volatile i1, i1 addrspace(1)* %a
Show All 14 Lines

test/CodeGen/AMDGPU/add_shl.ll

Show All 16 Lines
; GFX9-LABEL: add_shl: ; GFX9-LABEL: add_shl:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_add_lshl_u32 v0, v0, v1, v2 ; GFX9-NEXT: v_add_lshl_u32 v0, v0, v1, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add_shl: ; GFX10-LABEL: add_shl:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_add_lshl_u32 v0, v0, v1, v2 ; GFX10-NEXT: v_add_lshl_u32 v0, v0, v1, v2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = add i32 %a, %b %x = add i32 %a, %b
%result = shl i32 %x, %c %result = shl i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @add_shl_vgpr_c(i32 inreg %a, i32 inreg %b, i32 %c) { define amdgpu_ps float @add_shl_vgpr_c(i32 inreg %a, i32 inreg %b, i32 %c) {
; VI-LABEL: add_shl_vgpr_c: ; VI-LABEL: add_shl_vgpr_c:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: s_add_i32 s2, s2, s3 ; VI-NEXT: s_add_i32 s2, s2, s3
; VI-NEXT: v_lshlrev_b32_e64 v0, v0, s2 ; VI-NEXT: v_lshlrev_b32_e64 v0, v0, s2
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: add_shl_vgpr_c: ; GFX9-LABEL: add_shl_vgpr_c:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: s_add_i32 s2, s2, s3 ; GFX9-NEXT: s_add_i32 s2, s2, s3
; GFX9-NEXT: v_lshlrev_b32_e64 v0, v0, s2 ; GFX9-NEXT: v_lshlrev_b32_e64 v0, v0, s2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add_shl_vgpr_c: ; GFX10-LABEL: add_shl_vgpr_c:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_add_lshl_u32 v0, s2, s3, v0 ; GFX10-NEXT: v_add_lshl_u32 v0, s2, s3, v0
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = add i32 %a, %b %x = add i32 %a, %b
%result = shl i32 %x, %c %result = shl i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @add_shl_vgpr_ac(i32 %a, i32 inreg %b, i32 %c) { define amdgpu_ps float @add_shl_vgpr_ac(i32 %a, i32 inreg %b, i32 %c) {
; VI-LABEL: add_shl_vgpr_ac: ; VI-LABEL: add_shl_vgpr_ac:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v0 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v0
; VI-NEXT: v_lshlrev_b32_e32 v0, v1, v0 ; VI-NEXT: v_lshlrev_b32_e32 v0, v1, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: add_shl_vgpr_ac: ; GFX9-LABEL: add_shl_vgpr_ac:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_add_lshl_u32 v0, v0, s2, v1 ; GFX9-NEXT: v_add_lshl_u32 v0, v0, s2, v1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add_shl_vgpr_ac: ; GFX10-LABEL: add_shl_vgpr_ac:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_add_lshl_u32 v0, v0, s2, v1 ; GFX10-NEXT: v_add_lshl_u32 v0, v0, s2, v1
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = add i32 %a, %b %x = add i32 %a, %b
%result = shl i32 %x, %c %result = shl i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @add_shl_vgpr_const(i32 %a, i32 %b) { define amdgpu_ps float @add_shl_vgpr_const(i32 %a, i32 %b) {
; VI-LABEL: add_shl_vgpr_const: ; VI-LABEL: add_shl_vgpr_const:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1 ; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1
; VI-NEXT: v_lshlrev_b32_e32 v0, 9, v0 ; VI-NEXT: v_lshlrev_b32_e32 v0, 9, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: add_shl_vgpr_const: ; GFX9-LABEL: add_shl_vgpr_const:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_add_lshl_u32 v0, v0, v1, 9 ; GFX9-NEXT: v_add_lshl_u32 v0, v0, v1, 9
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add_shl_vgpr_const: ; GFX10-LABEL: add_shl_vgpr_const:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_add_lshl_u32 v0, v0, v1, 9 ; GFX10-NEXT: v_add_lshl_u32 v0, v0, v1, 9
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = add i32 %a, %b %x = add i32 %a, %b
%result = shl i32 %x, 9 %result = shl i32 %x, 9
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @add_shl_vgpr_const_inline_const(i32 %a) { define amdgpu_ps float @add_shl_vgpr_const_inline_const(i32 %a) {
; VI-LABEL: add_shl_vgpr_const_inline_const: ; VI-LABEL: add_shl_vgpr_const_inline_const:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_lshlrev_b32_e32 v0, 9, v0 ; VI-NEXT: v_lshlrev_b32_e32 v0, 9, v0
; VI-NEXT: v_add_u32_e32 v0, vcc, 0x7e800, v0 ; VI-NEXT: v_add_u32_e32 v0, vcc, 0x7e800, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: add_shl_vgpr_const_inline_const: ; GFX9-LABEL: add_shl_vgpr_const_inline_const:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_mov_b32_e32 v1, 0x7e800 ; GFX9-NEXT: v_mov_b32_e32 v1, 0x7e800
; GFX9-NEXT: v_lshl_add_u32 v0, v0, 9, v1 ; GFX9-NEXT: v_lshl_add_u32 v0, v0, 9, v1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add_shl_vgpr_const_inline_const: ; GFX10-LABEL: add_shl_vgpr_const_inline_const:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_add_u32 v0, v0, 9, 0x7e800 ; GFX10-NEXT: v_lshl_add_u32 v0, v0, 9, 0x7e800
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = add i32 %a, 1012 %x = add i32 %a, 1012
%result = shl i32 %x, 9 %result = shl i32 %x, 9
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
; TODO: Non-optimal code generation because SelectionDAG combines ; TODO: Non-optimal code generation because SelectionDAG combines
Show All 10 Lines
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_mov_b32_e32 v1, 0x600 ; GFX9-NEXT: v_mov_b32_e32 v1, 0x600
; GFX9-NEXT: v_lshl_add_u32 v0, v0, 9, v1 ; GFX9-NEXT: v_lshl_add_u32 v0, v0, 9, v1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: add_shl_vgpr_inline_const_x2: ; GFX10-LABEL: add_shl_vgpr_inline_const_x2:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_add_u32 v0, v0, 9, 0x600 ; GFX10-NEXT: v_lshl_add_u32 v0, v0, 9, 0x600
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = add i32 %a, 3 %x = add i32 %a, 3
%result = shl i32 %x, 9 %result = shl i32 %x, 9
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }

test/CodeGen/AMDGPU/and_or.ll

Show All 16 Lines
; GFX9-LABEL: and_or: ; GFX9-LABEL: and_or:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_and_or_b32 v0, v0, v1, v2 ; GFX9-NEXT: v_and_or_b32 v0, v0, v1, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: and_or: ; GFX10-LABEL: and_or:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_and_or_b32 v0, v0, v1, v2 ; GFX10-NEXT: v_and_or_b32 v0, v0, v1, v2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = and i32 %a, %b %x = and i32 %a, %b
%result = or i32 %x, %c %result = or i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
; ThreeOp instruction variant not used due to Constant Bus Limitations ; ThreeOp instruction variant not used due to Constant Bus Limitations
define amdgpu_ps float @and_or_vgpr_b(i32 inreg %a, i32 %b, i32 inreg %c) { define amdgpu_ps float @and_or_vgpr_b(i32 inreg %a, i32 %b, i32 inreg %c) {
; VI-LABEL: and_or_vgpr_b: ; VI-LABEL: and_or_vgpr_b:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_and_b32_e32 v0, s2, v0 ; VI-NEXT: v_and_b32_e32 v0, s2, v0
; VI-NEXT: v_or_b32_e32 v0, s3, v0 ; VI-NEXT: v_or_b32_e32 v0, s3, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: and_or_vgpr_b: ; GFX9-LABEL: and_or_vgpr_b:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_and_b32_e32 v0, s2, v0 ; GFX9-NEXT: v_and_b32_e32 v0, s2, v0
; GFX9-NEXT: v_or_b32_e32 v0, s3, v0 ; GFX9-NEXT: v_or_b32_e32 v0, s3, v0
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: and_or_vgpr_b: ; GFX10-LABEL: and_or_vgpr_b:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_and_or_b32 v0, s2, v0, s3 ; GFX10-NEXT: v_and_or_b32 v0, s2, v0, s3
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = and i32 %a, %b %x = and i32 %a, %b
%result = or i32 %x, %c %result = or i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @and_or_vgpr_ab(i32 %a, i32 %b, i32 inreg %c) { define amdgpu_ps float @and_or_vgpr_ab(i32 %a, i32 %b, i32 inreg %c) {
; VI-LABEL: and_or_vgpr_ab: ; VI-LABEL: and_or_vgpr_ab:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_and_b32_e32 v0, v0, v1 ; VI-NEXT: v_and_b32_e32 v0, v0, v1
; VI-NEXT: v_or_b32_e32 v0, s2, v0 ; VI-NEXT: v_or_b32_e32 v0, s2, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: and_or_vgpr_ab: ; GFX9-LABEL: and_or_vgpr_ab:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_and_or_b32 v0, v0, v1, s2 ; GFX9-NEXT: v_and_or_b32 v0, v0, v1, s2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: and_or_vgpr_ab: ; GFX10-LABEL: and_or_vgpr_ab:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_and_or_b32 v0, v0, v1, s2 ; GFX10-NEXT: v_and_or_b32 v0, v0, v1, s2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = and i32 %a, %b %x = and i32 %a, %b
%result = or i32 %x, %c %result = or i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @and_or_vgpr_const(i32 %a, i32 %b) { define amdgpu_ps float @and_or_vgpr_const(i32 %a, i32 %b) {
; VI-LABEL: and_or_vgpr_const: ; VI-LABEL: and_or_vgpr_const:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_and_b32_e32 v0, 4, v0 ; VI-NEXT: v_and_b32_e32 v0, 4, v0
; VI-NEXT: v_or_b32_e32 v0, v0, v1 ; VI-NEXT: v_or_b32_e32 v0, v0, v1
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: and_or_vgpr_const: ; GFX9-LABEL: and_or_vgpr_const:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_and_or_b32 v0, v0, 4, v1 ; GFX9-NEXT: v_and_or_b32 v0, v0, 4, v1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: and_or_vgpr_const: ; GFX10-LABEL: and_or_vgpr_const:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_and_or_b32 v0, v0, 4, v1 ; GFX10-NEXT: v_and_or_b32 v0, v0, 4, v1
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = and i32 4, %a %x = and i32 4, %a
%result = or i32 %x, %b %result = or i32 %x, %b
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @and_or_vgpr_const_inline_const(i32 %a) { define amdgpu_ps float @and_or_vgpr_const_inline_const(i32 %a) {
; VI-LABEL: and_or_vgpr_const_inline_const: ; VI-LABEL: and_or_vgpr_const_inline_const:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_and_b32_e32 v0, 20, v0 ; VI-NEXT: v_and_b32_e32 v0, 20, v0
; VI-NEXT: v_or_b32_e32 v0, 0x808, v0 ; VI-NEXT: v_or_b32_e32 v0, 0x808, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: and_or_vgpr_const_inline_const: ; GFX9-LABEL: and_or_vgpr_const_inline_const:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_mov_b32_e32 v1, 0x808 ; GFX9-NEXT: v_mov_b32_e32 v1, 0x808
; GFX9-NEXT: v_and_or_b32 v0, v0, 20, v1 ; GFX9-NEXT: v_and_or_b32 v0, v0, 20, v1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: and_or_vgpr_const_inline_const: ; GFX10-LABEL: and_or_vgpr_const_inline_const:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_and_or_b32 v0, v0, 20, 0x808 ; GFX10-NEXT: v_and_or_b32 v0, v0, 20, 0x808
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = and i32 20, %a %x = and i32 20, %a
%result = or i32 %x, 2056 %result = or i32 %x, 2056
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @and_or_vgpr_inline_const_x2(i32 %a) { define amdgpu_ps float @and_or_vgpr_inline_const_x2(i32 %a) {
; VI-LABEL: and_or_vgpr_inline_const_x2: ; VI-LABEL: and_or_vgpr_inline_const_x2:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_and_b32_e32 v0, 4, v0 ; VI-NEXT: v_and_b32_e32 v0, 4, v0
; VI-NEXT: v_or_b32_e32 v0, 1, v0 ; VI-NEXT: v_or_b32_e32 v0, 1, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: and_or_vgpr_inline_const_x2: ; GFX9-LABEL: and_or_vgpr_inline_const_x2:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_and_or_b32 v0, v0, 4, 1 ; GFX9-NEXT: v_and_or_b32 v0, v0, 4, 1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: and_or_vgpr_inline_const_x2: ; GFX10-LABEL: and_or_vgpr_inline_const_x2:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_and_or_b32 v0, v0, 4, 1 ; GFX10-NEXT: v_and_or_b32 v0, v0, 4, 1
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = and i32 4, %a %x = and i32 4, %a
%result = or i32 %x, 1 %result = or i32 %x, 1
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }

test/CodeGen/AMDGPU/basic-branch.ll

; RUN: llc -O0 -march=amdgcn -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=GCNNOOPT -check-prefix=GCN %s ; RUN: llc -O0 -march=amdgcn -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=GCNNOOPT -check-prefix=GCN %s
; RUN: llc -O0 -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -amdgpu-spill-sgpr-to-smem=0 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=GCNNOOPT -check-prefix=GCN %s ; RUN: llc -O0 -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -amdgpu-spill-sgpr-to-smem=0 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=GCNNOOPT -check-prefix=GCN %s
; RUN: llc -O0 -march=amdgcn -mcpu=gfx1010 -mattr=-flat-for-global,-WavefrontSize32,+WavefrontSize64 -amdgpu-spill-sgpr-to-smem=0 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=GCNNOOPT -check-prefix=GCN %s
; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=GCNOPT -check-prefix=GCN %s ; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=GCNOPT -check-prefix=GCN %s
; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=GCNOPT -check-prefix=GCN %s ; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=GCNOPT -check-prefix=GCN %s
; GCN-LABEL: {{^}}test_branch: ; GCN-LABEL: {{^}}test_branch:
; GCNNOOPT: v_writelane_b32 ; GCNNOOPT: v_writelane_b32
; GCNNOOPT: v_writelane_b32 ; GCNNOOPT: v_writelane_b32
; GCN: s_cbranch_scc1 [[END:BB[0-9]+_[0-9]+]] ; GCN: s_cbranch_scc1 [[END:BB[0-9]+_[0-9]+]]
▲ Show 20 Lines • Show All 44 Lines • Show Last 20 Lines

test/CodeGen/AMDGPU/diverge-switch-default.ll

Show All 32 Lines
; is sensitive to optimizations; so we just ensure that the relevant ; is sensitive to optimizations; so we just ensure that the relevant
; operations in the block body are indeed in the same block. ; operations in the block body are indeed in the same block.
; CHECK: [[PHI:%[a-zA-Z0-9._]+]] = phi i64 ; CHECK: [[PHI:%[a-zA-Z0-9._]+]] = phi i64
; CHECK-NOT: {{ br }} ; CHECK-NOT: {{ br }}
; CHECK: load i8 ; CHECK: load i8
; CHECK-NOT: {{ br }} ; CHECK-NOT: {{ br }}
; CHECK: [[ICMP:%[a-zA-Z0-9._]+]] = icmp eq ; CHECK: [[ICMP:%[a-zA-Z0-9._]+]] = icmp eq
; CHECK: [[IF:%[a-zA-Z0-9._]+]] = call i64 @llvm.amdgcn.if.break(i1 [[ICMP]], i64 [[PHI]]) ; CHECK: [[IF:%[a-zA-Z0-9._]+]] = call i64 @llvm.amdgcn.if.break.i64.i64(i1 [[ICMP]], i64 [[PHI]])
; CHECK: [[LOOP:%[a-zA-Z0-9._]+]] = call i1 @llvm.amdgcn.loop(i64 [[IF]]) ; CHECK: [[LOOP:%[a-zA-Z0-9._]+]] = call i1 @llvm.amdgcn.loop.i64(i64 [[IF]])
; CHECK: br i1 [[LOOP]] ; CHECK: br i1 [[LOOP]]
sw.while: sw.while:
%p = phi i8 addrspace(1)* [ %gep_in, %sw.epilog ], [ %incdec.ptr, %sw.while ] %p = phi i8 addrspace(1)* [ %gep_in, %sw.epilog ], [ %incdec.ptr, %sw.while ]
%count = phi i32 [ 0, %sw.epilog ], [ %count.inc, %sw.while ] %count = phi i32 [ 0, %sw.epilog ], [ %count.inc, %sw.while ]
%char = load i8, i8 addrspace(1)* %p, align 1 %char = load i8, i8 addrspace(1)* %p, align 1
%tobool = icmp eq i8 %char, 0 %tobool = icmp eq i8 %char, 0
%incdec.ptr = getelementptr inbounds i8, i8 addrspace(1)* %p, i64 1 %incdec.ptr = getelementptr inbounds i8, i8 addrspace(1)* %p, i64 1
Show All 14 Lines

test/CodeGen/AMDGPU/hazard-hidden-bundle.mir

# RUN: llc -march=amdgcn -mcpu=gfx902 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,XNACK,GCX9 %s # RUN: llc -march=amdgcn -mcpu=gfx902 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,XNACK,GCX9 %s
# RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,NOXNACK,GFX9 %s # RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,NOXNACK,GFX9 %s
# RUN: llc -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,NOXNACK,GFX10 %s # RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-WavefrontSize32,+WavefrontSize64 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,NOXNACK,GFX10 %s
# GCN-LABEL: name: break_smem_clause_simple_load_smrd8_ptr_hidden_bundle # GCN-LABEL: name: break_smem_clause_simple_load_smrd8_ptr_hidden_bundle
# GCN: } # GCN: }
# XNACK-NEXT: S_NOP # XNACK-NEXT: S_NOP
# NOXNACK-NOT: S_NOP # NOXNACK-NOT: S_NOP
# GCN: S_LOAD_DWORDX2_IMM # GCN: S_LOAD_DWORDX2_IMM
--- ---
name: break_smem_clause_simple_load_smrd8_ptr_hidden_bundle name: break_smem_clause_simple_load_smrd8_ptr_hidden_bundle
▲ Show 20 Lines • Show All 83 Lines • Show Last 20 Lines

test/CodeGen/AMDGPU/hazard-in-bundle.mir

# RUN: llc -march=amdgcn -mcpu=gfx902 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,XNACK,GFX9 %s # RUN: llc -march=amdgcn -mcpu=gfx902 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,XNACK,GFX9 %s
# RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,NOXNACK,GFX9 %s # RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,NOXNACK,GFX9 %s
# RUN: llc -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,NOXNACK,GFX10 %s # RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-WavefrontSize32,+WavefrontSize64 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,NOXNACK,GFX10 %s
# GCN-LABEL: name: break_smem_clause_max_look_ahead_in_bundle # GCN-LABEL: name: break_smem_clause_max_look_ahead_in_bundle
# GCN: S_LOAD_DWORDX2_IMM # GCN: S_LOAD_DWORDX2_IMM
# XNACK-NEXT: S_NOP # XNACK-NEXT: S_NOP
# NOXNACK-NOT: S_NOP # NOXNACK-NOT: S_NOP
# GCN: S_LOAD_DWORDX2 # GCN: S_LOAD_DWORDX2
# XNACK-NEXT: S_NOP # XNACK-NEXT: S_NOP
# NOXNACK-NOT: S_NOP # NOXNACK-NOT: S_NOP
▲ Show 20 Lines • Show All 55 Lines • Show Last 20 Lines

test/CodeGen/AMDGPU/huge-private-buffer.ll

; RUN: llc -mtriple=amdgcn-amd-amdhsa -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=GCN %s ; RUN: llc -mtriple=amdgcn-amd-amdhsa -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,WAVE64 %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,WAVE32 %s
; GCN-LABEL: {{^}}scratch_buffer_known_high_masklo14:
; GCN: v_mov_b32_e32 [[FI:v[0-9]+]], 4
; GCN: v_and_b32_e32 [[MASKED:v[0-9]+]], 0x3ffc, [[FI]]
; GCN: {{flat|global}}_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[MASKED]]
define amdgpu_kernel void @scratch_buffer_known_high_masklo14() #0 {
%alloca = alloca i32, align 4, addrspace(5)
store volatile i32 0, i32 addrspace(5)* %alloca
%toint = ptrtoint i32 addrspace(5)* %alloca to i32
%masked = and i32 %toint, 16383
store volatile i32 %masked, i32 addrspace(1)* undef
ret void
}
; GCN-LABEL: {{^}}scratch_buffer_known_high_masklo16: ; GCN-LABEL: {{^}}scratch_buffer_known_high_masklo16:
; GCN: v_mov_b32_e32 [[FI:v[0-9]+]], 4 ; GCN: v_mov_b32_e32 [[FI:v[0-9]+]], 4
; GCN: v_and_b32_e32 [[MASKED:v[0-9]+]], 0xfffc, [[FI]] ; GCN: v_and_b32_e32 [[MASKED:v[0-9]+]], 0xfffc, [[FI]]
; GCN: flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[MASKED]] ; GCN: {{flat|global}}_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[MASKED]]
define amdgpu_kernel void @scratch_buffer_known_high_masklo16() #0 { define amdgpu_kernel void @scratch_buffer_known_high_masklo16() #0 {
%alloca = alloca i32, align 4, addrspace(5) %alloca = alloca i32, align 4, addrspace(5)
store volatile i32 0, i32 addrspace(5)* %alloca store volatile i32 0, i32 addrspace(5)* %alloca
%toint = ptrtoint i32 addrspace(5)* %alloca to i32 %toint = ptrtoint i32 addrspace(5)* %alloca to i32
%masked = and i32 %toint, 65535 %masked = and i32 %toint, 65535
store volatile i32 %masked, i32 addrspace(1)* undef store volatile i32 %masked, i32 addrspace(1)* undef
ret void ret void
} }
; GCN-LABEL: {{^}}scratch_buffer_known_high_masklo17: ; GCN-LABEL: {{^}}scratch_buffer_known_high_masklo17:
; GCN: v_mov_b32_e32 [[FI:v[0-9]+]], 4 ; GCN: v_mov_b32_e32 [[FI:v[0-9]+]], 4
; GCN-NOT: [[FI]] ; WAVE64-NOT: [[FI]]
; GCN: flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[FI]] ; WAVE64: {{flat|global}}_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[FI]]
; WAVE32: v_and_b32_e32 [[MASKED:v[0-9]+]], 0x1fffc, [[FI]]
; WAVE32: {{flat|global}}_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[MASKED]]
define amdgpu_kernel void @scratch_buffer_known_high_masklo17() #0 { define amdgpu_kernel void @scratch_buffer_known_high_masklo17() #0 {
%alloca = alloca i32, align 4, addrspace(5) %alloca = alloca i32, align 4, addrspace(5)
store volatile i32 0, i32 addrspace(5)* %alloca store volatile i32 0, i32 addrspace(5)* %alloca
%toint = ptrtoint i32 addrspace(5)* %alloca to i32 %toint = ptrtoint i32 addrspace(5)* %alloca to i32
%masked = and i32 %toint, 131071 %masked = and i32 %toint, 131071
store volatile i32 %masked, i32 addrspace(1)* undef store volatile i32 %masked, i32 addrspace(1)* undef
ret void ret void
} }
; GCN-LABEL: {{^}}scratch_buffer_known_high_mask18: ; GCN-LABEL: {{^}}scratch_buffer_known_high_mask18:
; GCN: v_mov_b32_e32 [[FI:v[0-9]+]], 4 ; GCN: v_mov_b32_e32 [[FI:v[0-9]+]], 4
; GCN-NOT: [[FI]] ; GCN-NOT: [[FI]]
; GCN: flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[FI]] ; GCN: {{flat|global}}_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[FI]]
define amdgpu_kernel void @scratch_buffer_known_high_mask18() #0 { define amdgpu_kernel void @scratch_buffer_known_high_mask18() #0 {
%alloca = alloca i32, align 4, addrspace(5) %alloca = alloca i32, align 4, addrspace(5)
store volatile i32 0, i32 addrspace(5)* %alloca store volatile i32 0, i32 addrspace(5)* %alloca
%toint = ptrtoint i32 addrspace(5)* %alloca to i32 %toint = ptrtoint i32 addrspace(5)* %alloca to i32
%masked = and i32 %toint, 262143 %masked = and i32 %toint, 262143
store volatile i32 %masked, i32 addrspace(1)* undef store volatile i32 %masked, i32 addrspace(1)* undef
ret void ret void
} }
attributes #0 = { nounwind } attributes #0 = { nounwind }

test/CodeGen/AMDGPU/insert-skip-from-vcc.mir

# RUN: llc -march=amdgcn -mcpu=fiji -run-pass si-insert-skips -verify-machineinstrs -o - %s | FileCheck -check-prefix=GCN %s # RUN: llc -march=amdgcn -mcpu=fiji -run-pass si-insert-skips -verify-machineinstrs -o - %s | FileCheck -check-prefix=GCN %s
# RUN: llc -march=amdgcn -mcpu=gfx1010 -run-pass si-insert-skips -verify-machineinstrs -o - %s | FileCheck -check-prefix=W32 %s
--- ---
# GCN-LABEL: name: and_execz_mov_vccz # GCN-LABEL: name: and_execz_mov_vccz
# GCN-NOT: S_MOV_ # GCN-NOT: S_MOV_
# GCN-NOT: S_AND_ # GCN-NOT: S_AND_
# GCN: S_CBRANCH_EXECZ %bb.1, implicit $exec # GCN: S_CBRANCH_EXECZ %bb.1, implicit $exec
name: and_execz_mov_vccz name: and_execz_mov_vccz
body: | body: |
▲ Show 20 Lines • Show All 303 Lines • ▼ Show 20 Linesbody: |
bb.1: bb.1:
S_NOP 0 S_NOP 0
bb.2: bb.2:
$vcc = S_AND_B64 $exec, -1, implicit-def $scc $vcc = S_AND_B64 $exec, -1, implicit-def $scc
S_CBRANCH_VCCZ %bb.1, implicit killed $vcc S_CBRANCH_VCCZ %bb.1, implicit killed $vcc
S_ENDPGM 0, implicit $scc S_ENDPGM 0, implicit $scc
... ...
---
# W32-LABEL: name: and_execz_mov_vccz_w32
# W32-NOT: S_MOV_
# W32-NOT: S_AND_
# W32: S_CBRANCH_EXECZ %bb.1, implicit $exec
name: and_execz_mov_vccz_w32
body: |
bb.0:
S_NOP 0
bb.1:
S_NOP 0
bb.2:
$sgpr0 = S_MOV_B32 -1
$vcc_lo = S_AND_B32 $exec_lo, killed $sgpr0, implicit-def dead $scc
S_CBRANCH_VCCZ %bb.1, implicit killed $vcc
S_ENDPGM 0
...

test/CodeGen/AMDGPU/large-work-group-promote-alloca.ll

; RUN: opt -S -mtriple=amdgcn-unknown-unknown -amdgpu-promote-alloca -disable-promote-alloca-to-vector < %s | FileCheck --check-prefix=SI --check-prefix=ALL %s ; RUN: opt -S -mtriple=amdgcn-unknown-unknown -amdgpu-promote-alloca -disable-promote-alloca-to-vector < %s | FileCheck --check-prefixes=SI,SICI,ALL %s
; RUN: opt -S -mcpu=tonga -mtriple=amdgcn-unknown-unknown -amdgpu-promote-alloca -disable-promote-alloca-to-vector < %s | FileCheck --check-prefix=CI --check-prefix=ALL %s ; RUN: opt -S -mcpu=tonga -mtriple=amdgcn-unknown-unknown -amdgpu-promote-alloca -disable-promote-alloca-to-vector < %s | FileCheck --check-prefixes=CI,SICI,ALL %s
; RUN: opt -S -mcpu=gfx1010 -mtriple=amdgcn-unknown-unknown -amdgpu-promote-alloca -disable-promote-alloca-to-vector < %s | FileCheck --check-prefixes=GFX10,ALL %s
; SI-NOT: @promote_alloca_size_63.stack = internal unnamed_addr addrspace(3) global [63 x [5 x i32]] undef, align 4 ; SI-NOT: @promote_alloca_size_63.stack = internal unnamed_addr addrspace(3) global [63 x [5 x i32]] undef, align 4
; CI: @promote_alloca_size_63.stack = internal unnamed_addr addrspace(3) global [63 x [5 x i32]] undef, align 4 ; CI: @promote_alloca_size_63.stack = internal unnamed_addr addrspace(3) global [63 x [5 x i32]] undef, align 4
define amdgpu_kernel void @promote_alloca_size_63(i32 addrspace(1)* nocapture %out, i32 addrspace(1)* nocapture %in) #0 { define amdgpu_kernel void @promote_alloca_size_63(i32 addrspace(1)* nocapture %out, i32 addrspace(1)* nocapture %in) #0 {
entry: entry:
%stack = alloca [5 x i32], align 4 %stack = alloca [5 x i32], align 4
%0 = load i32, i32 addrspace(1)* %in, align 4 %0 = load i32, i32 addrspace(1)* %in, align 4
Show All 30 Linesentry:
store i32 %2, i32 addrspace(1)* %out, align 4 store i32 %2, i32 addrspace(1)* %out, align 4
%arrayidx12 = getelementptr inbounds [5 x i32], [5 x i32]* %stack, i32 0, i32 1 %arrayidx12 = getelementptr inbounds [5 x i32], [5 x i32]* %stack, i32 0, i32 1
%3 = load i32, i32* %arrayidx12 %3 = load i32, i32* %arrayidx12
%arrayidx13 = getelementptr inbounds i32, i32 addrspace(1)* %out, i32 1 %arrayidx13 = getelementptr inbounds i32, i32 addrspace(1)* %out, i32 1
store i32 %3, i32 addrspace(1)* %arrayidx13 store i32 %3, i32 addrspace(1)* %arrayidx13
ret void ret void
} }
; ALL: @promote_alloca_size_1600.stack = internal unnamed_addr addrspace(3) global [1600 x [5 x i32]] undef, align 4 ; SICI: @promote_alloca_size_1600.stack = internal unnamed_addr addrspace(3) global [1600 x [5 x i32]] undef, align 4
; GFX10: alloca [5 x i32]
define amdgpu_kernel void @promote_alloca_size_1600(i32 addrspace(1)* nocapture %out, i32 addrspace(1)* nocapture %in) #2 { define amdgpu_kernel void @promote_alloca_size_1600(i32 addrspace(1)* nocapture %out, i32 addrspace(1)* nocapture %in) #2 {
entry: entry:
%stack = alloca [5 x i32], align 4 %stack = alloca [5 x i32], align 4
%0 = load i32, i32 addrspace(1)* %in, align 4 %0 = load i32, i32 addrspace(1)* %in, align 4
%arrayidx1 = getelementptr inbounds [5 x i32], [5 x i32]* %stack, i32 0, i32 %0 %arrayidx1 = getelementptr inbounds [5 x i32], [5 x i32]* %stack, i32 0, i32 %0
store i32 4, i32* %arrayidx1, align 4 store i32 4, i32* %arrayidx1, align 4
%arrayidx2 = getelementptr inbounds i32, i32 addrspace(1)* %in, i32 1 %arrayidx2 = getelementptr inbounds i32, i32 addrspace(1)* %in, i32 1
▲ Show 20 Lines • Show All 78 Lines • ▼ Show 20 Linesentry:
%arrayidx12 = getelementptr inbounds [42 x i8], [42 x i8]* %stack, i64 0, i64 1 %arrayidx12 = getelementptr inbounds [42 x i8], [42 x i8]* %stack, i64 0, i64 1
%tmp3 = load i8, i8* %arrayidx12, align 1 %tmp3 = load i8, i8* %arrayidx12, align 1
%arrayidx13 = getelementptr inbounds i8, i8 addrspace(1)* %out, i64 1 %arrayidx13 = getelementptr inbounds i8, i8 addrspace(1)* %out, i64 1
store i8 %tmp3, i8 addrspace(1)* %arrayidx13, align 1 store i8 %tmp3, i8 addrspace(1)* %arrayidx13, align 1
ret void ret void
} }
; ALL-LABEL: @occupancy_6_over( ; ALL-LABEL: @occupancy_6_over(
; ALL: alloca [43 x i8] ; SICI: alloca [43 x i8]
; GFX10-NOT: alloca
define amdgpu_kernel void @occupancy_6_over(i8 addrspace(1)* nocapture %out, i8 addrspace(1)* nocapture %in) #5 { define amdgpu_kernel void @occupancy_6_over(i8 addrspace(1)* nocapture %out, i8 addrspace(1)* nocapture %in) #5 {
entry: entry:
%stack = alloca [43 x i8], align 4 %stack = alloca [43 x i8], align 4
%tmp = load i8, i8 addrspace(1)* %in, align 1 %tmp = load i8, i8 addrspace(1)* %in, align 1
%tmp4 = sext i8 %tmp to i64 %tmp4 = sext i8 %tmp to i64
%arrayidx1 = getelementptr inbounds [43 x i8], [43 x i8]* %stack, i64 0, i64 %tmp4 %arrayidx1 = getelementptr inbounds [43 x i8], [43 x i8]* %stack, i64 0, i64 %tmp4
store i8 4, i8* %arrayidx1, align 1 store i8 4, i8* %arrayidx1, align 1
%arrayidx2 = getelementptr inbounds i8, i8 addrspace(1)* %in, i64 1 %arrayidx2 = getelementptr inbounds i8, i8 addrspace(1)* %in, i64 1
Show All 33 Linesentry:
%arrayidx12 = getelementptr inbounds [32 x i8], [32 x i8]* %stack, i64 0, i64 1 %arrayidx12 = getelementptr inbounds [32 x i8], [32 x i8]* %stack, i64 0, i64 1
%tmp3 = load i8, i8* %arrayidx12, align 1 %tmp3 = load i8, i8* %arrayidx12, align 1
%arrayidx13 = getelementptr inbounds i8, i8 addrspace(1)* %out, i64 1 %arrayidx13 = getelementptr inbounds i8, i8 addrspace(1)* %out, i64 1
store i8 %tmp3, i8 addrspace(1)* %arrayidx13, align 1 store i8 %tmp3, i8 addrspace(1)* %arrayidx13, align 1
ret void ret void
} }
; ALL-LABEL: @occupancy_8_over( ; ALL-LABEL: @occupancy_8_over(
; ALL: alloca [33 x i8] ; SICI: alloca [33 x i8]
; GFX10-NOT: alloca
define amdgpu_kernel void @occupancy_8_over(i8 addrspace(1)* nocapture %out, i8 addrspace(1)* nocapture %in) #6 { define amdgpu_kernel void @occupancy_8_over(i8 addrspace(1)* nocapture %out, i8 addrspace(1)* nocapture %in) #6 {
entry: entry:
%stack = alloca [33 x i8], align 4 %stack = alloca [33 x i8], align 4
%tmp = load i8, i8 addrspace(1)* %in, align 1 %tmp = load i8, i8 addrspace(1)* %in, align 1
%tmp4 = sext i8 %tmp to i64 %tmp4 = sext i8 %tmp to i64
%arrayidx1 = getelementptr inbounds [33 x i8], [33 x i8]* %stack, i64 0, i64 %tmp4 %arrayidx1 = getelementptr inbounds [33 x i8], [33 x i8]* %stack, i64 0, i64 %tmp4
store i8 4, i8* %arrayidx1, align 1 store i8 4, i8* %arrayidx1, align 1
%arrayidx2 = getelementptr inbounds i8, i8 addrspace(1)* %in, i64 1 %arrayidx2 = getelementptr inbounds i8, i8 addrspace(1)* %in, i64 1
Show All 33 Linesentry:
%arrayidx12 = getelementptr inbounds [28 x i8], [28 x i8]* %stack, i64 0, i64 1 %arrayidx12 = getelementptr inbounds [28 x i8], [28 x i8]* %stack, i64 0, i64 1
%tmp3 = load i8, i8* %arrayidx12, align 1 %tmp3 = load i8, i8* %arrayidx12, align 1
%arrayidx13 = getelementptr inbounds i8, i8 addrspace(1)* %out, i64 1 %arrayidx13 = getelementptr inbounds i8, i8 addrspace(1)* %out, i64 1
store i8 %tmp3, i8 addrspace(1)* %arrayidx13, align 1 store i8 %tmp3, i8 addrspace(1)* %arrayidx13, align 1
ret void ret void
} }
; ALL-LABEL: @occupancy_9_over( ; ALL-LABEL: @occupancy_9_over(
; ALL: alloca [29 x i8] ; SICI: alloca [29 x i8]
; GFX10-NOT: alloca
define amdgpu_kernel void @occupancy_9_over(i8 addrspace(1)* nocapture %out, i8 addrspace(1)* nocapture %in) #7 { define amdgpu_kernel void @occupancy_9_over(i8 addrspace(1)* nocapture %out, i8 addrspace(1)* nocapture %in) #7 {
entry: entry:
%stack = alloca [29 x i8], align 4 %stack = alloca [29 x i8], align 4
%tmp = load i8, i8 addrspace(1)* %in, align 1 %tmp = load i8, i8 addrspace(1)* %in, align 1
%tmp4 = sext i8 %tmp to i64 %tmp4 = sext i8 %tmp to i64
%arrayidx1 = getelementptr inbounds [29 x i8], [29 x i8]* %stack, i64 0, i64 %tmp4 %arrayidx1 = getelementptr inbounds [29 x i8], [29 x i8]* %stack, i64 0, i64 %tmp4
store i8 4, i8* %arrayidx1, align 1 store i8 4, i8* %arrayidx1, align 1
%arrayidx2 = getelementptr inbounds i8, i8 addrspace(1)* %in, i64 1 %arrayidx2 = getelementptr inbounds i8, i8 addrspace(1)* %in, i64 1
Show All 22 Lines

test/CodeGen/AMDGPU/llvm.amdgcn.kill.ll

; RUN: llc -march=amdgcn -mcpu=verde -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=SI %s ; RUN: llc -march=amdgcn -mcpu=verde -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,SI %s
; RUN: llc -march=amdgcn -mcpu=tonga -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=SI %s ; RUN: llc -march=amdgcn -mcpu=tonga -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,SI %s
; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,GFX10 %s
; SI-LABEL: {{^}}gs_const:
; SI-NOT: v_cmpx ; GCN-LABEL: {{^}}gs_const:
; SI: s_mov_b64 exec, 0 ; GCN-NOT: v_cmpx
; GCN: s_mov_b64 exec, 0
define amdgpu_gs void @gs_const() { define amdgpu_gs void @gs_const() {
%tmp = icmp ule i32 0, 3 %tmp = icmp ule i32 0, 3
%tmp1 = select i1 %tmp, float 1.000000e+00, float -1.000000e+00 %tmp1 = select i1 %tmp, float 1.000000e+00, float -1.000000e+00
%c1 = fcmp oge float %tmp1, 0.0 %c1 = fcmp oge float %tmp1, 0.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
%tmp2 = icmp ule i32 3, 0 %tmp2 = icmp ule i32 3, 0
%tmp3 = select i1 %tmp2, float 1.000000e+00, float -1.000000e+00 %tmp3 = select i1 %tmp2, float 1.000000e+00, float -1.000000e+00
%c2 = fcmp oge float %tmp3, 0.0 %c2 = fcmp oge float %tmp3, 0.0
call void @llvm.amdgcn.kill(i1 %c2) call void @llvm.amdgcn.kill(i1 %c2)
ret void ret void
} }
; SI-LABEL: {{^}}vcc_implicit_def: ; GCN-LABEL: {{^}}vcc_implicit_def:
; SI-NOT: v_cmp_gt_f32_e32 vcc, ; GCN-NOT: v_cmp_gt_f32_e32 vcc,
; SI: v_cmp_gt_f32_e64 [[CMP:s\[[0-9]+:[0-9]+\]]], 0, v{{[0-9]+}} ; GCN: v_cmp_gt_f32_e64 [[CMP:s\[[0-9]+:[0-9]+\]]], 0, v{{[0-9]+}}
; SI: v_cmpx_le_f32_e32 vcc, 0, v{{[0-9]+}} ; SI: v_cmpx_le_f32_e32 vcc, 0, v{{[0-9]+}}
; SI: v_cndmask_b32_e64 v{{[0-9]+}}, 0, 1.0, [[CMP]] ; GFX10: v_cmpx_le_f32_e32 0, v{{[0-9]+}}
; GCN: v_cndmask_b32_e64 v{{[0-9]+}}, 0, 1.0, [[CMP]]
define amdgpu_ps void @vcc_implicit_def(float %arg13, float %arg14) { define amdgpu_ps void @vcc_implicit_def(float %arg13, float %arg14) {
%tmp0 = fcmp olt float %arg13, 0.000000e+00 %tmp0 = fcmp olt float %arg13, 0.000000e+00
%c1 = fcmp oge float %arg14, 0.0 %c1 = fcmp oge float %arg14, 0.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
%tmp1 = select i1 %tmp0, float 1.000000e+00, float 0.000000e+00 %tmp1 = select i1 %tmp0, float 1.000000e+00, float 0.000000e+00
call void @llvm.amdgcn.exp.f32(i32 1, i32 15, float %tmp1, float %tmp1, float %tmp1, float %tmp1, i1 true, i1 true) #0 call void @llvm.amdgcn.exp.f32(i32 1, i32 15, float %tmp1, float %tmp1, float %tmp1, float %tmp1, i1 true, i1 true) #0
ret void ret void
} }
; SI-LABEL: {{^}}true: ; GCN-LABEL: {{^}}true:
; SI-NEXT: %bb. ; GCN-NEXT: %bb.
; SI-NEXT: %bb. ; GCN-NEXT: %bb.
; SI-NEXT: s_endpgm ; GCN-NEXT: s_endpgm
define amdgpu_gs void @true() { define amdgpu_gs void @true() {
call void @llvm.amdgcn.kill(i1 true) call void @llvm.amdgcn.kill(i1 true)
ret void ret void
} }
; SI-LABEL: {{^}}false: ; GCN-LABEL: {{^}}false:
; SI-NOT: v_cmpx ; GCN-NOT: v_cmpx
; SI: s_mov_b64 exec, 0 ; GCN: s_mov_b64 exec, 0
define amdgpu_gs void @false() { define amdgpu_gs void @false() {
call void @llvm.amdgcn.kill(i1 false) call void @llvm.amdgcn.kill(i1 false)
ret void ret void
} }
; SI-LABEL: {{^}}and: ; GCN-LABEL: {{^}}and:
; SI: v_cmp_lt_i32 ; GCN: v_cmp_lt_i32
; SI: v_cmp_lt_i32 ; GCN: v_cmp_lt_i32
; SI: s_or_b64 s[0:1] ; GCN: s_or_b64 s[0:1]
; SI: s_and_b64 exec, exec, s[0:1] ; GCN: s_and_b64 exec, exec, s[0:1]
define amdgpu_gs void @and(i32 %a, i32 %b, i32 %c, i32 %d) { define amdgpu_gs void @and(i32 %a, i32 %b, i32 %c, i32 %d) {
%c1 = icmp slt i32 %a, %b %c1 = icmp slt i32 %a, %b
%c2 = icmp slt i32 %c, %d %c2 = icmp slt i32 %c, %d
%x = or i1 %c1, %c2 %x = or i1 %c1, %c2
call void @llvm.amdgcn.kill(i1 %x) call void @llvm.amdgcn.kill(i1 %x)
ret void ret void
} }
; SI-LABEL: {{^}}andn2: ; GCN-LABEL: {{^}}andn2:
; SI: v_cmp_lt_i32 ; GCN: v_cmp_lt_i32
; SI: v_cmp_lt_i32 ; GCN: v_cmp_lt_i32
; SI: s_xor_b64 s[0:1] ; GCN: s_xor_b64 s[0:1]
; SI: s_andn2_b64 exec, exec, s[0:1] ; GCN: s_andn2_b64 exec, exec, s[0:1]
define amdgpu_gs void @andn2(i32 %a, i32 %b, i32 %c, i32 %d) { define amdgpu_gs void @andn2(i32 %a, i32 %b, i32 %c, i32 %d) {
%c1 = icmp slt i32 %a, %b %c1 = icmp slt i32 %a, %b
%c2 = icmp slt i32 %c, %d %c2 = icmp slt i32 %c, %d
%x = xor i1 %c1, %c2 %x = xor i1 %c1, %c2
%y = xor i1 %x, 1 %y = xor i1 %x, 1
call void @llvm.amdgcn.kill(i1 %y) call void @llvm.amdgcn.kill(i1 %y)
ret void ret void
} }
; SI-LABEL: {{^}}oeq: ; GCN-LABEL: {{^}}oeq:
; SI: v_cmpx_eq_f32 ; GCN: v_cmpx_eq_f32
; SI-NOT: s_and ; GCN-NOT: s_and
define amdgpu_gs void @oeq(float %a) { define amdgpu_gs void @oeq(float %a) {
%c1 = fcmp oeq float %a, 0.0 %c1 = fcmp oeq float %a, 0.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}ogt: ; GCN-LABEL: {{^}}ogt:
; SI: v_cmpx_lt_f32 ; GCN: v_cmpx_lt_f32
; SI-NOT: s_and ; GCN-NOT: s_and
define amdgpu_gs void @ogt(float %a) { define amdgpu_gs void @ogt(float %a) {
%c1 = fcmp ogt float %a, 0.0 %c1 = fcmp ogt float %a, 0.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}oge: ; GCN-LABEL: {{^}}oge:
; SI: v_cmpx_le_f32 ; GCN: v_cmpx_le_f32
; SI-NOT: s_and ; GCN-NOT: s_and
define amdgpu_gs void @oge(float %a) { define amdgpu_gs void @oge(float %a) {
%c1 = fcmp oge float %a, 0.0 %c1 = fcmp oge float %a, 0.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}olt: ; GCN-LABEL: {{^}}olt:
; SI: v_cmpx_gt_f32 ; GCN: v_cmpx_gt_f32
; SI-NOT: s_and ; GCN-NOT: s_and
define amdgpu_gs void @olt(float %a) { define amdgpu_gs void @olt(float %a) {
%c1 = fcmp olt float %a, 0.0 %c1 = fcmp olt float %a, 0.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}ole: ; GCN-LABEL: {{^}}ole:
; SI: v_cmpx_ge_f32 ; GCN: v_cmpx_ge_f32
; SI-NOT: s_and ; GCN-NOT: s_and
define amdgpu_gs void @ole(float %a) { define amdgpu_gs void @ole(float %a) {
%c1 = fcmp ole float %a, 0.0 %c1 = fcmp ole float %a, 0.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}one: ; GCN-LABEL: {{^}}one:
; SI: v_cmpx_lg_f32 ; GCN: v_cmpx_lg_f32
; SI-NOT: s_and ; GCN-NOT: s_and
define amdgpu_gs void @one(float %a) { define amdgpu_gs void @one(float %a) {
%c1 = fcmp one float %a, 0.0 %c1 = fcmp one float %a, 0.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}ord: ; GCN-LABEL: {{^}}ord:
; FIXME: This is absolutely unimportant, but we could use the cmpx variant here. ; FIXME: This is absolutely unimportant, but we could use the cmpx variant here.
; SI: v_cmp_o_f32 ; GCN: v_cmp_o_f32
define amdgpu_gs void @ord(float %a) { define amdgpu_gs void @ord(float %a) {
%c1 = fcmp ord float %a, 0.0 %c1 = fcmp ord float %a, 0.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}uno: ; GCN-LABEL: {{^}}uno:
; FIXME: This is absolutely unimportant, but we could use the cmpx variant here. ; FIXME: This is absolutely unimportant, but we could use the cmpx variant here.
; SI: v_cmp_u_f32 ; GCN: v_cmp_u_f32
define amdgpu_gs void @uno(float %a) { define amdgpu_gs void @uno(float %a) {
%c1 = fcmp uno float %a, 0.0 %c1 = fcmp uno float %a, 0.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}ueq: ; GCN-LABEL: {{^}}ueq:
; SI: v_cmpx_nlg_f32 ; GCN: v_cmpx_nlg_f32
; SI-NOT: s_and ; GCN-NOT: s_and
define amdgpu_gs void @ueq(float %a) { define amdgpu_gs void @ueq(float %a) {
%c1 = fcmp ueq float %a, 0.0 %c1 = fcmp ueq float %a, 0.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}ugt: ; GCN-LABEL: {{^}}ugt:
; SI: v_cmpx_nge_f32 ; GCN: v_cmpx_nge_f32
; SI-NOT: s_and ; GCN-NOT: s_and
define amdgpu_gs void @ugt(float %a) { define amdgpu_gs void @ugt(float %a) {
%c1 = fcmp ugt float %a, 0.0 %c1 = fcmp ugt float %a, 0.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}uge: ; GCN-LABEL: {{^}}uge:
; SI: v_cmpx_ngt_f32_e32 vcc, -1.0 ; SI: v_cmpx_ngt_f32_e32 vcc, -1.0
; SI-NOT: s_and ; GFX10: v_cmpx_ngt_f32_e32 -1.0
; GCN-NOT: s_and
define amdgpu_gs void @uge(float %a) { define amdgpu_gs void @uge(float %a) {
%c1 = fcmp uge float %a, -1.0 %c1 = fcmp uge float %a, -1.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}ult: ; GCN-LABEL: {{^}}ult:
; SI: v_cmpx_nle_f32_e32 vcc, -2.0 ; SI: v_cmpx_nle_f32_e32 vcc, -2.0
; SI-NOT: s_and ; GFX10: v_cmpx_nle_f32_e32 -2.0
; GCN-NOT: s_and
define amdgpu_gs void @ult(float %a) { define amdgpu_gs void @ult(float %a) {
%c1 = fcmp ult float %a, -2.0 %c1 = fcmp ult float %a, -2.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}ule: ; GCN-LABEL: {{^}}ule:
; SI: v_cmpx_nlt_f32_e32 vcc, 2.0 ; SI: v_cmpx_nlt_f32_e32 vcc, 2.0
; SI-NOT: s_and ; GFX10: v_cmpx_nlt_f32_e32 2.0
; GCN-NOT: s_and
define amdgpu_gs void @ule(float %a) { define amdgpu_gs void @ule(float %a) {
%c1 = fcmp ule float %a, 2.0 %c1 = fcmp ule float %a, 2.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}une: ; GCN-LABEL: {{^}}une:
; SI: v_cmpx_neq_f32_e32 vcc, 0 ; SI: v_cmpx_neq_f32_e32 vcc, 0
; SI-NOT: s_and ; GFX10: v_cmpx_neq_f32_e32 0
; GCN-NOT: s_and
define amdgpu_gs void @une(float %a) { define amdgpu_gs void @une(float %a) {
%c1 = fcmp une float %a, 0.0 %c1 = fcmp une float %a, 0.0
call void @llvm.amdgcn.kill(i1 %c1) call void @llvm.amdgcn.kill(i1 %c1)
ret void ret void
} }
; SI-LABEL: {{^}}neg_olt: ; GCN-LABEL: {{^}}neg_olt:
; SI: v_cmpx_ngt_f32_e32 vcc, 1.0 ; SI: v_cmpx_ngt_f32_e32 vcc, 1.0
; SI-NOT: s_and ; GFX10: v_cmpx_ngt_f32_e32 1.0
; GCN-NOT: s_and
define amdgpu_gs void @neg_olt(float %a) { define amdgpu_gs void @neg_olt(float %a) {
%c1 = fcmp olt float %a, 1.0 %c1 = fcmp olt float %a, 1.0
%c2 = xor i1 %c1, 1 %c2 = xor i1 %c1, 1
call void @llvm.amdgcn.kill(i1 %c2) call void @llvm.amdgcn.kill(i1 %c2)
ret void ret void
} }
; SI-LABEL: {{^}}fcmp_x2: ; GCN-LABEL: {{^}}fcmp_x2:
; FIXME: LLVM should be able to combine these fcmp opcodes. ; FIXME: LLVM should be able to combine these fcmp opcodes.
; SI: v_cmp_lt_f32_e32 vcc, s{{[0-9]+}}, v0 ; SI: v_cmp_lt_f32_e32 vcc, s{{[0-9]+}}, v0
; SI: v_cndmask_b32 ; GFX10: v_cmp_lt_f32_e32 vcc, 0x3e800000, v0
; SI: v_cmpx_le_f32 ; GCN: v_cndmask_b32
; GCN: v_cmpx_le_f32
define amdgpu_ps void @fcmp_x2(float %a) #0 { define amdgpu_ps void @fcmp_x2(float %a) #0 {
%ogt = fcmp nsz ogt float %a, 2.500000e-01 %ogt = fcmp nsz ogt float %a, 2.500000e-01
%k = select i1 %ogt, float -1.000000e+00, float 0.000000e+00 %k = select i1 %ogt, float -1.000000e+00, float 0.000000e+00
%c = fcmp nsz oge float %k, 0.000000e+00 %c = fcmp nsz oge float %k, 0.000000e+00
call void @llvm.amdgcn.kill(i1 %c) #1 call void @llvm.amdgcn.kill(i1 %c) #1
ret void ret void
} }
; SI-LABEL: {{^}}wqm: ; GCN-LABEL: {{^}}wqm:
; SI: v_cmp_neq_f32_e32 vcc, 0 ; GCN: v_cmp_neq_f32_e32 vcc, 0
; SI: s_wqm_b64 s[0:1], vcc ; GCN: s_wqm_b64 s[0:1], vcc
; SI: s_and_b64 exec, exec, s[0:1] ; GCN: s_and_b64 exec, exec, s[0:1]
define amdgpu_ps void @wqm(float %a) { define amdgpu_ps void @wqm(float %a) {
%c1 = fcmp une float %a, 0.0 %c1 = fcmp une float %a, 0.0
%c2 = call i1 @llvm.amdgcn.wqm.vote(i1 %c1) %c2 = call i1 @llvm.amdgcn.wqm.vote(i1 %c1)
call void @llvm.amdgcn.kill(i1 %c2) call void @llvm.amdgcn.kill(i1 %c2)
ret void ret void
} }
; This checks that we use the 64-bit encoding when the operand is a SGPR. ; This checks that we use the 64-bit encoding when the operand is a SGPR.
; SI-LABEL: {{^}}test_sgpr: ; GCN-LABEL: {{^}}test_sgpr:
; SI: v_cmpx_ge_f32_e64 ; GCN: v_cmpx_ge_f32_e64
define amdgpu_ps void @test_sgpr(float inreg %a) #0 { define amdgpu_ps void @test_sgpr(float inreg %a) #0 {
%c = fcmp ole float %a, 1.000000e+00 %c = fcmp ole float %a, 1.000000e+00
call void @llvm.amdgcn.kill(i1 %c) #1 call void @llvm.amdgcn.kill(i1 %c) #1
ret void ret void
} }
; SI-LABEL: {{^}}test_non_inline_imm_sgpr: ; GCN-LABEL: {{^}}test_non_inline_imm_sgpr:
; SI-NOT: v_cmpx_ge_f32_e64 ; GCN-NOT: v_cmpx_ge_f32_e64
define amdgpu_ps void @test_non_inline_imm_sgpr(float inreg %a) #0 { define amdgpu_ps void @test_non_inline_imm_sgpr(float inreg %a) #0 {
%c = fcmp ole float %a, 1.500000e+00 %c = fcmp ole float %a, 1.500000e+00
call void @llvm.amdgcn.kill(i1 %c) #1 call void @llvm.amdgcn.kill(i1 %c) #1
ret void ret void
} }
; SI-LABEL: {{^}}test_scc_liveness: ; GCN-LABEL: {{^}}test_scc_liveness:
; SI: v_cmp ; GCN: v_cmp
; SI: s_and_b64 exec ; GCN: s_and_b64 exec
; SI: s_cmp ; GCN: s_cmp
; SI: s_cbranch_scc ; GCN: s_cbranch_scc
define amdgpu_ps void @test_scc_liveness() #0 { define amdgpu_ps void @test_scc_liveness() #0 {
main_body: main_body:
br label %loop3 br label %loop3
loop3: ; preds = %loop3, %main_body loop3: ; preds = %loop3, %main_body
%tmp = phi i32 [ 0, %main_body ], [ %tmp5, %loop3 ] %tmp = phi i32 [ 0, %main_body ], [ %tmp5, %loop3 ]
%tmp1 = icmp sgt i32 %tmp, 0 %tmp1 = icmp sgt i32 %tmp, 0
call void @llvm.amdgcn.kill(i1 %tmp1) #1 call void @llvm.amdgcn.kill(i1 %tmp1) #1
Show All 12 Lines

test/CodeGen/AMDGPU/loop_break.ll

Show All 9 Lines
; OPT: bb4: ; OPT: bb4:
; OPT: load volatile ; OPT: load volatile
; OPT: icmp slt i32 ; OPT: icmp slt i32
; OPT: xor i1 %cmp1 ; OPT: xor i1 %cmp1
; OPT: br label %Flow ; OPT: br label %Flow
; OPT: Flow: ; OPT: Flow:
; OPT: call i64 @llvm.amdgcn.if.break( ; OPT: call i64 @llvm.amdgcn.if.break.i64.i64(
; OPT: call i1 @llvm.amdgcn.loop(i64 ; OPT: call i1 @llvm.amdgcn.loop.i64(i64
; OPT: br i1 %{{[0-9]+}}, label %bb9, label %bb1 ; OPT: br i1 %{{[0-9]+}}, label %bb9, label %bb1
; OPT: bb9: ; OPT: bb9:
; OPT: call void @llvm.amdgcn.end.cf(i64 ; OPT: call void @llvm.amdgcn.end.cf.i64(i64
; GCN-LABEL: {{^}}break_loop: ; GCN-LABEL: {{^}}break_loop:
; GCN: s_mov_b64 [[OUTER_MASK:s\[[0-9]+:[0-9]+\]]], 0{{$}} ; GCN: s_mov_b64 [[OUTER_MASK:s\[[0-9]+:[0-9]+\]]], 0{{$}}
; GCN: [[LOOP_ENTRY:BB[0-9]+_[0-9]+]]: ; %bb1 ; GCN: [[LOOP_ENTRY:BB[0-9]+_[0-9]+]]: ; %bb1
; GCN: v_cmp_lt_i32_e32 vcc, -1 ; GCN: v_cmp_lt_i32_e32 vcc, -1
; GCN: s_and_b64 vcc, exec, vcc ; GCN: s_and_b64 vcc, exec, vcc
; GCN: s_or_b64 [[INNER_MASK:s\[[0-9]+:[0-9]+\]]], [[INNER_MASK]], exec ; GCN: s_or_b64 [[INNER_MASK:s\[[0-9]+:[0-9]+\]]], [[INNER_MASK]], exec
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 Lines
; OPT: bb4: ; OPT: bb4:
; OPT-NEXT: %load = load volatile i32, i32 addrspace(1)* undef, align 4 ; OPT-NEXT: %load = load volatile i32, i32 addrspace(1)* undef, align 4
; OPT-NEXT: %cmp1 = icmp sge i32 %tmp, %load ; OPT-NEXT: %cmp1 = icmp sge i32 %tmp, %load
; OPT-NEXT: br label %Flow ; OPT-NEXT: br label %Flow
; OPT: Flow: ; OPT: Flow:
; OPT-NEXT: %tmp2 = phi i32 [ %lsr.iv.next, %bb4 ], [ undef, %bb1 ] ; OPT-NEXT: %tmp2 = phi i32 [ %lsr.iv.next, %bb4 ], [ undef, %bb1 ]
; OPT-NEXT: %tmp3 = phi i1 [ %cmp1, %bb4 ], [ undef, %bb1 ] ; OPT-NEXT: %tmp3 = phi i1 [ %cmp1, %bb4 ], [ undef, %bb1 ]
; OPT-NEXT: %0 = call i64 @llvm.amdgcn.if.break(i1 %tmp3, i64 %phi.broken) ; OPT-NEXT: %0 = call i64 @llvm.amdgcn.if.break.i64.i64(i1 %tmp3, i64 %phi.broken)
; OPT-NEXT: %1 = call i1 @llvm.amdgcn.loop(i64 %0) ; OPT-NEXT: %1 = call i1 @llvm.amdgcn.loop.i64(i64 %0)
; OPT-NEXT: br i1 %1, label %bb9, label %bb1 ; OPT-NEXT: br i1 %1, label %bb9, label %bb1
; OPT: bb9: ; preds = %Flow ; OPT: bb9: ; preds = %Flow
; OPT-NEXT: call void @llvm.amdgcn.end.cf(i64 %0) ; OPT-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %0)
; OPT-NEXT: store volatile i32 7 ; OPT-NEXT: store volatile i32 7
; OPT-NEXT: ret void ; OPT-NEXT: ret void
define amdgpu_kernel void @undef_phi_cond_break_loop(i32 %arg) #0 { define amdgpu_kernel void @undef_phi_cond_break_loop(i32 %arg) #0 {
bb: bb:
%id = call i32 @llvm.amdgcn.workitem.id.x() %id = call i32 @llvm.amdgcn.workitem.id.x()
%tmp = sub i32 %id, %arg %tmp = sub i32 %id, %arg
br label %bb1 br label %bb1
Show All 32 Lines
; OPT: bb4: ; OPT: bb4:
; OPT-NEXT: %load = load volatile i32, i32 addrspace(1)* undef, align 4 ; OPT-NEXT: %load = load volatile i32, i32 addrspace(1)* undef, align 4
; OPT-NEXT: %cmp1 = icmp sge i32 %tmp, %load ; OPT-NEXT: %cmp1 = icmp sge i32 %tmp, %load
; OPT-NEXT: br label %Flow ; OPT-NEXT: br label %Flow
; OPT: Flow: ; OPT: Flow:
; OPT-NEXT: %tmp2 = phi i32 [ %lsr.iv.next, %bb4 ], [ undef, %bb1 ] ; OPT-NEXT: %tmp2 = phi i32 [ %lsr.iv.next, %bb4 ], [ undef, %bb1 ]
; OPT-NEXT: %tmp3 = phi i1 [ %cmp1, %bb4 ], [ icmp ne (i32 addrspace(3)* inttoptr (i32 4 to i32 addrspace(3)*), i32 addrspace(3)* @lds), %bb1 ] ; OPT-NEXT: %tmp3 = phi i1 [ %cmp1, %bb4 ], [ icmp ne (i32 addrspace(3)* inttoptr (i32 4 to i32 addrspace(3)*), i32 addrspace(3)* @lds), %bb1 ]
; OPT-NEXT: %0 = call i64 @llvm.amdgcn.if.break(i1 %tmp3, i64 %phi.broken) ; OPT-NEXT: %0 = call i64 @llvm.amdgcn.if.break.i64.i64(i1 %tmp3, i64 %phi.broken)
; OPT-NEXT: %1 = call i1 @llvm.amdgcn.loop(i64 %0) ; OPT-NEXT: %1 = call i1 @llvm.amdgcn.loop.i64(i64 %0)
; OPT-NEXT: br i1 %1, label %bb9, label %bb1 ; OPT-NEXT: br i1 %1, label %bb9, label %bb1
; OPT: bb9: ; preds = %Flow ; OPT: bb9: ; preds = %Flow
; OPT-NEXT: call void @llvm.amdgcn.end.cf(i64 %0) ; OPT-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %0)
; OPT-NEXT: store volatile i32 7 ; OPT-NEXT: store volatile i32 7
; OPT-NEXT: ret void ; OPT-NEXT: ret void
define amdgpu_kernel void @constexpr_phi_cond_break_loop(i32 %arg) #0 { define amdgpu_kernel void @constexpr_phi_cond_break_loop(i32 %arg) #0 {
bb: bb:
%id = call i32 @llvm.amdgcn.workitem.id.x() %id = call i32 @llvm.amdgcn.workitem.id.x()
%tmp = sub i32 %id, %arg %tmp = sub i32 %id, %arg
br label %bb1 br label %bb1
Show All 29 Lines
; OPT: bb4: ; OPT: bb4:
; OPT-NEXT: %load = load volatile i32, i32 addrspace(1)* undef, align 4 ; OPT-NEXT: %load = load volatile i32, i32 addrspace(1)* undef, align 4
; OPT-NEXT: %cmp1 = icmp sge i32 %tmp, %load ; OPT-NEXT: %cmp1 = icmp sge i32 %tmp, %load
; OPT-NEXT: br label %Flow ; OPT-NEXT: br label %Flow
; OPT: Flow: ; OPT: Flow:
; OPT-NEXT: %tmp2 = phi i32 [ %lsr.iv.next, %bb4 ], [ undef, %bb1 ] ; OPT-NEXT: %tmp2 = phi i32 [ %lsr.iv.next, %bb4 ], [ undef, %bb1 ]
; OPT-NEXT: %tmp3 = phi i1 [ %cmp1, %bb4 ], [ true, %bb1 ] ; OPT-NEXT: %tmp3 = phi i1 [ %cmp1, %bb4 ], [ true, %bb1 ]
; OPT-NEXT: %0 = call i64 @llvm.amdgcn.if.break(i1 %tmp3, i64 %phi.broken) ; OPT-NEXT: %0 = call i64 @llvm.amdgcn.if.break.i64.i64(i1 %tmp3, i64 %phi.broken)
; OPT-NEXT: %1 = call i1 @llvm.amdgcn.loop(i64 %0) ; OPT-NEXT: %1 = call i1 @llvm.amdgcn.loop.i64(i64 %0)
; OPT-NEXT: br i1 %1, label %bb9, label %bb1 ; OPT-NEXT: br i1 %1, label %bb9, label %bb1
; OPT: bb9: ; preds = %Flow ; OPT: bb9: ; preds = %Flow
; OPT-NEXT: call void @llvm.amdgcn.end.cf(i64 %0) ; OPT-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %0)
; OPT-NEXT: store volatile i32 7 ; OPT-NEXT: store volatile i32 7
; OPT-NEXT: ret void ; OPT-NEXT: ret void
define amdgpu_kernel void @true_phi_cond_break_loop(i32 %arg) #0 { define amdgpu_kernel void @true_phi_cond_break_loop(i32 %arg) #0 {
bb: bb:
%id = call i32 @llvm.amdgcn.workitem.id.x() %id = call i32 @llvm.amdgcn.workitem.id.x()
%tmp = sub i32 %id, %arg %tmp = sub i32 %id, %arg
br label %bb1 br label %bb1
Show All 28 Lines
; OPT: bb4: ; OPT: bb4:
; OPT-NEXT: %load = load volatile i32, i32 addrspace(1)* undef, align 4 ; OPT-NEXT: %load = load volatile i32, i32 addrspace(1)* undef, align 4
; OPT-NEXT: %cmp1 = icmp sge i32 %tmp, %load ; OPT-NEXT: %cmp1 = icmp sge i32 %tmp, %load
; OPT-NEXT: br label %Flow ; OPT-NEXT: br label %Flow
; OPT: Flow: ; OPT: Flow:
; OPT-NEXT: %tmp2 = phi i32 [ %lsr.iv.next, %bb4 ], [ undef, %bb1 ] ; OPT-NEXT: %tmp2 = phi i32 [ %lsr.iv.next, %bb4 ], [ undef, %bb1 ]
; OPT-NEXT: %tmp3 = phi i1 [ %cmp1, %bb4 ], [ false, %bb1 ] ; OPT-NEXT: %tmp3 = phi i1 [ %cmp1, %bb4 ], [ false, %bb1 ]
; OPT-NEXT: %0 = call i64 @llvm.amdgcn.if.break(i1 %tmp3, i64 %phi.broken) ; OPT-NEXT: %0 = call i64 @llvm.amdgcn.if.break.i64.i64(i1 %tmp3, i64 %phi.broken)
; OPT-NEXT: %1 = call i1 @llvm.amdgcn.loop(i64 %0) ; OPT-NEXT: %1 = call i1 @llvm.amdgcn.loop.i64(i64 %0)
; OPT-NEXT: br i1 %1, label %bb9, label %bb1 ; OPT-NEXT: br i1 %1, label %bb9, label %bb1
; OPT: bb9: ; preds = %Flow ; OPT: bb9: ; preds = %Flow
; OPT-NEXT: call void @llvm.amdgcn.end.cf(i64 %0) ; OPT-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %0)
; OPT-NEXT: store volatile i32 7 ; OPT-NEXT: store volatile i32 7
; OPT-NEXT: ret void ; OPT-NEXT: ret void
define amdgpu_kernel void @false_phi_cond_break_loop(i32 %arg) #0 { define amdgpu_kernel void @false_phi_cond_break_loop(i32 %arg) #0 {
bb: bb:
%id = call i32 @llvm.amdgcn.workitem.id.x() %id = call i32 @llvm.amdgcn.workitem.id.x()
%tmp = sub i32 %id, %arg %tmp = sub i32 %id, %arg
br label %bb1 br label %bb1
Show All 33 Lines
; OPT-NEXT: %load = load volatile i32, i32 addrspace(1)* undef, align 4 ; OPT-NEXT: %load = load volatile i32, i32 addrspace(1)* undef, align 4
; OPT-NEXT: %cmp1 = icmp sge i32 %tmp, %load ; OPT-NEXT: %cmp1 = icmp sge i32 %tmp, %load
; OPT-NEXT: br label %Flow ; OPT-NEXT: br label %Flow
; OPT: Flow: ; OPT: Flow:
; OPT-NEXT: %tmp2 = phi i32 [ %lsr.iv.next, %bb4 ], [ undef, %bb1 ] ; OPT-NEXT: %tmp2 = phi i32 [ %lsr.iv.next, %bb4 ], [ undef, %bb1 ]
; OPT-NEXT: %tmp3 = phi i1 [ %cmp1, %bb4 ], [ true, %bb1 ] ; OPT-NEXT: %tmp3 = phi i1 [ %cmp1, %bb4 ], [ true, %bb1 ]
; OPT-NEXT: %0 = xor i1 %tmp3, true ; OPT-NEXT: %0 = xor i1 %tmp3, true
; OPT-NEXT: %1 = call i64 @llvm.amdgcn.if.break(i1 %0, i64 %phi.broken) ; OPT-NEXT: %1 = call i64 @llvm.amdgcn.if.break.i64.i64(i1 %0, i64 %phi.broken)
; OPT-NEXT: %2 = call i1 @llvm.amdgcn.loop(i64 %1) ; OPT-NEXT: %2 = call i1 @llvm.amdgcn.loop.i64(i64 %1)
; OPT-NEXT: br i1 %2, label %bb9, label %bb1 ; OPT-NEXT: br i1 %2, label %bb9, label %bb1
; OPT: bb9: ; OPT: bb9:
; OPT-NEXT: call void @llvm.amdgcn.end.cf(i64 %1) ; OPT-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %1)
; OPT-NEXT: store volatile i32 7, i32 addrspace(3)* undef ; OPT-NEXT: store volatile i32 7, i32 addrspace(3)* undef
; OPT-NEXT: ret void ; OPT-NEXT: ret void
define amdgpu_kernel void @invert_true_phi_cond_break_loop(i32 %arg) #0 { define amdgpu_kernel void @invert_true_phi_cond_break_loop(i32 %arg) #0 {
bb: bb:
%id = call i32 @llvm.amdgcn.workitem.id.x() %id = call i32 @llvm.amdgcn.workitem.id.x()
%tmp = sub i32 %id, %arg %tmp = sub i32 %id, %arg
br label %bb1 br label %bb1
Show All 25 Lines

test/CodeGen/AMDGPU/mubuf-legalize-operands.mir

# RUN: llc -march=amdgcn -mcpu=gfx700 -verify-machineinstrs -verify-machine-dom-info --run-pass=si-fix-sgpr-copies -o - %s | FileCheck %s --check-prefixes=COMMON,ADDR64 # RUN: llc -march=amdgcn -mcpu=gfx700 -verify-machineinstrs -verify-machine-dom-info --run-pass=si-fix-sgpr-copies -o - %s | FileCheck %s --check-prefixes=W64,ADDR64
# RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs -verify-machine-dom-info --run-pass=si-fix-sgpr-copies -o - %s | FileCheck %s --check-prefixes=COMMON,NO-ADDR64 # RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs -verify-machine-dom-info --run-pass=si-fix-sgpr-copies -o - %s | FileCheck %s --check-prefixes=W64,W64-NO-ADDR64
# RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs -verify-machine-dom-info --run-pass=si-fix-sgpr-copies -o - %s | FileCheck %s --check-prefixes=W64,W64-NO-ADDR64
# RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -verify-machineinstrs -verify-machine-dom-info --run-pass=si-fix-sgpr-copies -o - %s | FileCheck %s --check-prefixes=W32
# Test that we correctly legalize VGPR Rsrc operands in MUBUF instructions. # Test that we correctly legalize VGPR Rsrc operands in MUBUF instructions.
# #
# On ADDR64 hardware we optimize the _ADDR64 and _OFFSET cases to avoid # On ADDR64 hardware we optimize the _ADDR64 and _OFFSET cases to avoid
# needing a waterfall. For all other instruction variants, and when we are # needing a waterfall. For all other instruction variants, and when we are
# on non-ADDR64 hardware, we emit a waterfall loop. # on non-ADDR64 hardware, we emit a waterfall loop.
# COMMON-LABEL: name: idxen # W64-LABEL: name: idxen
# COMMON-LABEL: bb.0: # W64-LABEL: bb.0:
# COMMON-NEXT: successors: %bb.1({{.*}}) # W64-NEXT: successors: %bb.1({{.*}})
# COMMON: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3 # W64: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
# COMMON: [[SAVEEXEC:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec # W64: [[SAVEEXEC:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec
# COMMON-LABEL: bb.1: # W64-LABEL: bb.1:
# COMMON-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}}) # W64-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}})
# COMMON: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec # W64: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec
# COMMON: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec # W64: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec
# COMMON: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec # W64: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec
# COMMON: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec # W64: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec
# COMMON: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3 # W64: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3
# COMMON: [[CMP0:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec # W64: [[CMP0:%[0-9]+]]:sreg_64_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec
# COMMON: [[CMP1:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec # W64: [[CMP1:%[0-9]+]]:sreg_64_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec
# COMMON: [[CMP:%[0-9]+]]:sreg_64 = S_AND_B64 [[CMP0]], [[CMP1]], implicit-def $scc # W64: [[CMP:%[0-9]+]]:sreg_64_xexec = S_AND_B64 [[CMP0]], [[CMP1]], implicit-def $scc
# COMMON: [[TMPEXEC:%[0-9]+]]:sreg_64 = S_AND_SAVEEXEC_B64 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec # W64: [[TMPEXEC:%[0-9]+]]:sreg_64_xexec = S_AND_SAVEEXEC_B64 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec
# COMMON: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN %4, killed [[SRSRC]], 0, 0, 0, 0, 0, 0, implicit $exec # W64: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN %4, killed [[SRSRC]], 0, 0, 0, 0, 0, 0, implicit $exec
# COMMON: $exec = S_XOR_B64_term $exec, [[TMPEXEC]], implicit-def $scc # W64: $exec = S_XOR_B64_term $exec, [[TMPEXEC]], implicit-def $scc
# COMMON: S_CBRANCH_EXECNZ %bb.1, implicit $exec # W64: S_CBRANCH_EXECNZ %bb.1, implicit $exec
# COMMON-LABEL bb.2: # W64-LABEL bb.2:
# COMMON: $exec = S_MOV_B64 [[SAVEEXEC]] # W64: $exec = S_MOV_B64 [[SAVEEXEC]]
# W32-LABEL: name: idxen
# W32-LABEL: bb.0:
# W32-NEXT: successors: %bb.1({{.*}})
# W32: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
# W32: [[SAVEEXEC:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
# W32-LABEL: bb.1:
# W32-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}})
# W32: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec
# W32: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec
# W32: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec
# W32: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec
# W32: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3
# W32: [[CMP0:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec
# W32: [[CMP1:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec
# W32: [[CMP:%[0-9]+]]:sreg_32_xm0_xexec = S_AND_B32 [[CMP0]], [[CMP1]], implicit-def $scc
# W32: [[TMPEXEC:%[0-9]+]]:sreg_32_xm0_xexec = S_AND_SAVEEXEC_B32 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec
# W32: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN %4, killed [[SRSRC]], 0, 0, 0, 0, 0, 0, implicit $exec
# TODO: S_XOR_B32_term should be `implicit-def $scc`
# W32: $exec_lo = S_XOR_B32_term $exec_lo, [[TMPEXEC]]
# W32: S_CBRANCH_EXECNZ %bb.1, implicit $exec
# W32-LABEL bb.2:
# W32: $exec_lo = S_MOV_B32 [[SAVEEXEC]]
--- ---
name: idxen name: idxen
liveins: liveins:
- { reg: '$vgpr0', virtual-reg: '%0' } - { reg: '$vgpr0', virtual-reg: '%0' }
- { reg: '$vgpr1', virtual-reg: '%1' } - { reg: '$vgpr1', virtual-reg: '%1' }
- { reg: '$vgpr2', virtual-reg: '%2' } - { reg: '$vgpr2', virtual-reg: '%2' }
- { reg: '$vgpr3', virtual-reg: '%3' } - { reg: '$vgpr3', virtual-reg: '%3' }
- { reg: '$vgpr4', virtual-reg: '%4' } - { reg: '$vgpr4', virtual-reg: '%4' }
Show All 9 Lines bb.0:
%0:vgpr_32 = COPY $vgpr0 %0:vgpr_32 = COPY $vgpr0
%6:sreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3 %6:sreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
%7:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN %4, killed %6, 0, 0, 0, 0, 0, 0, implicit $exec %7:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN %4, killed %6, 0, 0, 0, 0, 0, 0, implicit $exec
$sgpr30_sgpr31 = COPY %5 $sgpr30_sgpr31 = COPY %5
$vgpr0 = COPY %7 $vgpr0 = COPY %7
S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0 S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0
... ...
# COMMON-LABEL: name: offen # W64-LABEL: name: offen
# COMMON-LABEL: bb.0: # W64-LABEL: bb.0:
# COMMON-NEXT: successors: %bb.1({{.*}}) # W64-NEXT: successors: %bb.1({{.*}})
# COMMON: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3 # W64: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
# COMMON: [[SAVEEXEC:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec # W64: [[SAVEEXEC:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec
# COMMON-LABEL: bb.1: # W64-LABEL: bb.1:
# COMMON-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}}) # W64-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}})
# COMMON: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec # W64: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec
# COMMON: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec # W64: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec
# COMMON: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec # W64: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec
# COMMON: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec # W64: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec
# COMMON: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3 # W64: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3
# COMMON: [[CMP0:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec # W64: [[CMP0:%[0-9]+]]:sreg_64_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec
# COMMON: [[CMP1:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec # W64: [[CMP1:%[0-9]+]]:sreg_64_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec
# COMMON: [[CMP:%[0-9]+]]:sreg_64 = S_AND_B64 [[CMP0]], [[CMP1]], implicit-def $scc # W64: [[CMP:%[0-9]+]]:sreg_64_xexec = S_AND_B64 [[CMP0]], [[CMP1]], implicit-def $scc
# COMMON: [[TMPEXEC:%[0-9]+]]:sreg_64 = S_AND_SAVEEXEC_B64 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec # W64: [[TMPEXEC:%[0-9]+]]:sreg_64_xexec = S_AND_SAVEEXEC_B64 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec
# COMMON: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_OFFEN %4, killed [[SRSRC]], 0, 0, 0, 0, 0, 0, implicit $exec # W64: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_OFFEN %4, killed [[SRSRC]], 0, 0, 0, 0, 0, 0, implicit $exec
# COMMON: $exec = S_XOR_B64_term $exec, [[TMPEXEC]], implicit-def $scc # W64: $exec = S_XOR_B64_term $exec, [[TMPEXEC]], implicit-def $scc
# COMMON: S_CBRANCH_EXECNZ %bb.1, implicit $exec # W64: S_CBRANCH_EXECNZ %bb.1, implicit $exec
# COMMON-LABEL bb.2: # W64-LABEL bb.2:
# COMMON: $exec = S_MOV_B64 [[SAVEEXEC]] # W64: $exec = S_MOV_B64 [[SAVEEXEC]]
# W32-LABEL: name: offen
# W32-LABEL: bb.0:
# W32-NEXT: successors: %bb.1({{.*}})
# W32: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
# W32: [[SAVEEXEC:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
# W32-LABEL: bb.1:
# W32-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}})
# W32: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec
# W32: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec
# W32: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec
# W32: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec
# W32: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3
# W32: [[CMP0:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec
# W32: [[CMP1:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec
# W32: [[CMP:%[0-9]+]]:sreg_32_xm0_xexec = S_AND_B32 [[CMP0]], [[CMP1]], implicit-def $scc
# W32: [[TMPEXEC:%[0-9]+]]:sreg_32_xm0_xexec = S_AND_SAVEEXEC_B32 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec
# W32: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_OFFEN %4, killed [[SRSRC]], 0, 0, 0, 0, 0, 0, implicit $exec
# TODO: S_XOR_B32_term should be `implicit-def $scc`
# W32: $exec_lo = S_XOR_B32_term $exec_lo, [[TMPEXEC]]
# W32: S_CBRANCH_EXECNZ %bb.1, implicit $exec
# W32-LABEL bb.2:
# W32: $exec_lo = S_MOV_B32 [[SAVEEXEC]]
--- ---
name: offen name: offen
liveins: liveins:
- { reg: '$vgpr0', virtual-reg: '%0' } - { reg: '$vgpr0', virtual-reg: '%0' }
- { reg: '$vgpr1', virtual-reg: '%1' } - { reg: '$vgpr1', virtual-reg: '%1' }
- { reg: '$vgpr2', virtual-reg: '%2' } - { reg: '$vgpr2', virtual-reg: '%2' }
- { reg: '$vgpr3', virtual-reg: '%3' } - { reg: '$vgpr3', virtual-reg: '%3' }
- { reg: '$vgpr4', virtual-reg: '%4' } - { reg: '$vgpr4', virtual-reg: '%4' }
Show All 9 Lines bb.0:
%0:vgpr_32 = COPY $vgpr0 %0:vgpr_32 = COPY $vgpr0
%6:sreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3 %6:sreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
%7:vgpr_32 = BUFFER_LOAD_FORMAT_X_OFFEN %4, killed %6, 0, 0, 0, 0, 0, 0, implicit $exec %7:vgpr_32 = BUFFER_LOAD_FORMAT_X_OFFEN %4, killed %6, 0, 0, 0, 0, 0, 0, implicit $exec
$sgpr30_sgpr31 = COPY %5 $sgpr30_sgpr31 = COPY %5
$vgpr0 = COPY %7 $vgpr0 = COPY %7
S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0 S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0
... ...
# COMMON-LABEL: name: bothen # W64-LABEL: name: bothen
# COMMON-LABEL: bb.0: # W64-LABEL: bb.0:
# COMMON-NEXT: successors: %bb.1({{.*}}) # W64-NEXT: successors: %bb.1({{.*}})
# COMMON: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3 # W64: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
# COMMON: [[SAVEEXEC:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec # W64: [[SAVEEXEC:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec
# COMMON-LABEL: bb.1: # W64-LABEL: bb.1:
# COMMON-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}}) # W64-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}})
# COMMON: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec # W64: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec
# COMMON: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec # W64: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec
# COMMON: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec # W64: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec
# COMMON: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec # W64: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec
# COMMON: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3 # W64: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3
# COMMON: [[CMP0:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec # W64: [[CMP0:%[0-9]+]]:sreg_64_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec
# COMMON: [[CMP1:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec # W64: [[CMP1:%[0-9]+]]:sreg_64_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec
# COMMON: [[CMP:%[0-9]+]]:sreg_64 = S_AND_B64 [[CMP0]], [[CMP1]], implicit-def $scc # W64: [[CMP:%[0-9]+]]:sreg_64_xexec = S_AND_B64 [[CMP0]], [[CMP1]], implicit-def $scc
# COMMON: [[TMPEXEC:%[0-9]+]]:sreg_64 = S_AND_SAVEEXEC_B64 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec # W64: [[TMPEXEC:%[0-9]+]]:sreg_64_xexec = S_AND_SAVEEXEC_B64 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec
# COMMON: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_BOTHEN %4, killed [[SRSRC]], 0, 0, 0, 0, 0, 0, implicit $exec # W64: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_BOTHEN %4, killed [[SRSRC]], 0, 0, 0, 0, 0, 0, implicit $exec
# COMMON: $exec = S_XOR_B64_term $exec, [[TMPEXEC]], implicit-def $scc # W64: $exec = S_XOR_B64_term $exec, [[TMPEXEC]], implicit-def $scc
# COMMON: S_CBRANCH_EXECNZ %bb.1, implicit $exec # W64: S_CBRANCH_EXECNZ %bb.1, implicit $exec
# COMMON-LABEL bb.2: # W64-LABEL bb.2:
# COMMON: $exec = S_MOV_B64 [[SAVEEXEC]] # W64: $exec = S_MOV_B64 [[SAVEEXEC]]
# W32-LABEL: name: bothen
# W32-LABEL: bb.0:
# W32-NEXT: successors: %bb.1({{.*}})
# W32: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
# W32: [[SAVEEXEC:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
# W32-LABEL: bb.1:
# W32-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}})
# W32: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec
# W32: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec
# W32: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec
# W32: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec
# W32: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3
# W32: [[CMP0:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec
# W32: [[CMP1:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec
# W32: [[CMP:%[0-9]+]]:sreg_32_xm0_xexec = S_AND_B32 [[CMP0]], [[CMP1]], implicit-def $scc
# W32: [[TMPEXEC:%[0-9]+]]:sreg_32_xm0_xexec = S_AND_SAVEEXEC_B32 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec
# W32: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_BOTHEN %4, killed [[SRSRC]], 0, 0, 0, 0, 0, 0, implicit $exec
# TODO: S_XOR_B32_term should be `implicit-def $scc`
# W32: $exec_lo = S_XOR_B32_term $exec_lo, [[TMPEXEC]]
# W32: S_CBRANCH_EXECNZ %bb.1, implicit $exec
# W32-LABEL bb.2:
# W32: $exec_lo = S_MOV_B32 [[SAVEEXEC]]
--- ---
name: bothen name: bothen
liveins: liveins:
- { reg: '$vgpr0', virtual-reg: '%0' } - { reg: '$vgpr0', virtual-reg: '%0' }
- { reg: '$vgpr1', virtual-reg: '%1' } - { reg: '$vgpr1', virtual-reg: '%1' }
- { reg: '$vgpr2', virtual-reg: '%2' } - { reg: '$vgpr2', virtual-reg: '%2' }
- { reg: '$vgpr3', virtual-reg: '%3' } - { reg: '$vgpr3', virtual-reg: '%3' }
- { reg: '$vgpr4_vgpr5', virtual-reg: '%4' } - { reg: '$vgpr4_vgpr5', virtual-reg: '%4' }
Show All 9 Lines bb.0:
%0:vgpr_32 = COPY $vgpr0 %0:vgpr_32 = COPY $vgpr0
%6:sreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3 %6:sreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
%7:vgpr_32 = BUFFER_LOAD_FORMAT_X_BOTHEN %4, killed %6, 0, 0, 0, 0, 0, 0, implicit $exec %7:vgpr_32 = BUFFER_LOAD_FORMAT_X_BOTHEN %4, killed %6, 0, 0, 0, 0, 0, 0, implicit $exec
$sgpr30_sgpr31 = COPY %5 $sgpr30_sgpr31 = COPY %5
$vgpr0 = COPY %7 $vgpr0 = COPY %7
S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0 S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0
... ...
# COMMON-LABEL: name: addr64 # ADDR64-LABEL: name: addr64
# COMMON-LABEL: bb.0: # ADDR64-LABEL: bb.0:
# COMMON: %12:vreg_64 = COPY %8.sub0_sub1 # ADDR64: %12:vreg_64 = COPY %8.sub0_sub1
# COMMON: %13:sreg_64 = S_MOV_B64 0 # ADDR64: %13:sreg_64 = S_MOV_B64 0
# COMMON: %14:sgpr_32 = S_MOV_B32 0 # ADDR64: %14:sgpr_32 = S_MOV_B32 0
# COMMON: %15:sgpr_32 = S_MOV_B32 61440 # ADDR64: %15:sgpr_32 = S_MOV_B32 61440
# COMMON: %16:sreg_128 = REG_SEQUENCE %13, %subreg.sub0_sub1, %14, %subreg.sub2, %15, %subreg.sub3 # ADDR64: %16:sreg_128 = REG_SEQUENCE %13, %subreg.sub0_sub1, %14, %subreg.sub2, %15, %subreg.sub3
# COMMON: %9:vgpr_32 = V_ADD_I32_e32 %12.sub0, %4.sub0, implicit-def $vcc, implicit $exec # ADDR64: %9:vgpr_32 = V_ADD_I32_e32 %12.sub0, %4.sub0, implicit-def $vcc, implicit $exec
# COMMON: %10:vgpr_32 = V_ADDC_U32_e32 %12.sub1, %4.sub1, implicit-def $vcc, implicit $vcc, implicit $exec # ADDR64: %10:vgpr_32 = V_ADDC_U32_e32 %12.sub1, %4.sub1, implicit-def $vcc, implicit $vcc, implicit $exec
# COMMON: %11:vreg_64 = REG_SEQUENCE %9, %subreg.sub0, %10, %subreg.sub1 # ADDR64: %11:vreg_64 = REG_SEQUENCE %9, %subreg.sub0, %10, %subreg.sub1
# COMMON: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_ADDR64 %11, killed %16, 0, 0, 0, 0, 0, 0, implicit $exec # ADDR64: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_ADDR64 %11, killed %16, 0, 0, 0, 0, 0, 0, implicit $exec
--- ---
name: addr64 name: addr64
liveins: liveins:
- { reg: '$vgpr0', virtual-reg: '%0' } - { reg: '$vgpr0', virtual-reg: '%0' }
- { reg: '$vgpr1', virtual-reg: '%1' } - { reg: '$vgpr1', virtual-reg: '%1' }
- { reg: '$vgpr2', virtual-reg: '%2' } - { reg: '$vgpr2', virtual-reg: '%2' }
- { reg: '$vgpr3', virtual-reg: '%3' } - { reg: '$vgpr3', virtual-reg: '%3' }
- { reg: '$vgpr4_vgpr5', virtual-reg: '%4' } - { reg: '$vgpr4_vgpr5', virtual-reg: '%4' }
Show All 9 Lines bb.0:
%0:vgpr_32 = COPY $vgpr0 %0:vgpr_32 = COPY $vgpr0
%6:sreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3 %6:sreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
%7:vgpr_32 = BUFFER_LOAD_FORMAT_X_ADDR64 %4, killed %6, 0, 0, 0, 0, 0, 0, implicit $exec %7:vgpr_32 = BUFFER_LOAD_FORMAT_X_ADDR64 %4, killed %6, 0, 0, 0, 0, 0, 0, implicit $exec
$sgpr30_sgpr31 = COPY %5 $sgpr30_sgpr31 = COPY %5
$vgpr0 = COPY %7 $vgpr0 = COPY %7
S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0 S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0
... ...
# COMMON-LABEL: name: offset # W64-LABEL: name: offset
# COMMON-LABEL: bb.0: # W64-LABEL: bb.0:
# W64-NO-ADDR64: successors: %bb.1({{.*}})
# W64-NO-ADDR64: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
# W64-NO-ADDR64: [[SAVEEXEC:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec
# W64-NO-ADDR64-LABEL: bb.1:
# W64-NO-ADDR64-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}})
# W64-NO-ADDR64: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec
# W64-NO-ADDR64: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec
# W64-NO-ADDR64: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec
# W64-NO-ADDR64: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec
# W64-NO-ADDR64: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3
# W64-NO-ADDR64: [[CMP0:%[0-9]+]]:sreg_64_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec
# W64-NO-ADDR64: [[CMP1:%[0-9]+]]:sreg_64_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec
# W64-NO-ADDR64: [[CMP:%[0-9]+]]:sreg_64_xexec = S_AND_B64 [[CMP0]], [[CMP1]], implicit-def $scc
# W64-NO-ADDR64: [[TMPEXEC:%[0-9]+]]:sreg_64_xexec = S_AND_SAVEEXEC_B64 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec
# W64-NO-ADDR64: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_OFFSET killed [[SRSRC]], 0, 0, 0, 0, 0, 0, implicit $exec
# W64-NO-ADDR64: $exec = S_XOR_B64_term $exec, [[TMPEXEC]], implicit-def $scc
# W64-NO-ADDR64: S_CBRANCH_EXECNZ %bb.1, implicit $exec
# W64-NO-ADDR64-LABEL bb.2:
# W64-NO-ADDR64: $exec = S_MOV_B64 [[SAVEEXEC]]
# NO-ADDR64-NEXT: successors: %bb.1({{.*}}) # W32: successors: %bb.1({{.*}})
# NO-ADDR64: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3 # W32: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
# NO-ADDR64: [[SAVEEXEC:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec # W32: [[SAVEEXEC:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
# NO-ADDR64-LABEL: bb.1: # W32-LABEL: bb.1:
# NO-ADDR64-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}}) # W32-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}})
# NO-ADDR64: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec # W32: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec
# NO-ADDR64: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec # W32: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec
# NO-ADDR64: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec # W32: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec
# NO-ADDR64: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec # W32: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec
# NO-ADDR64: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3 # W32: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3
# NO-ADDR64: [[CMP0:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec # W32: [[CMP0:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec
# NO-ADDR64: [[CMP1:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec # W32: [[CMP1:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec
# NO-ADDR64: [[CMP:%[0-9]+]]:sreg_64 = S_AND_B64 [[CMP0]], [[CMP1]], implicit-def $scc # W32: [[CMP:%[0-9]+]]:sreg_32_xm0_xexec = S_AND_B32 [[CMP0]], [[CMP1]], implicit-def $scc
# NO-ADDR64: [[TMPEXEC:%[0-9]+]]:sreg_64 = S_AND_SAVEEXEC_B64 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec # W32: [[TMPEXEC:%[0-9]+]]:sreg_32_xm0_xexec = S_AND_SAVEEXEC_B32 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec
# NO-ADDR64: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_OFFSET killed [[SRSRC]], 0, 0, 0, 0, 0, 0, implicit $exec # W32: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_OFFSET killed [[SRSRC]], 0, 0, 0, 0, 0, 0, implicit $exec
# NO-ADDR64: $exec = S_XOR_B64_term $exec, [[TMPEXEC]], implicit-def $scc # TODO: S_XOR_B32_term should be `implicit-def $scc`
# NO-ADDR64: S_CBRANCH_EXECNZ %bb.1, implicit $exec # W32: $exec_lo = S_XOR_B32_term $exec_lo, [[TMPEXEC]]
# NO-ADDR64-LABEL bb.2: # W32: S_CBRANCH_EXECNZ %bb.1, implicit $exec
# NO-ADDR64: $exec = S_MOV_B64 [[SAVEEXEC]] # W32-LABEL bb.2:
# W32: $exec_lo = S_MOV_B32 [[SAVEEXEC]]
# ADDR64: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3 # ADDR64: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
# ADDR64: [[RSRCPTR:%[0-9]+]]:vreg_64 = COPY [[VRSRC]].sub0_sub1 # ADDR64: [[RSRCPTR:%[0-9]+]]:vreg_64 = COPY [[VRSRC]].sub0_sub1
# ADDR64: [[ZERO64:%[0-9]+]]:sreg_64 = S_MOV_B64 0 # ADDR64: [[ZERO64:%[0-9]+]]:sreg_64 = S_MOV_B64 0
# ADDR64: [[RSRCFMTLO:%[0-9]+]]:sgpr_32 = S_MOV_B32 0 # ADDR64: [[RSRCFMTLO:%[0-9]+]]:sgpr_32 = S_MOV_B32 0
# ADDR64: [[RSRCFMTHI:%[0-9]+]]:sgpr_32 = S_MOV_B32 61440 # ADDR64: [[RSRCFMTHI:%[0-9]+]]:sgpr_32 = S_MOV_B32 61440
# ADDR64: [[ZERORSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[ZERO64]], %subreg.sub0_sub1, [[RSRCFMTLO]], %subreg.sub2, [[RSRCFMTHI]], %subreg.sub3 # ADDR64: [[ZERORSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[ZERO64]], %subreg.sub0_sub1, [[RSRCFMTLO]], %subreg.sub2, [[RSRCFMTHI]], %subreg.sub3
# ADDR64: [[VADDR64:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[RSRCPTR]].sub0, %subreg.sub0, [[RSRCPTR]].sub1, %subreg.sub1 # ADDR64: [[VADDR64:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[RSRCPTR]].sub0, %subreg.sub0, [[RSRCPTR]].sub1, %subreg.sub1
Show All 26 Lines

test/CodeGen/AMDGPU/multi-divergent-exit-region.ll

; RUN: opt -mtriple=amdgcn-- -S -amdgpu-unify-divergent-exit-nodes -verify -structurizecfg -verify -si-annotate-control-flow %s | FileCheck -check-prefix=IR %s ; RUN: opt -mtriple=amdgcn-- -S -amdgpu-unify-divergent-exit-nodes -verify -structurizecfg -verify -si-annotate-control-flow %s | FileCheck -check-prefix=IR %s
; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s ; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
; Add an extra verifier runs. There were some cases where invalid IR ; Add an extra verifier runs. There were some cases where invalid IR
; was produced but happened to be fixed by the later passes. ; was produced but happened to be fixed by the later passes.
; Make sure divergent control flow with multiple exits from a region ; Make sure divergent control flow with multiple exits from a region
; is properly handled. UnifyFunctionExitNodes should be run before ; is properly handled. UnifyFunctionExitNodes should be run before
; StructurizeCFG. ; StructurizeCFG.
; IR-LABEL: @multi_divergent_region_exit_ret_ret( ; IR-LABEL: @multi_divergent_region_exit_ret_ret(
; IR: %1 = call { i1, i64 } @llvm.amdgcn.if(i1 %0) ; IR: %1 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %0)
; IR: %2 = extractvalue { i1, i64 } %1, 0 ; IR: %2 = extractvalue { i1, i64 } %1, 0
; IR: %3 = extractvalue { i1, i64 } %1, 1 ; IR: %3 = extractvalue { i1, i64 } %1, 1
; IR: br i1 %2, label %LeafBlock1, label %Flow ; IR: br i1 %2, label %LeafBlock1, label %Flow
; IR: Flow: ; IR: Flow:
; IR: %4 = phi i1 [ true, %LeafBlock1 ], [ false, %entry ] ; IR: %4 = phi i1 [ true, %LeafBlock1 ], [ false, %entry ]
; IR: %5 = phi i1 [ %10, %LeafBlock1 ], [ false, %entry ] ; IR: %5 = phi i1 [ %10, %LeafBlock1 ], [ false, %entry ]
; IR: %6 = call { i1, i64 } @llvm.amdgcn.else(i64 %3) ; IR: %6 = call { i1, i64 } @llvm.amdgcn.else.i64.i64(i64 %3)
; IR: %7 = extractvalue { i1, i64 } %6, 0 ; IR: %7 = extractvalue { i1, i64 } %6, 0
; IR: %8 = extractvalue { i1, i64 } %6, 1 ; IR: %8 = extractvalue { i1, i64 } %6, 1
; IR: br i1 %7, label %LeafBlock, label %Flow1 ; IR: br i1 %7, label %LeafBlock, label %Flow1
; IR: LeafBlock: ; IR: LeafBlock:
; IR: br label %Flow1 ; IR: br label %Flow1
; IR: LeafBlock1: ; IR: LeafBlock1:
; IR: br label %Flow{{$}} ; IR: br label %Flow{{$}}
; IR: Flow2: ; IR: Flow2:
; IR: %11 = phi i1 [ false, %exit1 ], [ %15, %Flow1 ] ; IR: %11 = phi i1 [ false, %exit1 ], [ %15, %Flow1 ]
; IR: call void @llvm.amdgcn.end.cf(i64 %19) ; IR: call void @llvm.amdgcn.end.cf.i64(i64 %19)
; IR: %12 = call { i1, i64 } @llvm.amdgcn.if(i1 %11) ; IR: %12 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %11)
; IR: %13 = extractvalue { i1, i64 } %12, 0 ; IR: %13 = extractvalue { i1, i64 } %12, 0
; IR: %14 = extractvalue { i1, i64 } %12, 1 ; IR: %14 = extractvalue { i1, i64 } %12, 1
; IR: br i1 %13, label %exit0, label %UnifiedReturnBlock ; IR: br i1 %13, label %exit0, label %UnifiedReturnBlock
; IR: exit0: ; IR: exit0:
; IR: store volatile i32 9, i32 addrspace(1)* undef ; IR: store volatile i32 9, i32 addrspace(1)* undef
; IR: br label %UnifiedReturnBlock ; IR: br label %UnifiedReturnBlock
; IR: Flow1: ; IR: Flow1:
; IR: %15 = phi i1 [ %SwitchLeaf, %LeafBlock ], [ %4, %Flow ] ; IR: %15 = phi i1 [ %SwitchLeaf, %LeafBlock ], [ %4, %Flow ]
; IR: %16 = phi i1 [ %9, %LeafBlock ], [ %5, %Flow ] ; IR: %16 = phi i1 [ %9, %LeafBlock ], [ %5, %Flow ]
; IR: call void @llvm.amdgcn.end.cf(i64 %8) ; IR: call void @llvm.amdgcn.end.cf.i64(i64 %8)
; IR: %17 = call { i1, i64 } @llvm.amdgcn.if(i1 %16) ; IR: %17 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %16)
; IR: %18 = extractvalue { i1, i64 } %17, 0 ; IR: %18 = extractvalue { i1, i64 } %17, 0
; IR: %19 = extractvalue { i1, i64 } %17, 1 ; IR: %19 = extractvalue { i1, i64 } %17, 1
; IR: br i1 %18, label %exit1, label %Flow2 ; IR: br i1 %18, label %exit1, label %Flow2
; IR: exit1: ; IR: exit1:
; IR: store volatile i32 17, i32 addrspace(3)* undef ; IR: store volatile i32 17, i32 addrspace(3)* undef
; IR: br label %Flow2 ; IR: br label %Flow2
; IR: UnifiedReturnBlock: ; IR: UnifiedReturnBlock:
; IR: call void @llvm.amdgcn.end.cf(i64 %14) ; IR: call void @llvm.amdgcn.end.cf.i64(i64 %14)
; IR: ret void ; IR: ret void
; GCN-LABEL: {{^}}multi_divergent_region_exit_ret_ret: ; GCN-LABEL: {{^}}multi_divergent_region_exit_ret_ret:
; GCN-DAG: s_mov_b64 [[EXIT1:s\[[0-9]+:[0-9]+\]]], 0 ; GCN-DAG: s_mov_b64 [[EXIT1:s\[[0-9]+:[0-9]+\]]], 0
; GCN-DAG: v_cmp_lt_i32_e32 vcc, 1, ; GCN-DAG: v_cmp_lt_i32_e32 vcc, 1,
; GCN-DAG: s_mov_b64 [[EXIT0:s\[[0-9]+:[0-9]+\]]], 0 ; GCN-DAG: s_mov_b64 [[EXIT0:s\[[0-9]+:[0-9]+\]]], 0
▲ Show 20 Lines • Show All 70 Lines • ▼ Show 20 Linesexit0: ; preds = %LeafBlock, %LeafBlock1
ret void ret void
exit1: ; preds = %LeafBlock, %LeafBlock1 exit1: ; preds = %LeafBlock, %LeafBlock1
store volatile i32 17, i32 addrspace(3)* undef store volatile i32 17, i32 addrspace(3)* undef
ret void ret void
} }
; IR-LABEL: @multi_divergent_region_exit_unreachable_unreachable( ; IR-LABEL: @multi_divergent_region_exit_unreachable_unreachable(
; IR: %1 = call { i1, i64 } @llvm.amdgcn.if(i1 %0) ; IR: %1 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %0)
; IR: %6 = call { i1, i64 } @llvm.amdgcn.else(i64 %3) ; IR: %6 = call { i1, i64 } @llvm.amdgcn.else.i64.i64(i64 %3)
; IR: %11 = phi i1 [ false, %exit1 ], [ %15, %Flow1 ] ; IR: %11 = phi i1 [ false, %exit1 ], [ %15, %Flow1 ]
; IR: call void @llvm.amdgcn.end.cf(i64 %19) ; IR: call void @llvm.amdgcn.end.cf.i64(i64 %19)
; IR: %12 = call { i1, i64 } @llvm.amdgcn.if(i1 %11) ; IR: %12 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %11)
; IR: br i1 %13, label %exit0, label %UnifiedUnreachableBlock ; IR: br i1 %13, label %exit0, label %UnifiedUnreachableBlock
; IR: UnifiedUnreachableBlock: ; IR: UnifiedUnreachableBlock:
; IR-NEXT: unreachable ; IR-NEXT: unreachable
; FIXME: Probably should insert an s_endpgm anyway. ; FIXME: Probably should insert an s_endpgm anyway.
Show All 39 Lines
; IR-LABEL: @multi_exit_region_divergent_ret_uniform_ret( ; IR-LABEL: @multi_exit_region_divergent_ret_uniform_ret(
; IR: %divergent.cond0 = icmp slt i32 %tmp16, 2 ; IR: %divergent.cond0 = icmp slt i32 %tmp16, 2
; IR: llvm.amdgcn.if ; IR: llvm.amdgcn.if
; IR: br i1 ; IR: br i1
; IR: {{^}}Flow: ; IR: {{^}}Flow:
; IR: %4 = phi i1 [ true, %LeafBlock1 ], [ false, %entry ] ; IR: %4 = phi i1 [ true, %LeafBlock1 ], [ false, %entry ]
; IR: %5 = phi i1 [ %10, %LeafBlock1 ], [ false, %entry ] ; IR: %5 = phi i1 [ %10, %LeafBlock1 ], [ false, %entry ]
; IR: %6 = call { i1, i64 } @llvm.amdgcn.else(i64 %3) ; IR: %6 = call { i1, i64 } @llvm.amdgcn.else.i64.i64(i64 %3)
; IR: br i1 %7, label %LeafBlock, label %Flow1 ; IR: br i1 %7, label %LeafBlock, label %Flow1
; IR: {{^}}LeafBlock: ; IR: {{^}}LeafBlock:
; IR: %divergent.cond1 = icmp eq i32 %tmp16, 1 ; IR: %divergent.cond1 = icmp eq i32 %tmp16, 1
; IR: %9 = xor i1 %divergent.cond1, true ; IR: %9 = xor i1 %divergent.cond1, true
; IR: br label %Flow1 ; IR: br label %Flow1
; IR: LeafBlock1: ; IR: LeafBlock1:
; IR: %uniform.cond0 = icmp eq i32 %arg3, 2 ; IR: %uniform.cond0 = icmp eq i32 %arg3, 2
; IR: %10 = xor i1 %uniform.cond0, true ; IR: %10 = xor i1 %uniform.cond0, true
; IR: br label %Flow ; IR: br label %Flow
; IR: Flow2: ; IR: Flow2:
; IR: %11 = phi i1 [ false, %exit1 ], [ %15, %Flow1 ] ; IR: %11 = phi i1 [ false, %exit1 ], [ %15, %Flow1 ]
; IR: call void @llvm.amdgcn.end.cf(i64 %19) ; IR: call void @llvm.amdgcn.end.cf.i64(i64 %19)
; IR: %12 = call { i1, i64 } @llvm.amdgcn.if(i1 %11) ; IR: %12 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %11)
; IR: br i1 %13, label %exit0, label %UnifiedReturnBlock ; IR: br i1 %13, label %exit0, label %UnifiedReturnBlock
; IR: exit0: ; IR: exit0:
; IR: store volatile i32 9, i32 addrspace(1)* undef ; IR: store volatile i32 9, i32 addrspace(1)* undef
; IR: br label %UnifiedReturnBlock ; IR: br label %UnifiedReturnBlock
; IR: {{^}}Flow1: ; IR: {{^}}Flow1:
; IR: %15 = phi i1 [ %divergent.cond1, %LeafBlock ], [ %4, %Flow ] ; IR: %15 = phi i1 [ %divergent.cond1, %LeafBlock ], [ %4, %Flow ]
; IR: %16 = phi i1 [ %9, %LeafBlock ], [ %5, %Flow ] ; IR: %16 = phi i1 [ %9, %LeafBlock ], [ %5, %Flow ]
; IR: call void @llvm.amdgcn.end.cf(i64 %8) ; IR: call void @llvm.amdgcn.end.cf.i64(i64 %8)
; IR: %17 = call { i1, i64 } @llvm.amdgcn.if(i1 %16) ; IR: %17 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %16)
; IR: %18 = extractvalue { i1, i64 } %17, 0 ; IR: %18 = extractvalue { i1, i64 } %17, 0
; IR: %19 = extractvalue { i1, i64 } %17, 1 ; IR: %19 = extractvalue { i1, i64 } %17, 1
; IR: br i1 %18, label %exit1, label %Flow2 ; IR: br i1 %18, label %exit1, label %Flow2
; IR: exit1: ; IR: exit1:
; IR: store volatile i32 17, i32 addrspace(3)* undef ; IR: store volatile i32 17, i32 addrspace(3)* undef
; IR: br label %Flow2 ; IR: br label %Flow2
; IR: UnifiedReturnBlock: ; IR: UnifiedReturnBlock:
; IR: call void @llvm.amdgcn.end.cf(i64 %14) ; IR: call void @llvm.amdgcn.end.cf.i64(i64 %14)
; IR: ret void ; IR: ret void
define amdgpu_kernel void @multi_exit_region_divergent_ret_uniform_ret(i32 addrspace(1)* nocapture %arg0, i32 addrspace(1)* nocapture %arg1, i32 addrspace(1)* nocapture %arg2, i32 %arg3) #0 { define amdgpu_kernel void @multi_exit_region_divergent_ret_uniform_ret(i32 addrspace(1)* nocapture %arg0, i32 addrspace(1)* nocapture %arg1, i32 addrspace(1)* nocapture %arg2, i32 %arg3) #0 {
entry: entry:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x() #1 %tmp = tail call i32 @llvm.amdgcn.workitem.id.x() #1
%tmp1 = add i32 0, %tmp %tmp1 = add i32 0, %tmp
%tmp2 = zext i32 %tmp1 to i64 %tmp2 = zext i32 %tmp1 to i64
%tmp3 = add i64 0, %tmp2 %tmp3 = add i64 0, %tmp2
%tmp4 = shl i64 %tmp3, 32 %tmp4 = shl i64 %tmp3, 32
Show All 22 Linesexit0: ; preds = %LeafBlock, %LeafBlock1
ret void ret void
exit1: ; preds = %LeafBlock, %LeafBlock1 exit1: ; preds = %LeafBlock, %LeafBlock1
store volatile i32 17, i32 addrspace(3)* undef store volatile i32 17, i32 addrspace(3)* undef
ret void ret void
} }
; IR-LABEL: @multi_exit_region_uniform_ret_divergent_ret( ; IR-LABEL: @multi_exit_region_uniform_ret_divergent_ret(
; IR: %1 = call { i1, i64 } @llvm.amdgcn.if(i1 %0) ; IR: %1 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %0)
; IR: br i1 %2, label %LeafBlock1, label %Flow ; IR: br i1 %2, label %LeafBlock1, label %Flow
; IR: Flow: ; IR: Flow:
; IR: %4 = phi i1 [ true, %LeafBlock1 ], [ false, %entry ] ; IR: %4 = phi i1 [ true, %LeafBlock1 ], [ false, %entry ]
; IR: %5 = phi i1 [ %10, %LeafBlock1 ], [ false, %entry ] ; IR: %5 = phi i1 [ %10, %LeafBlock1 ], [ false, %entry ]
; IR: %6 = call { i1, i64 } @llvm.amdgcn.else(i64 %3) ; IR: %6 = call { i1, i64 } @llvm.amdgcn.else.i64.i64(i64 %3)
; IR: %11 = phi i1 [ false, %exit1 ], [ %15, %Flow1 ] ; IR: %11 = phi i1 [ false, %exit1 ], [ %15, %Flow1 ]
; IR: call void @llvm.amdgcn.end.cf(i64 %19) ; IR: call void @llvm.amdgcn.end.cf.i64(i64 %19)
; IR: %12 = call { i1, i64 } @llvm.amdgcn.if(i1 %11) ; IR: %12 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %11)
define amdgpu_kernel void @multi_exit_region_uniform_ret_divergent_ret(i32 addrspace(1)* nocapture %arg0, i32 addrspace(1)* nocapture %arg1, i32 addrspace(1)* nocapture %arg2, i32 %arg3) #0 { define amdgpu_kernel void @multi_exit_region_uniform_ret_divergent_ret(i32 addrspace(1)* nocapture %arg0, i32 addrspace(1)* nocapture %arg1, i32 addrspace(1)* nocapture %arg2, i32 %arg3) #0 {
entry: entry:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x() #1 %tmp = tail call i32 @llvm.amdgcn.workitem.id.x() #1
%tmp1 = add i32 0, %tmp %tmp1 = add i32 0, %tmp
%tmp2 = zext i32 %tmp1 to i64 %tmp2 = zext i32 %tmp1 to i64
%tmp3 = add i64 0, %tmp2 %tmp3 = add i64 0, %tmp2
%tmp4 = shl i64 %tmp3, 32 %tmp4 = shl i64 %tmp3, 32
Show All 24 Lines
exit1: ; preds = %LeafBlock, %LeafBlock1 exit1: ; preds = %LeafBlock, %LeafBlock1
store volatile i32 17, i32 addrspace(3)* undef store volatile i32 17, i32 addrspace(3)* undef
ret void ret void
} }
; IR-LABEL: @multi_divergent_region_exit_ret_ret_return_value( ; IR-LABEL: @multi_divergent_region_exit_ret_ret_return_value(
; IR: Flow2: ; IR: Flow2:
; IR: %11 = phi i1 [ false, %exit1 ], [ %15, %Flow1 ] ; IR: %11 = phi i1 [ false, %exit1 ], [ %15, %Flow1 ]
; IR: call void @llvm.amdgcn.end.cf(i64 %19) ; IR: call void @llvm.amdgcn.end.cf.i64(i64 %19)
; IR: UnifiedReturnBlock: ; IR: UnifiedReturnBlock:
; IR: %UnifiedRetVal = phi float [ 2.000000e+00, %Flow2 ], [ 1.000000e+00, %exit0 ] ; IR: %UnifiedRetVal = phi float [ 2.000000e+00, %Flow2 ], [ 1.000000e+00, %exit0 ]
; IR: call void @llvm.amdgcn.end.cf(i64 %14) ; IR: call void @llvm.amdgcn.end.cf.i64(i64 %14)
; IR: ret float %UnifiedRetVal ; IR: ret float %UnifiedRetVal
define amdgpu_ps float @multi_divergent_region_exit_ret_ret_return_value(i32 %vgpr) #0 { define amdgpu_ps float @multi_divergent_region_exit_ret_ret_return_value(i32 %vgpr) #0 {
entry: entry:
%Pivot = icmp slt i32 %vgpr, 2 %Pivot = icmp slt i32 %vgpr, 2
br i1 %Pivot, label %LeafBlock, label %LeafBlock1 br i1 %Pivot, label %LeafBlock, label %LeafBlock1
LeafBlock: ; preds = %entry LeafBlock: ; preds = %entry
%SwitchLeaf = icmp eq i32 %vgpr, 1 %SwitchLeaf = icmp eq i32 %vgpr, 1
▲ Show 20 Lines • Show All 51 Lines • ▼ Show 20 Linesexit0: ; preds = %LeafBlock, %LeafBlock1
ret float 1.0 ret float 1.0
exit1: ; preds = %LeafBlock, %LeafBlock1 exit1: ; preds = %LeafBlock, %LeafBlock1
store i32 17, i32 addrspace(3)* undef store i32 17, i32 addrspace(3)* undef
ret float 2.0 ret float 2.0
} }
; IR-LABEL: @multi_divergent_region_exit_ret_unreachable( ; IR-LABEL: @multi_divergent_region_exit_ret_unreachable(
; IR: %1 = call { i1, i64 } @llvm.amdgcn.if(i1 %0) ; IR: %1 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %0)
; IR: Flow: ; IR: Flow:
; IR: %4 = phi i1 [ true, %LeafBlock1 ], [ false, %entry ] ; IR: %4 = phi i1 [ true, %LeafBlock1 ], [ false, %entry ]
; IR: %5 = phi i1 [ %10, %LeafBlock1 ], [ false, %entry ] ; IR: %5 = phi i1 [ %10, %LeafBlock1 ], [ false, %entry ]
; IR: %6 = call { i1, i64 } @llvm.amdgcn.else(i64 %3) ; IR: %6 = call { i1, i64 } @llvm.amdgcn.else.i64.i64(i64 %3)
; IR: Flow2: ; IR: Flow2:
; IR: %11 = phi i1 [ false, %exit1 ], [ %15, %Flow1 ] ; IR: %11 = phi i1 [ false, %exit1 ], [ %15, %Flow1 ]
; IR: call void @llvm.amdgcn.end.cf(i64 %19) ; IR: call void @llvm.amdgcn.end.cf.i64(i64 %19)
; IR: %12 = call { i1, i64 } @llvm.amdgcn.if(i1 %11) ; IR: %12 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %11)
; IR: br i1 %13, label %exit0, label %UnifiedReturnBlock ; IR: br i1 %13, label %exit0, label %UnifiedReturnBlock
; IR: exit0: ; IR: exit0:
; IR-NEXT: store volatile i32 17, i32 addrspace(3)* undef ; IR-NEXT: store volatile i32 17, i32 addrspace(3)* undef
; IR-NEXT: br label %UnifiedReturnBlock ; IR-NEXT: br label %UnifiedReturnBlock
; IR: Flow1: ; IR: Flow1:
; IR: %15 = phi i1 [ %SwitchLeaf, %LeafBlock ], [ %4, %Flow ] ; IR: %15 = phi i1 [ %SwitchLeaf, %LeafBlock ], [ %4, %Flow ]
; IR: %16 = phi i1 [ %9, %LeafBlock ], [ %5, %Flow ] ; IR: %16 = phi i1 [ %9, %LeafBlock ], [ %5, %Flow ]
; IR: call void @llvm.amdgcn.end.cf(i64 %8) ; IR: call void @llvm.amdgcn.end.cf.i64(i64 %8)
; IR: %17 = call { i1, i64 } @llvm.amdgcn.if(i1 %16) ; IR: %17 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %16)
; IR: %18 = extractvalue { i1, i64 } %17, 0 ; IR: %18 = extractvalue { i1, i64 } %17, 0
; IR: %19 = extractvalue { i1, i64 } %17, 1 ; IR: %19 = extractvalue { i1, i64 } %17, 1
; IR: br i1 %18, label %exit1, label %Flow2 ; IR: br i1 %18, label %exit1, label %Flow2
; IR: exit1: ; IR: exit1:
; IR-NEXT: store volatile i32 9, i32 addrspace(1)* undef ; IR-NEXT: store volatile i32 9, i32 addrspace(1)* undef
; IR-NEXT: call void @llvm.amdgcn.unreachable() ; IR-NEXT: call void @llvm.amdgcn.unreachable()
; IR-NEXT: br label %Flow2 ; IR-NEXT: br label %Flow2
; IR: UnifiedReturnBlock: ; IR: UnifiedReturnBlock:
; IR-NEXT: call void @llvm.amdgcn.end.cf(i64 %14) ; IR-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %14)
; IR-NEXT: ret void ; IR-NEXT: ret void
define amdgpu_kernel void @multi_divergent_region_exit_ret_unreachable(i32 addrspace(1)* nocapture %arg0, i32 addrspace(1)* nocapture %arg1, i32 addrspace(1)* nocapture %arg2) #0 { define amdgpu_kernel void @multi_divergent_region_exit_ret_unreachable(i32 addrspace(1)* nocapture %arg0, i32 addrspace(1)* nocapture %arg1, i32 addrspace(1)* nocapture %arg2) #0 {
entry: entry:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x() #1 %tmp = tail call i32 @llvm.amdgcn.workitem.id.x() #1
%tmp1 = add i32 0, %tmp %tmp1 = add i32 0, %tmp
%tmp2 = zext i32 %tmp1 to i64 %tmp2 = zext i32 %tmp1 to i64
%tmp3 = add i64 0, %tmp2 %tmp3 = add i64 0, %tmp2
%tmp4 = shl i64 %tmp3, 32 %tmp4 = shl i64 %tmp3, 32
Show All 39 Lines
; IR: indirect.exit1: ; IR: indirect.exit1:
; IR: %load = load volatile i32, i32 addrspace(1)* undef ; IR: %load = load volatile i32, i32 addrspace(1)* undef
; IR: store volatile i32 %load, i32 addrspace(1)* undef ; IR: store volatile i32 %load, i32 addrspace(1)* undef
; IR: store volatile i32 9, i32 addrspace(1)* undef ; IR: store volatile i32 9, i32 addrspace(1)* undef
; IR: call void @llvm.amdgcn.unreachable() ; IR: call void @llvm.amdgcn.unreachable()
; IR-NEXT: br label %Flow2 ; IR-NEXT: br label %Flow2
; IR: UnifiedReturnBlock: ; preds = %exit0, %Flow2 ; IR: UnifiedReturnBlock: ; preds = %exit0, %Flow2
; IR-NEXT: call void @llvm.amdgcn.end.cf(i64 %14) ; IR-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %14)
; IR-NEXT: ret void ; IR-NEXT: ret void
define amdgpu_kernel void @indirect_multi_divergent_region_exit_ret_unreachable(i32 addrspace(1)* nocapture %arg0, i32 addrspace(1)* nocapture %arg1, i32 addrspace(1)* nocapture %arg2) #0 { define amdgpu_kernel void @indirect_multi_divergent_region_exit_ret_unreachable(i32 addrspace(1)* nocapture %arg0, i32 addrspace(1)* nocapture %arg1, i32 addrspace(1)* nocapture %arg2) #0 {
entry: entry:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x() #1 %tmp = tail call i32 @llvm.amdgcn.workitem.id.x() #1
%tmp1 = add i32 0, %tmp %tmp1 = add i32 0, %tmp
%tmp2 = zext i32 %tmp1 to i64 %tmp2 = zext i32 %tmp1 to i64
%tmp3 = add i64 0, %tmp2 %tmp3 = add i64 0, %tmp2
%tmp4 = shl i64 %tmp3, 32 %tmp4 = shl i64 %tmp3, 32
▲ Show 20 Lines • Show All 138 Lines • ▼ Show 20 Lines
; IR: %8 = phi i1 [ false, %uniform.ret1 ], [ true, %uniform.multi.exit.region ] ; IR: %8 = phi i1 [ false, %uniform.ret1 ], [ true, %uniform.multi.exit.region ]
; IR: br i1 %8, label %uniform.if, label %Flow2 ; IR: br i1 %8, label %uniform.if, label %Flow2
; IR: Flow: ; preds = %uniform.then, %uniform.if ; IR: Flow: ; preds = %uniform.then, %uniform.if
; IR: %11 = phi i1 [ %10, %uniform.then ], [ %9, %uniform.if ] ; IR: %11 = phi i1 [ %10, %uniform.then ], [ %9, %uniform.if ]
; IR: br i1 %11, label %uniform.endif, label %uniform.ret0 ; IR: br i1 %11, label %uniform.endif, label %uniform.ret0
; IR: UnifiedReturnBlock: ; preds = %Flow3, %Flow2 ; IR: UnifiedReturnBlock: ; preds = %Flow3, %Flow2
; IR-NEXT: call void @llvm.amdgcn.end.cf(i64 %6) ; IR-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %6)
; IR-NEXT: ret void ; IR-NEXT: ret void
define amdgpu_kernel void @uniform_complex_multi_ret_nest_in_divergent_triangle(i32 %arg0) #0 { define amdgpu_kernel void @uniform_complex_multi_ret_nest_in_divergent_triangle(i32 %arg0) #0 {
entry: entry:
%id.x = tail call i32 @llvm.amdgcn.workitem.id.x() %id.x = tail call i32 @llvm.amdgcn.workitem.id.x()
%divergent.cond0 = icmp eq i32 %id.x, 0 %divergent.cond0 = icmp eq i32 %id.x, 0
br i1 %divergent.cond0, label %uniform.multi.exit.region, label %divergent.ret br i1 %divergent.cond0, label %uniform.multi.exit.region, label %divergent.ret
uniform.multi.exit.region: uniform.multi.exit.region:
Show All 29 Lines
} }
; IR-LABEL: @multi_divergent_unreachable_exit( ; IR-LABEL: @multi_divergent_unreachable_exit(
; IR: UnifiedUnreachableBlock: ; IR: UnifiedUnreachableBlock:
; IR-NEXT: call void @llvm.amdgcn.unreachable() ; IR-NEXT: call void @llvm.amdgcn.unreachable()
; IR-NEXT: br label %UnifiedReturnBlock ; IR-NEXT: br label %UnifiedReturnBlock
; IR: UnifiedReturnBlock: ; IR: UnifiedReturnBlock:
; IR-NEXT: call void @llvm.amdgcn.end.cf(i64 ; IR-NEXT: call void @llvm.amdgcn.end.cf.i64(i64
; IR-NEXT: ret void ; IR-NEXT: ret void
define amdgpu_kernel void @multi_divergent_unreachable_exit() #0 { define amdgpu_kernel void @multi_divergent_unreachable_exit() #0 {
bb: bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x() %tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
switch i32 %tmp, label %bb3 [ switch i32 %tmp, label %bb3 [
i32 2, label %bb1 i32 2, label %bb1
i32 0, label %bb2 i32 0, label %bb2
] ]
Show All 23 Lines

test/CodeGen/AMDGPU/multilevel-break.ll

; RUN: opt -S -mtriple=amdgcn-- -structurizecfg -si-annotate-control-flow < %s | FileCheck -check-prefix=OPT %s ; RUN: opt -S -mtriple=amdgcn-- -structurizecfg -si-annotate-control-flow < %s | FileCheck -check-prefix=OPT %s
; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s ; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
; OPT-LABEL: {{^}}define amdgpu_vs void @multi_else_break( ; OPT-LABEL: {{^}}define amdgpu_vs void @multi_else_break(
; OPT: main_body: ; OPT: main_body:
; OPT: LOOP.outer: ; OPT: LOOP.outer:
; OPT: LOOP: ; OPT: LOOP:
; OPT: [[if:%[0-9]+]] = call { i1, i64 } @llvm.amdgcn.if( ; OPT: [[if:%[0-9]+]] = call { i1, i64 } @llvm.amdgcn.if.i64(
; OPT: [[if_exec:%[0-9]+]] = extractvalue { i1, i64 } [[if]], 1 ; OPT: [[if_exec:%[0-9]+]] = extractvalue { i1, i64 } [[if]], 1
; ;
; OPT: Flow: ; OPT: Flow:
; ;
; Ensure two if.break calls, for both the inner and outer loops ; Ensure two if.break calls, for both the inner and outer loops
; OPT: call void @llvm.amdgcn.end.cf ; OPT: call void @llvm.amdgcn.end.cf
; OPT-NEXT: call i64 @llvm.amdgcn.if.break(i1 ; OPT-NEXT: call i64 @llvm.amdgcn.if.break.i64.i64(i1
; OPT-NEXT: call i1 @llvm.amdgcn.loop(i64 ; OPT-NEXT: call i1 @llvm.amdgcn.loop.i64(i64
; OPT-NEXT: call i64 @llvm.amdgcn.if.break(i1 ; OPT-NEXT: call i64 @llvm.amdgcn.if.break.i64.i64(i1
; ;
; OPT: Flow1: ; OPT: Flow1:
; GCN-LABEL: {{^}}multi_else_break: ; GCN-LABEL: {{^}}multi_else_break:
; GCN: ; %main_body ; GCN: ; %main_body
; GCN: s_mov_b64 [[LEFT_OUTER:s\[[0-9]+:[0-9]+\]]], 0{{$}} ; GCN: s_mov_b64 [[LEFT_OUTER:s\[[0-9]+:[0-9]+\]]], 0{{$}}
▲ Show 20 Lines • Show All 134 Lines • Show Last 20 Lines

test/CodeGen/AMDGPU/nested-loop-conditions.ll

; RUN: opt -mtriple=amdgcn-- -S -structurizecfg -si-annotate-control-flow %s | FileCheck -check-prefix=IR %s ; RUN: opt -mtriple=amdgcn-- -S -structurizecfg -si-annotate-control-flow %s | FileCheck -check-prefix=IR %s
; RUN: llc -march=amdgcn -mcpu=hawaii -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s ; RUN: llc -march=amdgcn -mcpu=hawaii -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
; After structurizing, there are 3 levels of loops. The i1 phi ; After structurizing, there are 3 levels of loops. The i1 phi
; conditions mutually depend on each other, so it isn't safe to delete ; conditions mutually depend on each other, so it isn't safe to delete
; the condition that appears to have no uses until the loop is ; the condition that appears to have no uses until the loop is
; completely processed. ; completely processed.
; IR-LABEL: @reduced_nested_loop_conditions( ; IR-LABEL: @reduced_nested_loop_conditions(
; IR: bb5: ; IR: bb5:
; IR-NEXT: %phi.broken = phi i64 [ %3, %bb10 ], [ 0, %bb ] ; IR-NEXT: %phi.broken = phi i64 [ %3, %bb10 ], [ 0, %bb ]
; IR-NEXT: %tmp6 = phi i32 [ 0, %bb ], [ %tmp11, %bb10 ] ; IR-NEXT: %tmp6 = phi i32 [ 0, %bb ], [ %tmp11, %bb10 ]
; IR-NEXT: %tmp7 = icmp eq i32 %tmp6, 1 ; IR-NEXT: %tmp7 = icmp eq i32 %tmp6, 1
; IR-NEXT: %0 = call { i1, i64 } @llvm.amdgcn.if(i1 %tmp7) ; IR-NEXT: %0 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %tmp7)
; IR-NEXT: %1 = extractvalue { i1, i64 } %0, 0 ; IR-NEXT: %1 = extractvalue { i1, i64 } %0, 0
; IR-NEXT: %2 = extractvalue { i1, i64 } %0, 1 ; IR-NEXT: %2 = extractvalue { i1, i64 } %0, 1
; IR-NEXT: br i1 %1, label %bb8, label %Flow ; IR-NEXT: br i1 %1, label %bb8, label %Flow
; IR: bb8: ; IR: bb8:
; IR-NEXT: br label %bb13 ; IR-NEXT: br label %bb13
; IR: bb10: ; IR: bb10:
; IR-NEXT: %tmp11 = phi i32 [ %6, %Flow ] ; IR-NEXT: %tmp11 = phi i32 [ %6, %Flow ]
; IR-NEXT: %tmp12 = phi i1 [ %5, %Flow ] ; IR-NEXT: %tmp12 = phi i1 [ %5, %Flow ]
; IR-NEXT: %3 = call i64 @llvm.amdgcn.if.break(i1 %tmp12, i64 %phi.broken) ; IR-NEXT: %3 = call i64 @llvm.amdgcn.if.break.i64.i64(i1 %tmp12, i64 %phi.broken)
; IR-NEXT: %4 = call i1 @llvm.amdgcn.loop(i64 %3) ; IR-NEXT: %4 = call i1 @llvm.amdgcn.loop.i64(i64 %3)
; IR-NEXT: br i1 %4, label %bb23, label %bb5 ; IR-NEXT: br i1 %4, label %bb23, label %bb5
; IR: Flow: ; IR: Flow:
; IR-NEXT: %5 = phi i1 [ %tmp22, %bb4 ], [ true, %bb5 ] ; IR-NEXT: %5 = phi i1 [ %tmp22, %bb4 ], [ true, %bb5 ]
; IR-NEXT: %6 = phi i32 [ %tmp21, %bb4 ], [ undef, %bb5 ] ; IR-NEXT: %6 = phi i32 [ %tmp21, %bb4 ], [ undef, %bb5 ]
; IR-NEXT: call void @llvm.amdgcn.end.cf(i64 %2) ; IR-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %2)
; IR-NEXT: br label %bb10 ; IR-NEXT: br label %bb10
; IR: bb13: ; IR: bb13:
; IR-NEXT: %tmp14 = phi i1 [ %tmp22, %bb3 ], [ true, %bb8 ] ; IR-NEXT: %tmp14 = phi i1 [ %tmp22, %bb3 ], [ true, %bb8 ]
; IR-NEXT: %tmp15 = bitcast i64 %tmp2 to <2 x i32> ; IR-NEXT: %tmp15 = bitcast i64 %tmp2 to <2 x i32>
; IR-NEXT: br i1 %tmp14, label %bb16, label %bb20 ; IR-NEXT: br i1 %tmp14, label %bb16, label %bb20
; IR: bb16: ; IR: bb16:
; IR-NEXT: %tmp17 = extractelement <2 x i32> %tmp15, i64 1 ; IR-NEXT: %tmp17 = extractelement <2 x i32> %tmp15, i64 1
; IR-NEXT: %tmp18 = getelementptr inbounds i32, i32 addrspace(3)* undef, i32 %tmp17 ; IR-NEXT: %tmp18 = getelementptr inbounds i32, i32 addrspace(3)* undef, i32 %tmp17
; IR-NEXT: %tmp19 = load volatile i32, i32 addrspace(3)* %tmp18 ; IR-NEXT: %tmp19 = load volatile i32, i32 addrspace(3)* %tmp18
; IR-NEXT: br label %bb20 ; IR-NEXT: br label %bb20
; IR: bb20: ; IR: bb20:
; IR-NEXT: %tmp21 = phi i32 [ %tmp19, %bb16 ], [ 0, %bb13 ] ; IR-NEXT: %tmp21 = phi i32 [ %tmp19, %bb16 ], [ 0, %bb13 ]
; IR-NEXT: %tmp22 = phi i1 [ false, %bb16 ], [ %tmp14, %bb13 ] ; IR-NEXT: %tmp22 = phi i1 [ false, %bb16 ], [ %tmp14, %bb13 ]
; IR-NEXT: br label %bb9 ; IR-NEXT: br label %bb9
; IR: bb23: ; IR: bb23:
; IR-NEXT: call void @llvm.amdgcn.end.cf(i64 %3) ; IR-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %3)
; IR-NEXT: ret void ; IR-NEXT: ret void
; GCN-LABEL: {{^}}reduced_nested_loop_conditions: ; GCN-LABEL: {{^}}reduced_nested_loop_conditions:
; GCN: s_cmp_lg_u32 s{{[0-9]+}}, 1 ; GCN: s_cmp_lg_u32 s{{[0-9]+}}, 1
; GCN-NEXT: s_cbranch_scc0 ; GCN-NEXT: s_cbranch_scc0
; FIXME: Should fold to unconditional branch? ; FIXME: Should fold to unconditional branch?
▲ Show 20 Lines • Show All 53 Lines • ▼ Show 20 Lines
bb23: ; preds = %bb10 bb23: ; preds = %bb10
ret void ret void
} }
; Earlier version of above, before a run of the structurizer. ; Earlier version of above, before a run of the structurizer.
; IR-LABEL: @nested_loop_conditions( ; IR-LABEL: @nested_loop_conditions(
; IR: Flow3: ; IR: Flow3:
; IR-NEXT: call void @llvm.amdgcn.end.cf(i64 %21) ; IR-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %21)
; IR-NEXT: %0 = call { i1, i64 } @llvm.amdgcn.if(i1 %14) ; IR-NEXT: %0 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %14)
; IR-NEXT: %1 = extractvalue { i1, i64 } %0, 0 ; IR-NEXT: %1 = extractvalue { i1, i64 } %0, 0
; IR-NEXT: %2 = extractvalue { i1, i64 } %0, 1 ; IR-NEXT: %2 = extractvalue { i1, i64 } %0, 1
; IR-NEXT: br i1 %1, label %bb4.bb13_crit_edge, label %Flow4 ; IR-NEXT: br i1 %1, label %bb4.bb13_crit_edge, label %Flow4
; IR: Flow4: ; IR: Flow4:
; IR-NEXT: %3 = phi i1 [ true, %bb4.bb13_crit_edge ], [ false, %Flow3 ] ; IR-NEXT: %3 = phi i1 [ true, %bb4.bb13_crit_edge ], [ false, %Flow3 ]
; IR-NEXT: call void @llvm.amdgcn.end.cf(i64 %2) ; IR-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %2)
; IR-NEXT: br label %Flow ; IR-NEXT: br label %Flow
; IR: Flow: ; IR: Flow:
; IR-NEXT: %4 = phi i1 [ %3, %Flow4 ], [ true, %bb ] ; IR-NEXT: %4 = phi i1 [ %3, %Flow4 ], [ true, %bb ]
; IR-NEXT: %5 = call { i1, i64 } @llvm.amdgcn.if(i1 %4) ; IR-NEXT: %5 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %4)
; IR-NEXT: %6 = extractvalue { i1, i64 } %5, 0 ; IR-NEXT: %6 = extractvalue { i1, i64 } %5, 0
; IR-NEXT: %7 = extractvalue { i1, i64 } %5, 1 ; IR-NEXT: %7 = extractvalue { i1, i64 } %5, 1
; IR-NEXT: br i1 %6, label %bb13, label %bb31 ; IR-NEXT: br i1 %6, label %bb13, label %bb31
; IR: bb14: ; IR: bb14:
; IR: %tmp15 = icmp eq i32 %tmp1037, 1 ; IR: %tmp15 = icmp eq i32 %tmp1037, 1
; IR-NEXT: %8 = call { i1, i64 } @llvm.amdgcn.if(i1 %tmp15) ; IR-NEXT: %8 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %tmp15)
; IR: Flow1: ; IR: Flow1:
; IR-NEXT: %11 = phi <4 x i32> [ %tmp9, %bb21 ], [ undef, %bb14 ] ; IR-NEXT: %11 = phi <4 x i32> [ %tmp9, %bb21 ], [ undef, %bb14 ]
; IR-NEXT: %12 = phi i32 [ %tmp10, %bb21 ], [ undef, %bb14 ] ; IR-NEXT: %12 = phi i32 [ %tmp10, %bb21 ], [ undef, %bb14 ]
; IR-NEXT: %13 = phi i1 [ %18, %bb21 ], [ true, %bb14 ] ; IR-NEXT: %13 = phi i1 [ %18, %bb21 ], [ true, %bb14 ]
; IR-NEXT: %14 = phi i1 [ %18, %bb21 ], [ false, %bb14 ] ; IR-NEXT: %14 = phi i1 [ %18, %bb21 ], [ false, %bb14 ]
; IR-NEXT: %15 = phi i1 [ false, %bb21 ], [ true, %bb14 ] ; IR-NEXT: %15 = phi i1 [ false, %bb21 ], [ true, %bb14 ]
; IR-NEXT: call void @llvm.amdgcn.end.cf(i64 %10) ; IR-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %10)
; IR-NEXT: %16 = call i64 @llvm.amdgcn.if.break(i1 %13, i64 %phi.broken) ; IR-NEXT: %16 = call i64 @llvm.amdgcn.if.break.i64.i64(i1 %13, i64 %phi.broken)
; IR-NEXT: %17 = call i1 @llvm.amdgcn.loop(i64 %16) ; IR-NEXT: %17 = call i1 @llvm.amdgcn.loop.i64(i64 %16)
; IR-NEXT: br i1 %17, label %Flow2, label %bb14 ; IR-NEXT: br i1 %17, label %Flow2, label %bb14
; IR: bb21: ; IR: bb21:
; IR: %tmp12 = icmp slt i32 %tmp11, 9 ; IR: %tmp12 = icmp slt i32 %tmp11, 9
; IR-NEXT: %18 = xor i1 %tmp12, true ; IR-NEXT: %18 = xor i1 %tmp12, true
; IR-NEXT: br label %Flow1 ; IR-NEXT: br label %Flow1
; IR: Flow2: ; IR: Flow2:
; IR-NEXT: call void @llvm.amdgcn.end.cf(i64 %16) ; IR-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %16)
; IR-NEXT: %19 = call { i1, i64 } @llvm.amdgcn.if(i1 %15) ; IR-NEXT: %19 = call { i1, i64 } @llvm.amdgcn.if.i64(i1 %15)
; IR-NEXT: %20 = extractvalue { i1, i64 } %19, 0 ; IR-NEXT: %20 = extractvalue { i1, i64 } %19, 0
; IR-NEXT: %21 = extractvalue { i1, i64 } %19, 1 ; IR-NEXT: %21 = extractvalue { i1, i64 } %19, 1
; IR-NEXT: br i1 %20, label %bb31.loopexit, label %Flow3 ; IR-NEXT: br i1 %20, label %bb31.loopexit, label %Flow3
; IR: bb31: ; IR: bb31:
; IR-NEXT: call void @llvm.amdgcn.end.cf(i64 %7) ; IR-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 %7)
; IR-NEXT: store volatile i32 0, i32 addrspace(1)* undef ; IR-NEXT: store volatile i32 0, i32 addrspace(1)* undef
; IR-NEXT: ret void ; IR-NEXT: ret void
; GCN-LABEL: {{^}}nested_loop_conditions: ; GCN-LABEL: {{^}}nested_loop_conditions:
; GCN: v_cmp_lt_i32_e32 vcc, 8, v ; GCN: v_cmp_lt_i32_e32 vcc, 8, v
; GCN: s_and_b64 vcc, exec, vcc ; GCN: s_and_b64 vcc, exec, vcc
▲ Show 20 Lines • Show All 88 Lines • Show Last 20 Lines

test/CodeGen/AMDGPU/or3.ll

Show All 16 Lines
; GFX9-LABEL: or3: ; GFX9-LABEL: or3:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_or3_b32 v0, v0, v1, v2 ; GFX9-NEXT: v_or3_b32 v0, v0, v1, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: or3: ; GFX10-LABEL: or3:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_or3_b32 v0, v0, v1, v2 ; GFX10-NEXT: v_or3_b32 v0, v0, v1, v2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = or i32 %a, %b %x = or i32 %a, %b
%result = or i32 %x, %c %result = or i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
; ThreeOp instruction variant not used due to Constant Bus Limitations ; ThreeOp instruction variant not used due to Constant Bus Limitations
Show All 9 Lines
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_or_b32_e32 v0, s2, v0 ; GFX9-NEXT: v_or_b32_e32 v0, s2, v0
; GFX9-NEXT: v_or_b32_e32 v0, s3, v0 ; GFX9-NEXT: v_or_b32_e32 v0, s3, v0
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: or3_vgpr_a: ; GFX10-LABEL: or3_vgpr_a:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_or3_b32 v0, v0, s2, s3 ; GFX10-NEXT: v_or3_b32 v0, v0, s2, s3
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = or i32 %a, %b %x = or i32 %a, %b
%result = or i32 %x, %c %result = or i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @or3_vgpr_all2(i32 %a, i32 %b, i32 %c) { define amdgpu_ps float @or3_vgpr_all2(i32 %a, i32 %b, i32 %c) {
; VI-LABEL: or3_vgpr_all2: ; VI-LABEL: or3_vgpr_all2:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_or_b32_e32 v1, v1, v2 ; VI-NEXT: v_or_b32_e32 v1, v1, v2
; VI-NEXT: v_or_b32_e32 v0, v0, v1 ; VI-NEXT: v_or_b32_e32 v0, v0, v1
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: or3_vgpr_all2: ; GFX9-LABEL: or3_vgpr_all2:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_or3_b32 v0, v1, v2, v0 ; GFX9-NEXT: v_or3_b32 v0, v1, v2, v0
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: or3_vgpr_all2: ; GFX10-LABEL: or3_vgpr_all2:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_or3_b32 v0, v1, v2, v0 ; GFX10-NEXT: v_or3_b32 v0, v1, v2, v0
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = or i32 %b, %c %x = or i32 %b, %c
%result = or i32 %a, %x %result = or i32 %a, %x
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @or3_vgpr_bc(i32 inreg %a, i32 %b, i32 %c) { define amdgpu_ps float @or3_vgpr_bc(i32 inreg %a, i32 %b, i32 %c) {
; VI-LABEL: or3_vgpr_bc: ; VI-LABEL: or3_vgpr_bc:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_or_b32_e32 v0, s2, v0 ; VI-NEXT: v_or_b32_e32 v0, s2, v0
; VI-NEXT: v_or_b32_e32 v0, v0, v1 ; VI-NEXT: v_or_b32_e32 v0, v0, v1
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: or3_vgpr_bc: ; GFX9-LABEL: or3_vgpr_bc:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_or3_b32 v0, s2, v0, v1 ; GFX9-NEXT: v_or3_b32 v0, s2, v0, v1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: or3_vgpr_bc: ; GFX10-LABEL: or3_vgpr_bc:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_or3_b32 v0, s2, v0, v1 ; GFX10-NEXT: v_or3_b32 v0, s2, v0, v1
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = or i32 %a, %b %x = or i32 %a, %b
%result = or i32 %x, %c %result = or i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @or3_vgpr_const(i32 %a, i32 %b) { define amdgpu_ps float @or3_vgpr_const(i32 %a, i32 %b) {
; VI-LABEL: or3_vgpr_const: ; VI-LABEL: or3_vgpr_const:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_or_b32_e32 v0, v1, v0 ; VI-NEXT: v_or_b32_e32 v0, v1, v0
; VI-NEXT: v_or_b32_e32 v0, 64, v0 ; VI-NEXT: v_or_b32_e32 v0, 64, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: or3_vgpr_const: ; GFX9-LABEL: or3_vgpr_const:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_or3_b32 v0, v1, v0, 64 ; GFX9-NEXT: v_or3_b32 v0, v1, v0, 64
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: or3_vgpr_const: ; GFX10-LABEL: or3_vgpr_const:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_or3_b32 v0, v1, v0, 64 ; GFX10-NEXT: v_or3_b32 v0, v1, v0, 64
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = or i32 64, %b %x = or i32 64, %b
%result = or i32 %x, %a %result = or i32 %x, %a
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }

test/CodeGen/AMDGPU/regbank-reassign.mir

Show First 20 Lines • Show All 43 Lines • ▼ Show 20 Lines
body: | body: |
bb.0: bb.0:
%0 = IMPLICIT_DEF %0 = IMPLICIT_DEF
%1 = IMPLICIT_DEF %1 = IMPLICIT_DEF
GLOBAL_STORE_DWORDX2 %1, %0, 0, 0, 0, 0, implicit $exec GLOBAL_STORE_DWORDX2 %1, %0, 0, 0, 0, 0, implicit $exec
S_ENDPGM 0 S_ENDPGM 0
... ...
# GCN-LABEL: s11_vs_vcc{{$}}
# GCN: $vgpr0, $vcc_lo = V_ADDC_U32_e64 killed $sgpr14, killed $vgpr0, killed $vcc_lo, 0
---
name: s11_vs_vcc
tracksRegLiveness: true
registers:
- { id: 0, class: sgpr_32, preferred-register: '$sgpr11' }
- { id: 1, class: vgpr_32 }
- { id: 2, class: vgpr_32 }
body: |
bb.0:
%0 = IMPLICIT_DEF
%1 = IMPLICIT_DEF
$vcc_lo = IMPLICIT_DEF
%2, $vcc_lo = V_ADDC_U32_e64 killed %0, killed %1, killed $vcc_lo, 0, implicit $exec
S_ENDPGM 0
...
# GCN-LABEL: s0_vs_s16{{$}} # GCN-LABEL: s0_vs_s16{{$}}
# GCN: S_AND_B32 killed renamable $sgpr14, $sgpr0, # GCN: S_AND_B32 killed renamable $sgpr14, $sgpr0,
--- ---
name: s0_vs_s16 name: s0_vs_s16
tracksRegLiveness: true tracksRegLiveness: true
registers: registers:
- { id: 0, class: sgpr_32, preferred-register: '$sgpr16' } - { id: 0, class: sgpr_32, preferred-register: '$sgpr16' }
- { id: 1, class: sgpr_32 } - { id: 1, class: sgpr_32 }
▲ Show 20 Lines • Show All 277 Lines • Show Last 20 Lines

test/CodeGen/AMDGPU/shl_add.ll

Show All 16 Lines
; GFX9-LABEL: shl_add: ; GFX9-LABEL: shl_add:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_lshl_add_u32 v0, v0, v1, v2 ; GFX9-NEXT: v_lshl_add_u32 v0, v0, v1, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: shl_add: ; GFX10-LABEL: shl_add:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_add_u32 v0, v0, v1, v2 ; GFX10-NEXT: v_lshl_add_u32 v0, v0, v1, v2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = shl i32 %a, %b %x = shl i32 %a, %b
%result = add i32 %x, %c %result = add i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
; ThreeOp instruction variant not used due to Constant Bus Limitations ; ThreeOp instruction variant not used due to Constant Bus Limitations
define amdgpu_ps float @shl_add_vgpr_a(i32 %a, i32 inreg %b, i32 inreg %c) { define amdgpu_ps float @shl_add_vgpr_a(i32 %a, i32 inreg %b, i32 inreg %c) {
; VI-LABEL: shl_add_vgpr_a: ; VI-LABEL: shl_add_vgpr_a:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_lshlrev_b32_e32 v0, s2, v0 ; VI-NEXT: v_lshlrev_b32_e32 v0, s2, v0
; VI-NEXT: v_add_u32_e32 v0, vcc, s3, v0 ; VI-NEXT: v_add_u32_e32 v0, vcc, s3, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: shl_add_vgpr_a: ; GFX9-LABEL: shl_add_vgpr_a:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_lshlrev_b32_e32 v0, s2, v0 ; GFX9-NEXT: v_lshlrev_b32_e32 v0, s2, v0
; GFX9-NEXT: v_add_u32_e32 v0, s3, v0 ; GFX9-NEXT: v_add_u32_e32 v0, s3, v0
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: shl_add_vgpr_a: ; GFX10-LABEL: shl_add_vgpr_a:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_add_u32 v0, v0, s2, s3 ; GFX10-NEXT: v_lshl_add_u32 v0, v0, s2, s3
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = shl i32 %a, %b %x = shl i32 %a, %b
%result = add i32 %x, %c %result = add i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @shl_add_vgpr_all(i32 %a, i32 %b, i32 %c) { define amdgpu_ps float @shl_add_vgpr_all(i32 %a, i32 %b, i32 %c) {
; VI-LABEL: shl_add_vgpr_all: ; VI-LABEL: shl_add_vgpr_all:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_lshlrev_b32_e32 v0, v1, v0 ; VI-NEXT: v_lshlrev_b32_e32 v0, v1, v0
; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v2 ; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v2
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: shl_add_vgpr_all: ; GFX9-LABEL: shl_add_vgpr_all:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_lshl_add_u32 v0, v0, v1, v2 ; GFX9-NEXT: v_lshl_add_u32 v0, v0, v1, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: shl_add_vgpr_all: ; GFX10-LABEL: shl_add_vgpr_all:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_add_u32 v0, v0, v1, v2 ; GFX10-NEXT: v_lshl_add_u32 v0, v0, v1, v2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = shl i32 %a, %b %x = shl i32 %a, %b
%result = add i32 %x, %c %result = add i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @shl_add_vgpr_ab(i32 %a, i32 %b, i32 inreg %c) { define amdgpu_ps float @shl_add_vgpr_ab(i32 %a, i32 %b, i32 inreg %c) {
; VI-LABEL: shl_add_vgpr_ab: ; VI-LABEL: shl_add_vgpr_ab:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_lshlrev_b32_e32 v0, v1, v0 ; VI-NEXT: v_lshlrev_b32_e32 v0, v1, v0
; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v0 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: shl_add_vgpr_ab: ; GFX9-LABEL: shl_add_vgpr_ab:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_lshl_add_u32 v0, v0, v1, s2 ; GFX9-NEXT: v_lshl_add_u32 v0, v0, v1, s2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: shl_add_vgpr_ab: ; GFX10-LABEL: shl_add_vgpr_ab:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_add_u32 v0, v0, v1, s2 ; GFX10-NEXT: v_lshl_add_u32 v0, v0, v1, s2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = shl i32 %a, %b %x = shl i32 %a, %b
%result = add i32 %x, %c %result = add i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @shl_add_vgpr_const(i32 %a, i32 %b) { define amdgpu_ps float @shl_add_vgpr_const(i32 %a, i32 %b) {
; VI-LABEL: shl_add_vgpr_const: ; VI-LABEL: shl_add_vgpr_const:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_lshlrev_b32_e32 v0, 3, v0 ; VI-NEXT: v_lshlrev_b32_e32 v0, 3, v0
; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1 ; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: shl_add_vgpr_const: ; GFX9-LABEL: shl_add_vgpr_const:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_lshl_add_u32 v0, v0, 3, v1 ; GFX9-NEXT: v_lshl_add_u32 v0, v0, 3, v1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: shl_add_vgpr_const: ; GFX10-LABEL: shl_add_vgpr_const:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_add_u32 v0, v0, 3, v1 ; GFX10-NEXT: v_lshl_add_u32 v0, v0, 3, v1
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = shl i32 %a, 3 %x = shl i32 %a, 3
%result = add i32 %x, %b %result = add i32 %x, %b
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }

test/CodeGen/AMDGPU/shl_or.ll

Show All 16 Lines
; GFX9-LABEL: shl_or: ; GFX9-LABEL: shl_or:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_lshl_or_b32 v0, v0, v1, v2 ; GFX9-NEXT: v_lshl_or_b32 v0, v0, v1, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: shl_or: ; GFX10-LABEL: shl_or:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_or_b32 v0, v0, v1, v2 ; GFX10-NEXT: v_lshl_or_b32 v0, v0, v1, v2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = shl i32 %a, %b %x = shl i32 %a, %b
%result = or i32 %x, %c %result = or i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @shl_or_vgpr_c(i32 inreg %a, i32 inreg %b, i32 %c) { define amdgpu_ps float @shl_or_vgpr_c(i32 inreg %a, i32 inreg %b, i32 %c) {
; VI-LABEL: shl_or_vgpr_c: ; VI-LABEL: shl_or_vgpr_c:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: s_lshl_b32 s0, s2, s3 ; VI-NEXT: s_lshl_b32 s0, s2, s3
; VI-NEXT: v_or_b32_e32 v0, s0, v0 ; VI-NEXT: v_or_b32_e32 v0, s0, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: shl_or_vgpr_c: ; GFX9-LABEL: shl_or_vgpr_c:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: s_lshl_b32 s0, s2, s3 ; GFX9-NEXT: s_lshl_b32 s0, s2, s3
; GFX9-NEXT: v_or_b32_e32 v0, s0, v0 ; GFX9-NEXT: v_or_b32_e32 v0, s0, v0
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: shl_or_vgpr_c: ; GFX10-LABEL: shl_or_vgpr_c:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_or_b32 v0, s2, s3, v0 ; GFX10-NEXT: v_lshl_or_b32 v0, s2, s3, v0
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = shl i32 %a, %b %x = shl i32 %a, %b
%result = or i32 %x, %c %result = or i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @shl_or_vgpr_all2(i32 %a, i32 %b, i32 %c) { define amdgpu_ps float @shl_or_vgpr_all2(i32 %a, i32 %b, i32 %c) {
; VI-LABEL: shl_or_vgpr_all2: ; VI-LABEL: shl_or_vgpr_all2:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_lshlrev_b32_e32 v0, v1, v0 ; VI-NEXT: v_lshlrev_b32_e32 v0, v1, v0
; VI-NEXT: v_or_b32_e32 v0, v2, v0 ; VI-NEXT: v_or_b32_e32 v0, v2, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: shl_or_vgpr_all2: ; GFX9-LABEL: shl_or_vgpr_all2:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_lshl_or_b32 v0, v0, v1, v2 ; GFX9-NEXT: v_lshl_or_b32 v0, v0, v1, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: shl_or_vgpr_all2: ; GFX10-LABEL: shl_or_vgpr_all2:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_or_b32 v0, v0, v1, v2 ; GFX10-NEXT: v_lshl_or_b32 v0, v0, v1, v2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = shl i32 %a, %b %x = shl i32 %a, %b
%result = or i32 %c, %x %result = or i32 %c, %x
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @shl_or_vgpr_ac(i32 %a, i32 inreg %b, i32 %c) { define amdgpu_ps float @shl_or_vgpr_ac(i32 %a, i32 inreg %b, i32 %c) {
; VI-LABEL: shl_or_vgpr_ac: ; VI-LABEL: shl_or_vgpr_ac:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_lshlrev_b32_e32 v0, s2, v0 ; VI-NEXT: v_lshlrev_b32_e32 v0, s2, v0
; VI-NEXT: v_or_b32_e32 v0, v0, v1 ; VI-NEXT: v_or_b32_e32 v0, v0, v1
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: shl_or_vgpr_ac: ; GFX9-LABEL: shl_or_vgpr_ac:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_lshl_or_b32 v0, v0, s2, v1 ; GFX9-NEXT: v_lshl_or_b32 v0, v0, s2, v1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: shl_or_vgpr_ac: ; GFX10-LABEL: shl_or_vgpr_ac:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_or_b32 v0, v0, s2, v1 ; GFX10-NEXT: v_lshl_or_b32 v0, v0, s2, v1
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = shl i32 %a, %b %x = shl i32 %a, %b
%result = or i32 %x, %c %result = or i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @shl_or_vgpr_const(i32 %a, i32 %b) { define amdgpu_ps float @shl_or_vgpr_const(i32 %a, i32 %b) {
; VI-LABEL: shl_or_vgpr_const: ; VI-LABEL: shl_or_vgpr_const:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_lshlrev_b32_e32 v0, v1, v0 ; VI-NEXT: v_lshlrev_b32_e32 v0, v1, v0
; VI-NEXT: v_or_b32_e32 v0, 6, v0 ; VI-NEXT: v_or_b32_e32 v0, 6, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: shl_or_vgpr_const: ; GFX9-LABEL: shl_or_vgpr_const:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_lshl_or_b32 v0, v0, v1, 6 ; GFX9-NEXT: v_lshl_or_b32 v0, v0, v1, 6
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: shl_or_vgpr_const: ; GFX10-LABEL: shl_or_vgpr_const:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_or_b32 v0, v0, v1, 6 ; GFX10-NEXT: v_lshl_or_b32 v0, v0, v1, 6
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = shl i32 %a, %b %x = shl i32 %a, %b
%result = or i32 %x, 6 %result = or i32 %x, 6
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @shl_or_vgpr_const2(i32 %a, i32 %b) { define amdgpu_ps float @shl_or_vgpr_const2(i32 %a, i32 %b) {
; VI-LABEL: shl_or_vgpr_const2: ; VI-LABEL: shl_or_vgpr_const2:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_lshlrev_b32_e32 v0, 6, v0 ; VI-NEXT: v_lshlrev_b32_e32 v0, 6, v0
; VI-NEXT: v_or_b32_e32 v0, v0, v1 ; VI-NEXT: v_or_b32_e32 v0, v0, v1
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: shl_or_vgpr_const2: ; GFX9-LABEL: shl_or_vgpr_const2:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_lshl_or_b32 v0, v0, 6, v1 ; GFX9-NEXT: v_lshl_or_b32 v0, v0, 6, v1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: shl_or_vgpr_const2: ; GFX10-LABEL: shl_or_vgpr_const2:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_or_b32 v0, v0, 6, v1 ; GFX10-NEXT: v_lshl_or_b32 v0, v0, 6, v1
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = shl i32 %a, 6 %x = shl i32 %a, 6
%result = or i32 %x, %b %result = or i32 %x, %b
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @shl_or_vgpr_const_scalar1(i32 inreg %a, i32 %b) { define amdgpu_ps float @shl_or_vgpr_const_scalar1(i32 inreg %a, i32 %b) {
; VI-LABEL: shl_or_vgpr_const_scalar1: ; VI-LABEL: shl_or_vgpr_const_scalar1:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: s_lshl_b32 s0, s2, 6 ; VI-NEXT: s_lshl_b32 s0, s2, 6
; VI-NEXT: v_or_b32_e32 v0, s0, v0 ; VI-NEXT: v_or_b32_e32 v0, s0, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: shl_or_vgpr_const_scalar1: ; GFX9-LABEL: shl_or_vgpr_const_scalar1:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_lshl_or_b32 v0, s2, 6, v0 ; GFX9-NEXT: v_lshl_or_b32 v0, s2, 6, v0
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: shl_or_vgpr_const_scalar1: ; GFX10-LABEL: shl_or_vgpr_const_scalar1:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_or_b32 v0, s2, 6, v0 ; GFX10-NEXT: v_lshl_or_b32 v0, s2, 6, v0
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = shl i32 %a, 6 %x = shl i32 %a, 6
%result = or i32 %x, %b %result = or i32 %x, %b
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @shl_or_vgpr_const_scalar2(i32 %a, i32 inreg %b) { define amdgpu_ps float @shl_or_vgpr_const_scalar2(i32 %a, i32 inreg %b) {
; VI-LABEL: shl_or_vgpr_const_scalar2: ; VI-LABEL: shl_or_vgpr_const_scalar2:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_lshlrev_b32_e32 v0, 6, v0 ; VI-NEXT: v_lshlrev_b32_e32 v0, 6, v0
; VI-NEXT: v_or_b32_e32 v0, s2, v0 ; VI-NEXT: v_or_b32_e32 v0, s2, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: shl_or_vgpr_const_scalar2: ; GFX9-LABEL: shl_or_vgpr_const_scalar2:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_lshl_or_b32 v0, v0, 6, s2 ; GFX9-NEXT: v_lshl_or_b32 v0, v0, 6, s2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: shl_or_vgpr_const_scalar2: ; GFX10-LABEL: shl_or_vgpr_const_scalar2:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_lshl_or_b32 v0, v0, 6, s2 ; GFX10-NEXT: v_lshl_or_b32 v0, v0, 6, s2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = shl i32 %a, 6 %x = shl i32 %a, 6
%result = or i32 %x, %b %result = or i32 %x, %b
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }

test/CodeGen/AMDGPU/si-annotate-cf-unreachable.ll

; RUN: opt -mtriple=amdgcn-- -S -structurizecfg -si-annotate-control-flow %s | FileCheck -check-prefix=OPT %s ; RUN: opt -mtriple=amdgcn-- -S -structurizecfg -si-annotate-control-flow %s | FileCheck -check-prefix=OPT %s
; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s ; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
; OPT-LABEL: @annotate_unreachable( ; OPT-LABEL: @annotate_unreachable(
; OPT: call { i1, i64 } @llvm.amdgcn.if( ; OPT: call { i1, i64 } @llvm.amdgcn.if.i64(
; OPT-NOT: call void @llvm.amdgcn.end.cf( ; OPT-NOT: call void @llvm.amdgcn.end.cf
; GCN-LABEL: {{^}}annotate_unreachable: ; GCN-LABEL: {{^}}annotate_unreachable:
; GCN: s_and_saveexec_b64 ; GCN: s_and_saveexec_b64
; GCN-NOT: s_endpgm ; GCN-NOT: s_endpgm
; GCN: .Lfunc_end0 ; GCN: .Lfunc_end0
define amdgpu_kernel void @annotate_unreachable(<4 x float> addrspace(1)* noalias nocapture readonly %arg) #0 { define amdgpu_kernel void @annotate_unreachable(<4 x float> addrspace(1)* noalias nocapture readonly %arg) #0 {
bb: bb:
Show All 25 Lines

test/CodeGen/AMDGPU/si-annotatecfg-multiple-backedges.ll

Show All 11 Lines
; OPT-NEXT: [[TMP2:%.*]] = shl nsw i32 [[ARG:%.*]], 1 ; OPT-NEXT: [[TMP2:%.*]] = shl nsw i32 [[ARG:%.*]], 1
; OPT-NEXT: br label [[LOOP:%.*]] ; OPT-NEXT: br label [[LOOP:%.*]]
; OPT: loop: ; OPT: loop:
; OPT-NEXT: [[PHI_BROKEN1:%.*]] = phi i64 [ [[TMP7:%.*]], [[LOOP_END:%.*]] ], [ [[PHI_BROKEN1]], [[LOOP]] ], [ 0, [[ENTRY:%.*]] ] ; OPT-NEXT: [[PHI_BROKEN1:%.*]] = phi i64 [ [[TMP7:%.*]], [[LOOP_END:%.*]] ], [ [[PHI_BROKEN1]], [[LOOP]] ], [ 0, [[ENTRY:%.*]] ]
; OPT-NEXT: [[PHI_BROKEN:%.*]] = phi i64 [ 0, [[LOOP_END]] ], [ [[TMP0:%.*]], [[LOOP]] ], [ 0, [[ENTRY]] ] ; OPT-NEXT: [[PHI_BROKEN:%.*]] = phi i64 [ 0, [[LOOP_END]] ], [ [[TMP0:%.*]], [[LOOP]] ], [ 0, [[ENTRY]] ]
; OPT-NEXT: [[TMP4:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[TMP5:%.*]], [[LOOP]] ], [ 0, [[LOOP_END]] ] ; OPT-NEXT: [[TMP4:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[TMP5:%.*]], [[LOOP]] ], [ 0, [[LOOP_END]] ]
; OPT-NEXT: [[TMP5]] = add nsw i32 [[TMP4]], [[TMP]] ; OPT-NEXT: [[TMP5]] = add nsw i32 [[TMP4]], [[TMP]]
; OPT-NEXT: [[TMP6:%.*]] = icmp slt i32 [[ARG]], [[TMP5]] ; OPT-NEXT: [[TMP6:%.*]] = icmp slt i32 [[ARG]], [[TMP5]]
; OPT-NEXT: [[TMP0]] = call i64 @llvm.amdgcn.if.break(i1 [[TMP6]], i64 [[PHI_BROKEN]]) ; OPT-NEXT: [[TMP0]] = call i64 @llvm.amdgcn.if.break.i64.i64(i1 [[TMP6]], i64 [[PHI_BROKEN]])
; OPT-NEXT: [[TMP1:%.*]] = call i1 @llvm.amdgcn.loop(i64 [[TMP0]]) ; OPT-NEXT: [[TMP1:%.*]] = call i1 @llvm.amdgcn.loop.i64(i64 [[TMP0]])
; OPT-NEXT: br i1 [[TMP1]], label [[LOOP_END]], label [[LOOP]] ; OPT-NEXT: br i1 [[TMP1]], label [[LOOP_END]], label [[LOOP]]
; OPT: loop_end: ; OPT: loop_end:
; OPT-NEXT: call void @llvm.amdgcn.end.cf(i64 [[TMP0]]) ; OPT-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 [[TMP0]])
; OPT-NEXT: [[EXIT:%.*]] = icmp sgt i32 [[TMP5]], [[TMP2]] ; OPT-NEXT: [[EXIT:%.*]] = icmp sgt i32 [[TMP5]], [[TMP2]]
; OPT-NEXT: [[TMP7]] = call i64 @llvm.amdgcn.if.break(i1 [[EXIT]], i64 [[PHI_BROKEN1]]) ; OPT-NEXT: [[TMP7]] = call i64 @llvm.amdgcn.if.break.i64.i64(i1 [[EXIT]], i64 [[PHI_BROKEN1]])
; OPT-NEXT: [[TMP3:%.*]] = call i1 @llvm.amdgcn.loop(i64 [[TMP7]]) ; OPT-NEXT: [[TMP3:%.*]] = call i1 @llvm.amdgcn.loop.i64(i64 [[TMP7]])
; OPT-NEXT: br i1 [[TMP3]], label [[LOOP_EXIT:%.*]], label [[LOOP]] ; OPT-NEXT: br i1 [[TMP3]], label [[LOOP_EXIT:%.*]], label [[LOOP]]
; OPT: loop_exit: ; OPT: loop_exit:
; OPT-NEXT: call void @llvm.amdgcn.end.cf(i64 [[TMP7]]) ; OPT-NEXT: call void @llvm.amdgcn.end.cf.i64(i64 [[TMP7]])
; OPT-NEXT: [[TMP12:%.*]] = zext i32 [[TMP]] to i64 ; OPT-NEXT: [[TMP12:%.*]] = zext i32 [[TMP]] to i64
; OPT-NEXT: [[TMP13:%.*]] = getelementptr inbounds i32, i32* [[ARG1:%.*]], i64 [[TMP12]] ; OPT-NEXT: [[TMP13:%.*]] = getelementptr inbounds i32, i32* [[ARG1:%.*]], i64 [[TMP12]]
; OPT-NEXT: [[TMP14:%.*]] = addrspacecast i32* [[TMP13]] to i32 addrspace(1)* ; OPT-NEXT: [[TMP14:%.*]] = addrspacecast i32* [[TMP13]] to i32 addrspace(1)*
; OPT-NEXT: store i32 [[TMP5]], i32 addrspace(1)* [[TMP14]], align 4 ; OPT-NEXT: store i32 [[TMP5]], i32 addrspace(1)* [[TMP14]], align 4
; OPT-NEXT: ret void ; OPT-NEXT: ret void
; ;
entry: entry:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x() %tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
Show All 23 Lines

test/CodeGen/AMDGPU/smem-war-hazard.mir

# RUN: llc -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs -run-pass post-RA-hazard-rec -o - %s | FileCheck -check-prefix=GCN %s # RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-WavefrontSize32,+WavefrontSize64 -verify-machineinstrs -run-pass post-RA-hazard-rec -o - %s | FileCheck -check-prefix=GCN %s
# GCN-LABEL: name: hazard_smem_war # GCN-LABEL: name: hazard_smem_war
# GCN: S_LOAD_DWORD_IMM # GCN: S_LOAD_DWORD_IMM
# GCN: $sgpr_null = S_MOV_B32 0 # GCN: $sgpr_null = S_MOV_B32 0
# GCN-NEXT: V_CMP_EQ_F32 # GCN-NEXT: V_CMP_EQ_F32
--- ---
name: hazard_smem_war name: hazard_smem_war
body: | body: |
▲ Show 20 Lines • Show All 310 Lines • Show Last 20 Lines

test/CodeGen/AMDGPU/sub_i1.ll

; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s ; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,WAVE64 %s
; RUN: llc -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,WAVE32 %s
; GCN-LABEL: {{^}}sub_var_var_i1: ; GCN-LABEL: {{^}}sub_var_var_i1:
; GCN: s_xor_b64 ; WAVE32: s_xor_b32
; WAVE64: s_xor_b64
define amdgpu_kernel void @sub_var_var_i1(i1 addrspace(1)* %out, i1 addrspace(1)* %in0, i1 addrspace(1)* %in1) { define amdgpu_kernel void @sub_var_var_i1(i1 addrspace(1)* %out, i1 addrspace(1)* %in0, i1 addrspace(1)* %in1) {
%a = load volatile i1, i1 addrspace(1)* %in0 %a = load volatile i1, i1 addrspace(1)* %in0
%b = load volatile i1, i1 addrspace(1)* %in1 %b = load volatile i1, i1 addrspace(1)* %in1
%sub = sub i1 %a, %b %sub = sub i1 %a, %b
store i1 %sub, i1 addrspace(1)* %out store i1 %sub, i1 addrspace(1)* %out
ret void ret void
} }
; GCN-LABEL: {{^}}sub_var_imm_i1: ; GCN-LABEL: {{^}}sub_var_imm_i1:
; GCN: s_not_b64 ; WAVE32: s_not_b32
; WAVE64: s_not_b64
define amdgpu_kernel void @sub_var_imm_i1(i1 addrspace(1)* %out, i1 addrspace(1)* %in) { define amdgpu_kernel void @sub_var_imm_i1(i1 addrspace(1)* %out, i1 addrspace(1)* %in) {
%a = load volatile i1, i1 addrspace(1)* %in %a = load volatile i1, i1 addrspace(1)* %in
%sub = sub i1 %a, 1 %sub = sub i1 %a, 1
store i1 %sub, i1 addrspace(1)* %out store i1 %sub, i1 addrspace(1)* %out
ret void ret void
} }
; GCN-LABEL: {{^}}sub_i1_cf: ; GCN-LABEL: {{^}}sub_i1_cf:
; GCN: ; %endif ; GCN: ; %endif
; GCN: s_not_b64 ; WAVE32: s_not_b32
; WAVE64: s_not_b64
define amdgpu_kernel void @sub_i1_cf(i1 addrspace(1)* %out, i1 addrspace(1)* %a, i1 addrspace(1)* %b) { define amdgpu_kernel void @sub_i1_cf(i1 addrspace(1)* %out, i1 addrspace(1)* %a, i1 addrspace(1)* %b) {
entry: entry:
%tid = call i32 @llvm.amdgcn.workitem.id.x() %tid = call i32 @llvm.amdgcn.workitem.id.x()
%d_cmp = icmp ult i32 %tid, 16 %d_cmp = icmp ult i32 %tid, 16
br i1 %d_cmp, label %if, label %else br i1 %d_cmp, label %if, label %else
if: if:
%0 = load volatile i1, i1 addrspace(1)* %a %0 = load volatile i1, i1 addrspace(1)* %a
Show All 14 Lines

test/CodeGen/AMDGPU/v_cndmask.ll

; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,SI,SIVI %s ; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,SI,SIVI %s
; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,VI,SIVI %s ; RUN: llc -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,VI,SIVI %s
; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,GFX10 %s ; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-flat-for-global,+WavefrontSize64 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,GFX10 %s
declare i32 @llvm.amdgcn.workitem.id.x() #1 declare i32 @llvm.amdgcn.workitem.id.x() #1
; GCN-LABEL: {{^}}v_cnd_nan_nosgpr: ; GCN-LABEL: {{^}}v_cnd_nan_nosgpr:
; GCN: v_cmp_eq_u32_e64 [[COND:vcc|s\[[0-9]+:[0-9]+\]]], s{{[0-9]+}}, 0 ; GCN: v_cmp_eq_u32_e64 [[COND:vcc|s\[[0-9]+:[0-9]+\]]], s{{[0-9]+}}, 0
; GCN: v_cndmask_b32_e{{32|64}} v{{[0-9]}}, -1, v{{[0-9]+}}, [[COND]] ; GCN: v_cndmask_b32_e{{32|64}} v{{[0-9]}}, -1, v{{[0-9]+}}, [[COND]]
; GCN-DAG: v{{[0-9]}} ; GCN-DAG: v{{[0-9]}}
; All nan values are converted to 0xffffffff ; All nan values are converted to 0xffffffff
▲ Show 20 Lines • Show All 409 Lines • Show Last 20 Lines

test/CodeGen/AMDGPU/vcmpx-exec-war-hazard.mir

# RUN: llc -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs -run-pass si-insert-skips,post-RA-hazard-rec -o - %s | FileCheck -check-prefix=GCN %s # RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-WavefrontSize32,+WavefrontSize64 -verify-machineinstrs -run-pass si-insert-skips,post-RA-hazard-rec -o - %s | FileCheck -check-prefix=GCN %s
# GCN-LABEL: name: hazard_vcmpx_smov_exec_lo # GCN-LABEL: name: hazard_vcmpx_smov_exec_lo
# GCN: $sgpr0 = S_MOV_B32 $exec_lo # GCN: $sgpr0 = S_MOV_B32 $exec_lo
# GCN-NEXT: S_WAITCNT_DEPCTR 65534 # GCN-NEXT: S_WAITCNT_DEPCTR 65534
# GCN-NEXT: V_CMPX_LE_F32_nosdst_e32 # GCN-NEXT: V_CMPX_LE_F32_nosdst_e32
--- ---
name: hazard_vcmpx_smov_exec_lo name: hazard_vcmpx_smov_exec_lo
body: | body: |
▲ Show 20 Lines • Show All 155 Lines • Show Last 20 Lines

test/CodeGen/AMDGPU/wave32.ll

  • This file was added.
; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX1032 %s
; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX1064 %s
; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -amdgpu-early-ifcvt=1 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX1032 %s
; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -amdgpu-early-ifcvt=1 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX1064 %s
; RUN: llc -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX1032,GFX10DEFWAVE %s
; GCN-LABEL: {{^}}test_vopc_i32:
; GFX1032: v_cmp_lt_i32_e32 vcc_lo, 0, v{{[0-9]+}}
; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 2, 1, vcc_lo
; GFX1064: v_cmp_lt_i32_e32 vcc, 0, v{{[0-9]+}}
; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 2, 1, vcc{{$}}
define amdgpu_kernel void @test_vopc_i32(i32 addrspace(1)* %arg) {
%lid = tail call i32 @llvm.amdgcn.workitem.id.x()
%gep = getelementptr inbounds i32, i32 addrspace(1)* %arg, i32 %lid
%load = load i32, i32 addrspace(1)* %gep, align 4
%cmp = icmp sgt i32 %load, 0
%sel = select i1 %cmp, i32 1, i32 2
store i32 %sel, i32 addrspace(1)* %gep, align 4
ret void
}
; GCN-LABEL: {{^}}test_vopc_f32:
; GFX1032: v_cmp_nge_f32_e32 vcc_lo, 0, v{{[0-9]+}}
; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 2.0, 1.0, vcc_lo
; GFX1064: v_cmp_nge_f32_e32 vcc, 0, v{{[0-9]+}}
; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 2.0, 1.0, vcc{{$}}
define amdgpu_kernel void @test_vopc_f32(float addrspace(1)* %arg) {
%lid = tail call i32 @llvm.amdgcn.workitem.id.x()
%gep = getelementptr inbounds float, float addrspace(1)* %arg, i32 %lid
%load = load float, float addrspace(1)* %gep, align 4
%cmp = fcmp ugt float %load, 0.0
%sel = select i1 %cmp, float 1.0, float 2.0
store float %sel, float addrspace(1)* %gep, align 4
ret void
}
; GCN-LABEL: {{^}}test_vopc_vcmpx:
; GFX1032: v_cmpx_le_f32_e32 0, v{{[0-9]+}}
; GFX1064: v_cmpx_le_f32_e32 0, v{{[0-9]+}}
define amdgpu_ps void @test_vopc_vcmpx(float %x) {
%cmp = fcmp oge float %x, 0.0
call void @llvm.amdgcn.kill(i1 %cmp)
ret void
}
; GCN-LABEL: {{^}}test_vopc_2xf16:
; GFX1032: v_cmp_le_f16_sdwa [[SC:s[0-9]+]], {{[vs][0-9]+}}, v{{[0-9]+}} src0_sel:WORD_1 src1_sel:DWORD
; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 0x3c003c00, v{{[0-9]+}}, [[SC]]
; GFX1064: v_cmp_le_f16_sdwa [[SC:s\[[0-9:]+\]]], {{[vs][0-9]+}}, v{{[0-9]+}} src0_sel:WORD_1 src1_sel:DWORD
; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 0x3c003c00, v{{[0-9]+}}, [[SC]]
define amdgpu_kernel void @test_vopc_2xf16(<2 x half> addrspace(1)* %arg) {
%lid = tail call i32 @llvm.amdgcn.workitem.id.x()
%gep = getelementptr inbounds <2 x half>, <2 x half> addrspace(1)* %arg, i32 %lid
%load = load <2 x half>, <2 x half> addrspace(1)* %gep, align 4
%elt = extractelement <2 x half> %load, i32 1
%cmp = fcmp ugt half %elt, 0.0
%sel = select i1 %cmp, <2 x half> <half 1.0, half 1.0>, <2 x half> %load
store <2 x half> %sel, <2 x half> addrspace(1)* %gep, align 4
ret void
}
; GCN-LABEL: {{^}}test_vopc_class:
; GFX1032: v_cmp_class_f32_e64 [[C:vcc_lo|s[0-9:]+]], s{{[0-9]+}}, 0x204
; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 0, 1, [[C]]
; GFX1064: v_cmp_class_f32_e64 [[C:vcc|s\[[0-9:]+\]]], s{{[0-9]+}}, 0x204
; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 0, 1, [[C]]{{$}}
define amdgpu_kernel void @test_vopc_class(i32 addrspace(1)* %out, float %x) #0 {
%fabs = tail call float @llvm.fabs.f32(float %x)
%cmp = fcmp oeq float %fabs, 0x7FF0000000000000
%ext = zext i1 %cmp to i32
store i32 %ext, i32 addrspace(1)* %out, align 4
ret void
}
; GCN-LABEL: {{^}}test_vcmp_vcnd_f16:
; GFX1032: v_cmp_neq_f16_e64 [[C:vcc_lo|s\[[0-9:]+\]]], 0x7c00, s{{[0-9]+}}
; GFX1032: v_cndmask_b32_e32 v{{[0-9]+}}, 0x3c00, v{{[0-9]+}}, [[C]]
; GFX1064: v_cmp_neq_f16_e64 [[C:vcc|s\[[0-9:]+\]]], 0x7c00, s{{[0-9]+}}
; GFX1064: v_cndmask_b32_e32 v{{[0-9]+}}, 0x3c00, v{{[0-9]+}}, [[C]]{{$}}
define amdgpu_kernel void @test_vcmp_vcnd_f16(half addrspace(1)* %out, half %x) #0 {
%cmp = fcmp oeq half %x, 0x7FF0000000000000
%sel = select i1 %cmp, half 1.0, half %x
store half %sel, half addrspace(1)* %out, align 2
ret void
}
; GCN-LABEL: {{^}}test_vop3_cmp_f32_sop_and:
; GFX1032: v_cmp_nge_f32_e32 vcc_lo, 0, v{{[0-9]+}}
; GFX1032: v_cmp_nle_f32_e64 [[C2:s[0-9]+]], 1.0, v{{[0-9]+}}
; GFX1032: s_and_b32 [[AND:s[0-9]+]], vcc_lo, [[C2]]
; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 2.0, 1.0, [[AND]]
; GFX1064: v_cmp_nge_f32_e32 vcc, 0, v{{[0-9]+}}
; GFX1064: v_cmp_nle_f32_e64 [[C2:s\[[0-9:]+\]]], 1.0, v{{[0-9]+}}
; GFX1064: s_and_b64 [[AND:s\[[0-9:]+\]]], vcc, [[C2]]
; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 2.0, 1.0, [[AND]]
define amdgpu_kernel void @test_vop3_cmp_f32_sop_and(float addrspace(1)* %arg) {
%lid = tail call i32 @llvm.amdgcn.workitem.id.x()
%gep = getelementptr inbounds float, float addrspace(1)* %arg, i32 %lid
%load = load float, float addrspace(1)* %gep, align 4
%cmp = fcmp ugt float %load, 0.0
%cmp2 = fcmp ult float %load, 1.0
%and = and i1 %cmp, %cmp2
%sel = select i1 %and, float 1.0, float 2.0
store float %sel, float addrspace(1)* %gep, align 4
ret void
}
; GCN-LABEL: {{^}}test_vop3_cmp_i32_sop_xor:
; GFX1032: v_cmp_lt_i32_e32 vcc_lo, 0, v{{[0-9]+}}
; GFX1032: v_cmp_gt_i32_e64 [[C2:s[0-9]+]], 1, v{{[0-9]+}}
; GFX1032: s_xor_b32 [[AND:s[0-9]+]], vcc_lo, [[C2]]
; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 2, 1, [[AND]]
; GFX1064: v_cmp_lt_i32_e32 vcc, 0, v{{[0-9]+}}
; GFX1064: v_cmp_gt_i32_e64 [[C2:s\[[0-9:]+\]]], 1, v{{[0-9]+}}
; GFX1064: s_xor_b64 [[AND:s\[[0-9:]+\]]], vcc, [[C2]]
; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 2, 1, [[AND]]
define amdgpu_kernel void @test_vop3_cmp_i32_sop_xor(i32 addrspace(1)* %arg) {
%lid = tail call i32 @llvm.amdgcn.workitem.id.x()
%gep = getelementptr inbounds i32, i32 addrspace(1)* %arg, i32 %lid
%load = load i32, i32 addrspace(1)* %gep, align 4
%cmp = icmp sgt i32 %load, 0
%cmp2 = icmp slt i32 %load, 1
%xor = xor i1 %cmp, %cmp2
%sel = select i1 %xor, i32 1, i32 2
store i32 %sel, i32 addrspace(1)* %gep, align 4
ret void
}
; GCN-LABEL: {{^}}test_vop3_cmp_u32_sop_or:
; GFX1032: v_cmp_lt_u32_e32 vcc_lo, 3, v{{[0-9]+}}
; GFX1032: v_cmp_gt_u32_e64 [[C2:s[0-9]+]], 2, v{{[0-9]+}}
; GFX1032: s_or_b32 [[AND:s[0-9]+]], vcc_lo, [[C2]]
; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 2, 1, [[AND]]
; GFX1064: v_cmp_lt_u32_e32 vcc, 3, v{{[0-9]+}}
; GFX1064: v_cmp_gt_u32_e64 [[C2:s\[[0-9:]+\]]], 2, v{{[0-9]+}}
; GFX1064: s_or_b64 [[AND:s\[[0-9:]+\]]], vcc, [[C2]]
; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 2, 1, [[AND]]
define amdgpu_kernel void @test_vop3_cmp_u32_sop_or(i32 addrspace(1)* %arg) {
%lid = tail call i32 @llvm.amdgcn.workitem.id.x()
%gep = getelementptr inbounds i32, i32 addrspace(1)* %arg, i32 %lid
%load = load i32, i32 addrspace(1)* %gep, align 4
%cmp = icmp ugt i32 %load, 3
%cmp2 = icmp ult i32 %load, 2
%or = or i1 %cmp, %cmp2
%sel = select i1 %or, i32 1, i32 2
store i32 %sel, i32 addrspace(1)* %gep, align 4
ret void
}
; GCN-LABEL: {{^}}test_mask_if:
; GFX1032: s_and_saveexec_b32 s{{[0-9]+}}, vcc_lo
; GFX1064: s_and_saveexec_b64 s[{{[0-9:]+}}], vcc{{$}}
; GCN: ; mask branch
define amdgpu_kernel void @test_mask_if(i32 addrspace(1)* %arg) #0 {
%lid = tail call i32 @llvm.amdgcn.workitem.id.x()
%cmp = icmp ugt i32 %lid, 10
br i1 %cmp, label %if, label %endif
if:
store i32 0, i32 addrspace(1)* %arg, align 4
br label %endif
endif:
ret void
}
; GCN-LABEL: {{^}}test_loop_with_if:
; GFX1032: s_and_saveexec_b32 s{{[0-9]+}}, vcc_lo
; GFX1064: s_and_saveexec_b64 s[{{[0-9:]+}}], vcc{{$}}
; GCN: ; mask branch BB10_3
; GCN: BB{{.*}}:
; GCN: BB{{.*}}:
; GFX1032: s_or_b32 exec_lo, exec_lo, s{{[0-9]+}}
; GFX1032: s_and_saveexec_b32 s{{[0-9]+}}, s{{[0-9]+}}
; GFX1032: s_xor_b32 s{{[0-9]+}}, exec_lo, s{{[0-9]+}}
; GFX1064: s_or_b64 exec, exec, s[{{[0-9:]+}}]
; GFX1064: s_and_saveexec_b64 s[{{[0-9:]+}}], s[{{[0-9:]+}}]{{$}}
; GFX1064: s_xor_b64 s[{{[0-9:]+}}], exec, s[{{[0-9:]+}}]
; GCN: ; mask branch BB
; GCN: BB{{.*}}:
; GCN: BB{{.*}}:
; GFX1032: s_or_b32 exec_lo, exec_lo, s{{[0-9]+}}
; GFX1032: s_and_saveexec_b32 s{{[0-9]+}}, s{{[0-9]+}}
; GFX1064: s_or_b64 exec, exec, s[{{[0-9:]+}}]
; GFX1064: s_and_saveexec_b64 s[{{[0-9:]+}}], s[{{[0-9:]+}}]{{$}}
; GCN: ; mask branch BB
; GCN: BB{{.*}}:
; GCN: BB{{.*}}:
; GFX1032: s_or_b32 exec_lo, exec_lo, s{{[0-9]+}}
; GFX1032: s_or_b32 s{{[0-9]+}}, vcc_lo, s{{[0-9]+}}
; GFX1032: s_andn2_b32 exec_lo, exec_lo, s{{[0-9]+}}
; GFX1064: s_or_b64 exec, exec, s[{{[0-9:]+}}]
; GFX1064: s_or_b64 s[{{[0-9:]+}}], vcc, s[{{[0-9:]+}}]
; GFX1064: s_andn2_b64 exec, exec, s[{{[0-9:]+}}]
; GCN: s_cbranch_execnz
define amdgpu_kernel void @test_loop_with_if(i32 addrspace(1)* %arg) #0 {
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
br label %bb2
bb1:
ret void
bb2:
%tmp3 = phi i32 [ 0, %bb ], [ %tmp15, %bb13 ]
%tmp4 = icmp slt i32 %tmp3, %tmp
br i1 %tmp4, label %bb5, label %bb11
bb5:
%tmp6 = sext i32 %tmp3 to i64
%tmp7 = getelementptr inbounds i32, i32 addrspace(1)* %arg, i64 %tmp6
%tmp8 = load i32, i32 addrspace(1)* %tmp7, align 4
%tmp9 = icmp sgt i32 %tmp8, 10
br i1 %tmp9, label %bb10, label %bb11
bb10:
store i32 %tmp, i32 addrspace(1)* %tmp7, align 4
br label %bb13
bb11:
%tmp12 = sdiv i32 %tmp3, 2
br label %bb13
bb13:
%tmp14 = phi i32 [ %tmp3, %bb10 ], [ %tmp12, %bb11 ]
%tmp15 = add nsw i32 %tmp14, 1
%tmp16 = icmp slt i32 %tmp14, 255
br i1 %tmp16, label %bb2, label %bb1
}
; GCN-LABEL: {{^}}test_loop_with_if_else_break:
; GFX1032: s_and_saveexec_b32 s{{[0-9]+}}, vcc_lo
; GFX1064: s_and_saveexec_b64 s[{{[0-9:]+}}], vcc{{$}}
; GCN: ; mask branch
; GCN: s_cbranch_execz
; GCN: BB{{.*}}:
; GCN: BB{{.*}}:
; GFX1032: s_or_b32 s{{[0-9]+}}, s{{[0-9]+}}, exec_lo
; GFX1064: s_or_b64 s[{{[0-9:]+}}], s[{{[0-9:]+}}], exec
; GCN: s_cbranch_scc1
; GCN: ; {{BB|%bb}}{{.*}}:
; GFX1032: s_or_b32 s{{[0-9]+}}, vcc_lo, s{{[0-9]+}}
; GFX1032: s_or_b32 s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
; GFX1064: s_or_b64 s[{{[0-9:]+}}], vcc, s[{{[0-9:]+}}]
; GFX1064: s_or_b64 s[{{[0-9:]+}}], s[{{[0-9:]+}}], s[{{[0-9:]+}}]
; GCN: s_branch
; GCN: BB{{.*}}:
; GFX1032: s_or_b32 s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
; GFX1032: s_mov_b32 s{{[0-9]+}}, s{{[0-9]+}}
; GFX1032: s_andn2_b32 exec_lo, exec_lo, s{{[0-9]+}}
; GFX1064: s_or_b64 s[{{[0-9:]+}}], s[{{[0-9:]+}}], s[{{[0-9:]+}}]
; GFX1064: s_mov_b64 s[{{[0-9:]+}}], s[{{[0-9:]+}}]
; GFX1064: s_andn2_b64 exec, exec, s[{{[0-9:]+}}]
; GCN: s_cbranch_execnz
; GCN: BB{{.*}}:
define amdgpu_kernel void @test_loop_with_if_else_break(i32 addrspace(1)* %arg) #0 {
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
%tmp1 = icmp eq i32 %tmp, 0
br i1 %tmp1, label %.loopexit, label %.preheader
.preheader:
br label %bb2
bb2:
%tmp3 = phi i32 [ %tmp9, %bb8 ], [ 0, %.preheader ]
%tmp4 = zext i32 %tmp3 to i64
%tmp5 = getelementptr inbounds i32, i32 addrspace(1)* %arg, i64 %tmp4
%tmp6 = load i32, i32 addrspace(1)* %tmp5, align 4
%tmp7 = icmp sgt i32 %tmp6, 10
br i1 %tmp7, label %bb8, label %.loopexit
bb8:
store i32 %tmp, i32 addrspace(1)* %tmp5, align 4
%tmp9 = add nuw nsw i32 %tmp3, 1
%tmp10 = icmp ult i32 %tmp9, 256
%tmp11 = icmp ult i32 %tmp9, %tmp
%tmp12 = and i1 %tmp10, %tmp11
br i1 %tmp12, label %bb2, label %.loopexit
.loopexit:
ret void
}
; GCN-LABEL: {{^}}test_addc_vop2b:
; GFX1032: v_add_co_u32_e64 v{{[0-9]+}}, vcc_lo, v{{[0-9]+}}, s{{[0-9]+}}
; GFX1032: v_add_co_ci_u32_e32 v{{[0-9]+}}, vcc_lo, s{{[0-9]+}}, v{{[0-9]+}}, vcc_lo
; GFX1064: v_add_co_u32_e64 v{{[0-9]+}}, vcc, v{{[0-9]+}}, s{{[0-9]+}}
; GFX1064: v_add_co_ci_u32_e32 v{{[0-9]+}}, vcc, s{{[0-9]+}}, v{{[0-9]+}}, vcc{{$}}
define amdgpu_kernel void @test_addc_vop2b(i64 addrspace(1)* %arg, i64 %arg1) #0 {
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
%tmp3 = getelementptr inbounds i64, i64 addrspace(1)* %arg, i32 %tmp
%tmp4 = load i64, i64 addrspace(1)* %tmp3, align 8
%tmp5 = add nsw i64 %tmp4, %arg1
store i64 %tmp5, i64 addrspace(1)* %tmp3, align 8
ret void
}
; GCN-LABEL: {{^}}test_subbrev_vop2b:
; GFX1032: v_sub_co_u32_e64 v{{[0-9]+}}, [[A0:s[0-9]+|vcc_lo]], v{{[0-9]+}}, s{{[0-9]+}}{{$}}
; GFX1032: v_subrev_co_ci_u32_e32 v{{[0-9]+}}, vcc_lo, {{[vs][0-9]+}}, {{[vs][0-9]+}}, [[A0]]{{$}}
; GFX1064: v_sub_co_u32_e64 v{{[0-9]+}}, [[A0:s\[[0-9:]+\]|vcc]], v{{[0-9]+}}, s{{[0-9]+}}{{$}}
; GFX1064: v_subrev_co_ci_u32_e32 v{{[0-9]+}}, vcc, {{[vs][0-9]+}}, {{[vs][0-9]+}}, [[A0]]{{$}}
define amdgpu_kernel void @test_subbrev_vop2b(i64 addrspace(1)* %arg, i64 %arg1) #0 {
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
%tmp3 = getelementptr inbounds i64, i64 addrspace(1)* %arg, i32 %tmp
%tmp4 = load i64, i64 addrspace(1)* %tmp3, align 8
%tmp5 = sub nsw i64 %tmp4, %arg1
store i64 %tmp5, i64 addrspace(1)* %tmp3, align 8
ret void
}
; GCN-LABEL: {{^}}test_subb_vop2b:
; GFX1032: v_sub_co_u32_e64 v{{[0-9]+}}, [[A0:s[0-9]+|vcc_lo]], s{{[0-9]+}}, v{{[0-9]+}}{{$}}
; GFX1032: v_sub_co_ci_u32_e32 v{{[0-9]+}}, vcc_lo, {{[vs][0-9]+}}, v{{[0-9]+}}, [[A0]]{{$}}
; GFX1064: v_sub_co_u32_e64 v{{[0-9]+}}, [[A0:s\[[0-9:]+\]|vcc]], s{{[0-9]+}}, v{{[0-9]+}}{{$}}
; GFX1064: v_sub_co_ci_u32_e32 v{{[0-9]+}}, vcc, {{[vs][0-9]+}}, v{{[0-9]+}}, [[A0]]{{$}}
define amdgpu_kernel void @test_subb_vop2b(i64 addrspace(1)* %arg, i64 %arg1) #0 {
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
%tmp3 = getelementptr inbounds i64, i64 addrspace(1)* %arg, i32 %tmp
%tmp4 = load i64, i64 addrspace(1)* %tmp3, align 8
%tmp5 = sub nsw i64 %arg1, %tmp4
store i64 %tmp5, i64 addrspace(1)* %tmp3, align 8
ret void
}
; GCN-LABEL: {{^}}test_udiv64:
; GFX1032: v_add_co_u32_e64 v{{[0-9]+}}, [[SDST:s[0-9]+]], v{{[0-9]+}}, v{{[0-9]+}}
; GFX1032: v_add_co_ci_u32_e32 v{{[0-9]+}}, vcc_lo, 0, v{{[0-9]+}}, vcc_lo
; GFX1032: v_add_co_ci_u32_e64 v{{[0-9]+}}, vcc_lo, v{{[0-9]+}}, v{{[0-9]+}}, [[SDST]]
; GFX1032: v_add_co_u32_e64 v{{[0-9]+}}, vcc_lo, v{{[0-9]+}}, v{{[0-9]+}}
; GFX1032: v_add_co_u32_e64 v{{[0-9]+}}, vcc_lo, v{{[0-9]+}}, v{{[0-9]+}}
; GFX1032: v_add_co_u32_e64 v{{[0-9]+}}, vcc_lo, v{{[0-9]+}}, v{{[0-9]+}}
; GFX1032: v_add_co_ci_u32_e32 v{{[0-9]+}}, vcc_lo, 0, v{{[0-9]+}}, vcc_lo
; GFX1032: v_sub_co_u32_e64 v{{[0-9]+}}, vcc_lo, s{{[0-9]+}}, v{{[0-9]+}}
; GFX1032: v_sub_co_ci_u32_e64 v{{[0-9]+}}, s{{[0-9]+}}, {{[vs][0-9]+}}, v{{[0-9]+}}, vcc_lo
; GFX1032: v_subrev_co_ci_u32_e32 v{{[0-9]+}}, vcc_lo, {{[vs][0-9]+}}, v{{[0-9]+}}, vcc_lo
; GFX1064: v_add_co_u32_e64 v{{[0-9]+}}, [[SDST:s\[[0-9:]+\]]], v{{[0-9]+}}, v{{[0-9]+}}
; GFX1064: v_add_co_ci_u32_e32 v{{[0-9]+}}, vcc, 0, v{{[0-9]+}}, vcc{{$}}
; GFX1064: v_add_co_ci_u32_e64 v{{[0-9]+}}, vcc, v{{[0-9]+}}, v{{[0-9]+}}, [[SDST]]
; GFX1064: v_add_co_u32_e64 v{{[0-9]+}}, vcc, v{{[0-9]+}}, v{{[0-9]+}}
; GFX1064: v_add_co_u32_e64 v{{[0-9]+}}, vcc, v{{[0-9]+}}, v{{[0-9]+}}
; GFX1064: v_add_co_u32_e64 v{{[0-9]+}}, vcc, v{{[0-9]+}}, v{{[0-9]+}}
; GFX1064: v_add_co_ci_u32_e32 v{{[0-9]+}}, vcc, 0, v{{[0-9]+}}, vcc{{$}}
; GFX1064: v_sub_co_u32_e64 v{{[0-9]+}}, vcc, s{{[0-9]+}}, v{{[0-9]+}}
; GFX1064: v_sub_co_ci_u32_e64 v{{[0-9]+}}, s[{{[0-9:]+}}], {{[vs][0-9]+}}, v{{[0-9]+}}, vcc{{$}}
; GFX1064: v_subrev_co_ci_u32_e32 v{{[0-9]+}}, vcc, {{[vs][0-9]+}}, v{{[0-9]+}}, vcc{{$}}
define amdgpu_kernel void @test_udiv64(i64 addrspace(1)* %arg) #0 {
bb:
%tmp = getelementptr inbounds i64, i64 addrspace(1)* %arg, i64 1
%tmp1 = load i64, i64 addrspace(1)* %tmp, align 8
%tmp2 = load i64, i64 addrspace(1)* %arg, align 8
%tmp3 = udiv i64 %tmp1, %tmp2
%tmp4 = getelementptr inbounds i64, i64 addrspace(1)* %arg, i64 2
store i64 %tmp3, i64 addrspace(1)* %tmp4, align 8
ret void
}
; GCN-LABEL: {{^}}test_div_scale_f32:
; GFX1032: v_div_scale_f32 v{{[0-9]+}}, s{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; GFX1064: v_div_scale_f32 v{{[0-9]+}}, s[{{[0-9:]+}}], v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
define amdgpu_kernel void @test_div_scale_f32(float addrspace(1)* %out, float addrspace(1)* %in) #0 {
%tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
%gep.0 = getelementptr float, float addrspace(1)* %in, i32 %tid
%gep.1 = getelementptr float, float addrspace(1)* %gep.0, i32 1
%a = load volatile float, float addrspace(1)* %gep.0, align 4
%b = load volatile float, float addrspace(1)* %gep.1, align 4
%result = call { float, i1 } @llvm.amdgcn.div.scale.f32(float %a, float %b, i1 false) nounwind readnone
%result0 = extractvalue { float, i1 } %result, 0
store float %result0, float addrspace(1)* %out, align 4
ret void
}
; GCN-LABEL: {{^}}test_div_scale_f64:
; GFX1032: v_div_scale_f64 v[{{[0-9:]+}}], s{{[0-9]+}}, v[{{[0-9:]+}}], v[{{[0-9:]+}}], v[{{[0-9:]+}}]
; GFX1064: v_div_scale_f64 v[{{[0-9:]+}}], s[{{[0-9:]+}}], v[{{[0-9:]+}}], v[{{[0-9:]+}}], v[{{[0-9:]+}}]
define amdgpu_kernel void @test_div_scale_f64(double addrspace(1)* %out, double addrspace(1)* %aptr, double addrspace(1)* %in) #0 {
%tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
%gep.0 = getelementptr double, double addrspace(1)* %in, i32 %tid
%gep.1 = getelementptr double, double addrspace(1)* %gep.0, i32 1
%a = load volatile double, double addrspace(1)* %gep.0, align 8
%b = load volatile double, double addrspace(1)* %gep.1, align 8
%result = call { double, i1 } @llvm.amdgcn.div.scale.f64(double %a, double %b, i1 true) nounwind readnone
%result0 = extractvalue { double, i1 } %result, 0
store double %result0, double addrspace(1)* %out, align 8
ret void
}
; GCN-LABEL: {{^}}test_mad_i64_i32:
; GFX1032: v_mad_i64_i32 v[{{[0-9:]+}}], s{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, v[{{[0-9:]+}}]
; GFX1064: v_mad_i64_i32 v[{{[0-9:]+}}], s[{{[0-9:]+}}], v{{[0-9]+}}, v{{[0-9]+}}, v[{{[0-9:]+}}]
define i64 @test_mad_i64_i32(i32 %arg0, i32 %arg1, i64 %arg2) #0 {
%sext0 = sext i32 %arg0 to i64
%sext1 = sext i32 %arg1 to i64
%mul = mul i64 %sext0, %sext1
%mad = add i64 %mul, %arg2
ret i64 %mad
}
; GCN-LABEL: {{^}}test_mad_u64_u32:
; GFX1032: v_mad_u64_u32 v[{{[0-9:]+}}], s{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, v[{{[0-9:]+}}]
; GFX1064: v_mad_u64_u32 v[{{[0-9:]+}}], s[{{[0-9:]+}}], v{{[0-9]+}}, v{{[0-9]+}}, v[{{[0-9:]+}}]
define i64 @test_mad_u64_u32(i32 %arg0, i32 %arg1, i64 %arg2) #0 {
%sext0 = zext i32 %arg0 to i64
%sext1 = zext i32 %arg1 to i64
%mul = mul i64 %sext0, %sext1
%mad = add i64 %mul, %arg2
ret i64 %mad
}
; GCN-LABEL: {{^}}test_div_fmas_f32:
; GFX1032: v_cmp_eq_u32_e64 vcc_lo,
; GFX1064: v_cmp_eq_u32_e64 vcc,
; GCN: v_div_fmas_f32 v{{[0-9]+}}, {{[vs][0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
define amdgpu_kernel void @test_div_fmas_f32(float addrspace(1)* %out, float %a, float %b, float %c, i1 %d) nounwind {
%result = call float @llvm.amdgcn.div.fmas.f32(float %a, float %b, float %c, i1 %d) nounwind readnone
store float %result, float addrspace(1)* %out, align 4
ret void
}
; GCN-LABEL: {{^}}test_div_fmas_f64:
; GFX1032: v_cmp_eq_u32_e64 vcc_lo,
; GFX1064: v_cmp_eq_u32_e64 vcc,
; GCN-DAG: v_div_fmas_f64 v[{{[0-9:]+}}], {{[vs]}}[{{[0-9:]+}}], v[{{[0-9:]+}}], v[{{[0-9:]+}}]
define amdgpu_kernel void @test_div_fmas_f64(double addrspace(1)* %out, double %a, double %b, double %c, i1 %d) nounwind {
%result = call double @llvm.amdgcn.div.fmas.f64(double %a, double %b, double %c, i1 %d) nounwind readnone
store double %result, double addrspace(1)* %out, align 8
ret void
}
; GCN-LABEL: {{^}}test_div_fmas_f32_i1_phi_vcc:
; GFX1032: s_mov_b32 [[VCC:vcc_lo]], 0{{$}}
; GFX1064: s_mov_b64 [[VCC:vcc]], 0{{$}}
; GFX1032: s_and_saveexec_b32 [[SAVE:s[0-9]+]], s{{[0-9]+}}{{$}}
; GFX1064: s_and_saveexec_b64 [[SAVE:s\[[0-9]+:[0-9]+\]]], s[{{[0-9:]+}}]{{$}}
; GCN: load_dword [[LOAD:v[0-9]+]]
; GCN: v_cmp_ne_u32_e32 [[VCC]], 0, [[LOAD]]
; GCN: BB{{[0-9_]+}}:
; GFX1032: s_or_b32 exec_lo, exec_lo, [[SAVE]]
; GFX1064: s_or_b64 exec, exec, [[SAVE]]
; GCN: v_div_fmas_f32 {{v[0-9]+}}, {{v[0-9]+}}, {{v[0-9]+}}, {{v[0-9]+}}
define amdgpu_kernel void @test_div_fmas_f32_i1_phi_vcc(float addrspace(1)* %out, float addrspace(1)* %in, i32 addrspace(1)* %dummy) #0 {
entry:
%tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
%gep.out = getelementptr float, float addrspace(1)* %out, i32 2
%gep.a = getelementptr float, float addrspace(1)* %in, i32 %tid
%gep.b = getelementptr float, float addrspace(1)* %gep.a, i32 1
%gep.c = getelementptr float, float addrspace(1)* %gep.a, i32 2
%a = load float, float addrspace(1)* %gep.a
%b = load float, float addrspace(1)* %gep.b
%c = load float, float addrspace(1)* %gep.c
%cmp0 = icmp eq i32 %tid, 0
br i1 %cmp0, label %bb, label %exit
bb:
%val = load volatile i32, i32 addrspace(1)* %dummy
%cmp1 = icmp ne i32 %val, 0
br label %exit
exit:
%cond = phi i1 [false, %entry], [%cmp1, %bb]
%result = call float @llvm.amdgcn.div.fmas.f32(float %a, float %b, float %c, i1 %cond) nounwind readnone
store float %result, float addrspace(1)* %gep.out, align 4
ret void
}
; GCN-LABEL: {{^}}fdiv_f32:
; GFC1032: v_div_scale_f32 v{{[0-9]+}}, vcc_lo, s{{[0-9]+}}, v{{[0-9]+}}, s{{[0-9]+}}
; GFC1064: v_div_scale_f32 v{{[0-9]+}}, vcc, s{{[0-9]+}}, v{{[0-9]+}}, s{{[0-9]+}}
; GCN: v_rcp_f32_e32 v{{[0-9]+}}, v{{[0-9]+}}
; GCN-NOT: vcc
; GCN: v_div_fmas_f32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
define amdgpu_kernel void @fdiv_f32(float addrspace(1)* %out, float %a, float %b) #0 {
entry:
%fdiv = fdiv float %a, %b
store float %fdiv, float addrspace(1)* %out
ret void
}
; GCN-LABEL: {{^}}test_br_cc_f16:
; GFX1032: v_cmp_nlt_f16_e32 vcc_lo,
; GFX1032-NEXT: s_and_b32 vcc_lo, exec_lo, vcc_lo
; GFX1064: v_cmp_nlt_f16_e32 vcc,
; GFX1064-NEXT: s_and_b64 vcc, exec, vcc{{$}}
; GCN-NEXT: s_cbranch_vccnz
define amdgpu_kernel void @test_br_cc_f16(
half addrspace(1)* %r,
half addrspace(1)* %a,
half addrspace(1)* %b) {
entry:
%a.val = load half, half addrspace(1)* %a
%b.val = load half, half addrspace(1)* %b
%fcmp = fcmp olt half %a.val, %b.val
br i1 %fcmp, label %one, label %two
one:
store half %a.val, half addrspace(1)* %r
ret void
two:
store half %b.val, half addrspace(1)* %r
ret void
}
; GCN-LABEL: {{^}}test_brcc_i1:
; GCN: s_cmp_eq_u32 s{{[0-9]+}}, 0
; GCN-NEXT: s_cbranch_scc1
define amdgpu_kernel void @test_brcc_i1(i32 addrspace(1)* noalias %out, i32 addrspace(1)* noalias %in, i1 %val) #0 {
%cmp0 = icmp ne i1 %val, 0
br i1 %cmp0, label %store, label %end
store:
store i32 222, i32 addrspace(1)* %out
ret void
end:
ret void
}
; GCN-LABEL: {{^}}test_preserve_condition_undef_flag:
; GFX1032: v_cmp_nlt_f32_e64 s{{[0-9]+}}, s{{[0-9]+}}, 1.0
; GFX1032: v_cmp_ngt_f32_e64 s{{[0-9]+}}, s{{[0-9]+}}, 0
; GFX1032: v_cmp_nlt_f32_e64 s{{[0-9]+}}, s{{[0-9]+}}, 1.0
; GFX1032: s_or_b32 [[OR1:s[0-9]+]], s{{[0-9]+}}, s{{[0-9]+}}
; GFX1032: s_or_b32 [[OR2:s[0-9]+]], [[OR1]], s{{[0-9]+}}
; GFX1032: s_and_b32 vcc_lo, exec_lo, [[OR2]]
; GFX1064: v_cmp_nlt_f32_e64 s[{{[0-9:]+}}], s{{[0-9]+}}, 1.0
; GFX1064: v_cmp_ngt_f32_e64 s[{{[0-9:]+}}], s{{[0-9]+}}, 0
; GFX1064: v_cmp_nlt_f32_e64 s[{{[0-9:]+}}], s{{[0-9]+}}, 1.0
; GFX1064: s_or_b64 [[OR1:s\[[0-9:]+\]]], s[{{[0-9:]+}}], s[{{[0-9:]+}}]
; GFX1064: s_or_b64 [[OR2:s\[[0-9:]+\]]], [[OR1]], s[{{[0-9:]+}}]
; GFX1064: s_and_b64 vcc, exec, [[OR2]]
; GCN: s_cbranch_vccnz
define amdgpu_kernel void @test_preserve_condition_undef_flag(float %arg, i32 %arg1, float %arg2) #0 {
bb0:
%tmp = icmp sgt i32 %arg1, 4
%undef = call i1 @llvm.amdgcn.class.f32(float undef, i32 undef)
%tmp4 = select i1 %undef, float %arg, float 1.000000e+00
%tmp5 = fcmp ogt float %arg2, 0.000000e+00
%tmp6 = fcmp olt float %arg2, 1.000000e+00
%tmp7 = fcmp olt float %arg, %tmp4
%tmp8 = and i1 %tmp5, %tmp6
%tmp9 = and i1 %tmp8, %tmp7
br i1 %tmp9, label %bb1, label %bb2
bb1:
store volatile i32 0, i32 addrspace(1)* undef
br label %bb2
bb2:
ret void
}
; GCN-LABEL: {{^}}test_invert_true_phi_cond_break_loop:
; GFX1032: s_xor_b32 s{{[0-9]+}}, s{{[0-9]+}}, -1
; GFX1032: s_or_b32 s{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
; GFX1064: s_xor_b64 s[{{[0-9:]+}}], s[{{[0-9:]+}}], -1
; GFX1064: s_or_b64 s[{{[0-9:]+}}], s[{{[0-9:]+}}], s[{{[0-9:]+}}]
define amdgpu_kernel void @test_invert_true_phi_cond_break_loop(i32 %arg) #0 {
bb:
%id = call i32 @llvm.amdgcn.workitem.id.x()
%tmp = sub i32 %id, %arg
br label %bb1
bb1: ; preds = %Flow, %bb
%lsr.iv = phi i32 [ undef, %bb ], [ %tmp2, %Flow ]
%lsr.iv.next = add i32 %lsr.iv, 1
%cmp0 = icmp slt i32 %lsr.iv.next, 0
br i1 %cmp0, label %bb4, label %Flow
bb4: ; preds = %bb1
%load = load volatile i32, i32 addrspace(1)* undef, align 4
%cmp1 = icmp sge i32 %tmp, %load
br label %Flow
Flow: ; preds = %bb4, %bb1
%tmp2 = phi i32 [ %lsr.iv.next, %bb4 ], [ undef, %bb1 ]
%tmp3 = phi i1 [ %cmp1, %bb4 ], [ true, %bb1 ]
br i1 %tmp3, label %bb1, label %bb9
bb9: ; preds = %Flow
store volatile i32 7, i32 addrspace(3)* undef
ret void
}
; GCN-LABEL: {{^}}test_movrels_extract_neg_offset_vgpr:
; GFX1032: v_cmp_eq_u32_e32 vcc_lo, 1, v{{[0-9]+}}
; GFX1032: v_cndmask_b32_e64 v{{[0-9]+}}, 0, 1, vcc_lo
; GFX1032: v_cmp_ne_u32_e32 vcc_lo, 2, v{{[0-9]+}}
; GFX1032: v_cndmask_b32_e32 v{{[0-9]+}}, 2, v{{[0-9]+}}, vcc_lo
; GFX1032: v_cmp_ne_u32_e32 vcc_lo, 3, v{{[0-9]+}}
; GFX1032: v_cndmask_b32_e32 v{{[0-9]+}}, 3, v{{[0-9]+}}, vcc_lo
; GFX1064: v_cmp_eq_u32_e32 vcc, 1, v{{[0-9]+}}
; GFX1064: v_cndmask_b32_e64 v{{[0-9]+}}, 0, 1, vcc
; GFX1064: v_cmp_ne_u32_e32 vcc, 2, v{{[0-9]+}}
; GFX1064: v_cndmask_b32_e32 v{{[0-9]+}}, 2, v{{[0-9]+}}, vcc
; GFX1064: v_cmp_ne_u32_e32 vcc, 3, v{{[0-9]+}}
; GFX1064: v_cndmask_b32_e32 v{{[0-9]+}}, 3, v{{[0-9]+}}, vcc
define amdgpu_kernel void @test_movrels_extract_neg_offset_vgpr(i32 addrspace(1)* %out) #0 {
entry:
%id = call i32 @llvm.amdgcn.workitem.id.x() #1
%index = add i32 %id, -512
%value = extractelement <4 x i32> <i32 0, i32 1, i32 2, i32 3>, i32 %index
store i32 %value, i32 addrspace(1)* %out
ret void
}
; GCN-LABEL: {{^}}test_set_inactive:
; GFX1032: s_not_b32 exec_lo, exec_lo
; GFX1032: v_mov_b32_e32 {{v[0-9]+}}, 42
; GFX1032: s_not_b32 exec_lo, exec_lo
; GFX1064: s_not_b64 exec, exec{{$}}
; GFX1064: v_mov_b32_e32 {{v[0-9]+}}, 42
; GFX1064: s_not_b64 exec, exec{{$}}
define amdgpu_kernel void @test_set_inactive(i32 addrspace(1)* %out, i32 %in) #0 {
%tmp = call i32 @llvm.amdgcn.set.inactive.i32(i32 %in, i32 42)
store i32 %tmp, i32 addrspace(1)* %out
ret void
}
; GCN-LABEL: {{^}}test_set_inactive_64:
; GFX1032: s_not_b32 exec_lo, exec_lo
; GFX1032: v_mov_b32_e32 {{v[0-9]+}}, 0
; GFX1032: v_mov_b32_e32 {{v[0-9]+}}, 0
; GFX1032: s_not_b32 exec_lo, exec_lo
; GFX1064: s_not_b64 exec, exec{{$}}
; GFX1064: v_mov_b32_e32 {{v[0-9]+}}, 0
; GFX1064: v_mov_b32_e32 {{v[0-9]+}}, 0
; GFX1064: s_not_b64 exec, exec{{$}}
define amdgpu_kernel void @test_set_inactive_64(i64 addrspace(1)* %out, i64 %in) #0 {
%tmp = call i64 @llvm.amdgcn.set.inactive.i64(i64 %in, i64 0)
store i64 %tmp, i64 addrspace(1)* %out
ret void
}
; GCN-LABEL: {{^}}test_kill_i1_terminator_float:
; GFX1032: s_mov_b32 exec_lo, 0
; GFX1064: s_mov_b64 exec, 0
define amdgpu_ps void @test_kill_i1_terminator_float() #0 {
call void @llvm.amdgcn.kill(i1 false)
ret void
}
; GCN-LABEL: {{^}}test_kill_i1_terminator_i1:
; GFX1032: s_or_b32 [[OR:s[0-9]+]],
; GFX1032: s_and_b32 exec_lo, exec_lo, [[OR]]
; GFX1064: s_or_b64 [[OR:s\[[0-9:]+\]]],
; GFX1064: s_and_b64 exec, exec, [[OR]]
define amdgpu_gs void @test_kill_i1_terminator_i1(i32 %a, i32 %b, i32 %c, i32 %d) #0 {
%c1 = icmp slt i32 %a, %b
%c2 = icmp slt i32 %c, %d
%x = or i1 %c1, %c2
call void @llvm.amdgcn.kill(i1 %x)
ret void
}
; GCN-LABEL: {{^}}test_loop_vcc:
; GFX1032: v_cmp_lt_f32_e32 vcc_lo,
; GFX1064: v_cmp_lt_f32_e32 vcc,
; GCN: s_cbranch_vccz
define amdgpu_ps <4 x float> @test_loop_vcc(<4 x float> %in) #0 {
entry:
br label %loop
loop:
%ctr.iv = phi float [ 0.0, %entry ], [ %ctr.next, %body ]
%c.iv = phi <4 x float> [ %in, %entry ], [ %c.next, %body ]
%cc = fcmp ogt float %ctr.iv, 7.0
br i1 %cc, label %break, label %body
body:
%c.iv0 = extractelement <4 x float> %c.iv, i32 0
%c.next = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %c.iv0, <8 x i32> undef, <4 x i32> undef, i1 0, i32 0, i32 0)
%ctr.next = fadd float %ctr.iv, 2.0
br label %loop
break:
ret <4 x float> %c.iv
}
; GCN-LABEL: {{^}}test_wwm1:
; GFX1032: s_or_saveexec_b32 [[SAVE:s[0-9]+]], -1
; GFX1032: s_mov_b32 exec_lo, [[SAVE]]
; GFX1064: s_or_saveexec_b64 [[SAVE:s\[[0-9]+:[0-9]+\]]], -1
; GFX1064: s_mov_b64 exec, [[SAVE]]
define amdgpu_ps float @test_wwm1(i32 inreg %idx0, i32 inreg %idx1, float %src0, float %src1) {
main_body:
%out = fadd float %src0, %src1
%out.0 = call float @llvm.amdgcn.wwm.f32(float %out)
ret float %out.0
}
; GCN-LABEL: {{^}}test_wwm2:
; GFX1032: v_cmp_gt_u32_e32 vcc_lo, 32, v{{[0-9]+}}
; GFX1032: s_and_saveexec_b32 [[SAVE1:s[0-9]+]], vcc_lo
; GFX1032: s_or_saveexec_b32 [[SAVE2:s[0-9]+]], -1
; GFX1032: s_mov_b32 exec_lo, [[SAVE2]]
; GFX1032: s_or_b32 exec_lo, exec_lo, [[SAVE1]]
; GFX1064: v_cmp_gt_u32_e32 vcc, 32, v{{[0-9]+}}
; GFX1064: s_and_saveexec_b64 [[SAVE1:s\[[0-9:]+\]]], vcc{{$}}
; GFX1064: s_or_saveexec_b64 [[SAVE2:s\[[0-9:]+\]]], -1
; GFX1064: s_mov_b64 exec, [[SAVE2]]
; GFX1064: s_or_b64 exec, exec, [[SAVE1]]
define amdgpu_ps float @test_wwm2(i32 inreg %idx) {
main_body:
; use mbcnt to make sure the branch is divergent
%lo = call i32 @llvm.amdgcn.mbcnt.lo(i32 -1, i32 0)
%hi = call i32 @llvm.amdgcn.mbcnt.hi(i32 -1, i32 %lo)
%cc = icmp uge i32 %hi, 32
br i1 %cc, label %endif, label %if
if:
%src = call float @llvm.amdgcn.buffer.load.f32(<4 x i32> undef, i32 %idx, i32 0, i1 0, i1 0)
%out = fadd float %src, %src
%out.0 = call float @llvm.amdgcn.wwm.f32(float %out)
%out.1 = fadd float %src, %out.0
br label %endif
endif:
%out.2 = phi float [ %out.1, %if ], [ 0.0, %main_body ]
ret float %out.2
}
; GCN-LABEL: {{^}}test_wqm1:
; GFX1032: s_mov_b32 [[ORIG:s[0-9]+]], exec_lo
; GFX1032: s_wqm_b32 exec_lo, exec_lo
; GFX1032: s_and_b32 exec_lo, exec_lo, [[ORIG]]
; GFX1064: s_mov_b64 [[ORIG:s\[[0-9]+:[0-9]+\]]], exec{{$}}
; GFX1064: s_wqm_b64 exec, exec{{$}}
; GFX1064: s_and_b64 exec, exec, [[ORIG]]
define amdgpu_ps <4 x float> @test_wqm1(i32 inreg, i32 inreg, i32 inreg, i32 inreg %m0, <8 x i32> inreg %rsrc, <4 x i32> inreg %sampler, <2 x float> %pos) #0 {
main_body:
%inst23 = extractelement <2 x float> %pos, i32 0
%inst24 = extractelement <2 x float> %pos, i32 1
%inst25 = tail call float @llvm.amdgcn.interp.p1(float %inst23, i32 0, i32 0, i32 %m0)
%inst26 = tail call float @llvm.amdgcn.interp.p2(float %inst25, float %inst24, i32 0, i32 0, i32 %m0)
%inst28 = tail call float @llvm.amdgcn.interp.p1(float %inst23, i32 1, i32 0, i32 %m0)
%inst29 = tail call float @llvm.amdgcn.interp.p2(float %inst28, float %inst24, i32 1, i32 0, i32 %m0)
%tex = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float %inst26, float %inst29, <8 x i32> %rsrc, <4 x i32> %sampler, i1 0, i32 0, i32 0)
ret <4 x float> %tex
}
; GCN-LABEL: {{^}}test_wqm2:
; GFX1032: s_wqm_b32 exec_lo, exec_lo
; GFX1032: s_and_b32 exec_lo, exec_lo, s{{[0-9+]}}
; GFX1064: s_wqm_b64 exec, exec{{$}}
; GFX1064: s_and_b64 exec, exec, s[{{[0-9:]+}}]
define amdgpu_ps float @test_wqm2(i32 inreg %idx0, i32 inreg %idx1) #0 {
main_body:
%src0 = call float @llvm.amdgcn.buffer.load.f32(<4 x i32> undef, i32 %idx0, i32 0, i1 0, i1 0)
%src1 = call float @llvm.amdgcn.buffer.load.f32(<4 x i32> undef, i32 %idx1, i32 0, i1 0, i1 0)
%out = fadd float %src0, %src1
%out.0 = bitcast float %out to i32
%out.1 = call i32 @llvm.amdgcn.wqm.i32(i32 %out.0)
%out.2 = bitcast i32 %out.1 to float
ret float %out.2
}
; GCN-LABEL: {{^}}test_intr_fcmp_i64:
; GFX1032-DAG: v_mov_b32_e32 v[[V_HI:[0-9]+]], 0{{$}}
; GFX1032-DAG: v_cmp_eq_f32_e64 s[[C_LO:[0-9]+]], {{s[0-9]+}}, |{{[vs][0-9]+}}|
; GFX1032-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]
; GFX1064: v_cmp_eq_f32_e64 s{{\[}}[[C_LO:[0-9]+]]:[[C_HI:[0-9]+]]], {{s[0-9]+}}, |{{[vs][0-9]+}}|
; GFX1064-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]
; GFX1064-DAG: v_mov_b32_e32 v[[V_HI:[0-9]+]], s[[C_HI]]
; GCN: store_dwordx2 v[{{[0-9:]+}}], v{{\[}}[[V_LO]]:[[V_HI]]],
define amdgpu_kernel void @test_intr_fcmp_i64(i64 addrspace(1)* %out, float %src, float %a) {
%temp = call float @llvm.fabs.f32(float %a)
%result = call i64 @llvm.amdgcn.fcmp.i64.f32(float %src, float %temp, i32 1)
store i64 %result, i64 addrspace(1)* %out
ret void
}
; GCN-LABEL: {{^}}test_intr_icmp_i64:
; GFX1032-DAG: v_mov_b32_e32 v[[V_HI:[0-9]+]], 0{{$}}
; GFX1032-DAG: v_cmp_eq_u32_e64 [[C_LO:vcc_lo|s[0-9]+]], 0x64, {{s[0-9]+}}
; GFX1032-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], [[C_LO]]
; GFX1064: v_cmp_eq_u32_e64 s{{\[}}[[C_LO:[0-9]+]]:[[C_HI:[0-9]+]]], 0x64, {{s[0-9]+}}
; GFX1064-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]
; GFX1064-DAG: v_mov_b32_e32 v[[V_HI:[0-9]+]], s[[C_HI]]
; GCN: store_dwordx2 v[{{[0-9:]+}}], v{{\[}}[[V_LO]]:[[V_HI]]],
define amdgpu_kernel void @test_intr_icmp_i64(i64 addrspace(1)* %out, i32 %src) {
%result = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %src, i32 100, i32 32)
store i64 %result, i64 addrspace(1)* %out
ret void
}
; GCN-LABEL: {{^}}test_intr_fcmp_i32:
; GFX1032-DAG: v_cmp_eq_f32_e64 s[[C_LO:[0-9]+]], {{s[0-9]+}}, |{{[vs][0-9]+}}|
; GFX1032-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]
; GFX1064: v_cmp_eq_f32_e64 s{{\[}}[[C_LO:[0-9]+]]:[[C_HI:[0-9]+]]], {{s[0-9]+}}, |{{[vs][0-9]+}}|
; GFX1064-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]
; GCN: store_dword v[{{[0-9:]+}}], v[[V_LO]],
define amdgpu_kernel void @test_intr_fcmp_i32(i32 addrspace(1)* %out, float %src, float %a) {
%temp = call float @llvm.fabs.f32(float %a)
%result = call i32 @llvm.amdgcn.fcmp.i32.f32(float %src, float %temp, i32 1)
store i32 %result, i32 addrspace(1)* %out
ret void
}
; GCN-LABEL: {{^}}test_intr_icmp_i32:
; GFX1032-DAG: v_cmp_eq_u32_e64 s[[C_LO:[0-9]+]], 0x64, {{s[0-9]+}}
; GFX1032-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]{{$}}
; GFX1064: v_cmp_eq_u32_e64 s{{\[}}[[C_LO:[0-9]+]]:{{[0-9]+}}], 0x64, {{s[0-9]+}}
; GFX1064-DAG: v_mov_b32_e32 v[[V_LO:[0-9]+]], s[[C_LO]]{{$}}
; GCN: store_dword v[{{[0-9:]+}}], v[[V_LO]],
define amdgpu_kernel void @test_intr_icmp_i32(i32 addrspace(1)* %out, i32 %src) {
%result = call i32 @llvm.amdgcn.icmp.i32.i32(i32 %src, i32 100, i32 32)
store i32 %result, i32 addrspace(1)* %out
ret void
}
; GCN-LABEL: {{^}}test_wqm_vote:
; GFX1032: v_cmp_neq_f32_e32 vcc_lo, 0
; GFX1032: s_wqm_b32 [[WQM:s[0-9]+]], vcc_lo
; GFX1032: s_and_b32 exec_lo, exec_lo, [[WQM]]
; GFX1064: v_cmp_neq_f32_e32 vcc, 0
; GFX1064: s_wqm_b64 [[WQM:s\[[0-9:]+\]]], vcc{{$}}
; GFX1064: s_and_b64 exec, exec, [[WQM]]
define amdgpu_ps void @test_wqm_vote(float %a) {
%c1 = fcmp une float %a, 0.0
%c2 = call i1 @llvm.amdgcn.wqm.vote(i1 %c1)
call void @llvm.amdgcn.kill(i1 %c2)
ret void
}
; GCN-LABEL: {{^}}test_branch_true:
; GFX1032: s_and_b32 vcc_lo, exec_lo, -1
; GFX1064: s_and_b64 vcc, exec, -1
define amdgpu_kernel void @test_branch_true() #2 {
entry:
br i1 true, label %for.end, label %for.body.lr.ph
for.body.lr.ph: ; preds = %entry
br label %for.body
for.body: ; preds = %for.body, %for.body.lr.ph
br i1 undef, label %for.end, label %for.body
for.end: ; preds = %for.body, %entry
ret void
}
; GCN-LABEL: {{^}}test_ps_live:
; GFX1032: s_mov_b32 [[C:s[0-9]+]], exec_lo
; GFX1064: s_mov_b64 [[C:s\[[0-9:]+\]]], exec{{$}}
; GCN: v_cndmask_b32_e64 v{{[0-9]+}}, 0, 1, [[C]]
define amdgpu_ps float @test_ps_live() #0 {
%live = call i1 @llvm.amdgcn.ps.live()
%live.32 = zext i1 %live to i32
%r = bitcast i32 %live.32 to float
ret float %r
}
; GCN-LABEL: {{^}}test_vccnz_ifcvt_triangle64:
; GFX1032: v_cmp_neq_f64_e64 [[C:s[0-9]+]], s[{{[0-9:]+}}], 1.0
; GFX1032: s_and_b32 vcc_lo, exec_lo, [[C]]
; GFX1064: v_cmp_neq_f64_e64 [[C:s\[[0-9:]+\]]], s[{{[0-9:]+}}], 1.0
; GFX1064: s_and_b64 vcc, exec, [[C]]
define amdgpu_kernel void @test_vccnz_ifcvt_triangle64(double addrspace(1)* %out, double addrspace(1)* %in) #0 {
entry:
%v = load double, double addrspace(1)* %in
%cc = fcmp oeq double %v, 1.000000e+00
br i1 %cc, label %if, label %endif
if:
%u = fadd double %v, %v
br label %endif
endif:
%r = phi double [ %v, %entry ], [ %u, %if ]
store double %r, double addrspace(1)* %out
ret void
}
; GCN-LABEL: {{^}}test_init_exec:
; GFX1032: s_mov_b32 exec_lo, 0x12345
; GFX1064: s_mov_b64 exec, 0x12345
; GCN: v_add_f32_e32 v0,
define amdgpu_ps float @test_init_exec(float %a, float %b) {
main_body:
%s = fadd float %a, %b
call void @llvm.amdgcn.init.exec(i64 74565)
ret float %s
}
; GCN-LABEL: {{^}}test_init_exec_from_input:
; GCN: s_bfe_u32 s0, s3, 0x70008
; GFX1032: s_bfm_b32 exec_lo, s0, 0
; GFX1032: s_cmp_eq_u32 s0, 32
; GFX1032: s_cmov_b32 exec_lo, -1
; GFX1064: s_bfm_b64 exec, s0, 0
; GFX1064: s_cmp_eq_u32 s0, 64
; GFX1064: s_cmov_b64 exec, -1
; GCN: v_add_f32_e32 v0,
define amdgpu_ps float @test_init_exec_from_input(i32 inreg, i32 inreg, i32 inreg, i32 inreg %count, float %a, float %b) {
main_body:
%s = fadd float %a, %b
call void @llvm.amdgcn.init.exec.from.input(i32 %count, i32 8)
ret float %s
}
; GCN-LABEL: {{^}}test_vgprblocks_w32_attr:
; Test that the wave size can be overridden in function attributes and that the block size is correct as a result
; GFX10DEFWAVE: ; VGPRBlocks: 1
define amdgpu_gs float @test_vgprblocks_w32_attr(float %a, float %b, float %c, float %d, float %e,
float %f, float %g, float %h, float %i, float %j, float %k, float %l) #3 {
main_body:
%s = fadd float %a, %b
%s.1 = fadd float %s, %c
%s.2 = fadd float %s.1, %d
%s.3 = fadd float %s.2, %e
%s.4 = fadd float %s.3, %f
%s.5 = fadd float %s.4, %g
%s.6 = fadd float %s.5, %h
%s.7 = fadd float %s.6, %i
%s.8 = fadd float %s.7, %j
%s.9 = fadd float %s.8, %k
%s.10 = fadd float %s.9, %l
ret float %s.10
}
; GCN-LABEL: {{^}}test_vgprblocks_w64_attr:
; Test that the wave size can be overridden in function attributes and that the block size is correct as a result
; GFX10DEFWAVE: ; VGPRBlocks: 2
define amdgpu_gs float @test_vgprblocks_w64_attr(float %a, float %b, float %c, float %d, float %e,
float %f, float %g, float %h, float %i, float %j, float %k, float %l) #4 {
main_body:
%s = fadd float %a, %b
%s.1 = fadd float %s, %c
%s.2 = fadd float %s.1, %d
%s.3 = fadd float %s.2, %e
%s.4 = fadd float %s.3, %f
%s.5 = fadd float %s.4, %g
%s.6 = fadd float %s.5, %h
%s.7 = fadd float %s.6, %i
%s.8 = fadd float %s.7, %j
%s.9 = fadd float %s.8, %k
%s.10 = fadd float %s.9, %l
ret float %s.10
}
; GCN-LABEL: {{^}}icmp64:
; GFX1032: v_cmp_eq_u32_e32 vcc_lo, 0, v
; GFX1064: v_cmp_eq_u32_e32 vcc, 0, v
define amdgpu_kernel void @icmp64(i32 %n, i32 %s) {
entry:
%id = tail call i32 @llvm.amdgcn.workitem.id.x()
%mul4 = mul nsw i32 %s, %n
%cmp = icmp slt i32 0, %mul4
br label %if.end
if.end: ; preds = %entry
%rem = urem i32 %id, %s
%icmp = tail call i64 @llvm.amdgcn.icmp.i64.i32(i32 %rem, i32 0, i32 32)
%shr = lshr i64 %icmp, 1
%notmask = shl nsw i64 -1, 0
%and = and i64 %notmask, %shr
%or = or i64 %and, -9223372036854775808
%cttz = tail call i64 @llvm.cttz.i64(i64 %or, i1 true)
%cast = trunc i64 %cttz to i32
%cmp3 = icmp ugt i32 10, %cast
%cmp6 = icmp ne i32 %rem, 0
%brmerge = or i1 %cmp6, %cmp3
br i1 %brmerge, label %if.end2, label %if.then
if.then: ; preds = %if.end
unreachable
if.end2: ; preds = %if.end
ret void
}
; GCN-LABEL: {{^}}fcmp64:
; GFX1032: v_cmp_eq_f32_e32 vcc_lo, 0, v
; GFX1064: v_cmp_eq_f32_e32 vcc, 0, v
define amdgpu_kernel void @fcmp64(float %n, float %s) {
entry:
%id = tail call i32 @llvm.amdgcn.workitem.id.x()
%id.f = uitofp i32 %id to float
%mul4 = fmul float %s, %n
%cmp = fcmp ult float 0.0, %mul4
br label %if.end
if.end: ; preds = %entry
%rem.f = frem float %id.f, %s
%fcmp = tail call i64 @llvm.amdgcn.fcmp.i64.f32(float %rem.f, float 0.0, i32 1)
%shr = lshr i64 %fcmp, 1
%notmask = shl nsw i64 -1, 0
%and = and i64 %notmask, %shr
%or = or i64 %and, -9223372036854775808
%cttz = tail call i64 @llvm.cttz.i64(i64 %or, i1 true)
%cast = trunc i64 %cttz to i32
%cmp3 = icmp ugt i32 10, %cast
%cmp6 = fcmp one float %rem.f, 0.0
%brmerge = or i1 %cmp6, %cmp3
br i1 %brmerge, label %if.end2, label %if.then
if.then: ; preds = %if.end
unreachable
if.end2: ; preds = %if.end
ret void
}
; GCN-LABEL: {{^}}icmp32:
; GFX1032: v_cmp_eq_u32_e32 vcc_lo, 0, v
; GFX1064: v_cmp_eq_u32_e32 vcc, 0, v
define amdgpu_kernel void @icmp32(i32 %n, i32 %s) {
entry:
%id = tail call i32 @llvm.amdgcn.workitem.id.x()
%mul4 = mul nsw i32 %s, %n
%cmp = icmp slt i32 0, %mul4
br label %if.end
if.end: ; preds = %entry
%rem = urem i32 %id, %s
%icmp = tail call i32 @llvm.amdgcn.icmp.i32.i32(i32 %rem, i32 0, i32 32)
%shr = lshr i32 %icmp, 1
%notmask = shl nsw i32 -1, 0
%and = and i32 %notmask, %shr
%or = or i32 %and, 2147483648
%cttz = tail call i32 @llvm.cttz.i32(i32 %or, i1 true)
%cmp3 = icmp ugt i32 10, %cttz
%cmp6 = icmp ne i32 %rem, 0
%brmerge = or i1 %cmp6, %cmp3
br i1 %brmerge, label %if.end2, label %if.then
if.then: ; preds = %if.end
unreachable
if.end2: ; preds = %if.end
ret void
}
; GCN-LABEL: {{^}}fcmp32:
; GFX1032: v_cmp_eq_f32_e32 vcc_lo, 0, v
; GFX1064: v_cmp_eq_f32_e32 vcc, 0, v
define amdgpu_kernel void @fcmp32(float %n, float %s) {
entry:
%id = tail call i32 @llvm.amdgcn.workitem.id.x()
%id.f = uitofp i32 %id to float
%mul4 = fmul float %s, %n
%cmp = fcmp ult float 0.0, %mul4
br label %if.end
if.end: ; preds = %entry
%rem.f = frem float %id.f, %s
%fcmp = tail call i32 @llvm.amdgcn.fcmp.i32.f32(float %rem.f, float 0.0, i32 1)
%shr = lshr i32 %fcmp, 1
%notmask = shl nsw i32 -1, 0
%and = and i32 %notmask, %shr
%or = or i32 %and, 2147483648
%cttz = tail call i32 @llvm.cttz.i32(i32 %or, i1 true)
%cmp3 = icmp ugt i32 10, %cttz
%cmp6 = fcmp one float %rem.f, 0.0
%brmerge = or i1 %cmp6, %cmp3
br i1 %brmerge, label %if.end2, label %if.then
if.then: ; preds = %if.end
unreachable
if.end2: ; preds = %if.end
ret void
}
declare void @external_void_func_void() #1
; Test save/restore of VGPR needed for SGPR spilling.
; GCN-LABEL: {{^}}callee_no_stack_with_call:
; GCN: s_waitcnt
; GCN: s_mov_b32 s5, s32
; GFX1064: s_add_u32 s32, s32, 0x400
; GFX1032: s_add_u32 s32, s32, 0x200
; GFX1064: s_or_saveexec_b64 [[COPY_EXEC0:s\[[0-9]+:[0-9]+\]]], -1{{$}}
; GFX1032: s_or_saveexec_b32 [[COPY_EXEC0:s[0-9]]], -1{{$}}
; GCN-NEXT: buffer_store_dword v32, off, s[0:3], s5 ; 4-byte Folded Spill
; GFX1064-NEXT: s_mov_b64 exec, [[COPY_EXEC0]]
; GFX1032-NEXT: s_mov_b32 exec_lo, [[COPY_EXEC0]]
; GCN-DAG: v_writelane_b32 v32, s33, 0
; GCN-DAG: v_writelane_b32 v32, s34, 1
; GCN-DAG: s_mov_b32 s33, s5
; GCN: s_swappc_b64
; GCN-DAG: s_mov_b32 s5, s33
; GCN-DAG: v_readlane_b32 s34, v32, 1
; GCN-DAG: v_readlane_b32 s33, v32, 0
; GFX1064: s_or_saveexec_b64 [[COPY_EXEC1:s\[[0-9]+:[0-9]+\]]], -1{{$}}
; GFX1032: s_or_saveexec_b32 [[COPY_EXEC1:s[0-9]]], -1{{$}}
; GCN-NEXT: buffer_load_dword v32, off, s[0:3], s5 ; 4-byte Folded Reload
; GFX1064-NEXT: s_mov_b64 exec, [[COPY_EXEC1]]
; GFX1032-NEXT: s_mov_b32 exec_lo, [[COPY_EXEC1]]
; GFX1064: s_sub_u32 s32, s32, 0x400
; GFX1032: s_sub_u32 s32, s32, 0x200
; GCN: s_setpc_b64
define void @callee_no_stack_with_call() #1 {
call void @external_void_func_void()
ret void
}
declare i32 @llvm.amdgcn.workitem.id.x()
declare float @llvm.fabs.f32(float)
declare { float, i1 } @llvm.amdgcn.div.scale.f32(float, float, i1)
declare { double, i1 } @llvm.amdgcn.div.scale.f64(double, double, i1)
declare float @llvm.amdgcn.div.fmas.f32(float, float, float, i1)
declare double @llvm.amdgcn.div.fmas.f64(double, double, double, i1)
declare i1 @llvm.amdgcn.class.f32(float, i32)
declare i32 @llvm.amdgcn.set.inactive.i32(i32, i32)
declare i64 @llvm.amdgcn.set.inactive.i64(i64, i64)
declare <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32, float, <8 x i32>, <4 x i32>, i1, i32, i32)
declare <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32, float, float, <8 x i32>, <4 x i32>, i1, i32, i32)
declare float @llvm.amdgcn.wwm.f32(float)
declare i32 @llvm.amdgcn.wqm.i32(i32)
declare float @llvm.amdgcn.interp.p1(float, i32, i32, i32)
declare float @llvm.amdgcn.interp.p2(float, float, i32, i32, i32)
declare float @llvm.amdgcn.buffer.load.f32(<4 x i32>, i32, i32, i1, i1)
declare i32 @llvm.amdgcn.mbcnt.lo(i32, i32)
declare i32 @llvm.amdgcn.mbcnt.hi(i32, i32)
declare i64 @llvm.amdgcn.fcmp.i64.f32(float, float, i32)
declare i64 @llvm.amdgcn.icmp.i64.i32(i32, i32, i32)
declare i32 @llvm.amdgcn.fcmp.i32.f32(float, float, i32)
declare i32 @llvm.amdgcn.icmp.i32.i32(i32, i32, i32)
declare void @llvm.amdgcn.kill(i1)
declare i1 @llvm.amdgcn.wqm.vote(i1)
declare i1 @llvm.amdgcn.ps.live()
declare void @llvm.amdgcn.init.exec(i64)
declare void @llvm.amdgcn.init.exec.from.input(i32, i32)
declare i64 @llvm.cttz.i64(i64, i1)
declare i32 @llvm.cttz.i32(i32, i1)
attributes #0 = { nounwind readnone speculatable }
attributes #1 = { nounwind }
attributes #2 = { nounwind readnone optnone noinline }
attributes #3 = { "target-features"="+wavefrontsize32" }
attributes #4 = { "target-features"="+wavefrontsize64" }

test/CodeGen/AMDGPU/xor3.ll

Show All 10 Lines
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_xor_b32_e32 v0, v0, v1 ; GFX9-NEXT: v_xor_b32_e32 v0, v0, v1
; GFX9-NEXT: v_xor_b32_e32 v0, v0, v2 ; GFX9-NEXT: v_xor_b32_e32 v0, v0, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: xor3: ; GFX10-LABEL: xor3:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_xor3_b32 v0, v0, v1, v2 ; GFX10-NEXT: v_xor3_b32 v0, v0, v1, v2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = xor i32 %a, %b %x = xor i32 %a, %b
%result = xor i32 %x, %c %result = xor i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @xor3_vgpr_b(i32 inreg %a, i32 %b, i32 inreg %c) { define amdgpu_ps float @xor3_vgpr_b(i32 inreg %a, i32 %b, i32 inreg %c) {
; GFX9-LABEL: xor3_vgpr_b: ; GFX9-LABEL: xor3_vgpr_b:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_xor_b32_e32 v0, s2, v0 ; GFX9-NEXT: v_xor_b32_e32 v0, s2, v0
; GFX9-NEXT: v_xor_b32_e32 v0, s3, v0 ; GFX9-NEXT: v_xor_b32_e32 v0, s3, v0
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: xor3_vgpr_b: ; GFX10-LABEL: xor3_vgpr_b:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_xor3_b32 v0, s2, v0, s3 ; GFX10-NEXT: v_xor3_b32 v0, s2, v0, s3
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = xor i32 %a, %b %x = xor i32 %a, %b
%result = xor i32 %x, %c %result = xor i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @xor3_vgpr_all2(i32 %a, i32 %b, i32 %c) { define amdgpu_ps float @xor3_vgpr_all2(i32 %a, i32 %b, i32 %c) {
; GFX9-LABEL: xor3_vgpr_all2: ; GFX9-LABEL: xor3_vgpr_all2:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_xor_b32_e32 v1, v1, v2 ; GFX9-NEXT: v_xor_b32_e32 v1, v1, v2
; GFX9-NEXT: v_xor_b32_e32 v0, v0, v1 ; GFX9-NEXT: v_xor_b32_e32 v0, v0, v1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: xor3_vgpr_all2: ; GFX10-LABEL: xor3_vgpr_all2:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_xor3_b32 v0, v1, v2, v0 ; GFX10-NEXT: v_xor3_b32 v0, v1, v2, v0
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = xor i32 %b, %c %x = xor i32 %b, %c
%result = xor i32 %a, %x %result = xor i32 %a, %x
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @xor3_vgpr_bc(i32 inreg %a, i32 %b, i32 %c) { define amdgpu_ps float @xor3_vgpr_bc(i32 inreg %a, i32 %b, i32 %c) {
; GFX9-LABEL: xor3_vgpr_bc: ; GFX9-LABEL: xor3_vgpr_bc:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_xor_b32_e32 v0, s2, v0 ; GFX9-NEXT: v_xor_b32_e32 v0, s2, v0
; GFX9-NEXT: v_xor_b32_e32 v0, v0, v1 ; GFX9-NEXT: v_xor_b32_e32 v0, v0, v1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: xor3_vgpr_bc: ; GFX10-LABEL: xor3_vgpr_bc:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_xor3_b32 v0, s2, v0, v1 ; GFX10-NEXT: v_xor3_b32 v0, s2, v0, v1
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = xor i32 %a, %b %x = xor i32 %a, %b
%result = xor i32 %x, %c %result = xor i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @xor3_vgpr_const(i32 %a, i32 %b) { define amdgpu_ps float @xor3_vgpr_const(i32 %a, i32 %b) {
; GFX9-LABEL: xor3_vgpr_const: ; GFX9-LABEL: xor3_vgpr_const:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_xor_b32_e32 v0, v0, v1 ; GFX9-NEXT: v_xor_b32_e32 v0, v0, v1
; GFX9-NEXT: v_xor_b32_e32 v0, 16, v0 ; GFX9-NEXT: v_xor_b32_e32 v0, 16, v0
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: xor3_vgpr_const: ; GFX10-LABEL: xor3_vgpr_const:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_xor3_b32 v0, v0, v1, 16 ; GFX10-NEXT: v_xor3_b32 v0, v0, v1, 16
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = xor i32 %a, %b %x = xor i32 %a, %b
%result = xor i32 %x, 16 %result = xor i32 %x, 16
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps <2 x float> @xor3_multiuse_outer(i32 %a, i32 %b, i32 %c, i32 %x) { define amdgpu_ps <2 x float> @xor3_multiuse_outer(i32 %a, i32 %b, i32 %c, i32 %x) {
; GFX9-LABEL: xor3_multiuse_outer: ; GFX9-LABEL: xor3_multiuse_outer:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_xor_b32_e32 v0, v0, v1 ; GFX9-NEXT: v_xor_b32_e32 v0, v0, v1
; GFX9-NEXT: v_xor_b32_e32 v0, v0, v2 ; GFX9-NEXT: v_xor_b32_e32 v0, v0, v2
; GFX9-NEXT: v_mul_lo_u32 v1, v0, v3 ; GFX9-NEXT: v_mul_lo_u32 v1, v0, v3
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: xor3_multiuse_outer: ; GFX10-LABEL: xor3_multiuse_outer:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_xor3_b32 v0, v0, v1, v2 ; GFX10-NEXT: v_xor3_b32 v0, v0, v1, v2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: v_mul_lo_u32 v1, v0, v3 ; GFX10-NEXT: v_mul_lo_u32 v1, v0, v3
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%inner = xor i32 %a, %b %inner = xor i32 %a, %b
%outer = xor i32 %inner, %c %outer = xor i32 %inner, %c
%x1 = mul i32 %outer, %x %x1 = mul i32 %outer, %x
%r1 = insertelement <2 x i32> undef, i32 %outer, i32 0 %r1 = insertelement <2 x i32> undef, i32 %outer, i32 0
%r0 = insertelement <2 x i32> %r1, i32 %x1, i32 1 %r0 = insertelement <2 x i32> %r1, i32 %x1, i32 1
%bc = bitcast <2 x i32> %r0 to <2 x float> %bc = bitcast <2 x i32> %r0 to <2 x float>
ret <2 x float> %bc ret <2 x float> %bc
} }
define amdgpu_ps <2 x float> @xor3_multiuse_inner(i32 %a, i32 %b, i32 %c) { define amdgpu_ps <2 x float> @xor3_multiuse_inner(i32 %a, i32 %b, i32 %c) {
; GFX9-LABEL: xor3_multiuse_inner: ; GFX9-LABEL: xor3_multiuse_inner:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_xor_b32_e32 v0, v0, v1 ; GFX9-NEXT: v_xor_b32_e32 v0, v0, v1
; GFX9-NEXT: v_xor_b32_e32 v1, v0, v2 ; GFX9-NEXT: v_xor_b32_e32 v1, v0, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: xor3_multiuse_inner: ; GFX10-LABEL: xor3_multiuse_inner:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_xor_b32_e32 v0, v0, v1 ; GFX10-NEXT: v_xor_b32_e32 v0, v0, v1
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: v_xor_b32_e32 v1, v0, v2 ; GFX10-NEXT: v_xor_b32_e32 v1, v0, v2
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%inner = xor i32 %a, %b %inner = xor i32 %a, %b
%outer = xor i32 %inner, %c %outer = xor i32 %inner, %c
%r1 = insertelement <2 x i32> undef, i32 %inner, i32 0 %r1 = insertelement <2 x i32> undef, i32 %inner, i32 0
%r0 = insertelement <2 x i32> %r1, i32 %outer, i32 1 %r0 = insertelement <2 x i32> %r1, i32 %outer, i32 1
%bc = bitcast <2 x i32> %r0 to <2 x float> %bc = bitcast <2 x i32> %r0 to <2 x float>
ret <2 x float> %bc ret <2 x float> %bc
Show All 12 Lines
; GFX9-NEXT: v_xor_b32_e32 v0, v0, v2 ; GFX9-NEXT: v_xor_b32_e32 v0, v0, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: xor3_uniform_vgpr: ; GFX10-LABEL: xor3_uniform_vgpr:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_add_f32_e64 v1, s3, 2.0 ; GFX10-NEXT: v_add_f32_e64 v1, s3, 2.0
; GFX10-NEXT: v_add_f32_e64 v2, s2, 1.0 ; GFX10-NEXT: v_add_f32_e64 v2, s2, 1.0
; GFX10-NEXT: v_add_f32_e64 v0, 0x40400000, s4 ; GFX10-NEXT: v_add_f32_e64 v0, 0x40400000, s4
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: v_xor_b32_e32 v1, v2, v1 ; GFX10-NEXT: v_xor_b32_e32 v1, v2, v1
; GFX10-NEXT: v_xor_b32_e32 v0, v1, v0 ; GFX10-NEXT: v_xor_b32_e32 v0, v1, v0
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%a1 = fadd float %a, 1.0 %a1 = fadd float %a, 1.0
%b2 = fadd float %b, 2.0 %b2 = fadd float %b, 2.0
%c3 = fadd float %c, 3.0 %c3 = fadd float %c, 3.0
%bc.a = bitcast float %a1 to i32 %bc.a = bitcast float %a1 to i32
%bc.b = bitcast float %b2 to i32 %bc.b = bitcast float %b2 to i32
%bc.c = bitcast float %c3 to i32 %bc.c = bitcast float %c3 to i32
%x = xor i32 %bc.a, %bc.b %x = xor i32 %bc.a, %bc.b
%result = xor i32 %x, %bc.c %result = xor i32 %x, %bc.c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }

test/CodeGen/AMDGPU/xor_add.ll

Show All 16 Lines
; GFX9-LABEL: xor_add: ; GFX9-LABEL: xor_add:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_xad_u32 v0, v0, v1, v2 ; GFX9-NEXT: v_xad_u32 v0, v0, v1, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: xor_add: ; GFX10-LABEL: xor_add:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_xad_u32 v0, v0, v1, v2 ; GFX10-NEXT: v_xad_u32 v0, v0, v1, v2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = xor i32 %a, %b %x = xor i32 %a, %b
%result = add i32 %x, %c %result = add i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
; ThreeOp instruction variant not used due to Constant Bus Limitations ; ThreeOp instruction variant not used due to Constant Bus Limitations
define amdgpu_ps float @xor_add_vgpr_a(i32 %a, i32 inreg %b, i32 inreg %c) { define amdgpu_ps float @xor_add_vgpr_a(i32 %a, i32 inreg %b, i32 inreg %c) {
; VI-LABEL: xor_add_vgpr_a: ; VI-LABEL: xor_add_vgpr_a:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_xor_b32_e32 v0, s2, v0 ; VI-NEXT: v_xor_b32_e32 v0, s2, v0
; VI-NEXT: v_add_u32_e32 v0, vcc, s3, v0 ; VI-NEXT: v_add_u32_e32 v0, vcc, s3, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: xor_add_vgpr_a: ; GFX9-LABEL: xor_add_vgpr_a:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_xor_b32_e32 v0, s2, v0 ; GFX9-NEXT: v_xor_b32_e32 v0, s2, v0
; GFX9-NEXT: v_add_u32_e32 v0, s3, v0 ; GFX9-NEXT: v_add_u32_e32 v0, s3, v0
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: xor_add_vgpr_a: ; GFX10-LABEL: xor_add_vgpr_a:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_xad_u32 v0, v0, s2, s3 ; GFX10-NEXT: v_xad_u32 v0, v0, s2, s3
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = xor i32 %a, %b %x = xor i32 %a, %b
%result = add i32 %x, %c %result = add i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @xor_add_vgpr_all(i32 %a, i32 %b, i32 %c) { define amdgpu_ps float @xor_add_vgpr_all(i32 %a, i32 %b, i32 %c) {
; VI-LABEL: xor_add_vgpr_all: ; VI-LABEL: xor_add_vgpr_all:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_xor_b32_e32 v0, v0, v1 ; VI-NEXT: v_xor_b32_e32 v0, v0, v1
; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v2 ; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v2
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: xor_add_vgpr_all: ; GFX9-LABEL: xor_add_vgpr_all:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_xad_u32 v0, v0, v1, v2 ; GFX9-NEXT: v_xad_u32 v0, v0, v1, v2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: xor_add_vgpr_all: ; GFX10-LABEL: xor_add_vgpr_all:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_xad_u32 v0, v0, v1, v2 ; GFX10-NEXT: v_xad_u32 v0, v0, v1, v2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = xor i32 %a, %b %x = xor i32 %a, %b
%result = add i32 %x, %c %result = add i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @xor_add_vgpr_ab(i32 %a, i32 %b, i32 inreg %c) { define amdgpu_ps float @xor_add_vgpr_ab(i32 %a, i32 %b, i32 inreg %c) {
; VI-LABEL: xor_add_vgpr_ab: ; VI-LABEL: xor_add_vgpr_ab:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_xor_b32_e32 v0, v0, v1 ; VI-NEXT: v_xor_b32_e32 v0, v0, v1
; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v0 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v0
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: xor_add_vgpr_ab: ; GFX9-LABEL: xor_add_vgpr_ab:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_xad_u32 v0, v0, v1, s2 ; GFX9-NEXT: v_xad_u32 v0, v0, v1, s2
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: xor_add_vgpr_ab: ; GFX10-LABEL: xor_add_vgpr_ab:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_xad_u32 v0, v0, v1, s2 ; GFX10-NEXT: v_xad_u32 v0, v0, v1, s2
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = xor i32 %a, %b %x = xor i32 %a, %b
%result = add i32 %x, %c %result = add i32 %x, %c
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }
define amdgpu_ps float @xor_add_vgpr_const(i32 %a, i32 %b) { define amdgpu_ps float @xor_add_vgpr_const(i32 %a, i32 %b) {
; VI-LABEL: xor_add_vgpr_const: ; VI-LABEL: xor_add_vgpr_const:
; VI: ; %bb.0: ; VI: ; %bb.0:
; VI-NEXT: v_xor_b32_e32 v0, 3, v0 ; VI-NEXT: v_xor_b32_e32 v0, 3, v0
; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1 ; VI-NEXT: v_add_u32_e32 v0, vcc, v0, v1
; VI-NEXT: ; return to shader part epilog ; VI-NEXT: ; return to shader part epilog
; ;
; GFX9-LABEL: xor_add_vgpr_const: ; GFX9-LABEL: xor_add_vgpr_const:
; GFX9: ; %bb.0: ; GFX9: ; %bb.0:
; GFX9-NEXT: v_xad_u32 v0, v0, 3, v1 ; GFX9-NEXT: v_xad_u32 v0, v0, 3, v1
; GFX9-NEXT: ; return to shader part epilog ; GFX9-NEXT: ; return to shader part epilog
; ;
; GFX10-LABEL: xor_add_vgpr_const: ; GFX10-LABEL: xor_add_vgpr_const:
; GFX10: ; %bb.0: ; GFX10: ; %bb.0:
; GFX10-NEXT: v_xad_u32 v0, v0, 3, v1 ; GFX10-NEXT: v_xad_u32 v0, v0, 3, v1
; GFX10-NEXT: ; implicit-def: $vcc_hi
; GFX10-NEXT: ; return to shader part epilog ; GFX10-NEXT: ; return to shader part epilog
%x = xor i32 %a, 3 %x = xor i32 %a, 3
%result = add i32 %x, %b %result = add i32 %x, %b
%bc = bitcast i32 %result to float %bc = bitcast i32 %result to float
ret float %bc ret float %bc
} }

test/MC/AMDGPU/expressions-gfx10.s

// RUN: llvm-mc -arch=amdgcn -mcpu=gfx1010 -show-encoding %s | FileCheck %s --check-prefix=GFX10 // RUN: llvm-mc -arch=amdgcn -mcpu=gfx1010 -mattr=-WavefrontSize32,+WavefrontSize64 -show-encoding %s | FileCheck %s --check-prefix=GFX10
i1=1 i1=1
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Constant expressions may be used where literals are accepted. // Constant expressions may be used where literals are accepted.
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
v_bfe_u32 v0, i1+100, v1, v2 v_bfe_u32 v0, i1+100, v1, v2
Show All 40 Lines

test/MC/AMDGPU/gfx10-constant-bus.s

Show All 27 Lines
v_div_fmas_f64 v[5:6], 0x12345678, 0x12345678, 0x12345678 v_div_fmas_f64 v[5:6], 0x12345678, 0x12345678, 0x12345678
// GFX10: v_div_fmas_f64 v[5:6], 0x12345678, 0x12345678, 0x12345678 ; encoding: [0x05,0x00,0x70,0xd5,0xff,0xfe,0xfd,0x03,0x78,0x56,0x34,0x12] // GFX10: v_div_fmas_f64 v[5:6], 0x12345678, 0x12345678, 0x12345678 ; encoding: [0x05,0x00,0x70,0xd5,0xff,0xfe,0xfd,0x03,0x78,0x56,0x34,0x12]
v_div_fmas_f64 v[5:6], v[1:2], s[2:3], v[3:4] v_div_fmas_f64 v[5:6], v[1:2], s[2:3], v[3:4]
// GFX10: v_div_fmas_f64 v[5:6], v[1:2], s[2:3], v[3:4] ; encoding: [0x05,0x00,0x70,0xd5,0x01,0x05,0x0c,0x04] // GFX10: v_div_fmas_f64 v[5:6], v[1:2], s[2:3], v[3:4] ; encoding: [0x05,0x00,0x70,0xd5,0x01,0x05,0x0c,0x04]
v_div_fmas_f64 v[5:6], v[1:2], s[2:3], 0x123456 v_div_fmas_f64 v[5:6], v[1:2], s[2:3], 0x123456
// GFX10-ERR: error: invalid operand (violates constant bus restrictions) // GFX10-ERR: error: invalid operand (violates constant bus restrictions)
//-----------------------------------------------------------------------------------------
// v_mad_u64_u32 has operands of different sizes.
// When these operands are literals, they are counted as 2 scalar values even if literals are identical.
v_mad_u64_u32 v[5:6], s12, v1, 0x12345678, 0x12345678
// GFX10: v_mad_u64_u32 v[5:6], s12, v1, 0x12345678, 0x12345678 ; encoding: [0x05,0x0c,0x76,0xd5,0x01,0xff,0xfd,0x03,0x78,0x56,0x34,0x12]
v_mad_u64_u32 v[5:6], s12, s1, 0x12345678, 0x12345678
// GFX10-ERR: error: invalid operand (violates constant bus restrictions)

test/MC/AMDGPU/wave32.s

  • This file was added.
// RUN: not llvm-mc -arch=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -show-encoding %s | FileCheck -check-prefix=GFX1032 %s
// RUN: not llvm-mc -arch=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -show-encoding %s | FileCheck -check-prefix=GFX1064 %s
// RUN: not llvm-mc -arch=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -show-encoding %s 2>&1 | FileCheck -check-prefix=GFX1032-ERR %s
// RUN: not llvm-mc -arch=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -show-encoding %s 2>&1 | FileCheck -check-prefix=GFX1064-ERR %s
v_cmp_ge_i32_e32 s0, v0
// GFX1032: v_cmp_ge_i32_e32 vcc_lo, s0, v0 ; encoding: [0x00,0x00,0x0c,0x7d]
// GFX1064: v_cmp_ge_i32_e32 vcc, s0, v0 ; encoding: [0x00,0x00,0x0c,0x7d]
v_cmp_ge_i32_e32 vcc_lo, s0, v1
// GFX1032: v_cmp_ge_i32_e32 vcc_lo, s0, v1 ; encoding: [0x00,0x02,0x0c,0x7d]
// GFX1064-ERR: error: instruction not supported on this GPU
v_cmp_ge_i32_e32 vcc, s0, v2
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_cmp_ge_i32_e32 vcc, s0, v2 ; encoding: [0x00,0x04,0x0c,0x7d]
v_cmp_le_f16_sdwa s0, v3, v4 src0_sel:WORD_1 src1_sel:DWORD
// GFX1032: v_cmp_le_f16_sdwa s0, v3, v4 src0_sel:WORD_1 src1_sel:DWORD ; encoding: [0xf9,0x08,0x96,0x7d,0x03,0x80,0x05,0x06]
// GFX1064-ERR: error: invalid operand for instruction
v_cmp_le_f16_sdwa s[0:1], v3, v4 src0_sel:WORD_1 src1_sel:DWORD
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_cmp_le_f16_sdwa s[0:1], v3, v4 src0_sel:WORD_1 src1_sel:DWORD ; encoding: [0xf9,0x08,0x96,0x7d,0x03,0x80,0x05,0x06]
v_cmp_class_f32_e32 vcc_lo, s0, v0
// GFX1032: v_cmp_class_f32_e32 vcc_lo, s0, v0 ; encoding: [0x00,0x00,0x10,0x7d]
// GFX1064-ERR: error: instruction not supported on this GPU
v_cmp_class_f32_e32 vcc, s0, v0
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_cmp_class_f32_e32 vcc, s0, v0 ; encoding: [0x00,0x00,0x10,0x7d]
// TODO-GFX10: The following encoding does not match SP3's encoding, which is:
// [0xf9,0x04,0x1e,0x7d,0x01,0x06,0x06,0x06]
v_cmp_class_f16_sdwa vcc_lo, v1, v2 src0_sel:DWORD src1_sel:DWORD
// GFX1032: v_cmp_class_f16_sdwa vcc_lo, v1, v2 src0_sel:DWORD src1_sel:DWORD ; encoding: [0xf9,0x04,0x1e,0x7d,0x01,0x00,0x06,0x06]
// GFX1064-ERR: error: invalid operand for instruction
// TODO-GFX10: The following encoding does not match SP3's encoding, which is:
// [0xf9,0x04,0x1e,0x7d,0x01,0x06,0x06,0x06]
v_cmp_class_f16_sdwa vcc, v1, v2 src0_sel:DWORD src1_sel:DWORD
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_cmp_class_f16_sdwa vcc, v1, v2 src0_sel:DWORD src1_sel:DWORD ; encoding: [0xf9,0x04,0x1e,0x7d,0x01,0x00,0x06,0x06]
v_cmp_class_f16_sdwa s0, v1, v2 src0_sel:DWORD src1_sel:DWORD
// GFX1032: v_cmp_class_f16_sdwa s0, v1, v2 src0_sel:DWORD src1_sel:DWORD ; encoding: [0xf9,0x04,0x1e,0x7d,0x01,0x80,0x06,0x06]
// GFX1064-ERR: error: invalid operand for instruction
v_cmp_class_f16_sdwa s[0:1], v1, v2 src0_sel:DWORD src1_sel:DWORD
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_cmp_class_f16_sdwa s[0:1], v1, v2 src0_sel:DWORD src1_sel:DWORD ; encoding: [0xf9,0x04,0x1e,0x7d,0x01,0x80,0x06,0x06]
v_cndmask_b32_e32 v1, v2, v3,
// GFX1032: v_cndmask_b32_e32 v1, v2, v3, vcc_lo ; encoding: [0x02,0x07,0x02,0x02]
// GFX1064: v_cndmask_b32_e32 v1, v2, v3, vcc ; encoding: [0x02,0x07,0x02,0x02]
v_cndmask_b32_e32 v1, v2, v3, vcc_lo
// GFX1032: v_cndmask_b32_e32 v1, v2, v3, vcc_lo ; encoding: [0x02,0x07,0x02,0x02]
// GFX1064-ERR: error: instruction not supported on this GPU
v_cndmask_b32_e32 v1, v2, v3, vcc
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_cndmask_b32_e32 v1, v2, v3, vcc ; encoding: [0x02,0x07,0x02,0x02]
v_add_co_u32_e32 v2, vcc_lo, s0, v2
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064-ERR: error: instruction not supported on this GPU
v_add_co_u32_e32 v2, vcc, s0, v2
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064-ERR: error: instruction not supported on this GPU
v_add_co_ci_u32_e32 v3, vcc_lo, v3, v4, vcc_lo
// GFX1032: v_add_co_ci_u32_e32 v3, vcc_lo, v3, v4, vcc_lo ; encoding: [0x03,0x09,0x06,0x50]
// GFX1064-ERR: error: instruction not supported on this GPU
v_add_co_ci_u32_e32 v3, vcc, v3, v4, vcc
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_add_co_ci_u32_e32 v3, vcc, v3, v4, vcc ; encoding: [0x03,0x09,0x06,0x50]
v_add_co_ci_u32_e32 v3, v3, v4
// GFX1032: v_add_co_ci_u32_e32 v3, vcc_lo, v3, v4, vcc_lo ; encoding: [0x03,0x09,0x06,0x50]
// GFX1064: v_add_co_ci_u32_e32 v3, vcc, v3, v4, vcc ; encoding: [0x03,0x09,0x06,0x50]
v_sub_co_u32_e32 v2, vcc_lo, s0, v2
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064-ERR: error: instruction not supported on this GPU
v_sub_co_u32_e32 v2, vcc, s0, v2
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064-ERR: error: instruction not supported on this GPU
v_subrev_co_u32_e32 v2, vcc_lo, s0, v2
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064-ERR: error: instruction not supported on this GPU
v_subrev_co_u32_e32 v2, vcc, s0, v2
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064-ERR: error: instruction not supported on this GPU
v_sub_co_ci_u32_e32 v3, vcc_lo, v3, v4, vcc_lo
// GFX1032: v_sub_co_ci_u32_e32 v3, vcc_lo, v3, v4, vcc_lo ; encoding: [0x03,0x09,0x06,0x52]
// GFX1064-ERR: error: instruction not supported on this GPU
v_sub_co_ci_u32_e32 v3, vcc, v3, v4, vcc
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_sub_co_ci_u32_e32 v3, vcc, v3, v4, vcc ; encoding: [0x03,0x09,0x06,0x52]
v_sub_co_ci_u32_e32 v3, v3, v4
// GFX1032: v_sub_co_ci_u32_e32 v3, vcc_lo, v3, v4, vcc_lo ; encoding: [0x03,0x09,0x06,0x52]
// GFX1064: v_sub_co_ci_u32_e32 v3, vcc, v3, v4, vcc ; encoding: [0x03,0x09,0x06,0x52]
v_subrev_co_ci_u32_e32 v1, vcc_lo, 0, v1, vcc_lo
// GFX1032: v_subrev_co_ci_u32_e32 v1, vcc_lo, 0, v1, vcc_lo ; encoding: [0x80,0x02,0x02,0x54]
// GFX1064-ERR: error: instruction not supported on this GPU
v_subrev_co_ci_u32_e32 v1, vcc, 0, v1, vcc
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_subrev_co_ci_u32_e32 v1, vcc, 0, v1, vcc ; encoding: [0x80,0x02,0x02,0x54]
v_subrev_co_ci_u32_e32 v1, 0, v1
// GFX1032: v_subrev_co_ci_u32_e32 v1, vcc_lo, 0, v1, vcc_lo ; encoding: [0x80,0x02,0x02,0x54]
// GFX1064: v_subrev_co_ci_u32_e32 v1, vcc, 0, v1, vcc ; encoding: [0x80,0x02,0x02,0x54]
v_add_co_u32_sdwa v0, vcc_lo, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
// GFX1032-ERR: error: invalid operand
// GFX1064-ERR: error: invalid operand
v_add_co_u32_sdwa v0, vcc, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
// GFX1032-ERR: error: instruction not supported
// GFX1064-ERR: error: instruction not supported
v_add_co_u32_sdwa v0, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
// GFX1032-ERR: error: not a valid operand
// GFX1064-ERR: error: not a valid operand
v_add_co_ci_u32_sdwa v1, vcc_lo, v1, v4, vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
// GFX1032: v_add_co_ci_u32_sdwa v1, vcc_lo, v1, v4, vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x50,0x01,0x06,0x00,0x06]
// GFX1064-ERR: error: instruction not supported on this GPU
v_add_co_ci_u32_sdwa v1, vcc, v1, v4, vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_add_co_ci_u32_sdwa v1, vcc, v1, v4, vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x50,0x01,0x06,0x00,0x06]
v_add_co_ci_u32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
// GFX1032: v_add_co_ci_u32_sdwa v1, vcc_lo, v1, v4, vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x50,0x01,0x06,0x00,0x06]
// GFX1064: v_add_co_ci_u32_sdwa v1, vcc, v1, v4, vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x50,0x01,0x06,0x00,0x06]
v_sub_co_u32_sdwa v0, vcc_lo, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
// GFX1032-ERR: error: invalid operand
// GFX1064-ERR: error: invalid operand
v_sub_co_u32_sdwa v0, vcc, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
// GFX1032-ERR: error: instruction not supported
// GFX1064-ERR: error: instruction not supported
v_sub_co_u32_sdwa v0, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
// GFX1032-ERR: error: not a valid operand
// GFX1064-ERR: error: not a valid operand
v_subrev_co_u32_sdwa v0, vcc_lo, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
// GFX1032-ERR: error: invalid operand
// GFX1064-ERR: error: invalid operand
v_subrev_co_u32_sdwa v0, vcc, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
// GFX1032-ERR: error: instruction not supported
// GFX1064-ERR: error: instruction not supported
v_subrev_co_u32_sdwa v0, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
// GFX1032-ERR: error: not a valid operand
// GFX1064-ERR: error: not a valid operand
v_sub_co_ci_u32_sdwa v1, vcc_lo, v1, v4, vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
// GFX1032: v_sub_co_ci_u32_sdwa v1, vcc_lo, v1, v4, vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x52,0x01,0x06,0x00,0x06]
// GFX1064-ERR: error: instruction not supported on this GPU
v_sub_co_ci_u32_sdwa v1, vcc, v1, v4, vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_sub_co_ci_u32_sdwa v1, vcc, v1, v4, vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x52,0x01,0x06,0x00,0x06]
v_sub_co_ci_u32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
// GFX1032: v_sub_co_ci_u32_sdwa v1, vcc_lo, v1, v4, vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x52,0x01,0x06,0x00,0x06]
// GFX1064: v_sub_co_ci_u32_sdwa v1, vcc, v1, v4, vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x52,0x01,0x06,0x00,0x06]
v_subrev_co_ci_u32_sdwa v1, vcc_lo, v1, v4, vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
// GFX1032: v_subrev_co_ci_u32_sdwa v1, vcc_lo, v1, v4, vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x54,0x01,0x06,0x00,0x06]
// GFX1064-ERR: error: instruction not supported on this GPU
v_subrev_co_ci_u32_sdwa v1, vcc, v1, v4, vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_subrev_co_ci_u32_sdwa v1, vcc, v1, v4, vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x54,0x01,0x06,0x00,0x06]
v_subrev_co_ci_u32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
// GFX1032: v_subrev_co_ci_u32_sdwa v1, vcc_lo, v1, v4, vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x54,0x01,0x06,0x00,0x06]
// GFX1064: v_subrev_co_ci_u32_sdwa v1, vcc, v1, v4, vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x54,0x01,0x06,0x00,0x06]
v_add_co_ci_u32 v1, sext(v1), sext(v4) dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
// GFX1032: v_add_co_ci_u32_sdwa v1, vcc_lo, sext(v1), sext(v4), vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x50,0x01,0x06,0x08,0x0e]
// GFX1064: v_add_co_ci_u32_sdwa v1, vcc, sext(v1), sext(v4), vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x50,0x01,0x06,0x08,0x0e]
v_add_co_ci_u32_sdwa v1, vcc_lo, sext(v1), sext(v4), vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
// GFX1032: v_add_co_ci_u32_sdwa v1, vcc_lo, sext(v1), sext(v4), vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x50,0x01,0x06,0x08,0x0e]
// GFX1064-ERR: error: instruction not supported on this GPU
v_add_co_ci_u32_sdwa v1, vcc, sext(v1), sext(v4), vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_add_co_ci_u32_sdwa v1, vcc, sext(v1), sext(v4), vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD ; encoding: [0xf9,0x08,0x02,0x50,0x01,0x06,0x08,0x0e]
v_add_co_u32_dpp v5, v1, v2 quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032-ERR: error: not a valid operand
// GFX1064-ERR: error: not a valid operand
v_add_co_u32_dpp v5, vcc_lo, v1, v2 quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032-ERR: error: not a valid operand
// GFX1064-ERR: error: not a valid operand
v_add_co_u32_dpp v5, vcc, v1, v2 quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032-ERR: error: not a valid operand
// GFX1064-ERR: error: not a valid operand
v_add_co_ci_u32_dpp v5, v1, v2 quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032: v_add_co_ci_u32_dpp v5, vcc_lo, v1, v2, vcc_lo quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0 ; encoding: [0xfa,0x04,0x0a,0x50,0x01,0xe4,0x00,0x00]
// GFX1064: v_add_co_ci_u32_dpp v5, vcc, v1, v2, vcc quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0 ; encoding: [0xfa,0x04,0x0a,0x50,0x01,0xe4,0x00,0x00]
v_add_co_ci_u32_dpp v5, vcc_lo, v1, v2, vcc_lo quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032: v_add_co_ci_u32_dpp v5, vcc_lo, v1, v2, vcc_lo quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0 ; encoding: [0xfa,0x04,0x0a,0x50,0x01,0xe4,0x00,0x00]
// GFX1064-ERR: error: instruction not supported on this GPU
v_add_co_ci_u32_dpp v5, vcc, v1, v2, vcc quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_add_co_ci_u32_dpp v5, vcc, v1, v2, vcc quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0 ; encoding: [0xfa,0x04,0x0a,0x50,0x01,0xe4,0x00,0x00]
v_sub_co_u32_dpp v5, v1, v2 quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032-ERR: error: not a valid operand
// GFX1064-ERR: error: not a valid operand
v_sub_co_u32_dpp v5, vcc_lo, v1, v2 quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032-ERR: error: not a valid operand
// GFX1064-ERR: error: not a valid operand
v_sub_co_u32_dpp v5, vcc, v1, v2 quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032-ERR: error: not a valid operand
// GFX1064-ERR: error: not a valid operand
v_sub_co_ci_u32_dpp v5, vcc_lo, v1, v2, vcc_lo quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032: v_sub_co_ci_u32_dpp v5, vcc_lo, v1, v2, vcc_lo quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0 ; encoding: [0xfa,0x04,0x0a,0x52,0x01,0xe4,0x00,0x00]
// GFX1064-ERR: error: instruction not supported on this GPU
v_sub_co_ci_u32_dpp v5, vcc, v1, v2, vcc quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_sub_co_ci_u32_dpp v5, vcc, v1, v2, vcc quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0 ; encoding: [0xfa,0x04,0x0a,0x52,0x01,0xe4,0x00,0x00]
v_subrev_co_u32_dpp v5, v1, v2 quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032-ERR: error: not a valid operand
// GFX1064-ERR: error: not a valid operand
v_subrev_co_u32_dpp v5, vcc_lo, v1, v2 quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032-ERR: error: not a valid operand
// GFX1064-ERR: error: not a valid operand
v_subrev_co_u32_dpp v5, vcc, v1, v2 quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032-ERR: error: not a valid operand
// GFX1064-ERR: error: not a valid operand
v_subrev_co_ci_u32_dpp v5, vcc_lo, v1, v2, vcc_lo quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032: v_subrev_co_ci_u32_dpp v5, vcc_lo, v1, v2, vcc_lo quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0 ; encoding: [0xfa,0x04,0x0a,0x54,0x01,0xe4,0x00,0x00]
// GFX1064-ERR: error: instruction not supported on this GPU
v_subrev_co_ci_u32_dpp v5, vcc, v1, v2, vcc quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
// GFX1032-ERR: error: instruction not supported on this GPU
// GFX1064: v_subrev_co_ci_u32_dpp v5, vcc, v1, v2, vcc quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0 ; encoding: [0xfa,0x04,0x0a,0x54,0x01,0xe4,0x00,0x00]
v_add_co_u32 v0, s0, v0, v2
// GFX1032: v_add_co_u32_e64 v0, s0, v0, v2 ; encoding: [0x00,0x00,0x0f,0xd7,0x00,0x05,0x02,0x00]
// GFX1064-ERR: error: invalid operand for instruction
v_add_co_u32_e64 v0, s0, v0, v2
// GFX1032: v_add_co_u32_e64 v0, s0, v0, v2 ; encoding: [0x00,0x00,0x0f,0xd7,0x00,0x05,0x02,0x00]
// GFX1064-ERR: error: invalid operand for instruction
v_add_co_ci_u32_e64 v4, s0, v1, v5, s2
// GFX1032: v_add_co_ci_u32_e64 v4, s0, v1, v5, s2 ; encoding: [0x04,0x00,0x28,0xd5,0x01,0x0b,0x0a,0x00]
// GFX1064-ERR: error: invalid operand for instruction
v_sub_co_u32 v0, s0, v0, v2
// GFX1032: v_sub_co_u32_e64 v0, s0, v0, v2 ; encoding: [0x00,0x00,0x10,0xd7,0x00,0x05,0x02,0x00]
// GFX1064-ERR: error: invalid operand for instruction
v_sub_co_u32_e64 v0, s0, v0, v2
// GFX1032: v_sub_co_u32_e64 v0, s0, v0, v2 ; encoding: [0x00,0x00,0x10,0xd7,0x00,0x05,0x02,0x00]
// GFX1064-ERR: error: invalid operand for instruction
v_sub_co_ci_u32_e64 v4, s0, v1, v5, s2
// GFX1032: v_sub_co_ci_u32_e64 v4, s0, v1, v5, s2 ; encoding: [0x04,0x00,0x29,0xd5,0x01,0x0b,0x0a,0x00]
// GFX1064-ERR: error: invalid operand for instruction
v_subrev_co_u32 v0, s0, v0, v2
// GFX1032: v_subrev_co_u32_e64 v0, s0, v0, v2 ; encoding: [0x00,0x00,0x19,0xd7,0x00,0x05,0x02,0x00]
// GFX1064-ERR: error: invalid operand for instruction
v_subrev_co_u32_e64 v0, s0, v0, v2
// GFX1032: v_subrev_co_u32_e64 v0, s0, v0, v2 ; encoding: [0x00,0x00,0x19,0xd7,0x00,0x05,0x02,0x00]
// GFX1064-ERR: error: invalid operand for instruction
v_subrev_co_ci_u32_e64 v4, s0, v1, v5, s2
// GFX1032: v_subrev_co_ci_u32_e64 v4, s0, v1, v5, s2 ; encoding: [0x04,0x00,0x2a,0xd5,0x01,0x0b,0x0a,0x00]
// GFX1064-ERR: error: invalid operand for instruction
v_add_co_u32 v0, s[0:1], v0, v2
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_add_co_u32_e64 v0, s[0:1], v0, v2 ; encoding: [0x00,0x00,0x0f,0xd7,0x00,0x05,0x02,0x00]
v_add_co_u32_e64 v0, s[0:1], v0, v2
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_add_co_u32_e64 v0, s[0:1], v0, v2 ; encoding: [0x00,0x00,0x0f,0xd7,0x00,0x05,0x02,0x00]
v_add_co_ci_u32_e64 v4, s[0:1], v1, v5, s[2:3]
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_add_co_ci_u32_e64 v4, s[0:1], v1, v5, s[2:3] ; encoding: [0x04,0x00,0x28,0xd5,0x01,0x0b,0x0a,0x00]
v_sub_co_u32 v0, s[0:1], v0, v2
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_sub_co_u32_e64 v0, s[0:1], v0, v2 ; encoding: [0x00,0x00,0x10,0xd7,0x00,0x05,0x02,0x00]
v_sub_co_u32_e64 v0, s[0:1], v0, v2
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_sub_co_u32_e64 v0, s[0:1], v0, v2 ; encoding: [0x00,0x00,0x10,0xd7,0x00,0x05,0x02,0x00]
v_sub_co_ci_u32_e64 v4, s[0:1], v1, v5, s[2:3]
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_sub_co_ci_u32_e64 v4, s[0:1], v1, v5, s[2:3] ; encoding: [0x04,0x00,0x29,0xd5,0x01,0x0b,0x0a,0x00]
v_subrev_co_u32 v0, s[0:1], v0, v2
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_subrev_co_u32_e64 v0, s[0:1], v0, v2 ; encoding: [0x00,0x00,0x19,0xd7,0x00,0x05,0x02,0x00]
v_subrev_co_u32_e64 v0, s[0:1], v0, v2
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_subrev_co_u32_e64 v0, s[0:1], v0, v2 ; encoding: [0x00,0x00,0x19,0xd7,0x00,0x05,0x02,0x00]
v_subrev_co_ci_u32_e64 v4, s[0:1], v1, v5, s[2:3]
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_subrev_co_ci_u32_e64 v4, s[0:1], v1, v5, s[2:3] ; encoding: [0x04,0x00,0x2a,0xd5,0x01,0x0b,0x0a,0x00]
v_add_co_ci_u32_e64 v4, vcc_lo, v1, v5, s2
// GFX1032: v_add_co_ci_u32_e64 v4, vcc_lo, v1, v5, s2 ; encoding: [0x04,0x6a,0x28,0xd5,0x01,0x0b,0x0a,0x00]
// GFX1064-ERR: error: invalid operand for instruction
v_add_co_ci_u32_e64 v4, vcc, v1, v5, s[2:3]
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_add_co_ci_u32_e64 v4, vcc, v1, v5, s[2:3] ; encoding: [0x04,0x6a,0x28,0xd5,0x01,0x0b,0x0a,0x00]
v_add_co_ci_u32_e64 v4, s0, v1, v5, vcc_lo
// GFX1032: v_add_co_ci_u32_e64 v4, s0, v1, v5, vcc_lo ; encoding: [0x04,0x00,0x28,0xd5,0x01,0x0b,0xaa,0x01]
// GFX1064-ERR: error: invalid operand for instruction
v_add_co_ci_u32_e64 v4, s[0:1], v1, v5, vcc
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_add_co_ci_u32_e64 v4, s[0:1], v1, v5, vcc ; encoding: [0x04,0x00,0x28,0xd5,0x01,0x0b,0xaa,0x01]
v_div_scale_f32 v2, s2, v0, v0, v2
// GFX1032: v_div_scale_f32 v2, s2, v0, v0, v2 ; encoding: [0x02,0x02,0x6d,0xd5,0x00,0x01,0x0a,0x04]
// GFX1064-ERR: error: invalid operand for instruction
v_div_scale_f32 v2, s[2:3], v0, v0, v2
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_div_scale_f32 v2, s[2:3], v0, v0, v2 ; encoding: [0x02,0x02,0x6d,0xd5,0x00,0x01,0x0a,0x04]
v_div_scale_f64 v[2:3], s2, v[0:1], v[0:1], v[2:3]
// GFX1032: v_div_scale_f64 v[2:3], s2, v[0:1], v[0:1], v[2:3] ; encoding: [0x02,0x02,0x6e,0xd5,0x00,0x01,0x0a,0x04]
// GFX1064-ERR: error: invalid operand for instruction
v_div_scale_f64 v[2:3], s[2:3], v[0:1], v[0:1], v[2:3]
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_div_scale_f64 v[2:3], s[2:3], v[0:1], v[0:1], v[2:3] ; encoding: [0x02,0x02,0x6e,0xd5,0x00,0x01,0x0a,0x04]
v_mad_i64_i32 v[0:1], s6, v0, v1, v[2:3]
// GFX1032: v_mad_i64_i32 v[0:1], s6, v0, v1, v[2:3] ; encoding: [0x00,0x06,0x77,0xd5,0x00,0x03,0x0a,0x04]
// GFX1064-ERR: error: invalid operand for instruction
v_mad_i64_i32 v[0:1], s[6:7], v0, v1, v[2:3]
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_mad_i64_i32 v[0:1], s[6:7], v0, v1, v[2:3] ; encoding: [0x00,0x06,0x77,0xd5,0x00,0x03,0x0a,0x04]
v_mad_u64_u32 v[0:1], s6, v0, v1, v[2:3]
// GFX1032: v_mad_u64_u32 v[0:1], s6, v0, v1, v[2:3] ; encoding: [0x00,0x06,0x76,0xd5,0x00,0x03,0x0a,0x04]
// GFX1064-ERR: error: invalid operand for instruction
v_mad_u64_u32 v[0:1], s[6:7], v0, v1, v[2:3]
// GFX1032-ERR: error: invalid operand for instruction
// GFX1064: v_mad_u64_u32 v[0:1], s[6:7], v0, v1, v[2:3] ; encoding: [0x00,0x06,0x76,0xd5,0x00,0x03,0x0a,0x04]
v_cmpx_neq_f32_e32 v0, v1
// GFX1032: v_cmpx_neq_f32_e32 v0, v1 ; encoding: [0x00,0x03,0x3a,0x7c]
// GFX1064: v_cmpx_neq_f32_e32 v0, v1 ; encoding: [0x00,0x03,0x3a,0x7c]
v_cmpx_neq_f32_sdwa v0, v1 src0_sel:WORD_1 src1_sel:DWORD
// GFX1032: v_cmpx_neq_f32_sdwa v0, v1 src0_sel:WORD_1 src1_sel:DWORD ; encoding: [0xf9,0x02,0x3a,0x7c,0x00,0x00,0x05,0x06]
// GFX1064: v_cmpx_neq_f32_sdwa v0, v1 src0_sel:WORD_1 src1_sel:DWORD ; encoding: [0xf9,0x02,0x3a,0x7c,0x00,0x00,0x05,0x06]
v_cmpx_eq_u32_sdwa v0, 1 src0_sel:WORD_1 src1_sel:DWORD
// GFX1032: v_cmpx_eq_u32_sdwa v0, 1 src0_sel:WORD_1 src1_sel:DWORD ; encoding: [0xf9,0x02,0xa5,0x7d,0x00,0x00,0x05,0x86]
// GFX1064: v_cmpx_eq_u32_sdwa v0, 1 src0_sel:WORD_1 src1_sel:DWORD ; encoding: [0xf9,0x02,0xa5,0x7d,0x00,0x00,0x05,0x86]
v_cmpx_class_f32_e64 v0, 1
// GFX1032: v_cmpx_class_f32_e64 v0, 1 ; encoding: [0x00,0x00,0x98,0xd4,0x00,0x03,0x01,0x00]
// GFX1064: v_cmpx_class_f32_e64 v0, 1 ; encoding: [0x00,0x00,0x98,0xd4,0x00,0x03,0x01,0x00]
v_cmpx_class_f32_sdwa v0, 1 src0_sel:WORD_1 src1_sel:DWORD
// GFX1032: v_cmpx_class_f32_sdwa v0, 1 src0_sel:WORD_1 src1_sel:DWORD ; encoding: [0xf9,0x02,0x31,0x7d,0x00,0x00,0x05,0x86]
// GFX1064: v_cmpx_class_f32_sdwa v0, 1 src0_sel:WORD_1 src1_sel:DWORD ; encoding: [0xf9,0x02,0x31,0x7d,0x00,0x00,0x05,0x86]

test/MC/Disassembler/AMDGPU/gfx10-sgpr-max.txt

# RUN: llvm-mc -arch=amdgcn -mcpu=gfx1010 -disassemble -show-encoding < %s | FileCheck -check-prefix=GFX10 %s # RUN: llvm-mc -arch=amdgcn -mcpu=gfx1010 -disassemble -show-encoding < %s | FileCheck -check-prefix=GFX10 %s
# GFX10: v_mov_b32_e32 v0, s105 ; encoding: [0x69,0x02,0x00,0x7e] # GFX10: v_mov_b32_e32 v0, s105 ; encoding: [0x69,0x02,0x00,0x7e]
0x69,0x02,0x00,0x7e 0x69,0x02,0x00,0x7e
# GFX10: v_mov_b32_sdwa v0, s105 dst_sel:DWORD dst_unused:UNUSED_PRESERVE src0_sel:DWORD # GFX10: v_mov_b32_sdwa v0, s105 dst_sel:DWORD dst_unused:UNUSED_PRESERVE src0_sel:DWORD
0xf9,0x02,0x00,0x7e,0x69,0x16,0x86,0x00 0xf9,0x02,0x00,0x7e,0x69,0x16,0x86,0x00
# GFX10: s_mov_b32 s105, s104 ; encoding: [0x68,0x03,0xe9,0xbe] # GFX10: s_mov_b32 s105, s104 ; encoding: [0x68,0x03,0xe9,0xbe]
0x68,0x03,0xe9,0xbe 0x68,0x03,0xe9,0xbe
# GFX10: v_cmp_eq_f32_e64 s105, v0, s105
0x69,0x00,0x02,0xd4,0x00,0xd3,0x00,0x00
# GFX10: v_cmp_eq_f32_sdwa s105, v0, s105 src0_sel:DWORD src1_sel:DWORD
0xf9,0xd2,0x04,0x7c,0x00,0xe9,0x06,0x86

test/MC/Disassembler/AMDGPU/wave32.txt

  • This file was added.
# RUN: llvm-mc -arch=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -disassemble -show-encoding < %s | FileCheck -check-prefix=GFX1032 %s
# RUN: llvm-mc -arch=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize64,-wavefrontsize32 -disassemble -show-encoding < %s | FileCheck -check-prefix=GFX1064 %s
# GFX1032: v_cmp_lt_f32_e32 vcc_lo, s2, v4
# GFX1064: v_cmp_lt_f32_e32 vcc, s2, v4
0x02,0x08,0x02,0x7c
# GFX1032: v_cmp_ge_i32_e64 s2, s0, v2
# GFX1064: v_cmp_ge_i32_e64 s[2:3], s0, v2
0x02,0x00,0x86,0xd4,0x00,0x04,0x02,0x00
# GFX1032: v_cmp_ge_i32_sdwa vcc_lo, v0, v2 src0_sel:WORD_1 src1_sel:DWORD
# GFX1064: v_cmp_ge_i32_sdwa vcc, v0, v2 src0_sel:WORD_1 src1_sel:DWORD
0xf9,0x04,0x0c,0x7d,0x00,0x00,0x05,0x06
# GFX1032: v_cmp_le_f16_sdwa s0, v3, v4 src0_sel:WORD_1 src1_sel:DWORD
# GFX1064: v_cmp_le_f16_sdwa s[0:1], v3, v4 src0_sel:WORD_1 src1_sel:DWORD
0xf9,0x08,0x96,0x7d,0x03,0x80,0x05,0x06
# GFX1032: v_cmp_class_f32_e32 vcc_lo, s0, v0
# GFX1064: v_cmp_class_f32_e32 vcc, s0, v0
0x00,0x00,0x10,0x7d
# GFX1032: v_cmp_class_f16_sdwa vcc_lo, v1, v2 src0_sel:DWORD src1_sel:DWORD
# GFX1064: v_cmp_class_f16_sdwa vcc, v1, v2 src0_sel:DWORD src1_sel:DWORD
0xf9,0x04,0x1e,0x7d,0x01,0x00,0x06,0x06
# GFX1032: v_cmp_class_f16_sdwa s0, v1, v2 src0_sel:DWORD src1_sel:DWORD
# GFX1064: v_cmp_class_f16_sdwa s[0:1], v1, v2 src0_sel:DWORD src1_sel:DWORD
0xf9,0x04,0x1e,0x7d,0x01,0x80,0x06,0x06
# GFX1032: v_cndmask_b32_e32 v5, 0, v2, vcc_lo
# GFX1064: v_cndmask_b32_e32 v5, 0, v2, vcc ;
0x80,0x04,0x0a,0x02
# GFX1032: v_cndmask_b32_e32 v1, v2, v3, vcc_lo
# GFX1064: v_cndmask_b32_e32 v1, v2, v3, vcc ;
0x02,0x07,0x02,0x02
# GFX1032: v_add_co_u32_e64 v2, vcc_lo, s0, v2
# GFX1064: v_add_co_u32_e64 v2, vcc, s0, v2
0x02,0x6a,0x0f,0xd7,0x00,0x04,0x02,0x00
# GFX1032: v_add_co_ci_u32_e32 v3, vcc_lo, v3, v4, vcc_lo
# GFX1064: v_add_co_ci_u32_e32 v3, vcc, v3, v4, vcc ;
0x03,0x09,0x06,0x50
# GFX1032: v_sub_co_u32_e64 v2, vcc_lo, s0, v2
# GFX1064: v_sub_co_u32_e64 v2, vcc, s0, v2
0x02,0x6a,0x10,0xd7,0x00,0x04,0x02,0x00
# GFX1032: v_subrev_co_u32_e64 v2, vcc_lo, s0, v2
# GFX1064: v_subrev_co_u32_e64 v2, vcc, s0, v2
0x02,0x6a,0x19,0xd7,0x00,0x04,0x02,0x00
# GFX1032: v_sub_co_ci_u32_e32 v3, vcc_lo, v3, v4, vcc_lo
# GFX1064: v_sub_co_ci_u32_e32 v3, vcc, v3, v4, vcc ;
0x03,0x09,0x06,0x52
# GFX1032: v_subrev_co_ci_u32_e32 v1, vcc_lo, 0, v1, vcc_lo
# GFX1064: v_subrev_co_ci_u32_e32 v1, vcc, 0, v1, vcc ;
0x80,0x02,0x02,0x54
# GFX1032: v_add_co_ci_u32_sdwa v1, vcc_lo, v1, v4, vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
# GFX1064: v_add_co_ci_u32_sdwa v1, vcc, v1, v4, vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
0xf9,0x08,0x02,0x50,0x01,0x06,0x00,0x06
# GFX1032: v_sub_co_ci_u32_sdwa v1, vcc_lo, v1, v4, vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
# GFX1064: v_sub_co_ci_u32_sdwa v1, vcc, v1, v4, vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
0xf9,0x08,0x02,0x52,0x01,0x06,0x00,0x06
# GFX1032: v_subrev_co_ci_u32_sdwa v1, vcc_lo, v1, v4, vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
# GFX1064: v_subrev_co_ci_u32_sdwa v1, vcc, v1, v4, vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
0xf9,0x08,0x02,0x54,0x01,0x06,0x00,0x06
# GFX1032: v_add_co_ci_u32_sdwa v1, vcc_lo, sext(v1), sext(v4), vcc_lo dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
# GFX1064: v_add_co_ci_u32_sdwa v1, vcc, sext(v1), sext(v4), vcc dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
0xf9,0x08,0x02,0x50,0x01,0x06,0x08,0x0e
# GFX1032: v_add_nc_u32_dpp v5, v1, v2 quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
# GFX1064: v_add_nc_u32_dpp v5, v1, v2 quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
0xfa,0x04,0x0a,0x4a,0x01,0xe4,0x00,0x00
# FIXME: Results in invalid v_subrev_u16_dpp which apparently has the same encoding but does not exist in GFX10
# gfx1032: v_add_co_ci_u32_dpp v5, vcc_lo, v1, v2, vcc_lo quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
# gfx1064: v_add_co_ci_u32_dpp v5, vcc, v1, v2, vcc quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
# 0xfa,0x04,0x0a,0x50,0x01,0xe4,0x00,0x00
# FIXME: Results in v_mul_lo_u16_dpp
# gfx1032: v_sub_co_ci_u32_dpp v5, vcc_lo, v1, v2, vcc_lo quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
# gfx1064: v_sub_co_ci_u32_dpp v5, vcc, v1, v2, vcc quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
# 0xfa,0x04,0x0a,0x52,0x01,0xe4,0x00,0x00
# FIXME: gives v_lshlrev_b16_dpp
# gfx1032: v_subrev_co_ci_u32_dpp v5, vcc_lo, v1, v2, vcc_lo quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
# gfx1064: v_subrev_co_ci_u32_dpp v5, vcc, v1, v2, vcc quad_perm:[0,1,2,3] row_mask:0x0 bank_mask:0x0
# 0xfa,0x04,0x0a,0x54,0x01,0xe4,0x00,0x00
# GFX1032: v_add_co_u32_e64 v0, s0, v0, v2
# GFX1064: v_add_co_u32_e64 v0, s[0:1], v0, v2
0x00,0x00,0x0f,0xd7,0x00,0x05,0x02,0x00
# GFX1032: v_add_co_ci_u32_e64 v4, s0, v1, v5, s2
# GFX1064: v_add_co_ci_u32_e64 v4, s[0:1], v1, v5, s[2:3]
0x04,0x00,0x28,0xd5,0x01,0x0b,0x0a,0x00
# GFX1032: v_sub_co_u32_e64 v0, s0, v0, v2
# GFX1064: v_sub_co_u32_e64 v0, s[0:1], v0, v2
0x00,0x00,0x10,0xd7,0x00,0x05,0x02,0x00
# GFX1032: v_sub_co_ci_u32_e64 v4, s0, v1, v5, s2
# GFX1064: v_sub_co_ci_u32_e64 v4, s[0:1], v1, v5, s[2:3]
0x04,0x00,0x29,0xd5,0x01,0x0b,0x0a,0x00
# GFX1032: v_subrev_co_u32_e64 v0, s0, v0, v2
# GFX1064: v_subrev_co_u32_e64 v0, s[0:1], v0, v2
0x00,0x00,0x19,0xd7,0x00,0x05,0x02,0x00
# GFX1032: v_subrev_co_ci_u32_e64 v4, s0, v1, v5, s2
# GFX1064: v_subrev_co_ci_u32_e64 v4, s[0:1], v1, v5, s[2:3]
0x04,0x00,0x2a,0xd5,0x01,0x0b,0x0a,0x00
# GFX1032: v_add_co_ci_u32_e64 v4, vcc_lo, v1, v5, s2
# GFX1064: v_add_co_ci_u32_e64 v4, vcc, v1, v5, s[2:3]
0x04,0x6a,0x28,0xd5,0x01,0x0b,0x0a,0x00
# GFX1032: v_add_co_ci_u32_e64 v4, s0, v1, v5, vcc_lo
# GFX1064: v_add_co_ci_u32_e64 v4, s[0:1], v1, v5, vcc ;
0x04,0x00,0x28,0xd5,0x01,0x0b,0xaa,0x01
# GFX1032: v_div_scale_f32 v2, s2, v0, v0, v2
# GFX1064: v_div_scale_f32 v2, s[2:3], v0, v0, v2
0x02,0x02,0x6d,0xd5,0x00,0x01,0x0a,0x04
# GFX1032: v_div_scale_f64 v[2:3], s2, v[0:1], v[0:1], v[2:3]
# GFX1064: v_div_scale_f64 v[2:3], s[2:3], v[0:1], v[0:1], v[2:3]
0x02,0x02,0x6e,0xd5,0x00,0x01,0x0a,0x04
# GFX1032: v_mad_i64_i32 v[0:1], s6, v0, v1, v[2:3]
# GFX1064: v_mad_i64_i32 v[0:1], s[6:7], v0, v1, v[2:3]
0x00,0x06,0x77,0xd5,0x00,0x03,0x0a,0x04
# GFX1032: v_mad_u64_u32 v[0:1], s6, v0, v1, v[2:3]
# GFX1064: v_mad_u64_u32 v[0:1], s[6:7], v0, v1, v[2:3]
0x00,0x06,0x76,0xd5,0x00,0x03,0x0a,0x04
# GFX1032: v_cmpx_neq_f32_e32 v0, v1
# GFX1064: v_cmpx_neq_f32_e32 v0, v1
0x00,0x03,0x3a,0x7c
# GFX1032: v_cmpx_neq_f32_sdwa v0, v1 src0_sel:WORD_1 src1_sel:DWORD
# GFX1064: v_cmpx_neq_f32_sdwa v0, v1 src0_sel:WORD_1 src1_sel:DWORD
0xf9,0x02,0x3a,0x7c,0x00,0x00,0x05,0x06
# GFX1032: v_cmpx_class_f32_e64 v0, 1
# GFX1064: v_cmpx_class_f32_e64 v0, 1
0x00,0x00,0x98,0xd4,0x00,0x03,0x01,0x00
# GFX1032: v_cmpx_class_f32_sdwa v0, 1 src0_sel:WORD_1 src1_sel:DWORD
# GFX1064: v_cmpx_class_f32_sdwa v0, 1 src0_sel:WORD_1 src1_sel:DWORD
0xf9,0x02,0x31,0x7d,0x00,0x00,0x05,0x86

test/Transforms/InstCombine/AMDGPU/amdgcn-intrinsics.ll

Show First 20 Lines • Show All 81 Lines • ▼ Show 20 Lines
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
; llvm.amdgcn.frexp.mant ; llvm.amdgcn.frexp.mant
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
declare float @llvm.amdgcn.frexp.mant.f32(float) nounwind readnone declare float @llvm.amdgcn.frexp.mant.f32(float) nounwind readnone
declare double @llvm.amdgcn.frexp.mant.f64(double) nounwind readnone declare double @llvm.amdgcn.frexp.mant.f64(double) nounwind readnone
define float @test_constant_fold_frexp_mant_f32_undef() nounwind { define float @test_constant_fold_frexp_mant_f32_undef() nounwind {
; CHECK-LABEL: @test_constant_fold_frexp_mant_f32_undef( ; CHECK-LABEL: @test_constant_fold_frexp_mant_f32_undef(
; CHECK-NEXT: ret float undef ; CHECK-NEXT: ret float undef
; ;
%val = call float @llvm.amdgcn.frexp.mant.f32(float undef) %val = call float @llvm.amdgcn.frexp.mant.f32(float undef)
ret float %val ret float %val
} }
▲ Show 20 Lines • Show All 144 Lines • ▼ Show 20 Lines
define double @test_constant_fold_frexp_mant_f64_min_num() nounwind { define double @test_constant_fold_frexp_mant_f64_min_num() nounwind {
; CHECK-LABEL: @test_constant_fold_frexp_mant_f64_min_num( ; CHECK-LABEL: @test_constant_fold_frexp_mant_f64_min_num(
; CHECK-NEXT: ret double 5.000000e-01 ; CHECK-NEXT: ret double 5.000000e-01
; ;
%val = call double @llvm.amdgcn.frexp.mant.f64(double 4.940656e-324) %val = call double @llvm.amdgcn.frexp.mant.f64(double 4.940656e-324)
ret double %val ret double %val
} }
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
; llvm.amdgcn.frexp.exp ; llvm.amdgcn.frexp.exp
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
declare i32 @llvm.amdgcn.frexp.exp.f32(float) nounwind readnone declare i32 @llvm.amdgcn.frexp.exp.f32(float) nounwind readnone
declare i32 @llvm.amdgcn.frexp.exp.f64(double) nounwind readnone declare i32 @llvm.amdgcn.frexp.exp.f64(double) nounwind readnone
define i32 @test_constant_fold_frexp_exp_f32_undef() nounwind { define i32 @test_constant_fold_frexp_exp_f32_undef() nounwind {
▲ Show 20 Lines • Show All 621 Lines • ▼ Show 20 Lines
define <2 x half> @constant_cvt_pkrtz() { define <2 x half> @constant_cvt_pkrtz() {
; CHECK-LABEL: @constant_cvt_pkrtz( ; CHECK-LABEL: @constant_cvt_pkrtz(
; CHECK-NEXT: ret <2 x half> <half 0xH4000, half 0xH4400> ; CHECK-NEXT: ret <2 x half> <half 0xH4000, half 0xH4400>
; ;
%cvt = call <2 x half> @llvm.amdgcn.cvt.pkrtz(float 2.0, float 4.0) %cvt = call <2 x half> @llvm.amdgcn.cvt.pkrtz(float 2.0, float 4.0)
ret <2 x half> %cvt ret <2 x half> %cvt
} }
; Test constant values where rtz changes result
define <2 x half> @constant_rtz_pkrtz() { define <2 x half> @constant_rtz_pkrtz() {
; CHECK-LABEL: @constant_rtz_pkrtz( ; CHECK-LABEL: @constant_rtz_pkrtz(
; CHECK-NEXT: ret <2 x half> <half 0xH7BFF, half 0xH7BFF> ; CHECK-NEXT: ret <2 x half> <half 0xH7BFF, half 0xH7BFF>
; ;
%cvt = call <2 x half> @llvm.amdgcn.cvt.pkrtz(float 65535.0, float 65535.0) %cvt = call <2 x half> @llvm.amdgcn.cvt.pkrtz(float 65535.0, float 65535.0)
ret <2 x half> %cvt ret <2 x half> %cvt
} }
▲ Show 20 Lines • Show All 370 Lines • ▼ Show 20 Lines
} }
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
; llvm.amdgcn.exp ; llvm.amdgcn.exp
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
declare void @llvm.amdgcn.exp.f32(i32 immarg, i32 immarg, float, float, float, float, i1 immarg, i1 immarg) nounwind inaccessiblememonly declare void @llvm.amdgcn.exp.f32(i32 immarg, i32 immarg, float, float, float, float, i1 immarg, i1 immarg) nounwind inaccessiblememonly
define void @exp_disabled_inputs_to_undef(float %x, float %y, float %z, float %w) { define void @exp_disabled_inputs_to_undef(float %x, float %y, float %z, float %w) {
; enable src0..src3 constants
; CHECK-LABEL: @exp_disabled_inputs_to_undef( ; CHECK-LABEL: @exp_disabled_inputs_to_undef(
; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 1, float 1.000000e+00, float undef, float undef, float undef, i1 true, i1 false) ; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 1, float 1.000000e+00, float undef, float undef, float undef, i1 true, i1 false)
; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 2, float undef, float 2.000000e+00, float undef, float undef, i1 true, i1 false) ; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 2, float undef, float 2.000000e+00, float undef, float undef, i1 true, i1 false)
; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 4, float undef, float undef, float 5.000000e-01, float undef, i1 true, i1 false) ; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 4, float undef, float undef, float 5.000000e-01, float undef, i1 true, i1 false)
; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 8, float undef, float undef, float undef, float 4.000000e+00, i1 true, i1 false) ; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 8, float undef, float undef, float undef, float 4.000000e+00, i1 true, i1 false)
; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 1, float [[X:%.*]], float undef, float undef, float undef, i1 true, i1 false) ; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 1, float [[X:%.*]], float undef, float undef, float undef, i1 true, i1 false)
; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 2, float undef, float [[Y:%.*]], float undef, float undef, i1 true, i1 false) ; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 2, float undef, float [[Y:%.*]], float undef, float undef, i1 true, i1 false)
; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 4, float undef, float undef, float [[Z:%.*]], float undef, i1 true, i1 false) ; CHECK-NEXT: call void @llvm.amdgcn.exp.f32(i32 0, i32 4, float undef, float undef, float [[Z:%.*]], float undef, i1 true, i1 false)
Show All 29 Lines
} }
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
; llvm.amdgcn.exp.compr ; llvm.amdgcn.exp.compr
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
declare void @llvm.amdgcn.exp.compr.v2f16(i32 immarg, i32 immarg, <2 x half>, <2 x half>, i1 immarg, i1 immarg) nounwind inaccessiblememonly declare void @llvm.amdgcn.exp.compr.v2f16(i32 immarg, i32 immarg, <2 x half>, <2 x half>, i1 immarg, i1 immarg) nounwind inaccessiblememonly
define void @exp_compr_disabled_inputs_to_undef(<2 x half> %xy, <2 x half> %zw) { define void @exp_compr_disabled_inputs_to_undef(<2 x half> %xy, <2 x half> %zw) {
; CHECK-LABEL: @exp_compr_disabled_inputs_to_undef( ; CHECK-LABEL: @exp_compr_disabled_inputs_to_undef(
; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 0, <2 x half> undef, <2 x half> undef, i1 true, i1 false) ; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 0, <2 x half> undef, <2 x half> undef, i1 true, i1 false)
; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 1, <2 x half> <half 0xH3C00, half 0xH4000>, <2 x half> undef, i1 true, i1 false) ; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 1, <2 x half> <half 0xH3C00, half 0xH4000>, <2 x half> undef, i1 true, i1 false)
; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 2, <2 x half> <half 0xH3C00, half 0xH4000>, <2 x half> undef, i1 true, i1 false) ; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 2, <2 x half> <half 0xH3C00, half 0xH4000>, <2 x half> undef, i1 true, i1 false)
; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 3, <2 x half> <half 0xH3C00, half 0xH4000>, <2 x half> undef, i1 true, i1 false) ; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 3, <2 x half> <half 0xH3C00, half 0xH4000>, <2 x half> undef, i1 true, i1 false)
; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 0, <2 x half> undef, <2 x half> undef, i1 true, i1 false) ; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 0, <2 x half> undef, <2 x half> undef, i1 true, i1 false)
; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 1, <2 x half> [[XY:%.*]], <2 x half> undef, i1 true, i1 false) ; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 1, <2 x half> [[XY:%.*]], <2 x half> undef, i1 true, i1 false)
▲ Show 20 Lines • Show All 154 Lines • ▼ Show 20 Lines
; CHECK-LABEL: @fmed3_qnan1_x_y_f32( ; CHECK-LABEL: @fmed3_qnan1_x_y_f32(
; CHECK-NEXT: [[MED3:%.*]] = call float @llvm.minnum.f32(float [[X:%.*]], float [[Y:%.*]]) ; CHECK-NEXT: [[MED3:%.*]] = call float @llvm.minnum.f32(float [[X:%.*]], float [[Y:%.*]])
; CHECK-NEXT: ret float [[MED3]] ; CHECK-NEXT: ret float [[MED3]]
; ;
%med3 = call float @llvm.amdgcn.fmed3.f32(float 0x7FF8000100000000, float %x, float %y) %med3 = call float @llvm.amdgcn.fmed3.f32(float 0x7FF8000100000000, float %x, float %y)
ret float %med3 ret float %med3
} }
; This can return any of the qnans.
define float @fmed3_qnan0_qnan1_qnan2_f32(float %x, float %y) { define float @fmed3_qnan0_qnan1_qnan2_f32(float %x, float %y) {
; CHECK-LABEL: @fmed3_qnan0_qnan1_qnan2_f32( ; CHECK-LABEL: @fmed3_qnan0_qnan1_qnan2_f32(
; CHECK-NEXT: ret float 0x7FF8030000000000 ; CHECK-NEXT: ret float 0x7FF8030000000000
; ;
%med3 = call float @llvm.amdgcn.fmed3.f32(float 0x7FF8000100000000, float 0x7FF8002000000000, float 0x7FF8030000000000) %med3 = call float @llvm.amdgcn.fmed3.f32(float 0x7FF8000100000000, float 0x7FF8002000000000, float 0x7FF8030000000000)
ret float %med3 ret float %med3
} }
▲ Show 20 Lines • Show All 116 Lines • ▼ Show 20 Lines;
%med = call float @llvm.amdgcn.fmed3.f32(float 0.0, float 1.0, float undef) %med = call float @llvm.amdgcn.fmed3.f32(float 0.0, float 1.0, float undef)
ret float %med ret float %med
} }
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
; llvm.amdgcn.icmp ; llvm.amdgcn.icmp
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
declare i64 @llvm.amdgcn.icmp.i32(i32, i32, i32 immarg) nounwind readnone convergent declare i64 @llvm.amdgcn.icmp.i64.i32(i32, i32, i32 immarg) nounwind readnone convergent
declare i64 @llvm.amdgcn.icmp.i64(i64, i64, i32 immarg) nounwind readnone convergent declare i64 @llvm.amdgcn.icmp.i64.i64(i64, i64, i32 immarg) nounwind readnone convergent
declare i64 @llvm.amdgcn.icmp.i1(i1, i1, i32 immarg) nounwind readnone convergent declare i64 @llvm.amdgcn.icmp.i64.i1(i1, i1, i32 immarg) nounwind readnone convergent
define i64 @invalid_icmp_code(i32 %a, i32 %b) { define i64 @invalid_icmp_code(i32 %a, i32 %b) {
; CHECK-LABEL: @invalid_icmp_code( ; CHECK-LABEL: @invalid_icmp_code(
; CHECK-NEXT: [[UNDER:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 31) ; CHECK-NEXT: [[UNDER:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 31)
; CHECK-NEXT: [[OVER:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[A]], i32 [[B]], i32 42) ; CHECK-NEXT: [[OVER:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[A]], i32 [[B]], i32 42)
; CHECK-NEXT: [[OR:%.*]] = or i64 [[UNDER]], [[OVER]] ; CHECK-NEXT: [[OR:%.*]] = or i64 [[UNDER]], [[OVER]]
; CHECK-NEXT: ret i64 [[OR]] ; CHECK-NEXT: ret i64 [[OR]]
; ;
%under = call i64 @llvm.amdgcn.icmp.i32(i32 %a, i32 %b, i32 31) %under = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %a, i32 %b, i32 31)
%over = call i64 @llvm.amdgcn.icmp.i32(i32 %a, i32 %b, i32 42) %over = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %a, i32 %b, i32 42)
%or = or i64 %under, %over %or = or i64 %under, %over
ret i64 %or ret i64 %or
} }
define i64 @icmp_constant_inputs_false() { define i64 @icmp_constant_inputs_false() {
; CHECK-LABEL: @icmp_constant_inputs_false( ; CHECK-LABEL: @icmp_constant_inputs_false(
; CHECK-NEXT: ret i64 0 ; CHECK-NEXT: ret i64 0
; ;
%result = call i64 @llvm.amdgcn.icmp.i32(i32 9, i32 8, i32 32) %result = call i64 @llvm.amdgcn.icmp.i64.i32(i32 9, i32 8, i32 32)
ret i64 %result ret i64 %result
} }
define i64 @icmp_constant_inputs_true() { define i64 @icmp_constant_inputs_true() {
; CHECK-LABEL: @icmp_constant_inputs_true( ; CHECK-LABEL: @icmp_constant_inputs_true(
; CHECK-NEXT: [[RESULT:%.*]] = call i64 @llvm.read_register.i64(metadata !0) #5 ; CHECK-NEXT: [[RESULT:%.*]] = call i64 @llvm.read_register.i64(metadata !0) #5
; CHECK-NEXT: ret i64 [[RESULT]] ; CHECK-NEXT: ret i64 [[RESULT]]
; ;
%result = call i64 @llvm.amdgcn.icmp.i32(i32 9, i32 8, i32 34) %result = call i64 @llvm.amdgcn.icmp.i64.i32(i32 9, i32 8, i32 34)
ret i64 %result ret i64 %result
} }
define i64 @icmp_constant_to_rhs_slt(i32 %x) { define i64 @icmp_constant_to_rhs_slt(i32 %x) {
; CHECK-LABEL: @icmp_constant_to_rhs_slt( ; CHECK-LABEL: @icmp_constant_to_rhs_slt(
; CHECK-NEXT: [[RESULT:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[X:%.*]], i32 9, i32 38) ; CHECK-NEXT: [[RESULT:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[X:%.*]], i32 9, i32 38)
; CHECK-NEXT: ret i64 [[RESULT]] ; CHECK-NEXT: ret i64 [[RESULT]]
; ;
%result = call i64 @llvm.amdgcn.icmp.i32(i32 9, i32 %x, i32 40) %result = call i64 @llvm.amdgcn.icmp.i64.i32(i32 9, i32 %x, i32 40)
ret i64 %result ret i64 %result
} }
define i64 @fold_icmp_ne_0_zext_icmp_eq_i32(i32 %a, i32 %b) { define i64 @fold_icmp_ne_0_zext_icmp_eq_i32(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_eq_i32( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_eq_i32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 32) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 32)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i32 %a, %b %cmp = icmp eq i32 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_ne_i32(i32 %a, i32 %b) { define i64 @fold_icmp_ne_0_zext_icmp_ne_i32(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_ne_i32( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_ne_i32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp ne i32 %a, %b %cmp = icmp ne i32 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_sle_i32(i32 %a, i32 %b) { define i64 @fold_icmp_ne_0_zext_icmp_sle_i32(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_sle_i32( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_sle_i32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 41) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 41)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp sle i32 %a, %b %cmp = icmp sle i32 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_ugt_i64(i64 %a, i64 %b) { define i64 @fold_icmp_ne_0_zext_icmp_ugt_i64(i64 %a, i64 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_ugt_i64( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_ugt_i64(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64(i64 [[A:%.*]], i64 [[B:%.*]], i32 34) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i64(i64 [[A:%.*]], i64 [[B:%.*]], i32 34)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp ugt i64 %a, %b %cmp = icmp ugt i64 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_ult_swap_i64(i64 %a, i64 %b) { define i64 @fold_icmp_ne_0_zext_icmp_ult_swap_i64(i64 %a, i64 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_ult_swap_i64( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_ult_swap_i64(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64(i64 [[A:%.*]], i64 [[B:%.*]], i32 34) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i64(i64 [[A:%.*]], i64 [[B:%.*]], i32 34)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp ugt i64 %a, %b %cmp = icmp ugt i64 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 0, i32 %zext.cmp, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 0, i32 %zext.cmp, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_fcmp_oeq_f32(float %a, float %b) { define i64 @fold_icmp_ne_0_zext_fcmp_oeq_f32(float %a, float %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_fcmp_oeq_f32( ; CHECK-LABEL: @fold_icmp_ne_0_zext_fcmp_oeq_f32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.f32(float [[A:%.*]], float [[B:%.*]], i32 1) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.i64.f32(float [[A:%.*]], float [[B:%.*]], i32 1)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = fcmp oeq float %a, %b %cmp = fcmp oeq float %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_fcmp_une_f32(float %a, float %b) { define i64 @fold_icmp_ne_0_zext_fcmp_une_f32(float %a, float %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_fcmp_une_f32( ; CHECK-LABEL: @fold_icmp_ne_0_zext_fcmp_une_f32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.f32(float [[A:%.*]], float [[B:%.*]], i32 14) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.i64.f32(float [[A:%.*]], float [[B:%.*]], i32 14)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = fcmp une float %a, %b %cmp = fcmp une float %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_fcmp_olt_f64(double %a, double %b) { define i64 @fold_icmp_ne_0_zext_fcmp_olt_f64(double %a, double %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_fcmp_olt_f64( ; CHECK-LABEL: @fold_icmp_ne_0_zext_fcmp_olt_f64(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.f64(double [[A:%.*]], double [[B:%.*]], i32 4) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.i64.f64(double [[A:%.*]], double [[B:%.*]], i32 4)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = fcmp olt double %a, %b %cmp = fcmp olt double %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_sext_icmp_ne_0_i32(i32 %a, i32 %b) { define i64 @fold_icmp_sext_icmp_ne_0_i32(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_icmp_sext_icmp_ne_0_i32( ; CHECK-LABEL: @fold_icmp_sext_icmp_ne_0_i32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 32) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 32)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i32 %a, %b %cmp = icmp eq i32 %a, %b
%sext.cmp = sext i1 %cmp to i32 %sext.cmp = sext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %sext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %sext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_eq_0_zext_icmp_eq_i32(i32 %a, i32 %b) { define i64 @fold_icmp_eq_0_zext_icmp_eq_i32(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_icmp_eq_0_zext_icmp_eq_i32( ; CHECK-LABEL: @fold_icmp_eq_0_zext_icmp_eq_i32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i32 %a, %b %cmp = icmp eq i32 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 32)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_eq_0_zext_icmp_slt_i32(i32 %a, i32 %b) { define i64 @fold_icmp_eq_0_zext_icmp_slt_i32(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_icmp_eq_0_zext_icmp_slt_i32( ; CHECK-LABEL: @fold_icmp_eq_0_zext_icmp_slt_i32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 39) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 39)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp slt i32 %a, %b %cmp = icmp slt i32 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 32)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_eq_0_zext_fcmp_oeq_f32(float %a, float %b) { define i64 @fold_icmp_eq_0_zext_fcmp_oeq_f32(float %a, float %b) {
; CHECK-LABEL: @fold_icmp_eq_0_zext_fcmp_oeq_f32( ; CHECK-LABEL: @fold_icmp_eq_0_zext_fcmp_oeq_f32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.f32(float [[A:%.*]], float [[B:%.*]], i32 14) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.i64.f32(float [[A:%.*]], float [[B:%.*]], i32 14)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = fcmp oeq float %a, %b %cmp = fcmp oeq float %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 32)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_eq_0_zext_fcmp_ule_f32(float %a, float %b) { define i64 @fold_icmp_eq_0_zext_fcmp_ule_f32(float %a, float %b) {
; CHECK-LABEL: @fold_icmp_eq_0_zext_fcmp_ule_f32( ; CHECK-LABEL: @fold_icmp_eq_0_zext_fcmp_ule_f32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.f32(float [[A:%.*]], float [[B:%.*]], i32 2) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.i64.f32(float [[A:%.*]], float [[B:%.*]], i32 2)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = fcmp ule float %a, %b %cmp = fcmp ule float %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 32)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_eq_0_zext_fcmp_ogt_f32(float %a, float %b) { define i64 @fold_icmp_eq_0_zext_fcmp_ogt_f32(float %a, float %b) {
; CHECK-LABEL: @fold_icmp_eq_0_zext_fcmp_ogt_f32( ; CHECK-LABEL: @fold_icmp_eq_0_zext_fcmp_ogt_f32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.f32(float [[A:%.*]], float [[B:%.*]], i32 13) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.i64.f32(float [[A:%.*]], float [[B:%.*]], i32 13)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = fcmp ogt float %a, %b %cmp = fcmp ogt float %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 32)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_zext_icmp_eq_1_i32(i32 %a, i32 %b) { define i64 @fold_icmp_zext_icmp_eq_1_i32(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_icmp_zext_icmp_eq_1_i32( ; CHECK-LABEL: @fold_icmp_zext_icmp_eq_1_i32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 32) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 32)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i32 %a, %b %cmp = icmp eq i32 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 1, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 1, i32 32)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_zext_argi1_eq_1_i32(i1 %cond) { define i64 @fold_icmp_zext_argi1_eq_1_i32(i1 %cond) {
; CHECK-LABEL: @fold_icmp_zext_argi1_eq_1_i32( ; CHECK-LABEL: @fold_icmp_zext_argi1_eq_1_i32(
; CHECK-NEXT: [[ZEXT_COND:%.*]] = zext i1 [[COND:%.*]] to i32 ; CHECK-NEXT: [[ZEXT_COND:%.*]] = zext i1 [[COND:%.*]] to i32
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[ZEXT_COND]], i32 0, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[ZEXT_COND]], i32 0, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%zext.cond = zext i1 %cond to i32 %zext.cond = zext i1 %cond to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cond, i32 1, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cond, i32 1, i32 32)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_zext_argi1_eq_neg1_i32(i1 %cond) { define i64 @fold_icmp_zext_argi1_eq_neg1_i32(i1 %cond) {
; CHECK-LABEL: @fold_icmp_zext_argi1_eq_neg1_i32( ; CHECK-LABEL: @fold_icmp_zext_argi1_eq_neg1_i32(
; CHECK-NEXT: [[ZEXT_COND:%.*]] = zext i1 [[COND:%.*]] to i32 ; CHECK-NEXT: [[ZEXT_COND:%.*]] = zext i1 [[COND:%.*]] to i32
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[ZEXT_COND]], i32 -1, i32 32) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[ZEXT_COND]], i32 -1, i32 32)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%zext.cond = zext i1 %cond to i32 %zext.cond = zext i1 %cond to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cond, i32 -1, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cond, i32 -1, i32 32)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_sext_argi1_eq_1_i32(i1 %cond) { define i64 @fold_icmp_sext_argi1_eq_1_i32(i1 %cond) {
; CHECK-LABEL: @fold_icmp_sext_argi1_eq_1_i32( ; CHECK-LABEL: @fold_icmp_sext_argi1_eq_1_i32(
; CHECK-NEXT: [[SEXT_COND:%.*]] = sext i1 [[COND:%.*]] to i32 ; CHECK-NEXT: [[SEXT_COND:%.*]] = sext i1 [[COND:%.*]] to i32
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[SEXT_COND]], i32 1, i32 32) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[SEXT_COND]], i32 1, i32 32)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%sext.cond = sext i1 %cond to i32 %sext.cond = sext i1 %cond to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %sext.cond, i32 1, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %sext.cond, i32 1, i32 32)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_sext_argi1_eq_neg1_i32(i1 %cond) { define i64 @fold_icmp_sext_argi1_eq_neg1_i32(i1 %cond) {
; CHECK-LABEL: @fold_icmp_sext_argi1_eq_neg1_i32( ; CHECK-LABEL: @fold_icmp_sext_argi1_eq_neg1_i32(
; CHECK-NEXT: [[SEXT_COND:%.*]] = sext i1 [[COND:%.*]] to i32 ; CHECK-NEXT: [[SEXT_COND:%.*]] = sext i1 [[COND:%.*]] to i32
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[SEXT_COND]], i32 0, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[SEXT_COND]], i32 0, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%sext.cond = sext i1 %cond to i32 %sext.cond = sext i1 %cond to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %sext.cond, i32 -1, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %sext.cond, i32 -1, i32 32)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_sext_argi1_eq_neg1_i64(i1 %cond) { define i64 @fold_icmp_sext_argi1_eq_neg1_i64(i1 %cond) {
; CHECK-LABEL: @fold_icmp_sext_argi1_eq_neg1_i64( ; CHECK-LABEL: @fold_icmp_sext_argi1_eq_neg1_i64(
; CHECK-NEXT: [[SEXT_COND:%.*]] = sext i1 [[COND:%.*]] to i64 ; CHECK-NEXT: [[SEXT_COND:%.*]] = sext i1 [[COND:%.*]] to i64
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64(i64 [[SEXT_COND]], i64 0, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i64(i64 [[SEXT_COND]], i64 0, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%sext.cond = sext i1 %cond to i64 %sext.cond = sext i1 %cond to i64
%mask = call i64 @llvm.amdgcn.icmp.i64(i64 %sext.cond, i64 -1, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i64(i64 %sext.cond, i64 -1, i32 32)
ret i64 %mask ret i64 %mask
} }
; TODO: Should be able to fold to false
define i64 @fold_icmp_sext_icmp_eq_1_i32(i32 %a, i32 %b) { define i64 @fold_icmp_sext_icmp_eq_1_i32(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_icmp_sext_icmp_eq_1_i32( ; CHECK-LABEL: @fold_icmp_sext_icmp_eq_1_i32(
; CHECK-NEXT: [[CMP:%.*]] = icmp eq i32 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp eq i32 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[SEXT_CMP:%.*]] = sext i1 [[CMP]] to i32 ; CHECK-NEXT: [[SEXT_CMP:%.*]] = sext i1 [[CMP]] to i32
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[SEXT_CMP]], i32 1, i32 32) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[SEXT_CMP]], i32 1, i32 32)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i32 %a, %b %cmp = icmp eq i32 %a, %b
%sext.cmp = sext i1 %cmp to i32 %sext.cmp = sext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %sext.cmp, i32 1, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %sext.cmp, i32 1, i32 32)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_sext_icmp_eq_neg1_i32(i32 %a, i32 %b) { define i64 @fold_icmp_sext_icmp_eq_neg1_i32(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_icmp_sext_icmp_eq_neg1_i32( ; CHECK-LABEL: @fold_icmp_sext_icmp_eq_neg1_i32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 32) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 32)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i32 %a, %b %cmp = icmp eq i32 %a, %b
%sext.cmp = sext i1 %cmp to i32 %sext.cmp = sext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %sext.cmp, i32 -1, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %sext.cmp, i32 -1, i32 32)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_sext_icmp_sge_neg1_i32(i32 %a, i32 %b) { define i64 @fold_icmp_sext_icmp_sge_neg1_i32(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_icmp_sext_icmp_sge_neg1_i32( ; CHECK-LABEL: @fold_icmp_sext_icmp_sge_neg1_i32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 39) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 39)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp sge i32 %a, %b %cmp = icmp sge i32 %a, %b
%sext.cmp = sext i1 %cmp to i32 %sext.cmp = sext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %sext.cmp, i32 -1, i32 32) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %sext.cmp, i32 -1, i32 32)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_not_icmp_ne_0_zext_icmp_sle_i32(i32 %a, i32 %b) { define i64 @fold_not_icmp_ne_0_zext_icmp_sle_i32(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_not_icmp_ne_0_zext_icmp_sle_i32( ; CHECK-LABEL: @fold_not_icmp_ne_0_zext_icmp_sle_i32(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 38) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[A:%.*]], i32 [[B:%.*]], i32 38)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp sle i32 %a, %b %cmp = icmp sle i32 %a, %b
%not = xor i1 %cmp, true %not = xor i1 %cmp, true
%zext.cmp = zext i1 %not to i32 %zext.cmp = zext i1 %not to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_eq_i4(i4 %a, i4 %b) { define i64 @fold_icmp_ne_0_zext_icmp_eq_i4(i4 %a, i4 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_eq_i4( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_eq_i4(
; CHECK-NEXT: [[TMP1:%.*]] = zext i4 [[A:%.*]] to i16 ; CHECK-NEXT: [[TMP1:%.*]] = zext i4 [[A:%.*]] to i16
; CHECK-NEXT: [[TMP2:%.*]] = zext i4 [[B:%.*]] to i16 ; CHECK-NEXT: [[TMP2:%.*]] = zext i4 [[B:%.*]] to i16
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i16(i16 [[TMP1]], i16 [[TMP2]], i32 32) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i16(i16 [[TMP1]], i16 [[TMP2]], i32 32)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i4 %a, %b %cmp = icmp eq i4 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_eq_i8(i8 %a, i8 %b) { define i64 @fold_icmp_ne_0_zext_icmp_eq_i8(i8 %a, i8 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_eq_i8( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_eq_i8(
; CHECK-NEXT: [[TMP1:%.*]] = zext i8 [[A:%.*]] to i16 ; CHECK-NEXT: [[TMP1:%.*]] = zext i8 [[A:%.*]] to i16
; CHECK-NEXT: [[TMP2:%.*]] = zext i8 [[B:%.*]] to i16 ; CHECK-NEXT: [[TMP2:%.*]] = zext i8 [[B:%.*]] to i16
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i16(i16 [[TMP1]], i16 [[TMP2]], i32 32) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i16(i16 [[TMP1]], i16 [[TMP2]], i32 32)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i8 %a, %b %cmp = icmp eq i8 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_eq_i16(i16 %a, i16 %b) { define i64 @fold_icmp_ne_0_zext_icmp_eq_i16(i16 %a, i16 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_eq_i16( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_eq_i16(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i16(i16 [[A:%.*]], i16 [[B:%.*]], i32 32) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i16(i16 [[A:%.*]], i16 [[B:%.*]], i32 32)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i16 %a, %b %cmp = icmp eq i16 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_eq_i36(i36 %a, i36 %b) { define i64 @fold_icmp_ne_0_zext_icmp_eq_i36(i36 %a, i36 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_eq_i36( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_eq_i36(
; CHECK-NEXT: [[TMP1:%.*]] = zext i36 [[A:%.*]] to i64 ; CHECK-NEXT: [[TMP1:%.*]] = zext i36 [[A:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = zext i36 [[B:%.*]] to i64 ; CHECK-NEXT: [[TMP2:%.*]] = zext i36 [[B:%.*]] to i64
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64(i64 [[TMP1]], i64 [[TMP2]], i32 32) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i64(i64 [[TMP1]], i64 [[TMP2]], i32 32)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i36 %a, %b %cmp = icmp eq i36 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_eq_i128(i128 %a, i128 %b) { define i64 @fold_icmp_ne_0_zext_icmp_eq_i128(i128 %a, i128 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_eq_i128( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_eq_i128(
; CHECK-NEXT: [[CMP:%.*]] = icmp eq i128 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp eq i128 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[ZEXT_CMP:%.*]] = zext i1 [[CMP]] to i32 ; CHECK-NEXT: [[ZEXT_CMP:%.*]] = zext i1 [[CMP]] to i32
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[ZEXT_CMP]], i32 0, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[ZEXT_CMP]], i32 0, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i128 %a, %b %cmp = icmp eq i128 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_fcmp_oeq_f16(half %a, half %b) { define i64 @fold_icmp_ne_0_zext_fcmp_oeq_f16(half %a, half %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_fcmp_oeq_f16( ; CHECK-LABEL: @fold_icmp_ne_0_zext_fcmp_oeq_f16(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.f16(half [[A:%.*]], half [[B:%.*]], i32 1) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.fcmp.i64.f16(half [[A:%.*]], half [[B:%.*]], i32 1)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = fcmp oeq half %a, %b %cmp = fcmp oeq half %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_fcmp_oeq_f128(fp128 %a, fp128 %b) { define i64 @fold_icmp_ne_0_zext_fcmp_oeq_f128(fp128 %a, fp128 %b) {
;
; CHECK-LABEL: @fold_icmp_ne_0_zext_fcmp_oeq_f128( ; CHECK-LABEL: @fold_icmp_ne_0_zext_fcmp_oeq_f128(
; CHECK-NEXT: [[CMP:%.*]] = fcmp oeq fp128 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = fcmp oeq fp128 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[ZEXT_CMP:%.*]] = zext i1 [[CMP]] to i32 ; CHECK-NEXT: [[ZEXT_CMP:%.*]] = zext i1 [[CMP]] to i32
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i32(i32 [[ZEXT_CMP]], i32 0, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i32(i32 [[ZEXT_CMP]], i32 0, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = fcmp oeq fp128 %a, %b %cmp = fcmp oeq fp128 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_slt_i4(i4 %a, i4 %b) { define i64 @fold_icmp_ne_0_zext_icmp_slt_i4(i4 %a, i4 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_slt_i4( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_slt_i4(
; CHECK-NEXT: [[TMP1:%.*]] = sext i4 [[A:%.*]] to i16 ; CHECK-NEXT: [[TMP1:%.*]] = sext i4 [[A:%.*]] to i16
; CHECK-NEXT: [[TMP2:%.*]] = sext i4 [[B:%.*]] to i16 ; CHECK-NEXT: [[TMP2:%.*]] = sext i4 [[B:%.*]] to i16
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i16(i16 [[TMP1]], i16 [[TMP2]], i32 40) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i16(i16 [[TMP1]], i16 [[TMP2]], i32 40)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp slt i4 %a, %b %cmp = icmp slt i4 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_slt_i8(i8 %a, i8 %b) { define i64 @fold_icmp_ne_0_zext_icmp_slt_i8(i8 %a, i8 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_slt_i8( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_slt_i8(
; CHECK-NEXT: [[TMP1:%.*]] = sext i8 [[A:%.*]] to i16 ; CHECK-NEXT: [[TMP1:%.*]] = sext i8 [[A:%.*]] to i16
; CHECK-NEXT: [[TMP2:%.*]] = sext i8 [[B:%.*]] to i16 ; CHECK-NEXT: [[TMP2:%.*]] = sext i8 [[B:%.*]] to i16
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i16(i16 [[TMP1]], i16 [[TMP2]], i32 40) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i16(i16 [[TMP1]], i16 [[TMP2]], i32 40)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp slt i8 %a, %b %cmp = icmp slt i8 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_slt_i16(i16 %a, i16 %b) { define i64 @fold_icmp_ne_0_zext_icmp_slt_i16(i16 %a, i16 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_slt_i16( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_slt_i16(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i16(i16 [[A:%.*]], i16 [[B:%.*]], i32 40) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i16(i16 [[A:%.*]], i16 [[B:%.*]], i32 40)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp slt i16 %a, %b %cmp = icmp slt i16 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_ult_i4(i4 %a, i4 %b) { define i64 @fold_icmp_ne_0_zext_icmp_ult_i4(i4 %a, i4 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_ult_i4( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_ult_i4(
; CHECK-NEXT: [[TMP1:%.*]] = zext i4 [[A:%.*]] to i16 ; CHECK-NEXT: [[TMP1:%.*]] = zext i4 [[A:%.*]] to i16
; CHECK-NEXT: [[TMP2:%.*]] = zext i4 [[B:%.*]] to i16 ; CHECK-NEXT: [[TMP2:%.*]] = zext i4 [[B:%.*]] to i16
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i16(i16 [[TMP1]], i16 [[TMP2]], i32 36) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i16(i16 [[TMP1]], i16 [[TMP2]], i32 36)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp ult i4 %a, %b %cmp = icmp ult i4 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_ult_i8(i8 %a, i8 %b) { define i64 @fold_icmp_ne_0_zext_icmp_ult_i8(i8 %a, i8 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_ult_i8( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_ult_i8(
; CHECK-NEXT: [[TMP1:%.*]] = zext i8 [[A:%.*]] to i16 ; CHECK-NEXT: [[TMP1:%.*]] = zext i8 [[A:%.*]] to i16
; CHECK-NEXT: [[TMP2:%.*]] = zext i8 [[B:%.*]] to i16 ; CHECK-NEXT: [[TMP2:%.*]] = zext i8 [[B:%.*]] to i16
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i16(i16 [[TMP1]], i16 [[TMP2]], i32 36) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i16(i16 [[TMP1]], i16 [[TMP2]], i32 36)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp ult i8 %a, %b %cmp = icmp ult i8 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_ne_0_zext_icmp_ult_i16(i16 %a, i16 %b) { define i64 @fold_icmp_ne_0_zext_icmp_ult_i16(i16 %a, i16 %b) {
; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_ult_i16( ; CHECK-LABEL: @fold_icmp_ne_0_zext_icmp_ult_i16(
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i16(i16 [[A:%.*]], i16 [[B:%.*]], i32 36) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i16(i16 [[A:%.*]], i16 [[B:%.*]], i32 36)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp ult i16 %a, %b %cmp = icmp ult i16 %a, %b
%zext.cmp = zext i1 %cmp to i32 %zext.cmp = zext i1 %cmp to i32
%mask = call i64 @llvm.amdgcn.icmp.i32(i32 %zext.cmp, i32 0, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %zext.cmp, i32 0, i32 33)
ret i64 %mask ret i64 %mask
} }
; 1-bit NE comparisons
define i64 @fold_icmp_i1_ne_0_icmp_eq_i1(i32 %a, i32 %b) { define i64 @fold_icmp_i1_ne_0_icmp_eq_i1(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_eq_i1( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_eq_i1(
; CHECK-NEXT: [[CMP:%.*]] = icmp eq i32 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp eq i32 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i32 %a, %b %cmp = icmp eq i32 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_ne_i1(i32 %a, i32 %b) { define i64 @fold_icmp_i1_ne_0_icmp_ne_i1(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_ne_i1( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_ne_i1(
; CHECK-NEXT: [[CMP:%.*]] = icmp ne i32 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp ne i32 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp ne i32 %a, %b %cmp = icmp ne i32 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_sle_i1(i32 %a, i32 %b) { define i64 @fold_icmp_i1_ne_0_icmp_sle_i1(i32 %a, i32 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_sle_i1( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_sle_i1(
; CHECK-NEXT: [[CMP:%.*]] = icmp sle i32 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp sle i32 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp sle i32 %a, %b %cmp = icmp sle i32 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_ugt_i64(i64 %a, i64 %b) { define i64 @fold_icmp_i1_ne_0_icmp_ugt_i64(i64 %a, i64 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_ugt_i64( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_ugt_i64(
; CHECK-NEXT: [[CMP:%.*]] = icmp ugt i64 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp ugt i64 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp ugt i64 %a, %b %cmp = icmp ugt i64 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_ult_swap_i64(i64 %a, i64 %b) { define i64 @fold_icmp_i1_ne_0_icmp_ult_swap_i64(i64 %a, i64 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_ult_swap_i64( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_ult_swap_i64(
; CHECK-NEXT: [[CMP:%.*]] = icmp ugt i64 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp ugt i64 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp ugt i64 %a, %b %cmp = icmp ugt i64 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 false, i1 %cmp, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 false, i1 %cmp, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_fcmp_oeq_f32(float %a, float %b) { define i64 @fold_icmp_i1_ne_0_fcmp_oeq_f32(float %a, float %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_fcmp_oeq_f32( ; CHECK-LABEL: @fold_icmp_i1_ne_0_fcmp_oeq_f32(
; CHECK-NEXT: [[CMP:%.*]] = fcmp oeq float [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = fcmp oeq float [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = fcmp oeq float %a, %b %cmp = fcmp oeq float %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_fcmp_une_f32(float %a, float %b) { define i64 @fold_icmp_i1_ne_0_fcmp_une_f32(float %a, float %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_fcmp_une_f32( ; CHECK-LABEL: @fold_icmp_i1_ne_0_fcmp_une_f32(
; CHECK-NEXT: [[CMP:%.*]] = fcmp une float [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = fcmp une float [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = fcmp une float %a, %b %cmp = fcmp une float %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_fcmp_olt_f64(double %a, double %b) { define i64 @fold_icmp_i1_ne_0_fcmp_olt_f64(double %a, double %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_fcmp_olt_f64( ; CHECK-LABEL: @fold_icmp_i1_ne_0_fcmp_olt_f64(
; CHECK-NEXT: [[CMP:%.*]] = fcmp olt double [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = fcmp olt double [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = fcmp olt double %a, %b %cmp = fcmp olt double %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_eq_i4(i4 %a, i4 %b) { define i64 @fold_icmp_i1_ne_0_icmp_eq_i4(i4 %a, i4 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_eq_i4( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_eq_i4(
; CHECK-NEXT: [[CMP:%.*]] = icmp eq i4 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp eq i4 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i4 %a, %b %cmp = icmp eq i4 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_eq_i8(i8 %a, i8 %b) { define i64 @fold_icmp_i1_ne_0_icmp_eq_i8(i8 %a, i8 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_eq_i8( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_eq_i8(
; CHECK-NEXT: [[CMP:%.*]] = icmp eq i8 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp eq i8 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i8 %a, %b %cmp = icmp eq i8 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_eq_i16(i16 %a, i16 %b) { define i64 @fold_icmp_i1_ne_0_icmp_eq_i16(i16 %a, i16 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_eq_i16( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_eq_i16(
; CHECK-NEXT: [[CMP:%.*]] = icmp eq i16 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp eq i16 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i16 %a, %b %cmp = icmp eq i16 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_eq_i36(i36 %a, i36 %b) { define i64 @fold_icmp_i1_ne_0_icmp_eq_i36(i36 %a, i36 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_eq_i36( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_eq_i36(
; CHECK-NEXT: [[CMP:%.*]] = icmp eq i36 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp eq i36 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i36 %a, %b %cmp = icmp eq i36 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_eq_i128(i128 %a, i128 %b) { define i64 @fold_icmp_i1_ne_0_icmp_eq_i128(i128 %a, i128 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_eq_i128( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_eq_i128(
; CHECK-NEXT: [[CMP:%.*]] = icmp eq i128 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp eq i128 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp eq i128 %a, %b %cmp = icmp eq i128 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_fcmp_oeq_f16(half %a, half %b) { define i64 @fold_icmp_i1_ne_0_fcmp_oeq_f16(half %a, half %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_fcmp_oeq_f16( ; CHECK-LABEL: @fold_icmp_i1_ne_0_fcmp_oeq_f16(
; CHECK-NEXT: [[CMP:%.*]] = fcmp oeq half [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = fcmp oeq half [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = fcmp oeq half %a, %b %cmp = fcmp oeq half %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_fcmp_oeq_f128(fp128 %a, fp128 %b) { define i64 @fold_icmp_i1_ne_0_fcmp_oeq_f128(fp128 %a, fp128 %b) {
;
; CHECK-LABEL: @fold_icmp_i1_ne_0_fcmp_oeq_f128( ; CHECK-LABEL: @fold_icmp_i1_ne_0_fcmp_oeq_f128(
; CHECK-NEXT: [[CMP:%.*]] = fcmp oeq fp128 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = fcmp oeq fp128 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = fcmp oeq fp128 %a, %b %cmp = fcmp oeq fp128 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_slt_i4(i4 %a, i4 %b) { define i64 @fold_icmp_i1_ne_0_icmp_slt_i4(i4 %a, i4 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_slt_i4( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_slt_i4(
; CHECK-NEXT: [[CMP:%.*]] = icmp slt i4 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp slt i4 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp slt i4 %a, %b %cmp = icmp slt i4 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_slt_i8(i8 %a, i8 %b) { define i64 @fold_icmp_i1_ne_0_icmp_slt_i8(i8 %a, i8 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_slt_i8( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_slt_i8(
; CHECK-NEXT: [[CMP:%.*]] = icmp slt i8 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp slt i8 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp slt i8 %a, %b %cmp = icmp slt i8 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_slt_i16(i16 %a, i16 %b) { define i64 @fold_icmp_i1_ne_0_icmp_slt_i16(i16 %a, i16 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_slt_i16( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_slt_i16(
; CHECK-NEXT: [[CMP:%.*]] = icmp slt i16 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp slt i16 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp slt i16 %a, %b %cmp = icmp slt i16 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_ult_i4(i4 %a, i4 %b) { define i64 @fold_icmp_i1_ne_0_icmp_ult_i4(i4 %a, i4 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_ult_i4( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_ult_i4(
; CHECK-NEXT: [[CMP:%.*]] = icmp ult i4 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp ult i4 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp ult i4 %a, %b %cmp = icmp ult i4 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_ult_i8(i8 %a, i8 %b) { define i64 @fold_icmp_i1_ne_0_icmp_ult_i8(i8 %a, i8 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_ult_i8( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_ult_i8(
; CHECK-NEXT: [[CMP:%.*]] = icmp ult i8 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp ult i8 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp ult i8 %a, %b %cmp = icmp ult i8 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
define i64 @fold_icmp_i1_ne_0_icmp_ult_i16(i16 %a, i16 %b) { define i64 @fold_icmp_i1_ne_0_icmp_ult_i16(i16 %a, i16 %b) {
; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_ult_i16( ; CHECK-LABEL: @fold_icmp_i1_ne_0_icmp_ult_i16(
; CHECK-NEXT: [[CMP:%.*]] = icmp ult i16 [[A:%.*]], [[B:%.*]] ; CHECK-NEXT: [[CMP:%.*]] = icmp ult i16 [[A:%.*]], [[B:%.*]]
; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i1(i1 [[CMP]], i1 false, i32 33) ; CHECK-NEXT: [[MASK:%.*]] = call i64 @llvm.amdgcn.icmp.i64.i1(i1 [[CMP]], i1 false, i32 33)
; CHECK-NEXT: ret i64 [[MASK]] ; CHECK-NEXT: ret i64 [[MASK]]
; ;
%cmp = icmp ult i16 %a, %b %cmp = icmp ult i16 %a, %b
%mask = call i64 @llvm.amdgcn.icmp.i1(i1 %cmp, i1 false, i32 33) %mask = call i64 @llvm.amdgcn.icmp.i64.i1(i1 %cmp, i1 false, i32 33)
ret i64 %mask ret i64 %mask
} }
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
; llvm.amdgcn.fcmp ; llvm.amdgcn.fcmp
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
declare i64 @llvm.amdgcn.fcmp.f32(float, float, i32 immarg) nounwind readnone convergent declare i64 @llvm.amdgcn.fcmp.i64.f32(float, float, i32 immarg) nounwind readnone convergent
define i64 @invalid_fcmp_code(float %a, float %b) { define i64 @invalid_fcmp_code(float %a, float %b) {
; CHECK-LABEL: @invalid_fcmp_code( ; CHECK-LABEL: @invalid_fcmp_code(
; CHECK-NEXT: [[UNDER:%.*]] = call i64 @llvm.amdgcn.fcmp.f32(float [[A:%.*]], float [[B:%.*]], i32 -1) ; CHECK-NEXT: [[UNDER:%.*]] = call i64 @llvm.amdgcn.fcmp.i64.f32(float [[A:%.*]], float [[B:%.*]], i32 -1)
; CHECK-NEXT: [[OVER:%.*]] = call i64 @llvm.amdgcn.fcmp.f32(float [[A]], float [[B]], i32 16) ; CHECK-NEXT: [[OVER:%.*]] = call i64 @llvm.amdgcn.fcmp.i64.f32(float [[A]], float [[B]], i32 16)
; CHECK-NEXT: [[OR:%.*]] = or i64 [[UNDER]], [[OVER]] ; CHECK-NEXT: [[OR:%.*]] = or i64 [[UNDER]], [[OVER]]
; CHECK-NEXT: ret i64 [[OR]] ; CHECK-NEXT: ret i64 [[OR]]
; ;
%under = call i64 @llvm.amdgcn.fcmp.f32(float %a, float %b, i32 -1) %under = call i64 @llvm.amdgcn.fcmp.i64.f32(float %a, float %b, i32 -1)
%over = call i64 @llvm.amdgcn.fcmp.f32(float %a, float %b, i32 16) %over = call i64 @llvm.amdgcn.fcmp.i64.f32(float %a, float %b, i32 16)
%or = or i64 %under, %over %or = or i64 %under, %over
ret i64 %or ret i64 %or
} }
define i64 @fcmp_constant_inputs_false() { define i64 @fcmp_constant_inputs_false() {
; CHECK-LABEL: @fcmp_constant_inputs_false( ; CHECK-LABEL: @fcmp_constant_inputs_false(
; CHECK-NEXT: ret i64 0 ; CHECK-NEXT: ret i64 0
; ;
%result = call i64 @llvm.amdgcn.fcmp.f32(float 2.0, float 4.0, i32 1) %result = call i64 @llvm.amdgcn.fcmp.i64.f32(float 2.0, float 4.0, i32 1)
ret i64 %result ret i64 %result
} }
define i64 @fcmp_constant_inputs_true() { define i64 @fcmp_constant_inputs_true() {
; CHECK-LABEL: @fcmp_constant_inputs_true( ; CHECK-LABEL: @fcmp_constant_inputs_true(
; CHECK-NEXT: [[RESULT:%.*]] = call i64 @llvm.read_register.i64(metadata !0) #5 ; CHECK-NEXT: [[RESULT:%.*]] = call i64 @llvm.read_register.i64(metadata !0) #5
; CHECK-NEXT: ret i64 [[RESULT]] ; CHECK-NEXT: ret i64 [[RESULT]]
; ;
%result = call i64 @llvm.amdgcn.fcmp.f32(float 2.0, float 4.0, i32 4) %result = call i64 @llvm.amdgcn.fcmp.i64.f32(float 2.0, float 4.0, i32 4)
ret i64 %result ret i64 %result
} }
define i64 @fcmp_constant_to_rhs_olt(float %x) { define i64 @fcmp_constant_to_rhs_olt(float %x) {
; CHECK-LABEL: @fcmp_constant_to_rhs_olt( ; CHECK-LABEL: @fcmp_constant_to_rhs_olt(
; CHECK-NEXT: [[RESULT:%.*]] = call i64 @llvm.amdgcn.fcmp.f32(float [[X:%.*]], float 4.000000e+00, i32 2) ; CHECK-NEXT: [[RESULT:%.*]] = call i64 @llvm.amdgcn.fcmp.i64.f32(float [[X:%.*]], float 4.000000e+00, i32 2)
; CHECK-NEXT: ret i64 [[RESULT]] ; CHECK-NEXT: ret i64 [[RESULT]]
; ;
%result = call i64 @llvm.amdgcn.fcmp.f32(float 4.0, float %x, i32 4) %result = call i64 @llvm.amdgcn.fcmp.i64.f32(float 4.0, float %x, i32 4)
ret i64 %result ret i64 %result
} }
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
; llvm.amdgcn.wqm.vote ; llvm.amdgcn.wqm.vote
; -------------------------------------------------------------------- ; --------------------------------------------------------------------
declare i1 @llvm.amdgcn.wqm.vote(i1) declare i1 @llvm.amdgcn.wqm.vote(i1)
▲ Show 20 Lines • Show All 80 Lines • ▼ Show 20 Lines
; CHECK-NEXT: ret void ; CHECK-NEXT: ret void
; ;
%tmp0 = call i32 @llvm.amdgcn.update.dpp.i32(i32 undef, i32 %in1, i32 4, i32 15, i32 15, i1 1) %tmp0 = call i32 @llvm.amdgcn.update.dpp.i32(i32 undef, i32 %in1, i32 4, i32 15, i32 15, i1 1)
store i32 %tmp0, i32 addrspace(1)* %out store i32 %tmp0, i32 addrspace(1)* %out
ret void ret void
} }
; CHECK: attributes #5 = { convergent } ; CHECK: attributes #5 = { convergent }

test/Verifier/AMDGPU/intrinsic-immarg.ll

Show First 20 Lines • Show All 117 Lines • ▼ Show 20 Linesdefine void @exp_compr_invalid_inputs(i32 %tgt, i32 %en, i1 %bool) {
; CHECK: immarg operand has non-immediate parameter ; CHECK: immarg operand has non-immediate parameter
; CHECK-NEXT: i1 %bool ; CHECK-NEXT: i1 %bool
; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 5, <2 x half> <half 0xH3C00, half 0xH4000>, <2 x half> <half 0xH3800, half 0xH4400>, i1 false, i1 %bool) ; CHECK-NEXT: call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 5, <2 x half> <half 0xH3C00, half 0xH4000>, <2 x half> <half 0xH3800, half 0xH4400>, i1 false, i1 %bool)
call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 5, <2 x half> <half 1.0, half 2.0>, <2 x half> <half 0.5, half 4.0>, i1 false, i1 %bool) call void @llvm.amdgcn.exp.compr.v2f16(i32 0, i32 5, <2 x half> <half 1.0, half 2.0>, <2 x half> <half 0.5, half 4.0>, i1 false, i1 %bool)
ret void ret void
} }
declare i64 @llvm.amdgcn.icmp.i32(i32, i32, i32) declare i64 @llvm.amdgcn.icmp.i64.i32(i32, i32, i32)
define i64 @invalid_nonconstant_icmp_code(i32 %a, i32 %b, i32 %c) { define i64 @invalid_nonconstant_icmp_code(i32 %a, i32 %b, i32 %c) {
; CHECK: immarg operand has non-immediate parameter ; CHECK: immarg operand has non-immediate parameter
; CHECK-NEXT: i32 %c ; CHECK-NEXT: i32 %c
; CHECK-NEXT: %result = call i64 @llvm.amdgcn.icmp.i32(i32 %a, i32 %b, i32 %c) ; CHECK-NEXT: %result = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %a, i32 %b, i32 %c)
%result = call i64 @llvm.amdgcn.icmp.i32(i32 %a, i32 %b, i32 %c) %result = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %a, i32 %b, i32 %c)
ret i64 %result ret i64 %result
} }
declare i64 @llvm.amdgcn.fcmp.f32(float, float, i32) declare i64 @llvm.amdgcn.fcmp.i64.f32(float, float, i32)
define i64 @invalid_nonconstant_fcmp_code(float %a, float %b, i32 %c) { define i64 @invalid_nonconstant_fcmp_code(float %a, float %b, i32 %c) {
; CHECK: immarg operand has non-immediate parameter ; CHECK: immarg operand has non-immediate parameter
; CHECK-NEXT: i32 %c ; CHECK-NEXT: i32 %c
; CHECK-NEXT: %result = call i64 @llvm.amdgcn.fcmp.f32(float %a, float %b, i32 %c) ; CHECK-NEXT: %result = call i64 @llvm.amdgcn.fcmp.i64.f32(float %a, float %b, i32 %c)
%result = call i64 @llvm.amdgcn.fcmp.f32(float %a, float %b, i32 %c) %result = call i64 @llvm.amdgcn.fcmp.i64.f32(float %a, float %b, i32 %c)
ret i64 %result ret i64 %result
} }
declare i32 @llvm.amdgcn.atomic.inc.i32.p3i32(i32 addrspace(3)* nocapture, i32, i32, i32, i1) declare i32 @llvm.amdgcn.atomic.inc.i32.p3i32(i32 addrspace(3)* nocapture, i32, i32, i32, i1)
define amdgpu_kernel void @invalid_atomic_inc(i32 addrspace(1)* %out, i32 addrspace(3)* %ptr, i32 %var, i1 %bool) { define amdgpu_kernel void @invalid_atomic_inc(i32 addrspace(1)* %out, i32 addrspace(3)* %ptr, i32 %var, i1 %bool) {
; CHECK: immarg operand has non-immediate parameter ; CHECK: immarg operand has non-immediate parameter
; CHECK-NEXT: i32 %var ; CHECK-NEXT: i32 %var
; CHECK-NEXT: %result0 = call i32 @llvm.amdgcn.atomic.inc.i32.p3i32(i32 addrspace(3)* %ptr, i32 42, i32 %var, i32 0, i1 false) ; CHECK-NEXT: %result0 = call i32 @llvm.amdgcn.atomic.inc.i32.p3i32(i32 addrspace(3)* %ptr, i32 42, i32 %var, i32 0, i1 false)
▲ Show 20 Lines • Show All 403 Lines • Show Last 20 Lines