This is an archive of the discontinued LLVM Phabricator instance.

Differential D86003

[AMDGPU] Fix MAI ld/st hazard handling
ClosedPublic

Authored by rampitec on Aug 14 2020, 3:42 PM.

Details

Reviewers
arsenm
Commits
rG43a38dc25173: [AMDGPU] Fix MAI ld/st hazard handling
Summary

It did not process hazard for ds_permute because it does not
load or store even though it is DS.

Diff Detail

Event Timeline

rampitec created this revision.Aug 14 2020, 3:42 PM
Herald added a project: Restricted Project. · View Herald TranscriptAug 14 2020, 3:42 PM
Herald added subscribers: kerbowa, hiraditya, t-tye and 6 others. · View Herald Transcript
rampitec requested review of this revision.Aug 14 2020, 3:42 PM
Herald added a subscriber: wdng. · View Herald TranscriptAug 14 2020, 3:42 PM
arsenm accepted this revision.Aug 14 2020, 5:03 PM
This revision is now accepted and ready to land.Aug 14 2020, 5:03 PM
Closed by commit rG43a38dc25173: [AMDGPU] Fix MAI ld/st hazard handling (authored by rampitec). · Explain WhyAug 14 2020, 5:08 PM
This revision was automatically updated to reflect the committed changes.
rampitec added a commit: rG43a38dc25173: [AMDGPU] Fix MAI ld/st hazard handling.

Revision Contents

PathSize
llvm/
lib/
Target/
AMDGPU/
8 lines
test/
CodeGen/
AMDGPU/
39 lines

Diff 285775

llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp

Show First 20 Lines • Show All 179 Lines • ▼ Show 20 LinesGCNHazardRecognizer::getHazardType(SUnit *SU, int Stalls) {
if (ST.hasReadM0SendMsgHazard() && isSendMsgTraceDataOrGDS(TII, *MI) && if (ST.hasReadM0SendMsgHazard() && isSendMsgTraceDataOrGDS(TII, *MI) &&
checkReadM0Hazards(MI) > 0) checkReadM0Hazards(MI) > 0)
return NoopHazard; return NoopHazard;
if (SIInstrInfo::isMAI(*MI) && checkMAIHazards(MI) > 0) if (SIInstrInfo::isMAI(*MI) && checkMAIHazards(MI) > 0)
return NoopHazard; return NoopHazard;
if (MI->mayLoadOrStore() && checkMAILdStHazards(MI) > 0) if ((SIInstrInfo::isVMEM(*MI) ||
SIInstrInfo::isFLAT(*MI) ||
SIInstrInfo::isDS(*MI)) && checkMAILdStHazards(MI) > 0)
return NoopHazard; return NoopHazard;
if (MI->isInlineAsm() && checkInlineAsmHazards(MI) > 0) if (MI->isInlineAsm() && checkInlineAsmHazards(MI) > 0)
return NoopHazard; return NoopHazard;
return NoHazard; return NoHazard;
} }
▲ Show 20 Lines • Show All 86 Lines • ▼ Show 20 Lines if (ST.hasReadM0MovRelInterpHazard() && (TII.isVINTRP(*MI) ||
return std::max(WaitStates, checkReadM0Hazards(MI)); return std::max(WaitStates, checkReadM0Hazards(MI));
if (ST.hasReadM0SendMsgHazard() && isSendMsgTraceDataOrGDS(TII, *MI)) if (ST.hasReadM0SendMsgHazard() && isSendMsgTraceDataOrGDS(TII, *MI))
return std::max(WaitStates, checkReadM0Hazards(MI)); return std::max(WaitStates, checkReadM0Hazards(MI));
if (SIInstrInfo::isMAI(*MI)) if (SIInstrInfo::isMAI(*MI))
return std::max(WaitStates, checkMAIHazards(MI)); return std::max(WaitStates, checkMAIHazards(MI));
if (MI->mayLoadOrStore()) if (SIInstrInfo::isVMEM(*MI) ||
SIInstrInfo::isFLAT(*MI) ||
SIInstrInfo::isDS(*MI))
return std::max(WaitStates, checkMAILdStHazards(MI)); return std::max(WaitStates, checkMAILdStHazards(MI));
return WaitStates; return WaitStates;
} }
void GCNHazardRecognizer::EmitNoop() { void GCNHazardRecognizer::EmitNoop() {
EmittedInstrs.push_front(nullptr); EmittedInstrs.push_front(nullptr);
} }
▲ Show 20 Lines • Show All 1,111 Lines • Show Last 20 Lines

llvm/test/CodeGen/AMDGPU/mai-hazards.mir

Show First 20 Lines • Show All 395 Lines • ▼ Show 20 Lines
name: accvgpr_read_write_vgpr_load name: accvgpr_read_write_vgpr_load
body: | body: |
bb.0: bb.0:
$vgpr0 = V_ACCVGPR_READ_B32 killed $agpr0, implicit $exec $vgpr0 = V_ACCVGPR_READ_B32 killed $agpr0, implicit $exec
$vgpr3 = FLAT_LOAD_DWORD $vgpr0_vgpr1, 0, 0, 0, 0, implicit $exec, implicit $flat_scr $vgpr3 = FLAT_LOAD_DWORD $vgpr0_vgpr1, 0, 0, 0, 0, implicit $exec, implicit $flat_scr
... ...
--- ---
# GCN-LABEL: name: accvgpr_read_write_vgpr_ds_permute
# GCN: V_ACCVGPR_READ_B32
# GCN-NEXT: S_NOP 0
# GCN-NEXT: S_NOP 0
# GCN-NEXT: DS_PERMUTE_B32
name: accvgpr_read_write_vgpr_ds_permute
body: |
bb.0:
$vgpr0 = V_ACCVGPR_READ_B32 killed $agpr0, implicit $exec
$vgpr1 = DS_PERMUTE_B32 $vgpr0, $vgpr1, 0, implicit $exec
...
---
# GCN-LABEL: name: accvgpr_read_write_vgpr_flat_load
# GCN: V_ACCVGPR_READ_B32
# GCN-NEXT: S_NOP 0
# GCN-NEXT: S_NOP 0
# GCN-NEXT: FLAT_LOAD_DWORD
name: accvgpr_read_write_vgpr_flat_load
body: |
bb.0:
$vgpr0 = V_ACCVGPR_READ_B32 killed $agpr0, implicit $exec
$vgpr4 = FLAT_LOAD_DWORD $vgpr0_vgpr1, 0, 0, 0, 0, implicit $exec, implicit $flat_scr
...
---
# GCN-LABEL: name: accvgpr_read_write_vgpr_buffer_store
# GCN: V_ACCVGPR_READ_B32
# GCN-NEXT: S_NOP 0
# GCN-NEXT: S_NOP 0
# GCN-NEXT: BUFFER_STORE_DWORD_OFFSET
name: accvgpr_read_write_vgpr_buffer_store
body: |
bb.0:
$vgpr0 = V_ACCVGPR_READ_B32 killed $agpr0, implicit $exec
BUFFER_STORE_DWORD_OFFSET $vgpr0, $sgpr0_sgpr1_sgpr2_sgpr3, 0, 0, 0, 0, 0, 0, 0, implicit $exec
...
---
# GCN-LABEL: name: accvgpr_read_write_vgpr_store # GCN-LABEL: name: accvgpr_read_write_vgpr_store
# GCN: V_ACCVGPR_READ_B32 # GCN: V_ACCVGPR_READ_B32
# GCN-NEXT: S_NOP 0 # GCN-NEXT: S_NOP 0
# GCN-NEXT: S_NOP 0 # GCN-NEXT: S_NOP 0
# GCN-NEXT: DS_WRITE_B32 # GCN-NEXT: DS_WRITE_B32
name: accvgpr_read_write_vgpr_store name: accvgpr_read_write_vgpr_store
body: | body: |
bb.0: bb.0:
▲ Show 20 Lines • Show All 46 Lines • Show Last 20 Lines