Page MenuHomePhabricator

[libomptarget] [amdgpu] After a kernel dispatch packet is published, its contents must not be accessed.
ClosedPublic

Authored by dhruvachak on Sep 28 2021, 11:59 PM.

Details

Summary

Fixes: SWDEV-275232 (With contributions from Ammar Elwazir, Laurent Morichetti, and Tony Tye)

The current code is racy. After the packet is submitted, the GPU will increment the read index. If this wraps around before the memory is read from it'll refer to a signal from an unrelated packet. Change avoids reading from the packet post-submission.

Diff Detail

Event Timeline

dhruvachak created this revision.Sep 28 2021, 11:59 PM
dhruvachak requested review of this revision.Sep 28 2021, 11:59 PM
Herald added a project: Restricted Project. · View Herald TranscriptSep 28 2021, 11:59 PM
dhruvachak edited the summary of this revision. (Show Details)Sep 29 2021, 12:18 AM

Change looks non-functional to me. Lifetime of signal s increased, accesses to signal and kernarg memory are made through variables instead of lifting back out of the queue. So that's fine.

What's the problem with the current code? SWDEV-275232 looks like an internal link, and I don't know of anything in the HSA spec to say the contents of the queue buffer can't be touched after the signal store + atomic write to the top combination. Is this requirement documented somewhere?

JonChesterfield accepted this revision.Sep 29 2021, 9:08 AM
JonChesterfield edited the summary of this revision. (Show Details)

Some discussion occurred offline. I don't have a reference to somewhere in the HSA spec saying that this memory cannot safely be read after publishing but it follows directly from another thread potentially reusing the same memory before it is accessed here.

This revision is now accepted and ready to land.Sep 29 2021, 9:10 AM
This revision was landed with ongoing or failed builds.Sep 29 2021, 9:22 AM
This revision was automatically updated to reflect the committed changes.

@t-tye found the reference
http://hsa.glossner.org/wp-content/uploads/2021/02/HSA-SysArch-1.2.pdf

Section 2.8.3 Queue Mechanics
When an agent assigns an AQL packet to the packet processor, the ownership of the AQL
packet is transferred. The packet processor may update the AQL packet at any time after
packet submission, and the submitting agent should not rely on the content of the AQL packet
after submission.