Change the default number of teams.
Based on kernel register usage, adjust the number of threads in a team.
Includes a corner case fix.
This change is dependent on https://reviews.llvm.org/D98829
dhruvachak on Mar 17 2021, 5:52 PM.Authored by
This is really interesting. The idea seems to be to choose the dispatch parameters based on the kernel metadata and the limits of the machine.
What's the underlying heuristic? Break across N CU's in chunks that match the occupancy limits of each CU?
If so we probably want to compare LDS usage as well to avoid partitioning poorly for that.
Maybe others - there might be a performance cliff on amount of private memory too.
Yes, that's the idea.
Agreed. However, I don't see LDS usage in the metadata table in the image. Is it present there?
In theory, a very high sgpr count can limit the number of available workgroups if that's not factored in for determining the number of threads. But in practice, VGPRs tend to be the primary limiting factor. So perhaps we can start with using VGPRs for this purpose and have experience guide us in the future.
Yes, see https://llvm.org/docs/AMDGPUUsage.html for the list of what we can expect. What may not be obvious is that the metadata calls it ".group_segment_fixed_size". I don't know the origin of the terminology, maybe opencl?
If I understand correctly, occupancy rules all look something like (resource used / resource available) == number simultaneous, where one of the resources tends to be limiting. Offhand, I think that's VGPR, SGPR, LDS (group segment). I think there's also an architecture dependent upper bound on how many things can run at once even if they use very little of those, maybe 8 for gfx9 and 16 for gfx10.
If that's right, perhaps the calculation should look something like:
uint vgpr_occupancy = vgpr_used / vgpr_available; uint sgpr_occupancy = sgpr_used / sgpr_available; uint lds_occupancy = lds_used / lds_available; uint limiting_occupancy = min(vgpr_occupancy, sgpr_occupacny, lds_occupancy);
and then we derive threadsPerGroup from that occupancy and the various other considerations.
Thanks for the pointer to the group segment. Yes, in general, my idea is similar to what you outlined above. However, note that SGPRs and LDS are at different granularities compared to VGPRs. VGPRs are per-thread, SGPRs are shared within a wavefront, and LDS is shared within a workgroup. So while VGPRs can be used to limit the number of threads, perhaps SGPRs and LDS can be used to limit the number of teams.
Let me split up this patch further. I would like to land the default num_teams change sooner rather than later since that's a simple change and has shown improved performance. So let me separate that out. Incorporating SGPRs/LDS to constrain teams/threads will need more experimentation.