AMDGPU/SI: Set INDEX_STRIDE for scratch coalescing

[mareko]

Summary

Mesa and other users must set this to enable coalescing:

• STRIDE = 0
• SWIZZLE_ENABLE = 1

DiRT Showdown - apitrace results:

Before: 17.73 FPS
After:  19.77 FPS

[arsenm]

Can you add a test that shows the bit change in the resource setup? I would expect the tests would be breaking on this already

lib/Target/AMDGPU/SIInstrInfo.cpp

3098

C++ style comments

[mareko]

Why would it be breaking? This is the only place that clears DATA_FORMAT and sets ADD_TID_ENABLE, both of which are required for scratch to even work. Or did you mean something else?

[arsenm]

In D21136#452923, @mareko wrote:

Why would it be breaking? This is the only place that clears DATA_FORMAT and sets ADD_TID_ENABLE, both of which are required for scratch to even work. Or did you mean something else?

The tests check the complete 32-bit value moved into the register

[mareko]

Fixed the tests and the comment.

[qcolombet]

+llvm-commit

[arsenm]

LGTM

lib/Target/AMDGPU/SIInstrInfo.cpp

3098

Ditto

lib/Target/AMDGPU/SIInstrInfo.h

553

UINT64_C() instead of LL

lib/Target/AMDGPU/SIInstrInfo.h

namespace AMDGPU {
 
   LLVM_READONLY
   int getAtomicRetOp(uint16_t Opcode);
 
   LLVM_READONLY
   int getAtomicNoRetOp(uint16_t Opcode);
 
   const uint64_t RSRC_DATA_FORMAT = 0xf00000000000LL;
-  const uint64_t RSRC_TID_ENABLE = 1LL << 55;
-  const uint64_t RSRC_ELEMENT_SIZE_SHIFT = 51;
+  const uint64_t RSRC_ELEMENT_SIZE_SHIFT = (32 + 19);
+  const uint64_t RSRC_INDEX_STRIDE_SHIFT = (32 + 21);
+  const uint64_t RSRC_TID_ENABLE = 1LL << (32 + 23);
 } // End namespace AMDGPU
 
 namespace SI {
 namespace KernelInputOffsets {
 
 /// Offsets in bytes from the start of the input buffer
 enum Offsets {
   NGROUPS_X = 0,

lib/Target/AMDGPU/SIInstrInfo.cpp

 
 uint64_t SIInstrInfo::getScratchRsrcWords23() const {
   uint64_t Rsrc23 = getDefaultRsrcDataFormat() |
                     AMDGPU::RSRC_TID_ENABLE |
                     0xffffffff; // Size;
 
   uint64_t EltSizeValue = Log2_32(ST.getMaxPrivateElementSize()) - 1;
 
-  Rsrc23 |= (EltSizeValue << AMDGPU::RSRC_ELEMENT_SIZE_SHIFT);
+  Rsrc23 |= (EltSizeValue << AMDGPU::RSRC_ELEMENT_SIZE_SHIFT) |
+            (3LL << AMDGPU::RSRC_INDEX_STRIDE_SHIFT); /* IndexStride = 64 */
 
   // If TID_ENABLE is set, DATA_FORMAT specifies stride bits [14:17].
   // Clear them unless we want a huge stride.
   if (ST.getGeneration() >= AMDGPUSubtarget::VOLCANIC_ISLANDS)
     Rsrc23 &= ~AMDGPU::RSRC_DATA_FORMAT;
 
   return Rsrc23;
 }