[AMDGPU] Set optional PAL metadata

[sebastian-ne]

Summary

Set informational fields in the .shader_functions table.

Also correct the documentation, .scratch_memory_size and .lds_size are
integers.

[foad]

llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp

1406

Is this the number of vgprs that are actually used by the generated ISA? From the name "NumVGPRsForWavesPerEU" it sounds like it might be the maximum number that could be used while still achieving some specified occupancy.

[sebastian-ne]

llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp

1406

I’m not sure either by looking at the name.
But it seems to be the number of used registers, at least in the test case.
Also, this variable is used to set the normal .vgpr_count too (with normal, I mean the .vgpr_count that we already set, in contrast to the one in .shader_functions, which is added here).

[sebastian-ne]

llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp

1406

I looked a bit more, NumVGPRsForWavesPerEU is the amount of VGPRs used by the ISA, except if the function attribute amdgpu-waves-per-eu sets a *maximum* of waves per eu. Then NumVGPRsForWavesPerEU will be raised artificially, so that no more waves than specified fit in.
I guess this is a fair thing to do.
(For the record, if it is not obvious from the comments somewhere, .vgpr_count is only used by tools like RGP and does not affect actual execution.)

llvm/docs/AMDGPUUsage.rst

.. table:: AMDPAL Code Object Shader Function Metadata Map
      :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table
 
      ============================= ============== =================================================================
      String Key                    Value Type     Description
      ============================= ============== =================================================================
      ".api_shader_hash"            sequence of    Input shader hash, typically passed in from the client. The value
                                    2 integers     is implementation defined, and can not be relied on between
                                                   different builds of the compiler.
-     ".scratch_memory_size"        sequence of    Size in bytes of scratch memory used by the shader.
-                                   2 integers
-     ".lds_size"                   sequence of    Size in bytes of LDS memory.
-                                   2 integers
+     ".scratch_memory_size"        integer        Size in bytes of scratch memory used by the shader.
+     ".lds_size"                   integer        Size in bytes of LDS memory.
      ".vgpr_count"                 integer        Number of VGPRs used by the shader.
      ".sgpr_count"                 integer        Number of SGPRs used by the shader.
      ".stack_frame_size_in_bytes"  integer        Amount of stack size used by the shader.
      ".shader_subtype"             string         Shader subtype/kind. Values include:
 
                                                     - "Unknown"
 
      ============================= ============== =================================================================

llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp

void AMDGPUAsmPrinter::EmitPALMetadata(const MachineFunction &MF,
   if (STM.isWave32())
     MD->setWave32(MF.getFunction().getCallingConv());
 }
 
 void AMDGPUAsmPrinter::emitPALFunctionMetadata(const MachineFunction &MF) {
   auto *MD = getTargetStreamer()->getPALMetadata();
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   MD->setFunctionScratchSize(MF, MFI.getStackSize());
+
   // Set compute registers
   MD->setRsrc1(CallingConv::AMDGPU_CS,
                CurrentProgramInfo.getPGMRSrc1(CallingConv::AMDGPU_CS));
   MD->setRsrc2(CallingConv::AMDGPU_CS, CurrentProgramInfo.ComputePGMRSrc2);
+
+  // Set optional info
+  MD->setFunctionLdsSize(MF, CurrentProgramInfo.LDSSize);
+  MD->setFunctionNumUsedVgprs(MF, CurrentProgramInfo.NumVGPRsForWavesPerEU);
+  MD->setFunctionNumUsedSgprs(MF, CurrentProgramInfo.NumSGPRsForWavesPerEU);
 }
 
 // This is supposed to be log2(Size)
 static amd_element_byte_size_t getElementByteSizeValue(unsigned Size) {
   switch (Size) {
   case 4:
     return AMD_ELEMENT_4_BYTES;
   case 8:

llvm/lib/Target/AMDGPU/Utils/AMDGPUPALMetadata.h

public:
   void setNumUsedSgprs(unsigned CC, unsigned Val);
 
   // Set the scratch size in the metadata.
   void setScratchSize(unsigned CC, unsigned Val);
 
   // Set the stack frame size of a function in the metadata.
   void setFunctionScratchSize(const MachineFunction &MF, unsigned Val);
 
+  // Set the amount of LDS used in bytes in the metadata. This is an optional
+  // advisory record for logging etc; wave dispatch actually uses the rsrc1
+  // register for the shader stage to determine the amount of LDS to allocate.
+  void setFunctionLdsSize(const MachineFunction &MF, unsigned Val);
+
+  // Set the number of used vgprs in the metadata. This is an optional advisory
+  // record for logging etc; wave dispatch actually uses the rsrc1 register for
+  // the shader stage to determine the number of vgprs to allocate.
+  void setFunctionNumUsedVgprs(const MachineFunction &MF, unsigned Val);
+
+  // Set the number of used sgprs in the metadata. This is an optional advisory
+  // record for logging etc; wave dispatch actually uses the rsrc1 register for
+  // the shader stage to determine the number of sgprs to allocate.
+  void setFunctionNumUsedSgprs(const MachineFunction &MF, unsigned Val);
+
   // Set the hardware register bit in PAL metadata to enable wave32 on the
   // shader of the given calling convention.
   void setWave32(unsigned CC);
 
   // Emit the accumulated PAL metadata as asm directives.
   // This is called from AMDGPUTargetAsmStreamer::Finish().
   void toString(std::string &S);
 

llvm/lib/Target/AMDGPU/Utils/AMDGPUPALMetadata.cpp

 
 // Set the stack frame size of a function in the metadata.
 void AMDGPUPALMetadata::setFunctionScratchSize(const MachineFunction &MF,
                                                unsigned Val) {
   auto Node = getShaderFunction(MF.getFunction().getName());
   Node[".stack_frame_size_in_bytes"] = MsgPackDoc.getNode(Val);
 }
 
+// Set the amount of LDS used in bytes in the metadata.
+void AMDGPUPALMetadata::setFunctionLdsSize(const MachineFunction &MF,
+                                           unsigned Val) {
+  auto Node = getShaderFunction(MF.getFunction().getName());
+  Node[".lds_size"] = MsgPackDoc.getNode(Val);
+}
+
+// Set the number of used vgprs in the metadata.
+void AMDGPUPALMetadata::setFunctionNumUsedVgprs(const MachineFunction &MF,
+                                                unsigned Val) {
+  auto Node = getShaderFunction(MF.getFunction().getName());
+  Node[".vgpr_count"] = MsgPackDoc.getNode(Val);
+}
+
+// Set the number of used vgprs in the metadata.
+void AMDGPUPALMetadata::setFunctionNumUsedSgprs(const MachineFunction &MF,
+                                                unsigned Val) {
+  auto Node = getShaderFunction(MF.getFunction().getName());
+  Node[".sgpr_count"] = MsgPackDoc.getNode(Val);
+}
+
 // Set the hardware register bit in PAL metadata to enable wave32 on the
 // shader of the given calling convention.
 void AMDGPUPALMetadata::setWave32(unsigned CC) {
   switch (CC) {
   case CallingConv::AMDGPU_HS:
     setRegister(PALMD::R_A2D5_VGT_SHADER_STAGES_EN, S_028B54_HS_W32_EN(1));
     break;
   case CallingConv::AMDGPU_GS:

llvm/test/CodeGen/AMDGPU/amdpal-callable.ll

-; RUN: llc -mtriple=amdgcn--amdpal -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,SDAG -enable-var-scope %s
-; RUN: llc -mtriple=amdgcn--amdpal -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,SDAG -enable-var-scope %s
-; RUN: llc -global-isel -mtriple=amdgcn--amdpal -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GISEL -enable-var-scope %s
+; RUN: llc -mtriple=amdgcn--amdpal -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,SDAG,GFX8 -enable-var-scope %s
+; RUN: llc -mtriple=amdgcn--amdpal -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,SDAG,GFX9 -enable-var-scope %s
+; RUN: llc -global-isel -mtriple=amdgcn--amdpal -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GISEL,GFX9 -enable-var-scope %s
 
 ; Make sure this interacts well with -amdgpu-fixed-function-abi
-; RUN: llc -mtriple=amdgcn--amdpal -mcpu=gfx900 -amdgpu-fixed-function-abi -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,SDAG -enable-var-scope %s
+; RUN: llc -mtriple=amdgcn--amdpal -mcpu=gfx900 -amdgpu-fixed-function-abi -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,SDAG,GFX9 -enable-var-scope %s
 
 declare float @extern_func(float) #0
 declare float @extern_func_many_args(<64 x float>) #0
 
 @funcptr = external hidden unnamed_addr addrspace(4) constant void()*, align 4
 
 define amdgpu_gfx float @no_stack(float %arg0) #0 {
   %add = fadd float %arg0, 1.0
define amdgpu_gfx float @simple_lds_recurse(float %arg0) #0 {
   %res = call amdgpu_gfx float @simple_lds_recurse(float %val)
   ret float %res
 }
 
 attributes #0 = { nounwind }
 
 ; GCN: amdpal.pipelines:
 ; GCN-NEXT:  - .registers:
-; SDAG-NEXT:      0x2e12 (COMPUTE_PGM_RSRC1): 0xaf03cf{{$}}
-; SDAG-NEXT:      0x2e13 (COMPUTE_PGM_RSRC2): 0x8001{{$}}
-; GISEL-NEXT:      0x2e12 (COMPUTE_PGM_RSRC1): 0xaf03cf{{$}}
-; GISEL-NEXT:      0x2e13 (COMPUTE_PGM_RSRC2): 0x8001{{$}}
+; GCN-NEXT:      0x2e12 (COMPUTE_PGM_RSRC1): 0xaf03cf{{$}}
+; GCN-NEXT:      0x2e13 (COMPUTE_PGM_RSRC2): 0x8001{{$}}
 ; GCN-NEXT:    .shader_functions:
 ; GCN-NEXT:      dynamic_stack:
+; GCN-NEXT:        .lds_size:       0{{$}}
+; GCN-NEXT:        .sgpr_count:     0x24{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x10{{$}}
+; SDAG-NEXT:        .vgpr_count:     0x2{{$}}
+; GISEL-NEXT:        .vgpr_count:     0x3{{$}}
 ; GCN-NEXT:      dynamic_stack_loop:
+; GCN-NEXT:        .lds_size:       0{{$}}
+; SDAG-NEXT:        .sgpr_count:     0x22{{$}}
+; GISEL-NEXT:        .sgpr_count:     0x24{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x10{{$}}
+; SDAG-NEXT:        .vgpr_count:     0x3{{$}}
+; GISEL-NEXT:        .vgpr_count:     0x4{{$}}
 ; GCN-NEXT:      multiple_stack:
+; GCN-NEXT:        .lds_size:       0{{$}}
+; GCN-NEXT:        .sgpr_count:     0x21{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x24{{$}}
+; GCN-NEXT:        .vgpr_count:     0x3{{$}}
 ; GCN-NEXT:      no_stack:
+; GCN-NEXT:        .lds_size:       0{{$}}
+; GCN-NEXT:        .sgpr_count:     0x20{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0{{$}}
+; GCN-NEXT:        .vgpr_count:     0x1{{$}}
 ; GCN-NEXT:      no_stack_call:
+; GCN-NEXT:        .lds_size:       0{{$}}
+; GCN-NEXT:        .sgpr_count:     0x20{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0{{$}}
+; GCN-NEXT:        .vgpr_count:     0x1{{$}}
 ; GCN-NEXT:      no_stack_extern_call:
+; GCN-NEXT:        .lds_size:       0{{$}}
+; GFX8-NEXT:        .sgpr_count:     0x68{{$}}
+; GFX9-NEXT:        .sgpr_count:     0x66{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x10{{$}}
+; GCN-NEXT:        .vgpr_count:     0x40{{$}}
 ; GCN-NEXT:      no_stack_extern_call_many_args:
-; SDAG-NEXT:        .stack_frame_size_in_bytes: 0x90{{$}}
-; GISEL-NEXT:        .stack_frame_size_in_bytes: 0x90{{$}}
+; GCN-NEXT:        .lds_size:       0{{$}}
+; GFX8-NEXT:        .sgpr_count:     0x68{{$}}
+; GFX9-NEXT:        .sgpr_count:     0x66{{$}}
+; GCN-NEXT:        .stack_frame_size_in_bytes: 0x90{{$}}
+; GCN-NEXT:        .vgpr_count:     0x40{{$}}
 ; GCN-NEXT:      no_stack_indirect_call:
+; GCN-NEXT:        .lds_size:       0{{$}}
+; GFX8-NEXT:        .sgpr_count:     0x68{{$}}
+; GFX9-NEXT:        .sgpr_count:     0x66{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x10{{$}}
+; GCN-NEXT:        .vgpr_count:     0x40{{$}}
 ; GCN-NEXT:      simple_lds:
+; GCN-NEXT:        .lds_size:       0x100{{$}}
+; GCN-NEXT:        .sgpr_count:     0x20{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0{{$}}
+; GCN-NEXT:        .vgpr_count:     0x1{{$}}
 ; GCN-NEXT:      simple_lds_recurse:
+; GCN-NEXT:        .lds_size:       0x100{{$}}
+; GFX8-NEXT:        .sgpr_count:     0x68{{$}}
+; GFX9-NEXT:        .sgpr_count:     0x66{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x10{{$}}
+; GCN-NEXT:        .vgpr_count:     0x40{{$}}
 ; GCN-NEXT:      simple_stack:
+; GCN-NEXT:        .lds_size:       0{{$}}
+; GCN-NEXT:        .sgpr_count:     0x21{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x14{{$}}
+; GCN-NEXT:        .vgpr_count:     0x2{{$}}
 ; GCN-NEXT:      simple_stack_call:
+; GCN-NEXT:        .lds_size:       0{{$}}
+; GCN-NEXT:        .sgpr_count:     0x22{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x20{{$}}
+; GCN-NEXT:        .vgpr_count:     0x3{{$}}
 ; GCN-NEXT:      simple_stack_extern_call:
+; GCN-NEXT:        .lds_size:       0{{$}}
+; GFX8-NEXT:        .sgpr_count:     0x68{{$}}
+; GFX9-NEXT:        .sgpr_count:     0x66{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x20{{$}}
+; GCN-NEXT:        .vgpr_count:     0x40{{$}}
 ; GCN-NEXT:      simple_stack_indirect_call:
+; GCN-NEXT:        .lds_size:       0{{$}}
+; GFX8-NEXT:        .sgpr_count:     0x68{{$}}
+; GFX9-NEXT:        .sgpr_count:     0x66{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x20{{$}}
+; GCN-NEXT:        .vgpr_count:     0x40{{$}}
 ; GCN-NEXT:      simple_stack_recurse:
+; GCN-NEXT:        .lds_size:       0{{$}}
+; GFX8-NEXT:        .sgpr_count:     0x68{{$}}
+; GFX9-NEXT:        .sgpr_count:     0x66{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x20{{$}}
+; GCN-NEXT:        .vgpr_count:     0x40{{$}}
 ; GCN-NEXT: ...