[AMDGPU] Add gfx10 assembler directive to specify shared VGPR count

[t-tye]

llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

4881

How does the value relate to the total VGPR count? Are these in addition, or this just indicates how many of the VGPRs are shared? If the later, is there an error for when this value exceeds the requested VGPR count?

How does new_vgpr work? Is it agnostic to which VGPRs are private and shared? Seems it probably is so that present no issues with adding shared VGPRs but wanted to check.

[rampitec]

Also needs disassembler change.

llvm/docs/AMDGPUUsage.rst

12377

Should probably specify it is GFX10 only.

llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUTargetStreamer.cpp

450

Only print it if IVersion.Major >= 10.

[kzhuravl]

In D105507#2860701, @rampitec wrote:

Also needs disassembler change.

Disassembler support is already here, because the asm printing is done AMDGPUTargetAsmStreamer.

llvm/docs/AMDGPUUsage.rst

12377

It already says "GFX10" in one of the columns.

llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

4881

Will post a follow up patch shortly.

llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUTargetStreamer.cpp

450

It is already the case. See line 427 in this patch set.

[lamb-j]

Add check for shared_vgpr_count exceeding next_free_vgpr

[lamb-j]

llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

4881

I've added a check/error for when shared_vgpr_count exceeds next_free_vgpr. I'm still looking into "new_vgpr" and if any changes need to be made related to that.

[lamb-j]

Adding checks for wavefront size and multiple of 8

[lamb-j]

Updating description to match hardware description, adding appropriate checks

[lamb-j]

Removing extra newline

[arsenm]

llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

4974

Braces

4977

Leftover debug printing

4978

Braces, spaces around *

4981

I don't think this needs the newline

[lamb-j]

Removing accidental print statement and fixing formatting

[lamb-j]

llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

4974

Do we want braces here because the TokError() string is split over two lines?

[t-tye]

llvm/docs/AMDGPUUsage.rst

4454–4458

Suggest reword to:

Number of shared VGPR blocks for wavefront size 64 when executing in subvector mode. For wavefront size 64 the value is 0-15, representing 0-120 VGPRs (granularity of 8), such that (compute_pgm_rsrc1.vgprs +1)*4 + shared_vgpr_count*8 does not exceed 256. For wavefront size 32 must be 0.

[lamb-j]

Clarifying description

[lamb-j]

Adding revision info to commit message

[lamb-j]

Address clang-format comment

[lamb-j]

Addressing clang-format comments

[kzhuravl]

LGTM, thanks for taking this over!

llvm/docs/AMDGPUUsage.rst

 ..
 
   .. table:: compute_pgm_rsrc3 for GFX10
      :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table
 
      ======= ======= =============================== ===========================================================================
      Bits    Size    Field Name                      Description
      ======= ======= =============================== ===========================================================================
-     3:0     4 bits  SHARED_VGPR_COUNT               Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120.
-                                                     compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64.
+     3:0     4 bits  SHARED_VGPR_COUNT               Number of shared VGPR blocks when executing in subvector mode. For
+                                                     wavefront size 64 the value is 0-15, representing 0-120 VGPRs (granularity
+                                                     of 8), such that (compute_pgm_rsrc1.vgprs +1)*4 + shared_vgpr_count*8 does
+                                                     not exceed 256. For wavefront size 32 shared_vgpr_count must be 0.
      31:4    28                                      Reserved, must be 0.
              bits
      32      **Total size 4 bytes.**
      ======= ===================================================================================================================
 
 ..
 
   .. table:: Floating Point Rounding Mode Enumeration Values
      :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table
.. table:: AMDHSA Kernel Assembler Directives
      ``.amdhsa_workgroup_processor_mode``                     Target              GFX10        Controls ENABLE_WGP_MODE in
                                                               Feature                          :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
                                                               Specific
                                                               (cumode)
      ``.amdhsa_memory_ordered``                               1                   GFX10        Controls MEM_ORDERED in
                                                                                                :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`.
      ``.amdhsa_forward_progress``                             0                   GFX10        Controls FWD_PROGRESS in
                                                                                                :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`.
+     ``.amdhsa_shared_vgpr_count``                            0                   GFX10        Controls SHARED_VGPR_COUNT in
+                                                                                               :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`.
      ``.amdhsa_exception_fp_ieee_invalid_op``                 0                   GFX6-GFX10   Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in
                                                                                                :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`.
      ``.amdhsa_exception_fp_denorm_src``                      0                   GFX6-GFX10   Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in
                                                                                                :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`.
      ``.amdhsa_exception_fp_ieee_div_zero``                   0                   GFX6-GFX10   Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in
                                                                                                :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`.
      ``.amdhsa_exception_fp_ieee_overflow``                   0                   GFX6-GFX10   Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in
                                                                                                :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`.

llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

bool AMDGPUAsmParser::ParseDirectiveAMDHSAKernel() {
 
   StringSet<> Seen;
 
   IsaVersion IVersion = getIsaVersion(getSTI().getCPU());
 
   SMRange VGPRRange;
   uint64_t NextFreeVGPR = 0;
   uint64_t AccumOffset = 0;
+  uint64_t SharedVGPRCount = 0;
   SMRange SGPRRange;
   uint64_t NextFreeSGPR = 0;
 
   // Count the number of user SGPRs implied from the enabled feature bits.
   unsigned ImpliedUserSGPRCount = 0;
 
   // Track if the asm explicitly contains the directive for the user SGPR
   // count.
if (ID == ".amdhsa_group_segment_fixed_size") {
         return Error(IDRange.Start, "directive requires gfx10+", IDRange);
       PARSE_BITS_ENTRY(KD.compute_pgm_rsrc1, COMPUTE_PGM_RSRC1_MEM_ORDERED, Val,
                        ValRange);
     } else if (ID == ".amdhsa_forward_progress") {
       if (IVersion.Major < 10)
         return Error(IDRange.Start, "directive requires gfx10+", IDRange);
       PARSE_BITS_ENTRY(KD.compute_pgm_rsrc1, COMPUTE_PGM_RSRC1_FWD_PROGRESS, Val,
                        ValRange);
+    } else if (ID == ".amdhsa_shared_vgpr_count") {
+      if (IVersion.Major < 10)
+        return Error(IDRange.Start, "directive requires gfx10+", IDRange);
+      SharedVGPRCount = Val;
+      PARSE_BITS_ENTRY(KD.compute_pgm_rsrc3,
+                       COMPUTE_PGM_RSRC3_GFX10_SHARED_VGPR_COUNT, Val,
+                       ValRange);
     } else if (ID == ".amdhsa_exception_fp_ieee_invalid_op") {
       PARSE_BITS_ENTRY(
           KD.compute_pgm_rsrc2,
           COMPUTE_PGM_RSRC2_ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION, Val,
           ValRange);
     } else if (ID == ".amdhsa_exception_fp_denorm_src") {
       PARSE_BITS_ENTRY(KD.compute_pgm_rsrc2,
                        COMPUTE_PGM_RSRC2_ENABLE_EXCEPTION_FP_DENORMAL_SOURCE,
if (AccumOffset < 4 || AccumOffset > 256 || (AccumOffset & 3))
       return TokError("accum_offset should be in range [4..256] in "
                       "increments of 4");
     if (AccumOffset > alignTo(std::max((uint64_t)1, NextFreeVGPR), 4))
       return TokError("accum_offset exceeds total VGPR allocation");
     AMDHSA_BITS_SET(KD.compute_pgm_rsrc3, COMPUTE_PGM_RSRC3_GFX90A_ACCUM_OFFSET,
                     (AccumOffset / 4 - 1));
   }
 
+  if (IVersion.Major == 10) {
+    // SharedVGPRCount < 16 checked by PARSE_ENTRY_BITS
+    if (SharedVGPRCount && EnableWavefrontSize32) {
+      return TokError("shared_vgpr_count directive not valid on "
+                      "wavefront size 32");
+    }
+    if (SharedVGPRCount * 2 + VGPRBlocks > 63) {
+      return TokError("shared_vgpr_count*2 + "
+                      "compute_pgm_rsrc1.GRANULATED_WORKITEM_VGPR_COUNT cannot "
+                      "exceed 63\n");
+    }
+  }
+
   getTargetStreamer().EmitAmdhsaKernelDescriptor(
       getSTI(), KernelName, KD, NextFreeVGPR, NextFreeSGPR, ReserveVCC,
       ReserveFlatScr);
   return false;
 }
 
 bool AMDGPUAsmParser::ParseDirectiveHSACodeObjectVersion() {
   uint32_t Major;

llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUTargetStreamer.cpp

PRINT_FIELD(OS, ".amdhsa_workgroup_processor_mode", KD,
                 compute_pgm_rsrc1,
                 amdhsa::COMPUTE_PGM_RSRC1_WGP_MODE);
     PRINT_FIELD(OS, ".amdhsa_memory_ordered", KD,
                 compute_pgm_rsrc1,
                 amdhsa::COMPUTE_PGM_RSRC1_MEM_ORDERED);
     PRINT_FIELD(OS, ".amdhsa_forward_progress", KD,
                 compute_pgm_rsrc1,
                 amdhsa::COMPUTE_PGM_RSRC1_FWD_PROGRESS);
+    PRINT_FIELD(OS, ".amdhsa_shared_vgpr_count", KD, compute_pgm_rsrc3,
+                amdhsa::COMPUTE_PGM_RSRC3_GFX10_SHARED_VGPR_COUNT);
   }
   PRINT_FIELD(
       OS, ".amdhsa_exception_fp_ieee_invalid_op", KD,
       compute_pgm_rsrc2,
       amdhsa::COMPUTE_PGM_RSRC2_ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION);
   PRINT_FIELD(OS, ".amdhsa_exception_fp_denorm_src", KD,
               compute_pgm_rsrc2,
               amdhsa::COMPUTE_PGM_RSRC2_ENABLE_EXCEPTION_FP_DENORMAL_SOURCE);

llvm/test/MC/AMDGPU/hsa-diag-v3.s

 // NONGFX10: error: directive requires gfx10+
 // GFX10: error: value out of range
 // NONAMDHSA: error: unknown directive
 .warning "test_amdhsa_forward_progress_invalid"
 .amdhsa_kernel test_amdhsa_forward_progress_invalid
   .amdhsa_forward_progress 5
 .end_amdhsa_kernel
 
+// GCN-LABEL: warning: test_amdhsa_shared_vgpr_count_invalid1
+// NONGFX10: error: directive requires gfx10+
+// GFX10: error: .amdhsa_next_free_vgpr directive is required
+// NONAMDHSA: error: unknown directive
+.warning "test_amdhsa_shared_vgpr_count_invalid1"
+.amdhsa_kernel test_amdhsa_shared_vgpr_count_invalid1
+  .amdhsa_shared_vgpr_count 8
+.end_amdhsa_kernel
+
+// GCN-LABEL: warning: test_amdhsa_shared_vgpr_count_invalid2
+// NONGFX10: error: directive requires gfx10+
+// GFX10: error: shared_vgpr_count directive not valid on wavefront size 32
+// NONAMDHSA: error: unknown directive
+.warning "test_amdhsa_shared_vgpr_count_invalid2"
+.amdhsa_kernel test_amdhsa_shared_vgpr_count_invalid2
+  .amdhsa_next_free_vgpr 16
+  .amdhsa_next_free_sgpr 0
+  .amdhsa_shared_vgpr_count 8
+  .amdhsa_wavefront_size32 1
+.end_amdhsa_kernel
+
+// GCN-LABEL: warning: test_amdhsa_shared_vgpr_count_invalid3
+// NONGFX10: error: directive requires gfx10+
+// GFX10: error: value out of range
+// NONAMDHSA: error: unknown directive
+.warning "test_amdhsa_shared_vgpr_count_invalid3"
+.amdhsa_kernel test_amdhsa_shared_vgpr_count_invalid3
+  .amdhsa_next_free_vgpr 32
+  .amdhsa_next_free_sgpr 0
+  .amdhsa_shared_vgpr_count 16
+.end_amdhsa_kernel
+
+// GCN-LABEL: warning: test_amdhsa_shared_vgpr_count_invalid4
+// NONGFX10: error: directive requires gfx10+
+// GFX10: error: shared_vgpr_count*2 + compute_pgm_rsrc1.GRANULATED_WORKITEM_VGPR_COUNT cannot exceed 63
+// NONAMDHSA: error: unknown directive
+.warning "test_amdhsa_shared_vgpr_count_invalid4"
+.amdhsa_kernel test_amdhsa_shared_vgpr_count_invalid4
+  .amdhsa_next_free_vgpr 273
+  .amdhsa_next_free_sgpr 0
+  .amdhsa_shared_vgpr_count 15
+.end_amdhsa_kernel
+
 // GCN-LABEL: warning: test_next_free_vgpr_invalid
 // AMDHSA: error: .amdgcn.next_free_{v,s}gpr symbols must be absolute expressions
 // NONAMDHSA-NOT: error:
 .warning "test_next_free_vgpr_invalid"
 .set .amdgcn.next_free_vgpr, "foo"
 v_mov_b32_e32 v0, s0
 
 // GCN-LABEL: warning: test_end
 .warning "test_end"

llvm/test/MC/AMDGPU/hsa-gfx10-v3.s

 // ASM: .rodata
 
 // Test that only specifying required directives is allowed, and that defaulted
 // values are omitted.
 .p2align 6
 .amdhsa_kernel minimal
   .amdhsa_next_free_vgpr 0
   .amdhsa_next_free_sgpr 0
+  .amdhsa_shared_vgpr_count 0
 .end_amdhsa_kernel
 
 // ASM: .amdhsa_kernel minimal
 // ASM: .amdhsa_next_free_vgpr 0
 // ASM-NEXT: .amdhsa_next_free_sgpr 0
+// ASM: .amdhsa_shared_vgpr_count 0
 // ASM: .end_amdhsa_kernel
 
 // Test that we can specify all available directives with non-default values.
 .p2align 6
 .amdhsa_kernel complete
   .amdhsa_group_segment_fixed_size 1
   .amdhsa_private_segment_fixed_size 1
   .amdhsa_kernarg_size 8

 // ASM-NEXT: .amdhsa_float_denorm_mode_32 1
 // ASM-NEXT: .amdhsa_float_denorm_mode_16_64 0
 // ASM-NEXT: .amdhsa_dx10_clamp 0
 // ASM-NEXT: .amdhsa_ieee_mode 0
 // ASM-NEXT: .amdhsa_fp16_overflow 1
 // ASM-NEXT: .amdhsa_workgroup_processor_mode 1
 // ASM-NEXT: .amdhsa_memory_ordered 1
 // ASM-NEXT: .amdhsa_forward_progress 1
+// ASM-NEXT: .amdhsa_shared_vgpr_count 0
 // ASM-NEXT: .amdhsa_exception_fp_ieee_invalid_op 1
 // ASM-NEXT: .amdhsa_exception_fp_denorm_src 1
 // ASM-NEXT: .amdhsa_exception_fp_ieee_div_zero 1
 // ASM-NEXT: .amdhsa_exception_fp_ieee_overflow 1
 // ASM-NEXT: .amdhsa_exception_fp_ieee_underflow 1
 // ASM-NEXT: .amdhsa_exception_fp_ieee_inexact 1
 // ASM-NEXT: .amdhsa_exception_int_div_zero 1
 // ASM-NEXT: .end_amdhsa_kernel