AMDGPU: Update code object description

[t-tye]

Summary

• Use ELF header flags to identify processor.
• Remove isa note record.
• Add target feature section.
• Make metadata for NumVGPRs, NumSGPRs and MaxFlatWorkGroupSize required.
• Add FixedWorkGroupSize to CodeProps metadata.
• Add ReqdWorkGroupSize* to kernel descriptor and move MaxFlatWorkGroupSize to be adjacent.
• Move IsXNACKEnabled in the kernel descriptor to be at the end of the unused flags.
• Remove IsDynamicCallStack from the metadata and kernel descriptor.
• Remove legacy debugger metadata.

[t-tye]

• Updated the target features section.

[kzhuravl]

LGTM.

[t-tye]

• Remove old XNACK enabled processor names.

[kzhuravl]

LGTM.

[tstellar]

Does this mean the code object format has changed?

[kzhuravl]

In D39828#921228, @tstellar wrote:

Does this mean the code object format has changed?

I am in the process of updating the implementation.

[tstellar]

Will this change be backwards compatible? If not, could you add to the documentation which version of the ROCm runtime is required for the new format.

[kzhuravl]

In D39828#921234, @tstellar wrote:

Will this change be backwards compatible?

No.

If not, could you add to the documentation which version of the ROCm runtime is required for the new format.

Sure, will do.

docs/AMDGPUUsage.rst

 .. _amdgpu-target-triples:
 
 Target Triples
 --------------
 
 Use the ``clang -target <Architecture>-<Vendor>-<OS>-<Environment>`` option to
 specify the target triple:
 
-  .. table:: AMDGPU Target Triples
-     :name: amdgpu-target-triples-table
+  .. table:: AMDGPU Architectures
+     :name: amdgpu-architecture-table
 
-     ============ ======== ========= ===========
-     Architecture Vendor   OS        Environment
-     ============ ======== ========= ===========
-     r600         amd      <empty>   <empty>
-     amdgcn       amd      <empty>   <empty>
-     amdgcn       amd      amdhsa    <empty>
-     amdgcn       amd      amdhsa    opencl
-     amdgcn       amd      amdhsa    amdgizcl
-     amdgcn       amd      amdhsa    amdgiz
-     amdgcn       amd      amdhsa    hcc
-     ============ ======== ========= ===========
-
-``r600-amd--``
-  Supports AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders executed on
-  the MESA runtime.
-
-``amdgcn-amd--``
-  Supports AMD GPUs GCN GFX6 onwards for graphics and compute shaders executed on
-  the MESA runtime.
-
-``amdgcn-amd-amdhsa-``
-  Supports AMD GCN GPUs GFX6 onwards for compute kernels executed on HSA [HSA]_
-  compatible runtimes such as AMD's ROCm [AMD-ROCm]_.
-
-``amdgcn-amd-amdhsa-opencl``
-  Supports AMD GCN GPUs GFX6 onwards for OpenCL compute kernels executed on HSA
-  [HSA]_ compatible runtimes such as AMD's ROCm [AMD-ROCm]_. See
-  :ref:`amdgpu-opencl`.
-
-``amdgcn-amd-amdhsa-amdgizcl``
-  Same as ``amdgcn-amd-amdhsa-opencl`` except a different address space mapping
-  is used (see :ref:`amdgpu-address-spaces`).
+     ============ ==============================================================
+     Architecture Description
+     ============ ==============================================================
+     ``r600``     AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders.
+     ``amdgcn``   AMD GPUs GCN GFX6 onwards for graphics and compute shaders.
+     ============ ==============================================================
 
-``amdgcn-amd-amdhsa-amdgiz``
-  Same as ``amdgcn-amd-amdhsa-`` except a different address space mapping is
-  used (see :ref:`amdgpu-address-spaces`).
+  .. table:: AMDGPU Vendors
+     :name: amdgpu-vendor-table
 
-``amdgcn-amd-amdhsa-hcc``
-  Supports AMD GCN GPUs GFX6 onwards for AMD HC language compute kernels
-  executed on HSA [HSA]_ compatible runtimes such as AMD's ROCm [AMD-ROCm]_. See
-  :ref:`amdgpu-hcc`.
+     ============ ==============================================================
+     Vendor       Description
+     ============ ==============================================================
+     ``amd``      Can be used for all AMD GPU usage.
+     ``mesa3d``   Can be used if the OS is ``mesa3d``.
+     ============ ==============================================================
+
+  .. table:: AMDGPU Operating Systems
+     :name: amdgpu-os-table
+
+     ============== ============================================================
+     OS             Description
+     ============== ============================================================
+     *<empty>*      Defaults to the *unknown* OS.
+     ``amdhsa``     Compute kernels executed on HSA [HSA]_ compatible runtimes
+                    such as AMD's ROCm [AMD-ROCm]_.
+     ``amdpal``     Graphic shaders and compute kernels executed on AMD PAL
+                    runtime.
+     ``mesa3d``     Graphic shaders and compute kernels executed on Mesa 3D
+                    runtime.
+     ============== ============================================================
+
+  .. table:: AMDGPU Environments
+     :name: amdgpu-environment-table
+
+     ============ ==============================================================
+     Environment  Description
+     ============ ==============================================================
+     *<empty>*    Defaults to ``opencl``.
+     ``opencl``   OpenCL compute kernel (see :ref:`amdgpu-opencl`).
+     ``amdgizcl`` Same as ``opencl`` except a different address space mapping is
+                  used (see :ref:`amdgpu-address-spaces`).
+     ``amdgiz``   Same as ``opencl`` except a different address space mapping is
+                  used (see :ref:`amdgpu-address-spaces`).
+     ``hcc``      AMD HC language compute kernel (see :ref:`amdgpu-hcc`).
+     ============ ==============================================================
 
 .. _amdgpu-processors:
 
 Processors
 ----------
 
 Use the ``clang -mcpu <Processor>`` option to specify the AMD GPU processor. The
 names from both the *Processor* and *Alternative Processor* can be used.
 
   .. table:: AMDGPU Processors
-     :name: amdgpu-processors-table
+     :name: amdgpu-processor-table
 
-     ========== =========== ============ ===== ======= ==================
-     Processor  Alternative Target       dGPU/ Runtime Example
-                Processor   Triple       APU   Support Products
+     =========== =============== ============ ===== ======= ==================
+     Processor   Alternative     Target       dGPU/ ROCm    Example
+                 Processor       Triple       APU   Support Products
                                  Architecture
-     ========== =========== ============ ===== ======= ==================
+     =========== =============== ============ ===== ======= ==================
      **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_
-     --------------------------------------------------------------------
-     r600                   r600         dGPU
-     r630                   r600         dGPU
-     rs880                  r600         dGPU
-     rv670                  r600         dGPU
+     -------------------------------------------------------------------------
+     ``r600``                    ``r600``     dGPU
+     ``r630``                    ``r600``     dGPU
+     ``rs880``                   ``r600``     dGPU
+     ``rv670``                   ``r600``     dGPU
      **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_
-     --------------------------------------------------------------------
-     rv710                  r600         dGPU
-     rv730                  r600         dGPU
-     rv770                  r600         dGPU
+     -------------------------------------------------------------------------
+     ``rv710``                   ``r600``     dGPU
+     ``rv730``                   ``r600``     dGPU
+     ``rv770``                   ``r600``     dGPU
      **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_
-     --------------------------------------------------------------------
-     cedar                  r600         dGPU
-     redwood                r600         dGPU
-     sumo                   r600         dGPU
-     juniper                r600         dGPU
-     cypress                r600         dGPU
+     -------------------------------------------------------------------------
+     ``cedar``                   ``r600``     dGPU
+     ``redwood``                 ``r600``     dGPU
+     ``sumo``                    ``r600``     dGPU
+     ``juniper``                 ``r600``     dGPU
+     ``cypress``                 ``r600``     dGPU
      **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_
-     --------------------------------------------------------------------
-     barts                  r600         dGPU
-     turks                  r600         dGPU
-     caicos                 r600         dGPU
-     cayman                 r600         dGPU
+     -------------------------------------------------------------------------
+     ``barts``                   ``r600``     dGPU
+     ``turks``                   ``r600``     dGPU
+     ``caicos``                  ``r600``     dGPU
+     ``cayman``                  ``r600``     dGPU
      **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_
-     --------------------------------------------------------------------
-     gfx600     - tahiti    amdgcn       dGPU
-     gfx601     - pitcairn  amdgcn       dGPU
-                - verde
-                - oland
-                - hainan
+     -------------------------------------------------------------------------
+     ``gfx600``  - ``tahiti``    ``amdgcn``   dGPU
+     ``gfx601``  - ``pitcairn``  ``amdgcn``   dGPU
+                 - ``verde``
+                 - ``oland``
+                 - ``hainan``
      **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_
-     --------------------------------------------------------------------
-     gfx700     - bonaire   amdgcn       dGPU          - Radeon HD 7790
+     -------------------------------------------------------------------------
+     ``gfx700``  - ``bonaire``   ``amdgcn``   dGPU          - Radeon HD 7790
                                                             - Radeon HD 8770
                                                             - R7 260
                                                             - R7 260X
-     \          - kaveri    amdgcn       APU           - A6-7000
+     \           - ``kaveri``    ``amdgcn``   APU           - A6-7000
                                                             - A6 Pro-7050B
                                                             - A8-7100
                                                             - A8 Pro-7150B
                                                             - A10-7300
                                                             - A10 Pro-7350B
                                                             - FX-7500
                                                             - A8-7200P
                                                             - A10-7400P
                                                             - FX-7600P
-     gfx701     - hawaii    amdgcn       dGPU  ROCm    - FirePro W8100
+     ``gfx701``  - ``hawaii``    ``amdgcn``   dGPU  ROCm    - FirePro W8100
                                                             - FirePro W9100
                                                             - FirePro S9150
                                                             - FirePro S9170
-     gfx702                              dGPU  ROCm    - Radeon R9 290
+     ``gfx702``                  ``amdgcn``   dGPU  ROCm    - Radeon R9 290
                                                             - Radeon R9 290x
                                                             - Radeon R390
                                                             - Radeon R390x
-     gfx703     - kabini    amdgcn       APU           - E1-2100
-                - mullins                              - E1-2200
+     ``gfx703``  - ``kabini``    ``amdgcn``   APU           - E1-2100
+                 - ``mullins``                              - E1-2200
                                                             - E1-2500
                                                             - E2-3000
                                                             - E2-3800
                                                             - A4-5000
                                                             - A4-5100
                                                             - A6-5200
                                                             - A4 Pro-3340B
      **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_
-     --------------------------------------------------------------------
-     gfx800     - iceland   amdgcn       dGPU          - FirePro S7150
+     -------------------------------------------------------------------------
+     ``gfx800``  - ``iceland``   ``amdgcn``   dGPU          - FirePro S7150
                                                             - FirePro S7100
                                                             - FirePro W7100
                                                             - Radeon R285
                                                             - Radeon R9 380
                                                             - Radeon R9 385
                                                             - Mobile FirePro
                                                               M7170
-     gfx801     - carrizo   amdgcn       APU           - A6-8500P
+     ``gfx801``  - ``carrizo``   ``amdgcn``   APU           - A6-8500P
                                                             - Pro A6-8500B
                                                             - A8-8600P
                                                             - Pro A8-8600B
                                                             - FX-8800P
                                                             - Pro A12-8800B
-     \                      amdgcn       APU   ROCm    - A10-8700P
+     \                           ``amdgcn``   APU   ROCm    - A10-8700P
                                                             - Pro A10-8700B
                                                             - A10-8780P
-     \                      amdgcn       APU           - A10-9600P
+     \                           ``amdgcn``   APU           - A10-9600P
                                                             - A10-9630P
                                                             - A12-9700P
                                                             - A12-9730P
                                                             - FX-9800P
                                                             - FX-9830P
-     \                      amdgcn       APU           - E2-9010
+     \                           ``amdgcn``   APU           - E2-9010
                                                             - A6-9210
                                                             - A9-9410
-     gfx802     - tonga     amdgcn       dGPU  ROCm    Same as gfx800
-     gfx803     - fiji      amdgcn       dGPU  ROCm    - Radeon R9 Nano
-                                                       - Radeon R9 Fury
+     ``gfx802``  - ``tonga``     ``amdgcn``   dGPU  ROCm    Same as gfx800
+     ``gfx803``  - ``fiji``      ``amdgcn``   dGPU  ROCm    - Radeon R9 Nano
+                 - ``gfx804``                               - Radeon R9 Fury
                                                             - Radeon R9 FuryX
                                                             - Radeon Pro Duo
                                                             - FirePro S9300x2
                                                             - Radeon Instinct MI8
-     \          - polaris10 amdgcn       dGPU  ROCm    - Radeon RX 470
+     \           - ``polaris10`` ``amdgcn``   dGPU  ROCm    - Radeon RX 470
                                                             - Radeon RX 480
                                                             - Radeon Instinct MI6
-     \          - polaris11 amdgcn       dGPU  ROCm    - Radeon RX 460
-     gfx804                 amdgcn       dGPU          Same as gfx803
-     gfx810     - stoney    amdgcn       APU
+     \           - ``polaris11`` ``amdgcn``   dGPU  ROCm    - Radeon RX 460
+     ``gfx810``  - ``stoney``    ``amdgcn``   APU
      **GCN GFX9** [AMD-GCN-GFX9]_
-     --------------------------------------------------------------------
-     gfx900                 amdgcn       dGPU          - Radeon Vega
+     -------------------------------------------------------------------------
+     ``gfx900``  - ``gfx901``    ``amdgcn``   dGPU  ROCm    - Radeon Vega
                                                               Frontier Edition
                                                             - Radeon RX Vega 56
                                                             - Radeon RX Vega 64
                                                             - Radeon RX Vega 64
                                                               Liquid
                                                             - Radeon Instinct MI25
-     gfx901                 amdgcn       dGPU  ROCm    Same as gfx900
-                                                       except XNACK is
-                                                       enabled
-     gfx902                 amdgcn       APU           *TBA*
+     ``gfx902``  - ``gfx903``    ``amdgcn``   APU           *TBA*
 
                                                             .. TODO
                                                                Add product
                                                                names.
-     gfx903                 amdgcn       APU           Same as gfx902
-                                                       except XNACK is
-                                                       enabled
-     ========== =========== ============ ===== ======= ==================
+     =========== =============== ============ ===== ======= ==================
+
+.. _amdgpu-target-features:
+
+Target Features
+---------------
+
+Target features control how code is generated to support certain
+features. Not all target features are supported by all processors. The
+runtime must ensure that the features supported by the device used to
+execute the code match the features enabled when generating the
+code. A mismatch of features may result in incorrect execution, or a
+reduction in performance.
+
+Use the ``clang -m[no-]<TargetFeature>`` option to specify the AMD GPU
+target features.
+
+For example:
+
+``-mxnack``
+  Enable the *XNACK* feature.
+``-mno-xnack``
+  Disable the *XNACK* feature.
+
+  .. table:: AMDGPU Target Features
+     :name: amdgpu-target-feature-table
+
+     ============== ======== ==================================================
+     Target Feature Default  Description
+     ============== ======== ==================================================
+     -m[no-]xnack   disabled Enable/disable generating code that has
+                             memory clauses that are compatible with
+                             having XNACK replay enabled.
+
+                             This is used for demand paging and page
+                             migration. If XNACK replay is enabled in
+                             the device, then if a page fault occurs
+                             the code may execute incorrectly if the
+                             XNACK feature is not enabled. Executing
+                             code that has the feature enabled on a
+                             device that does not have XNACK replay
+                             enabled will execute correctly, but may
+                             be less performant than code with the
+                             feature disabled.
+
+                             This feature is supported by the
+                             ``amdgcn`` architecture for GFX7-GFX9.
+     ============== ======== ==================================================
 
 .. _amdgpu-address-spaces:
 
 Address Spaces
 --------------
 
 The AMDGPU backend uses the following address space mappings.
 

 This section provides LLVM memory synchronization scopes supported by the AMDGPU
 backend memory model when the target triple OS is ``amdhsa`` (see
 :ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`).
 
 The memory model supported is based on the HSA memory model [HSA]_ which is
 based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before
 relation is transitive over the synchonizes-with relation independent of scope,
 and synchonizes-with allows the memory scope instances to be inclusive (see
-table :ref:`amdgpu-amdhsa-llvm-sync-scopes-amdhsa-table`).
+table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`).
 
 This is different to the OpenCL [OpenCL]_ memory model which does not have scope
 inclusion and requires the memory scopes to exactly match. However, this
 is conservatively correct for OpenCL.
 
-  .. table:: AMDHSA LLVM Sync Scopes for AMDHSA
-     :name: amdgpu-amdhsa-llvm-sync-scopes-amdhsa-table
+  .. table:: AMDHSA LLVM Sync Scopes
+     :name: amdgpu-amdhsa-llvm-sync-scopes-table
 
      ================ ==========================================================
      LLVM Sync Scope  Description
      ================ ==========================================================
      *none*           The default: ``system``.
 
                       Synchronizes with, and participates in modification and
                       seq_cst total orderings with, other operations (except
The AMDGPU backend uses the following ELF header:
   .. table:: AMDGPU ELF Header
      :name: amdgpu-elf-header-table
 
      ========================== ===============================
      Field                      Value
      ========================== ===============================
      ``e_ident[EI_CLASS]``      ``ELFCLASS64``
      ``e_ident[EI_DATA]``       ``ELFDATA2LSB``
-     ``e_ident[EI_OSABI]``      ``ELFOSABI_AMDGPU_HSA``,
-                                ``ELFOSABI_AMDGPU_PAL`` or
-                                ``ELFOSABI_AMDGPU_MESA3D``
-     ``e_ident[EI_ABIVERSION]`` ``ELFABIVERSION_AMDGPU_HSA``,
-                                ``ELFABIVERSION_AMDGPU_PAL`` or
-                                ``ELFABIVERSION_AMDGPU_MESA3D``
-     ``e_type``                 ``ET_REL`` or ``ET_DYN``
+     ``e_ident[EI_OSABI]``      - ``ELFOSABI_NONE``
+                                - ``ELFOSABI_AMDGPU_HSA``
+                                - ``ELFOSABI_AMDGPU_PAL``
+                                - ``ELFOSABI_AMDGPU_MESA3D``
+     ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA``
+                                - ``ELFABIVERSION_AMDGPU_PAL``
+                                - ``ELFABIVERSION_AMDGPU_MESA3D``
+     ``e_type``                 - ``ET_REL``
+                                - ``ET_DYN``
      ``e_machine``              ``EM_AMDGPU``
      ``e_entry``                0
-     ``e_flags``                0
+     ``e_flags``                See :ref:`amdgpu-elf-header-e_flags-table`
      ========================== ===============================
 
 ..
 
   .. table:: AMDGPU ELF Header Enumeration Values
      :name: amdgpu-elf-header-enumeration-values-table
 
      =============================== =====
      Name                            Value
      =============================== =====
      ``EM_AMDGPU``                   224
+     ``ELFOSABI_NONE``               0
      ``ELFOSABI_AMDGPU_HSA``         64
      ``ELFOSABI_AMDGPU_PAL``         65
      ``ELFOSABI_AMDGPU_MESA3D``      66
      ``ELFABIVERSION_AMDGPU_HSA``    1
      ``ELFABIVERSION_AMDGPU_PAL``    0
      ``ELFABIVERSION_AMDGPU_MESA3D`` 0
      =============================== =====
 
 ``e_ident[EI_CLASS]``
-  The ELF class is always ``ELFCLASS64``. The AMDGPU backend only supports 64
+  The ELF class is:
+
+  * ``ELFCLASS32`` for ``r600`` architecture.
+
+  * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64
     bit applications.
 
 ``e_ident[EI_DATA]``
-  All AMDGPU targets use ELFDATA2LSB for little-endian byte ordering.
+  All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering.
 
 ``e_ident[EI_OSABI]``
-  One of the following AMD GPU architecture specific OS ABIs:
+  One of the following AMD GPU architecture specific OS ABIs
+  (see :ref:`amdgpu-os-table`):
+
+  * ``ELFOSABI_NONE`` for *unknown* OS.
 
-  * ``ELFOSABI_AMDGPU_HSA`` is used to specify that the code object conforms to
-    the AMD HSA runtime ABI [HSA]_.
+  * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS.
 
-  * ``ELFOSABI_AMDGPU_PAL`` is used to specify that the code object conforms to
-    the AMD PAL runtime ABI.
+  * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS.
 
-  * ``ELFOSABI_AMDGPU_MESA3D`` is used to specify that the code object conforms
-    to the AMD MESA runtime ABI.
+  * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS.
 
 ``e_ident[EI_ABIVERSION]``
   The ABI version of the AMD GPU architecture specific OS ABI to which the code
   object conforms:
 
   * ``ELFABIVERSION_AMDGPU_HSA`` is used to specify the version of AMD HSA
     runtime ABI.
 
   * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL
     runtime ABI.
 
   * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA
-    runtime ABI.
+    3D runtime ABI.
 
 ``e_type``
   Can be one of the following values:
 
 
   ``ET_REL``
     The type produced by the AMD GPU backend compiler as it is relocatable code
     object.
 
   ``ET_DYN``
     The type produced by the linker as it is a shared code object.
 
   The AMD HSA runtime loader requires a ``ET_DYN`` code object.
 
 ``e_machine``
-  The value ``EM_AMDGPU`` is used for the machine for all members of the AMD GPU
-  architecture family. The specific member is specified in the
-  ``NT_AMD_AMDGPU_ISA`` entry in the ``.note`` section (see
-  :ref:`amdgpu-note-records`).
+  The value ``EM_AMDGPU`` is used for the machine for all processors supported
+  by the ``r600`` and ``amdgcn`` architectures (see
+  :ref:`amdgpu-processor-table`). The specific processor is specified in the
+  ``EF_AMDGPU_MACH`` bit field of the ``e_flags`` (see
+  :ref:`amdgpu-elf-header-e_flags-table`).
 
 ``e_entry``
   The entry point is 0 as the entry points for individual kernels must be
   selected in order to invoke them through AQL packets.
 
 ``e_flags``
-  The value is 0 as no flags are used.
+  The AMDGPU backend uses the following ELF header flags:
+
+  .. table:: AMDGPU ELF Header ``e_flags``
+     :name: amdgpu-elf-header-e_flags-table
+
+     ================================= ========== =============================
+     Name                              Value      Description
+     ================================= ========== =============================
+     **AMDGPU Processor Flag**                    See :ref:`amdgpu-processor-table`.
+     -------------------------------------------- -----------------------------
+     ``EF_AMDGPU_MACH``                0x000000ff AMDGPU processor selection
+                                                  mask for
+                                                  ``EF_AMDGPU_MACH_xxx`` values
+                                                  defined in
+                                                  :ref:`amdgpu-ef-amdgpu-mach-table`.
+     ================================= ========== =============================
+
+  .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values
+     :name: amdgpu-ef-amdgpu-mach-table
+
+     ================================= ========== =============================
+     Name                              Value      Description (see
+                                                  :ref:`amdgpu-processor-table`)
+     ================================= ========== =============================
+     ``EF_AMDGPU_MACH_NONE``           0          *not specified*
+     ``EF_AMDGPU_MACH_R600_R600``      1          ``r600``
+     ``EF_AMDGPU_MACH_R600_R630``      2          ``r630``
+     ``EF_AMDGPU_MACH_R600_RS880``     3          ``rs880``
+     ``EF_AMDGPU_MACH_R600_RV670``     4          ``rv670``
+     ``EF_AMDGPU_MACH_R600_RV710``     5          ``rv710``
+     ``EF_AMDGPU_MACH_R600_RV730``     6          ``rv730``
+     ``EF_AMDGPU_MACH_R600_RV770``     7          ``rv770``
+     ``EF_AMDGPU_MACH_R600_CEDAR``     8          ``cedar``
+     ``EF_AMDGPU_MACH_R600_REDWOOD``   9          ``redwood``
+     ``EF_AMDGPU_MACH_R600_SUMO``      10         ``sumo``
+     ``EF_AMDGPU_MACH_R600_JUNIPER``   11         ``juniper``
+     ``EF_AMDGPU_MACH_R600_CYPRESS``   12         ``cypress``
+     ``EF_AMDGPU_MACH_R600_BARTS``     13         ``barts``
+     ``EF_AMDGPU_MACH_R600_TURKS``     14         ``turks``
+     ``EF_AMDGPU_MACH_R600_CAICOS``    15         ``caicos``
+     ``EF_AMDGPU_MACH_R600_CAYMAN``    16         ``cayman``
+     *reserved*                        17-31      Reserved for ``r600``
+                                                  architecture processors.
+     ``EF_AMDGPU_MACH_AMDGCN_GFX600``  32         ``gfx600``
+     ``EF_AMDGPU_MACH_AMDGCN_GFX601``  33         ``gfx601``
+     ``EF_AMDGPU_MACH_AMDGCN_GFX700``  34         ``gfx700``
+     ``EF_AMDGPU_MACH_AMDGCN_GFX701``  35         ``gfx701``
+     ``EF_AMDGPU_MACH_AMDGCN_GFX702``  36         ``gfx702``
+     ``EF_AMDGPU_MACH_AMDGCN_GFX703``  37         ``gfx703``
+     ``EF_AMDGPU_MACH_AMDGCN_GFX800``  38         ``gfx800``
+     ``EF_AMDGPU_MACH_AMDGCN_GFX801``  39         ``gfx801``
+     ``EF_AMDGPU_MACH_AMDGCN_GFX802``  40         ``gfx802``
+     ``EF_AMDGPU_MACH_AMDGCN_GFX803``  41         ``gfx803``
+     ``EF_AMDGPU_MACH_AMDGCN_GFX810``  42         ``gfx810``
+     ``EF_AMDGPU_MACH_AMDGCN_GFX900``  43         ``gfx900``
+     ``EF_AMDGPU_MACH_AMDGCN_GFX902``  44         ``gfx902``
+     ================================= ========== =============================
 
 Sections
 --------
 
 An AMDGPU target ELF code object has the standard ELF sections which include:
 
   .. table:: AMDGPU ELF Sections
      :name: amdgpu-elf-sections-table
``.text``
   as position independent code. See :ref:`amdgpu-code-conventions` for
   information on conventions used in the isa generation.
 
 .. _amdgpu-note-records:
 
 Note Records
 ------------
 
-As required by ``ELFCLASS64``, minimal zero byte padding must be generated after
-the ``name`` field to ensure the ``desc`` field is 4 byte aligned. In addition,
-minimal zero byte padding must be generated to ensure the ``desc`` field size is
-a multiple of 4 bytes. The ``sh_addralign`` field of the ``.note`` section must
-be at least 4 to indicate at least 8 byte alignment.
+As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero byte padding must
+be generated after the ``name`` field to ensure the ``desc`` field is 4 byte
+aligned. In addition, minimal zero byte padding must be generated to ensure the
+``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` field of the
+``.note`` section must be at least 4 to indicate at least 8 byte alignment.
 
 The AMDGPU backend code object uses the following ELF note records in the
 ``.note`` section. The *Description* column specifies the layout of the note
 record's ``desc`` field. All fields are consecutive bytes. Note records with
 variable size strings have a corresponding ``*_size`` field that specifies the
 number of bytes, including the terminating null character, in the string. The
 string(s) come immediately after the preceding fields.
 
 Additional note records can be present.
 
   .. table:: AMDGPU ELF Note Records
      :name: amdgpu-elf-note-records-table
 
      ===== ============================== ======================================
      Name  Type                           Description
      ===== ============================== ======================================
      "AMD" ``NT_AMD_AMDGPU_HSA_METADATA`` <metadata null terminated string>
-     "AMD" ``NT_AMD_AMDGPU_ISA``          <isa name null terminated string>
      ===== ============================== ======================================
 
 ..
 
   .. table:: AMDGPU ELF Note Record Enumeration Values
      :name: amdgpu-elf-note-record-enumeration-values-table
 
      ============================== =====
      Name                           Value
      ============================== =====
      *reserved*                       0-9
      ``NT_AMD_AMDGPU_HSA_METADATA``    10
-     ``NT_AMD_AMDGPU_ISA``             11
+     *reserved*                        11
      ============================== =====
 
-``NT_AMD_AMDGPU_ISA``
-  Specifies the instruction set architecture used by the machine code contained
-  in the code object.
-
-  This note record is required for code objects containing machine code for
-  processors matching the ``amdgcn`` architecture in table
-  :ref:`amdgpu-processors`.
-
-  The null terminated string has the following syntax:
-
-    *architecture*\ ``-``\ *vendor*\ ``-``\ *os*\ ``-``\ *environment*\ ``-``\ *processor*
-
-  where:
-
-    *architecture*
-      The architecture from table :ref:`amdgpu-target-triples-table`.
-
-      This is always ``amdgcn`` when the target triple OS is ``amdhsa`` (see
-      :ref:`amdgpu-target-triples`).
-
-    *vendor*
-      The vendor from table :ref:`amdgpu-target-triples-table`.
-
-      For the AMDGPU backend this is always ``amd``.
-
-    *os*
-      The OS from table :ref:`amdgpu-target-triples-table`.
-
-    *environment*
-      An environment from table :ref:`amdgpu-target-triples-table`, or blank if
-      the environment has no affect on the execution of the code object.
-
-      For the AMDGPU backend this is currently always blank.
-    *processor*
-      The processor from table :ref:`amdgpu-processors-table`.
-
-  For example:
-
-    ``amdgcn-amd-amdhsa--gfx901``
-
 ``NT_AMD_AMDGPU_HSA_METADATA``
   Specifies extensible metadata associated with the code objects executed on HSA
   [HSA]_ compatible runtimes such as AMD's ROCm [AMD-ROCm]_. It is required when
   the target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See
   :ref:`amdgpu-amdhsa-hsa-code-object-metadata` for the syntax of the code
   object metadata string.
 
 .. _amdgpu-symbols:
.. table:: AMDHSA Code Object Kernel Metadata Mapping
      "Args"            sequence of              Sequence of mappings of the
                        mapping                  kernel arguments. See
                                                 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-mapping-table`
                                                 for the definition of the mapping.
      "CodeProps"       mapping                  Mapping of properties related to
                                                 the kernel code. See
                                                 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-mapping-table`
                                                 for the mapping definition.
-     "DebugProps"      mapping                  Mapping of properties related to
-                                                the kernel debugging. See
-                                                :ref:`amdgpu-amdhsa-code-object-kernel-debug-properties-metadata-mapping-table`
-                                                for the mapping definition.
      ================= ============== ========= ================================
 
 ..
 
   .. table:: AMDHSA Code Object Kernel Attribute Metadata Mapping
      :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-mapping-table
 
      =================== ============== ========= ==============================
"GroupSegmentFixedSize"      integer        Required  The amount of group
                                                            group segment memory
                                                            that may be added
                                                            when the kernel is
                                                            dispatched.
      "PrivateSegmentFixedSize"    integer        Required  The amount of fixed
                                                            private address space
                                                            memory required for a
                                                            work-item in
-                                                           bytes. If
-                                                           IsDynamicCallstack
-                                                           is 1 then additional
+                                                           bytes. If the kernel
+                                                           uses a dynamic call
+                                                           stack then additional
                                                            space must be added
                                                            to this value for the
                                                            call stack.
      "KernargSegmentAlign"        integer        Required  The maximum byte
                                                            alignment of
                                                            arguments in the
                                                            kernarg segment. Must
                                                            be a power of 2.
      "WavefrontSize"              integer        Required  Wavefront size. Must
                                                            be a power of 2.
-     "NumSGPRs"                   integer                  Number of scalar
+     "NumSGPRs"                   integer        Required  Number of scalar
                                                            registers used by a
                                                            wavefront for
                                                            GFX6-GFX9. This
                                                            includes the special
                                                            SGPRs for VCC, Flat
                                                            Scratch (GFX7-GFX9)
                                                            and XNACK (for
                                                            GFX8-GFX9). It does
                                                            not include the 16
                                                            SGPR added if a trap
                                                            handler is
                                                            enabled. It is not
                                                            rounded up to the
                                                            allocation
                                                            granularity.
-     "NumVGPRs"                   integer                  Number of vector
+     "NumVGPRs"                   integer        Required  Number of vector
                                                            registers used by
                                                            each work-item for
                                                            GFX6-GFX9
-     "MaxFlatWorkGroupSize"       integer                  Maximum flat
+     "MaxFlatWorkGroupSize"       integer        Required  Maximum flat
                                                            work-group size
                                                            supported by the
                                                            kernel in work-items.
-     "IsDynamicCallStack"         boolean                  Indicates if the
-                                                           generated machine
-                                                           code is using a
-                                                           dynamically sized
-                                                           call stack.
+                                                           Must be >=1 and
+                                                           consistent with any
+                                                           non-0 values in
+                                                           FixedWorkGroupSize.
+     "FixedWorkGroupSize"         sequence of              Corresponds to the
+                                  3 integers               dispatch work-group
+                                                           size X, Y, Z. If
+                                                           omitted, defaults to
+                                                           0, 0, 0. If an
+                                                           element is non-0 then
+                                                           the kernel must only
+                                                           be launched with a
+                                                           matching corresponding
+                                                           work-group size.
      "IsXNACKEnabled"             boolean                  Indicates if the
                                                            generated machine
                                                            code is capable of
-                                                           supporting XNACK.
+                                                           supporting XNACK. See
+                                                           :ref:`amdgpu-target-features`.
      ============================ ============== ========= =====================
 
 ..
 
-  .. table:: AMDHSA Code Object Kernel Debug Properties Metadata Mapping
-     :name: amdgpu-amdhsa-code-object-kernel-debug-properties-metadata-mapping-table
-
-     =================================== ============== ========= ==============
-     String Key                          Value Type     Required? Description
-     =================================== ============== ========= ==============
-     "DebuggerABIVersion"                sequence of
-                                         2 integers
-     "ReservedNumVGPRs"                  integer
-     "ReservedFirstVGPR"                 integer
-     "PrivateSegmentBufferSGPR"          integer
-     "WavefrontPrivateSegmentOffsetSGPR" integer
-     =================================== ============== ========= ==============
-
-.. TODO
-   Plan to remove the debug properties metadata.
-
 Kernel Dispatch
 ~~~~~~~~~~~~~~~
 
 The HSA architected queuing language (AQL) defines a user space memory interface
 that can be used to control the dispatch of kernels, in an agent independent
 way. An agent can have zero or more AQL queues created for it using the ROCm
 runtime, in which AQL packets (all of which are 64 bytes) can be placed. See the
 *HSA Platform System Architecture Specification* [HSA]_ for the AQL queue
.. table:: Kernel Descriptor for GFX6-GFX9
      63:32   4 bytes PrivateSegmentFixedSize         The amount of fixed
                                                      private address space
                                                      memory required for a
                                                      work-item in bytes. If
                                                      is_dynamic_callstack is 1
                                                      then additional space must
                                                      be added to this value for
                                                      the call stack.
-     95:64   4 bytes MaxFlatWorkGroupSize            Maximum flat work-group
-                                                     size supported by the
-                                                     kernel in work-items.
-     96      1 bit   IsDynamicCallStack              Indicates if the generated
-                                                     machine code is using a
-                                                     dynamically sized call
-                                                     stack.
-     97      1 bit   IsXNACKEnabled                  Indicates if the generated
-                                                     machine code is capable of
-                                                     suppoting XNACK.
-     127:98  30 bits                                 Reserved, must be 0.
+     127:64  8 bytes                                 Reserved, must be 0.
      191:128 8 bytes KernelCodeEntryByteOffset       Byte offset (possibly
                                                      negative) from base
                                                      address of kernel
                                                      descriptor to kernel's
                                                      entry point instruction
                                                      which must be 256 byte
                                                      aligned.
-     383:192 24                                      Reserved, must be 0.
+     223:192 4 bytes MaxFlatWorkGroupSize            Maximum flat work-group
+                                                     size supported by the
+                                                     kernel in work-items. If
+                                                     an exact work-group size
+                                                     is required then must be
+                                                     omitted or 0 and
+                                                     ReqdWorkGroupSize* must
+                                                     be set to non-0.
+     239:224 2 bytes ReqdWorkGroupSizeX              If present and non-0 then
+                                                     the kernel
+                                                     must be executed with the
+                                                     specified work-group size
+                                                     for X.
+     255:240 2 bytes ReqdWorkGroupSizeY              If present and non-0 then
+                                                     the kernel
+                                                     must be executed with the
+                                                     specified work-group size
+                                                     for Y.
+     271:256 2 bytes ReqdWorkGroupSizeZ              If present and non-0 then
+                                                     the kernel
+                                                     must be executed with the
+                                                     specified work-group size
+                                                     for Z.
+     383:271 14                                      Reserved, must be 0.
              bytes
      415:384 4 bytes ComputePgmRsrc1                 Compute Shader (CS)
                                                      program settings used by
                                                      CP to set up
                                                      ``COMPUTE_PGM_RSRC1``
                                                      configuration
                                                      register. See
                                                      :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx9-table`.
.. table:: Kernel Descriptor for GFX6-GFX9
      453     1 bit   EnableSGPRFlatScratchInit       *see above*
      454     1 bit   EnableSGPRPrivateSegmentSize    *see above*
      455     1 bit   EnableSGPRGridWorkgroupCountX   Not implemented in CP and
                                                      should always be 0.
      456     1 bit   EnableSGPRGridWorkgroupCountY   Not implemented in CP and
                                                      should always be 0.
      457     1 bit   EnableSGPRGridWorkgroupCountZ   Not implemented in CP and
                                                      should always be 0.
-     463:458 6 bits                                  Reserved, must be 0.
+     462:458 5 bits                                  Reserved, must be 0.
+     463     1 bit   IsXNACKEnabled                  Indicates if the generated
+                                                     machine code is capable of
+                                                     supporting XNACK.
      511:464 6                                       Reserved, must be 0.
              bytes
      512     **Total size 64 bytes.**
      ======= ====================================================================
 
 ..
 
   .. table:: compute_pgm_rsrc1 for GFX6-GFX9
      :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx9-table
 
      ======= ======= =============================== ===========================================================================
      Bits    Size    Field Name                      Description
      ======= ======= =============================== ===========================================================================
      5:0     6 bits  GRANULATED_WORKITEM_VGPR_COUNT  Number of vector registers
                                                      used by each work-item,
                                                      granularity is device
                                                      specific:
 
-                                                     GFX6-9
+                                                     GFX6-GFX9
                                                        - max_vgpr 1..256
                                                        - roundup((max_vgpg + 1)
                                                          / 4) - 1
 
                                                      Used by CP to set up
                                                      ``COMPUTE_PGM_RSRC1.VGPRS``.
      9:6     4 bits  GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar registers
                                                      used by a wavefront,
                                                      granularity is device
                                                      specific:
 
-                                                     GFX6-8
+                                                     GFX6-GFX8
                                                        - max_sgpr 1..112
                                                        - roundup((max_sgpg + 1)
                                                          / 8) - 1
                                                      GFX9
                                                        - max_sgpr 1..112
                                                        - roundup((max_sgpg + 1)
                                                          / 16) - 1
 
.. table:: compute_pgm_rsrc1 for GFX6-GFX9
      25      1 bit   CDBG_USER                       Must be 0.
 
                                                      Flag that can be used to
                                                      control debugging code.
 
                                                      CP is responsible for
                                                      filling in
                                                      ``COMPUTE_PGM_RSRC1.CDBG_USER``.
-     26      1 bit   FP16_OVFL                       GFX6-8
+     26      1 bit   FP16_OVFL                       GFX6-GFX8
                                                        Reserved, must be 0.
                                                      GFX9
                                                        Wavefront starts execution
                                                        with specified fp16 overflow
                                                        mode.
 
                                                        - If 0, fp16 overflow generates
                                                          +/-INF values.
3. Scratch Wave Offset is set by SPI in a per wave basis which is why its value
    cannot included with the flat scratch init value which is per queue.
 4. The VGPRs are set by SPI which only supports specifying either (X), (X, Y)
    or (X, Y, Z).
 
 Flat Scratch register pair are adjacent SGRRs so they can be moved as a 64 bit
 value to the hardware required SGPRn-3 and SGPRn-4 respectively.
 
 The global segment can be accessed either using buffer instructions (GFX6 which
-has V# 64 bit address support), flat instructions (GFX7-9), or global
+has V# 64 bit address support), flat instructions (GFX7-GFX9), or global
 instructions (GFX9).
 
 If buffer operations are used then the compiler can generate a V# with the
 following properties:
 
 * base address of 0
 * no swizzle
 * ATC: 1 if IOMMU present (such as APU)

 If the kernel may use flat operations to access scratch memory, the prolog code
 must set up FLAT_SCRATCH register pair (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which
 are in SGPRn-4/SGPRn-3). Initialization uses Flat Scratch Init and Scratch Wave
 Offset SGPR registers (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`):
 
 GFX6
   Flat scratch is not supported.
 
-GFX7-8
+GFX7-GFX8
   1. The low word of Flat Scratch Init is 32 bit byte offset from
      ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory
      being managed by SPI for the queue executing the kernel dispatch. This is
      the same value used in the Scratch Segment Buffer V# base address. The
      prolog must add the value of Scratch Wave Offset to get the wave's byte
      scratch backing memory offset from ``SH_HIDDEN_PRIVATE_BASE_VIMID``. Since
      FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right shifted
      by 8 before moving into FLAT_SCRATCH_LO.
* The LDS memory has multiple request queues shared by the SIMDs of a
   can be reordered relative to each other, which can result in reordering the
   visibility of vector memory operations with respect to LDS operations of other
   wavefronts in the same work-group. A ``s_waitcnt lgkmcnt(0)`` is required to
   ensure synchronization between LDS operations and vector memory operations
   between waves of a work-group, but not between operations performed by the
   same wavefront.
 * The vector memory operations are performed as wavefront wide operations and
   completion is reported to a wavefront in execution order. The exception is
-  that for GFX7-9 ``flat_load/store/atomic`` instructions can report out of
+  that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of
   vector memory order if they access LDS memory, and out of LDS operation order
   if they access global memory.
 * The vector memory operations access a single vector L1 cache shared by all
   SIMDs a CU. Therefore, no special action is required for coherence between the
   lanes of a single wavefront, or for coherence between wavefronts in the same
   work-group. A ``buffer_wbinvl1_vol`` is required for coherence between waves
   executing in different work-groups as they may be executing on different CUs.
 * The scalar memory operations access a scalar L1 cache shared by all wavefronts
* Each CU has a separate request queue per channel. Therefore, the vector and
   relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure
   synchronization between vector memory operations of different CUs. It ensures a
   previous vector memory operation has completed before executing a subsequent
   vector memory or LDS operation and so can be used to meet the requirements of
   acquire and release.
 * The L2 cache can be kept coherent with other agents on some targets, or ranges
   of virtual addresses can be set up to bypass it to ensure system coherence.
 
-Private address space uses ``buffer_load/store`` using the scratch V# (GFX6-8),
+Private address space uses ``buffer_load/store`` using the scratch V# (GFX6-GFX8),
 or ``scratch_load/store`` (GFX9). Since only a single thread is accessing the
 memory, atomic memory orderings are not meaningful and all accesses are treated
 as non-atomic.
 
 Constant address space uses ``buffer/global_load`` instructions (or equivalent
 scalar memory instructions). Since the constant address space contents do not
 change during the execution of a kernel dispatch it is not legal to perform
 stores, and atomic memory orderings are not meaningful and all access are