This is an archive of the discontinued LLVM Phabricator instance.

[AMDGPU] Corrections to memory model description.
ClosedPublic

Authored by t-tye on Oct 18 2017, 2:50 PM.

Download Raw Diff

Details

Reviewers

Commits

rG6baa6d21e8d0: [AMDGPU] Corrections to memory model description.
rL316131: [AMDGPU] Corrections to memory model description.

Summary

Add description on nontemporal support.
Correct OpenCL sequentially consistent and fence code sequences.
Minor test cleanup.

Diff Detail

Repository: rL LLVM

Event Timeline

t-tye created this revision.Oct 18 2017, 2:50 PM

Herald added subscribers: Anastasia, tpr, dstuttard and 4 others. · View Herald TranscriptOct 18 2017, 2:50 PM

t-tye added a reviewer: kzhuravl.Oct 18 2017, 2:51 PM

LGTM with typos fixed.

docs/AMDGPUUsage.rst
2683 ↗	(On Diff #119526)	vmcnt.
2807 ↗	(On Diff #119526)	vmcnt.
3326 ↗	(On Diff #119526)	vmcnt.

This revision is now accepted and ready to land.Oct 18 2017, 3:00 PM

Address review comments.

t-tye marked 3 inline comments as done.Oct 18 2017, 3:04 PM

kzhuravl accepted this revision.Oct 18 2017, 3:05 PM

Closed by commit rL316131: [AMDGPU] Corrections to memory model description. (authored by t-tye). · Explain WhyOct 18 2017, 3:19 PM

This revision was automatically updated to reflect the committed changes.

Revision Contents

Path

Size

llvm/

trunk/

docs/

AMDGPUUsage.rst

548 lines

Diff 119532

llvm/trunk/docs/AMDGPUUsage.rst

Show First 20 Lines • Show All 1,234 Lines • ▼ Show 20 Lines	"DebuggerABIVersion" sequence of
2 integers		2 integers
"ReservedNumVGPRs" integer		"ReservedNumVGPRs" integer
"ReservedFirstVGPR" integer		"ReservedFirstVGPR" integer
"PrivateSegmentBufferSGPR" integer		"PrivateSegmentBufferSGPR" integer
"WavefrontPrivateSegmentOffsetSGPR" integer		"WavefrontPrivateSegmentOffsetSGPR" integer
=================================== ============== ========= ==============		=================================== ============== ========= ==============

.. TODO		.. TODO
Plan to remove the debug properties metadata.		Plan to remove the debug properties metadata.

Kernel Dispatch		Kernel Dispatch
~~~~~~~~~~~~~~~		~~~~~~~~~~~~~~~

The HSA architected queuing language (AQL) defines a user space memory interface		The HSA architected queuing language (AQL) defines a user space memory interface
that can be used to control the dispatch of kernels, in an agent independent		that can be used to control the dispatch of kernels, in an agent independent
way. An agent can have zero or more AQL queues created for it using the ROCm		way. An agent can have zero or more AQL queues created for it using the ROCm
runtime, in which AQL packets (all of which are 64 bytes) can be placed. See the		runtime, in which AQL packets (all of which are 64 bytes) can be placed. See the
▲ Show 20 Lines • Show All 174 Lines • ▼ Show 20 Lines
Kernel Descriptor for GFX6-GFX9		Kernel Descriptor for GFX6-GFX9
+++++++++++++++++++++++++++++++		+++++++++++++++++++++++++++++++

CP microcode requires the Kernel descritor to be allocated on 64 byte alignment.		CP microcode requires the Kernel descritor to be allocated on 64 byte alignment.

.. table:: Kernel Descriptor for GFX6-GFX9		.. table:: Kernel Descriptor for GFX6-GFX9
:name: amdgpu-amdhsa-kernel-descriptor-gfx6-gfx9-table		:name: amdgpu-amdhsa-kernel-descriptor-gfx6-gfx9-table

======= ======= =============================== ===========================		======= ======= =============================== ============================
Bits Size Field Name Description		Bits Size Field Name Description
======= ======= =============================== ===========================		======= ======= =============================== ============================
31:0 4 bytes GroupSegmentFixedSize The amount of fixed local		31:0 4 bytes GroupSegmentFixedSize The amount of fixed local
address space memory		address space memory
required for a work-group		required for a work-group
in bytes. This does not		in bytes. This does not
include any dynamically		include any dynamically
allocated local address		allocated local address
space memory that may be		space memory that may be
added when the kernel is		added when the kernel is
Show All 11 Lines	95:64 4 bytes MaxFlatWorkGroupSize Maximum flat work-group
kernel in work-items.		kernel in work-items.
96 1 bit IsDynamicCallStack Indicates if the generated		96 1 bit IsDynamicCallStack Indicates if the generated
machine code is using a		machine code is using a
dynamically sized call		dynamically sized call
stack.		stack.
97 1 bit IsXNACKEnabled Indicates if the generated		97 1 bit IsXNACKEnabled Indicates if the generated
machine code is capable of		machine code is capable of
suppoting XNACK.		suppoting XNACK.
127:98 30 bits Reserved. Must be 0.		127:98 30 bits Reserved, must be 0.
191:128 8 bytes KernelCodeEntryByteOffset Byte offset (possibly		191:128 8 bytes KernelCodeEntryByteOffset Byte offset (possibly
negative) from base		negative) from base
address of kernel		address of kernel
descriptor to kernel's		descriptor to kernel's
entry point instruction		entry point instruction
which must be 256 byte		which must be 256 byte
aligned.		aligned.
383:192 24 Reserved. Must be 0.		383:192 24 Reserved, must be 0.
bytes		bytes
415:384 4 bytes ComputePgmRsrc1 Compute Shader (CS)		415:384 4 bytes ComputePgmRsrc1 Compute Shader (CS)
program settings used by		program settings used by
CP to set up		CP to set up
``COMPUTE_PGM_RSRC1``		``COMPUTE_PGM_RSRC1``
configuration		configuration
register. See		register. See
:ref:`amdgpu-amdhsa-compute_pgm_rsrc1_t-gfx6-gfx9-table`.		:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx9-table`.
447:416 4 bytes ComputePgmRsrc2 Compute Shader (CS)		447:416 4 bytes ComputePgmRsrc2 Compute Shader (CS)
program settings used by		program settings used by
CP to set up		CP to set up
``COMPUTE_PGM_RSRC2``		``COMPUTE_PGM_RSRC2``
configuration		configuration
register. See		register. See
:ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx9-table`.		:ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx9-table`.
448 1 bit EnableSGPRPrivateSegmentBuffer Enable the setup of the		448 1 bit EnableSGPRPrivateSegmentBuffer Enable the setup of the
Show All 15 Lines	.. table:: Kernel Descriptor for GFX6-GFX9
453 1 bit EnableSGPRFlatScratchInit see above		453 1 bit EnableSGPRFlatScratchInit see above
454 1 bit EnableSGPRPrivateSegmentSize see above		454 1 bit EnableSGPRPrivateSegmentSize see above
455 1 bit EnableSGPRGridWorkgroupCountX Not implemented in CP and		455 1 bit EnableSGPRGridWorkgroupCountX Not implemented in CP and
should always be 0.		should always be 0.
456 1 bit EnableSGPRGridWorkgroupCountY Not implemented in CP and		456 1 bit EnableSGPRGridWorkgroupCountY Not implemented in CP and
should always be 0.		should always be 0.
457 1 bit EnableSGPRGridWorkgroupCountZ Not implemented in CP and		457 1 bit EnableSGPRGridWorkgroupCountZ Not implemented in CP and
should always be 0.		should always be 0.
463:458 6 bits Reserved. Must be 0.		463:458 6 bits Reserved, must be 0.
511:464 6 Reserved. Must be 0.		511:464 6 Reserved, must be 0.
bytes		bytes
512 Total size 64 bytes.		512 Total size 64 bytes.
======= ===================================================================		======= ====================================================================

..		..

.. table:: compute_pgm_rsrc1 for GFX6-GFX9		.. table:: compute_pgm_rsrc1 for GFX6-GFX9
:name: amdgpu-amdhsa-compute_pgm_rsrc1_t-gfx6-gfx9-table		:name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx9-table

======= ======= =============================== ===========================================================================		======= ======= =============================== ===========================================================================
Bits Size Field Name Description		Bits Size Field Name Description
======= ======= =============================== ===========================================================================		======= ======= =============================== ===========================================================================
5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector registers		5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector registers
used by each work-item,		used by each work-item,
granularity is device		granularity is device
specific:		specific:

GFX6-9		GFX6-9
roundup((max-vgpg + 1)		- max_vgpr 1..256
		- roundup((max_vgpg + 1)
/ 4) - 1		/ 4) - 1

Used by CP to set up		Used by CP to set up
``COMPUTE_PGM_RSRC1.VGPRS``.		``COMPUTE_PGM_RSRC1.VGPRS``.
9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar registers		9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar registers
used by a wavefront,		used by a wavefront,
granularity is device		granularity is device
specific:		specific:

GFX6-8		GFX6-8
roundup((max-sgpg + 1)		- max_sgpr 1..112
		- roundup((max_sgpg + 1)
/ 8) - 1		/ 8) - 1
GFX9		GFX9
roundup((max-sgpg + 1)		- max_sgpr 1..112
		- roundup((max_sgpg + 1)
/ 16) - 1		/ 16) - 1

Includes the special SGPRs		Includes the special SGPRs
for VCC, Flat Scratch (for		for VCC, Flat Scratch (for
GFX7 onwards) and XNACK		GFX7 onwards) and XNACK
(for GFX8 onwards). It does		(for GFX8 onwards). It does
not include the 16 SGPR		not include the 16 SGPR
added if a trap handler is		added if a trap handler is
enabled.		enabled.
▲ Show 20 Lines • Show All 67 Lines • ▼ Show 20 Lines	20 1 bit PRIV Must be 0.
mode.		mode.

CP is responsible for		CP is responsible for
filling in		filling in
``COMPUTE_PGM_RSRC1.PRIV``.		``COMPUTE_PGM_RSRC1.PRIV``.
21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution		21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution
with DX10 clamp mode		with DX10 clamp mode
enabled. Used by the vector		enabled. Used by the vector
ALU to force DX-10 style		ALU to force DX10 style
treatment of NaN's (when		treatment of NaN's (when
set, clamp NaN to zero,		set, clamp NaN to zero,
otherwise pass NaN		otherwise pass NaN
through).		through).

Used by CP to set up		Used by CP to set up
``COMPUTE_PGM_RSRC1.DX10_CLAMP``.		``COMPUTE_PGM_RSRC1.DX10_CLAMP``.
22 1 bit DEBUG_MODE Must be 0.		22 1 bit DEBUG_MODE Must be 0.
Show All 31 Lines	.. table:: compute_pgm_rsrc1 for GFX6-GFX9
25 1 bit CDBG_USER Must be 0.		25 1 bit CDBG_USER Must be 0.

Flag that can be used to		Flag that can be used to
control debugging code.		control debugging code.

CP is responsible for		CP is responsible for
filling in		filling in
``COMPUTE_PGM_RSRC1.CDBG_USER``.		``COMPUTE_PGM_RSRC1.CDBG_USER``.
26 1 bit FP16_OVFL GFX6-8:		26 1 bit FP16_OVFL GFX6-8
Reserved. Must be 0.		Reserved, must be 0.
GFX9:		GFX9
Wavefront starts		Wavefront starts execution
execution with specified		with specified fp16 overflow
fp16 overflow mode.		mode.

- If 0, then fp16		- If 0, fp16 overflow generates
overflow generates
+/-INF values.		+/-INF values.
- If 1, then fp16		- If 1, fp16 overflow that is the
overflow that is the		result of an +/-INF input value
result of an +/-INF		or divide by 0 produces a +/-INF,
input value or divide		otherwise clamps computed
by 0 generates a		overflow to +/-MAX_FP16 as
+/-INF, otherwise		appropriate.
clamps computed
overflow to +/-MAX_FP16
as appropriate.

Used by CP to set up		Used by CP to set up
``COMPUTE_PGM_RSRC1.FP16_OVFL``.		``COMPUTE_PGM_RSRC1.FP16_OVFL``.
31:27 5 bits Reserved. Must be 0.		31:27 5 bits Reserved, must be 0.
32 Total size 4 bytes		32 Total size 4 bytes
======= ===================================================================================================================		======= ===================================================================================================================

..		..

.. table:: compute_pgm_rsrc2 for GFX6-GFX9		.. table:: compute_pgm_rsrc2 for GFX6-GFX9
:name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx9-table		:name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx9-table

▲ Show 20 Lines • Show All 140 Lines • ▼ Show 20 Lines	27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow
_OVERFLOW		_OVERFLOW
28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow		28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow
_UNDERFLOW		_UNDERFLOW
29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact		29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact
_INEXACT		_INEXACT
30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero		30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero
_ZERO (rcp_iflag_f32 instruction		_ZERO (rcp_iflag_f32 instruction
only)		only)
31 1 bit Reserved. Must be 0.		31 1 bit Reserved, must be 0.
32 Total size 4 bytes.		32 Total size 4 bytes.
======= ===================================================================================================================		======= ===================================================================================================================

..		..

.. table:: Floating Point Rounding Mode Enumeration Values		.. table:: Floating Point Rounding Mode Enumeration Values
:name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table		:name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table

▲ Show 20 Lines • Show All 373 Lines • ▼ Show 20 Lines
~~~~~~~~~~~~		~~~~~~~~~~~~

This section describes the mapping of LLVM memory model onto AMDGPU machine code		This section describes the mapping of LLVM memory model onto AMDGPU machine code
(see :ref:`memmodel`). The implementation is WIP.		(see :ref:`memmodel`). The implementation is WIP.

.. TODO		.. TODO
Update when implementation complete.		Update when implementation complete.

Support more relaxed OpenCL memory model to be controlled by environment
component of target triple.

The AMDGPU backend supports the memory synchronization scopes specified in		The AMDGPU backend supports the memory synchronization scopes specified in
:ref:`amdgpu-memory-scopes`.		:ref:`amdgpu-memory-scopes`.

The code sequences used to implement the memory model are defined in table		The code sequences used to implement the memory model are defined in table
:ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`.		:ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`.

The sequences specify the order of instructions that a single thread must		The sequences specify the order of instructions that a single thread must
execute. The ``s_waitcnt`` and ``buffer_wbinvl1_vol`` are defined with respect		execute. The ``s_waitcnt`` and ``buffer_wbinvl1_vol`` are defined with respect
to other memory instructions executed by the same thread. This allows them to be		to other memory instructions executed by the same thread. This allows them to be
moved earlier or later which can allow them to be combined with other instances		moved earlier or later which can allow them to be combined with other instances
of the same instruction, or hoisted/sunk out of loops to improve		of the same instruction, or hoisted/sunk out of loops to improve
performance. Only the instructions related to the memory model are given;		performance. Only the instructions related to the memory model are given;
additional ``s_waitcnt`` instructions are required to ensure registers are		additional ``s_waitcnt`` instructions are required to ensure registers are
defined before being used. These may be able to be combined with the memory		defined before being used. These may be able to be combined with the memory
model ``s_waitcnt`` instructions as described above.		model ``s_waitcnt`` instructions as described above.

The AMDGPU memory model supports both the HSA [HSA]_ memory model, and the		The AMDGPU backend supports the following memory models:
OpenCL [OpenCL]_ memory model. The HSA memory model uses a single happens-before
relation for all address spaces (see :ref:`amdgpu-address-spaces`). The OpenCL		HSA Memory Model [HSA]_
memory model which has separate happens-before relations for the global and		The HSA memory model uses a single happens-before relation for all address
local address spaces, and only a fence specifying both global and local address		spaces (see :ref:`amdgpu-address-spaces`).
space joins the relationships. Since the LLVM ``memfence`` instruction does not		OpenCL Memory Model [OpenCL]_
allow an address space to be specified the OpenCL fence has to convervatively		The OpenCL memory model which has separate happens-before relations for the
assume both local and global address space was specified. However, optimizations		global and local address spaces. Only a fence specifying both global and
can often be done to eliminate the additional ``s_waitcnt``instructions when		local address space, and seq_cst instructions join the relationships. Since
there are no intervening corresponding ``ds/flat_load/store/atomic`` memory		the LLVM ``memfence`` instruction does not allow an address space to be
instructions. The code sequences in the table indicate what can be omitted for		specified the OpenCL fence has to convervatively assume both local and
the OpenCL memory. The target triple environment is used to determine if the		global address space was specified. However, optimizations can often be
		done to eliminate the additional ``s_waitcnt`` instructions when there are
		no intervening memory instructions which access the corresponding address
		space. The code sequences in the table indicate what can be omitted for the
		OpenCL memory. The target triple environment is used to determine if the
source language is OpenCL (see :ref:`amdgpu-opencl`).		source language is OpenCL (see :ref:`amdgpu-opencl`).

``ds/flat_load/store/atomic`` instructions to local memory are termed LDS		``ds/flat_load/store/atomic`` instructions to local memory are termed LDS
operations.		operations.

``buffer/global/flat_load/store/atomic`` instructions to global memory are		``buffer/global/flat_load/store/atomic`` instructions to global memory are
termed vector memory operations.		termed vector memory operations.

For GFX6-GFX9:		For GFX6-GFX9:
Show All 15 Lines	* The LDS memory has multiple request queues shared by the SIMDs of a
ensure synchronization between LDS operations and vector memory operations		ensure synchronization between LDS operations and vector memory operations
between waves of a work-group, but not between operations performed by the		between waves of a work-group, but not between operations performed by the
same wavefront.		same wavefront.
* The vector memory operations are performed as wavefront wide operations and		* The vector memory operations are performed as wavefront wide operations and
completion is reported to a wavefront in execution order. The exception is		completion is reported to a wavefront in execution order. The exception is
that for GFX7-9 ``flat_load/store/atomic`` instructions can report out of		that for GFX7-9 ``flat_load/store/atomic`` instructions can report out of
vector memory order if they access LDS memory, and out of LDS operation order		vector memory order if they access LDS memory, and out of LDS operation order
if they access global memory.		if they access global memory.
* The vector memory operations access a vector L1 cache shared by all wavefronts		* The vector memory operations access a single vector L1 cache shared by all
on a CU. Therefore, no special action is required for coherence between		SIMDs a CU. Therefore, no special action is required for coherence between the
wavefronts in the same work-group. A ``buffer_wbinvl1_vol`` is required for		lanes of a single wavefront, or for coherence between wavefronts in the same
coherence between waves executing in different work-groups as they may be		work-group. A ``buffer_wbinvl1_vol`` is required for coherence between waves
executing on different CUs.		executing in different work-groups as they may be executing on different CUs.
* The scalar memory operations access a scalar L1 cache shared by all wavefronts		* The scalar memory operations access a scalar L1 cache shared by all wavefronts
on a group of CUs. The scalar and vector L1 caches are not coherent. However,		on a group of CUs. The scalar and vector L1 caches are not coherent. However,
scalar operations are used in a restricted way so do not impact the memory		scalar operations are used in a restricted way so do not impact the memory
model. See :ref:`amdgpu-amdhsa-memory-spaces`.		model. See :ref:`amdgpu-amdhsa-memory-spaces`.
* The vector and scalar memory operations use an L2 cache shared by all CUs on		* The vector and scalar memory operations use an L2 cache shared by all CUs on
the same agent.		the same agent.
* The L2 cache has independent channels to service disjoint ranges of virtual		* The L2 cache has independent channels to service disjoint ranges of virtual
addresses.		addresses.
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 Lines
case the AMDGPU backend ensures the memory location used to spill is never		case the AMDGPU backend ensures the memory location used to spill is never
accessed by vector memory operations at the same time. If scalar writes are used		accessed by vector memory operations at the same time. If scalar writes are used
then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function		then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function
return since the locations may be used for vector memory instructions by a		return since the locations may be used for vector memory instructions by a
future wave that uses the same scratch area, or a function call that creates a		future wave that uses the same scratch area, or a function call that creates a
frame at the same address, respectively. There is no need for a ``s_dcache_inv``		frame at the same address, respectively. There is no need for a ``s_dcache_inv``
as all scalar writes are write-before-read in the same thread.		as all scalar writes are write-before-read in the same thread.

Scratch backing memory (which is used for the private address space) is accessed		Scratch backing memory (which is used for the private address space)
with MTYPE NC_NV (non-coherenent non-volatile). Since the private address space		is accessed with MTYPE NC_NV (non-coherenent non-volatile). Since the private
is only accessed by a single thread, and is always write-before-read,		address space is only accessed by a single thread, and is always
there is never a need to invalidate these entries from the L1 cache. Hence all		write-before-read, there is never a need to invalidate these entries from the L1
cache invalidates are done as ``*_vol`` to only invalidate the volatile cache		cache. Hence all cache invalidates are done as ``*_vol`` to only invalidate the
lines.		volatile cache lines.

On dGPU the kernarg backing memory is accessed as UC (uncached) to avoid needing		On dGPU the kernarg backing memory is accessed as UC (uncached) to avoid needing
to invalidate the L2 cache. This also causes it to be treated as non-volatile		to invalidate the L2 cache. This also causes it to be treated as
and so is not invalidated by ``*_vol``. On APU it is accessed as CC (cache		non-volatile and so is not invalidated by ``*_vol``. On APU it is accessed as CC
coherent) and so the L2 cache will coherent with the CPU and other agents.		(cache coherent) and so the L2 cache will coherent with the CPU and other
		agents.

.. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9		.. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9
:name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table		:name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table

============ ============ ============== ========== =======================		============ ============ ============== ========== ===============================
LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code		LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code
Ordering Sync Scope Address		Ordering Sync Scope Address
Space		Space
============ ============ ============== ========== =======================		============ ============ ============== ========== ===============================
Non-Atomic		Non-Atomic
---------------------------------------------------------------------------		-----------------------------------------------------------------------------------
load none none - global non-volatile		load none none - global - !volatile & !nontemporal
- generic 1. buffer/global/flat_load		- generic
volatile		- private 1. buffer/global/flat_load
		- constant
		- volatile & !nontemporal

1. buffer/global/flat_load		1. buffer/global/flat_load
glc=1		glc=1

		- nontemporal

		1. buffer/global/flat_load
		glc=1 slc=1

load none none - local 1. ds_load		load none none - local 1. ds_load
store none none - global 1. buffer/global/flat_store		store none none - global - !nontemporal
- generic		- generic
		- private 1. buffer/global/flat_store
		- constant
		- nontemporal

		1. buffer/global/flat_stote
		glc=1 slc=1

store none none - local 1. ds_store		store none none - local 1. ds_store
Unordered Atomic		Unordered Atomic
---------------------------------------------------------------------------		-----------------------------------------------------------------------------------
load atomic unordered any any Same as non-atomic.		load atomic unordered any any Same as non-atomic.
store atomic unordered any any Same as non-atomic.		store atomic unordered any any Same as non-atomic.
atomicrmw unordered any any *Same as monotonic		atomicrmw unordered any any *Same as monotonic
atomic*.		atomic*.
Monotonic Atomic		Monotonic Atomic
---------------------------------------------------------------------------		-----------------------------------------------------------------------------------
load atomic monotonic - singlethread - global 1. buffer/global/flat_load		load atomic monotonic - singlethread - global 1. buffer/global/flat_load
- wavefront - generic		- wavefront - generic
- workgroup		- workgroup
load atomic monotonic - singlethread - local 1. ds_load		load atomic monotonic - singlethread - local 1. ds_load
- wavefront		- wavefront
- workgroup		- workgroup
load atomic monotonic - agent - global 1. buffer/global/flat_load		load atomic monotonic - agent - global 1. buffer/global/flat_load
- system - generic glc=1		- system - generic glc=1
Show All 9 Lines	atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic
- wavefront - generic		- wavefront - generic
- workgroup		- workgroup
- agent		- agent
- system		- system
atomicrmw monotonic - singlethread - local 1. ds_atomic		atomicrmw monotonic - singlethread - local 1. ds_atomic
- wavefront		- wavefront
- workgroup		- workgroup
Acquire Atomic		Acquire Atomic
---------------------------------------------------------------------------		-----------------------------------------------------------------------------------
load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load		load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load
- wavefront - local		- wavefront - local
- generic		- generic
load atomic acquire - workgroup - global 1. buffer/global_load		load atomic acquire - workgroup - global 1. buffer/global/flat_load
load atomic acquire - workgroup - local 1. ds/flat_load		load atomic acquire - workgroup - local 1. ds_load
- generic 2. s_waitcnt lgkmcnt(0)		2. s_waitcnt lgkmcnt(0)

- If OpenCL, omit		- If OpenCL, omit.
waitcnt.
- Must happen before		- Must happen before
any following		any following
global/generic		global/generic
load/load		load/load
atomic/store/store		atomic/store/store
atomic/atomicrmw.		atomic/atomicrmw.
- Ensures any		- Ensures any
following global		following global
data read is no		data read is no
older than the load		older than the load
atomic value being		atomic value being
acquired.		acquired.
		load atomic acquire - workgroup - generic 1. flat_load
		2. s_waitcnt lgkmcnt(0)

load atomic acquire - agent - global 1. buffer/global_load		- If OpenCL, omit.
		- Must happen before
		any following
		global/generic
		load/load
		atomic/store/store
		atomic/atomicrmw.
		- Ensures any
		following global
		data read is no
		older than the load
		atomic value being
		acquired.
		load atomic acquire - agent - global 1. buffer/global/flat_load
- system glc=1		- system glc=1
2. s_waitcnt vmcnt(0)		2. s_waitcnt vmcnt(0)

- Must happen before		- Must happen before
following		following
buffer_wbinvl1_vol.		buffer_wbinvl1_vol.
- Ensures the load		- Ensures the load
has completed		has completed
Show All 36 Lines	load atomic acquire - agent - generic 1. flat_load glc=1
- Ensures that		- Ensures that
following loads		following loads
will not see stale		will not see stale
global data.		global data.

atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic		atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic
- wavefront - local		- wavefront - local
- generic		- generic
atomicrmw acquire - workgroup - global 1. buffer/global_atomic		atomicrmw acquire - workgroup - global 1. buffer/global/flat_atomic
atomicrmw acquire - workgroup - local 1. ds/flat_atomic		atomicrmw acquire - workgroup - local 1. ds_atomic
- generic 2. waitcnt lgkmcnt(0)		2. waitcnt lgkmcnt(0)

- If OpenCL, omit		- If OpenCL, omit.
waitcnt.
- Must happen before		- Must happen before
any following		any following
global/generic		global/generic
load/load		load/load
atomic/store/store		atomic/store/store
atomic/atomicrmw.		atomic/atomicrmw.
- Ensures any		- Ensures any
following global		following global
data read is no		data read is no
older than the		older than the
atomicrmw value		atomicrmw value
being acquired.		being acquired.

atomicrmw acquire - agent - global 1. buffer/global_atomic		atomicrmw acquire - workgroup - generic 1. flat_atomic
		2. waitcnt lgkmcnt(0)

		- If OpenCL, omit.
		- Must happen before
		any following
		global/generic
		load/load
		atomic/store/store
		atomic/atomicrmw.
		- Ensures any
		following global
		data read is no
		older than the
		atomicrmw value
		being acquired.

		atomicrmw acquire - agent - global 1. buffer/global/flat_atomic
- system 2. s_waitcnt vmcnt(0)		- system 2. s_waitcnt vmcnt(0)

- Must happen before		- Must happen before
following		following
buffer_wbinvl1_vol.		buffer_wbinvl1_vol.
- Ensures the		- Ensures the
atomicrmw has		atomicrmw has
completed before		completed before
Show All 40 Lines	atomicrmw acquire - agent - generic 1. flat_atomic
global data.		global data.

fence acquire - singlethread none none		fence acquire - singlethread none none
- wavefront		- wavefront
fence acquire - workgroup none 1. s_waitcnt lgkmcnt(0)		fence acquire - workgroup none 1. s_waitcnt lgkmcnt(0)

- If OpenCL and		- If OpenCL and
address space is		address space is
not generic, omit		not generic, omit.
waitcnt. However,		- However, since LLVM
since LLVM
currently has no		currently has no
address space on		address space on
the fence need to		the fence need to
conservatively		conservatively
always generate. If		always generate. If
fence had an		fence had an
address space then		address space then
set to address		set to address
Show All 22 Lines	fence acquire - workgroup none 1. s_waitcnt lgkmcnt(0)
atomic/atomicrmw.		atomic/atomicrmw.
- Ensures any		- Ensures any
following global		following global
data read is no		data read is no
older than the		older than the
value read by the		value read by the
fence-paired-atomic.		fence-paired-atomic.

fence acquire - agent none 1. s_waitcnt vmcnt(0) &		fence acquire - agent none 1. s_waitcnt lgkmcnt(0) &
- system lgkmcnt(0)		- system vmcnt(0)

- If OpenCL and		- If OpenCL and
address space is		address space is
not generic, omit		not generic, omit
lgkmcnt(0).		lgkmcnt(0).
However, since LLVM		- However, since LLVM
currently has no		currently has no
address space on		address space on
the fence need to		the fence need to
conservatively		conservatively
always generate		always generate
(see comment for		(see comment for
previous fence).		previous fence).
- Could be split into		- Could be split into
Show All 15 Lines	fence acquire - agent none 1. s_waitcnt lgkmcnt(0) &
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic).		fence-paired-atomic).
- s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after		must happen after
any preceding		any preceding
group/generic load		local/generic load
atomic/atomicrmw		atomic/atomicrmw
with an equal or		with an equal or
wider sync scope		wider sync scope
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic).		fence-paired-atomic).
Show All 10 Lines	fence acquire - agent none 1. s_waitcnt lgkmcnt(0) &
locations read must		locations read must
be no older than		be no older than
the value read by		the value read by
the		the
fence-paired-atomic.		fence-paired-atomic.

2. buffer_wbinvl1_vol		2. buffer_wbinvl1_vol

- Must happen before		- Must happen before any
any following global/generic		following global/generic
load/load		load/load
atomic/store/store		atomic/store/store
atomic/atomicrmw.		atomic/atomicrmw.
- Ensures that		- Ensures that
following loads		following loads
will not see stale		will not see stale
global data.		global data.

Release Atomic		Release Atomic
---------------------------------------------------------------------------		-----------------------------------------------------------------------------------
store atomic release - singlethread - global 1. buffer/global/ds/flat_store		store atomic release - singlethread - global 1. buffer/global/ds/flat_store
- wavefront - local		- wavefront - local
- generic		- generic
store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0)		store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0)
- generic
- If OpenCL, omit		- If OpenCL, omit.
waitcnt.
- Must happen after		- Must happen after
any preceding		any preceding
local/generic		local/generic
load/store/load		load/store/load
atomic/store		atomic/store
atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before		- Must happen before
the following		the following
store.		store.
- Ensures that all		- Ensures that all
memory operations		memory operations
to local have		to local have
completed before		completed before
performing the		performing the
store that is being		store that is being
released.		released.

2. buffer/global/flat_store		2. buffer/global/flat_store
store atomic release - workgroup - local 1. ds_store		store atomic release - workgroup - local 1. ds_store
store atomic release - agent - global 1. s_waitcnt vmcnt(0) &		store atomic release - workgroup - generic 1. s_waitcnt lgkmcnt(0)
- system - generic lgkmcnt(0)
		- If OpenCL, omit.
		- Must happen after
		any preceding
		local/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- Must happen before
		the following
		store.
		- Ensures that all
		memory operations
		to local have
		completed before
		performing the
		store that is being
		released.

		2. flat_store
		store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) &
		- system - generic vmcnt(0)

- If OpenCL, omit		- If OpenCL, omit
lgkmcnt(0).		lgkmcnt(0).
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0) and		vmcnt(0) and
s_waitcnt		s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
Show All 15 Lines	store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) &
load/store/load		load/store/load
atomic/store		atomic/store
atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before		- Must happen before
the following		the following
store.		store.
- Ensures that all		- Ensures that all
memory operations		memory operations
to global have		to memory have
completed before		completed before
performing the		performing the
store that is being		store that is being
released.		released.

2. buffer/global/ds/flat_store		2. buffer/global/ds/flat_store
atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic		atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic
- wavefront - local		- wavefront - local
- generic		- generic
atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0)		atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0)
- generic
- If OpenCL, omit		- If OpenCL, omit.
waitcnt.
- Must happen after		- Must happen after
any preceding		any preceding
local/generic		local/generic
load/store/load		load/store/load
atomic/store		atomic/store
atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before		- Must happen before
the following		the following
atomicrmw.		atomicrmw.
- Ensures that all		- Ensures that all
memory operations		memory operations
to local have		to local have
completed before		completed before
performing the		performing the
atomicrmw that is		atomicrmw that is
being released.		being released.

2. buffer/global/flat_atomic		2. buffer/global/flat_atomic
atomicrmw release - workgroup - local 1. ds_atomic		atomicrmw release - workgroup - local 1. ds_atomic
atomicrmw release - agent - global 1. s_waitcnt vmcnt(0) &		atomicrmw release - workgroup - generic 1. s_waitcnt lgkmcnt(0)
- system - generic lgkmcnt(0)
		- If OpenCL, omit.
		- Must happen after
		any preceding
		local/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- Must happen before
		the following
		atomicrmw.
		- Ensures that all
		memory operations
		to local have
		completed before
		performing the
		atomicrmw that is
		being released.

		2. flat_atomic
		atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) &
		- system - generic vmcnt(0)

- If OpenCL, omit		- If OpenCL, omit
lgkmcnt(0).		lgkmcnt(0).
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0) and		vmcnt(0) and
s_waitcnt		s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
Show All 21 Lines	atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) &
- Ensures that all		- Ensures that all
memory operations		memory operations
to global and local		to global and local
have completed		have completed
before performing		before performing
the atomicrmw that		the atomicrmw that
is being released.		is being released.

2. buffer/global/ds/flat_atomic*		2. buffer/global/ds/flat_atomic
fence release - singlethread none none		fence release - singlethread none none
- wavefront		- wavefront
fence release - workgroup none 1. s_waitcnt lgkmcnt(0)		fence release - workgroup none 1. s_waitcnt lgkmcnt(0)

- If OpenCL and		- If OpenCL and
address space is		address space is
not generic, omit		not generic, omit.
waitcnt. However,		- However, since LLVM
since LLVM
currently has no		currently has no
address space on		address space on
the fence need to		the fence need to
conservatively		conservatively
always generate		always generate. If
(see comment for		fence had an
previous fence).		address space then
		set to address
		space of OpenCL
		fence flag, or to
		generic if both
		local and global
		flags are
		specified.
- Must happen after		- Must happen after
any preceding		any preceding
local/generic		local/generic
load/load		load/load
atomic/store/store		atomic/store/store
atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before		- Must happen before
any following store		any following store
atomic/atomicrmw		atomic/atomicrmw
with an equal or		with an equal or
wider sync scope		wider sync scope
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic).		fence-paired-atomic).
- Ensures that all		- Ensures that all
memory operations		memory operations
to local have		to local have
completed before		completed before
performing the		performing the
following		following
fence-paired-atomic.		fence-paired-atomic.

fence release - agent none 1. s_waitcnt vmcnt(0) &		fence release - agent none 1. s_waitcnt lgkmcnt(0) &
- system lgkmcnt(0)		- system vmcnt(0)

- If OpenCL and		- If OpenCL and
address space is		address space is
not generic, omit		not generic, omit
lgkmcnt(0).		lgkmcnt(0).
However, since LLVM		- If OpenCL and
		address space is
		local, omit
		vmcnt(0).
		- However, since LLVM
currently has no		currently has no
address space on		address space on
the fence need to		the fence need to
conservatively		conservatively
always generate		always generate. If
(see comment for		fence had an
previous fence).		address space then
		set to address
		space of OpenCL
		fence flag, or to
		generic if both
		local and global
		flags are
		specified.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0) and		vmcnt(0) and
s_waitcnt		s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
Show All 19 Lines	fence release - agent none 1. s_waitcnt lgkmcnt(0) &
wider sync scope		wider sync scope
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic).		fence-paired-atomic).
- Ensures that all		- Ensures that all
memory operations		memory operations
to global have		have
completed before		completed before
performing the		performing the
following		following
fence-paired-atomic.		fence-paired-atomic.

Acquire-Release Atomic		Acquire-Release Atomic
---------------------------------------------------------------------------		-----------------------------------------------------------------------------------
atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic		atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic
- wavefront - local		- wavefront - local
- generic		- generic
atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0)		atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0)

- If OpenCL, omit		- If OpenCL, omit.
waitcnt.
- Must happen after		- Must happen after
any preceding		any preceding
local/generic		local/generic
load/store/load		load/store/load
atomic/store		atomic/store
atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before		- Must happen before
the following		the following
atomicrmw.		atomicrmw.
- Ensures that all		- Ensures that all
memory operations		memory operations
to local have		to local have
completed before		completed before
performing the		performing the
atomicrmw that is		atomicrmw that is
being released.		being released.

2. buffer/global_atomic		2. buffer/global/flat_atomic
atomicrmw acq_rel - workgroup - local 1. ds_atomic		atomicrmw acq_rel - workgroup - local 1. ds_atomic
2. s_waitcnt lgkmcnt(0)		2. s_waitcnt lgkmcnt(0)

- If OpenCL, omit		- If OpenCL, omit.
waitcnt.
- Must happen before		- Must happen before
any following		any following
global/generic		global/generic
load/load		load/load
atomic/store/store		atomic/store/store
atomic/atomicrmw.		atomic/atomicrmw.
- Ensures any		- Ensures any
following global		following global
data read is no		data read is no
older than the load		older than the load
atomic value being		atomic value being
acquired.		acquired.

atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0)		atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0)

- If OpenCL, omit		- If OpenCL, omit.
waitcnt.
- Must happen after		- Must happen after
any preceding		any preceding
local/generic		local/generic
load/store/load		load/store/load
atomic/store		atomic/store
atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before		- Must happen before
the following		the following
atomicrmw.		atomicrmw.
- Ensures that all		- Ensures that all
memory operations		memory operations
to local have		to local have
completed before		completed before
performing the		performing the
atomicrmw that is		atomicrmw that is
being released.		being released.

2. flat_atomic		2. flat_atomic
3. s_waitcnt lgkmcnt(0)		3. s_waitcnt lgkmcnt(0)

- If OpenCL, omit		- If OpenCL, omit.
waitcnt.
- Must happen before		- Must happen before
any following		any following
global/generic		global/generic
load/load		load/load
atomic/store/store		atomic/store/store
atomic/atomicrmw.		atomic/atomicrmw.
- Ensures any		- Ensures any
following global		following global
data read is no		data read is no
older than the load		older than the load
atomic value being		atomic value being
acquired.		acquired.
atomicrmw acq_rel - agent - global 1. s_waitcnt vmcnt(0) &
- system lgkmcnt(0)		atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) &
		- system vmcnt(0)

- If OpenCL, omit		- If OpenCL, omit
lgkmcnt(0).		lgkmcnt(0).
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0) and		vmcnt(0) and
s_waitcnt		s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
Show All 21 Lines	atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) &
- Ensures that all		- Ensures that all
memory operations		memory operations
to global have		to global have
completed before		completed before
performing the		performing the
atomicrmw that is		atomicrmw that is
being released.		being released.

2. buffer/global_atomic		2. buffer/global/flat_atomic
3. s_waitcnt vmcnt(0)		3. s_waitcnt vmcnt(0)

- Must happen before		- Must happen before
following		following
buffer_wbinvl1_vol.		buffer_wbinvl1_vol.
- Ensures the		- Ensures the
atomicrmw has		atomicrmw has
completed before		completed before
invalidating the		invalidating the
cache.		cache.

4. buffer_wbinvl1_vol		4. buffer_wbinvl1_vol

- Must happen before		- Must happen before
any following		any following
global/generic		global/generic
load/load		load/load
atomic/atomicrmw.		atomic/atomicrmw.
- Ensures that		- Ensures that
following loads		following loads
will not see stale		will not see stale
global data.		global data.

atomicrmw acq_rel - agent - generic 1. s_waitcnt vmcnt(0) &		atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) &
- system lgkmcnt(0)		- system vmcnt(0)

- If OpenCL, omit		- If OpenCL, omit
lgkmcnt(0).		lgkmcnt(0).
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0) and		vmcnt(0) and
s_waitcnt		s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
▲ Show 20 Lines • Show All 54 Lines • ▼ Show 20 Lines	atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) &
global data.		global data.

fence acq_rel - singlethread none none		fence acq_rel - singlethread none none
- wavefront		- wavefront
fence acq_rel - workgroup none 1. s_waitcnt lgkmcnt(0)		fence acq_rel - workgroup none 1. s_waitcnt lgkmcnt(0)

- If OpenCL and		- If OpenCL and
address space is		address space is
not generic, omit		not generic, omit.
waitcnt. However,		- However,
since LLVM		since LLVM
currently has no		currently has no
address space on		address space on
the fence need to		the fence need to
conservatively		conservatively
always generate		always generate
(see comment for		(see comment for
previous fence).		previous fence).
Show All 21 Lines	fence acq_rel - workgroup none 1. s_waitcnt lgkmcnt(0)
local/generic load		local/generic load
atomic/atomicrmw		atomic/atomicrmw
with an equal or		with an equal or
wider sync scope		wider sync scope
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic)		acquire-fence-paired-atomic
has completed		) has completed
before following		before following
global memory		global memory
operations. This		operations. This
satisfies the		satisfies the
requirements of		requirements of
acquire.		acquire.
- Ensures that all		- Ensures that all
previous memory		previous memory
operations have		operations have
completed before a		completed before a
following		following
local/generic store		local/generic store
atomic/atomicrmw		atomic/atomicrmw
with an equal or		with an equal or
wider sync scope		wider sync scope
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic).		release-fence-paired-atomic
This satisfies the		). This satisfies the
requirements of		requirements of
release.		release.

fence acq_rel - agent none 1. s_waitcnt vmcnt(0) &		fence acq_rel - agent none 1. s_waitcnt lgkmcnt(0) &
- system lgkmcnt(0)		- system vmcnt(0)

- If OpenCL and		- If OpenCL and
address space is		address space is
not generic, omit		not generic, omit
lgkmcnt(0).		lgkmcnt(0).
However, since LLVM		- However, since LLVM
currently has no		currently has no
address space on		address space on
the fence need to		the fence need to
conservatively		conservatively
always generate		always generate
(see comment for		(see comment for
previous fence).		previous fence).
- Could be split into		- Could be split into
Show All 28 Lines	fence acq_rel - agent none 1. s_waitcnt lgkmcnt(0) &
load		load
atomic/atomicrmw		atomic/atomicrmw
with an equal or		with an equal or
wider sync scope		wider sync scope
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic)		acquire-fence-paired-atomic
has completed		) has completed
before invalidating		before invalidating
the cache. This		the cache. This
satisfies the		satisfies the
requirements of		requirements of
acquire.		acquire.
- Ensures that all		- Ensures that all
previous memory		previous memory
operations have		operations have
completed before a		completed before a
following		following
global/local/generic		global/local/generic
store		store
atomic/atomicrmw		atomic/atomicrmw
with an equal or		with an equal or
wider sync scope		wider sync scope
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic).		release-fence-paired-atomic
This satisfies the		). This satisfies the
requirements of		requirements of
release.		release.

2. buffer_wbinvl1_vol		2. buffer_wbinvl1_vol

- Must happen before		- Must happen before
any following		any following
global/generic		global/generic
load/load		load/load
atomic/store/store		atomic/store/store
atomic/atomicrmw.		atomic/atomicrmw.
- Ensures that		- Ensures that
following loads		following loads
will not see stale		will not see stale
global data. This		global data. This
satisfies the		satisfies the
requirements of		requirements of
acquire.		acquire.

Sequential Consistent Atomic		Sequential Consistent Atomic
---------------------------------------------------------------------------		-----------------------------------------------------------------------------------
load atomic seq_cst - singlethread - global *Same as corresponding		load atomic seq_cst - singlethread - global *Same as corresponding
- wavefront - local load atomic acquire*.		- wavefront - local load atomic acquire,
- workgroup - generic		- generic except must generated
load atomic seq_cst - agent - global 1. s_waitcnt vmcnt(0)		all instructions even
- system - local		for OpenCL.*
- generic - Must happen after		load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0)
		- generic
		- Must
		happen after
		preceding
		global/generic load
		atomic/store
		atomic/atomicrmw
		with memory
		ordering of seq_cst
		and with equal or
		wider sync scope.
		(Note that seq_cst
		fences have their
		own s_waitcnt
		lgkmcnt(0) and so do
		not need to be
		considered.)
		- Ensures any
		preceding
		sequential
		consistent local
		memory instructions
		have completed
		before executing
		this sequentially
		consistent
		instruction. This
		prevents reordering
		a seq_cst store
		followed by a
		seq_cst load. (Note
		that seq_cst is
		stronger than
		acquire/release as
		the reordering of
		load acquire
		followed by a store
		release is
		prevented by the
		waitcnt of
		the release, but
		there is nothing
		preventing a store
		release followed by
		load acquire from
		competing out of
		order.)

		2. *Following
		instructions same as
		corresponding load
		atomic acquire,
		except must generated
		all instructions even
		for OpenCL.*
		load atomic seq_cst - workgroup - local *Same as corresponding
		load atomic acquire,
		except must generated
		all instructions even
		for OpenCL.*
		load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) &
		- system - generic vmcnt(0)

		- Could be split into
		separate s_waitcnt
		vmcnt(0)
		and s_waitcnt
		lgkmcnt(0) to allow
		them to be
		independently moved
		according to the
		following rules.
		- waitcnt lgkmcnt(0)
		must happen after
		preceding
		global/generic load
		atomic/store
		atomic/atomicrmw
		with memory
		ordering of seq_cst
		and with equal or
		wider sync scope.
		(Note that seq_cst
		fences have their
		own s_waitcnt
		lgkmcnt(0) and so do
		not need to be
		considered.)
		- waitcnt vmcnt(0)
		must happen after
preceding		preceding
global/generic load		global/generic load
atomic/store		atomic/store
atomic/atomicrmw		atomic/atomicrmw
with memory		with memory
ordering of seq_cst		ordering of seq_cst
and with equal or		and with equal or
wider sync scope.		wider sync scope.
Show All 11 Lines	load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) &
have completed		have completed
before executing		before executing
this sequentially		this sequentially
consistent		consistent
instruction. This		instruction. This
prevents reordering		prevents reordering
a seq_cst store		a seq_cst store
followed by a		followed by a
seq_cst load (Note		seq_cst load. (Note
that seq_cst is		that seq_cst is
stronger than		stronger than
acquire/release as		acquire/release as
the reordering of		the reordering of
load acquire		load acquire
followed by a store		followed by a store
release is		release is
prevented by the		prevented by the
waitcnt vmcnt(0) of		waitcnt of
the release, but		the release, but
there is nothing		there is nothing
preventing a store		preventing a store
release followed by		release followed by
load acquire from		load acquire from
competing out of		competing out of
order.)		order.)

2. *Following		2. *Following
instructions same as		instructions same as
corresponding load		corresponding load
atomic acquire*.		atomic acquire,
		except must generated
		all instructions even
		for OpenCL.*
store atomic seq_cst - singlethread - global *Same as corresponding		store atomic seq_cst - singlethread - global *Same as corresponding
- wavefront - local store atomic release*.		- wavefront - local store atomic release,
- workgroup - generic		- workgroup - generic except must generated
		all instructions even
		for OpenCL.*
store atomic seq_cst - agent - global *Same as corresponding		store atomic seq_cst - agent - global *Same as corresponding
- system - generic store atomic release*.		- system - generic store atomic release,
		except must generated
		all instructions even
		for OpenCL.*
atomicrmw seq_cst - singlethread - global *Same as corresponding		atomicrmw seq_cst - singlethread - global *Same as corresponding
- wavefront - local atomicrmw acq_rel*.		- wavefront - local atomicrmw acq_rel,
- workgroup - generic		- workgroup - generic except must generated
		all instructions even
		for OpenCL.*
atomicrmw seq_cst - agent - global *Same as corresponding		atomicrmw seq_cst - agent - global *Same as corresponding
- system - generic atomicrmw acq_rel*.		- system - generic atomicrmw acq_rel,
		except must generated
		all instructions even
		for OpenCL.*
fence seq_cst - singlethread none *Same as corresponding		fence seq_cst - singlethread none *Same as corresponding
- wavefront fence acq_rel*.		- wavefront fence acq_rel,
- workgroup		- workgroup except must generated
- agent		- agent all instructions even
- system		- system for OpenCL.*
============ ============ ============== ========== =======================		============ ============ ============== ========== ===============================

The memory order also adds the single thread optimization constrains defined in		The memory order also adds the single thread optimization constrains defined in
table		table
:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx9-table`.		:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx9-table`.

.. table:: AMDHSA Memory Model Single Thread Optimization Constraints GFX6-GFX9		.. table:: AMDHSA Memory Model Single Thread Optimization Constraints GFX6-GFX9
:name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx9-table		:name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx9-table

▲ Show 20 Lines • Show All 393 Lines • ▼ Show 20 Lines
- kernel_code_version_major defaults to 1.		- kernel_code_version_major defaults to 1.
- machine_kind defaults to 1.		- machine_kind defaults to 1.
- machine_version_major, machine_version_minor, and		- machine_version_major, machine_version_minor, and
machine_version_stepping are derived from the value of the -mcpu option		machine_version_stepping are derived from the value of the -mcpu option
that is passed to the assembler.		that is passed to the assembler.
- kernel_code_entry_byte_offset defaults to 256.		- kernel_code_entry_byte_offset defaults to 256.
- wavefront_size defaults to 6.		- wavefront_size defaults to 6.
- kernarg_segment_alignment, group_segment_alignment, and		- kernarg_segment_alignment, group_segment_alignment, and
private_segment_alignment default to 4. Note that alignments are specified		private_segment_alignment default to 4. Note that alignments are specified
as a power of two, so a value of n means an alignment of 2^ n.		as a power of two, so a value of n means an alignment of 2^ n.

The .amd_kernel_code_t directive must be placed immediately after the		The .amd_kernel_code_t directive must be placed immediately after the
function label and before any instructions.		function label and before any instructions.

For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document,		For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document,
comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s.		comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s.

▲ Show 20 Lines • Show All 57 Lines • Show Last 20 Lines