This is an archive of the discontinued LLVM Phabricator instance.

Paths

Table of Contentst

-
llvm/docs/
-
docs/
17/20
AMDGPUUsage.rst

Differential D90548

[NFC][AMDGPU] Restructure the AMDGPU memory model description
ClosedPublic

Authored by t-tye on Nov 1 2020, 1:28 AM.

Download Raw Diff

Details

Reviewers

kzhuravl
scott.linder
b-sumner
rampitec

Commits

rG68160789c120: [NFC][AMDGPU] Restructure the AMDGPU memory model description

Summary

Separate the AMDGPU memory model description into separate sections
for each architecture.

Diff Detail

Repository: rG LLVM Github Monorepo

Event Timeline

t-tye created this revision.Nov 1 2020, 1:28 AM

Herald added a project: Restricted Project. · View Herald TranscriptNov 1 2020, 1:28 AM

Herald added subscribers: llvm-commits, kerbowa, jfb and 5 others. · View Herald Transcript

t-tye requested review of this revision.Nov 1 2020, 1:28 AM

Herald added a subscriber: wdng. · View Herald TranscriptNov 1 2020, 1:28 AM

Harbormaster completed remote builds in B77166: Diff 302127.Nov 1 2020, 1:06 AM

Unify further similar rows.
Clarify s_waicnt lgkmcnt(0) at workgroup scope to ensure LDS operation is complete.

Harbormaster completed remote builds in B77199: Diff 302174.Nov 1 2020, 2:13 PM

Some small nits, and a note that we had discussed already about widening the table

llvm/docs/AMDGPUUsage.rst
4154	management
4154	instructions
4192	accesses
4224	Remove extra space
4229	Remove extra space
4307–4313	used
4309	Should this be either "constant" or `const` (not sure how to include literal backticks in Phabricator markup)
4384	Can avoid wrapping this
4837–4903	It can be something we do down the line, but I think making the table wider would make it easier to digest. My screen is wider than it is tall, so I would gladly sacrifice some horizontal real-estate in order to see each row in its entirety without scrolling. This would also make it easier to read some of these longer bullets, and it would compress the number of lines in the file each table takes up. For example, wrapping at 120 columns this table shrinks from ~1200 source lines to ~600.
5560	used
5562	Same question as above
5769	Is the change from "the" to "a" significant?
5900	Same question here
6300	Redundant, delete
6395	lgkmcnt

Correct typos.

scott.linder accepted this revision.Nov 2 2020, 1:30 PM

This revision is now accepted and ready to land.Nov 2 2020, 1:30 PM

This revision was landed with ongoing or failed builds.Nov 2 2020, 1:32 PM

Closed by commit rG68160789c120: [NFC][AMDGPU] Restructure the AMDGPU memory model description (authored by t-tye). · Explain Why

This revision was automatically updated to reflect the committed changes.

t-tye added a commit: rG68160789c120: [NFC][AMDGPU] Restructure the AMDGPU memory model description.

Harbormaster completed remote builds in B77313: Diff 302385.Nov 2 2020, 2:11 PM

t-tye marked 12 inline comments as done.Nov 2 2020, 6:27 PM

t-tye added inline comments.

llvm/docs/AMDGPUUsage.rst
4309	Done in next commit.
4384	Done in next commit.
4837–4903	I agree but would rather do that as a separate commit.
5769	Yes. This case is for generic and so if may be "a" local address space, but the local case is always "the" address space.
5900	Same answer.

Revision Contents

Path

Size

llvm/

docs/

AMDGPUUsage.rst

5286 lines

Diff 302389

llvm/docs/AMDGPUUsage.rst

This file is larger than 256 KB, so syntax highlighting is disabled by default.

Show First 20 Lines • Show All 3,462 Lines • ▼ Show 20 Lines	29 1 bit WGP_MODE GFX6-GFX9
See :ref:`amdgpu-amdhsa-memory-model`.		See :ref:`amdgpu-amdhsa-memory-model`.

Used by CP to set up		Used by CP to set up
``COMPUTE_PGM_RSRC1.WGP_MODE``.		``COMPUTE_PGM_RSRC1.WGP_MODE``.
30 1 bit MEM_ORDERED GFX6-9		30 1 bit MEM_ORDERED GFX6-9
Reserved, must be 0.		Reserved, must be 0.
GFX10		GFX10
Controls the behavior of the		Controls the behavior of the
waitcnt's vmcnt and vscnt		s_waitcnt's vmcnt and vscnt
counters.		counters.

- If 0 vmcnt reports completion		- If 0 vmcnt reports completion
of load and atomic with return		of load and atomic with return
out of order with sample		out of order with sample
instructions, and the vscnt		instructions, and the vscnt
reports the completion of		reports the completion of
store and atomic without		store and atomic without
▲ Show 20 Lines • Show All 655 Lines • ▼ Show 20 Lines
Wavefront Offset is only requested if it is used (see		Wavefront Offset is only requested if it is used (see
:ref:`amdgpu-amdhsa-initial-kernel-execution-state`).		:ref:`amdgpu-amdhsa-initial-kernel-execution-state`).

.. _amdgpu-amdhsa-memory-model:		.. _amdgpu-amdhsa-memory-model:

Memory Model		Memory Model
~~~~~~~~~~~~		~~~~~~~~~~~~

This section describes the mapping of LLVM memory model onto AMDGPU machine code		This section describes the mapping of the LLVM memory model onto AMDGPU machine
(see :ref:`memmodel`).		code (see :ref:`memmodel`).

The AMDGPU backend supports the memory synchronization scopes specified in		The AMDGPU backend supports the memory synchronization scopes specified in
:ref:`amdgpu-memory-scopes`.		:ref:`amdgpu-memory-scopes`.

The code sequences used to implement the memory model are defined in table		The code sequences used to implement the memory model specify the order of
:ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table`.		instructions that a single thread must execute. The ``s_waitcnt`` and cache
		management instructions such as ``buffer_wbinvl1_vol`` are defined with respect
The sequences specify the order of instructions that a single thread must
execute. The ``s_waitcnt`` and ``buffer_wbinvl1_vol`` are defined with respect
to other memory instructions executed by the same thread. This allows them to be		to other memory instructions executed by the same thread. This allows them to be
moved earlier or later which can allow them to be combined with other instances		moved earlier or later which can allow them to be combined with other instances
of the same instruction, or hoisted/sunk out of loops to improve		of the same instruction, or hoisted/sunk out of loops to improve performance.
		scott.linderUnsubmitted Done Reply Inline Actions management scott.linder: management
		scott.linderUnsubmitted Done Reply Inline Actions instructions scott.linder: instructions
performance. Only the instructions related to the memory model are given;		Only the instructions related to the memory model are given; additional
additional ``s_waitcnt`` instructions are required to ensure registers are		``s_waitcnt`` instructions are required to ensure registers are defined before
defined before being used. These may be able to be combined with the memory		being used. These may be able to be combined with the memory model ``s_waitcnt``
model ``s_waitcnt`` instructions as described above.		instructions as described above.

The AMDGPU backend supports the following memory models:		The AMDGPU backend supports the following memory models:

HSA Memory Model [HSA]_		HSA Memory Model [HSA]_
The HSA memory model uses a single happens-before relation for all address		The HSA memory model uses a single happens-before relation for all address
spaces (see :ref:`amdgpu-address-spaces`).		spaces (see :ref:`amdgpu-address-spaces`).
OpenCL Memory Model [OpenCL]_		OpenCL Memory Model [OpenCL]_
The OpenCL memory model which has separate happens-before relations for the		The OpenCL memory model which has separate happens-before relations for the
Show All 9 Lines	OpenCL Memory Model [OpenCL]_
source language is OpenCL (see :ref:`amdgpu-opencl`).		source language is OpenCL (see :ref:`amdgpu-opencl`).

``ds/flat_load/store/atomic`` instructions to local memory are termed LDS		``ds/flat_load/store/atomic`` instructions to local memory are termed LDS
operations.		operations.

``buffer/global/flat_load/store/atomic`` instructions to global memory are		``buffer/global/flat_load/store/atomic`` instructions to global memory are
termed vector memory operations.		termed vector memory operations.

		Private address space uses ``buffer_load/store`` using the scratch V#
		(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread
		is accessing the memory, atomic memory orderings are not meaningful, and all
		accesses are treated as non-atomic.

		Constant address space uses ``buffer/global_load`` instructions (or equivalent
		scalar memory instructions). Since the constant address space contents do not
		change during the execution of a kernel dispatch it is not legal to perform
		stores, and atomic memory orderings are not meaningful, and all accesses are
		scott.linderUnsubmitted Done Reply Inline Actions accesses scott.linder: accesses
		treated as non-atomic.

		A memory synchronization scope wider than work-group is not meaningful for the
		group (LDS) address space and is treated as work-group.

		The memory model does not support the region address space which is treated as
		non-atomic.

		Acquire memory ordering is not meaningful on store atomic instructions and is
		treated as non-atomic.

		Release memory ordering is not meaningful on load atomic instructions and is
		treated a non-atomic.

		Acquire-release memory ordering is not meaningful on load or store atomic
		instructions and is treated as acquire and release respectively.

		The memory order also adds the single thread optimization constraints defined in
		table
		:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`.

		.. table:: AMDHSA Memory Model Single Thread Optimization Constraints
		:name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table

		============ ==============================================================
		LLVM Memory Optimization Constraints
		Ordering
		============ ==============================================================
		unordered none
		monotonic none
		acquire - If a load atomic/atomicrmw then no following load/load
		atomic/store/store atomic/atomicrmw/fence instruction can be
		scott.linderUnsubmitted Done Reply Inline Actions Remove extra space scott.linder: Remove extra space
		moved before the acquire.
		- If a fence then same as load atomic, plus no preceding
		associated fence-paired-atomic can be moved after the fence.
		release - If a store atomic/atomicrmw then no preceding load/load
		atomic/store/store atomic/atomicrmw/fence instruction can be
		scott.linderUnsubmitted Done Reply Inline Actions Remove extra space scott.linder: Remove extra space
		moved after the release.
		- If a fence then same as store atomic, plus no following
		associated fence-paired-atomic can be moved before the
		fence.
		acq_rel Same constraints as both acquire and release.
		seq_cst - If a load atomic then same constraints as acquire, plus no
		preceding sequentially consistent load atomic/store
		atomic/atomicrmw/fence instruction can be moved after the
		seq_cst.
		- If a store atomic then the same constraints as release, plus
		no following sequentially consistent load atomic/store
		atomic/atomicrmw/fence instruction can be moved before the
		seq_cst.
		- If an atomicrmw/fence then same constraints as acq_rel.
		============ ==============================================================

		The code sequences used to implement the memory model are defined in the
		following sections:

		* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9`
		* :ref:`amdgpu-amdhsa-memory-model-gfx10`

		.. _amdgpu-amdhsa-memory-model-gfx6-gfx9:

		Memory Model GFX6-GFX9
		++++++++++++++++++++++

For GFX6-GFX9:		For GFX6-GFX9:

* Each agent has multiple shader arrays (SA).		* Each agent has multiple shader arrays (SA).
* Each SA has multiple compute units (CU).		* Each SA has multiple compute units (CU).
* Each CU has multiple SIMDs that execute wavefronts.		* Each CU has multiple SIMDs that execute wavefronts.
* The wavefronts for a single work-group are executed in the same CU but may be		* The wavefronts for a single work-group are executed in the same CU but may be
executed by different SIMDs.		executed by different SIMDs.
* Each CU has a single LDS memory shared by the wavefronts of the work-groups		* Each CU has a single LDS memory shared by the wavefronts of the work-groups
Show All 34 Lines	* Each CU has a separate request queue per channel. Therefore, the vector and
reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to		reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to
ensure synchronization between vector memory operations of different CUs. It		ensure synchronization between vector memory operations of different CUs. It
ensures a previous vector memory operation has completed before executing a		ensures a previous vector memory operation has completed before executing a
subsequent vector memory or LDS operation and so can be used to meet the		subsequent vector memory or LDS operation and so can be used to meet the
requirements of acquire and release.		requirements of acquire and release.
* The L2 cache can be kept coherent with other agents on some targets, or ranges		* The L2 cache can be kept coherent with other agents on some targets, or ranges
of virtual addresses can be set up to bypass it to ensure system coherence.		of virtual addresses can be set up to bypass it to ensure system coherence.

		Scalar memory operations are only used to access memory that is proven to not
		change during the execution of the kernel dispatch. This includes constant
		address space and global address space for program scope const variables.
		scott.linderUnsubmitted Not Done Reply Inline Actions Should this be either "constant" or `const` (not sure how to include literal backticks in Phabricator markup) scott.linder: Should this be either "constant" or `const` (not sure how to include literal backticks in…
		t-tyeAuthorUnsubmitted Done Reply Inline Actions Done in next commit. t-tye: Done in next commit.
		Therefore, the kernel machine code does not have to maintain the scalar cache to
		ensure it is coherent with the vector caches. The scalar and vector caches are
		invalidated between kernel dispatches by CP since constant address space data
		may change between kernel dispatch executions. See
		scott.linderUnsubmitted Done Reply Inline Actions used scott.linder: used
		:ref:`amdgpu-amdhsa-memory-spaces`.

		The one exception is if scalar writes are used to spill SGPR registers. In this
		case the AMDGPU backend ensures the memory location used to spill is never
		accessed by vector memory operations at the same time. If scalar writes are used
		then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function
		return since the locations may be used for vector memory instructions by a
		future wavefront that uses the same scratch area, or a function call that
		creates a frame at the same address, respectively. There is no need for a
		``s_dcache_inv`` as all scalar writes are write-before-read in the same thread.

		For kernarg backing memory:

		* CP invalidates the L1 cache at the start of each kernel dispatch.
		* On dGPU the kernarg backing memory is allocated in host memory accessed as
		MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also
		causes it to be treated as non-volatile and so is not invalidated by
		``*_vol``.
		* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent)
		and so the L2 cache will be coherent with the CPU and other agents.

		Scratch backing memory (which is used for the private address space) is accessed
		with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is
		only accessed by a single thread, and is always write-before-read, there is
		never a need to invalidate these entries from the L1 cache. Hence all cache
		invalidates are done as ``*_vol`` to only invalidate the volatile cache lines.

		The code sequences used to implement the memory model for GFX6-GFX9 are defined
		in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`.

		.. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9
		:name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table

		============ ============ ============== ========== ================================
		LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code
		Ordering Sync Scope Address GFX6-9
		Space
		============ ============ ============== ========== ================================
		Non-Atomic
		------------------------------------------------------------------------------------
		load none none - global - !volatile & !nontemporal
		- generic
		- private 1. buffer/global/flat_load
		- constant
		- volatile & !nontemporal

		1. buffer/global/flat_load
		glc=1

		- nontemporal

		1. buffer/global/flat_load
		glc=1 slc=1

		load none none - local 1. ds_load
		store none none - global - !nontemporal
		- generic
		- private 1. buffer/global/flat_store
		- constant
		- nontemporal

		1. buffer/global/flat_store
		glc=1 slc=1

		store none none - local 1. ds_store
		Unordered Atomic
		------------------------------------------------------------------------------------
		load atomic unordered any any Same as non-atomic.
		store atomic unordered any any Same as non-atomic.
		atomicrmw unordered any any *Same as monotonic
		atomic*.
		scott.linderUnsubmitted Not Done Reply Inline Actions Can avoid wrapping this scott.linder: Can avoid wrapping this
		t-tyeAuthorUnsubmitted Done Reply Inline Actions Done in next commit. t-tye: Done in next commit.
		Monotonic Atomic
		------------------------------------------------------------------------------------
		load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load
		- wavefront - local
		- workgroup - generic
		load atomic monotonic - agent - global 1. buffer/global/flat_load
		- system - generic glc=1
		store atomic monotonic - singlethread - global 1. buffer/global/flat_store
		- wavefront - generic
		- workgroup
		- agent
		- system
		store atomic monotonic - singlethread - local 1. ds_store
		- wavefront
		- workgroup
		atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic
		- wavefront - generic
		- workgroup
		- agent
		- system
		atomicrmw monotonic - singlethread - local 1. ds_atomic
		- wavefront
		- workgroup
		Acquire Atomic
		------------------------------------------------------------------------------------
		load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load
		- wavefront - local
		- generic
		load atomic acquire - workgroup - global 1. buffer/global_load
		load atomic acquire - workgroup - local 1. ds/flat_load
		- generic 2. s_waitcnt lgkmcnt(0)

		- If OpenCL, omit.
		- Must happen before
		any following
		global/generic
		load/load
		atomic/store/store
		atomic/atomicrmw.
		- Ensures any
		following global
		data read is no
		older than a local load
		atomic value being
		acquired.

		load atomic acquire - agent - global 1. buffer/global_load
		- system glc=1
		2. s_waitcnt vmcnt(0)

		- Must happen before
		following
		buffer_wbinvl1_vol.
		- Ensures the load
		has completed
		before invalidating
		the cache.

		3. buffer_wbinvl1_vol

		- Must happen before
		any following
		global/generic
		load/load
		atomic/atomicrmw.
		- Ensures that
		following
		loads will not see
		stale global data.

		load atomic acquire - agent - generic 1. flat_load glc=1
		- system 2. s_waitcnt vmcnt(0) &
		lgkmcnt(0)

		- If OpenCL omit
		lgkmcnt(0).
		- Must happen before
		following
		buffer_wbinvl1_vol.
		- Ensures the flat_load
		has completed
		before invalidating
		the cache.

		3. buffer_wbinvl1_vol

		- Must happen before
		any following
		global/generic
		load/load
		atomic/atomicrmw.
		- Ensures that
		following loads
		will not see stale
		global data.

		atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic
		- wavefront - local
		- generic
		atomicrmw acquire - workgroup - global 1. buffer/global_atomic
		atomicrmw acquire - workgroup - local 1. ds/flat_atomic
		- generic 2. s_waitcnt lgkmcnt(0)

		- If OpenCL, omit.
		- Must happen before
		any following
		global/generic
		load/load
		atomic/store/store
		atomic/atomicrmw.
		- Ensures any
		following global
		data read is no
		older than a local
		atomicrmw value
		being acquired.

		atomicrmw acquire - agent - global 1. buffer/global_atomic
		- system 2. s_waitcnt vmcnt(0)

		- Must happen before
		following
		buffer_wbinvl1_vol.
		- Ensures the
		atomicrmw has
		completed before
		invalidating the
		cache.

		3. buffer_wbinvl1_vol

		- Must happen before
		any following
		global/generic
		load/load
		atomic/atomicrmw.
		- Ensures that
		following loads
		will not see stale
		global data.

		atomicrmw acquire - agent - generic 1. flat_atomic
		- system 2. s_waitcnt vmcnt(0) &
		lgkmcnt(0)

		- If OpenCL, omit
		lgkmcnt(0).
		- Must happen before
		following
		buffer_wbinvl1_vol.
		- Ensures the
		atomicrmw has
		completed before
		invalidating the
		cache.

		3. buffer_wbinvl1_vol

		- Must happen before
		any following
		global/generic
		load/load
		atomic/atomicrmw.
		- Ensures that
		following loads
		will not see stale
		global data.

		fence acquire - singlethread none none
		- wavefront
		fence acquire - workgroup none 1. s_waitcnt lgkmcnt(0)

		- If OpenCL and
		address space is
		not generic, omit.
		- However, since LLVM
		currently has no
		address space on
		the fence need to
		conservatively
		always generate. If
		fence had an
		address space then
		set to address
		space of OpenCL
		fence flag, or to
		generic if both
		local and global
		flags are
		specified.
		- Must happen after
		any preceding
		local/generic load
		atomic/atomicrmw
		with an equal or
		wider sync scope
		and memory ordering
		stronger than
		unordered (this is
		termed the
		fence-paired-atomic).
		- Must happen before
		any following
		global/generic
		load/load
		atomic/store/store
		atomic/atomicrmw.
		- Ensures any
		following global
		data read is no
		older than the
		value read by the
		fence-paired-atomic.

		fence acquire - agent none 1. s_waitcnt lgkmcnt(0) &
		- system vmcnt(0)

		- If OpenCL and
		address space is
		not generic, omit
		lgkmcnt(0).
		- However, since LLVM
		currently has no
		address space on
		the fence need to
		conservatively
		always generate
		(see comment for
		previous fence).
		- Could be split into
		separate s_waitcnt
		vmcnt(0) and
		s_waitcnt
		lgkmcnt(0) to allow
		them to be
		independently moved
		according to the
		following rules.
		- s_waitcnt vmcnt(0)
		must happen after
		any preceding
		global/generic load
		atomic/atomicrmw
		with an equal or
		wider sync scope
		and memory ordering
		stronger than
		unordered (this is
		termed the
		fence-paired-atomic).
		- s_waitcnt lgkmcnt(0)
		must happen after
		any preceding
		local/generic load
		atomic/atomicrmw
		with an equal or
		wider sync scope
		and memory ordering
		stronger than
		unordered (this is
		termed the
		fence-paired-atomic).
		- Must happen before
		the following
		buffer_wbinvl1_vol.
		- Ensures that the
		fence-paired atomic
		has completed
		before invalidating
		the
		cache. Therefore
		any following
		locations read must
		be no older than
		the value read by
		the
		fence-paired-atomic.

		2. buffer_wbinvl1_vol

		- Must happen before any
		following global/generic
		load/load
		atomic/store/store
		atomic/atomicrmw.
		- Ensures that
		following loads
		will not see stale
		global data.

		Release Atomic
		------------------------------------------------------------------------------------
		store atomic release - singlethread - global 1. buffer/global/ds/flat_store
		- wavefront - local
		- generic
		store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0)
		- generic
		- If OpenCL, omit.
		- Must happen after
		any preceding
		local/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- Must happen before
		the following
		store.
		- Ensures that all
		memory operations
		to local have
		completed before
		performing the
		store that is being
		released.

		2. buffer/global/flat_store
		store atomic release - workgroup - local 1. ds_store
		store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) &
		- system - generic vmcnt(0)

		- If OpenCL and
		address space is
		not generic, omit
		lgkmcnt(0).
		- Could be split into
		separate s_waitcnt
		vmcnt(0) and
		s_waitcnt
		lgkmcnt(0) to allow
		them to be
		independently moved
		according to the
		following rules.
		- s_waitcnt vmcnt(0)
		must happen after
		any preceding
		global/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- s_waitcnt lgkmcnt(0)
		must happen after
		any preceding
		local/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- Must happen before
		the following
		store.
		- Ensures that all
		memory operations
		to memory have
		completed before
		performing the
		store that is being
		released.

		2. buffer/global/flat_store
		atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic
		- wavefront - local
		- generic
		atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0)
		- generic
		- If OpenCL, omit.
		- Must happen after
		any preceding
		local/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- Must happen before
		the following
		atomicrmw.
		- Ensures that all
		memory operations
		to local have
		completed before
		performing the
		atomicrmw that is
		being released.

		2. buffer/global/flat_atomic
		atomicrmw release - workgroup - local 1. ds_atomic
		atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) &
		- system - generic vmcnt(0)

		- If OpenCL, omit
		lgkmcnt(0).
		- Could be split into
		separate s_waitcnt
		vmcnt(0) and
		s_waitcnt
		lgkmcnt(0) to allow
		them to be
		independently moved
		according to the
		following rules.
		- s_waitcnt vmcnt(0)
		must happen after
		any preceding
		global/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- s_waitcnt lgkmcnt(0)
		must happen after
		any preceding
		local/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- Must happen before
		the following
		atomicrmw.
		- Ensures that all
		memory operations
		to global and local
		have completed
		before performing
		the atomicrmw that
		is being released.

		2. buffer/global/flat_atomic
		fence release - singlethread none none
		- wavefront
		fence release - workgroup none 1. s_waitcnt lgkmcnt(0)

		- If OpenCL and
		address space is
		not generic, omit.
		- However, since LLVM
		currently has no
		address space on
		the fence need to
		conservatively
		always generate. If
		fence had an
		address space then
		set to address
		space of OpenCL
		fence flag, or to
		generic if both
		local and global
		flags are
		specified.
		- Must happen after
		any preceding
		local/generic
		load/load
		atomic/store/store
		atomic/atomicrmw.
		- Must happen before
		any following store
		atomic/atomicrmw
		with an equal or
		wider sync scope
		and memory ordering
		stronger than
		unordered (this is
		termed the
		fence-paired-atomic).
		- Ensures that all
		memory operations
		to local have
		completed before
		performing the
		following
		fence-paired-atomic.

		fence release - agent none 1. s_waitcnt lgkmcnt(0) &
		- system vmcnt(0)

		- If OpenCL and
		address space is
		not generic, omit
		lgkmcnt(0).
		- If OpenCL and
		address space is
		local, omit
		vmcnt(0).
		- However, since LLVM
		currently has no
		address space on
		the fence need to
		conservatively
		always generate. If
		fence had an
		address space then
		set to address
		space of OpenCL
		fence flag, or to
		generic if both
		local and global
		flags are
		specified.
		- Could be split into
		separate s_waitcnt
		vmcnt(0) and
		s_waitcnt
		lgkmcnt(0) to allow
		them to be
		independently moved
		according to the
		following rules.
		- s_waitcnt vmcnt(0)
		must happen after
		any preceding
		global/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- s_waitcnt lgkmcnt(0)
		must happen after
		any preceding
		local/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		scott.linderUnsubmitted Not Done Reply Inline Actions It can be something we do down the line, but I think making the table wider would make it easier to digest. My screen is wider than it is tall, so I would gladly sacrifice some horizontal real-estate in order to see each row in its entirety without scrolling. This would also make it easier to read some of these longer bullets, and it would compress the number of lines in the file each table takes up. For example, wrapping at 120 columns this table shrinks from ~1200 source lines to ~600. scott.linder: It can be something we do down the line, but I think making the table wider would make it…
		t-tyeAuthorUnsubmitted Done Reply Inline Actions I agree but would rather do that as a separate commit. t-tye: I agree but would rather do that as a separate commit.
		- Must happen before
		any following store
		atomic/atomicrmw
		with an equal or
		wider sync scope
		and memory ordering
		stronger than
		unordered (this is
		termed the
		fence-paired-atomic).
		- Ensures that all
		memory operations
		have
		completed before
		performing the
		following
		fence-paired-atomic.

		Acquire-Release Atomic
		------------------------------------------------------------------------------------
		atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic
		- wavefront - local
		- generic
		atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0)

		- If OpenCL, omit.
		- Must happen after
		any preceding
		local/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- Must happen before
		the following
		atomicrmw.
		- Ensures that all
		memory operations
		to local have
		completed before
		performing the
		atomicrmw that is
		being released.

		2. buffer/global_atomic

		atomicrmw acq_rel - workgroup - local 1. ds_atomic
		2. s_waitcnt lgkmcnt(0)

		- If OpenCL, omit.
		- Must happen before
		any following
		global/generic
		load/load
		atomic/store/store
		atomic/atomicrmw.
		- Ensures any
		following global
		data read is no
		older than the local load
		atomic value being
		acquired.

		atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0)

		- If OpenCL, omit.
		- Must happen after
		any preceding
		local/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- Must happen before
		the following
		atomicrmw.
		- Ensures that all
		memory operations
		to local have
		completed before
		performing the
		atomicrmw that is
		being released.

		2. flat_atomic
		3. s_waitcnt lgkmcnt(0)

		- If OpenCL, omit.
		- Must happen before
		any following
		global/generic
		load/load
		atomic/store/store
		atomic/atomicrmw.
		- Ensures any
		following global
		data read is no
		older than a local load
		atomic value being
		acquired.

		atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) &
		- system vmcnt(0)

		- If OpenCL, omit
		lgkmcnt(0).
		- Could be split into
		separate s_waitcnt
		vmcnt(0) and
		s_waitcnt
		lgkmcnt(0) to allow
		them to be
		independently moved
		according to the
		following rules.
		- s_waitcnt vmcnt(0)
		must happen after
		any preceding
		global/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- s_waitcnt lgkmcnt(0)
		must happen after
		any preceding
		local/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- Must happen before
		the following
		atomicrmw.
		- Ensures that all
		memory operations
		to global have
		completed before
		performing the
		atomicrmw that is
		being released.

		2. buffer/global_atomic
		3. s_waitcnt vmcnt(0)

		- Must happen before
		following
		buffer_wbinvl1_vol.
		- Ensures the
		atomicrmw has
		completed before
		invalidating the
		cache.

		4. buffer_wbinvl1_vol

		- Must happen before
		any following
		global/generic
		load/load
		atomic/atomicrmw.
		- Ensures that
		following loads
		will not see stale
		global data.

		atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) &
		- system vmcnt(0)

		- If OpenCL, omit
		lgkmcnt(0).
		- Could be split into
		separate s_waitcnt
		vmcnt(0) and
		s_waitcnt
		lgkmcnt(0) to allow
		them to be
		independently moved
		according to the
		following rules.
		- s_waitcnt vmcnt(0)
		must happen after
		any preceding
		global/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- s_waitcnt lgkmcnt(0)
		must happen after
		any preceding
		local/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- Must happen before
		the following
		atomicrmw.
		- Ensures that all
		memory operations
		to global have
		completed before
		performing the
		atomicrmw that is
		being released.

		2. flat_atomic
		3. s_waitcnt vmcnt(0) &
		lgkmcnt(0)

		- If OpenCL, omit
		lgkmcnt(0).
		- Must happen before
		following
		buffer_wbinvl1_vol.
		- Ensures the
		atomicrmw has
		completed before
		invalidating the
		cache.

		4. buffer_wbinvl1_vol

		- Must happen before
		any following
		global/generic
		load/load
		atomic/atomicrmw.
		- Ensures that
		following loads
		will not see stale
		global data.

		fence acq_rel - singlethread none none
		- wavefront
		fence acq_rel - workgroup none 1. s_waitcnt lgkmcnt(0)

		- If OpenCL and
		address space is
		not generic, omit.
		- However,
		since LLVM
		currently has no
		address space on
		the fence need to
		conservatively
		always generate
		(see comment for
		previous fence).
		- Must happen after
		any preceding
		local/generic
		load/load
		atomic/store/store
		atomic/atomicrmw.
		- Must happen before
		any following
		global/generic
		load/load
		atomic/store/store
		atomic/atomicrmw.
		- Ensures that all
		memory operations
		to local have
		completed before
		performing any
		following global
		memory operations.
		- Ensures that the
		preceding
		local/generic load
		atomic/atomicrmw
		with an equal or
		wider sync scope
		and memory ordering
		stronger than
		unordered (this is
		termed the
		acquire-fence-paired-atomic)
		has completed
		before following
		global memory
		operations. This
		satisfies the
		requirements of
		acquire.
		- Ensures that all
		previous memory
		operations have
		completed before a
		following
		local/generic store
		atomic/atomicrmw
		with an equal or
		wider sync scope
		and memory ordering
		stronger than
		unordered (this is
		termed the
		release-fence-paired-atomic).
		This satisfies the
		requirements of
		release.

		fence acq_rel - agent none 1. s_waitcnt lgkmcnt(0) &
		- system vmcnt(0)

		- If OpenCL and
		address space is
		not generic, omit
		lgkmcnt(0).
		- However, since LLVM
		currently has no
		address space on
		the fence need to
		conservatively
		always generate
		(see comment for
		previous fence).
		- Could be split into
		separate s_waitcnt
		vmcnt(0) and
		s_waitcnt
		lgkmcnt(0) to allow
		them to be
		independently moved
		according to the
		following rules.
		- s_waitcnt vmcnt(0)
		must happen after
		any preceding
		global/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- s_waitcnt lgkmcnt(0)
		must happen after
		any preceding
		local/generic
		load/store/load
		atomic/store
		atomic/atomicrmw.
		- Must happen before
		the following
		buffer_wbinvl1_vol.
		- Ensures that the
		preceding
		global/local/generic
		load
		atomic/atomicrmw
		with an equal or
		wider sync scope
		and memory ordering
		stronger than
		unordered (this is
		termed the
		acquire-fence-paired-atomic)
		has completed
		before invalidating
		the cache. This
		satisfies the
		requirements of
		acquire.
		- Ensures that all
		previous memory
		operations have
		completed before a
		following
		global/local/generic
		store
		atomic/atomicrmw
		with an equal or
		wider sync scope
		and memory ordering
		stronger than
		unordered (this is
		termed the
		release-fence-paired-atomic).
		This satisfies the
		requirements of
		release.

		2. buffer_wbinvl1_vol

		- Must happen before
		any following
		global/generic
		load/load
		atomic/store/store
		atomic/atomicrmw.
		- Ensures that
		following loads
		will not see stale
		global data. This
		satisfies the
		requirements of
		acquire.

		Sequential Consistent Atomic
		------------------------------------------------------------------------------------
		load atomic seq_cst - singlethread - global *Same as corresponding
		- wavefront - local load atomic acquire,
		- generic except must generated
		all instructions even
		for OpenCL.*
		load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0)
		- generic

		- Must
		happen after
		preceding
		local/generic load
		atomic/store
		atomic/atomicrmw
		with memory
		ordering of seq_cst
		and with equal or
		wider sync scope.
		(Note that seq_cst
		fences have their
		own s_waitcnt
		lgkmcnt(0) and so do
		not need to be
		considered.)
		- Ensures any
		preceding
		sequential
		consistent local
		memory instructions
		have completed
		before executing
		this sequentially
		consistent
		instruction. This
		prevents reordering
		a seq_cst store
		followed by a
		seq_cst load. (Note
		that seq_cst is
		stronger than
		acquire/release as
		the reordering of
		load acquire
		followed by a store
		release is
		prevented by the
		s_waitcnt of
		the release, but
		there is nothing
		preventing a store
		release followed by
		load acquire from
		completing out of
		order. The s_waitcnt
		could be placed after
		seq_store or before
		the seq_load. We
		choose the load to
		make the s_waitcnt be
		as late as possible
		so that the store
		may have already
		completed.)

		2. *Following
		instructions same as
		corresponding load
		atomic acquire,
		except must generated
		all instructions even
		for OpenCL.*
		load atomic seq_cst - workgroup - local *Same as corresponding
		load atomic acquire,
		except must generated
		all instructions even
		for OpenCL.*

		load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) &
		- system - generic vmcnt(0)

		- Could be split into
		separate s_waitcnt
		vmcnt(0)
		and s_waitcnt
		lgkmcnt(0) to allow
		them to be
		independently moved
		according to the
		following rules.
		- s_waitcnt lgkmcnt(0)
		must happen after
		preceding
		global/generic load
		atomic/store
		atomic/atomicrmw
		with memory
		ordering of seq_cst
		and with equal or
		wider sync scope.
		(Note that seq_cst
		fences have their
		own s_waitcnt
		lgkmcnt(0) and so do
		not need to be
		considered.)
		- s_waitcnt vmcnt(0)
		must happen after
		preceding
		global/generic load
		atomic/store
		atomic/atomicrmw
		with memory
		ordering of seq_cst
		and with equal or
		wider sync scope.
		(Note that seq_cst
		fences have their
		own s_waitcnt
		vmcnt(0) and so do
		not need to be
		considered.)
		- Ensures any
		preceding
		sequential
		consistent global
		memory instructions
		have completed
		before executing
		this sequentially
		consistent
		instruction. This
		prevents reordering
		a seq_cst store
		followed by a
		seq_cst load. (Note
		that seq_cst is
		stronger than
		acquire/release as
		the reordering of
		load acquire
		followed by a store
		release is
		prevented by the
		s_waitcnt of
		the release, but
		there is nothing
		preventing a store
		release followed by
		load acquire from
		completing out of
		order. The s_waitcnt
		could be placed after
		seq_store or before
		the seq_load. We
		choose the load to
		make the s_waitcnt be
		as late as possible
		so that the store
		may have already
		completed.)

		2. *Following
		instructions same as
		corresponding load
		atomic acquire,
		except must generated
		all instructions even
		for OpenCL.*
		store atomic seq_cst - singlethread - global *Same as corresponding
		- wavefront - local store atomic release,
		- workgroup - generic except must generated
		- agent all instructions even
		- system for OpenCL.*
		atomicrmw seq_cst - singlethread - global *Same as corresponding
		- wavefront - local atomicrmw acq_rel,
		- workgroup - generic except must generated
		- agent all instructions even
		- system for OpenCL.*
		fence seq_cst - singlethread none *Same as corresponding
		- wavefront fence acq_rel,
		- workgroup except must generated
		- agent all instructions even
		- system for OpenCL.*
		============ ============ ============== ========== ================================

		.. _amdgpu-amdhsa-memory-model-gfx10:

		Memory Model GFX10
		++++++++++++++++++

For GFX10:		For GFX10:

* Each agent has multiple shader arrays (SA).		* Each agent has multiple shader arrays (SA).
* Each SA has multiple work-group processors (WGP).		* Each SA has multiple work-group processors (WGP).
* Each WGP has multiple compute units (CU).		* Each WGP has multiple compute units (CU).
* Each CU has multiple SIMDs that execute wavefronts.		* Each CU has multiple SIMDs that execute wavefronts.
* The wavefronts for a single work-group are executed in the same		* The wavefronts for a single work-group are executed in the same
WGP. In CU wavefront execution mode the wavefronts may be executed by		WGP. In CU wavefront execution mode the wavefronts may be executed by
▲ Show 20 Lines • Show All 55 Lines • ▼ Show 20 Lines	* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1
reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is		reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is
required to ensure synchronization between vector memory operations of		required to ensure synchronization between vector memory operations of
different SAs. It ensures a previous vector memory operation has completed		different SAs. It ensures a previous vector memory operation has completed
before executing a subsequent vector memory and so can be used to meet the		before executing a subsequent vector memory and so can be used to meet the
requirements of acquire, release and sequential consistency.		requirements of acquire, release and sequential consistency.
* The L2 cache can be kept coherent with other agents on some targets, or ranges		* The L2 cache can be kept coherent with other agents on some targets, or ranges
of virtual addresses can be set up to bypass it to ensure system coherence.		of virtual addresses can be set up to bypass it to ensure system coherence.

Private address space uses ``buffer_load/store`` using the scratch V#		Scalar memory operations are only used to access memory that is proven to not
		scott.linderUnsubmitted Done Reply Inline Actions used scott.linder: used
(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread		change during the execution of the kernel dispatch. This includes constant
is accessing the memory, atomic memory orderings are not meaningful, and all		address space and global address space for program scope const variables.
		scott.linderUnsubmitted Done Reply Inline Actions Same question as above scott.linder: Same question as above
accesses are treated as non-atomic.		Therefore, the kernel machine code does not have to maintain the scalar cache to
		ensure it is coherent with the vector caches. The scalar and vector caches are
Constant address space uses ``buffer/global_load`` instructions (or equivalent		invalidated between kernel dispatches by CP since constant address space data
scalar memory instructions). Since the constant address space contents do not		may change between kernel dispatch executions. See
change during the execution of a kernel dispatch it is not legal to perform
stores, and atomic memory orderings are not meaningful, and all access are
treated as non-atomic.

A memory synchronization scope wider than work-group is not meaningful for the
group (LDS) address space and is treated as work-group.

The memory model does not support the region address space which is treated as
non-atomic.

Acquire memory ordering is not meaningful on store atomic instructions and is
treated as non-atomic.

Release memory ordering is not meaningful on load atomic instructions and is
treated a non-atomic.

Acquire-release memory ordering is not meaningful on load or store atomic
instructions and is treated as acquire and release respectively.

AMDGPU backend only uses scalar memory operations to access memory that is
proven to not change during the execution of the kernel dispatch. This includes
constant address space and global address space for program scope const
variables. Therefore, the kernel machine code does not have to maintain the
scalar L1 cache to ensure it is coherent with the vector L1 cache. The scalar
and vector L1 caches are invalidated between kernel dispatches by CP since
constant address space data may change between kernel dispatch executions. See
:ref:`amdgpu-amdhsa-memory-spaces`.		:ref:`amdgpu-amdhsa-memory-spaces`.

The one exception is if scalar writes are used to spill SGPR registers. In this		The one exception is if scalar writes are used to spill SGPR registers. In this
case the AMDGPU backend ensures the memory location used to spill is never		case the AMDGPU backend ensures the memory location used to spill is never
accessed by vector memory operations at the same time. If scalar writes are used		accessed by vector memory operations at the same time. If scalar writes are used
then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function		then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function
return since the locations may be used for vector memory instructions by a		return since the locations may be used for vector memory instructions by a
future wavefront that uses the same scratch area, or a function call that		future wavefront that uses the same scratch area, or a function call that
creates a frame at the same address, respectively. There is no need for a		creates a frame at the same address, respectively. There is no need for a
``s_dcache_inv`` as all scalar writes are write-before-read in the same thread.		``s_dcache_inv`` as all scalar writes are write-before-read in the same thread.

For GFX6-GFX9, scratch backing memory (which is used for the private address		For kernarg backing memory:
space) is accessed with MTYPE NC_NV (non-coherent non-volatile). Since the
private address space is only accessed by a single thread, and is always
write-before-read, there is never a need to invalidate these entries from the L1
cache. Hence all cache invalidates are done as ``*_vol`` to only invalidate the
volatile cache lines.

For GFX10, scratch backing memory (which is used for the private address space)
is accessed with MTYPE NC (non-coherent). Since the private address space is
only accessed by a single thread, and is always write-before-read, there is
never a need to invalidate these entries from the L0 or L1 caches.

For GFX10, wavefronts are executed in native mode with in-order reporting of		* CP invalidates the L0 and L1 caches at the start of each kernel dispatch.
loads and sample instructions. In this mode vmcnt reports completion of load,		* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid
atomic with return and sample instructions in order, and the vscnt reports the		needing to invalidate the L2 cache.
completion of store and atomic without return in order. See ``MEM_ORDERED``		* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and
field in :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`.		so the L2 cache will be coherent with the CPU and other agents.

		Scratch backing memory (which is used for the private address space) is accessed
		with MTYPE NC (non-coherent). Since the private address space is only accessed
		by a single thread, and is always write-before-read, there is never a need to
		invalidate these entries from the L0 or L1 caches.

		Wavefronts are executed in native mode with in-order reporting of loads and
		sample instructions. In this mode vmcnt reports completion of load, atomic with
		return and sample instructions in order, and the vscnt reports the completion of
		store and atomic without return in order. See ``MEM_ORDERED`` field in
		:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`.

In GFX10, wavefronts can be executed in WGP or CU wavefront execution mode:		Wavefronts can be executed in WGP or CU wavefront execution mode:

* In WGP wavefront execution mode the wavefronts of a work-group are executed		* In WGP wavefront execution mode the wavefronts of a work-group are executed
on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per		on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per
CU L0 caches is required for work-group synchronization. Also accesses to L1		CU L0 caches is required for work-group synchronization. Also accesses to L1
at work-group scope need to be explicitly ordered as the accesses from		at work-group scope need to be explicitly ordered as the accesses from
different CUs are not ordered.		different CUs are not ordered.
* In CU wavefront execution mode the wavefronts of a work-group are executed on		* In CU wavefront execution mode the wavefronts of a work-group are executed on
the SIMDs of a single CU of the WGP. Therefore, all global memory access by		the SIMDs of a single CU of the WGP. Therefore, all global memory access by
the work-group access the same L0 which in turn ensures L1 accesses are		the work-group access the same L0 which in turn ensures L1 accesses are
ordered and so do not require explicit management of the caches for		ordered and so do not require explicit management of the caches for
work-group synchronization.		work-group synchronization.

See ``WGP_MODE`` field in		See ``WGP_MODE`` field in
:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and		:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and
:ref:`amdgpu-target-features`.		:ref:`amdgpu-target-features`.

On dGPU the kernarg backing memory is accessed as UC (uncached) to avoid needing		The code sequences used to implement the memory model for GFX10 are defined in
to invalidate the L2 cache. For GFX6-GFX9, this also causes it to be treated as		table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`.
non-volatile and so is not invalidated by ``*_vol``. On APU it is accessed as CC
(cache coherent) and so the L2 cache will be coherent with the CPU and other		.. table:: AMDHSA Memory Model Code Sequences GFX10
agents.		:name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table

.. table:: AMDHSA Memory Model Code Sequences GFX6-GFX10		============ ============ ============== ========== ================================
:name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table		LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code
		Ordering Sync Scope Address GFX10
============ ============ ============== ========== ================================ ================================
LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code AMDGPU Machine Code
Ordering Sync Scope Address GFX6-9 GFX10
Space		Space
============ ============ ============== ========== ================================ ================================		============ ============ ============== ========== ================================
Non-Atomic		Non-Atomic
---------------------------------------------------------------------------------------------------------------------		------------------------------------------------------------------------------------
load none none - global - !volatile & !nontemporal - !volatile & !nontemporal		load none none - global - !volatile & !nontemporal
- generic		- generic
- private 1. buffer/global/flat_load 1. buffer/global/flat_load		- private 1. buffer/global/flat_load
- constant		- constant
- volatile & !nontemporal - volatile & !nontemporal		- volatile & !nontemporal

1. buffer/global/flat_load 1. buffer/global/flat_load		1. buffer/global/flat_load
glc=1 glc=1 dlc=1		glc=1 dlc=1

- nontemporal - nontemporal		- nontemporal

1. buffer/global/flat_load 1. buffer/global/flat_load		1. buffer/global/flat_load
glc=1 slc=1 slc=1		slc=1

load none none - local 1. ds_load 1. ds_load		load none none - local 1. ds_load
store none none - global - !nontemporal - !nontemporal		store none none - global - !nontemporal
- generic		- generic
- private 1. buffer/global/flat_store 1. buffer/global/flat_store		- private 1. buffer/global/flat_store
- constant		- constant
- nontemporal - nontemporal		- nontemporal

1. buffer/global/flat_store 1. buffer/global/flat_store		1. buffer/global/flat_store
glc=1 slc=1 slc=1		slc=1

store none none - local 1. ds_store 1. ds_store		store none none - local 1. ds_store
Unordered Atomic		Unordered Atomic
---------------------------------------------------------------------------------------------------------------------		------------------------------------------------------------------------------------
load atomic unordered any any Same as non-atomic. Same as non-atomic.		load atomic unordered any any Same as non-atomic.
store atomic unordered any any Same as non-atomic. Same as non-atomic.		store atomic unordered any any Same as non-atomic.
atomicrmw unordered any any Same as monotonic Same as monotonic		atomicrmw unordered any any *Same as monotonic
atomic. atomic.		atomic*.
Monotonic Atomic		Monotonic Atomic
---------------------------------------------------------------------------------------------------------------------		------------------------------------------------------------------------------------
load atomic monotonic - singlethread - global 1. buffer/global/flat_load 1. buffer/global/flat_load		load atomic monotonic - singlethread - global 1. buffer/global/flat_load
- wavefront - generic		- wavefront - generic
load atomic monotonic - workgroup - global 1. buffer/global/flat_load 1. buffer/global/flat_load		load atomic monotonic - workgroup - global 1. buffer/global/flat_load
- generic glc=1		- generic glc=1

- If CU wavefront execution		- If CU wavefront execution
mode, omit glc=1.		mode, omit glc=1.

load atomic monotonic - singlethread - local 1. ds_load 1. ds_load		load atomic monotonic - singlethread - local 1. ds_load
- wavefront		- wavefront
- workgroup		- workgroup
load atomic monotonic - agent - global 1. buffer/global/flat_load 1. buffer/global/flat_load		load atomic monotonic - agent - global 1. buffer/global/flat_load
- system - generic glc=1 glc=1 dlc=1		- system - generic glc=1 dlc=1
store atomic monotonic - singlethread - global 1. buffer/global/flat_store 1. buffer/global/flat_store		store atomic monotonic - singlethread - global 1. buffer/global/flat_store
- wavefront - generic		- wavefront - generic
- workgroup		- workgroup
- agent		- agent
- system		- system
store atomic monotonic - singlethread - local 1. ds_store 1. ds_store		store atomic monotonic - singlethread - local 1. ds_store
- wavefront		- wavefront
- workgroup		- workgroup
atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 1. buffer/global/flat_atomic		atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic
- wavefront - generic		- wavefront - generic
- workgroup		- workgroup
- agent		- agent
- system		- system
atomicrmw monotonic - singlethread - local 1. ds_atomic 1. ds_atomic		atomicrmw monotonic - singlethread - local 1. ds_atomic
- wavefront		- wavefront
- workgroup		- workgroup
Acquire Atomic		Acquire Atomic
---------------------------------------------------------------------------------------------------------------------		------------------------------------------------------------------------------------
load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 1. buffer/global/ds/flat_load		load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load
- wavefront - local		- wavefront - local
- generic		- generic
load atomic acquire - workgroup - global 1. buffer/global_load 1. buffer/global_load glc=1		load atomic acquire - workgroup - global 1. buffer/global_load glc=1

- If CU wavefront execution		- If CU wavefront execution
mode, omit glc=1.		mode, omit glc=1.

2. s_waitcnt vmcnt(0)		2. s_waitcnt vmcnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- Must happen before		- Must happen before
the following buffer_gl0_inv		the following buffer_gl0_inv
and before any following		and before any following
global/generic		global/generic
load/load		load/load
atomic/store/store		atomic/store/store
atomic/atomicrmw.		atomic/atomicrmw.

3. buffer_gl0_inv		3. buffer_gl0_inv

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- Ensures that		- Ensures that
following		following
loads will not see		loads will not see
stale data.		stale data.

load atomic acquire - workgroup - local 1. ds_load 1. ds_load		load atomic acquire - workgroup - local 1. ds_load
2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0)		2. s_waitcnt lgkmcnt(0)

- If OpenCL, omit. - If OpenCL, omit.		- If OpenCL, omit.
- Must happen before - Must happen before		- Must happen before
any following the following buffer_gl0_inv		the following buffer_gl0_inv
global/generic and before any following		and before any following
load/load global/generic load/load		global/generic load/load
atomic/store/store atomic/store/store		atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Ensures any - Ensures any		- Ensures any
following global following global		following global
data read is no data read is no		data read is no
older than the load older than the load		older than the local load
atomic value being atomic value being		atomic value being
acquired. acquired.		acquired.

3. buffer_gl0_inv		3. buffer_gl0_inv

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- If OpenCL, omit.		- If OpenCL, omit.
- Ensures that		- Ensures that
following		following
loads will not see		loads will not see
stale data.		stale data.

load atomic acquire - workgroup - generic 1. flat_load 1. flat_load glc=1		load atomic acquire - workgroup - generic 1. flat_load glc=1

- If CU wavefront execution		- If CU wavefront execution
mode, omit glc=1.		mode, omit glc=1.

2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) &		2. s_waitcnt lgkmcnt(0) &
vmcnt(0)		vmcnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit vmcnt(0).		mode, omit vmcnt(0).
- If OpenCL, omit. - If OpenCL, omit		- If OpenCL, omit
lgkmcnt(0).		lgkmcnt(0).
- Must happen before - Must happen before		- Must happen before
any following the following		the following
global/generic buffer_gl0_inv and any		buffer_gl0_inv and any
load/load following global/generic		following global/generic
atomic/store/store load/load		load/load
atomic/atomicrmw. atomic/store/store		atomic/store/store
atomic/atomicrmw.		atomic/atomicrmw.
- Ensures any - Ensures any		- Ensures any
following global following global		following global
data read is no data read is no		data read is no
older than the load older than the load		older than a local load
		scott.linderUnsubmitted Not Done Reply Inline Actions Is the change from "the" to "a" significant? scott.linder: Is the change from "the" to "a" significant?
		t-tyeAuthorUnsubmitted Done Reply Inline Actions Yes. This case is for generic and so if may be "a" local address space, but the local case is always "the" address space. t-tye: Yes. This case is for generic and so if may be "a" local address space, but the local case is…
atomic value being atomic value being		atomic value being
acquired. acquired.		acquired.

3. buffer_gl0_inv		3. buffer_gl0_inv

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- Ensures that		- Ensures that
following		following
loads will not see		loads will not see
stale data.		stale data.

load atomic acquire - agent - global 1. buffer/global_load 1. buffer/global_load		load atomic acquire - agent - global 1. buffer/global_load
- system glc=1 glc=1 dlc=1		- system glc=1 dlc=1
2. s_waitcnt vmcnt(0) 2. s_waitcnt vmcnt(0)		2. s_waitcnt vmcnt(0)

- Must happen before - Must happen before		- Must happen before
following following		following
buffer_wbinvl1_vol. buffer_gl*_inv.		buffer_gl*_inv.
- Ensures the load - Ensures the load		- Ensures the load
has completed has completed		has completed
before invalidating before invalidating		before invalidating
the cache. the caches.		the caches.

3. buffer_wbinvl1_vol 3. buffer_gl0_inv;		3. buffer_gl0_inv;
buffer_gl1_inv		buffer_gl1_inv

- Must happen before - Must happen before		- Must happen before
any following any following		any following
global/generic global/generic		global/generic
load/load load/load		load/load
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Ensures that - Ensures that		- Ensures that
following following		following
loads will not see loads will not see		loads will not see
stale global data. stale global data.		stale global data.

load atomic acquire - agent - generic 1. flat_load glc=1 1. flat_load glc=1 dlc=1		load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1
- system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vmcnt(0) &		- system 2. s_waitcnt vmcnt(0) &
lgkmcnt(0) lgkmcnt(0)		lgkmcnt(0)

- If OpenCL omit - If OpenCL omit
lgkmcnt(0). lgkmcnt(0).
- Must happen before - Must happen before
following following
buffer_wbinvl1_vol. buffer_gl*_invl.
- Ensures the flat_load - Ensures the flat_load
has completed has completed
before invalidating before invalidating
the cache. the caches.

3. buffer_wbinvl1_vol 3. buffer_gl0_inv;		- If OpenCL omit
		lgkmcnt(0).
		- Must happen before
		following
		buffer_gl*_invl.
		- Ensures the flat_load
		has completed
		before invalidating
		the caches.

		3. buffer_gl0_inv;
buffer_gl1_inv		buffer_gl1_inv

- Must happen before - Must happen before		- Must happen before
any following any following		any following
global/generic global/generic		global/generic
load/load load/load		load/load
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Ensures that - Ensures that		- Ensures that
following loads following loads		following loads
will not see stale will not see stale		will not see stale
global data. global data.		global data.

atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic		atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic
- wavefront - local		- wavefront - local
- generic		- generic
atomicrmw acquire - workgroup - global 1. buffer/global_atomic 1. buffer/global_atomic		atomicrmw acquire - workgroup - global 1. buffer/global_atomic
2. s_waitcnt vm/vscnt(0)		2. s_waitcnt vm/vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- Use vmcnt(0) if atomic with		- Use vmcnt(0) if atomic with
return and vscnt(0) if		return and vscnt(0) if
atomic with no-return.		atomic with no-return.
- Must happen before		- Must happen before
the following buffer_gl0_inv		the following buffer_gl0_inv
and before any following		and before any following
global/generic		global/generic
load/load		load/load
atomic/store/store		atomic/store/store
atomic/atomicrmw.		atomic/atomicrmw.

3. buffer_gl0_inv		3. buffer_gl0_inv

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- Ensures that		- Ensures that
following		following
loads will not see		loads will not see
stale data.		stale data.

atomicrmw acquire - workgroup - local 1. ds_atomic 1. ds_atomic		atomicrmw acquire - workgroup - local 1. ds_atomic
2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0)		2. s_waitcnt lgkmcnt(0)

- If OpenCL, omit. - If OpenCL, omit.		- If OpenCL, omit.
- Must happen before - Must happen before		- Must happen before
any following the following		the following
global/generic buffer_gl0_inv.		buffer_gl0_inv.
load/load		- Ensures any
atomic/store/store		following global
atomic/atomicrmw.		data read is no
- Ensures any - Ensures any		older than the local
following global following global		atomicrmw value
data read is no data read is no		being acquired.
older than the older than the
atomicrmw value atomicrmw value
being acquired. being acquired.

3. buffer_gl0_inv		3. buffer_gl0_inv

- If OpenCL omit.		- If OpenCL omit.
- Ensures that		- Ensures that
following		following
loads will not see		loads will not see
stale data.		stale data.

atomicrmw acquire - workgroup - generic 1. flat_atomic 1. flat_atomic		atomicrmw acquire - workgroup - generic 1. flat_atomic
2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) &		2. s_waitcnt lgkmcnt(0) &
vm/vscnt(0)		vm/vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit vm/vscnt(0).		mode, omit vm/vscnt(0).
- If OpenCL, omit. - If OpenCL, omit		- If OpenCL, omit lgkmcnt(0).
waitcnt lgkmcnt(0).
- Use vmcnt(0) if atomic with		- Use vmcnt(0) if atomic with
return and vscnt(0) if		return and vscnt(0) if
atomic with no-return.		atomic with no-return.
- Must happen before - Must happen before		- Must happen before
any following the following		the following
global/generic buffer_gl0_inv.		buffer_gl0_inv.
load/load		- Ensures any
atomic/store/store		following global
atomic/atomicrmw.		data read is no
- Ensures any - Ensures any		older than a local
		scott.linderUnsubmitted Not Done Reply Inline Actions Same question here scott.linder: Same question here
		t-tyeAuthorUnsubmitted Done Reply Inline Actions Same answer. t-tye: Same answer.
following global following global		atomicrmw value
data read is no data read is no		being acquired.
older than the older than the
atomicrmw value atomicrmw value
being acquired. being acquired.

3. buffer_gl0_inv		3. buffer_gl0_inv

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- Ensures that		- Ensures that
following		following
loads will not see		loads will not see
stale data.		stale data.

atomicrmw acquire - agent - global 1. buffer/global_atomic 1. buffer/global_atomic		atomicrmw acquire - agent - global 1. buffer/global_atomic
- system 2. s_waitcnt vmcnt(0) 2. s_waitcnt vm/vscnt(0)		- system 2. s_waitcnt vm/vscnt(0)

- Use vmcnt(0) if atomic with		- Use vmcnt(0) if atomic with
return and vscnt(0) if		return and vscnt(0) if
atomic with no-return.		atomic with no-return.
waitcnt lgkmcnt(0).		- Must happen before
- Must happen before - Must happen before		following
following following		buffer_gl*_inv.
buffer_wbinvl1_vol. buffer_gl*_inv.		- Ensures the
- Ensures the - Ensures the		atomicrmw has
atomicrmw has atomicrmw has		completed before
completed before completed before		invalidating the
invalidating the invalidating the		caches.
cache. caches.

3. buffer_wbinvl1_vol 3. buffer_gl0_inv;		3. buffer_gl0_inv;
buffer_gl1_inv		buffer_gl1_inv

- Must happen before - Must happen before		- Must happen before
any following any following		any following
global/generic global/generic		global/generic
load/load load/load		load/load
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Ensures that - Ensures that		- Ensures that
following loads following loads		following loads
will not see stale will not see stale		will not see stale
global data. global data.		global data.

atomicrmw acquire - agent - generic 1. flat_atomic 1. flat_atomic		atomicrmw acquire - agent - generic 1. flat_atomic
- system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vm/vscnt(0) &		- system 2. s_waitcnt vm/vscnt(0) &
lgkmcnt(0) lgkmcnt(0)		lgkmcnt(0)

- If OpenCL, omit - If OpenCL, omit		- If OpenCL, omit
lgkmcnt(0). lgkmcnt(0).		lgkmcnt(0).
- Use vmcnt(0) if atomic with		- Use vmcnt(0) if atomic with
return and vscnt(0) if		return and vscnt(0) if
atomic with no-return.		atomic with no-return.
- Must happen before - Must happen before		- Must happen before
following following		following
buffer_wbinvl1_vol. buffer_gl*_inv.		buffer_gl*_inv.
- Ensures the - Ensures the		- Ensures the
atomicrmw has atomicrmw has		atomicrmw has
completed before completed before		completed before
invalidating the invalidating the		invalidating the
cache. caches.		caches.

3. buffer_wbinvl1_vol 3. buffer_gl0_inv;		3. buffer_gl0_inv;
buffer_gl1_inv		buffer_gl1_inv

- Must happen before - Must happen before		- Must happen before
any following any following		any following
global/generic global/generic		global/generic
load/load load/load		load/load
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Ensures that - Ensures that		- Ensures that
following loads following loads		following loads
will not see stale will not see stale		will not see stale
global data. global data.		global data.

fence acquire - singlethread none none none		fence acquire - singlethread none none
- wavefront		- wavefront
fence acquire - workgroup none 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &		fence acquire - workgroup none 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0)		vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit vmcnt(0) and		mode, omit vmcnt(0) and
vscnt(0).		vscnt(0).
- If OpenCL and - If OpenCL and		- If OpenCL and
address space is address space is		address space is
not generic, omit. not generic, omit		not generic, omit
lgkmcnt(0).		lgkmcnt(0).
- If OpenCL and		- If OpenCL and
address space is		address space is
local, omit		local, omit
vmcnt(0) and vscnt(0).		vmcnt(0) and vscnt(0).
- However, since LLVM - However, since LLVM		- However, since LLVM
currently has no currently has no		currently has no
address space on address space on		address space on
the fence need to the fence need to		the fence need to
conservatively conservatively		conservatively
always generate. If always generate. If		always generate. If
fence had an fence had an		fence had an
address space then address space then		address space then
set to address set to address		set to address
space of OpenCL space of OpenCL		space of OpenCL
fence flag, or to fence flag, or to		fence flag, or to
generic if both generic if both		generic if both
local and global local and global		local and global
flags are flags are		flags are
specified. specified.		specified.
- Must happen after
any preceding
local/generic load
atomic/atomicrmw
with an equal or
wider sync scope
and memory ordering
stronger than
unordered (this is
termed the
fence-paired-atomic).
- Must happen before
any following
global/generic
load/load
atomic/store/store
atomic/atomicrmw.
- Ensures any
following global
data read is no
older than the
value read by the
fence-paired-atomic.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt		vscnt(0) and s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic load		global/generic load
atomic/		atomic/
atomicrmw-with-return-value		atomicrmw-with-return-value
with an equal or		with an equal or
wider sync scope		wider sync scope
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic).		fence-paired-atomic).
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
atomicrmw-no-return-value		atomicrmw-no-return-value
with an equal or		with an equal or
wider sync scope		wider sync scope
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic).		fence-paired-atomic).
- s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after		must happen after
any preceding		any preceding
local/generic load		local/generic load
atomic/atomicrmw		atomic/atomicrmw
with an equal or		with an equal or
wider sync scope		wider sync scope
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic).		fence-paired-atomic).
- Must happen before		- Must happen before
the following		the following
buffer_gl0_inv.		buffer_gl0_inv.
- Ensures that the		- Ensures that the
fence-paired atomic		fence-paired atomic
has completed		has completed
before invalidating		before invalidating
the		the
cache. Therefore		cache. Therefore
any following		any following
locations read must		locations read must
be no older than		be no older than
the value read by		the value read by
the		the
fence-paired-atomic.		fence-paired-atomic.

3. buffer_gl0_inv		3. buffer_gl0_inv

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- Ensures that		- Ensures that
following		following
loads will not see		loads will not see
stale data.		stale data.

fence acquire - agent none 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &		fence acquire - agent none 1. s_waitcnt lgkmcnt(0) &
- system vmcnt(0) vmcnt(0) & vscnt(0)		- system vmcnt(0) & vscnt(0)

- If OpenCL and - If OpenCL and		- If OpenCL and
address space is address space is		address space is
not generic, omit not generic, omit		not generic, omit
lgkmcnt(0). lgkmcnt(0).		lgkmcnt(0).
- If OpenCL and		- If OpenCL and
address space is		address space is
local, omit		local, omit
vmcnt(0) and vscnt(0).		vmcnt(0) and vscnt(0).
- However, since LLVM - However, since LLVM		- However, since LLVM
currently has no currently has no		currently has no
address space on address space on		address space on
the fence need to the fence need to		the fence need to
conservatively conservatively		conservatively
always generate always generate		always generate
(see comment for (see comment for		(see comment for
previous fence). previous fence).		previous fence).
- Could be split into
separate s_waitcnt
vmcnt(0) and
s_waitcnt
lgkmcnt(0) to allow
them to be
independently moved
according to the
following rules.
- s_waitcnt vmcnt(0)
must happen after
any preceding
global/generic load
atomic/atomicrmw
with an equal or
wider sync scope
and memory ordering
stronger than
unordered (this is
termed the
fence-paired-atomic).
- s_waitcnt lgkmcnt(0)
must happen after
any preceding
local/generic load
atomic/atomicrmw
with an equal or
wider sync scope
and memory ordering
stronger than
unordered (this is
termed the
fence-paired-atomic).
- Must happen before
the following
buffer_wbinvl1_vol.
- Ensures that the
fence-paired atomic
has completed
before invalidating
the
cache. Therefore
any following
locations read must
be no older than
the value read by
the
fence-paired-atomic.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt		vscnt(0) and s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic load		global/generic load
atomic/		atomic/
atomicrmw-with-return-value		atomicrmw-with-return-value
with an equal or		with an equal or
wider sync scope		wider sync scope
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic).		fence-paired-atomic).
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
atomicrmw-no-return-value		atomicrmw-no-return-value
with an equal or		with an equal or
wider sync scope		wider sync scope
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic).		fence-paired-atomic).
- s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after		must happen after
any preceding		any preceding
local/generic load		local/generic load
atomic/atomicrmw		atomic/atomicrmw
with an equal or		with an equal or
wider sync scope		wider sync scope
and memory ordering		and memory ordering
stronger than		stronger than
unordered (this is		unordered (this is
termed the		termed the
fence-paired-atomic).		fence-paired-atomic).
- Must happen before		- Must happen before
the following		the following
buffer_gl*_inv.		buffer_gl*_inv.
- Ensures that the		- Ensures that the
fence-paired atomic		fence-paired atomic
has completed		has completed
before invalidating		before invalidating
the		the
caches. Therefore		caches. Therefore
any following		any following
locations read must		locations read must
be no older than		be no older than
the value read by		the value read by
the		the
fence-paired-atomic.		fence-paired-atomic.

2. buffer_wbinvl1_vol 2. buffer_gl0_inv;		2. buffer_gl0_inv;
buffer_gl1_inv		buffer_gl1_inv

- Must happen before any - Must happen before any		- Must happen before any
following global/generic following global/generic		following global/generic
load/load load/load		load/load
atomic/store/store atomic/store/store		atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Ensures that - Ensures that		- Ensures that
following loads following loads		following loads
will not see stale will not see stale		will not see stale
global data. global data.		global data.

Release Atomic		Release Atomic
---------------------------------------------------------------------------------------------------------------------		------------------------------------------------------------------------------------
store atomic release - singlethread - global 1. buffer/global/ds/flat_store 1. buffer/global/ds/flat_store		store atomic release - singlethread - global 1. buffer/global/ds/flat_store
- wavefront - local		- wavefront - local
- generic		- generic
store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &		store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0)		- generic vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit vmcnt(0) and		mode, omit vmcnt(0) and
vscnt(0).		vscnt(0).
- If OpenCL, omit. - If OpenCL, omit		- If OpenCL, omit
lgkmcnt(0).		lgkmcnt(0).
- Must happen after
any preceding
local/generic
load/store/load
atomic/store
atomic/atomicrmw.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt		vscnt(0) and s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic load/load		global/generic load/load
atomic/		atomic/
atomicrmw-with-return-value.		atomicrmw-with-return-value.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store		store/store
atomic/		atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after		must happen after
any preceding		any preceding
local/generic		local/generic
load/store/load		load/store/load
atomic/store		atomic/store
atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before - Must happen before		- Must happen before
the following the following		the following
store. store.		store.
- Ensures that all - Ensures that all		- Ensures that all
memory operations memory operations		memory operations
to local have have		have
completed before completed before		completed before
performing the performing the		performing the
store that is being store that is being		store that is being
released. released.		released.

2. buffer/global_store 2. buffer/global_store		2. buffer/global/flat_store
store atomic release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0)		store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- If OpenCL, omit.		- If OpenCL, omit.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0) and s_waitcnt		vmcnt(0) and s_waitcnt
vscnt(0) to allow		vscnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic load/load		global/generic load/load
atomic/		atomic/
atomicrmw-with-return-value.		atomicrmw-with-return-value.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store atomic/		store/store atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- Must happen before		- Must happen before
the following		the following
store.		store.
- Ensures that all		- Ensures that all
global memory		global memory
operations have		operations have
completed before		completed before
performing the		performing the
store that is being		store that is being
released.		released.

1. ds_store 2. ds_store		2. ds_store
store atomic release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &		store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0)		- system - generic vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If OpenCL and
mode, omit vmcnt(0) and		address space is
vscnt(0).		not generic, omit
- If OpenCL, omit. - If OpenCL, omit
lgkmcnt(0).		lgkmcnt(0).
- Must happen after
any preceding
local/generic
load/store/load
atomic/store
atomic/atomicrmw.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt vscnt(0)
vscnt(0) and s_waitcnt		and s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic load/load		global/generic
		load/load
atomic/		atomic/
atomicrmw-with-return-value.		atomicrmw-with-return-value.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store		store/store atomic/
atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after		must happen after
any preceding		any preceding
local/generic		local/generic
load/store/load		load/store/load
atomic/store		atomic/store
atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before - Must happen before		- Must happen before
the following the following		the following
store. store.		store.
- Ensures that all - Ensures that all		- Ensures that all
memory operations memory operations		memory operations
to local have have		have
		scott.linderUnsubmitted Done Reply Inline Actions Redundant, delete scott.linder: Redundant, delete
completed before completed before		completed before
performing the performing the		performing the
store that is being store that is being		store that is being
released. released.		released.

2. flat_store 2. flat_store
store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &
- system - generic vmcnt(0) vmcnt(0) & vscnt(0)

- If OpenCL, omit - If OpenCL, omit
lgkmcnt(0). lgkmcnt(0).
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0) and vmcnt(0), s_waitcnt vscnt(0)
s_waitcnt and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
load/store/load load/load
atomic/store atomic/
atomic/atomicrmw. atomicrmw-with-return-value.
- s_waitcnt vscnt(0)
must happen after
any preceding
global/generic
store/store atomic/
atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before
the following the following
store. store.
- Ensures that all - Ensures that all
memory operations memory operations
to memory have to memory have
completed before completed before
performing the performing the
store that is being store that is being
released. released.

2. buffer/global/flat_store 2. buffer/global/flat_store		2. buffer/global/flat_store
atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic		atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic
- wavefront - local		- wavefront - local
- generic		- generic
atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &		atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0)		- generic vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit vmcnt(0) and		mode, omit vmcnt(0) and
vscnt(0).		vscnt(0).
- If OpenCL, omit.		- If OpenCL, omit lgkmcnt(0).

- Must happen after
any preceding
local/generic
load/store/load
atomic/store
atomic/atomicrmw.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt		vscnt(0) and s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic load/load		global/generic load/load
atomic/		atomic/
atomicrmw-with-return-value.		atomicrmw-with-return-value.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store		store/store
atomic/		atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after		must happen after
any preceding		any preceding
local/generic		local/generic
load/store/load		load/store/load
atomic/store		atomic/store
atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before - Must happen before		- Must happen before
the following the following		the following
atomicrmw. atomicrmw.		atomicrmw.
- Ensures that all - Ensures that all		- Ensures that all
memory operations memory operations		memory operations
to local have have		have
completed before completed before		completed before
performing the performing the		performing the
atomicrmw that is atomicrmw that is		atomicrmw that is
being released. being released.		being released.

2. buffer/global_atomic 2. buffer/global_atomic		2. buffer/global/flat_atomic
atomicrmw release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0)		atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- If OpenCL, omit.		- If OpenCL, omit.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0) and s_waitcnt		vmcnt(0) and s_waitcnt
vscnt(0) to allow		vscnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic load/load		global/generic load/load
atomic/		atomic/
atomicrmw-with-return-value.		atomicrmw-with-return-value.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store atomic/		store/store atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- Must happen before		- Must happen before
the following		the following
store.		store.
- Ensures that all		- Ensures that all
global memory		global memory
operations have		operations have
completed before		completed before
performing the		performing the
store that is being		store that is being
released.		released.

1. ds_atomic 2. ds_atomic		2. ds_atomic
atomicrmw release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &		atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) &
		scott.linderUnsubmitted Not Done Reply Inline Actions lgkmcnt scott.linder: lgkmcnt
vmcnt(0) & vscnt(0)		- system - generic vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If OpenCL, omit
mode, omit vmcnt(0) and
vscnt(0).
- If OpenCL, omit. - If OpenCL, omit
lgkmcnt(0).		lgkmcnt(0).
- Must happen after
any preceding
local/generic
load/store/load
atomic/store
atomic/atomicrmw.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt		vscnt(0) and s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic load/load		global/generic
atomic/		load/load atomic/
atomicrmw-with-return-value.		atomicrmw-with-return-value.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store		store/store atomic/
atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after		must happen after
any preceding		any preceding
local/generic		local/generic
load/store/load		load/store/load
atomic/store		atomic/store
atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before - Must happen before		- Must happen before
the following the following		the following
atomicrmw. atomicrmw.		atomicrmw.
- Ensures that all - Ensures that all		- Ensures that all
memory operations memory operations		memory operations
to local have have		to global and local
completed before completed before		have completed
performing the performing the		before performing
atomicrmw that is atomicrmw that is		the atomicrmw that
being released. being released.		is being released.

2. flat_atomic 2. flat_atomic
atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lkkmcnt(0) &
- system - generic vmcnt(0) vmcnt(0) & vscnt(0)

- If OpenCL, omit - If OpenCL, omit
lgkmcnt(0). lgkmcnt(0).
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0) and vmcnt(0), s_waitcnt
s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
load/store/load load/load atomic/
atomic/store atomicrmw-with-return-value.
atomic/atomicrmw.
- s_waitcnt vscnt(0)
must happen after
any preceding
global/generic
store/store atomic/
atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before
the following the following
atomicrmw. atomicrmw.
- Ensures that all - Ensures that all
memory operations memory operations
to global and local to global and local
have completed have completed
before performing before performing
the atomicrmw that the atomicrmw that
is being released. is being released.

2. buffer/global/flat_atomic 2. buffer/global/flat_atomic		2. buffer/global/flat_atomic
fence release - singlethread none none none		fence release - singlethread none none
- wavefront		- wavefront
fence release - workgroup none 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &		fence release - workgroup none 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0)		vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit vmcnt(0) and		mode, omit vmcnt(0) and
vscnt(0).		vscnt(0).
- If OpenCL and - If OpenCL and		- If OpenCL and
address space is address space is		address space is
not generic, omit. not generic, omit		not generic, omit
lgkmcnt(0).		lgkmcnt(0).
- If OpenCL and		- If OpenCL and
address space is		address space is
local, omit		local, omit
vmcnt(0) and vscnt(0).		vmcnt(0) and vscnt(0).
- However, since LLVM - However, since LLVM		- However, since LLVM
currently has no currently has no		currently has no
address space on address space on		address space on
the fence need to the fence need to		the fence need to
conservatively conservatively		conservatively
always generate. If always generate. If		always generate. If
fence had an fence had an		fence had an
address space then address space then		address space then
set to address set to address		set to address
space of OpenCL space of OpenCL		space of OpenCL
fence flag, or to fence flag, or to		fence flag, or to
generic if both generic if both		generic if both
local and global local and global		local and global
flags are flags are		flags are
specified. specified.		specified.
- Must happen after
any preceding
local/generic
load/load
atomic/store/store
atomic/atomicrmw.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt		vscnt(0) and s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
load/load		load/load
atomic/		atomic/
atomicrmw-with-return-value.		atomicrmw-with-return-value.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store atomic/		store/store atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after		must happen after
any preceding		any preceding
local/generic		local/generic
load/store/load		load/store/load
atomic/store atomic/		atomic/store atomic/
atomicrmw.		atomicrmw.
- Must happen before - Must happen before		- Must happen before
any following store any following store		any following store
atomic/atomicrmw atomic/atomicrmw		atomic/atomicrmw
with an equal or with an equal or		with an equal or
wider sync scope wider sync scope		wider sync scope
and memory ordering and memory ordering		and memory ordering
stronger than stronger than		stronger than
unordered (this is unordered (this is		unordered (this is
termed the termed the		termed the
fence-paired-atomic). fence-paired-atomic).		fence-paired-atomic).
- Ensures that all - Ensures that all		- Ensures that all
memory operations memory operations		memory operations
to local have have		have
completed before completed before		completed before
performing the performing the		performing the
following following		following
fence-paired-atomic. fence-paired-atomic.		fence-paired-atomic.

fence release - agent none 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &		fence release - agent none 1. s_waitcnt lgkmcnt(0) &
- system vmcnt(0) vmcnt(0) & vscnt(0)		- system vmcnt(0) & vscnt(0)

- If OpenCL and - If OpenCL and		- If OpenCL and
address space is address space is		address space is
not generic, omit not generic, omit		not generic, omit
lgkmcnt(0). lgkmcnt(0).		lgkmcnt(0).
- If OpenCL and - If OpenCL and		- If OpenCL and
address space is address space is		address space is
local, omit local, omit		local, omit
vmcnt(0). vmcnt(0) and vscnt(0).		vmcnt(0) and vscnt(0).
- However, since LLVM - However, since LLVM		- However, since LLVM
currently has no currently has no		currently has no
address space on address space on		address space on
the fence need to the fence need to		the fence need to
conservatively conservatively		conservatively
always generate. If always generate. If		always generate. If
fence had an fence had an		fence had an
address space then address space then		address space then
set to address set to address		set to address
space of OpenCL space of OpenCL		space of OpenCL
fence flag, or to fence flag, or to		fence flag, or to
generic if both generic if both		generic if both
local and global local and global		local and global
flags are flags are		flags are
specified. specified.		specified.
- Could be split into - Could be split into		- Could be split into
separate s_waitcnt separate s_waitcnt		separate s_waitcnt
vmcnt(0) and vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
s_waitcnt vscnt(0) and s_waitcnt		vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be them to be		them to be
independently moved independently moved		independently moved
according to the according to the		according to the
following rules. following rules.		following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after must happen after		must happen after
any preceding any preceding		any preceding
global/generic global/generic		global/generic
load/store/load load/load atomic/		load/load atomic/
atomic/store atomicrmw-with-return-value.		atomicrmw-with-return-value.
atomic/atomicrmw.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store atomic/		store/store atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after must happen after		must happen after
any preceding any preceding		any preceding
local/generic local/generic		local/generic
load/store/load load/store/load		load/store/load
atomic/store atomic/store		atomic/store
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before - Must happen before		- Must happen before
any following store any following store		any following store
atomic/atomicrmw atomic/atomicrmw		atomic/atomicrmw
with an equal or with an equal or		with an equal or
wider sync scope wider sync scope		wider sync scope
and memory ordering and memory ordering		and memory ordering
stronger than stronger than		stronger than
unordered (this is unordered (this is		unordered (this is
termed the termed the		termed the
fence-paired-atomic). fence-paired-atomic).		fence-paired-atomic).
- Ensures that all - Ensures that all		- Ensures that all
memory operations memory operations		memory operations
have have		have
completed before completed before		completed before
performing the performing the		performing the
following following		following
fence-paired-atomic. fence-paired-atomic.		fence-paired-atomic.

Acquire-Release Atomic		Acquire-Release Atomic
---------------------------------------------------------------------------------------------------------------------		------------------------------------------------------------------------------------
atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic		atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic
- wavefront - local		- wavefront - local
- generic		- generic
atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &		atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0)		vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit vmcnt(0) and		mode, omit vmcnt(0) and
vscnt(0).		vscnt(0).
- If OpenCL, omit. - If OpenCL, omit		- If OpenCL, omit
lgkmcnt(0).		lgkmcnt(0).
- Must happen after - Must happen after		- Must happen after
any preceding any preceding		any preceding
local/generic local/generic		local/generic
load/store/load load/store/load		load/store/load
atomic/store atomic/store		atomic/store
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
vscnt(0), and s_waitcnt		vscnt(0), and s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic load/load		global/generic load/load
atomic/		atomic/
atomicrmw-with-return-value.		atomicrmw-with-return-value.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store		store/store
atomic/		atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after		must happen after
any preceding		any preceding
local/generic		local/generic
load/store/load		load/store/load
atomic/store		atomic/store
atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before - Must happen before		- Must happen before
the following the following		the following
atomicrmw. atomicrmw.		atomicrmw.
- Ensures that all - Ensures that all		- Ensures that all
memory operations memory operations		memory operations
to local have have		have
completed before completed before		completed before
performing the performing the		performing the
atomicrmw that is atomicrmw that is		atomicrmw that is
being released. being released.		being released.

2. buffer/global_atomic 2. buffer/global_atomic		2. buffer/global_atomic
3. s_waitcnt vm/vscnt(0)		3. s_waitcnt vm/vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- Use vmcnt(0) if atomic with		- Use vmcnt(0) if atomic with
return and vscnt(0) if		return and vscnt(0) if
atomic with no-return.		atomic with no-return.
- Must happen before		- Must happen before
the following		the following
buffer_gl0_inv.		buffer_gl0_inv.
- Ensures any		- Ensures any
following global		following global
data read is no		data read is no
older than the		older than the
atomicrmw value		atomicrmw value
being acquired.		being acquired.

4. buffer_gl0_inv		4. buffer_gl0_inv

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- Ensures that		- Ensures that
following		following
loads will not see		loads will not see
stale data.		stale data.

atomicrmw acq_rel - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0)		atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- If OpenCL, omit.		- If OpenCL, omit.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0) and s_waitcnt		vmcnt(0) and s_waitcnt
vscnt(0) to allow		vscnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic load/load		global/generic load/load
atomic/		atomic/
atomicrmw-with-return-value.		atomicrmw-with-return-value.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store atomic/		store/store atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- Must happen before		- Must happen before
the following		the following
store.		store.
- Ensures that all		- Ensures that all
global memory		global memory
operations have		operations have
completed before		completed before
performing the		performing the
store that is being		store that is being
released.		released.

1. ds_atomic 2. ds_atomic		2. ds_atomic
2. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0)		3. s_waitcnt lgkmcnt(0)

- If OpenCL, omit. - If OpenCL, omit.		- If OpenCL, omit.
- Must happen before - Must happen before		- Must happen before
any following the following		the following
global/generic buffer_gl0_inv.		buffer_gl0_inv.
load/load		- Ensures any
atomic/store/store		following global
atomic/atomicrmw.		data read is no
- Ensures any - Ensures any		older than the local load
following global following global		atomic value being
data read is no data read is no		acquired.
older than the load older than the load
atomic value being atomic value being
acquired. acquired.

4. buffer_gl0_inv		4. buffer_gl0_inv

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- If OpenCL omit.		- If OpenCL omit.
- Ensures that		- Ensures that
following		following
loads will not see		loads will not see
stale data.		stale data.

atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &		atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0)		vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit vmcnt(0) and		mode, omit vmcnt(0) and
vscnt(0).		vscnt(0).
- If OpenCL, omit. - If OpenCL, omit lgkmcnt(0).		- If OpenCL, omit lgkmcnt(0).
- Must happen after
any preceding
local/generic
load/store/load
atomic/store
atomic/atomicrmw.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt		vscnt(0) and s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic load/load		global/generic load/load
atomic/		atomic/
atomicrmw-with-return-value.		atomicrmw-with-return-value.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store		store/store
atomic/		atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after		must happen after
any preceding		any preceding
local/generic		local/generic
load/store/load		load/store/load
atomic/store		atomic/store
atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before - Must happen before		- Must happen before
the following the following		the following
atomicrmw. atomicrmw.		atomicrmw.
- Ensures that all - Ensures that all		- Ensures that all
memory operations memory operations		memory operations
to local have have		have
completed before completed before		completed before
performing the performing the		performing the
atomicrmw that is atomicrmw that is		atomicrmw that is
being released. being released.		being released.

2. flat_atomic 2. flat_atomic		2. flat_atomic
3. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) &		3. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0)		vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit vmcnt(0) and		mode, omit vmcnt(0) and
vscnt(0).		vscnt(0).
- If OpenCL, omit. - If OpenCL, omit lgkmcnt(0).		- If OpenCL, omit lgkmcnt(0).
- Must happen before - Must happen before		- Must happen before
any following the following		the following
global/generic buffer_gl0_inv.		buffer_gl0_inv.
load/load		- Ensures any
atomic/store/store		following global
atomic/atomicrmw.		data read is no
- Ensures any - Ensures any		older than the load
following global following global		atomic value being
data read is no data read is no		acquired.
older than the load older than the load
atomic value being atomic value being
acquired. acquired.

3. buffer_gl0_inv		3. buffer_gl0_inv

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- Ensures that		- Ensures that
following		following
loads will not see		loads will not see
stale data.		stale data.

atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &		atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) &
- system vmcnt(0) vmcnt(0) & vscnt(0)		- system vmcnt(0) & vscnt(0)

- If OpenCL, omit - If OpenCL, omit		- If OpenCL, omit
lgkmcnt(0). lgkmcnt(0).		lgkmcnt(0).
- Could be split into - Could be split into		- Could be split into
separate s_waitcnt separate s_waitcnt		separate s_waitcnt
vmcnt(0) and vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
s_waitcnt vscnt(0) and s_waitcnt		vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be them to be		them to be
independently moved independently moved		independently moved
according to the according to the		according to the
following rules. following rules.		following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after must happen after		must happen after
any preceding any preceding		any preceding
global/generic global/generic		global/generic
load/store/load load/load atomic/		load/load atomic/
atomic/store atomicrmw-with-return-value.		atomicrmw-with-return-value.
atomic/atomicrmw.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store atomic/		store/store atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after must happen after		must happen after
any preceding any preceding		any preceding
local/generic local/generic		local/generic
load/store/load load/store/load		load/store/load
atomic/store atomic/store		atomic/store
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before - Must happen before		- Must happen before
the following the following		the following
atomicrmw. atomicrmw.		atomicrmw.
- Ensures that all - Ensures that all		- Ensures that all
memory operations memory operations		memory operations
to global have to global have		to global have
completed before completed before		completed before
performing the performing the		performing the
atomicrmw that is atomicrmw that is		atomicrmw that is
being released. being released.		being released.

2. buffer/global_atomic 2. buffer/global_atomic		2. buffer/global_atomic
3. s_waitcnt vmcnt(0) 3. s_waitcnt vm/vscnt(0)		3. s_waitcnt vm/vscnt(0)

- Use vmcnt(0) if atomic with		- Use vmcnt(0) if atomic with
return and vscnt(0) if		return and vscnt(0) if
atomic with no-return.		atomic with no-return.
waitcnt lgkmcnt(0).		- Must happen before
- Must happen before - Must happen before		following
following following		buffer_gl*_inv.
buffer_wbinvl1_vol. buffer_gl*_inv.		- Ensures the
- Ensures the - Ensures the		atomicrmw has
atomicrmw has atomicrmw has		completed before
completed before completed before		invalidating the
invalidating the invalidating the		caches.
cache. caches.

4. buffer_wbinvl1_vol 4. buffer_gl0_inv;		4. buffer_gl0_inv;
buffer_gl1_inv		buffer_gl1_inv

- Must happen before - Must happen before		- Must happen before
any following any following		any following
global/generic global/generic		global/generic
load/load load/load		load/load
atomic/atomicrmw. atomic/atomicrmw.
- Ensures that - Ensures that
following loads following loads
will not see stale will not see stale
global data. global data.

atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &
- system vmcnt(0) vmcnt(0) & vscnt(0)

- If OpenCL, omit - If OpenCL, omit
lgkmcnt(0). lgkmcnt(0).
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0) and vmcnt(0), s_waitcnt
s_waitcnt vscnt(0), and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
load/store/load load/load atomic
atomic/store atomicrmw-with-return-value.
atomic/atomicrmw.		atomic/atomicrmw.
		- Ensures that
		following loads
		will not see stale
		global data.

		atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) &
		- system vmcnt(0) & vscnt(0)

		- If OpenCL, omit
		lgkmcnt(0).
		- Could be split into
		separate s_waitcnt
		vmcnt(0), s_waitcnt
		vscnt(0), and s_waitcnt
		lgkmcnt(0) to allow
		them to be
		independently moved
		according to the
		following rules.
		- s_waitcnt vmcnt(0)
		must happen after
		any preceding
		global/generic
		load/load atomic
		atomicrmw-with-return-value.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store atomic/		store/store atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after must happen after		must happen after
any preceding any preceding		any preceding
local/generic local/generic		local/generic
load/store/load load/store/load		load/store/load
atomic/store atomic/store		atomic/store
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before - Must happen before		- Must happen before
the following the following		the following
atomicrmw. atomicrmw.		atomicrmw.
- Ensures that all - Ensures that all		- Ensures that all
memory operations memory operations		memory operations
to global have have		have
completed before completed before		completed before
performing the performing the		performing the
atomicrmw that is atomicrmw that is		atomicrmw that is
being released. being released.		being released.

2. flat_atomic 2. flat_atomic
3. s_waitcnt vmcnt(0) & 3. s_waitcnt vm/vscnt(0) &
lgkmcnt(0) lgkmcnt(0)

- If OpenCL, omit - If OpenCL, omit		2. flat_atomic
lgkmcnt(0). lgkmcnt(0).		3. s_waitcnt vm/vscnt(0) &
		lgkmcnt(0)

		- If OpenCL, omit
		lgkmcnt(0).
- Use vmcnt(0) if atomic with		- Use vmcnt(0) if atomic with
return and vscnt(0) if		return and vscnt(0) if
atomic with no-return.		atomic with no-return.
- Must happen before - Must happen before		- Must happen before
following following		following
buffer_wbinvl1_vol. buffer_gl*_inv.		buffer_gl*_inv.
- Ensures the - Ensures the		- Ensures the
atomicrmw has atomicrmw has		atomicrmw has
completed before completed before		completed before
invalidating the invalidating the		invalidating the
cache. caches.		caches.

4. buffer_wbinvl1_vol 4. buffer_gl0_inv;		4. buffer_gl0_inv;
buffer_gl1_inv		buffer_gl1_inv

- Must happen before - Must happen before		- Must happen before
any following any following		any following
global/generic global/generic		global/generic
load/load load/load		load/load
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Ensures that - Ensures that		- Ensures that
following loads following loads		following loads
will not see stale will not see stale		will not see stale
global data. global data.		global data.

fence acq_rel - singlethread none none none		fence acq_rel - singlethread none none
- wavefront		- wavefront
fence acq_rel - workgroup none 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &		fence acq_rel - workgroup none 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0)		vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit vmcnt(0) and		mode, omit vmcnt(0) and
vscnt(0).		vscnt(0).
- If OpenCL and - If OpenCL and		- If OpenCL and
address space is address space is		address space is
not generic, omit. not generic, omit		not generic, omit
lgkmcnt(0).		lgkmcnt(0).
- If OpenCL and		- If OpenCL and
address space is		address space is
local, omit		local, omit
vmcnt(0) and vscnt(0).		vmcnt(0) and vscnt(0).
- However, - However,		- However,
since LLVM since LLVM		since LLVM
currently has no currently has no		currently has no
address space on address space on		address space on
the fence need to the fence need to		the fence need to
conservatively conservatively		conservatively
always generate always generate		always generate
(see comment for (see comment for		(see comment for
previous fence). previous fence).		previous fence).
- Must happen after
any preceding
local/generic
load/load
atomic/store/store
atomic/atomicrmw.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt		vscnt(0) and s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
load/load		load/load
atomic/		atomic/
atomicrmw-with-return-value.		atomicrmw-with-return-value.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store atomic/		store/store atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after		must happen after
any preceding		any preceding
local/generic		local/generic
load/store/load		load/store/load
atomic/store atomic/		atomic/store atomic/
atomicrmw.		atomicrmw.
- Must happen before - Must happen before		- Must happen before
any following any following		any following
global/generic global/generic		global/generic
load/load load/load		load/load
atomic/store/store atomic/store/store		atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Ensures that all - Ensures that all		- Ensures that all
memory operations memory operations		memory operations
to local have have		have
completed before completed before		completed before
performing any performing any		performing any
following global following global		following global
memory operations. memory operations.		memory operations.
- Ensures that the - Ensures that the		- Ensures that the
preceding preceding		preceding
local/generic load local/generic load		local/generic load
atomic/atomicrmw atomic/atomicrmw		atomic/atomicrmw
with an equal or with an equal or		with an equal or
wider sync scope wider sync scope		wider sync scope
and memory ordering and memory ordering		and memory ordering
stronger than stronger than		stronger than
unordered (this is unordered (this is		unordered (this is
termed the termed the		termed the
acquire-fence-paired-atomic acquire-fence-paired-atomic		acquire-fence-paired-atomic)
) has completed ) has completed		has completed
before following before following		before following
global memory global memory		global memory
operations. This operations. This		operations. This
satisfies the satisfies the		satisfies the
requirements of requirements of		requirements of
acquire. acquire.		acquire.
- Ensures that all - Ensures that all		- Ensures that all
previous memory previous memory		previous memory
operations have operations have		operations have
completed before a completed before a		completed before a
following following		following
local/generic store local/generic store		local/generic store
atomic/atomicrmw atomic/atomicrmw		atomic/atomicrmw
with an equal or with an equal or		with an equal or
wider sync scope wider sync scope		wider sync scope
and memory ordering and memory ordering		and memory ordering
stronger than stronger than		stronger than
unordered (this is unordered (this is		unordered (this is
termed the termed the		termed the
release-fence-paired-atomic release-fence-paired-atomic		release-fence-paired-atomic).
). This satisfies the ). This satisfies the		This satisfies the
requirements of requirements of		requirements of
release. release.		release.
- Must happen before		- Must happen before
the following		the following
buffer_gl0_inv.		buffer_gl0_inv.
- Ensures that the		- Ensures that the
acquire-fence-paired		acquire-fence-paired
atomic has completed		atomic has completed
before invalidating		before invalidating
the		the
cache. Therefore		cache. Therefore
any following		any following
locations read must		locations read must
be no older than		be no older than
the value read by		the value read by
the		the
acquire-fence-paired-atomic.		acquire-fence-paired-atomic.

3. buffer_gl0_inv		3. buffer_gl0_inv

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- Ensures that		- Ensures that
following		following
loads will not see		loads will not see
stale data.		stale data.

fence acq_rel - agent none 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &		fence acq_rel - agent none 1. s_waitcnt lgkmcnt(0) &
- system vmcnt(0) vmcnt(0) & vscnt(0)		- system vmcnt(0) & vscnt(0)

- If OpenCL and - If OpenCL and		- If OpenCL and
address space is address space is		address space is
not generic, omit not generic, omit		not generic, omit
lgkmcnt(0). lgkmcnt(0).		lgkmcnt(0).
- If OpenCL and		- If OpenCL and
address space is		address space is
local, omit		local, omit
vmcnt(0) and vscnt(0).		vmcnt(0) and vscnt(0).
- However, since LLVM - However, since LLVM		- However, since LLVM
currently has no currently has no		currently has no
address space on address space on		address space on
the fence need to the fence need to		the fence need to
conservatively conservatively		conservatively
always generate always generate		always generate
(see comment for (see comment for		(see comment for
previous fence). previous fence).		previous fence).
- Could be split into - Could be split into		- Could be split into
separate s_waitcnt separate s_waitcnt		separate s_waitcnt
vmcnt(0) and vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
s_waitcnt vscnt(0) and s_waitcnt		vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be them to be		them to be
independently moved independently moved		independently moved
according to the according to the		according to the
following rules. following rules.		following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after must happen after		must happen after
any preceding any preceding		any preceding
global/generic global/generic		global/generic
load/store/load load/load		load/load
atomic/store atomic/		atomic/
atomic/atomicrmw. atomicrmw-with-return-value.		atomicrmw-with-return-value.
- s_waitcnt vscnt(0)		- s_waitcnt vscnt(0)
must happen after		must happen after
any preceding		any preceding
global/generic		global/generic
store/store atomic/		store/store atomic/
atomicrmw-no-return-value.		atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after must happen after		must happen after
any preceding any preceding		any preceding
local/generic local/generic		local/generic
load/store/load load/store/load		load/store/load
atomic/store atomic/store		atomic/store
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Must happen before - Must happen before		- Must happen before
the following the following		the following
buffer_wbinvl1_vol. buffer_gl*_inv.		buffer_gl*_inv.
- Ensures that the - Ensures that the		- Ensures that the
preceding preceding		preceding
global/local/generic global/local/generic		global/local/generic
load load		load
atomic/atomicrmw atomic/atomicrmw		atomic/atomicrmw
with an equal or with an equal or		with an equal or
wider sync scope wider sync scope		wider sync scope
and memory ordering and memory ordering		and memory ordering
stronger than stronger than		stronger than
unordered (this is unordered (this is		unordered (this is
termed the termed the		termed the
acquire-fence-paired-atomic acquire-fence-paired-atomic		acquire-fence-paired-atomic)
) has completed ) has completed		has completed
before invalidating before invalidating		before invalidating
the cache. This the caches. This		the caches. This
satisfies the satisfies the		satisfies the
requirements of requirements of		requirements of
acquire. acquire.		acquire.
- Ensures that all - Ensures that all		- Ensures that all
previous memory previous memory		previous memory
operations have operations have		operations have
completed before a completed before a		completed before a
following following		following
global/local/generic global/local/generic		global/local/generic
store store		store
atomic/atomicrmw atomic/atomicrmw		atomic/atomicrmw
with an equal or with an equal or		with an equal or
wider sync scope wider sync scope		wider sync scope
and memory ordering and memory ordering		and memory ordering
stronger than stronger than		stronger than
unordered (this is unordered (this is		unordered (this is
termed the termed the		termed the
release-fence-paired-atomic release-fence-paired-atomic		release-fence-paired-atomic).
). This satisfies the ). This satisfies the		This satisfies the
requirements of requirements of		requirements of
release. release.		release.

2. buffer_wbinvl1_vol 2. buffer_gl0_inv;		2. buffer_gl0_inv;
buffer_gl1_inv		buffer_gl1_inv

- Must happen before - Must happen before		- Must happen before
any following any following		any following
global/generic global/generic		global/generic
load/load load/load		load/load
atomic/store/store atomic/store/store		atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.		atomic/atomicrmw.
- Ensures that - Ensures that		- Ensures that
following loads following loads		following loads
will not see stale will not see stale		will not see stale
global data. This global data. This		global data. This
satisfies the satisfies the		satisfies the
requirements of requirements of		requirements of
acquire. acquire.		acquire.

Sequential Consistent Atomic		Sequential Consistent Atomic
---------------------------------------------------------------------------------------------------------------------		------------------------------------------------------------------------------------
load atomic seq_cst - singlethread - global Same as corresponding Same as corresponding		load atomic seq_cst - singlethread - global *Same as corresponding
- wavefront - local load atomic acquire, load atomic acquire,		- wavefront - local load atomic acquire,
- generic except must generated except must generated		- generic except must generated
all instructions even all instructions even		all instructions even
for OpenCL.* for OpenCL.*		for OpenCL.*
load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &		load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) &
- generic vmcnt(0) & vscnt(0)		- generic vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit vmcnt(0) and		mode, omit vmcnt(0) and
vscnt(0).		vscnt(0).
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
vscnt(0), and s_waitcnt		vscnt(0), and s_waitcnt
lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- Must - waitcnt lgkmcnt(0) must		- s_waitcnt lgkmcnt(0) must
happen after happen after		happen after
preceding preceding		preceding
global/generic load local load		local/generic load
atomic/store atomic/store		atomic/store
atomic/atomicrmw atomic/atomicrmw		atomic/atomicrmw
with memory with memory		with memory
ordering of seq_cst ordering of seq_cst		ordering of seq_cst
and with equal or and with equal or		and with equal or
wider sync scope. wider sync scope.		wider sync scope.
(Note that seq_cst (Note that seq_cst		(Note that seq_cst
fences have their fences have their		fences have their
own s_waitcnt own s_waitcnt		own s_waitcnt
lgkmcnt(0) and so do lgkmcnt(0) and so do		lgkmcnt(0) and so do
not need to be not need to be		not need to be
considered.) considered.)		considered.)
- waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
Must happen after		must happen after
preceding		preceding
global/generic load		global/generic load
atomic/		atomic/
atomicrmw-with-return-value		atomicrmw-with-return-value
with memory		with memory
ordering of seq_cst		ordering of seq_cst
and with equal or		and with equal or
wider sync scope.		wider sync scope.
(Note that seq_cst		(Note that seq_cst
fences have their		fences have their
own s_waitcnt		own s_waitcnt
vmcnt(0) and so do		vmcnt(0) and so do
not need to be		not need to be
considered.)		considered.)
- waitcnt vscnt(0)		- s_waitcnt vscnt(0)
Must happen after		Must happen after
preceding		preceding
global/generic store		global/generic store
atomic/		atomic/
atomicrmw-no-return-value		atomicrmw-no-return-value
with memory		with memory
ordering of seq_cst		ordering of seq_cst
and with equal or		and with equal or
wider sync scope.		wider sync scope.
(Note that seq_cst		(Note that seq_cst
fences have their		fences have their
own s_waitcnt		own s_waitcnt
vscnt(0) and so do		vscnt(0) and so do
not need to be		not need to be
considered.)		considered.)
- Ensures any - Ensures any		- Ensures any
preceding preceding		preceding
sequential sequential		sequential
consistent local consistent global/local		consistent global/local
memory instructions memory instructions		memory instructions
have completed have completed		have completed
before executing before executing		before executing
this sequentially this sequentially		this sequentially
consistent consistent		consistent
instruction. This instruction. This		instruction. This
prevents reordering prevents reordering		prevents reordering
a seq_cst store a seq_cst store		a seq_cst store
followed by a followed by a		followed by a
seq_cst load. (Note seq_cst load. (Note		seq_cst load. (Note
that seq_cst is that seq_cst is		that seq_cst is
stronger than stronger than		stronger than
acquire/release as acquire/release as		acquire/release as
the reordering of the reordering of		the reordering of
load acquire load acquire		load acquire
followed by a store followed by a store		followed by a store
release is release is		release is
prevented by the prevented by the		prevented by the
waitcnt of waitcnt of		s_waitcnt of
the release, but the release, but		the release, but
there is nothing there is nothing		there is nothing
preventing a store preventing a store		preventing a store
release followed by release followed by		release followed by
load acquire from load acquire from		load acquire from
completing out of completing out of		completing out of
order. The waitcnt order. The waitcnt		order. The s_waitcnt
could be placed after could be placed after		could be placed after
seq_store or before seq_store or before		seq_store or before
the seq_load. We the seq_load. We		the seq_load. We
choose the load to choose the load to		choose the load to
make the waitcnt be make the waitcnt be		make the s_waitcnt be
as late as possible as late as possible		as late as possible
so that the store so that the store		so that the store
may have already may have already		may have already
completed.) completed.)		completed.)

2. Following 2. Following		2. *Following
instructions same as instructions same as		instructions same as
corresponding load corresponding load		corresponding load
atomic acquire, atomic acquire,		atomic acquire,
except must generated except must generated
all instructions even all instructions even
for OpenCL.* for OpenCL.*
load atomic seq_cst - workgroup - local *Same as corresponding
load atomic acquire,
except must generated		except must generated
all instructions even		all instructions even
for OpenCL.*		for OpenCL.*
		load atomic seq_cst - workgroup - local

1. s_waitcnt vmcnt(0) & vscnt(0)		1. s_waitcnt vmcnt(0) & vscnt(0)

- If CU wavefront execution		- If CU wavefront execution
mode, omit.		mode, omit.
- Could be split into		- Could be split into
separate s_waitcnt		separate s_waitcnt
vmcnt(0) and s_waitcnt		vmcnt(0) and s_waitcnt
vscnt(0) to allow		vscnt(0) to allow
them to be		them to be
independently moved		independently moved
according to the		according to the
following rules.		following rules.
- waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
Must happen after		Must happen after
preceding		preceding
global/generic load		global/generic load
atomic/		atomic/
atomicrmw-with-return-value		atomicrmw-with-return-value
with memory		with memory
ordering of seq_cst		ordering of seq_cst
and with equal or		and with equal or
wider sync scope.		wider sync scope.
(Note that seq_cst		(Note that seq_cst
fences have their		fences have their
own s_waitcnt		own s_waitcnt
vmcnt(0) and so do		vmcnt(0) and so do
not need to be		not need to be
considered.)		considered.)
- waitcnt vscnt(0)		- s_waitcnt vscnt(0)
Must happen after		Must happen after
preceding		preceding
global/generic store		global/generic store
atomic/		atomic/
atomicrmw-no-return-value		atomicrmw-no-return-value
with memory		with memory
ordering of seq_cst		ordering of seq_cst
and with equal or		and with equal or
wider sync scope.		wider sync scope.
(Note that seq_cst		(Note that seq_cst
fences have their		fences have their
own s_waitcnt		own s_waitcnt
vscnt(0) and so do		vscnt(0) and so do
not need to be		not need to be
considered.)		considered.)
- Ensures any		- Ensures any
preceding		preceding
sequential		sequential
consistent global		consistent global
memory instructions		memory instructions
have completed		have completed
before executing		before executing
this sequentially		this sequentially
consistent		consistent
instruction. This		instruction. This
prevents reordering		prevents reordering
a seq_cst store		a seq_cst store
followed by a		followed by a
seq_cst load. (Note		seq_cst load. (Note
that seq_cst is		that seq_cst is
stronger than		stronger than
acquire/release as		acquire/release as
the reordering of		the reordering of
load acquire		load acquire
followed by a store		followed by a store
release is		release is
prevented by the		prevented by the
waitcnt of		s_waitcnt of
the release, but		the release, but
there is nothing		there is nothing
preventing a store		preventing a store
release followed by		release followed by
load acquire from		load acquire from
completing out of		completing out of
order. The waitcnt		order. The s_waitcnt
could be placed after		could be placed after
seq_store or before		seq_store or before
the seq_load. We		the seq_load. We
choose the load to		choose the load to
make the waitcnt be		make the s_waitcnt be
as late as possible		as late as possible
so that the store		so that the store
may have already		may have already
completed.)		completed.)

2. *Following		2. *Following
instructions same as		instructions same as
corresponding load		corresponding load
atomic acquire,		atomic acquire,
except must generated		except must generated
all instructions even		all instructions even
for OpenCL.*		for OpenCL.*

load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &		load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) &
- system - generic vmcnt(0) vmcnt(0) & vscnt(0)		- system - generic vmcnt(0) & vscnt(0)

- Could be split into - Could be split into		- Could be split into
separate s_waitcnt separate s_waitcnt		separate s_waitcnt
vmcnt(0) vmcnt(0), s_waitcnt		vmcnt(0), s_waitcnt
and s_waitcnt vscnt(0) and s_waitcnt		vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow		lgkmcnt(0) to allow
them to be them to be		them to be
independently moved independently moved		independently moved
according to the according to the		according to the
following rules. following rules.		following rules.
- waitcnt lgkmcnt(0) - waitcnt lgkmcnt(0)		- s_waitcnt lgkmcnt(0)
must happen after must happen after		must happen after
preceding preceding		preceding
global/generic load local load		local load
atomic/store atomic/store		atomic/store
atomic/atomicrmw atomic/atomicrmw		atomic/atomicrmw
with memory with memory		with memory
ordering of seq_cst ordering of seq_cst		ordering of seq_cst
and with equal or and with equal or		and with equal or
wider sync scope. wider sync scope.		wider sync scope.
(Note that seq_cst (Note that seq_cst		(Note that seq_cst
fences have their fences have their		fences have their
own s_waitcnt own s_waitcnt		own s_waitcnt
lgkmcnt(0) and so do lgkmcnt(0) and so do		lgkmcnt(0) and so do
not need to be not need to be		not need to be
considered.) considered.)		considered.)
- waitcnt vmcnt(0) - waitcnt vmcnt(0)		- s_waitcnt vmcnt(0)
must happen after must happen after		must happen after
preceding preceding		preceding
global/generic load global/generic load		global/generic load
atomic/store atomic/		atomic/
atomic/atomicrmw atomicrmw-with-return-value		atomicrmw-with-return-value
with memory with memory		with memory
ordering of seq_cst ordering of seq_cst		ordering of seq_cst
and with equal or and with equal or		and with equal or
wider sync scope. wider sync scope.		wider sync scope.
(Note that seq_cst (Note that seq_cst		(Note that seq_cst
fences have their fences have their		fences have their
own s_waitcnt own s_waitcnt		own s_waitcnt
vmcnt(0) and so do vmcnt(0) and so do		vmcnt(0) and so do
not need to be not need to be		not need to be
considered.) considered.)		considered.)
- waitcnt vscnt(0)		- s_waitcnt vscnt(0)
Must happen after		Must happen after
preceding		preceding
global/generic store		global/generic store
atomic/		atomic/
atomicrmw-no-return-value		atomicrmw-no-return-value
with memory		with memory
ordering of seq_cst		ordering of seq_cst
and with equal or		and with equal or
wider sync scope.		wider sync scope.
(Note that seq_cst		(Note that seq_cst
fences have their		fences have their
own s_waitcnt		own s_waitcnt
vscnt(0) and so do		vscnt(0) and so do
not need to be		not need to be
considered.)		considered.)
- Ensures any - Ensures any		- Ensures any
preceding preceding		preceding
sequential sequential		sequential
consistent global consistent global		consistent global
memory instructions memory instructions		memory instructions
have completed have completed		have completed
before executing before executing		before executing
this sequentially this sequentially		this sequentially
consistent consistent		consistent
instruction. This instruction. This		instruction. This
prevents reordering prevents reordering		prevents reordering
a seq_cst store a seq_cst store		a seq_cst store
followed by a followed by a		followed by a
seq_cst load. (Note seq_cst load. (Note		seq_cst load. (Note
that seq_cst is that seq_cst is		that seq_cst is
stronger than stronger than		stronger than
acquire/release as acquire/release as		acquire/release as
the reordering of the reordering of		the reordering of
load acquire load acquire		load acquire
followed by a store followed by a store		followed by a store
release is release is		release is
prevented by the prevented by the		prevented by the
waitcnt of waitcnt of		s_waitcnt of
the release, but the release, but		the release, but
there is nothing there is nothing		there is nothing
preventing a store preventing a store		preventing a store
release followed by release followed by		release followed by
load acquire from load acquire from		load acquire from
completing out of completing out of		completing out of
order. The waitcnt order. The waitcnt		order. The s_waitcnt
could be placed after could be placed after		could be placed after
seq_store or before seq_store or before		seq_store or before
the seq_load. We the seq_load. We		the seq_load. We
choose the load to choose the load to		choose the load to
make the waitcnt be make the waitcnt be		make the s_waitcnt be
as late as possible as late as possible		as late as possible
so that the store so that the store		so that the store
may have already may have already		may have already
completed.) completed.)		completed.)

2. Following 2. Following
instructions same as instructions same as
corresponding load corresponding load
atomic acquire, atomic acquire,
except must generated except must generated
all instructions even all instructions even
for OpenCL.* for OpenCL.*
store atomic seq_cst - singlethread - global Same as corresponding Same as corresponding
- wavefront - local store atomic release, store atomic release,
- workgroup - generic except must generated except must generated
all instructions even all instructions even
for OpenCL.* for OpenCL.*
store atomic seq_cst - agent - global Same as corresponding Same as corresponding
- system - generic store atomic release, store atomic release,
except must generated except must generated
all instructions even all instructions even
for OpenCL.* for OpenCL.*
atomicrmw seq_cst - singlethread - global Same as corresponding Same as corresponding
- wavefront - local atomicrmw acq_rel, atomicrmw acq_rel,
- workgroup - generic except must generated except must generated
all instructions even all instructions even
for OpenCL.* for OpenCL.*
atomicrmw seq_cst - agent - global Same as corresponding Same as corresponding
- system - generic atomicrmw acq_rel, atomicrmw acq_rel,
except must generated except must generated
all instructions even all instructions even
for OpenCL.* for OpenCL.*
fence seq_cst - singlethread none Same as corresponding Same as corresponding
- wavefront fence acq_rel, fence acq_rel,
- workgroup except must generated except must generated
- agent all instructions even all instructions even
- system for OpenCL.* for OpenCL.*
============ ============ ============== ========== ================================ ================================

The memory order also adds the single thread optimization constrains defined in
table
:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table`.

.. table:: AMDHSA Memory Model Single Thread Optimization Constraints GFX6-GFX10
:name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table

============ ==============================================================		2. *Following
LLVM Memory Optimization Constraints		instructions same as
Ordering		corresponding load
============ ==============================================================		atomic acquire,
unordered none		except must generated
monotonic none		all instructions even
acquire - If a load atomic/atomicrmw then no following load/load		for OpenCL.*
atomic/store/ store atomic/atomicrmw/fence instruction can		store atomic seq_cst - singlethread - global *Same as corresponding
be moved before the acquire.		- wavefront - local store atomic release,
- If a fence then same as load atomic, plus no preceding		- workgroup - generic except must generated
associated fence-paired-atomic can be moved after the fence.		- agent all instructions even
release - If a store atomic/atomicrmw then no preceding load/load		- system for OpenCL.*
atomic/store/ store atomic/atomicrmw/fence instruction can		atomicrmw seq_cst - singlethread - global *Same as corresponding
be moved after the release.		- wavefront - local atomicrmw acq_rel,
- If a fence then same as store atomic, plus no following		- workgroup - generic except must generated
associated fence-paired-atomic can be moved before the		- agent all instructions even
fence.		- system for OpenCL.*
acq_rel Same constraints as both acquire and release.		fence seq_cst - singlethread none *Same as corresponding
seq_cst - If a load atomic then same constraints as acquire, plus no		- wavefront fence acq_rel,
preceding sequentially consistent load atomic/store		- workgroup except must generated
atomic/atomicrmw/fence instruction can be moved after the		- agent all instructions even
seq_cst.		- system for OpenCL.*
- If a store atomic then the same constraints as release, plus		============ ============ ============== ========== ================================
no following sequentially consistent load atomic/store
atomic/atomicrmw/fence instruction can be moved before the
seq_cst.
- If an atomicrmw/fence then same constraints as acq_rel.
============ ==============================================================

Trap Handler ABI		Trap Handler ABI
~~~~~~~~~~~~~~~~		~~~~~~~~~~~~~~~~

For code objects generated by AMDGPU backend for HSA [HSA]_ compatible runtimes		For code objects generated by AMDGPU backend for HSA [HSA]_ compatible runtimes
(such as ROCm [AMD-ROCm]_), the runtime installs a trap handler that supports		(such as ROCm [AMD-ROCm]_), the runtime installs a trap handler that supports
the ``s_trap`` instruction with the following usage:		the ``s_trap`` instruction with the following usage:

▲ Show 20 Lines • Show All 148 Lines • ▼ Show 20 Lines	8. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch
the function may dynamically allocate space for such things as runtime sized		the function may dynamically allocate space for such things as runtime sized
``alloca`` local allocations.		``alloca`` local allocations.

If the function calls another function, it will place any stack allocated		If the function calls another function, it will place any stack allocated
arguments after the last local allocation and adjust SGPR32 to the address		arguments after the last local allocation and adjust SGPR32 to the address
after the last local allocation.		after the last local allocation.

9. All other registers are unspecified.		9. All other registers are unspecified.
10. Any necessary ``waitcnt`` has been performed to ensure memory is available		10. Any necessary ``s_waitcnt`` has been performed to ensure memory is available
to the function.		to the function.

On exit from a function:		On exit from a function:

1. VGPR0-31 and SGPR4-29 are used to pass function result arguments as		1. VGPR0-31 and SGPR4-29 are used to pass function result arguments as
described below. Any registers used are considered clobbered registers.		described below. Any registers used are considered clobbered registers.
2. The following registers are preserved and have the same value as on entry:		2. The following registers are preserved and have the same value as on entry:

Show All 23 Lines	* VGPR40-47
For the AMDGPU backend, an inter-procedural register allocation (IPRA)		For the AMDGPU backend, an inter-procedural register allocation (IPRA)
optimization may mark some of clobbered SGPR and VGPR registers as		optimization may mark some of clobbered SGPR and VGPR registers as
preserved if it can be determined that the called function does not change		preserved if it can be determined that the called function does not change
their value.		their value.

2. The PC is set to the RA provided on entry.		2. The PC is set to the RA provided on entry.
3. MODE register: TBD.		3. MODE register: TBD.
4. All other registers are clobbered.		4. All other registers are clobbered.
5. Any necessary ``waitcnt`` has been performed to ensure memory accessed by		5. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by
function is available to the caller.		function is available to the caller.

.. TODO::		.. TODO::

- On gfx908 are all ACC registers clobbered?		- On gfx908 are all ACC registers clobbered?

- How are function results returned? The address of structured types is passed		- How are function results returned? The address of structured types is passed
by reference, but what about other types?		by reference, but what about other types?
▲ Show 20 Lines • Show All 1,232 Lines • Show Last 20 Lines