This is an archive of the discontinued LLVM Phabricator instance.

Differential D89977

[NFC][AMDGPU] Resize Memory Model columns in AMDGPUUsage.rst
ClosedPublic

Authored by scott.linder on Oct 22 2020, 11:06 AM.

Details

Reviewers
t-tye
kzhuravl
Commits
rG580f99bcff31: [NFC][AMDGPU] Resize Memory Model columns in AMDGPUUsage.rst
Summary

Make all of the "AMDGPU Machine Code GFX*" columns in the Memory Model
table a consistent width of 32-characters.

Best viewed with something like --word-diff

Diff Detail

Event Timeline

scott.linder created this revision.Oct 22 2020, 11:06 AM
Herald added a project: Restricted Project. · View Herald TranscriptOct 22 2020, 11:06 AM
Herald added subscribers: llvm-commits, kerbowa, jfb and 7 others. · View Herald Transcript
scott.linder requested review of this revision.Oct 22 2020, 11:06 AM
Herald added a subscriber: wdng. · View Herald TranscriptOct 22 2020, 11:06 AM
scott.linder added a parent revision: D89596: [AMDGPU] Update Memory Model in AMDGPUUsage.rst.Oct 22 2020, 11:07 AM
Harbormaster completed remote builds in B76067: Diff 300048.Oct 22 2020, 11:07 AM
scott.linder updated this revision to Diff 300077.Oct 22 2020, 12:37 PM

Rebase

Harbormaster completed remote builds in B76076: Diff 300077.Oct 22 2020, 1:22 PM
scott.linder added reviewers: t-tye, kzhuravl.Oct 27 2020, 1:42 PM
scott.linder added a comment.Oct 28 2020, 12:27 PM

I'll wait to rebase this again until the dependency is committed.

scott.linder updated this revision to Diff 301765.Oct 29 2020, 2:51 PM

Rebase

Harbormaster completed remote builds in B76987: Diff 301765.Oct 29 2020, 3:16 PM
This revision was not accepted when it landed; it landed in state Needs Review.Oct 29 2020, 4:10 PM
This revision was landed with ongoing or failed builds.
Closed by commit rG580f99bcff31: [NFC][AMDGPU] Resize Memory Model columns in AMDGPUUsage.rst (authored by scott.linder). · Explain Why
This revision was automatically updated to reflect the committed changes.
scott.linder added a commit: rG580f99bcff31: [NFC][AMDGPU] Resize Memory Model columns in AMDGPUUsage.rst.

Revision Contents

PathSize
llvm/
docs/
3916 lines

Diff 301782

llvm/docs/AMDGPUUsage.rst

  • This file is larger than 256 KB, so syntax highlighting is disabled by default.
Show First 20 Lines • Show All 4,386 Lines • ▼ Show 20 Lines
to invalidate the L2 cache. For GFX6-GFX9, this also causes it to be treated as to invalidate the L2 cache. For GFX6-GFX9, this also causes it to be treated as
non-volatile and so is not invalidated by ``*_vol``. On APU it is accessed as CC non-volatile and so is not invalidated by ``*_vol``. On APU it is accessed as CC
(cache coherent) and so the L2 cache will be coherent with the CPU and other (cache coherent) and so the L2 cache will be coherent with the CPU and other
agents. agents.
.. table:: AMDHSA Memory Model Code Sequences GFX6-GFX10 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX10
:name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table
============ ============ ============== ========== =============================== ================================== ============ ============ ============== ========== ================================ ================================
LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code AMDGPU Machine Code LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code AMDGPU Machine Code
Ordering Sync Scope Address GFX6-9 GFX10 Ordering Sync Scope Address GFX6-9 GFX10
Space Space
============ ============ ============== ========== =============================== ================================== ============ ============ ============== ========== ================================ ================================
**Non-Atomic** **Non-Atomic**
---------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------
load *none* *none* - global - !volatile & !nontemporal - !volatile & !nontemporal load *none* *none* - global - !volatile & !nontemporal - !volatile & !nontemporal
- generic - generic
- private 1. buffer/global/flat_load 1. buffer/global/flat_load - private 1. buffer/global/flat_load 1. buffer/global/flat_load
- constant - constant
- volatile & !nontemporal - volatile & !nontemporal - volatile & !nontemporal - volatile & !nontemporal
1. buffer/global/flat_load 1. buffer/global/flat_load 1. buffer/global/flat_load 1. buffer/global/flat_load
glc=1 glc=1 dlc=1 glc=1 glc=1 dlc=1
- nontemporal - nontemporal - nontemporal - nontemporal
1. buffer/global/flat_load 1. buffer/global/flat_load 1. buffer/global/flat_load 1. buffer/global/flat_load
glc=1 slc=1 slc=1 glc=1 slc=1 slc=1
load *none* *none* - local 1. ds_load 1. ds_load load *none* *none* - local 1. ds_load 1. ds_load
store *none* *none* - global - !nontemporal - !nontemporal store *none* *none* - global - !nontemporal - !nontemporal
- generic - generic
- private 1. buffer/global/flat_store 1. buffer/global/flat_store - private 1. buffer/global/flat_store 1. buffer/global/flat_store
- constant - constant
- nontemporal - nontemporal - nontemporal - nontemporal
1. buffer/global/flat_store 1. buffer/global/flat_store 1. buffer/global/flat_store 1. buffer/global/flat_store
glc=1 slc=1 slc=1 glc=1 slc=1 slc=1
store *none* *none* - local 1. ds_store 1. ds_store store *none* *none* - local 1. ds_store 1. ds_store
**Unordered Atomic** **Unordered Atomic**
---------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------
load atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*. load atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*.
store atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*. store atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*.
atomicrmw unordered *any* *any* *Same as monotonic *Same as monotonic atomicrmw unordered *any* *any* *Same as monotonic *Same as monotonic
atomic*. atomic*. atomic*. atomic*.
**Monotonic Atomic** **Monotonic Atomic**
---------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------
load atomic monotonic - singlethread - global 1. buffer/global/flat_load 1. buffer/global/flat_load load atomic monotonic - singlethread - global 1. buffer/global/flat_load 1. buffer/global/flat_load
- wavefront - generic - wavefront - generic
load atomic monotonic - workgroup - global 1. buffer/global/flat_load 1. buffer/global/flat_load load atomic monotonic - workgroup - global 1. buffer/global/flat_load 1. buffer/global/flat_load
- generic glc=1 - generic glc=1
- If CU wavefront execution - If CU wavefront execution
mode, omit glc=1. mode, omit glc=1.
load atomic monotonic - singlethread - local 1. ds_load 1. ds_load load atomic monotonic - singlethread - local 1. ds_load 1. ds_load
- wavefront - wavefront
- workgroup - workgroup
load atomic monotonic - agent - global 1. buffer/global/flat_load 1. buffer/global/flat_load load atomic monotonic - agent - global 1. buffer/global/flat_load 1. buffer/global/flat_load
- system - generic glc=1 glc=1 dlc=1 - system - generic glc=1 glc=1 dlc=1
store atomic monotonic - singlethread - global 1. buffer/global/flat_store 1. buffer/global/flat_store store atomic monotonic - singlethread - global 1. buffer/global/flat_store 1. buffer/global/flat_store
- wavefront - generic - wavefront - generic
- workgroup - workgroup
- agent - agent
- system - system
store atomic monotonic - singlethread - local 1. ds_store 1. ds_store store atomic monotonic - singlethread - local 1. ds_store 1. ds_store
- wavefront - wavefront
- workgroup - workgroup
atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 1. buffer/global/flat_atomic atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 1. buffer/global/flat_atomic
- wavefront - generic - wavefront - generic
- workgroup - workgroup
- agent - agent
- system - system
atomicrmw monotonic - singlethread - local 1. ds_atomic 1. ds_atomic atomicrmw monotonic - singlethread - local 1. ds_atomic 1. ds_atomic
- wavefront - wavefront
- workgroup - workgroup
**Acquire Atomic** **Acquire Atomic**
---------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------
load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 1. buffer/global/ds/flat_load load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 1. buffer/global/ds/flat_load
- wavefront - local - wavefront - local
- generic - generic
load atomic acquire - workgroup - global 1. buffer/global_load 1. buffer/global_load glc=1 load atomic acquire - workgroup - global 1. buffer/global_load 1. buffer/global_load glc=1
- If CU wavefront execution - If CU wavefront execution
mode, omit glc=1. mode, omit glc=1.
2. s_waitcnt vmcnt(0) 2. s_waitcnt vmcnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- Must happen before - Must happen before
the following buffer_gl0_inv the following buffer_gl0_inv
and before any following and before any following
global/generic global/generic
load/load load/load
atomic/store/store atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.
3. buffer_gl0_inv 3. buffer_gl0_inv
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- Ensures that - Ensures that
following following
loads will not see loads will not see
stale data. stale data.
load atomic acquire - workgroup - local 1. ds_load 1. ds_load load atomic acquire - workgroup - local 1. ds_load 1. ds_load
2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0)
- If OpenCL, omit. - If OpenCL, omit. - If OpenCL, omit. - If OpenCL, omit.
- Must happen before - Must happen before - Must happen before - Must happen before
any following the following buffer_gl0_inv any following the following buffer_gl0_inv
global/generic and before any following global/generic and before any following
load/load global/generic load/load load/load global/generic load/load
atomic/store/store atomic/store/store atomic/store/store atomic/store/store
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Ensures any - Ensures any - Ensures any - Ensures any
following global following global following global following global
data read is no data read is no data read is no data read is no
older than the load older than the load older than the load older than the load
atomic value being atomic value being atomic value being atomic value being
acquired. acquired. acquired. acquired.
3. buffer_gl0_inv 3. buffer_gl0_inv
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- If OpenCL, omit. - If OpenCL, omit.
- Ensures that - Ensures that
following following
loads will not see loads will not see
stale data. stale data.
load atomic acquire - workgroup - generic 1. flat_load 1. flat_load glc=1 load atomic acquire - workgroup - generic 1. flat_load 1. flat_load glc=1
- If CU wavefront execution - If CU wavefront execution
mode, omit glc=1. mode, omit glc=1.
2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) & 2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) &
vmcnt(0) vmcnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vmcnt(0). mode, omit vmcnt(0).
- If OpenCL, omit. - If OpenCL, omit - If OpenCL, omit. - If OpenCL, omit
lgkmcnt(0). lgkmcnt(0).
- Must happen before - Must happen before - Must happen before - Must happen before
any following the following any following the following
global/generic buffer_gl0_inv and any global/generic buffer_gl0_inv and any
load/load following global/generic load/load following global/generic
atomic/store/store load/load atomic/store/store load/load
atomic/atomicrmw. atomic/store/store atomic/atomicrmw. atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.
- Ensures any - Ensures any - Ensures any - Ensures any
following global following global following global following global
data read is no data read is no data read is no data read is no
older than the load older than the load older than the load older than the load
atomic value being atomic value being atomic value being atomic value being
acquired. acquired. acquired. acquired.
3. buffer_gl0_inv 3. buffer_gl0_inv
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- Ensures that - Ensures that
following following
loads will not see loads will not see
stale data. stale data.
load atomic acquire - agent - global 1. buffer/global_load 1. buffer/global_load load atomic acquire - agent - global 1. buffer/global_load 1. buffer/global_load
- system glc=1 glc=1 dlc=1 - system glc=1 glc=1 dlc=1
2. s_waitcnt vmcnt(0) 2. s_waitcnt vmcnt(0) 2. s_waitcnt vmcnt(0) 2. s_waitcnt vmcnt(0)
- Must happen before - Must happen before - Must happen before - Must happen before
following following following following
buffer_wbinvl1_vol. buffer_gl*_inv. buffer_wbinvl1_vol. buffer_gl*_inv.
- Ensures the load - Ensures the load - Ensures the load - Ensures the load
has completed has completed has completed has completed
before invalidating before invalidating before invalidating before invalidating
the cache. the caches. the cache. the caches.
3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 3. buffer_wbinvl1_vol 3. buffer_gl0_inv;
buffer_gl1_inv buffer_gl1_inv
- Must happen before - Must happen before - Must happen before - Must happen before
any following any following any following any following
global/generic global/generic global/generic global/generic
load/load load/load load/load load/load
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Ensures that - Ensures that - Ensures that - Ensures that
following following following following
loads will not see loads will not see loads will not see loads will not see
stale global data. stale global data. stale global data. stale global data.
load atomic acquire - agent - generic 1. flat_load glc=1 1. flat_load glc=1 dlc=1 load atomic acquire - agent - generic 1. flat_load glc=1 1. flat_load glc=1 dlc=1
- system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vmcnt(0) & - system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vmcnt(0) &
lgkmcnt(0) lgkmcnt(0) lgkmcnt(0) lgkmcnt(0)
- If OpenCL omit - If OpenCL omit - If OpenCL omit - If OpenCL omit
lgkmcnt(0). lgkmcnt(0). lgkmcnt(0). lgkmcnt(0).
- Must happen before - Must happen before - Must happen before - Must happen before
following following following following
buffer_wbinvl1_vol. buffer_gl*_invl. buffer_wbinvl1_vol. buffer_gl*_invl.
- Ensures the flat_load - Ensures the flat_load - Ensures the flat_load - Ensures the flat_load
has completed has completed has completed has completed
before invalidating before invalidating before invalidating before invalidating
the cache. the caches. the cache. the caches.
3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 3. buffer_wbinvl1_vol 3. buffer_gl0_inv;
buffer_gl1_inv buffer_gl1_inv
- Must happen before - Must happen before - Must happen before - Must happen before
any following any following any following any following
global/generic global/generic global/generic global/generic
load/load load/load load/load load/load
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Ensures that - Ensures that - Ensures that - Ensures that
following loads following loads following loads following loads
will not see stale will not see stale will not see stale will not see stale
global data. global data. global data. global data.
atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic
- wavefront - local - wavefront - local
- generic - generic
atomicrmw acquire - workgroup - global 1. buffer/global_atomic 1. buffer/global_atomic atomicrmw acquire - workgroup - global 1. buffer/global_atomic 1. buffer/global_atomic
2. s_waitcnt vm/vscnt(0) 2. s_waitcnt vm/vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- Use vmcnt(0) if atomic with - Use vmcnt(0) if atomic with
return and vscnt(0) if return and vscnt(0) if
atomic with no-return. atomic with no-return.
- Must happen before - Must happen before
the following buffer_gl0_inv the following buffer_gl0_inv
and before any following and before any following
global/generic global/generic
load/load load/load
atomic/store/store atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.
3. buffer_gl0_inv 3. buffer_gl0_inv
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- Ensures that - Ensures that
following following
loads will not see loads will not see
stale data. stale data.
atomicrmw acquire - workgroup - local 1. ds_atomic 1. ds_atomic atomicrmw acquire - workgroup - local 1. ds_atomic 1. ds_atomic
2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0)
- If OpenCL, omit. - If OpenCL, omit. - If OpenCL, omit. - If OpenCL, omit.
- Must happen before - Must happen before - Must happen before - Must happen before
any following the following any following the following
global/generic buffer_gl0_inv. global/generic buffer_gl0_inv.
load/load load/load
atomic/store/store atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.
- Ensures any - Ensures any - Ensures any - Ensures any
following global following global following global following global
data read is no data read is no data read is no data read is no
older than the older than the older than the older than the
atomicrmw value atomicrmw value atomicrmw value atomicrmw value
being acquired. being acquired. being acquired. being acquired.
3. buffer_gl0_inv 3. buffer_gl0_inv
- If OpenCL omit. - If OpenCL omit.
- Ensures that - Ensures that
following following
loads will not see loads will not see
stale data. stale data.
atomicrmw acquire - workgroup - generic 1. flat_atomic 1. flat_atomic atomicrmw acquire - workgroup - generic 1. flat_atomic 1. flat_atomic
2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) & 2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) &
vm/vscnt(0) vm/vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vm/vscnt(0). mode, omit vm/vscnt(0).
- If OpenCL, omit. - If OpenCL, omit - If OpenCL, omit. - If OpenCL, omit
waitcnt lgkmcnt(0). waitcnt lgkmcnt(0).
- Use vmcnt(0) if atomic with - Use vmcnt(0) if atomic with
return and vscnt(0) if return and vscnt(0) if
atomic with no-return. atomic with no-return.
- Must happen before - Must happen before - Must happen before - Must happen before
any following the following any following the following
global/generic buffer_gl0_inv. global/generic buffer_gl0_inv.
load/load load/load
atomic/store/store atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.
- Ensures any - Ensures any - Ensures any - Ensures any
following global following global following global following global
data read is no data read is no data read is no data read is no
older than the older than the older than the older than the
atomicrmw value atomicrmw value atomicrmw value atomicrmw value
being acquired. being acquired. being acquired. being acquired.
3. buffer_gl0_inv 3. buffer_gl0_inv
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- Ensures that - Ensures that
following following
loads will not see loads will not see
stale data. stale data.
atomicrmw acquire - agent - global 1. buffer/global_atomic 1. buffer/global_atomic atomicrmw acquire - agent - global 1. buffer/global_atomic 1. buffer/global_atomic
- system 2. s_waitcnt vmcnt(0) 2. s_waitcnt vm/vscnt(0) - system 2. s_waitcnt vmcnt(0) 2. s_waitcnt vm/vscnt(0)
- Use vmcnt(0) if atomic with - Use vmcnt(0) if atomic with
return and vscnt(0) if return and vscnt(0) if
atomic with no-return. atomic with no-return.
waitcnt lgkmcnt(0). waitcnt lgkmcnt(0).
- Must happen before - Must happen before - Must happen before - Must happen before
following following following following
buffer_wbinvl1_vol. buffer_gl*_inv. buffer_wbinvl1_vol. buffer_gl*_inv.
- Ensures the - Ensures the - Ensures the - Ensures the
atomicrmw has atomicrmw has atomicrmw has atomicrmw has
completed before completed before completed before completed before
invalidating the invalidating the invalidating the invalidating the
cache. caches. cache. caches.
3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 3. buffer_wbinvl1_vol 3. buffer_gl0_inv;
buffer_gl1_inv buffer_gl1_inv
- Must happen before - Must happen before - Must happen before - Must happen before
any following any following any following any following
global/generic global/generic global/generic global/generic
load/load load/load load/load load/load
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Ensures that - Ensures that - Ensures that - Ensures that
following loads following loads following loads following loads
will not see stale will not see stale will not see stale will not see stale
global data. global data. global data. global data.
atomicrmw acquire - agent - generic 1. flat_atomic 1. flat_atomic atomicrmw acquire - agent - generic 1. flat_atomic 1. flat_atomic
- system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vm/vscnt(0) & - system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vm/vscnt(0) &
lgkmcnt(0) lgkmcnt(0) lgkmcnt(0) lgkmcnt(0)
- If OpenCL, omit - If OpenCL, omit - If OpenCL, omit - If OpenCL, omit
lgkmcnt(0). lgkmcnt(0). lgkmcnt(0). lgkmcnt(0).
- Use vmcnt(0) if atomic with - Use vmcnt(0) if atomic with
return and vscnt(0) if return and vscnt(0) if
atomic with no-return. atomic with no-return.
- Must happen before - Must happen before - Must happen before - Must happen before
following following following following
buffer_wbinvl1_vol. buffer_gl*_inv. buffer_wbinvl1_vol. buffer_gl*_inv.
- Ensures the - Ensures the - Ensures the - Ensures the
atomicrmw has atomicrmw has atomicrmw has atomicrmw has
completed before completed before completed before completed before
invalidating the invalidating the invalidating the invalidating the
cache. caches. cache. caches.
3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 3. buffer_wbinvl1_vol 3. buffer_gl0_inv;
buffer_gl1_inv buffer_gl1_inv
- Must happen before - Must happen before - Must happen before - Must happen before
any following any following any following any following
global/generic global/generic global/generic global/generic
load/load load/load load/load load/load
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Ensures that - Ensures that - Ensures that - Ensures that
following loads following loads following loads following loads
will not see stale will not see stale will not see stale will not see stale
global data. global data. global data. global data.
fence acquire - singlethread *none* *none* *none* fence acquire - singlethread *none* *none* *none*
- wavefront - wavefront
fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0) vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vmcnt(0) and mode, omit vmcnt(0) and
vscnt(0). vscnt(0).
- If OpenCL and - If OpenCL and - If OpenCL and - If OpenCL and
address space is address space is address space is address space is
not generic, omit. not generic, omit not generic, omit. not generic, omit
lgkmcnt(0). lgkmcnt(0).
- If OpenCL and - If OpenCL and
address space is address space is
local, omit local, omit
vmcnt(0) and vscnt(0). vmcnt(0) and vscnt(0).
- However, since LLVM - However, since LLVM - However, since LLVM - However, since LLVM
currently has no currently has no currently has no currently has no
address space on address space on address space on address space on
the fence need to the fence need to the fence need to the fence need to
conservatively conservatively conservatively conservatively
always generate. If always generate. If always generate. If always generate. If
fence had an fence had an fence had an fence had an
address space then address space then address space then address space then
set to address set to address set to address set to address
space of OpenCL space of OpenCL space of OpenCL space of OpenCL
fence flag, or to fence flag, or to fence flag, or to fence flag, or to
generic if both generic if both generic if both generic if both
local and global local and global local and global local and global
flags are flags are flags are flags are
specified. specified. specified. specified.
- Must happen after - Must happen after
any preceding any preceding
local/generic load local/generic load
atomic/atomicrmw atomic/atomicrmw
with an equal or with an equal or
wider sync scope wider sync scope
and memory ordering and memory ordering
stronger than stronger than
unordered (this is unordered (this is
termed the termed the
fence-paired-atomic). fence-paired-atomic).
- Must happen before - Must happen before
any following any following
global/generic global/generic
load/load load/load
atomic/store/store atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.
- Ensures any - Ensures any
following global following global
data read is no data read is no
older than the older than the
value read by the value read by the
fence-paired-atomic. fence-paired-atomic.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0), s_waitcnt vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic load global/generic load
atomic/ atomic/
atomicrmw-with-return-value atomicrmw-with-return-value
with an equal or with an equal or
wider sync scope wider sync scope
and memory ordering and memory ordering
stronger than stronger than
unordered (this is unordered (this is
termed the termed the
fence-paired-atomic). fence-paired-atomic).
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
atomicrmw-no-return-value atomicrmw-no-return-value
with an equal or with an equal or
wider sync scope wider sync scope
and memory ordering and memory ordering
stronger than stronger than
unordered (this is unordered (this is
termed the termed the
fence-paired-atomic). fence-paired-atomic).
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after
any preceding any preceding
local/generic load local/generic load
atomic/atomicrmw atomic/atomicrmw
with an equal or with an equal or
wider sync scope wider sync scope
and memory ordering and memory ordering
stronger than stronger than
unordered (this is unordered (this is
termed the termed the
fence-paired-atomic). fence-paired-atomic).
- Must happen before - Must happen before
the following the following
buffer_gl0_inv. buffer_gl0_inv.
- Ensures that the - Ensures that the
fence-paired atomic fence-paired atomic
has completed has completed
before invalidating before invalidating
the the
cache. Therefore cache. Therefore
any following any following
locations read must locations read must
be no older than be no older than
the value read by the value read by
the the
fence-paired-atomic. fence-paired-atomic.
3. buffer_gl0_inv 3. buffer_gl0_inv
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- Ensures that - Ensures that
following following
loads will not see loads will not see
stale data. stale data.
fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &
- system vmcnt(0) vmcnt(0) & vscnt(0) - system vmcnt(0) vmcnt(0) & vscnt(0)
- If OpenCL and - If OpenCL and - If OpenCL and - If OpenCL and
address space is address space is address space is address space is
not generic, omit not generic, omit not generic, omit not generic, omit
lgkmcnt(0). lgkmcnt(0). lgkmcnt(0). lgkmcnt(0).
- If OpenCL and - If OpenCL and
address space is address space is
local, omit local, omit
vmcnt(0) and vscnt(0). vmcnt(0) and vscnt(0).
- However, since LLVM - However, since LLVM - However, since LLVM - However, since LLVM
currently has no currently has no currently has no currently has no
address space on address space on address space on address space on
the fence need to the fence need to the fence need to the fence need to
conservatively conservatively conservatively conservatively
always generate always generate always generate always generate
(see comment for (see comment for (see comment for (see comment for
previous fence). previous fence). previous fence). previous fence).
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0) and vmcnt(0) and
s_waitcnt s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
Show All 32 Lines fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &
the the
cache. Therefore cache. Therefore
any following any following
locations read must locations read must
be no older than be no older than
the value read by the value read by
the the
fence-paired-atomic. fence-paired-atomic.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0), s_waitcnt vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic load global/generic load
atomic/ atomic/
atomicrmw-with-return-value atomicrmw-with-return-value
with an equal or with an equal or
wider sync scope wider sync scope
and memory ordering and memory ordering
stronger than stronger than
unordered (this is unordered (this is
termed the termed the
fence-paired-atomic). fence-paired-atomic).
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
atomicrmw-no-return-value atomicrmw-no-return-value
with an equal or with an equal or
wider sync scope wider sync scope
and memory ordering and memory ordering
stronger than stronger than
unordered (this is unordered (this is
termed the termed the
fence-paired-atomic). fence-paired-atomic).
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after
any preceding any preceding
local/generic load local/generic load
atomic/atomicrmw atomic/atomicrmw
with an equal or with an equal or
wider sync scope wider sync scope
and memory ordering and memory ordering
stronger than stronger than
unordered (this is unordered (this is
termed the termed the
fence-paired-atomic). fence-paired-atomic).
- Must happen before - Must happen before
the following the following
buffer_gl*_inv. buffer_gl*_inv.
- Ensures that the - Ensures that the
fence-paired atomic fence-paired atomic
has completed has completed
before invalidating before invalidating
the the
caches. Therefore caches. Therefore
any following any following
locations read must locations read must
be no older than be no older than
the value read by the value read by
the the
fence-paired-atomic. fence-paired-atomic.
2. buffer_wbinvl1_vol 2. buffer_gl0_inv; 2. buffer_wbinvl1_vol 2. buffer_gl0_inv;
buffer_gl1_inv buffer_gl1_inv
- Must happen before any - Must happen before any - Must happen before any - Must happen before any
following global/generic following global/generic following global/generic following global/generic
load/load load/load load/load load/load
atomic/store/store atomic/store/store atomic/store/store atomic/store/store
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Ensures that - Ensures that - Ensures that - Ensures that
following loads following loads following loads following loads
will not see stale will not see stale will not see stale will not see stale
global data. global data. global data. global data.
**Release Atomic** **Release Atomic**
---------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------
store atomic release - singlethread - global 1. buffer/global/ds/flat_store 1. buffer/global/ds/flat_store store atomic release - singlethread - global 1. buffer/global/ds/flat_store 1. buffer/global/ds/flat_store
- wavefront - local - wavefront - local
- generic - generic
store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0) vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vmcnt(0) and mode, omit vmcnt(0) and
vscnt(0). vscnt(0).
- If OpenCL, omit. - If OpenCL, omit - If OpenCL, omit. - If OpenCL, omit
lgkmcnt(0). lgkmcnt(0).
- Must happen after - Must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0), s_waitcnt vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic load/load global/generic load/load
atomic/ atomic/
atomicrmw-with-return-value. atomicrmw-with-return-value.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store store/store
atomic/ atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
the following the following the following the following
store. store. store. store.
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
memory operations memory operations memory operations memory operations
to local have have to local have have
completed before completed before completed before completed before
performing the performing the performing the performing the
store that is being store that is being store that is being store that is being
released. released. released. released.
2. buffer/global_store 2. buffer/global_store 2. buffer/global_store 2. buffer/global_store
store atomic release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) store atomic release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- If OpenCL, omit. - If OpenCL, omit.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0) and s_waitcnt vmcnt(0) and s_waitcnt
vscnt(0) to allow vscnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic load/load global/generic load/load
atomic/ atomic/
atomicrmw-with-return-value. atomicrmw-with-return-value.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store atomic/ store/store atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- Must happen before - Must happen before
the following the following
store. store.
- Ensures that all - Ensures that all
global memory global memory
operations have operations have
completed before completed before
performing the performing the
store that is being store that is being
released. released.
1. ds_store 2. ds_store 1. ds_store 2. ds_store
store atomic release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & store atomic release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0) vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vmcnt(0) and mode, omit vmcnt(0) and
vscnt(0). vscnt(0).
- If OpenCL, omit. - If OpenCL, omit - If OpenCL, omit. - If OpenCL, omit
lgkmcnt(0). lgkmcnt(0).
- Must happen after - Must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0), s_waitcnt vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic load/load global/generic load/load
atomic/ atomic/
atomicrmw-with-return-value. atomicrmw-with-return-value.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store store/store
atomic/ atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
the following the following the following the following
store. store. store. store.
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
memory operations memory operations memory operations memory operations
to local have have to local have have
completed before completed before completed before completed before
performing the performing the performing the performing the
store that is being store that is being store that is being store that is being
released. released. released. released.
2. flat_store 2. flat_store 2. flat_store 2. flat_store
store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &
- system - generic vmcnt(0) vmcnt(0) & vscnt(0) - system - generic vmcnt(0) vmcnt(0) & vscnt(0)
- If OpenCL, omit - If OpenCL, omit - If OpenCL, omit - If OpenCL, omit
lgkmcnt(0). lgkmcnt(0). lgkmcnt(0). lgkmcnt(0).
- Could be split into - Could be split into - Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt separate s_waitcnt separate s_waitcnt
vmcnt(0) and vmcnt(0), s_waitcnt vscnt(0) vmcnt(0) and vmcnt(0), s_waitcnt vscnt(0)
s_waitcnt and s_waitcnt s_waitcnt and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be them to be them to be
independently moved independently moved independently moved independently moved
according to the according to the according to the according to the
following rules. following rules. following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after must happen after must happen after
any preceding any preceding any preceding any preceding
global/generic global/generic global/generic global/generic
load/store/load load/load load/store/load load/load
atomic/store atomic/ atomic/store atomic/
atomic/atomicrmw. atomicrmw-with-return-value. atomic/atomicrmw. atomicrmw-with-return-value.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store atomic/ store/store atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after must happen after must happen after
any preceding any preceding any preceding any preceding
local/generic local/generic local/generic local/generic
load/store/load load/store/load load/store/load load/store/load
atomic/store atomic/store atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
the following the following the following the following
store. store. store. store.
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
memory operations memory operations memory operations memory operations
to memory have to memory have to memory have to memory have
completed before completed before completed before completed before
performing the performing the performing the performing the
store that is being store that is being store that is being store that is being
released. released. released. released.
2. buffer/global/flat_store 2. buffer/global/flat_store 2. buffer/global/flat_store 2. buffer/global/flat_store
atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic
- wavefront - local - wavefront - local
- generic - generic
atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0) vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vmcnt(0) and mode, omit vmcnt(0) and
vscnt(0). vscnt(0).
- If OpenCL, omit. - If OpenCL, omit.
- Must happen after - Must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0), s_waitcnt vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic load/load global/generic load/load
atomic/ atomic/
atomicrmw-with-return-value. atomicrmw-with-return-value.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store store/store
atomic/ atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
the following the following the following the following
atomicrmw. atomicrmw. atomicrmw. atomicrmw.
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
memory operations memory operations memory operations memory operations
to local have have to local have have
completed before completed before completed before completed before
performing the performing the performing the performing the
atomicrmw that is atomicrmw that is atomicrmw that is atomicrmw that is
being released. being released. being released. being released.
2. buffer/global_atomic 2. buffer/global_atomic 2. buffer/global_atomic 2. buffer/global_atomic
atomicrmw release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) atomicrmw release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- If OpenCL, omit. - If OpenCL, omit.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0) and s_waitcnt vmcnt(0) and s_waitcnt
vscnt(0) to allow vscnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic load/load global/generic load/load
atomic/ atomic/
atomicrmw-with-return-value. atomicrmw-with-return-value.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store atomic/ store/store atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- Must happen before - Must happen before
the following the following
store. store.
- Ensures that all - Ensures that all
global memory global memory
operations have operations have
completed before completed before
performing the performing the
store that is being store that is being
released. released.
1. ds_atomic 2. ds_atomic 1. ds_atomic 2. ds_atomic
atomicrmw release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & atomicrmw release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0) vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vmcnt(0) and mode, omit vmcnt(0) and
vscnt(0). vscnt(0).
- If OpenCL, omit. - If OpenCL, omit - If OpenCL, omit. - If OpenCL, omit
waitcnt lgkmcnt(0). waitcnt lgkmcnt(0).
- Must happen after - Must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0), s_waitcnt vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic load/load global/generic load/load
atomic/ atomic/
atomicrmw-with-return-value. atomicrmw-with-return-value.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store store/store
atomic/ atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
the following the following the following the following
atomicrmw. atomicrmw. atomicrmw. atomicrmw.
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
memory operations memory operations memory operations memory operations
to local have have to local have have
completed before completed before completed before completed before
performing the performing the performing the performing the
atomicrmw that is atomicrmw that is atomicrmw that is atomicrmw that is
being released. being released. being released. being released.
2. flat_atomic 2. flat_atomic 2. flat_atomic 2. flat_atomic
atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lkkmcnt(0) & atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lkkmcnt(0) &
- system - generic vmcnt(0) vmcnt(0) & vscnt(0) - system - generic vmcnt(0) vmcnt(0) & vscnt(0)
- If OpenCL, omit - If OpenCL, omit - If OpenCL, omit - If OpenCL, omit
lgkmcnt(0). lgkmcnt(0). lgkmcnt(0). lgkmcnt(0).
- Could be split into - Could be split into - Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt separate s_waitcnt separate s_waitcnt
vmcnt(0) and vmcnt(0), s_waitcnt vmcnt(0) and vmcnt(0), s_waitcnt
s_waitcnt vscnt(0) and s_waitcnt s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be them to be them to be
independently moved independently moved independently moved independently moved
according to the according to the according to the according to the
following rules. following rules. following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after must happen after must happen after
any preceding any preceding any preceding any preceding
global/generic global/generic global/generic global/generic
load/store/load load/load atomic/ load/store/load load/load atomic/
atomic/store atomicrmw-with-return-value. atomic/store atomicrmw-with-return-value.
atomic/atomicrmw. atomic/atomicrmw.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store atomic/ store/store atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after must happen after must happen after
any preceding any preceding any preceding any preceding
local/generic local/generic local/generic local/generic
load/store/load load/store/load load/store/load load/store/load
atomic/store atomic/store atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
the following the following the following the following
atomicrmw. atomicrmw. atomicrmw. atomicrmw.
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
memory operations memory operations memory operations memory operations
to global and local to global and local to global and local to global and local
have completed have completed have completed have completed
before performing before performing before performing before performing
the atomicrmw that the atomicrmw that the atomicrmw that the atomicrmw that
is being released. is being released. is being released. is being released.
2. buffer/global/flat_atomic 2. buffer/global/flat_atomic 2. buffer/global/flat_atomic 2. buffer/global/flat_atomic
fence release - singlethread *none* *none* *none* fence release - singlethread *none* *none* *none*
- wavefront - wavefront
fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0) vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vmcnt(0) and mode, omit vmcnt(0) and
vscnt(0). vscnt(0).
- If OpenCL and - If OpenCL and - If OpenCL and - If OpenCL and
address space is address space is address space is address space is
not generic, omit. not generic, omit not generic, omit. not generic, omit
lgkmcnt(0). lgkmcnt(0).
- If OpenCL and - If OpenCL and
address space is address space is
local, omit local, omit
vmcnt(0) and vscnt(0). vmcnt(0) and vscnt(0).
- However, since LLVM - However, since LLVM - However, since LLVM - However, since LLVM
currently has no currently has no currently has no currently has no
address space on address space on address space on address space on
the fence need to the fence need to the fence need to the fence need to
conservatively conservatively conservatively conservatively
always generate. If always generate. If always generate. If always generate. If
fence had an fence had an fence had an fence had an
address space then address space then address space then address space then
set to address set to address set to address set to address
space of OpenCL space of OpenCL space of OpenCL space of OpenCL
fence flag, or to fence flag, or to fence flag, or to fence flag, or to
generic if both generic if both generic if both generic if both
local and global local and global local and global local and global
flags are flags are flags are flags are
specified. specified. specified. specified.
- Must happen after - Must happen after
any preceding any preceding
local/generic local/generic
load/load load/load
atomic/store/store atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0), s_waitcnt vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
load/load load/load
atomic/ atomic/
atomicrmw-with-return-value. atomicrmw-with-return-value.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store atomic/ store/store atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/ atomic/store atomic/
atomicrmw. atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
any following store any following store any following store any following store
atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw
with an equal or with an equal or with an equal or with an equal or
wider sync scope wider sync scope wider sync scope wider sync scope
and memory ordering and memory ordering and memory ordering and memory ordering
stronger than stronger than stronger than stronger than
unordered (this is unordered (this is unordered (this is unordered (this is
termed the termed the termed the termed the
fence-paired-atomic). fence-paired-atomic). fence-paired-atomic). fence-paired-atomic).
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
memory operations memory operations memory operations memory operations
to local have have to local have have
completed before completed before completed before completed before
performing the performing the performing the performing the
following following following following
fence-paired-atomic. fence-paired-atomic. fence-paired-atomic. fence-paired-atomic.
fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &
- system vmcnt(0) vmcnt(0) & vscnt(0) - system vmcnt(0) vmcnt(0) & vscnt(0)
- If OpenCL and - If OpenCL and - If OpenCL and - If OpenCL and
address space is address space is address space is address space is
not generic, omit not generic, omit not generic, omit not generic, omit
lgkmcnt(0). lgkmcnt(0). lgkmcnt(0). lgkmcnt(0).
- If OpenCL and - If OpenCL and - If OpenCL and - If OpenCL and
address space is address space is address space is address space is
local, omit local, omit local, omit local, omit
vmcnt(0). vmcnt(0) and vscnt(0). vmcnt(0). vmcnt(0) and vscnt(0).
- However, since LLVM - However, since LLVM - However, since LLVM - However, since LLVM
currently has no currently has no currently has no currently has no
address space on address space on address space on address space on
the fence need to the fence need to the fence need to the fence need to
conservatively conservatively conservatively conservatively
always generate. If always generate. If always generate. If always generate. If
fence had an fence had an fence had an fence had an
address space then address space then address space then address space then
set to address set to address set to address set to address
space of OpenCL space of OpenCL space of OpenCL space of OpenCL
fence flag, or to fence flag, or to fence flag, or to fence flag, or to
generic if both generic if both generic if both generic if both
local and global local and global local and global local and global
flags are flags are flags are flags are
specified. specified. specified. specified.
- Could be split into - Could be split into - Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt separate s_waitcnt separate s_waitcnt
vmcnt(0) and vmcnt(0), s_waitcnt vmcnt(0) and vmcnt(0), s_waitcnt
s_waitcnt vscnt(0) and s_waitcnt s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be them to be them to be
independently moved independently moved independently moved independently moved
according to the according to the according to the according to the
following rules. following rules. following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after must happen after must happen after
any preceding any preceding any preceding any preceding
global/generic global/generic global/generic global/generic
load/store/load load/load atomic/ load/store/load load/load atomic/
atomic/store atomicrmw-with-return-value. atomic/store atomicrmw-with-return-value.
atomic/atomicrmw. atomic/atomicrmw.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store atomic/ store/store atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after must happen after must happen after
any preceding any preceding any preceding any preceding
local/generic local/generic local/generic local/generic
load/store/load load/store/load load/store/load load/store/load
atomic/store atomic/store atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
any following store any following store any following store any following store
atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw
with an equal or with an equal or with an equal or with an equal or
wider sync scope wider sync scope wider sync scope wider sync scope
and memory ordering and memory ordering and memory ordering and memory ordering
stronger than stronger than stronger than stronger than
unordered (this is unordered (this is unordered (this is unordered (this is
termed the termed the termed the termed the
fence-paired-atomic). fence-paired-atomic). fence-paired-atomic). fence-paired-atomic).
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
memory operations memory operations memory operations memory operations
have have have have
completed before completed before completed before completed before
performing the performing the performing the performing the
following following following following
fence-paired-atomic. fence-paired-atomic. fence-paired-atomic. fence-paired-atomic.
**Acquire-Release Atomic** **Acquire-Release Atomic**
---------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------
atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic
- wavefront - local - wavefront - local
- generic - generic
atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0) vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vmcnt(0) and mode, omit vmcnt(0) and
vscnt(0). vscnt(0).
- If OpenCL, omit. - If OpenCL, omit - If OpenCL, omit. - If OpenCL, omit
s_waitcnt lgkmcnt(0). s_waitcnt lgkmcnt(0).
- Must happen after - Must happen after - Must happen after - Must happen after
any preceding any preceding any preceding any preceding
local/generic local/generic local/generic local/generic
load/store/load load/store/load load/store/load load/store/load
atomic/store atomic/store atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0), s_waitcnt vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic load/load global/generic load/load
atomic/ atomic/
atomicrmw-with-return-value. atomicrmw-with-return-value.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store store/store
atomic/ atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
the following the following the following the following
atomicrmw. atomicrmw. atomicrmw. atomicrmw.
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
memory operations memory operations memory operations memory operations
to local have have to local have have
completed before completed before completed before completed before
performing the performing the performing the performing the
atomicrmw that is atomicrmw that is atomicrmw that is atomicrmw that is
being released. being released. being released. being released.
2. buffer/global_atomic 2. buffer/global_atomic 2. buffer/global_atomic 2. buffer/global_atomic
3. s_waitcnt vm/vscnt(0) 3. s_waitcnt vm/vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vm/vscnt(0). mode, omit vm/vscnt(0).
- Use vmcnt(0) if atomic with - Use vmcnt(0) if atomic with
return and vscnt(0) if return and vscnt(0) if
atomic with no-return. atomic with no-return.
waitcnt lgkmcnt(0). waitcnt lgkmcnt(0).
- Must happen before - Must happen before
the following the following
buffer_gl0_inv. buffer_gl0_inv.
- Ensures any - Ensures any
following global following global
data read is no data read is no
older than the older than the
atomicrmw value atomicrmw value
being acquired. being acquired.
4. buffer_gl0_inv 4. buffer_gl0_inv
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- Ensures that - Ensures that
following following
loads will not see loads will not see
stale data. stale data.
atomicrmw acq_rel - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) atomicrmw acq_rel - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- If OpenCL, omit. - If OpenCL, omit.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0) and s_waitcnt vmcnt(0) and s_waitcnt
vscnt(0) to allow vscnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic load/load global/generic load/load
atomic/ atomic/
atomicrmw-with-return-value. atomicrmw-with-return-value.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store atomic/ store/store atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- Must happen before - Must happen before
the following the following
store. store.
- Ensures that all - Ensures that all
global memory global memory
operations have operations have
completed before completed before
performing the performing the
store that is being store that is being
released. released.
1. ds_atomic 2. ds_atomic 1. ds_atomic 2. ds_atomic
2. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0)
- If OpenCL, omit. - If OpenCL, omit. - If OpenCL, omit. - If OpenCL, omit.
- Must happen before - Must happen before - Must happen before - Must happen before
any following the following any following the following
global/generic buffer_gl0_inv. global/generic buffer_gl0_inv.
load/load load/load
atomic/store/store atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.
- Ensures any - Ensures any - Ensures any - Ensures any
following global following global following global following global
data read is no data read is no data read is no data read is no
older than the load older than the load older than the load older than the load
atomic value being atomic value being atomic value being atomic value being
acquired. acquired. acquired. acquired.
4. buffer_gl0_inv 4. buffer_gl0_inv
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- If OpenCL omit. - If OpenCL omit.
- Ensures that - Ensures that
following following
loads will not see loads will not see
stale data. stale data.
atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0) vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vmcnt(0) and mode, omit vmcnt(0) and
vscnt(0). vscnt(0).
- If OpenCL, omit. - If OpenCL, omit - If OpenCL, omit. - If OpenCL, omit
waitcnt lgkmcnt(0). waitcnt lgkmcnt(0).
- Must happen after - Must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0), s_waitcnt vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic load/load global/generic load/load
atomic/ atomic/
atomicrmw-with-return-value. atomicrmw-with-return-value.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store store/store
atomic/ atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
the following the following the following the following
atomicrmw. atomicrmw. atomicrmw. atomicrmw.
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
memory operations memory operations memory operations memory operations
to local have have to local have have
completed before completed before completed before completed before
performing the performing the performing the performing the
atomicrmw that is atomicrmw that is atomicrmw that is atomicrmw that is
being released. being released. being released. being released.
2. flat_atomic 2. flat_atomic 2. flat_atomic 2. flat_atomic
3. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) & 3. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) &
vm/vscnt(0) vm/vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vm/vscnt(0). mode, omit vm/vscnt(0).
- If OpenCL, omit. - If OpenCL, omit - If OpenCL, omit. - If OpenCL, omit
waitcnt lgkmcnt(0). waitcnt lgkmcnt(0).
- Must happen before - Must happen before - Must happen before - Must happen before
any following the following any following the following
global/generic buffer_gl0_inv. global/generic buffer_gl0_inv.
load/load load/load
atomic/store/store atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.
- Ensures any - Ensures any - Ensures any - Ensures any
following global following global following global following global
data read is no data read is no data read is no data read is no
older than the load older than the load older than the load older than the load
atomic value being atomic value being atomic value being atomic value being
acquired. acquired. acquired. acquired.
3. buffer_gl0_inv 3. buffer_gl0_inv
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- Ensures that - Ensures that
following following
loads will not see loads will not see
stale data. stale data.
atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &
- system vmcnt(0) vmcnt(0) & vscnt(0) - system vmcnt(0) vmcnt(0) & vscnt(0)
- If OpenCL, omit - If OpenCL, omit - If OpenCL, omit - If OpenCL, omit
lgkmcnt(0). lgkmcnt(0). lgkmcnt(0). lgkmcnt(0).
- Could be split into - Could be split into - Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt separate s_waitcnt separate s_waitcnt
vmcnt(0) and vmcnt(0), s_waitcnt vmcnt(0) and vmcnt(0), s_waitcnt
s_waitcnt vscnt(0) and s_waitcnt s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be them to be them to be
independently moved independently moved independently moved independently moved
according to the according to the according to the according to the
following rules. following rules. following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after must happen after must happen after
any preceding any preceding any preceding any preceding
global/generic global/generic global/generic global/generic
load/store/load load/load atomic/ load/store/load load/load atomic/
atomic/store atomicrmw-with-return-value. atomic/store atomicrmw-with-return-value.
atomic/atomicrmw. atomic/atomicrmw.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store atomic/ store/store atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after must happen after must happen after
any preceding any preceding any preceding any preceding
local/generic local/generic local/generic local/generic
load/store/load load/store/load load/store/load load/store/load
atomic/store atomic/store atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
the following the following the following the following
atomicrmw. atomicrmw. atomicrmw. atomicrmw.
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
memory operations memory operations memory operations memory operations
to global have to global have to global have to global have
completed before completed before completed before completed before
performing the performing the performing the performing the
atomicrmw that is atomicrmw that is atomicrmw that is atomicrmw that is
being released. being released. being released. being released.
2. buffer/global_atomic 2. buffer/global_atomic 2. buffer/global_atomic 2. buffer/global_atomic
3. s_waitcnt vmcnt(0) 3. s_waitcnt vm/vscnt(0) 3. s_waitcnt vmcnt(0) 3. s_waitcnt vm/vscnt(0)
- Use vmcnt(0) if atomic with - Use vmcnt(0) if atomic with
return and vscnt(0) if return and vscnt(0) if
atomic with no-return. atomic with no-return.
waitcnt lgkmcnt(0). waitcnt lgkmcnt(0).
- Must happen before - Must happen before - Must happen before - Must happen before
following following following following
buffer_wbinvl1_vol. buffer_gl*_inv. buffer_wbinvl1_vol. buffer_gl*_inv.
- Ensures the - Ensures the - Ensures the - Ensures the
atomicrmw has atomicrmw has atomicrmw has atomicrmw has
completed before completed before completed before completed before
invalidating the invalidating the invalidating the invalidating the
cache. caches. cache. caches.
4. buffer_wbinvl1_vol 4. buffer_gl0_inv; 4. buffer_wbinvl1_vol 4. buffer_gl0_inv;
buffer_gl1_inv buffer_gl1_inv
- Must happen before - Must happen before - Must happen before - Must happen before
any following any following any following any following
global/generic global/generic global/generic global/generic
load/load load/load load/load load/load
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Ensures that - Ensures that - Ensures that - Ensures that
following loads following loads following loads following loads
will not see stale will not see stale will not see stale will not see stale
global data. global data. global data. global data.
atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &
- system vmcnt(0) vmcnt(0) & vscnt(0) - system vmcnt(0) vmcnt(0) & vscnt(0)
- If OpenCL, omit - If OpenCL, omit - If OpenCL, omit - If OpenCL, omit
lgkmcnt(0). lgkmcnt(0). lgkmcnt(0). lgkmcnt(0).
- Could be split into - Could be split into - Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt separate s_waitcnt separate s_waitcnt
vmcnt(0) and vmcnt(0), s_waitcnt vmcnt(0) and vmcnt(0), s_waitcnt
s_waitcnt vscnt(0) and s_waitcnt s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be them to be them to be
independently moved independently moved independently moved independently moved
according to the according to the according to the according to the
following rules. following rules. following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after must happen after must happen after
any preceding any preceding any preceding any preceding
global/generic global/generic global/generic global/generic
load/store/load load/load atomic load/store/load load/load atomic
atomic/store atomicrmw-with-return-value. atomic/store atomicrmw-with-return-value.
atomic/atomicrmw. atomic/atomicrmw.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store atomic/ store/store atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after must happen after must happen after
any preceding any preceding any preceding any preceding
local/generic local/generic local/generic local/generic
load/store/load load/store/load load/store/load load/store/load
atomic/store atomic/store atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
the following the following the following the following
atomicrmw. atomicrmw. atomicrmw. atomicrmw.
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
memory operations memory operations memory operations memory operations
to global have have to global have have
completed before completed before completed before completed before
performing the performing the performing the performing the
atomicrmw that is atomicrmw that is atomicrmw that is atomicrmw that is
being released. being released. being released. being released.
2. flat_atomic 2. flat_atomic 2. flat_atomic 2. flat_atomic
3. s_waitcnt vmcnt(0) & 3. s_waitcnt vm/vscnt(0) & 3. s_waitcnt vmcnt(0) & 3. s_waitcnt vm/vscnt(0) &
lgkmcnt(0) lgkmcnt(0) lgkmcnt(0) lgkmcnt(0)
- If OpenCL, omit - If OpenCL, omit - If OpenCL, omit - If OpenCL, omit
lgkmcnt(0). lgkmcnt(0). lgkmcnt(0). lgkmcnt(0).
- Use vmcnt(0) if atomic with - Use vmcnt(0) if atomic with
return and vscnt(0) if return and vscnt(0) if
atomic with no-return. atomic with no-return.
- Must happen before - Must happen before - Must happen before - Must happen before
following following following following
buffer_wbinvl1_vol. buffer_gl*_inv. buffer_wbinvl1_vol. buffer_gl*_inv.
- Ensures the - Ensures the - Ensures the - Ensures the
atomicrmw has atomicrmw has atomicrmw has atomicrmw has
completed before completed before completed before completed before
invalidating the invalidating the invalidating the invalidating the
cache. caches. cache. caches.
4. buffer_wbinvl1_vol 4. buffer_gl0_inv; 4. buffer_wbinvl1_vol 4. buffer_gl0_inv;
buffer_gl1_inv buffer_gl1_inv
- Must happen before - Must happen before - Must happen before - Must happen before
any following any following any following any following
global/generic global/generic global/generic global/generic
load/load load/load load/load load/load
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Ensures that - Ensures that - Ensures that - Ensures that
following loads following loads following loads following loads
will not see stale will not see stale will not see stale will not see stale
global data. global data. global data. global data.
fence acq_rel - singlethread *none* *none* *none* fence acq_rel - singlethread *none* *none* *none*
- wavefront - wavefront
fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &
vmcnt(0) & vscnt(0) vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vmcnt(0) and mode, omit vmcnt(0) and
vscnt(0). vscnt(0).
- If OpenCL and - If OpenCL and - If OpenCL and - If OpenCL and
address space is address space is address space is address space is
not generic, omit. not generic, omit not generic, omit. not generic, omit
lgkmcnt(0). lgkmcnt(0).
- If OpenCL and - If OpenCL and
address space is address space is
local, omit local, omit
vmcnt(0) and vscnt(0). vmcnt(0) and vscnt(0).
- However, - However, - However, - However,
since LLVM since LLVM since LLVM since LLVM
currently has no currently has no currently has no currently has no
address space on address space on address space on address space on
the fence need to the fence need to the fence need to the fence need to
conservatively conservatively conservatively conservatively
always generate always generate always generate always generate
(see comment for (see comment for (see comment for (see comment for
previous fence). previous fence). previous fence). previous fence).
- Must happen after - Must happen after
any preceding any preceding
local/generic local/generic
load/load load/load
atomic/store/store atomic/store/store
atomic/atomicrmw. atomic/atomicrmw.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0), s_waitcnt vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
load/load load/load
atomic/ atomic/
atomicrmw-with-return-value. atomicrmw-with-return-value.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store atomic/ store/store atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after
any preceding any preceding
local/generic local/generic
load/store/load load/store/load
atomic/store atomic/ atomic/store atomic/
atomicrmw. atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
any following any following any following any following
global/generic global/generic global/generic global/generic
load/load load/load load/load load/load
atomic/store/store atomic/store/store atomic/store/store atomic/store/store
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
memory operations memory operations memory operations memory operations
to local have have to local have have
completed before completed before completed before completed before
performing any performing any performing any performing any
following global following global following global following global
memory operations. memory operations. memory operations. memory operations.
- Ensures that the - Ensures that the - Ensures that the - Ensures that the
preceding preceding preceding preceding
local/generic load local/generic load local/generic load local/generic load
atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw
with an equal or with an equal or with an equal or with an equal or
wider sync scope wider sync scope wider sync scope wider sync scope
and memory ordering and memory ordering and memory ordering and memory ordering
stronger than stronger than stronger than stronger than
unordered (this is unordered (this is unordered (this is unordered (this is
termed the termed the termed the termed the
acquire-fence-paired-atomic acquire-fence-paired-atomic acquire-fence-paired-atomic acquire-fence-paired-atomic
) has completed ) has completed ) has completed ) has completed
before following before following before following before following
global memory global memory global memory global memory
operations. This operations. This operations. This operations. This
satisfies the satisfies the satisfies the satisfies the
requirements of requirements of requirements of requirements of
acquire. acquire. acquire. acquire.
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
previous memory previous memory previous memory previous memory
operations have operations have operations have operations have
completed before a completed before a completed before a completed before a
following following following following
local/generic store local/generic store local/generic store local/generic store
atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw
with an equal or with an equal or with an equal or with an equal or
wider sync scope wider sync scope wider sync scope wider sync scope
and memory ordering and memory ordering and memory ordering and memory ordering
stronger than stronger than stronger than stronger than
unordered (this is unordered (this is unordered (this is unordered (this is
termed the termed the termed the termed the
release-fence-paired-atomic release-fence-paired-atomic release-fence-paired-atomic release-fence-paired-atomic
). This satisfies the ). This satisfies the ). This satisfies the ). This satisfies the
requirements of requirements of requirements of requirements of
release. release. release. release.
- Must happen before - Must happen before
the following the following
buffer_gl0_inv. buffer_gl0_inv.
- Ensures that the - Ensures that the
acquire-fence-paired acquire-fence-paired
atomic has completed atomic has completed
before invalidating before invalidating
the the
cache. Therefore cache. Therefore
any following any following
locations read must locations read must
be no older than be no older than
the value read by the value read by
the the
acquire-fence-paired-atomic. acquire-fence-paired-atomic.
3. buffer_gl0_inv 3. buffer_gl0_inv
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- Ensures that - Ensures that
following following
loads will not see loads will not see
stale data. stale data.
fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &
- system vmcnt(0) vmcnt(0) & vscnt(0) - system vmcnt(0) vmcnt(0) & vscnt(0)
- If OpenCL and - If OpenCL and - If OpenCL and - If OpenCL and
address space is address space is address space is address space is
not generic, omit not generic, omit not generic, omit not generic, omit
lgkmcnt(0). lgkmcnt(0). lgkmcnt(0). lgkmcnt(0).
- If OpenCL and - If OpenCL and
address space is address space is
local, omit local, omit
vmcnt(0) and vscnt(0). vmcnt(0) and vscnt(0).
- However, since LLVM - However, since LLVM - However, since LLVM - However, since LLVM
currently has no currently has no currently has no currently has no
address space on address space on address space on address space on
the fence need to the fence need to the fence need to the fence need to
conservatively conservatively conservatively conservatively
always generate always generate always generate always generate
(see comment for (see comment for (see comment for (see comment for
previous fence). previous fence). previous fence). previous fence).
- Could be split into - Could be split into - Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt separate s_waitcnt separate s_waitcnt
vmcnt(0) and vmcnt(0), s_waitcnt vmcnt(0) and vmcnt(0), s_waitcnt
s_waitcnt vscnt(0) and s_waitcnt s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be them to be them to be
independently moved independently moved independently moved independently moved
according to the according to the according to the according to the
following rules. following rules. following rules. following rules.
- s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0)
must happen after must happen after must happen after must happen after
any preceding any preceding any preceding any preceding
global/generic global/generic global/generic global/generic
load/store/load load/load load/store/load load/load
atomic/store atomic/ atomic/store atomic/
atomic/atomicrmw. atomicrmw-with-return-value. atomic/atomicrmw. atomicrmw-with-return-value.
- s_waitcnt vscnt(0) - s_waitcnt vscnt(0)
must happen after must happen after
any preceding any preceding
global/generic global/generic
store/store atomic/ store/store atomic/
atomicrmw-no-return-value. atomicrmw-no-return-value.
- s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0)
must happen after must happen after must happen after must happen after
any preceding any preceding any preceding any preceding
local/generic local/generic local/generic local/generic
load/store/load load/store/load load/store/load load/store/load
atomic/store atomic/store atomic/store atomic/store
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Must happen before - Must happen before - Must happen before - Must happen before
the following the following the following the following
buffer_wbinvl1_vol. buffer_gl*_inv. buffer_wbinvl1_vol. buffer_gl*_inv.
- Ensures that the - Ensures that the - Ensures that the - Ensures that the
preceding preceding preceding preceding
global/local/generic global/local/generic global/local/generic global/local/generic
load load load load
atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw
with an equal or with an equal or with an equal or with an equal or
wider sync scope wider sync scope wider sync scope wider sync scope
and memory ordering and memory ordering and memory ordering and memory ordering
stronger than stronger than stronger than stronger than
unordered (this is unordered (this is unordered (this is unordered (this is
termed the termed the termed the termed the
acquire-fence-paired-atomic acquire-fence-paired-atomic acquire-fence-paired-atomic acquire-fence-paired-atomic
) has completed ) has completed ) has completed ) has completed
before invalidating before invalidating before invalidating before invalidating
the cache. This the caches. This the cache. This the caches. This
satisfies the satisfies the satisfies the satisfies the
requirements of requirements of requirements of requirements of
acquire. acquire. acquire. acquire.
- Ensures that all - Ensures that all - Ensures that all - Ensures that all
previous memory previous memory previous memory previous memory
operations have operations have operations have operations have
completed before a completed before a completed before a completed before a
following following following following
global/local/generic global/local/generic global/local/generic global/local/generic
store store store store
atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw
with an equal or with an equal or with an equal or with an equal or
wider sync scope wider sync scope wider sync scope wider sync scope
and memory ordering and memory ordering and memory ordering and memory ordering
stronger than stronger than stronger than stronger than
unordered (this is unordered (this is unordered (this is unordered (this is
termed the termed the termed the termed the
release-fence-paired-atomic release-fence-paired-atomic release-fence-paired-atomic release-fence-paired-atomic
). This satisfies the ). This satisfies the ). This satisfies the ). This satisfies the
requirements of requirements of requirements of requirements of
release. release. release. release.
2. buffer_wbinvl1_vol 2. buffer_gl0_inv; 2. buffer_wbinvl1_vol 2. buffer_gl0_inv;
buffer_gl1_inv buffer_gl1_inv
- Must happen before - Must happen before - Must happen before - Must happen before
any following any following any following any following
global/generic global/generic global/generic global/generic
load/load load/load load/load load/load
atomic/store/store atomic/store/store atomic/store/store atomic/store/store
atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw. atomic/atomicrmw.
- Ensures that - Ensures that - Ensures that - Ensures that
following loads following loads following loads following loads
will not see stale will not see stale will not see stale will not see stale
global data. This global data. This global data. This global data. This
satisfies the satisfies the satisfies the satisfies the
requirements of requirements of requirements of requirements of
acquire. acquire. acquire. acquire.
**Sequential Consistent Atomic** **Sequential Consistent Atomic**
---------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------
load atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding load atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding
- wavefront - local load atomic acquire, load atomic acquire, - wavefront - local load atomic acquire, load atomic acquire,
- generic except must generated except must generated - generic except must generated except must generated
all instructions even all instructions even all instructions even all instructions even
for OpenCL.* for OpenCL.* for OpenCL.* for OpenCL.*
load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) &
- generic vmcnt(0) & vscnt(0) - generic vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit vmcnt(0) and mode, omit vmcnt(0) and
vscnt(0). vscnt(0).
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0), s_waitcnt vmcnt(0), s_waitcnt
vscnt(0) and s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- Must - waitcnt lgkmcnt(0) must - Must - waitcnt lgkmcnt(0) must
happen after happen after happen after happen after
preceding preceding preceding preceding
global/generic load local load global/generic load local load
atomic/store atomic/store atomic/store atomic/store
atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw
with memory with memory with memory with memory
ordering of seq_cst ordering of seq_cst ordering of seq_cst ordering of seq_cst
and with equal or and with equal or and with equal or and with equal or
wider sync scope. wider sync scope. wider sync scope. wider sync scope.
(Note that seq_cst (Note that seq_cst (Note that seq_cst (Note that seq_cst
fences have their fences have their fences have their fences have their
own s_waitcnt own s_waitcnt own s_waitcnt own s_waitcnt
lgkmcnt(0) and so do lgkmcnt(0) and so do lgkmcnt(0) and so do lgkmcnt(0) and so do
not need to be not need to be not need to be not need to be
considered.) considered.) considered.) considered.)
- waitcnt vmcnt(0) - waitcnt vmcnt(0)
Must happen after Must happen after
preceding preceding
global/generic load global/generic load
atomic/ atomic/
atomicrmw-with-return-value atomicrmw-with-return-value
with memory with memory
ordering of seq_cst ordering of seq_cst
and with equal or and with equal or
wider sync scope. wider sync scope.
(Note that seq_cst (Note that seq_cst
fences have their fences have their
own s_waitcnt own s_waitcnt
vmcnt(0) and so do vmcnt(0) and so do
not need to be not need to be
considered.) considered.)
- waitcnt vscnt(0) - waitcnt vscnt(0)
Must happen after Must happen after
preceding preceding
global/generic store global/generic store
atomic/ atomic/
atomicrmw-no-return-value atomicrmw-no-return-value
with memory with memory
ordering of seq_cst ordering of seq_cst
and with equal or and with equal or
wider sync scope. wider sync scope.
(Note that seq_cst (Note that seq_cst
fences have their fences have their
own s_waitcnt own s_waitcnt
vscnt(0) and so do vscnt(0) and so do
not need to be not need to be
considered.) considered.)
- Ensures any - Ensures any - Ensures any - Ensures any
preceding preceding preceding preceding
sequential sequential sequential sequential
consistent local consistent global/local consistent local consistent global/local
memory instructions memory instructions memory instructions memory instructions
have completed have completed have completed have completed
before executing before executing before executing before executing
this sequentially this sequentially this sequentially this sequentially
consistent consistent consistent consistent
instruction. This instruction. This instruction. This instruction. This
prevents reordering prevents reordering prevents reordering prevents reordering
a seq_cst store a seq_cst store a seq_cst store a seq_cst store
followed by a followed by a followed by a followed by a
seq_cst load. (Note seq_cst load. (Note seq_cst load. (Note seq_cst load. (Note
that seq_cst is that seq_cst is that seq_cst is that seq_cst is
stronger than stronger than stronger than stronger than
acquire/release as acquire/release as acquire/release as acquire/release as
the reordering of the reordering of the reordering of the reordering of
load acquire load acquire load acquire load acquire
followed by a store followed by a store followed by a store followed by a store
release is release is release is release is
prevented by the prevented by the prevented by the prevented by the
waitcnt of waitcnt of waitcnt of waitcnt of
the release, but the release, but the release, but the release, but
there is nothing there is nothing there is nothing there is nothing
preventing a store preventing a store preventing a store preventing a store
release followed by release followed by release followed by release followed by
load acquire from load acquire from load acquire from load acquire from
completing out of completing out of completing out of completing out of
order. The waitcnt order. The waitcnt order. The waitcnt order. The waitcnt
could be placed after could be placed after could be placed after could be placed after
seq_store or before seq_store or before seq_store or before seq_store or before
the seq_load. We the seq_load. We the seq_load. We the seq_load. We
choose the load to choose the load to choose the load to choose the load to
make the waitcnt be make the waitcnt be make the waitcnt be make the waitcnt be
as late as possible as late as possible as late as possible as late as possible
so that the store so that the store so that the store so that the store
may have already may have already may have already may have already
completed.) completed.) completed.) completed.)
2. *Following 2. *Following 2. *Following 2. *Following
instructions same as instructions same as instructions same as instructions same as
corresponding load corresponding load corresponding load corresponding load
atomic acquire, atomic acquire, atomic acquire, atomic acquire,
except must generated except must generated except must generated except must generated
all instructions even all instructions even all instructions even all instructions even
for OpenCL.* for OpenCL.* for OpenCL.* for OpenCL.*
load atomic seq_cst - workgroup - local *Same as corresponding load atomic seq_cst - workgroup - local *Same as corresponding
load atomic acquire, load atomic acquire,
except must generated except must generated
all instructions even all instructions even
for OpenCL.* for OpenCL.*
1. s_waitcnt vmcnt(0) & vscnt(0) 1. s_waitcnt vmcnt(0) & vscnt(0)
- If CU wavefront execution - If CU wavefront execution
mode, omit. mode, omit.
- Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt
vmcnt(0) and s_waitcnt vmcnt(0) and s_waitcnt
vscnt(0) to allow vscnt(0) to allow
them to be them to be
independently moved independently moved
according to the according to the
following rules. following rules.
- waitcnt vmcnt(0) - waitcnt vmcnt(0)
Must happen after Must happen after
preceding preceding
global/generic load global/generic load
atomic/ atomic/
atomicrmw-with-return-value atomicrmw-with-return-value
with memory with memory
ordering of seq_cst ordering of seq_cst
and with equal or and with equal or
wider sync scope. wider sync scope.
(Note that seq_cst (Note that seq_cst
fences have their fences have their
own s_waitcnt own s_waitcnt
vmcnt(0) and so do vmcnt(0) and so do
not need to be not need to be
considered.) considered.)
- waitcnt vscnt(0) - waitcnt vscnt(0)
Must happen after Must happen after
preceding preceding
global/generic store global/generic store
atomic/ atomic/
atomicrmw-no-return-value atomicrmw-no-return-value
with memory with memory
ordering of seq_cst ordering of seq_cst
and with equal or and with equal or
wider sync scope. wider sync scope.
(Note that seq_cst (Note that seq_cst
fences have their fences have their
own s_waitcnt own s_waitcnt
vscnt(0) and so do vscnt(0) and so do
not need to be not need to be
considered.) considered.)
- Ensures any - Ensures any
preceding preceding
sequential sequential
consistent global consistent global
memory instructions memory instructions
have completed have completed
before executing before executing
this sequentially this sequentially
consistent consistent
instruction. This instruction. This
prevents reordering prevents reordering
a seq_cst store a seq_cst store
followed by a followed by a
seq_cst load. (Note seq_cst load. (Note
that seq_cst is that seq_cst is
stronger than stronger than
acquire/release as acquire/release as
the reordering of the reordering of
load acquire load acquire
followed by a store followed by a store
release is release is
prevented by the prevented by the
waitcnt of waitcnt of
the release, but the release, but
there is nothing there is nothing
preventing a store preventing a store
release followed by release followed by
load acquire from load acquire from
completing out of completing out of
order. The waitcnt order. The waitcnt
could be placed after could be placed after
seq_store or before seq_store or before
the seq_load. We the seq_load. We
choose the load to choose the load to
make the waitcnt be make the waitcnt be
as late as possible as late as possible
so that the store so that the store
may have already may have already
completed.) completed.)
2. *Following 2. *Following
instructions same as instructions same as
corresponding load corresponding load
atomic acquire, atomic acquire,
except must generated except must generated
all instructions even all instructions even
for OpenCL.* for OpenCL.*
load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) &
- system - generic vmcnt(0) vmcnt(0) & vscnt(0) - system - generic vmcnt(0) vmcnt(0) & vscnt(0)
- Could be split into - Could be split into - Could be split into - Could be split into
separate s_waitcnt separate s_waitcnt separate s_waitcnt separate s_waitcnt
vmcnt(0) vmcnt(0), s_waitcnt vmcnt(0) vmcnt(0), s_waitcnt
and s_waitcnt vscnt(0) and s_waitcnt and s_waitcnt vscnt(0) and s_waitcnt
lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow lgkmcnt(0) to allow
them to be them to be them to be them to be
independently moved independently moved independently moved independently moved
according to the according to the according to the according to the
following rules. following rules. following rules. following rules.
- waitcnt lgkmcnt(0) - waitcnt lgkmcnt(0) - waitcnt lgkmcnt(0) - waitcnt lgkmcnt(0)
must happen after must happen after must happen after must happen after
preceding preceding preceding preceding
global/generic load local load global/generic load local load
atomic/store atomic/store atomic/store atomic/store
atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw atomic/atomicrmw
with memory with memory with memory with memory
ordering of seq_cst ordering of seq_cst ordering of seq_cst ordering of seq_cst
and with equal or and with equal or and with equal or and with equal or
wider sync scope. wider sync scope. wider sync scope. wider sync scope.
(Note that seq_cst (Note that seq_cst (Note that seq_cst (Note that seq_cst
fences have their fences have their fences have their fences have their
own s_waitcnt own s_waitcnt own s_waitcnt own s_waitcnt
lgkmcnt(0) and so do lgkmcnt(0) and so do lgkmcnt(0) and so do lgkmcnt(0) and so do
not need to be not need to be not need to be not need to be
considered.) considered.) considered.) considered.)
- waitcnt vmcnt(0) - waitcnt vmcnt(0) - waitcnt vmcnt(0) - waitcnt vmcnt(0)
must happen after must happen after must happen after must happen after
preceding preceding preceding preceding
global/generic load global/generic load global/generic load global/generic load
atomic/store atomic/ atomic/store atomic/
atomic/atomicrmw atomicrmw-with-return-value atomic/atomicrmw atomicrmw-with-return-value
with memory with memory with memory with memory
ordering of seq_cst ordering of seq_cst ordering of seq_cst ordering of seq_cst
and with equal or and with equal or and with equal or and with equal or
wider sync scope. wider sync scope. wider sync scope. wider sync scope.
(Note that seq_cst (Note that seq_cst (Note that seq_cst (Note that seq_cst
fences have their fences have their fences have their fences have their
own s_waitcnt own s_waitcnt own s_waitcnt own s_waitcnt
vmcnt(0) and so do vmcnt(0) and so do vmcnt(0) and so do vmcnt(0) and so do
not need to be not need to be not need to be not need to be
considered.) considered.) considered.) considered.)
- waitcnt vscnt(0) - waitcnt vscnt(0)
Must happen after Must happen after
preceding preceding
global/generic store global/generic store
atomic/ atomic/
atomicrmw-no-return-value atomicrmw-no-return-value
with memory with memory
ordering of seq_cst ordering of seq_cst
and with equal or and with equal or
wider sync scope. wider sync scope.
(Note that seq_cst (Note that seq_cst
fences have their fences have their
own s_waitcnt own s_waitcnt
vscnt(0) and so do vscnt(0) and so do
not need to be not need to be
considered.) considered.)
- Ensures any - Ensures any - Ensures any - Ensures any
preceding preceding preceding preceding
sequential sequential sequential sequential
consistent global consistent global consistent global consistent global
memory instructions memory instructions memory instructions memory instructions
have completed have completed have completed have completed
before executing before executing before executing before executing
this sequentially this sequentially this sequentially this sequentially
consistent consistent consistent consistent
instruction. This instruction. This instruction. This instruction. This
prevents reordering prevents reordering prevents reordering prevents reordering
a seq_cst store a seq_cst store a seq_cst store a seq_cst store
followed by a followed by a followed by a followed by a
seq_cst load. (Note seq_cst load. (Note seq_cst load. (Note seq_cst load. (Note
that seq_cst is that seq_cst is that seq_cst is that seq_cst is
stronger than stronger than stronger than stronger than
acquire/release as acquire/release as acquire/release as acquire/release as
the reordering of the reordering of the reordering of the reordering of
load acquire load acquire load acquire load acquire
followed by a store followed by a store followed by a store followed by a store
release is release is release is release is
prevented by the prevented by the prevented by the prevented by the
waitcnt of waitcnt of waitcnt of waitcnt of
the release, but the release, but the release, but the release, but
there is nothing there is nothing there is nothing there is nothing
preventing a store preventing a store preventing a store preventing a store
release followed by release followed by release followed by release followed by
load acquire from load acquire from load acquire from load acquire from
completing out of completing out of completing out of completing out of
order. The waitcnt order. The waitcnt order. The waitcnt order. The waitcnt
could be placed after could be placed after could be placed after could be placed after
seq_store or before seq_store or before seq_store or before seq_store or before
the seq_load. We the seq_load. We the seq_load. We the seq_load. We
choose the load to choose the load to choose the load to choose the load to
make the waitcnt be make the waitcnt be make the waitcnt be make the waitcnt be
as late as possible as late as possible as late as possible as late as possible
so that the store so that the store so that the store so that the store
may have already may have already may have already may have already
completed.) completed.) completed.) completed.)
2. *Following 2. *Following 2. *Following 2. *Following
instructions same as instructions same as instructions same as instructions same as
corresponding load corresponding load corresponding load corresponding load
atomic acquire, atomic acquire, atomic acquire, atomic acquire,
except must generated except must generated except must generated except must generated
all instructions even all instructions even all instructions even all instructions even
for OpenCL.* for OpenCL.* for OpenCL.* for OpenCL.*
store atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding store atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding
- wavefront - local store atomic release, store atomic release, - wavefront - local store atomic release, store atomic release,
- workgroup - generic except must generated except must generated - workgroup - generic except must generated except must generated
all instructions even all instructions even all instructions even all instructions even
for OpenCL.* for OpenCL.* for OpenCL.* for OpenCL.*
store atomic seq_cst - agent - global *Same as corresponding *Same as corresponding store atomic seq_cst - agent - global *Same as corresponding *Same as corresponding
- system - generic store atomic release, store atomic release, - system - generic store atomic release, store atomic release,
except must generated except must generated except must generated except must generated
all instructions even all instructions even all instructions even all instructions even
for OpenCL.* for OpenCL.* for OpenCL.* for OpenCL.*
atomicrmw seq_cst - singlethread - global *Same as corresponding *Same as corresponding atomicrmw seq_cst - singlethread - global *Same as corresponding *Same as corresponding
- wavefront - local atomicrmw acq_rel, atomicrmw acq_rel, - wavefront - local atomicrmw acq_rel, atomicrmw acq_rel,
- workgroup - generic except must generated except must generated - workgroup - generic except must generated except must generated
all instructions even all instructions even all instructions even all instructions even
for OpenCL.* for OpenCL.* for OpenCL.* for OpenCL.*
atomicrmw seq_cst - agent - global *Same as corresponding *Same as corresponding atomicrmw seq_cst - agent - global *Same as corresponding *Same as corresponding
- system - generic atomicrmw acq_rel, atomicrmw acq_rel, - system - generic atomicrmw acq_rel, atomicrmw acq_rel,
except must generated except must generated except must generated except must generated
all instructions even all instructions even all instructions even all instructions even
for OpenCL.* for OpenCL.* for OpenCL.* for OpenCL.*
fence seq_cst - singlethread *none* *Same as corresponding *Same as corresponding fence seq_cst - singlethread *none* *Same as corresponding *Same as corresponding
- wavefront fence acq_rel, fence acq_rel, - wavefront fence acq_rel, fence acq_rel,
- workgroup except must generated except must generated - workgroup except must generated except must generated
- agent all instructions even all instructions even - agent all instructions even all instructions even
- system for OpenCL.* for OpenCL.* - system for OpenCL.* for OpenCL.*
============ ============ ============== ========== =============================== ================================== ============ ============ ============== ========== ================================ ================================
The memory order also adds the single thread optimization constrains defined in The memory order also adds the single thread optimization constrains defined in
table table
:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table`. :ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table`.
.. table:: AMDHSA Memory Model Single Thread Optimization Constraints GFX6-GFX10 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints GFX6-GFX10
:name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table
▲ Show 20 Lines • Show All 1,473 Lines • Show Last 20 Lines