[AMDGPU] Omit unnecessary waitcnt before barriers

[kerbowa]

Summary

It is not necessary to wait for all outstanding memory operations before
barriers on hardware that can back off of the barrier in the event of an
exception when traps are enabled. Add a new subtarget feature which
tracks which HW has this ability.

[kerbowa]

The test waitcnt-vscnt.ll is kind of muddied by this patch. It was relying on the behavior of s_barrier always adding s_waitcnt_vscnt at barriers if there were any outstanding events. The singlethread fences were not actually doing anything. I've added workgroup fences instead. With workgroup fences we currently always add the waitcnt. The test will look as it did before after my memory legalizer/waitcnt overhaul so that these are optimized. Consider this a precommit of this test for now.

[foad]

In D120544#3344977, @kerbowa wrote:

The test waitcnt-vscnt.ll is kind of muddied by this patch. It was relying on the behavior of s_barrier always adding s_waitcnt_vscnt at barriers if there were any outstanding events. The singlethread fences were not actually doing anything. I've added workgroup fences instead. With workgroup fences we currently always add the waitcnt. The test will look as it did before after my memory legalizer/waitcnt overhaul so that these are optimized. Consider this a precommit of this test for now.

Can you split this into two patches: first change the test, and then rebase this patch?

[foad]

llvm/lib/Target/AMDGPU/GCNSubtarget.h

498

Stray " character

llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp

1139

Typo "subatarget"

llvm/test/CodeGen/AMDGPU/waitcnt-preexisting-vscnt.mir

38

Isn't this "waitcnt 1" redundant, since we have just done a "waitcnt 0" and then issued one store? Or am I completely misunderstanding?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPU.td

743

I don't understand what "backing off" means here

llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp

1143

Is this really a distinct feature if it's the same check as auto waitcnt?

[kerbowa]

llvm/lib/Target/AMDGPU/AMDGPU.td

743

Ya, I cannot think of a great name for this. Suggestions are welcome.

The SPG talks about it in these terms, but also uses "early barrier exit ".

The idea is that HW may need to "leave but not satisfy the barrier" to handle address-watch or MEMVIOL returned by in-flight memory ops.

llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp

1143

It's not the same as auto waitcnt. There are distinct subtargets that support each feature. This check is just saying we don't need an explicit waitcnt before barriers under any circumstances if there is an implicit wait by HW.

We don't actually use the auto waitcnt subtarget feature since it can be configured dynamically by HW. I was debating removing it entirely.

[rampitec]

llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp

1143

But then waitcount is still needed after the barrier? I.e. it does not mean that barrier wait for all outstanding memory operations, right?

[kerbowa]

llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp

1143

If auto waitcnt is enabled, I believe it does mean wait for all outstanding memory operations. However, the pass will still add redundant waitcnt after the barrier because the auto waitcnt feature is not optimized. But the feature is not used right now AFAIK.

If the subtarget has the "BackOffBarrier" feature, then waitcount are still needed after the barrier.

Normally barriers are paired with fences which may also enforce having waitcnt before the barrier. This change allows more flexibility and optimization in the cases where HW does not require waiting before barriers.

[kerbowa]

Fix typo and rebase.

[kerbowa]

llvm/test/CodeGen/AMDGPU/waitcnt-preexisting-vscnt.mir

38

You are right the waitcnt is redundant, but this is testing waitcnt which are inserted by the memory legalizer pass that are left "as is". This test will change when we attempt to optimize this.

[kerbowa]

Update test to use -mattr.

Forgot that there is a bug (feature?) where disabling a feature that
is in a "SubtargetFeatureGeneration" class disables ALL of the
features in that definition. Add BackOffBarrier to each gfx10
definition individually so that I can use -mattr in the test.

[rampitec]

LGTM, but could please also add it to gfx940?

[mceier]

This revision causes gpu hangups (Radeon 5700XT) on linux in many opengl apps, for example for godot I get something like this in dmesg:

[141585.653283] [drm:amdgpu_dm_commit_planes] *ERROR* Waiting for fences timed out!
[141587.702251] [drm:amdgpu_dm_commit_planes] *ERROR* Waiting for fences timed out!
[141590.783147] [drm:amdgpu_job_timedout] *ERROR* ring gfx_0.0.0 timeout, signaled seq=13399090, emitted seq=13399091
[141590.783726] [drm:amdgpu_job_timedout] *ERROR* Process information: process godot.x11.opt.t pid 527100 thread godot.x11.:cs0 pid 527120
[141590.783991] amdgpu 0000:03:00.0: amdgpu: GPU reset begin!
[141591.148991] amdgpu 0000:03:00.0: [drm:amdgpu_ring_test_helper] *ERROR* ring kiq_2.1.0 test failed (-110)
[141591.149000] [drm:gfx_v10_0_hw_fini] *ERROR* KGQ disable failed
[141591.334553] amdgpu 0000:03:00.0: [drm:amdgpu_ring_test_helper] *ERROR* ring kiq_2.1.0 test failed (-110)
[141591.334559] [drm:gfx_v10_0_hw_fini] *ERROR* KCQ disable failed
[141591.521657] [drm:gfx_v10_0_cp_gfx_enable.isra.0] *ERROR* failed to halt cp gfx
[141591.528453] [drm] free PSP TMR buffer
[141591.560524] CPU: 0 PID: 492277 Comm: kworker/u8:1 Not tainted 5.18.0-rc2-x86_64+ #1
[141591.560531] Hardware name: Gigabyte Technology Co., Ltd. Z170X-Gaming 7/Z170X-Gaming 7, BIOS F22j 01/11/2018
[141591.560534] Workqueue: amdgpu-reset-dev drm_sched_job_timedout
[141591.560540] Call Trace:
[141591.560541]  <TASK>
[141591.560543]  show_stack+0x52/0x58
[141591.560548]  dump_stack_lvl+0x49/0x5e
[141591.560551]  dump_stack+0x10/0x12
[141591.560553]  amdgpu_do_asic_reset+0x24/0x43b
[141591.560557]  amdgpu_device_gpu_recover_imp.cold+0x660/0x755
[141591.560560]  amdgpu_job_timedout+0x14f/0x180
[141591.560563]  ? set_next_entity+0xe1/0x160
[141591.560567]  drm_sched_job_timedout+0x6d/0x100
[141591.560569]  ? trace_hardirqs_on+0x37/0xf0
[141591.560573]  process_one_work+0x216/0x3f0
[141591.560576]  worker_thread+0x50/0x3e0
[141591.560578]  ? rescuer_thread+0x380/0x380
[141591.560580]  kthread+0xfc/0x120
[141591.560583]  ? kthread_complete_and_exit+0x20/0x20
[141591.560586]  ret_from_fork+0x22/0x30
[141591.560590]  </TASK>
[141591.560788] amdgpu 0000:03:00.0: amdgpu: BACO reset
[141593.710773] amdgpu 0000:03:00.0: amdgpu: GPU reset succeeded, trying to resume
...

git bisect log:

# bad: [6730b44480fcce18bfbbae0c46719250e9eae425] [HIP] Fix HIP include path
# good: [9b740c035c8b3e1a15effbfa73960fea259fd27d] Update normalizeAffineFor to canonicalize maps/operands before using them
git bisect start '--no-checkout' '6730b44480fcce18bfbbae0c46719250e9eae425' '9b740c035c8b3e1a15effbfa73960fea259fd27d' '--' 'llvm' 'cmake' 'third-party'
# bad: [9119eefe5fe513f188f8a2b8d5346ac0ce72d3f3] [X86] Add cheapX86FSETCC_SSE helper. NFC.
git bisect bad 9119eefe5fe513f188f8a2b8d5346ac0ce72d3f3
# bad: [f46fa4de4a95329b74ba0bb4ce9623b64ad84876] Revert "[clang][debug] port clang-cl /JMC flag to ELF"
git bisect bad f46fa4de4a95329b74ba0bb4ce9623b64ad84876
# bad: [0c2b43ab8cb1067dd1c7899094b824890803a7d2] [X86] Fix MCSymbolizer interface for X86Disassembler
git bisect bad 0c2b43ab8cb1067dd1c7899094b824890803a7d2
# bad: [e1069c1288d151f36f37c4c616b78b7b0a1e3a50] [AMDGPU] Ensure return address is save/restored if clobbered or when function has calls
git bisect bad e1069c1288d151f36f37c4c616b78b7b0a1e3a50
# good: [eadd1668d05df582281061460089562cfb6b3a90] update_analyze_test_checks.py: fix UTC_ARGS handling
git bisect good eadd1668d05df582281061460089562cfb6b3a90
# good: [20c4664552e2e5c1d85db13d3568b7d4a3e843ef] [gn build] Port 205557c908ff
git bisect good 20c4664552e2e5c1d85db13d3568b7d4a3e843ef
# bad: [8d0c34fd4fb66ea0d19563154a59658e4b7f35d4] [AMDGPU] Omit unnecessary waitcnt before barriers
git bisect bad 8d0c34fd4fb66ea0d19563154a59658e4b7f35d4
# first bad commit: [8d0c34fd4fb66ea0d19563154a59658e4b7f35d4] [AMDGPU] Omit unnecessary waitcnt before barriers

After reverting 8d0c34fd4fb66ea0d19563154a59658e4b7f35d4 on 4ffd0b6fde4da9a3ba4ee3a189504ce84c118b1c the gpu hangups disapear.

Steps to reproduce:

1. Start godot
2. Open project / create new opengl es project
3. Hangup happens either immediately or after few seconds of moving mouse

[arsenm]

In D120544#3456333, @mceier wrote:

This revision causes gpu hangups (Radeon 5700XT) on linux in many opengl apps, for example for godot I get something like this in dmesg:

[141585.653283] [drm:amdgpu_dm_commit_planes] *ERROR* Waiting for fences timed out!
[141587.702251] [drm:amdgpu_dm_commit_planes] *ERROR* Waiting for fences timed out!
[141590.783147] [drm:amdgpu_job_timedout] *ERROR* ring gfx_0.0.0 timeout, signaled seq=13399090, emitted seq=13399091
[141590.783726] [drm:amdgpu_job_timedout] *ERROR* Process information: process godot.x11.opt.t pid 527100 thread godot.x11.:cs0 pid 527120
[141590.783991] amdgpu 0000:03:00.0: amdgpu: GPU reset begin!
[141591.148991] amdgpu 0000:03:00.0: [drm:amdgpu_ring_test_helper] *ERROR* ring kiq_2.1.0 test failed (-110)
[141591.149000] [drm:gfx_v10_0_hw_fini] *ERROR* KGQ disable failed
[141591.334553] amdgpu 0000:03:00.0: [drm:amdgpu_ring_test_helper] *ERROR* ring kiq_2.1.0 test failed (-110)
[141591.334559] [drm:gfx_v10_0_hw_fini] *ERROR* KCQ disable failed
[141591.521657] [drm:gfx_v10_0_cp_gfx_enable.isra.0] *ERROR* failed to halt cp gfx
[141591.528453] [drm] free PSP TMR buffer
[141591.560524] CPU: 0 PID: 492277 Comm: kworker/u8:1 Not tainted 5.18.0-rc2-x86_64+ #1
[141591.560531] Hardware name: Gigabyte Technology Co., Ltd. Z170X-Gaming 7/Z170X-Gaming 7, BIOS F22j 01/11/2018
[141591.560534] Workqueue: amdgpu-reset-dev drm_sched_job_timedout
[141591.560540] Call Trace:
[141591.560541]  <TASK>
[141591.560543]  show_stack+0x52/0x58
[141591.560548]  dump_stack_lvl+0x49/0x5e
[141591.560551]  dump_stack+0x10/0x12
[141591.560553]  amdgpu_do_asic_reset+0x24/0x43b
[141591.560557]  amdgpu_device_gpu_recover_imp.cold+0x660/0x755
[141591.560560]  amdgpu_job_timedout+0x14f/0x180
[141591.560563]  ? set_next_entity+0xe1/0x160
[141591.560567]  drm_sched_job_timedout+0x6d/0x100
[141591.560569]  ? trace_hardirqs_on+0x37/0xf0
[141591.560573]  process_one_work+0x216/0x3f0
[141591.560576]  worker_thread+0x50/0x3e0
[141591.560578]  ? rescuer_thread+0x380/0x380
[141591.560580]  kthread+0xfc/0x120
[141591.560583]  ? kthread_complete_and_exit+0x20/0x20
[141591.560586]  ret_from_fork+0x22/0x30
[141591.560590]  </TASK>
[141591.560788] amdgpu 0000:03:00.0: amdgpu: BACO reset
[141593.710773] amdgpu 0000:03:00.0: amdgpu: GPU reset succeeded, trying to resume
...

git bisect log:

# bad: [6730b44480fcce18bfbbae0c46719250e9eae425] [HIP] Fix HIP include path
# good: [9b740c035c8b3e1a15effbfa73960fea259fd27d] Update normalizeAffineFor to canonicalize maps/operands before using them
git bisect start '--no-checkout' '6730b44480fcce18bfbbae0c46719250e9eae425' '9b740c035c8b3e1a15effbfa73960fea259fd27d' '--' 'llvm' 'cmake' 'third-party'
# bad: [9119eefe5fe513f188f8a2b8d5346ac0ce72d3f3] [X86] Add cheapX86FSETCC_SSE helper. NFC.
git bisect bad 9119eefe5fe513f188f8a2b8d5346ac0ce72d3f3
# bad: [f46fa4de4a95329b74ba0bb4ce9623b64ad84876] Revert "[clang][debug] port clang-cl /JMC flag to ELF"
git bisect bad f46fa4de4a95329b74ba0bb4ce9623b64ad84876
# bad: [0c2b43ab8cb1067dd1c7899094b824890803a7d2] [X86] Fix MCSymbolizer interface for X86Disassembler
git bisect bad 0c2b43ab8cb1067dd1c7899094b824890803a7d2
# bad: [e1069c1288d151f36f37c4c616b78b7b0a1e3a50] [AMDGPU] Ensure return address is save/restored if clobbered or when function has calls
git bisect bad e1069c1288d151f36f37c4c616b78b7b0a1e3a50
# good: [eadd1668d05df582281061460089562cfb6b3a90] update_analyze_test_checks.py: fix UTC_ARGS handling
git bisect good eadd1668d05df582281061460089562cfb6b3a90
# good: [20c4664552e2e5c1d85db13d3568b7d4a3e843ef] [gn build] Port 205557c908ff
git bisect good 20c4664552e2e5c1d85db13d3568b7d4a3e843ef
# bad: [8d0c34fd4fb66ea0d19563154a59658e4b7f35d4] [AMDGPU] Omit unnecessary waitcnt before barriers
git bisect bad 8d0c34fd4fb66ea0d19563154a59658e4b7f35d4
# first bad commit: [8d0c34fd4fb66ea0d19563154a59658e4b7f35d4] [AMDGPU] Omit unnecessary waitcnt before barriers

After reverting 8d0c34fd4fb66ea0d19563154a59658e4b7f35d4 on 4ffd0b6fde4da9a3ba4ee3a189504ce84c118b1c the gpu hangups disapear.

Steps to reproduce:

1. Start godot
2. Open project / create new opengl es project
3. Hangup happens either immediately or after few seconds of moving mouse

Can you post the IR and ISA for the failing shader?

[mceier]

In D120544#3457456, @arsenm wrote:

Can you post the IR and ISA for the failing shader?

I will try to get them this weekend. Before bisecting I tried running these programs with RADV_DEBUG=hang but that didn't produce any output.
I will try with MESA_GLSL=dump,log this weekend, hopefully it will capture something.

[kerbowa]

Reverted until I can investigate the failures.

[mceier]

In D120544#3457456, @arsenm wrote:

Can you post the IR and ISA for the failing shader?

I ran godot with MESA_SHADER_CACHE_DISABLE=true RADV_DEBUG=shaders AMD_DEBUG=vs,ps,preoptir environment variables set, are these logs enough:

• LLVM Without Revert (hangs GPU):
  llvm-without-revert-godot-mesa.log1 MBDownload
  dmesg:
  dmesg.log7 KBDownload
• LLVM With Revert:
  llvm-with-revert-godot-mesa.log1 MBDownload

Unfortunately I don't know how to pinpoint which exactly shader is at fault. I only noticed s_waitcnt is not emitted before s_barrier in the logs with hangup.

If these logs are not enough, I will probably need detailed instruction on how to get the IR/ISA files...
I know I can dump the shaders GLSL source by setting MESA_SHADER_DUMP_PATH, but I don't think these files are useful.

[nhaehnle]

I don't see how this change could possibly have ever been correct. Whether backoff is supported or not, in most cases there will be no backoff, and in that case, the waitcnt is required.

[nhaehnle]

Oh, I misunderstood what this patch does. It's possibly an unfortunate interaction between Mesa and LLVM, where Mesa assumed that the waitcnt was there, and your removal of it causes problems. In that case, can you please coordinate with Mesa to make sure it starts adding the required fence instructions in the LLVM IR? Cc @mareko

[mareko]

In D120544#3464886, @nhaehnle wrote:

Oh, I misunderstood what this patch does. It's possibly an unfortunate interaction between Mesa and LLVM, where Mesa assumed that the waitcnt was there, and your removal of it causes problems. In that case, can you please coordinate with Mesa to make sure it starts adding the required fence instructions in the LLVM IR? Cc @mareko

I'm interested in what to do in Mesa. Do we just need to add LLVMBuildFence(... LLVMAtomicOrderingRelease ...) before amdgcn.s.barrier?

[kerbowa]

I believe HIP adds a release fence before the barrier and an acquire directly after. Would something like that make sense?

[nhaehnle]

In D120544#3466638, @kerbowa wrote:

I believe HIP adds a release fence before the barrier and an acquire directly after. Would something like that make sense?

Yes, generally I expect that is what's required.

[mareko]

In D120544#3486402, @nhaehnle wrote:

In D120544#3466638, @kerbowa wrote:

I believe HIP adds a release fence before the barrier and an acquire directly after. Would something like that make sense?

Yes, generally I expect that is what's required.

What does the release fence add? Waiting for only LDS or everything?

[mareko]

We are going to insert amdgcn.s.waitcnt instead of a fence because fences wait for memory stores, which we don't want.

[kerbowa]

In D120544#3487040, @mareko wrote:

We are going to insert amdgcn.s.waitcnt instead of a fence because fences wait for memory stores, which we don't want.

Without this patch, the compiler will wait for everything, so I'm not sure how you will be differentiating between loads and stores.

What does the release fence add? Waiting for only LDS or everything?

It depends on the scope of the fence and the HW. See: https://llvm.org/docs/AMDGPUUsage.html#amdgpu-amdhsa-memory-model

[nhaehnle]

That document is pretty useless, I have to say. There is also a very obvious gap here in the expressiveness of the fence instruction which Marek implicitly pointed out: it should be possible to tell the fence which memory types should be affected. syncscope is not enough for that. @t-tye

We really need a change that allows us to put a "memory kind" modifier on fence instructions. This would correspond to the {Uniform,Workgroup,...}Memory bits in SPIR-V MemorySemantics fields.

[mareko]

The IR (NIR) that we translate to LLVM typically contains this sequence of instructions:

1. memory_barrier (fence)
2. control_barrier (s_barrier)

Either of them is optional, which means a memory barrier can occur without a control barrier, and vice versa. SPIR-V also works like this.

The memory barrier (fence) has the expressive power to wait for one or more of these:

• all shared memory opcodes
• only HS output shared memory opcodes
• all memory opcodes
• only global memory opcodes (not buffer or image)
• only buffer opcodes
• only image opcodes
• only buffer opcodes using shader storage buffer object variables (not other buffers even if aliased)
• only buffer/image opcodes using image variables (image variables can also access buffers, but shouldn't include any other buffers even if aliased)
• fences never wait for opcodes writing into write-only buffers
• fences never wait for opcodes reading from read-only buffers and images

What the LLVM fence can do:

• everything or nothing

[nhaehnle]

Thank you for the summary.

In D120544#3488714, @mareko wrote:

• only HS output shared memory opcodes

This one is tricky.

• all memory opcodes
• only global memory opcodes (not buffer or image)
• only buffer opcodes
• only image opcodes
• only buffer opcodes using shader storage buffer object variables (not other buffers if not aliased)
• only buffer/image opcodes using image variables (image variables can also access buffers, but shouldn't include any other buffers if not aliased)

I suppose the only difference this makes here is potentially in the counter value we wait for? I.e., we may wait for vmcnt(N) with N != 0?

• fences never wait for opcodes writing into write-only buffers
• fences never wait for opcodes reading from read-only buffers and images

IMO this one should be modeled like private/nonprivate in Vulkan. We had some discussions on this internally but not enough pressure to actually make it happen.

We should make all of this happen. I'm thinking in the direction of a memoryscopes operand, somewhat analogous to syncscope, except that while syncscope captures the set of threads we're potentially communicating with, memoryscope would capture the set of memoy through which we're potentially communicating.

[mareko]

I suppose the only difference this makes here is potentially in the counter value we wait for? I.e., we may wait for vmcnt(N) with N != 0?

LLVM IR can track everything accurately, and the compiler can choose what to do with it. The hw isn't that flexible, but it could either wait for nothing if that memory scope is not busy, vmcnt(N) with N > 0, or vmcnt(0).

I've changed the aliasing rules in my comment such that waiting for a memory scope shouldn't wait for any other memory scope even if they alias (the memory barrier is a user's choice).

[kerbowa]

JFYI, I'm planning to reintroduce this change in the next few weeks. Before this change, the compiler would ALWAYS wait for outstanding VMEM/LGKM at barriers. There is no HW requirement for this. As an optimization, we will be omitting these waitcnt on Navi/MI200. In order to retain the same behavior as before the change, fences must be added before barriers.

llvm/lib/Target/AMDGPU/AMDGPU.td

 
 def FeatureAutoWaitcntBeforeBarrier : SubtargetFeature <
   "auto-waitcnt-before-barrier",
   "AutoWaitcntBeforeBarrier",
   "true",
   "Hardware automatically inserts waitcnt before barrier"
 >;
 
+def FeatureBackOffBarrier : SubtargetFeature <"back-off-barrier",
+  "BackOffBarrier",
+  "true",
+  "Hardware supports backing off s_barrier if an exception occurs"
+>;
+
 def FeatureTrigReducedRange : SubtargetFeature<"trig-reduced-range",
   "HasTrigReducedRange",
   "true",
   "Requires use of fract on arguments to trig instructions"
 >;
 
 // Alignment enforcement is controlled by a configuration register:
 // SH_MEM_CONFIG.alignment_mode
def FeatureISAVersion9_0_A : FeatureSet<
    FeaturePackedFP32Ops,
    FeatureMAIInsts,
    FeaturePkFmacF16Inst,
    FeatureAtomicFaddInsts,
    FeatureImageInsts,
    FeatureMadMacF32Insts,
    FeatureSupportsSRAMECC,
    FeaturePackedTID,
-   FullRate64Ops]>;
+   FullRate64Ops,
+   FeatureBackOffBarrier]>;
 
 def FeatureISAVersion9_0_C : FeatureSet<
   [FeatureGFX9,
    FeatureMadMixInsts,
    FeatureLDSBankCount32,
    FeatureDsSrc2Insts,
    FeatureExtendedImageInsts,
    FeatureImageInsts,
def FeatureISAVersion9_4_0 : FeatureSet<
    Feature64BitDPP,
    FeaturePackedFP32Ops,
    FeatureMAIInsts,
    FeaturePkFmacF16Inst,
    FeatureAtomicFaddInsts,
    FeatureSupportsSRAMECC,
    FeaturePackedTID,
    FeatureArchitectedFlatScratch,
-   FullRate64Ops]>;
+   FullRate64Ops,
+   FeatureBackOffBarrier]>;
 
 // TODO: Organize more features into groups.
 def FeatureGroup {
   // Bugs present on gfx10.1.
   list<SubtargetFeature> GFX10_1_Bugs = [
     FeatureVcmpxPermlaneHazard,
     FeatureVMEMtoScalarWriteHazard,
     FeatureSMEMtoVectorWriteHazard,
!listconcat(FeatureGroup.GFX10_1_Bugs,
      FeatureWavefrontSize32,
      FeatureScalarStores,
      FeatureScalarAtomics,
      FeatureScalarFlatScratchInsts,
      FeatureGetWaveIdInst,
      FeatureMadMacF32Insts,
      FeatureDsSrc2Insts,
      FeatureLdsMisalignedBug,
-     FeatureSupportsXNACK])>;
+     FeatureSupportsXNACK,
+     FeatureBackOffBarrier])>;
 
 def FeatureISAVersion10_1_1 : FeatureSet<
   !listconcat(FeatureGroup.GFX10_1_Bugs,
     [FeatureGFX10,
      FeatureLDSBankCount32,
      FeatureDLInsts,
      FeatureDot1Insts,
      FeatureDot2Insts,
      FeatureDot5Insts,
      FeatureDot6Insts,
      FeatureDot7Insts,
      FeatureNSAEncoding,
      FeatureNSAMaxSize5,
      FeatureWavefrontSize32,
      FeatureScalarStores,
      FeatureScalarAtomics,
      FeatureScalarFlatScratchInsts,
      FeatureGetWaveIdInst,
      FeatureMadMacF32Insts,
      FeatureDsSrc2Insts,
      FeatureLdsMisalignedBug,
-     FeatureSupportsXNACK])>;
+     FeatureSupportsXNACK,
+     FeatureBackOffBarrier])>;
 
 def FeatureISAVersion10_1_2 : FeatureSet<
   !listconcat(FeatureGroup.GFX10_1_Bugs,
     [FeatureGFX10,
      FeatureLDSBankCount32,
      FeatureDLInsts,
      FeatureDot1Insts,
      FeatureDot2Insts,
      FeatureDot5Insts,
      FeatureDot6Insts,
      FeatureDot7Insts,
      FeatureNSAEncoding,
      FeatureNSAMaxSize5,
      FeatureWavefrontSize32,
      FeatureScalarStores,
      FeatureScalarAtomics,
      FeatureScalarFlatScratchInsts,
      FeatureGetWaveIdInst,
      FeatureMadMacF32Insts,
      FeatureDsSrc2Insts,
      FeatureLdsMisalignedBug,
-     FeatureSupportsXNACK])>;
+     FeatureSupportsXNACK,
+     FeatureBackOffBarrier])>;
 
 def FeatureISAVersion10_1_3 : FeatureSet<
   !listconcat(FeatureGroup.GFX10_1_Bugs,
     [FeatureGFX10,
      FeatureGFX10_AEncoding,
      FeatureLDSBankCount32,
      FeatureDLInsts,
      FeatureNSAEncoding,
      FeatureNSAMaxSize5,
      FeatureWavefrontSize32,
      FeatureScalarStores,
      FeatureScalarAtomics,
      FeatureScalarFlatScratchInsts,
      FeatureGetWaveIdInst,
      FeatureMadMacF32Insts,
      FeatureDsSrc2Insts,
      FeatureLdsMisalignedBug,
-     FeatureSupportsXNACK])>;
+     FeatureSupportsXNACK,
+     FeatureBackOffBarrier])>;
 
 def FeatureISAVersion10_3_0 : FeatureSet<
   [FeatureGFX10,
    FeatureGFX10_AEncoding,
    FeatureGFX10_BEncoding,
    FeatureGFX10_3Insts,
    FeatureLDSBankCount32,
    FeatureDLInsts,
    FeatureDot1Insts,
    FeatureDot2Insts,
    FeatureDot5Insts,
    FeatureDot6Insts,
    FeatureDot7Insts,
    FeatureNSAEncoding,
    FeatureNSAMaxSize13,
    FeatureWavefrontSize32,
-   FeatureShaderCyclesRegister]>;
+   FeatureShaderCyclesRegister,
+   FeatureBackOffBarrier]>;
 
 //===----------------------------------------------------------------------===//
 
 def AMDGPUInstrInfo : InstrInfo {
   let guessInstructionProperties = 1;
   let noNamedPositionallyEncodedOperands = 1;
 }
 

llvm/lib/Target/AMDGPU/GCNSubtarget.h

protected:
   bool FastFMAF32;
   bool FastDenormalF32;
   bool HalfRate64Ops;
   bool FullRate64Ops;
 
   // Dynamically set bits that enable features.
   bool FlatForGlobal;
   bool AutoWaitcntBeforeBarrier;
+  bool BackOffBarrier;
   bool UnalignedScratchAccess;
   bool UnalignedAccessMode;
   bool HasApertureRegs;
   bool SupportsXNACK;
 
   // This should not be used directly. 'TargetID' tracks the dynamic settings
   // for XNACK.
   bool EnableXNACK;
public:
   bool usePRTStrictNull() const {
     return EnablePRTStrictNull;
   }
 
   bool hasAutoWaitcntBeforeBarrier() const {
     return AutoWaitcntBeforeBarrier;
   }
 
+  /// \returns true if the target supports backing off of s_barrier instructions
+  /// when an exception is raised.
+  bool supportsBackOffBarrier() const {
+    return BackOffBarrier;
+  }
+
   bool hasUnalignedBufferAccess() const {
     return UnalignedBufferAccess;
   }
 
   bool hasUnalignedBufferAccessEnabled() const {
     return UnalignedBufferAccess && UnalignedAccessMode;
   }
 

llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp

if (MI.isCall() && callWaitsOnFunctionEntry(MI)) {
           }
           ScoreBrackets.determineWait(
               LGKM_CNT, ScoreBrackets.getRegScore(RegNo, LGKM_CNT), Wait);
         }
       }
     }
   }
 
-  // Check to see if this is an S_BARRIER, and if an implicit S_WAITCNT 0
-  // occurs before the instruction. Doing it here prevents any additional
-  // S_WAITCNTs from being emitted if the instruction was marked as
-  // requiring a WAITCNT beforehand.
+  // The subtarget may have an implicit S_WAITCNT 0 before barriers. If it does
+  // not, we need to ensure the subtarget is capable of backing off barrier
+  // instructions in case there are any outstanding memory operations that may
+  // cause an exception. Otherwise, insert an explicit S_WAITCNT 0 here.
   if (MI.getOpcode() == AMDGPU::S_BARRIER &&
-      !ST->hasAutoWaitcntBeforeBarrier()) {
+      !ST->hasAutoWaitcntBeforeBarrier() && !ST->supportsBackOffBarrier()) {
     Wait = Wait.combined(AMDGPU::Waitcnt::allZero(ST->hasVscnt()));
   }
 
   // TODO: Remove this work-around, enable the assert for Bug 457939
   //       after fixing the scheduler. Also, the Shader Compiler code is
   //       independent of target.
   if (readsVCCZ(MI) && ST->hasReadVCCZBug()) {
     if (ScoreBrackets.getScoreLB(LGKM_CNT) <

llvm/test/CodeGen/AMDGPU/back-off-barrier-subtarget-feature.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck --check-prefixes=GFX9-NO-BACKOFF %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a -verify-machineinstrs < %s | FileCheck --check-prefixes=GFX9-BACKOFF %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx940 -verify-machineinstrs < %s | FileCheck --check-prefixes=GFX9-BACKOFF %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a -mattr=-back-off-barrier -verify-machineinstrs < %s | FileCheck --check-prefixes=GFX9-NO-BACKOFF %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck --check-prefixes=GFX10-BACKOFF %s
+
+; Subtargets must wait for outstanding memory instructions before a barrier if
+; they cannot back off of the barrier.
+
+define void @back_off_barrier_no_fence(i32* %in, i32* %out) #0 {
+; GFX9-NO-BACKOFF-LABEL: back_off_barrier_no_fence:
+; GFX9-NO-BACKOFF:       ; %bb.0:
+; GFX9-NO-BACKOFF-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NO-BACKOFF-NEXT:    flat_load_dword v0, v[0:1]
+; GFX9-NO-BACKOFF-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX9-NO-BACKOFF-NEXT:    s_barrier
+; GFX9-NO-BACKOFF-NEXT:    flat_store_dword v[2:3], v0
+; GFX9-NO-BACKOFF-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX9-NO-BACKOFF-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9-BACKOFF-LABEL: back_off_barrier_no_fence:
+; GFX9-BACKOFF:       ; %bb.0:
+; GFX9-BACKOFF-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-BACKOFF-NEXT:    flat_load_dword v0, v[0:1]
+; GFX9-BACKOFF-NEXT:    s_barrier
+; GFX9-BACKOFF-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX9-BACKOFF-NEXT:    flat_store_dword v[2:3], v0
+; GFX9-BACKOFF-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX9-BACKOFF-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10-BACKOFF-LABEL: back_off_barrier_no_fence:
+; GFX10-BACKOFF:       ; %bb.0:
+; GFX10-BACKOFF-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10-BACKOFF-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-BACKOFF-NEXT:    flat_load_dword v0, v[0:1]
+; GFX10-BACKOFF-NEXT:    s_barrier
+; GFX10-BACKOFF-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX10-BACKOFF-NEXT:    flat_store_dword v[2:3], v0
+; GFX10-BACKOFF-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-BACKOFF-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-BACKOFF-NEXT:    s_setpc_b64 s[30:31]
+  %load = load i32, i32* %in
+  call void @llvm.amdgcn.s.barrier()
+  store i32 %load, i32* %out
+  ret void
+}
+
+define void @back_off_barrier_with_fence(i32* %in, i32* %out) #0 {
+; GFX9-NO-BACKOFF-LABEL: back_off_barrier_with_fence:
+; GFX9-NO-BACKOFF:       ; %bb.0:
+; GFX9-NO-BACKOFF-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NO-BACKOFF-NEXT:    flat_load_dword v0, v[0:1]
+; GFX9-NO-BACKOFF-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX9-NO-BACKOFF-NEXT:    s_barrier
+; GFX9-NO-BACKOFF-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NO-BACKOFF-NEXT:    flat_store_dword v[2:3], v0
+; GFX9-NO-BACKOFF-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX9-NO-BACKOFF-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9-BACKOFF-LABEL: back_off_barrier_with_fence:
+; GFX9-BACKOFF:       ; %bb.0:
+; GFX9-BACKOFF-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-BACKOFF-NEXT:    flat_load_dword v0, v[0:1]
+; GFX9-BACKOFF-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-BACKOFF-NEXT:    s_barrier
+; GFX9-BACKOFF-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX9-BACKOFF-NEXT:    flat_store_dword v[2:3], v0
+; GFX9-BACKOFF-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX9-BACKOFF-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX10-BACKOFF-LABEL: back_off_barrier_with_fence:
+; GFX10-BACKOFF:       ; %bb.0:
+; GFX10-BACKOFF-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10-BACKOFF-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-BACKOFF-NEXT:    flat_load_dword v0, v[0:1]
+; GFX10-BACKOFF-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX10-BACKOFF-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-BACKOFF-NEXT:    s_barrier
+; GFX10-BACKOFF-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX10-BACKOFF-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-BACKOFF-NEXT:    buffer_gl0_inv
+; GFX10-BACKOFF-NEXT:    flat_store_dword v[2:3], v0
+; GFX10-BACKOFF-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-BACKOFF-NEXT:    s_waitcnt_vscnt null, 0x0
+; GFX10-BACKOFF-NEXT:    s_setpc_b64 s[30:31]
+  %load = load i32, i32* %in
+  fence syncscope("workgroup") release
+  call void @llvm.amdgcn.s.barrier()
+  fence syncscope("workgroup") acquire
+  store i32 %load, i32* %out
+  ret void
+}
+
+declare void @llvm.amdgcn.s.barrier()
+
+attributes #0 = { nounwind }

llvm/test/CodeGen/AMDGPU/waitcnt-preexisting-vscnt.mir

 body:             |
   bb.0:
     liveins: $vgpr0_vgpr1, $vgpr2
 
     ; GFX10-LABEL: name: test_waitcnt_preexisting_vscnt_needs_vscnt
     ; GFX10: S_WAITCNT 0
     ; GFX10: S_WAITCNT_VSCNT undef $sgpr_null, 0
     ; GFX10: GLOBAL_STORE_DWORD $vgpr0_vgpr1, $vgpr2, 0, 0, implicit $exec
-    ; GFX10: S_WAITCNT_VSCNT undef $sgpr_null, 0
+    ; GFX10: S_WAITCNT_VSCNT undef $sgpr_null, 1
     ; GFX10: S_BARRIER
     ; GFX10: $vgpr0 = FLAT_LOAD_DWORD $vgpr0_vgpr1, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX10: S_WAITCNT 112
     ; GFX10: FLAT_STORE_DWORD $vgpr0_vgpr1, $vgpr0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX10: S_ENDPGM 0
     GLOBAL_STORE_DWORD $vgpr0_vgpr1, $vgpr2, 0, 0, implicit $exec
     S_WAITCNT_VSCNT undef $sgpr_null, 1
     S_BARRIER
body:             |
   bb.0:
     liveins: $vgpr0_vgpr1, $vgpr2
 
     ; GFX10-LABEL: name: test_waitcnt_preexisting_vscnt_combined_both_types
     ; GFX10: S_WAITCNT 0
     ; GFX10: S_WAITCNT_VSCNT undef $sgpr_null, 0
     ; GFX10: GLOBAL_STORE_DWORD $vgpr0_vgpr1, $vgpr2, 0, 0, implicit $exec
     ; GFX10: S_WAITCNT 0
-    ; GFX10: S_WAITCNT_VSCNT undef $sgpr_null, 0
+    ; GFX10: S_WAITCNT_VSCNT undef $sgpr_null, 1
     ; GFX10: S_BARRIER
     ; GFX10: $vgpr0 = FLAT_LOAD_DWORD $vgpr0_vgpr1, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX10: S_WAITCNT 112
     ; GFX10: FLAT_STORE_DWORD $vgpr0_vgpr1, $vgpr0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX10: S_ENDPGM 0
     GLOBAL_STORE_DWORD $vgpr0_vgpr1, $vgpr2, 0, 0, implicit $exec
     S_WAITCNT 0
     S_WAITCNT_VSCNT undef $sgpr_null, 1
     S_WAITCNT 0
     S_WAITCNT_VSCNT undef $sgpr_null, 2
     S_WAITCNT 0
     S_BARRIER
     $vgpr0 = FLAT_LOAD_DWORD $vgpr0_vgpr1, 0, 0, implicit $exec, implicit $flat_scr
     FLAT_STORE_DWORD $vgpr0_vgpr1, $vgpr0, 0, 0, implicit $exec, implicit $flat_scr
     S_ENDPGM 0
 ...

llvm/test/CodeGen/AMDGPU/waitcnt-vscnt.ll

 ; RUN: llc -march=amdgcn -mcpu=gfx802  -asm-verbose=0 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX8,GFX8_9 %s
 ; RUN: llc -march=amdgcn -mcpu=gfx900  -asm-verbose=0 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX9,GFX9_10,GFX8_9 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 -asm-verbose=0 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX10,GFX9_10 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-back-off-barrier -asm-verbose=0 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX10,GFX9_10 %s
 
 ; GCN-LABEL: barrier_vmcnt_global:
 ; GFX8:         flat_load_dword
 ; GFX9_10:      global_load_dword
 ; GFX8:         s_waitcnt vmcnt(0){{$}}
 ; GFX9_10:      s_waitcnt vmcnt(0){{$}}
 ; GCN-NEXT:     s_barrier
 define amdgpu_kernel void @barrier_vmcnt_global(i32 addrspace(1)* %arg) {
bb:
   %tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
   %tmp1 = zext i32 %tmp to i64
   %tmp2 = shl nuw nsw i64 %tmp1, 32
   %tmp3 = add nuw nsw i64 %tmp2, 8589934592
   %tmp4 = lshr exact i64 %tmp3, 32
   %tmp5 = getelementptr inbounds i32, i32 addrspace(1)* %arg, i64 %tmp4
   store i32 0, i32 addrspace(1)* %tmp5, align 4
   fence syncscope("singlethread") release
-  tail call void @llvm.amdgcn.s.barrier() #3
+  tail call void @llvm.amdgcn.s.barrier()
   fence syncscope("singlethread") acquire
   %tmp6 = add nuw nsw i64 %tmp2, 4294967296
   %tmp7 = lshr exact i64 %tmp6, 32
   %tmp8 = getelementptr inbounds i32, i32 addrspace(1)* %arg, i64 %tmp7
   store i32 1, i32 addrspace(1)* %tmp8, align 4
   ret void
 }
 
bb:
   %tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
   %tmp1 = zext i32 %tmp to i64
   %tmp2 = shl nuw nsw i64 %tmp1, 32
   %tmp3 = add nuw nsw i64 %tmp2, 8589934592
   %tmp4 = lshr exact i64 %tmp3, 32
   %tmp5 = getelementptr inbounds i32, i32* %arg, i64 %tmp4
   store i32 0, i32* %tmp5, align 4
   fence syncscope("singlethread") release
-  tail call void @llvm.amdgcn.s.barrier() #3
+  tail call void @llvm.amdgcn.s.barrier()
   fence syncscope("singlethread") acquire
   %tmp6 = add nuw nsw i64 %tmp2, 4294967296
   %tmp7 = lshr exact i64 %tmp6, 32
   %tmp8 = getelementptr inbounds i32, i32* %arg, i64 %tmp7
   store i32 1, i32* %tmp8, align 4
   ret void
 }