AMDGPU: Implement memory model

[t-tye]

lib/Target/AMDGPU/SIMemoryLegalizer.cpp

278

Is this needed only if the the instruction is a VMEM or FLAT instruction, not if a DS? It is only ensuring that the load has completed before doing the VMEM invalidate.

[t-tye]

lib/Target/AMDGPU/SIMemoryLegalizer.cpp

348

Should this be done? For rmw the glc bit controls whether the original value is returned, not whether the L1 cache is bypassed.

398

Ditto.

420

Ditto.

[t-tye]

lib/Target/AMDGPU/SIMemoryLegalizer.cpp

362

Is this required if a DS memory operation? Seems it is only required if a VMEM or FLAT instruction to ensure it has completed before invalidating the cache.

409

Is this required if a DS memory operation? Seems it is only required if a VMEM or FLAT instruction to ensure it has completed before invalidating the cache.

[arsenm]

test/CodeGen/AMDGPU/global_atomics.ll

1026–1027

Why does this lose the glc bit?

[arsenm]

lib/Target/AMDGPU/SIInstructions.td

105

There's an SPseduoInst or something like that which will avoid needing to set any of these bits

109

I don't think this is necessary, only mayLoad and mayStore

lib/Target/AMDGPU/SIMemoryLegalizer.cpp

195–196

If there are no memory operands, this should still work and be handled as conservatively as possible

[t-tye]

lib/Target/AMDGPU/SIMemoryLegalizer.cpp

234

I believe a waitcnt vmem(0) is required before the InsertBufferWbinvl1Vol to ensure any previous atomic load has completed that the fence will pair with to create a synchronizes-with relation.

[kzhuravl]

Update implementation.

[kzhuravl]

lib/Target/AMDGPU/SIInstructions.td

105

Already committed separately with comments taken care of.

109

Already committed separately with comments taken care of.

lib/Target/AMDGPU/SIMemoryLegalizer.cpp

195–196

Do you see an issue with this approach?

278

As discussed, this will be done in separate change.

348

As discussed, this will be done in separate change.

362

As discussed, this will be done in separate change.

398

As discussed, this will be done in separate change.

409

As discussed, this will be done in separate change.

420

As discussed, this will be done in separate change.

test/CodeGen/AMDGPU/global_atomics.ll

1026–1027

I think it does not need to be set in this case.

[arsenm]

lib/Target/AMDGPU/SIMemoryLegalizer.cpp

195–196

Yes. If there are no memory operands, this could skip an atomic load.

[t-tye]

lib/Target/AMDGPU/SIMemoryLegalizer.cpp

328

May be worth asserting that FailureOrdering is not AtomicOrdering::Release or AtomicOrdering::AcquireRelease as these are not allowed, and following code relies on that fact.

test/CodeGen/AMDGPU/flat_atomics.ll

1001–1002

Curious why glc is no longer being checked for?

[kzhuravl]

Address review feedback.

lib/Target/AMDGPU/SIMemoryLegalizer.cpp

328

Verifier takes care of those.

test/CodeGen/AMDGPU/flat_atomics.ll

1001–1002

Not required here. Discussed offline.

[t-tye]

One question, otherwise LGTM.

lib/Target/AMDGPU/SIInstructions.td

102

Since a fence itself does not modify memory, and has no memory address operand, should it be marked maybeatomic?

[kzhuravl]

Fix atomic fence expansion for workgroup/wavefront/singlethread scopes.

[t-tye]

LGTM

[kzhuravl]

rL308779, rL308781

lib/Target/AMDGPU/AMDGPU.h

 FunctionPass *createSIFoldOperandsPass();
 FunctionPass *createSIPeepholeSDWAPass();
 FunctionPass *createSILowerI1CopiesPass();
 FunctionPass *createSIShrinkInstructionsPass();
 FunctionPass *createSILoadStoreOptimizerPass();
 FunctionPass *createSIWholeQuadModePass();
 FunctionPass *createSIFixControlFlowLiveIntervalsPass();
 FunctionPass *createSIFixSGPRCopiesPass();
+FunctionPass *createSIMemoryLegalizerPass();
 FunctionPass *createSIDebuggerInsertNopsPass();
 FunctionPass *createSIInsertWaitsPass();
 FunctionPass *createSIInsertWaitcntsPass();
 FunctionPass *createAMDGPUCodeGenPreparePass();
 FunctionPass *createAMDGPUMachineCFGStructurizerPass();
 
 void initializeAMDGPUMachineCFGStructurizerPass(PassRegistry&);
 extern char &AMDGPUMachineCFGStructurizerID;

 extern char &AMDGPUAnnotateUniformValuesPassID;
 
 void initializeAMDGPUCodeGenPreparePass(PassRegistry&);
 extern char &AMDGPUCodeGenPrepareID;
 
 void initializeSIAnnotateControlFlowPass(PassRegistry&);
 extern char &SIAnnotateControlFlowPassID;
 
+void initializeSIMemoryLegalizerPass(PassRegistry&);
+extern char &SIMemoryLegalizerID;
+
 void initializeSIDebuggerInsertNopsPass(PassRegistry&);
 extern char &SIDebuggerInsertNopsID;
 
 void initializeSIInsertWaitsPass(PassRegistry&);
 extern char &SIInsertWaitsID;
 
 void initializeSIInsertWaitcntsPass(PassRegistry&);
 extern char &SIInsertWaitcntsID;

lib/Target/AMDGPU/AMDGPUMachineModuleInfo.h

+//===--- AMDGPUMachineModuleInfo.h ------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file
+/// \brief AMDGPU Machine Module Info.
+///
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIB_TARGET_AMDGPU_AMDGPUMACHINEMODULEINFO_H
+#define LLVM_LIB_TARGET_AMDGPU_AMDGPUMACHINEMODULEINFO_H
+
+#include "llvm/CodeGen/MachineModuleInfo.h"
+#include "llvm/CodeGen/MachineModuleInfoImpls.h"
+#include "llvm/IR/LLVMContext.h"
+
+namespace llvm {
+
+class AMDGPUMachineModuleInfo final : public MachineModuleInfoELF {
+private:
+
+  // All supported memory/synchronization scopes can be found here:
+  //   http://llvm.org/docs/AMDGPUUsage.html#memory-scopes
+
+  /// \brief Agent synchronization scope ID.
+  SyncScope::ID AgentSSID;
+  /// \brief Workgroup synchronization scope ID.
+  SyncScope::ID WorkgroupSSID;
+  /// \brief Wavefront synchronization scope ID.
+  SyncScope::ID WavefrontSSID;
+
+public:
+  AMDGPUMachineModuleInfo(const MachineModuleInfo &MMI);
+
+  /// \returns Agent synchronization scope ID.
+  SyncScope::ID getAgentSSID() const {
+    return AgentSSID;
+  }
+  /// \returns Workgroup synchronization scope ID.
+  SyncScope::ID getWorkgroupSSID() const {
+    return WorkgroupSSID;
+  }
+  /// \returns Wavefront synchronization scope ID.
+  SyncScope::ID getWavefrontSSID() const {
+    return WavefrontSSID;
+  }
+};
+
+} // end namespace llvm
+
+#endif // LLVM_LIB_TARGET_AMDGPU_AMDGPUMACHINEMODULEINFO_H

lib/Target/AMDGPU/AMDGPUMachineModuleInfo.cpp

+//===--- AMDGPUMachineModuleInfo.cpp ----------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file
+/// \brief AMDGPU Machine Module Info.
+///
+//
+//===----------------------------------------------------------------------===//
+
+#include "AMDGPUMachineModuleInfo.h"
+#include "llvm/IR/Module.h"
+
+namespace llvm {
+
+AMDGPUMachineModuleInfo::AMDGPUMachineModuleInfo(const MachineModuleInfo &MMI)
+    : MachineModuleInfoELF(MMI) {
+  LLVMContext &CTX = MMI.getModule()->getContext();
+  AgentSSID = CTX.getOrInsertSyncScopeID("agent");
+  WorkgroupSSID = CTX.getOrInsertSyncScopeID("workgroup");
+  WavefrontSSID = CTX.getOrInsertSyncScopeID("wavefront");
+}
+
+} // end namespace llvm

lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

extern "C" void LLVMInitializeAMDGPUTarget() {
   initializeAMDGPUCodeGenPreparePass(*PR);
   initializeAMDGPUUnifyMetadataPass(*PR);
   initializeSIAnnotateControlFlowPass(*PR);
   initializeSIInsertWaitsPass(*PR);
   initializeSIInsertWaitcntsPass(*PR);
   initializeSIWholeQuadModePass(*PR);
   initializeSILowerControlFlowPass(*PR);
   initializeSIInsertSkipsPass(*PR);
+  initializeSIMemoryLegalizerPass(*PR);
   initializeSIDebuggerInsertNopsPass(*PR);
   initializeSIOptimizeExecMaskingPass(*PR);
   initializeAMDGPUUnifyDivergentExitNodesPass(*PR);
   initializeAMDGPUAAWrapperPassPass(*PR);
 }
 
 static std::unique_ptr<TargetLoweringObjectFile> createTLOF(const Triple &TT) {
   return llvm::make_unique<AMDGPUTargetObjectFile>();
void GCNPassConfig::addPreEmitPass() {
   addPass(&PostRAHazardRecognizerID);
 
   if (EnableSIInsertWaitcntsPass)
     addPass(createSIInsertWaitcntsPass());
   else
     addPass(createSIInsertWaitsPass());
   addPass(createSIShrinkInstructionsPass());
   addPass(&SIInsertSkipsPassID);
+  addPass(createSIMemoryLegalizerPass());
   addPass(createSIDebuggerInsertNopsPass());
   addPass(&BranchRelaxationPassID);
 }
 
 TargetPassConfig *GCNTargetMachine::createPassConfig(PassManagerBase &PM) {
   return new GCNPassConfig(*this, PM);
 }
 

lib/Target/AMDGPU/BUFInstructions.td

: MUBUF_Pseudo<opName,
                  (outs vdataClass:$vdata),
                  getMUBUFIns<addrKindCopy>.ret,
                  " $vdata, " # getMUBUFAsmOps<addrKindCopy>.ret # "$glc$slc$tfe",
                  pattern>,
     MUBUF_SetupAddr<addrKindCopy> {
   let PseudoInstr = opName # "_" # getAddrName<addrKindCopy>.ret;
   let mayLoad = 1;
   let mayStore = 0;
+  let maybeAtomic = 1;
 }
 
 // FIXME: tfe can't be an operand because it requires a separate
 // opcode because it needs an N+1 register class dest register.
 multiclass MUBUF_Pseudo_Loads<string opName, RegisterClass vdataClass,
                               ValueType load_vt = i32,
                               SDPatternOperator ld = null_frag> {
 
: MUBUF_Pseudo<opName,
                  (outs),
                  getMUBUFIns<addrKindCopy, [vdataClassCopy]>.ret,
                  " $vdata, " # getMUBUFAsmOps<addrKindCopy>.ret # "$glc$slc$tfe",
                  pattern>,
     MUBUF_SetupAddr<addrKindCopy> {
   let PseudoInstr = opName # "_" # getAddrName<addrKindCopy>.ret;
   let mayLoad = 0;
   let mayStore = 1;
+  let maybeAtomic = 1;
 }
 
 multiclass MUBUF_Pseudo_Stores<string opName, RegisterClass vdataClass,
                                ValueType store_vt = i32,
                                SDPatternOperator st = null_frag> {
 
   def _OFFSET : MUBUF_Store_Pseudo <opName, BUFAddrKind.Offset, vdataClass,
     [(st store_vt:$vdata, (MUBUFOffset v4i32:$srsrc, i32:$soffset,
: MUBUF_Pseudo<opName, outs, ins, asmOps, pattern>,
     MUBUF_SetupAddr<addrKindCopy> {
   let mayStore = 1;
   let mayLoad = 1;
   let hasPostISelHook = 1;
   let hasSideEffects = 1;
   let DisableWQM = 1;
   let has_glc = 0;
   let has_tfe = 0;
+  let maybeAtomic = 1;
 }
 
 class MUBUF_AtomicNoRet_Pseudo<string opName, int addrKind,
                                RegisterClass vdataClass,
                                list<dag> pattern=[],
                                // Workaround bug bz30254
                                int addrKindCopy = addrKind,
                                RegisterClass vdataClassCopy = vdataClass>
class MUBUFLoad_PatternADDR64 <MUBUF_Pseudo Instr_ADDR64, ValueType vt,
      (Instr_ADDR64 $vaddr, $srsrc, $soffset, $offset, $glc, $slc, $tfe)
   >;
 
 multiclass MUBUFLoad_Atomic_Pattern <MUBUF_Pseudo Instr_ADDR64, MUBUF_Pseudo Instr_OFFSET,
                                      ValueType vt, PatFrag atomic_ld> {
   def : Pat <
      (vt (atomic_ld (MUBUFAddr64 v4i32:$srsrc, i64:$vaddr, i32:$soffset,
                                    i16:$offset, i1:$slc))),
-     (Instr_ADDR64 $vaddr, $srsrc, $soffset, $offset, 1, $slc, 0)
+     (Instr_ADDR64 $vaddr, $srsrc, $soffset, $offset, 0, $slc, 0)
   >;
 
   def : Pat <
     (vt (atomic_ld (MUBUFOffsetNoGLC v4i32:$rsrc, i32:$soffset, i16:$offset))),
-    (Instr_OFFSET $rsrc, $soffset, (as_i16imm $offset), 1, 0, 0)
+    (Instr_OFFSET $rsrc, $soffset, (as_i16imm $offset), 0, 0, 0)
   >;
 }
 
 let Predicates = [isSICI] in {
 def : MUBUFLoad_PatternADDR64 <BUFFER_LOAD_SBYTE_ADDR64, i32, sextloadi8_constant>;
 def : MUBUFLoad_PatternADDR64 <BUFFER_LOAD_UBYTE_ADDR64, i32, az_extloadi8_constant>;
 def : MUBUFLoad_PatternADDR64 <BUFFER_LOAD_SSHORT_ADDR64, i32, sextloadi16_constant>;
 def : MUBUFLoad_PatternADDR64 <BUFFER_LOAD_USHORT_ADDR64, i32, az_extloadi16_constant>;
defm : MUBUF_Load_Dword <v4i32, BUFFER_LOAD_DWORDX4_OFFSET, BUFFER_LOAD_DWORDX4_OFFEN,
                          BUFFER_LOAD_DWORDX4_IDXEN, BUFFER_LOAD_DWORDX4_BOTHEN>;
 
 multiclass MUBUFStore_Atomic_Pattern <MUBUF_Pseudo Instr_ADDR64, MUBUF_Pseudo Instr_OFFSET,
                                       ValueType vt, PatFrag atomic_st> {
   // Store follows atomic op convention so address is forst
   def : Pat <
      (atomic_st (MUBUFAddr64 v4i32:$srsrc, i64:$vaddr, i32:$soffset,
                                    i16:$offset, i1:$slc), vt:$val),
-     (Instr_ADDR64 $val, $vaddr, $srsrc, $soffset, $offset, 1, $slc, 0)
+     (Instr_ADDR64 $val, $vaddr, $srsrc, $soffset, $offset, 0, $slc, 0)
   >;
 
   def : Pat <
     (atomic_st (MUBUFOffsetNoGLC v4i32:$rsrc, i32:$soffset, i16:$offset), vt:$val),
-    (Instr_OFFSET $val, $rsrc, $soffset, (as_i16imm $offset), 1, 0, 0)
+    (Instr_OFFSET $val, $rsrc, $soffset, (as_i16imm $offset), 0, 0, 0)
   >;
 }
 let Predicates = [isSICI] in {
 defm : MUBUFStore_Atomic_Pattern <BUFFER_STORE_DWORD_ADDR64, BUFFER_STORE_DWORD_OFFSET, i32, global_store_atomic>;
 defm : MUBUFStore_Atomic_Pattern <BUFFER_STORE_DWORDX2_ADDR64, BUFFER_STORE_DWORDX2_OFFSET, i64, global_store_atomic>;
 } // End Predicates = [isSICI]
 
 

lib/Target/AMDGPU/CMakeLists.txt

add_llvm_target(AMDGPUCodeGen
   AMDGPUTargetObjectFile.cpp
   AMDGPUIntrinsicInfo.cpp
   AMDGPUISelDAGToDAG.cpp
   AMDGPULowerIntrinsics.cpp
   AMDGPUMacroFusion.cpp
   AMDGPUMCInstLower.cpp
   AMDGPUMachineCFGStructurizer.cpp
   AMDGPUMachineFunction.cpp
+  AMDGPUMachineModuleInfo.cpp
   AMDGPUUnifyMetadata.cpp
   AMDGPUOpenCLImageTypeLoweringPass.cpp
   AMDGPUSubtarget.cpp
   AMDGPUTargetMachine.cpp
   AMDGPUTargetTransformInfo.cpp
   AMDGPUISelLowering.cpp
   AMDGPUInstrInfo.cpp
   AMDGPUPromoteAlloca.cpp
add_llvm_target(AMDGPUCodeGen
   SIInsertWaitcnts.cpp
   SIInstrInfo.cpp
   SIISelLowering.cpp
   SILoadStoreOptimizer.cpp
   SILowerControlFlow.cpp
   SILowerI1Copies.cpp
   SIMachineFunctionInfo.cpp
   SIMachineScheduler.cpp
+  SIMemoryLegalizer.cpp
   SIOptimizeExecMasking.cpp
   SIPeepholeSDWA.cpp
   SIRegisterInfo.cpp
   SIShrinkInstructions.cpp
   SIWholeQuadMode.cpp
   GCNIterativeScheduler.cpp
   GCNMinRegStrategy.cpp
   GCNRegPressure.cpp

lib/Target/AMDGPU/FLATInstructions.td

class FLAT_Load_Pseudo <string opName, RegisterClass regClass,
   opName,
   (outs regClass:$vdst),
   !if(HasSignedOffset,
     (ins VReg_64:$vaddr, offset_s13:$offset, GLC:$glc, slc:$slc),
     (ins VReg_64:$vaddr, offset_u12:$offset, GLC:$glc, slc:$slc)),
   " $vdst, $vaddr$offset$glc$slc"> {
   let has_data = 0;
   let mayLoad = 1;
+  let maybeAtomic = 1;
 }
 
 class FLAT_Global_Load_Pseudo<string opName, RegisterClass regClass> :
   FLAT_Load_Pseudo<opName, regClass, 1> {
   let is_flat_global = 1;
 }
 
 class FLAT_Scratch_Load_Pseudo<string opName, RegisterClass regClass> :
   FLAT_Load_Pseudo<opName, regClass, 1> {
   let is_flat_scratch = 1;
 }
 
 class FLAT_Store_Pseudo <string opName, RegisterClass vdataClass,
   bit HasSignedOffset = 0> : FLAT_Pseudo<
   opName,
   (outs),
   !if(HasSignedOffset,
     (ins VReg_64:$vaddr, vdataClass:$vdata, offset_s13:$offset, GLC:$glc, slc:$slc),
     (ins VReg_64:$vaddr, vdataClass:$vdata, offset_u12:$offset, GLC:$glc, slc:$slc)),
   " $vaddr, $vdata$offset$glc$slc"> {
   let mayLoad  = 0;
   let mayStore = 1;
   let has_vdst = 0;
+  let maybeAtomic = 1;
 }
 
 class FLAT_Global_Store_Pseudo<string opName, RegisterClass regClass> :
   FLAT_Store_Pseudo<opName, regClass, 1> {
   let is_flat_global = 1;
 }
 
 class FLAT_Scratch_Store_Pseudo<string opName, RegisterClass regClass> :
def "" : FLAT_Pseudo <opName,
     []>,
     AtomicNoRet <NAME, 0> {
     let mayLoad = 1;
     let mayStore = 1;
     let has_glc  = 0;
     let glcValue = 0;
     let has_vdst = 0;
     let PseudoInstr = NAME;
+    let maybeAtomic = 1;
   }
 
   def _RTN : FLAT_Pseudo <opName,
     (outs vdst_rc:$vdst),
     !if(HasSignedOffset,
       (ins VReg_64:$vaddr, data_rc:$vdata, offset_s13:$offset, slc:$slc),
       (ins VReg_64:$vaddr, data_rc:$vdata, offset_u12:$offset, slc:$slc)),
     " $vdst, $vaddr, $vdata$offset glc$slc",
     [(set vt:$vdst,
       (atomic (FLATAtomic i64:$vaddr, i16:$offset, i1:$slc), data_vt:$vdata))]>,
     AtomicNoRet <NAME, 1> {
     let mayLoad  = 1;
     let mayStore = 1;
     let hasPostISelHook = 1;
     let has_glc  = 0;
     let glcValue = 1;
     let PseudoInstr = NAME # "_RTN";
+    let maybeAtomic = 1;
   }
 }
 
 class flat_binary_atomic_op<SDNode atomic_op> : PatFrag<
   (ops node:$ptr, node:$value),
   (atomic_op node:$ptr, node:$value),
   [{return cast<MemSDNode>(N)->getAddressSpace() == AMDGPUASI.FLAT_ADDRESS;}]
 >;

 // Patterns for global loads with no offset.
 class FlatLoadPat <FLAT_Pseudo inst, SDPatternOperator node, ValueType vt> : Pat <
   (vt (node (FLATAtomic i64:$vaddr, i16:$offset, i1:$slc))),
   (inst $vaddr, $offset, 0, $slc)
 >;
 
 class FlatLoadAtomicPat <FLAT_Pseudo inst, SDPatternOperator node, ValueType vt> : Pat <
   (vt (node (FLATAtomic i64:$vaddr, i16:$offset, i1:$slc))),
-  (inst $vaddr, $offset, 1, $slc)
+  (inst $vaddr, $offset, 0, $slc)
 >;
 
 class FlatStorePat <FLAT_Pseudo inst, SDPatternOperator node, ValueType vt> : Pat <
   (node vt:$data, (FLATAtomic i64:$vaddr, i16:$offset, i1:$slc)),
   (inst $vaddr, $data, $offset, 0, $slc)
 >;
 
 class FlatStoreAtomicPat <FLAT_Pseudo inst, SDPatternOperator node, ValueType vt> : Pat <
   // atomic store follows atomic binop convention so the address comes
   // first.
   (node (FLATAtomic i64:$vaddr, i16:$offset, i1:$slc), vt:$data),
-  (inst $vaddr, $data, $offset, 1, $slc)
+  (inst $vaddr, $data, $offset, 0, $slc)
 >;
 
 class FlatAtomicPat <FLAT_Pseudo inst, SDPatternOperator node, ValueType vt,
                      ValueType data_vt = vt> : Pat <
   (vt (node (FLATAtomic i64:$vaddr, i16:$offset, i1:$slc), data_vt:$data)),
   (inst $vaddr, $data, $offset, $slc)
 >;
 

lib/Target/AMDGPU/SIDefines.h

// TODO: Should this be spilt into VOP3 a and b?
 
   WQM = UINT64_C(1) << 35,
   DisableWQM = UINT64_C(1) << 36,
   Gather4 = UINT64_C(1) << 37,
   SOPK_ZEXT = UINT64_C(1) << 38,
   SCALAR_STORE = UINT64_C(1) << 39,
   FIXED_SIZE = UINT64_C(1) << 40,
   VOPAsmPrefer32Bit = UINT64_C(1) << 41,
-  HasFPClamp = UINT64_C(1) << 42
+  HasFPClamp = UINT64_C(1) << 42,
+
+  maybeAtomic = UINT64_C(1) << 43
 };
 
 // v_cmp_class_* etc. use a 10-bit mask for what operation is checked.
 // The result is true if any of these tests are true.
 enum ClassFlags {
   S_NAN = 1 << 0,        // Signaling NaN
   Q_NAN = 1 << 1,        // Quiet NaN
   N_INFINITY = 1 << 2,   // Negative infinity

lib/Target/AMDGPU/SIInstrFormats.td

class InstSI <dag outs, dag ins, string asm = "",
   field bit MUBUF = 0;
   field bit MTBUF = 0;
   field bit SMRD = 0;
   field bit MIMG = 0;
   field bit EXP = 0;
   field bit FLAT = 0;
   field bit DS = 0;
 
-   // Pseudo instruction formats.
   field bit VGPRSpill = 0;
   field bit SGPRSpill = 0;
 
   // High bits - other information.
   field bit VM_CNT = 0;
   field bit EXP_CNT = 0;
   field bit LGKM_CNT = 0;
 
class InstSI <dag outs, dag ins, string asm = "",
   // This bit tells the assembler to use the 32-bit encoding in case it
   // is unable to infer the encoding from the operands.
   field bit VOPAsmPrefer32Bit = 0;
 
   // This bit indicates that this has a floating point result type, so
   // the clamp modifier has floating point semantics.
   field bit FPClamp = 0;
 
+  // Is it possible for this instruction to be atomic?
+  field bit maybeAtomic = 0;
+
   // These need to be kept in sync with the enum in SIInstrFlags.
   let TSFlags{0} = SALU;
   let TSFlags{1} = VALU;
 
   let TSFlags{2} = SOP1;
   let TSFlags{3} = SOP2;
   let TSFlags{4} = SOPC;
   let TSFlags{5} = SOPK;
class InstSI <dag outs, dag ins, string asm = "",
   let TSFlags{37} = Gather4;
 
   let TSFlags{38} = SOPKZext;
   let TSFlags{39} = ScalarStore;
   let TSFlags{40} = FixedSize;
   let TSFlags{41} = VOPAsmPrefer32Bit;
   let TSFlags{42} = FPClamp;
 
+  let TSFlags{43} = maybeAtomic;
+
   let SchedRW = [Write32Bit];
 
   field bits<1> DisableSIDecoder = 0;
   field bits<1> DisableVIDecoder = 0;
   field bits<1> DisableDecoder = 0;
 
   let isAsmParserOnly = !if(!eq(DisableDecoder{0}, {0}), 0, 1);
   let AsmVariantName = AMDGPUAsmVariants.Default;

lib/Target/AMDGPU/SIInstructions.td

 //===----------------------------------------------------------------------===//
 // Pseudo Instructions
 //===----------------------------------------------------------------------===//
 def ATOMIC_FENCE : SPseudoInstSI<
   (outs), (ins i32imm:$ordering, i32imm:$scope),
   [(atomic_fence (i32 imm:$ordering), (i32 imm:$scope))],
   "ATOMIC_FENCE $ordering, $scope"> {
   let hasSideEffects = 1;
+  let maybeAtomic = 1;
 }
 
 let hasSideEffects = 0, mayLoad = 0, mayStore = 0, Uses = [EXEC] in {
 
 // For use in patterns
 def V_CNDMASK_B64_PSEUDO : VOP3Common <(outs VReg_64:$vdst),
   (ins VSrc_b64:$src0, VSrc_b64:$src1, SSrc_b64:$src2), "", []> {
   let isPseudo = 1;
   let isCodeGenOnly = 1;
   let usesCustomInserter = 1;
 }
 
 // 64-bit vector move instruction.  This is mainly used by the SIFoldOperands
 // pass to enable folding of inline immediates.
 def V_MOV_B64_PSEUDO : VPseudoInstSI <(outs VReg_64:$vdst),

lib/Target/AMDGPU/SIMemoryLegalizer.cpp

+//===--- SIMemoryLegalizer.cpp ----------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file
+/// \brief Memory legalizer - implements memory model. More information can be
+/// found here:
+///   http://llvm.org/docs/AMDGPUUsage.html#memory-model
+///
+//
+//===----------------------------------------------------------------------===//
+
+#include "AMDGPU.h"
+#include "AMDGPUMachineModuleInfo.h"
+#include "AMDGPUSubtarget.h"
+#include "Utils/AMDGPUBaseInfo.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/IR/DiagnosticInfo.h"
+
+using namespace llvm;
+using namespace llvm::AMDGPU;
+
+#define DEBUG_TYPE "si-memory-legalizer"
+#define PASS_NAME "SI Memory Legalizer"
+
+namespace {
+
+class SIMemoryLegalizer final : public MachineFunctionPass {
+private:
+  struct AtomicInfo final {
+    SyncScope::ID SSID = SyncScope::System;
+    AtomicOrdering Ordering = AtomicOrdering::SequentiallyConsistent;
+    AtomicOrdering FailureOrdering = AtomicOrdering::SequentiallyConsistent;
+
+    AtomicInfo() {}
+
+    AtomicInfo(SyncScope::ID SSID,
+               AtomicOrdering Ordering,
+               AtomicOrdering FailureOrdering)
+        : SSID(SSID),
+          Ordering(Ordering),
+          FailureOrdering(FailureOrdering) {}
+
+    AtomicInfo(const MachineMemOperand *MMO)
+        : SSID(MMO->getSyncScopeID()),
+          Ordering(MMO->getOrdering()),
+          FailureOrdering(MMO->getFailureOrdering()) {}
+  };
+
+  /// \brief LLVM context.
+  LLVMContext *CTX = nullptr;
+  /// \brief Machine module info.
+  const AMDGPUMachineModuleInfo *MMI = nullptr;
+  /// \brief Instruction info.
+  const SIInstrInfo *TII = nullptr;
+
+  /// \brief Immediate for "vmcnt(0)".
+  unsigned Vmcnt0Immediate = 0;
+  /// \brief Opcode for cache invalidation instruction (L1).
+  unsigned Wbinvl1Opcode = 0;
+
+  /// \brief List of atomic pseudo instructions.
+  std::list<MachineBasicBlock::iterator> AtomicPseudoMIs;
+
+  /// \brief Inserts "buffer_wbinvl1_vol" instruction \p Before or after \p MI.
+  /// Always returns true.
+  bool insertBufferWbinvl1Vol(MachineBasicBlock::iterator &MI,
+                              bool Before = true) const;
+  /// \brief Inserts "s_waitcnt vmcnt(0)" instruction \p Before or after \p MI.
+  /// Always returns true.
+  bool insertWaitcntVmcnt0(MachineBasicBlock::iterator &MI,
+                           bool Before = true) const;
+
+  /// \brief Sets GLC bit if present in \p MI. Returns true if \p MI is
+  /// modified, false otherwise.
+  bool setGLC(const MachineBasicBlock::iterator &MI) const;
+
+  /// \brief Removes all processed atomic pseudo instructions from the current
+  /// function. Returns true if current function is modified, false otherwise.
+  bool removeAtomicPseudoMIs();
+
+  /// \brief Reports unknown synchronization scope used in \p MI to LLVM
+  /// context.
+  void reportUnknownSynchScope(const MachineBasicBlock::iterator &MI);
+
+  /// \returns Atomic fence info if \p MI is an atomic fence operation,
+  /// "None" otherwise.
+  Optional<AtomicInfo> getAtomicFenceInfo(
+      const MachineBasicBlock::iterator &MI) const;
+  /// \returns Atomic load info if \p MI is an atomic load operation,
+  /// "None" otherwise.
+  Optional<AtomicInfo> getAtomicLoadInfo(
+      const MachineBasicBlock::iterator &MI) const;
+  /// \returns Atomic store info if \p MI is an atomic store operation,
+  /// "None" otherwise.
+  Optional<AtomicInfo> getAtomicStoreInfo(
+      const MachineBasicBlock::iterator &MI) const;
+  /// \returns Atomic cmpxchg info if \p MI is an atomic cmpxchg operation,
+  /// "None" otherwise.
+  Optional<AtomicInfo> getAtomicCmpxchgInfo(
+      const MachineBasicBlock::iterator &MI) const;
+  /// \returns Atomic rmw info if \p MI is an atomic rmw operation,
+  /// "None" otherwise.
+  Optional<AtomicInfo> getAtomicRmwInfo(
+      const MachineBasicBlock::iterator &MI) const;
+
+  /// \brief Expands atomic fence operation \p MI. Returns true if
+  /// instructions are added/deleted or \p MI is modified, false otherwise.
+  bool expandAtomicFence(const AtomicInfo &AI,
+                         MachineBasicBlock::iterator &MI);
+  /// \brief Expands atomic load operation \p MI. Returns true if
+  /// instructions are added/deleted or \p MI is modified, false otherwise.
+  bool expandAtomicLoad(const AtomicInfo &AI,
+                        MachineBasicBlock::iterator &MI);
+  /// \brief Expands atomic store operation \p MI. Returns true if
+  /// instructions are added/deleted or \p MI is modified, false otherwise.
+  bool expandAtomicStore(const AtomicInfo &AI,
+                         MachineBasicBlock::iterator &MI);
+  /// \brief Expands atomic cmpxchg operation \p MI. Returns true if
+  /// instructions are added/deleted or \p MI is modified, false otherwise.
+  bool expandAtomicCmpxchg(const AtomicInfo &AI,
+                           MachineBasicBlock::iterator &MI);
+  /// \brief Expands atomic rmw operation \p MI. Returns true if
+  /// instructions are added/deleted or \p MI is modified, false otherwise.
+  bool expandAtomicRmw(const AtomicInfo &AI,
+                       MachineBasicBlock::iterator &MI);
+
+public:
+  static char ID;
+
+  SIMemoryLegalizer()
+      : MachineFunctionPass(ID) {}
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.setPreservesCFG();
+    MachineFunctionPass::getAnalysisUsage(AU);
+  }
+
+  StringRef getPassName() const override {
+    return PASS_NAME;
+  }
+
+  bool runOnMachineFunction(MachineFunction &MF) override;
+};
+
+} // end namespace anonymous
+
+bool SIMemoryLegalizer::insertBufferWbinvl1Vol(MachineBasicBlock::iterator &MI,
+                                               bool Before) const {
+  MachineBasicBlock &MBB = *MI->getParent();
+  DebugLoc DL = MI->getDebugLoc();
+
+  if (!Before)
+    ++MI;
+
+  BuildMI(MBB, MI, DL, TII->get(Wbinvl1Opcode));
+
+  if (!Before)
+    --MI;
+
+  return true;
+}
+
+bool SIMemoryLegalizer::insertWaitcntVmcnt0(MachineBasicBlock::iterator &MI,
+                                            bool Before) const {
+  MachineBasicBlock &MBB = *MI->getParent();
+  DebugLoc DL = MI->getDebugLoc();
+
+  if (!Before)
+    ++MI;
+
+  BuildMI(MBB, MI, DL, TII->get(AMDGPU::S_WAITCNT)).addImm(Vmcnt0Immediate);
+
+  if (!Before)
+    --MI;
+
+  return true;
+}
+
+bool SIMemoryLegalizer::setGLC(const MachineBasicBlock::iterator &MI) const {
+  int GLCIdx = AMDGPU::getNamedOperandIdx(MI->getOpcode(), AMDGPU::OpName::glc);
+  if (GLCIdx == -1)
+    return false;
+
+  MachineOperand &GLC = MI->getOperand(GLCIdx);
+  if (GLC.getImm() == 1)
+    return false;
+
+  GLC.setImm(1);
+  return true;
+}
+
+bool SIMemoryLegalizer::removeAtomicPseudoMIs() {
+  if (AtomicPseudoMIs.empty())
+    return false;
+
+  for (auto &MI : AtomicPseudoMIs)
+    MI->eraseFromParent();
+
+  AtomicPseudoMIs.clear();
+  return true;
+}
+
+void SIMemoryLegalizer::reportUnknownSynchScope(
+    const MachineBasicBlock::iterator &MI) {
+  DiagnosticInfoUnsupported Diag(*MI->getParent()->getParent()->getFunction(),
+                                 "Unsupported synchronization scope");
+  CTX->diagnose(Diag);
+}
+
+Optional<SIMemoryLegalizer::AtomicInfo> SIMemoryLegalizer::getAtomicFenceInfo(
+    const MachineBasicBlock::iterator &MI) const {
+  assert(MI->getDesc().TSFlags & SIInstrFlags::maybeAtomic);
+
+  if (MI->getOpcode() != AMDGPU::ATOMIC_FENCE)
+    return None;
+
+  SyncScope::ID SSID =
+      static_cast<SyncScope::ID>(MI->getOperand(1).getImm());
+  AtomicOrdering Ordering =
+      static_cast<AtomicOrdering>(MI->getOperand(0).getImm());
+  return AtomicInfo(SSID, Ordering, AtomicOrdering::NotAtomic);
+}
+
+Optional<SIMemoryLegalizer::AtomicInfo> SIMemoryLegalizer::getAtomicLoadInfo(
+    const MachineBasicBlock::iterator &MI) const {
+  assert(MI->getDesc().TSFlags & SIInstrFlags::maybeAtomic);
+
+  if (!(MI->mayLoad() && !MI->mayStore()))
+    return None;
+  if (!MI->hasOneMemOperand())
+    return AtomicInfo();
+
+  const MachineMemOperand *MMO = *MI->memoperands_begin();
+  if (!MMO->isAtomic())
+    return None;
+
+  return AtomicInfo(MMO);
+}
+
+Optional<SIMemoryLegalizer::AtomicInfo> SIMemoryLegalizer::getAtomicStoreInfo(
+    const MachineBasicBlock::iterator &MI) const {
+  assert(MI->getDesc().TSFlags & SIInstrFlags::maybeAtomic);
+
+  if (!(!MI->mayLoad() && MI->mayStore()))
+    return None;
+  if (!MI->hasOneMemOperand())
+    return AtomicInfo();
+
+  const MachineMemOperand *MMO = *MI->memoperands_begin();
+  if (!MMO->isAtomic())
+    return None;
+
+  return AtomicInfo(MMO);
+}
+
+Optional<SIMemoryLegalizer::AtomicInfo> SIMemoryLegalizer::getAtomicCmpxchgInfo(
+    const MachineBasicBlock::iterator &MI) const {
+  assert(MI->getDesc().TSFlags & SIInstrFlags::maybeAtomic);
+
+  if (!(MI->mayLoad() && MI->mayStore()))
+    return None;
+  if (!MI->hasOneMemOperand())
+    return AtomicInfo();
+
+  const MachineMemOperand *MMO = *MI->memoperands_begin();
+  if (!MMO->isAtomic())
+    return None;
+  if (MMO->getFailureOrdering() == AtomicOrdering::NotAtomic)
+    return None;
+
+  return AtomicInfo(MMO);
+}
+
+Optional<SIMemoryLegalizer::AtomicInfo> SIMemoryLegalizer::getAtomicRmwInfo(
+    const MachineBasicBlock::iterator &MI) const {
+  assert(MI->getDesc().TSFlags & SIInstrFlags::maybeAtomic);
+
+  if (!(MI->mayLoad() && MI->mayStore()))
+    return None;
+  if (!MI->hasOneMemOperand())
+    return AtomicInfo();
+
+  const MachineMemOperand *MMO = *MI->memoperands_begin();
+  if (!MMO->isAtomic())
+    return None;
+  if (MMO->getFailureOrdering() != AtomicOrdering::NotAtomic)
+    return None;
+
+  return AtomicInfo(MMO);
+}
+
+bool SIMemoryLegalizer::expandAtomicFence(const AtomicInfo &AI,
+                                          MachineBasicBlock::iterator &MI) {
+  assert(MI->getOpcode() == AMDGPU::ATOMIC_FENCE);
+
+  bool Changed = false;
+  if (AI.SSID == SyncScope::System ||
+      AI.SSID == MMI->getAgentSSID()) {
+    if (AI.Ordering == AtomicOrdering::Acquire ||
+        AI.Ordering == AtomicOrdering::Release ||
+        AI.Ordering == AtomicOrdering::AcquireRelease ||
+        AI.Ordering == AtomicOrdering::SequentiallyConsistent)
+      Changed |= insertWaitcntVmcnt0(MI);
+
+    if (AI.Ordering == AtomicOrdering::Acquire ||
+        AI.Ordering == AtomicOrdering::AcquireRelease ||
+        AI.Ordering == AtomicOrdering::SequentiallyConsistent)
+      Changed |= insertBufferWbinvl1Vol(MI);
+
+    AtomicPseudoMIs.push_back(MI);
+    return Changed;
+  } else if (AI.SSID == SyncScope::SingleThread ||
+             AI.SSID == MMI->getWorkgroupSSID() ||
+             AI.SSID == MMI->getWavefrontSSID()) {
+    AtomicPseudoMIs.push_back(MI);
+    return Changed;
+  } else {
+    reportUnknownSynchScope(MI);
+    return Changed;
+  }
+}
+
+bool SIMemoryLegalizer::expandAtomicLoad(const AtomicInfo &AI,
+                                         MachineBasicBlock::iterator &MI) {
+  assert(MI->mayLoad() && !MI->mayStore());
+
+  bool Changed = false;
+  if (AI.SSID == SyncScope::System ||
+      AI.SSID == MMI->getAgentSSID()) {
+    if (AI.Ordering == AtomicOrdering::Acquire ||
+        AI.Ordering == AtomicOrdering::SequentiallyConsistent)
+      Changed |= setGLC(MI);
+
+    if (AI.Ordering == AtomicOrdering::SequentiallyConsistent)
+      Changed |= insertWaitcntVmcnt0(MI);
+
+    if (AI.Ordering == AtomicOrdering::Acquire ||
+        AI.Ordering == AtomicOrdering::SequentiallyConsistent) {
+      Changed |= insertWaitcntVmcnt0(MI, false);
+      Changed |= insertBufferWbinvl1Vol(MI, false);
+    }
+
+    return Changed;
+  } else if (AI.SSID == SyncScope::SingleThread ||
+             AI.SSID == MMI->getWorkgroupSSID() ||
+             AI.SSID == MMI->getWavefrontSSID()) {
+    return Changed;
+  } else {
+    reportUnknownSynchScope(MI);
+    return Changed;
+  }
+}
+
+bool SIMemoryLegalizer::expandAtomicStore(const AtomicInfo &AI,
+                                          MachineBasicBlock::iterator &MI) {
+  assert(!MI->mayLoad() && MI->mayStore());
+
+  bool Changed = false;
+  if (AI.SSID == SyncScope::System ||
+      AI.SSID == MMI->getAgentSSID()) {
+    if (AI.Ordering == AtomicOrdering::Release ||
+        AI.Ordering == AtomicOrdering::SequentiallyConsistent)
+      Changed |= insertWaitcntVmcnt0(MI);
+
+    return Changed;
+  } else if (AI.SSID == SyncScope::SingleThread ||
+             AI.SSID == MMI->getWorkgroupSSID() ||
+             AI.SSID == MMI->getWavefrontSSID()) {
+    return Changed;
+  } else {
+    reportUnknownSynchScope(MI);
+    return Changed;
+  }
+}
+
+bool SIMemoryLegalizer::expandAtomicCmpxchg(const AtomicInfo &AI,
+                                            MachineBasicBlock::iterator &MI) {
+  assert(MI->mayLoad() && MI->mayStore());
+
+  bool Changed = false;
+  if (AI.SSID == SyncScope::System ||
+      AI.SSID == MMI->getAgentSSID()) {
+    if (AI.Ordering == AtomicOrdering::Release ||
+        AI.Ordering == AtomicOrdering::AcquireRelease ||
+        AI.Ordering == AtomicOrdering::SequentiallyConsistent ||
+        AI.FailureOrdering == AtomicOrdering::SequentiallyConsistent)
+      Changed |= insertWaitcntVmcnt0(MI);
+
+    if (AI.Ordering == AtomicOrdering::Acquire ||
+        AI.Ordering == AtomicOrdering::AcquireRelease ||
+        AI.Ordering == AtomicOrdering::SequentiallyConsistent ||
+        AI.FailureOrdering == AtomicOrdering::Acquire ||
+        AI.FailureOrdering == AtomicOrdering::SequentiallyConsistent) {
+      Changed |= insertWaitcntVmcnt0(MI, false);
+      Changed |= insertBufferWbinvl1Vol(MI, false);
+    }
+
+    return Changed;
+  } else if (AI.SSID == SyncScope::SingleThread ||
+             AI.SSID == MMI->getWorkgroupSSID() ||
+             AI.SSID == MMI->getWavefrontSSID()) {
+    Changed |= setGLC(MI);
+    return Changed;
+  } else {
+    reportUnknownSynchScope(MI);
+    return Changed;
+  }
+}
+
+bool SIMemoryLegalizer::expandAtomicRmw(const AtomicInfo &AI,
+                                        MachineBasicBlock::iterator &MI) {
+  assert(MI->mayLoad() && MI->mayStore());
+
+  bool Changed = false;
+  if (AI.SSID == SyncScope::System ||
+      AI.SSID == MMI->getAgentSSID()) {
+    if (AI.Ordering == AtomicOrdering::Release ||
+        AI.Ordering == AtomicOrdering::AcquireRelease ||
+        AI.Ordering == AtomicOrdering::SequentiallyConsistent)
+      Changed |= insertWaitcntVmcnt0(MI);
+
+    if (AI.Ordering == AtomicOrdering::Acquire ||
+        AI.Ordering == AtomicOrdering::AcquireRelease ||
+        AI.Ordering == AtomicOrdering::SequentiallyConsistent) {
+      Changed |= insertWaitcntVmcnt0(MI, false);
+      Changed |= insertBufferWbinvl1Vol(MI, false);
+    }
+
+    return Changed;
+  } else if (AI.SSID == SyncScope::SingleThread ||
+             AI.SSID == MMI->getWorkgroupSSID() ||
+             AI.SSID == MMI->getWavefrontSSID()) {
+    Changed |= setGLC(MI);
+    return Changed;
+  } else {
+    reportUnknownSynchScope(MI);
+    return Changed;
+  }
+}
+
+bool SIMemoryLegalizer::runOnMachineFunction(MachineFunction &MF) {
+  bool Changed = false;
+  const SISubtarget &ST = MF.getSubtarget<SISubtarget>();
+  const IsaInfo::IsaVersion IV = IsaInfo::getIsaVersion(ST.getFeatureBits());
+
+  CTX = &MF.getFunction()->getContext();
+  MMI = &MF.getMMI().getObjFileInfo<AMDGPUMachineModuleInfo>();
+  TII = ST.getInstrInfo();
+
+  Vmcnt0Immediate =
+      AMDGPU::encodeWaitcnt(IV, 0, getExpcntBitMask(IV), getLgkmcntBitMask(IV));
+  Wbinvl1Opcode = ST.getGeneration() <= AMDGPUSubtarget::SOUTHERN_ISLANDS ?
+      AMDGPU::BUFFER_WBINVL1 : AMDGPU::BUFFER_WBINVL1_VOL;
+
+  for (auto &MBB : MF) {
+    for (auto MI = MBB.begin(); MI != MBB.end(); ++MI) {
+      if (!(MI->getDesc().TSFlags & SIInstrFlags::maybeAtomic))
+        continue;
+
+      if (const auto &AI = getAtomicFenceInfo(MI))
+        Changed |= expandAtomicFence(AI.getValue(), MI);
+      else if (const auto &AI = getAtomicLoadInfo(MI))
+        Changed |= expandAtomicLoad(AI.getValue(), MI);
+      else if (const auto &AI = getAtomicStoreInfo(MI))
+        Changed |= expandAtomicStore(AI.getValue(), MI);
+      else if (const auto &AI = getAtomicCmpxchgInfo(MI))
+        Changed |= expandAtomicCmpxchg(AI.getValue(), MI);
+      else if (const auto &AI = getAtomicRmwInfo(MI))
+        Changed |= expandAtomicRmw(AI.getValue(), MI);
+    }
+  }
+
+  Changed |= removeAtomicPseudoMIs();
+  return Changed;
+}
+
+INITIALIZE_PASS(SIMemoryLegalizer, DEBUG_TYPE, PASS_NAME, false, false)
+
+char SIMemoryLegalizer::ID = 0;
+char &llvm::SIMemoryLegalizerID = SIMemoryLegalizer::ID;
+
+FunctionPass *llvm::createSIMemoryLegalizerPass() {
+  return new SIMemoryLegalizer();
+}

test/CodeGen/AMDGPU/fence-amdgiz.ll

 ; RUN: llc < %s | FileCheck %s
 
 target datalayout = "e-p:64:64-p1:64:64-p2:64:64-p3:32:32-p4:32:32-p5:32:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-A5"
 target triple = "amdgcn-amd-amdhsa-amdgizcl"
 
-; CHECK_LABEL: atomic_fence
-; CHECK: BB#0:
-; CHECK: ATOMIC_FENCE 4, 1
-; CHECK: s_endpgm
-
+; CHECK-LABEL: atomic_fence
+; CHECK:       BB#0:
+; CHECK-NOT:   ATOMIC_FENCE
+; CHECK-NEXT:  s_waitcnt vmcnt(0)
+; CHECK-NEXT:  buffer_wbinvl1_vol
+; CHECK-NEXT:  s_endpgm
 define amdgpu_kernel void @atomic_fence() {
   fence acquire
   ret void
 }
 

test/CodeGen/AMDGPU/flat_atomics.ll

 entry:
   %ptr = getelementptr i32, i32 addrspace(4)* %in, i64 %index
   %val = load atomic i32, i32 addrspace(4)* %ptr seq_cst, align 4
   store i32 %val, i32 addrspace(4)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}atomic_store_i32_offset:
-; CIVI: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}} glc{{$}}
-; GFX9: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}} offset:16 glc{{$}}
+; CIVI: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+; GFX9: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}} offset:16{{$}}
 define amdgpu_kernel void @atomic_store_i32_offset(i32 %in, i32 addrspace(4)* %out) {
 entry:
   %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
   store atomic i32 %in, i32 addrspace(4)* %gep  seq_cst, align 4
   ret void
 }
 
 ; GCN-LABEL: {{^}}atomic_store_i32:
-; GCN: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}} glc{{$}}
+; GCN: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
 define amdgpu_kernel void @atomic_store_i32(i32 %in, i32 addrspace(4)* %out) {
 entry:
   store atomic i32 %in, i32 addrspace(4)* %out seq_cst, align 4
   ret void
 }
 
 ; GCN-LABEL: {{^}}atomic_store_i32_addr64_offset:
-; CIVI: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}} glc{{$}}
-; GFX9: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}} offset:16 glc{{$}}
+; CIVI: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+; GFX9: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}} offset:16{{$}}
 define amdgpu_kernel void @atomic_store_i32_addr64_offset(i32 %in, i32 addrspace(4)* %out, i64 %index) {
 entry:
   %ptr = getelementptr i32, i32 addrspace(4)* %out, i64 %index
   %gep = getelementptr i32, i32 addrspace(4)* %ptr, i32 4
   store atomic i32 %in, i32 addrspace(4)* %gep seq_cst, align 4
   ret void
 }
 
 ; GCN-LABEL: {{^}}atomic_store_i32_addr64:
-; GCN: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}} glc{{$}}
+; GCN: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
 define amdgpu_kernel void @atomic_store_i32_addr64(i32 %in, i32 addrspace(4)* %out, i64 %index) {
 entry:
   %ptr = getelementptr i32, i32 addrspace(4)* %out, i64 %index
   store atomic i32 %in, i32 addrspace(4)* %ptr seq_cst, align 4
   ret void
 }

test/CodeGen/AMDGPU/flat_atomics_i64.ll

 entry:
   %ptr = getelementptr i64, i64 addrspace(4)* %in, i64 %index
   %val = load atomic i64, i64 addrspace(4)* %ptr seq_cst, align 8
   store i64 %val, i64 addrspace(4)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}atomic_store_i64_offset:
-; GCN: flat_store_dwordx2 [[RET:v\[[0-9]+:[0-9]\]]], v[{{[0-9]+}}:{{[0-9]+}}] glc{{$}}
+; GCN: flat_store_dwordx2 [[RET:v\[[0-9]+:[0-9]\]]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
 define amdgpu_kernel void @atomic_store_i64_offset(i64 %in, i64 addrspace(4)* %out) {
 entry:
   %gep = getelementptr i64, i64 addrspace(4)* %out, i64 4
   store atomic i64 %in, i64 addrspace(4)* %gep  seq_cst, align 8
   ret void
 }
 
 ; GCN-LABEL: {{^}}atomic_store_i64:
-; GCN: flat_store_dwordx2 {{v\[[0-9]+:[0-9]\]}}, v[{{[0-9]+}}:{{[0-9]+}}] glc
+; GCN: flat_store_dwordx2 {{v\[[0-9]+:[0-9]\]}}, v[{{[0-9]+}}:{{[0-9]+}}]
 define amdgpu_kernel void @atomic_store_i64(i64 %in, i64 addrspace(4)* %out) {
 entry:
   store atomic i64 %in, i64 addrspace(4)* %out seq_cst, align 8
   ret void
 }
 
 ; GCN-LABEL: {{^}}atomic_store_i64_addr64_offset:
-; GCN: flat_store_dwordx2 {{v\[[0-9]+:[0-9]+\]}}, v[{{[0-9]+:[0-9]+}}] glc{{$}}
+; GCN: flat_store_dwordx2 {{v\[[0-9]+:[0-9]+\]}}, v[{{[0-9]+:[0-9]+}}]{{$}}
 define amdgpu_kernel void @atomic_store_i64_addr64_offset(i64 %in, i64 addrspace(4)* %out, i64 %index) {
 entry:
   %ptr = getelementptr i64, i64 addrspace(4)* %out, i64 %index
   %gep = getelementptr i64, i64 addrspace(4)* %ptr, i64 4
   store atomic i64 %in, i64 addrspace(4)* %gep seq_cst, align 8
   ret void
 }
 
 ; GCN-LABEL: {{^}}atomic_store_i64_addr64:
-; GCN: flat_store_dwordx2 {{v\[[0-9]+:[0-9]+\]}}, v[{{[0-9]+:[0-9]+}}] glc{{$}}
+; GCN: flat_store_dwordx2 {{v\[[0-9]+:[0-9]+\]}}, v[{{[0-9]+:[0-9]+}}]{{$}}
 define amdgpu_kernel void @atomic_store_i64_addr64(i64 %in, i64 addrspace(4)* %out, i64 %index) {
 entry:
   %ptr = getelementptr i64, i64 addrspace(4)* %out, i64 %index
   store atomic i64 %in, i64 addrspace(4)* %ptr seq_cst, align 8
   ret void
 }
 
 ; GCN-LABEL: {{^}}atomic_cmpxchg_i64_offset:

test/CodeGen/AMDGPU/global_atomics.ll

 entry:
   %ptr = getelementptr i32, i32 addrspace(1)* %in, i64 %index
   %val = load atomic i32, i32 addrspace(1)* %ptr seq_cst, align 4
   store i32 %val, i32 addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}atomic_store_i32_offset:
-; SI: buffer_store_dword {{v[0-9]+}}, off, s[{{[0-9]+}}:{{[0-9]+}}], 0 offset:16 glc{{$}}
-; VI: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}} glc{{$}}
+; SI: buffer_store_dword {{v[0-9]+}}, off, s[{{[0-9]+}}:{{[0-9]+}}], 0 offset:16{{$}}
+; VI: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
 define amdgpu_kernel void @atomic_store_i32_offset(i32 %in, i32 addrspace(1)* %out) {
 entry:
   %gep = getelementptr i32, i32 addrspace(1)* %out, i64 4
   store atomic i32 %in, i32 addrspace(1)* %gep  seq_cst, align 4
   ret void
 }
 
 ; FUNC-LABEL: {{^}}atomic_store_i32:
-; SI: buffer_store_dword {{v[0-9]+}}, off, s[{{[0-9]+}}:{{[0-9]+}}], 0 glc{{$}}
-; VI: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}} glc{{$}}
+; SI: buffer_store_dword {{v[0-9]+}}, off, s[{{[0-9]+}}:{{[0-9]+}}], 0{{$}}
+; VI: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
 define amdgpu_kernel void @atomic_store_i32(i32 %in, i32 addrspace(1)* %out) {
 entry:
   store atomic i32 %in, i32 addrspace(1)* %out seq_cst, align 4
   ret void
 }
 
 ; FUNC-LABEL: {{^}}atomic_store_i32_addr64_offset:
-; SI: buffer_store_dword {{v[0-9]+}}, v[{{[0-9]+}}:{{[0-9]+}}], s[{{[0-9]+}}:{{[0-9]+}}], 0 addr64 offset:16 glc{{$}}
-; VI: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}} glc{{$}}
+; SI: buffer_store_dword {{v[0-9]+}}, v[{{[0-9]+}}:{{[0-9]+}}], s[{{[0-9]+}}:{{[0-9]+}}], 0 addr64 offset:16{{$}}
+; VI: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
 define amdgpu_kernel void @atomic_store_i32_addr64_offset(i32 %in, i32 addrspace(1)* %out, i64 %index) {
 entry:
   %ptr = getelementptr i32, i32 addrspace(1)* %out, i64 %index
   %gep = getelementptr i32, i32 addrspace(1)* %ptr, i64 4
   store atomic i32 %in, i32 addrspace(1)* %gep seq_cst, align 4
   ret void
 }
 
 ; FUNC-LABEL: {{^}}atomic_store_i32_addr64:
-; SI: buffer_store_dword {{v[0-9]+}}, v[{{[0-9]+}}:{{[0-9]+}}], s[{{[0-9]+}}:{{[0-9]+}}], 0 addr64 glc{{$}}
-; VI: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}} glc{{$}}
+; SI: buffer_store_dword {{v[0-9]+}}, v[{{[0-9]+}}:{{[0-9]+}}], s[{{[0-9]+}}:{{[0-9]+}}], 0 addr64{{$}}
+; VI: flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
 define amdgpu_kernel void @atomic_store_i32_addr64(i32 %in, i32 addrspace(1)* %out, i64 %index) {
 entry:
   %ptr = getelementptr i32, i32 addrspace(1)* %out, i64 %index
   store atomic i32 %in, i32 addrspace(1)* %ptr seq_cst, align 4
   ret void
 }

test/CodeGen/AMDGPU/global_atomics_i64.ll

 entry:
   %ptr = getelementptr i64, i64 addrspace(1)* %in, i64 %index
   %val = load atomic i64, i64 addrspace(1)* %ptr seq_cst, align 8
   store i64 %val, i64 addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}atomic_store_i64_offset:
-; CI: buffer_store_dwordx2 [[RET:v\[[0-9]+:[0-9]+\]]], off, s[{{[0-9]+}}:{{[0-9]+}}], 0 offset:32 glc{{$}}
-; VI: flat_store_dwordx2 [[RET:v\[[0-9]+:[0-9]\]]], v[{{[0-9]+}}:{{[0-9]+}}] glc{{$}}
+; CI: buffer_store_dwordx2 [[RET:v\[[0-9]+:[0-9]+\]]], off, s[{{[0-9]+}}:{{[0-9]+}}], 0 offset:32{{$}}
+; VI: flat_store_dwordx2 [[RET:v\[[0-9]+:[0-9]\]]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
 define amdgpu_kernel void @atomic_store_i64_offset(i64 %in, i64 addrspace(1)* %out) {
 entry:
   %gep = getelementptr i64, i64 addrspace(1)* %out, i64 4
   store atomic i64 %in, i64 addrspace(1)* %gep  seq_cst, align 8
   ret void
 }
 
 ; FUNC-LABEL: {{^}}atomic_store_i64:
-; CI: buffer_store_dwordx2 {{v\[[0-9]+:[0-9]\]}}, off, s[{{[0-9]+}}:{{[0-9]+}}], 0 glc
-; VI: flat_store_dwordx2 {{v\[[0-9]+:[0-9]\]}}, v[{{[0-9]+}}:{{[0-9]+}}] glc
+; CI: buffer_store_dwordx2 {{v\[[0-9]+:[0-9]\]}}, off, s[{{[0-9]+}}:{{[0-9]+}}], 0{{$}}
+; VI: flat_store_dwordx2 {{v\[[0-9]+:[0-9]\]}}, v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
 define amdgpu_kernel void @atomic_store_i64(i64 %in, i64 addrspace(1)* %out) {
 entry:
   store atomic i64 %in, i64 addrspace(1)* %out seq_cst, align 8
   ret void
 }
 
 ; FUNC-LABEL: {{^}}atomic_store_i64_addr64_offset:
-; CI: buffer_store_dwordx2 {{v\[[0-9]+:[0-9]+\]}}, v[{{[0-9]+}}:{{[0-9]+}}], s[{{[0-9]+}}:{{[0-9]+}}], 0 addr64 offset:32 glc{{$}}
-; VI: flat_store_dwordx2 {{v\[[0-9]+:[0-9]+\]}}, v[{{[0-9]+:[0-9]+}}] glc{{$}}
+; CI: buffer_store_dwordx2 {{v\[[0-9]+:[0-9]+\]}}, v[{{[0-9]+}}:{{[0-9]+}}], s[{{[0-9]+}}:{{[0-9]+}}], 0 addr64 offset:32{{$}}
+; VI: flat_store_dwordx2 {{v\[[0-9]+:[0-9]+\]}}, v[{{[0-9]+:[0-9]+}}]{{$}}
 define amdgpu_kernel void @atomic_store_i64_addr64_offset(i64 %in, i64 addrspace(1)* %out, i64 %index) {
 entry:
   %ptr = getelementptr i64, i64 addrspace(1)* %out, i64 %index
   %gep = getelementptr i64, i64 addrspace(1)* %ptr, i64 4
   store atomic i64 %in, i64 addrspace(1)* %gep seq_cst, align 8
   ret void
 }
 
 ; FUNC-LABEL: {{^}}atomic_store_i64_addr64:
-; CI: buffer_store_dwordx2 {{v\[[0-9]+:[0-9]\]}}, v[{{[0-9]+}}:{{[0-9]+}}], s[{{[0-9]+}}:{{[0-9]+}}], 0 addr64 glc{{$}}
-; VI: flat_store_dwordx2 {{v\[[0-9]+:[0-9]+\]}}, v[{{[0-9]+:[0-9]+}}] glc{{$}}
+; CI: buffer_store_dwordx2 {{v\[[0-9]+:[0-9]\]}}, v[{{[0-9]+}}:{{[0-9]+}}], s[{{[0-9]+}}:{{[0-9]+}}], 0 addr64{{$}}
+; VI: flat_store_dwordx2 {{v\[[0-9]+:[0-9]+\]}}, v[{{[0-9]+:[0-9]+}}]{{$}}
 define amdgpu_kernel void @atomic_store_i64_addr64(i64 %in, i64 addrspace(1)* %out, i64 %index) {
 entry:
   %ptr = getelementptr i64, i64 addrspace(1)* %out, i64 %index
   store atomic i64 %in, i64 addrspace(1)* %ptr seq_cst, align 8
   ret void
 }

test/CodeGen/AMDGPU/memory-legalizer-atomic-cmpxchg.ll

+; RUN: llc -mtriple=amdgcn-amd- -mcpu=gfx803 -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx803 -verify-machineinstrs < %s | FileCheck %s
+
+; CHECK-LABEL: {{^}}system_monotonic_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @system_monotonic_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in monotonic monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_acquire_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @system_acquire_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in acquire monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_release_monotonic
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @system_release_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in release monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_acq_rel_monotonic
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @system_acq_rel_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in acq_rel monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_seq_cst_monotonic
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @system_seq_cst_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in seq_cst monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_acquire_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @system_acquire_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in acquire acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_release_acquire
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @system_release_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in release acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_acq_rel_acquire
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @system_acq_rel_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in acq_rel acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_seq_cst_acquire
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @system_seq_cst_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in seq_cst acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_seq_cst_seq_cst
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @system_seq_cst_seq_cst(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in seq_cst seq_cst
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_monotonic_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_monotonic_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("singlethread") monotonic monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_acquire_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_acquire_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("singlethread") acquire monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_release_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_release_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("singlethread") release monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_acq_rel_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_acq_rel_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("singlethread") acq_rel monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_seq_cst_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_seq_cst_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("singlethread") seq_cst monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_acquire_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_acquire_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("singlethread") acquire acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_release_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_release_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("singlethread") release acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_acq_rel_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_acq_rel_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("singlethread") acq_rel acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_seq_cst_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_seq_cst_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("singlethread") seq_cst acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_seq_cst_seq_cst
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_seq_cst_seq_cst(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("singlethread") seq_cst seq_cst
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_monotonic_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_monotonic_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("agent") monotonic monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_acquire_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_acquire_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("agent") acquire monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_release_monotonic
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_release_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("agent") release monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_acq_rel_monotonic
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_acq_rel_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("agent") acq_rel monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_seq_cst_monotonic
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_seq_cst_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("agent") seq_cst monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_acquire_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_acquire_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("agent") acquire acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_release_acquire
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_release_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("agent") release acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_acq_rel_acquire
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_acq_rel_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("agent") acq_rel acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_seq_cst_acquire
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_seq_cst_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("agent") seq_cst acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_seq_cst_seq_cst
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_seq_cst_seq_cst(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("agent") seq_cst seq_cst
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_monotonic_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_monotonic_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("workgroup") monotonic monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_acquire_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_acquire_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("workgroup") acquire monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_release_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_release_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("workgroup") release monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_acq_rel_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_acq_rel_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("workgroup") acq_rel monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_seq_cst_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_seq_cst_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("workgroup") seq_cst monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_acquire_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_acquire_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("workgroup") acquire acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_release_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_release_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("workgroup") release acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_acq_rel_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_acq_rel_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("workgroup") acq_rel acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_seq_cst_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_seq_cst_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("workgroup") seq_cst acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_seq_cst_seq_cst
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_seq_cst_seq_cst(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("workgroup") seq_cst seq_cst
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_monotonic_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_monotonic_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("wavefront") monotonic monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_acquire_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_acquire_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("wavefront") acquire monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_release_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_release_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("wavefront") release monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_acq_rel_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_acq_rel_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("wavefront") acq_rel monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_seq_cst_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_seq_cst_monotonic(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("wavefront") seq_cst monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_acquire_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_acquire_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("wavefront") acquire acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_release_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_release_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("wavefront") release acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_acq_rel_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_acq_rel_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("wavefront") acq_rel acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_seq_cst_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_seq_cst_acquire(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("wavefront") seq_cst acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_seq_cst_seq_cst
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_cmpswap v[{{[0-9]+\:[0-9]+}}], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_seq_cst_seq_cst(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("wavefront") seq_cst seq_cst
+  ret void
+}

test/CodeGen/AMDGPU/memory-legalizer-atomic-fence.ll

+; RUN: llc -mtriple=amdgcn-amd- -mcpu=gfx600 -verify-machineinstrs < %s | FileCheck -check-prefix=FUNC -check-prefix=GCN -check-prefix=GFX6 %s
+; RUN: llc -mtriple=amdgcn-amd- -mcpu=gfx803 -verify-machineinstrs < %s | FileCheck -check-prefix=FUNC -check-prefix=GCN -check-prefix=GFX8 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx803 -verify-machineinstrs < %s | FileCheck -check-prefix=FUNC -check-prefix=GCN -check-prefix=GFX8 %s
+
+; FUNC-LABEL: {{^}}system_acquire
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GFX6:       s_waitcnt vmcnt(0){{$}}
+; GFX6-NEXT:  buffer_wbinvl1{{$}}
+; GFX8:       s_waitcnt vmcnt(0){{$}}
+; GFX8-NEXT:  buffer_wbinvl1_vol{{$}}
+; GCN:        s_endpgm
+define amdgpu_kernel void @system_acquire() {
+entry:
+  fence acquire
+  ret void
+}
+
+; FUNC-LABEL: {{^}}system_release
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_waitcnt vmcnt(0){{$}}
+; GCN:        s_endpgm
+define amdgpu_kernel void @system_release() {
+entry:
+  fence release
+  ret void
+}
+
+; FUNC-LABEL: {{^}}system_acq_rel
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_waitcnt vmcnt(0){{$}}
+; GFX6:       buffer_wbinvl1{{$}}
+; GFX8:       buffer_wbinvl1_vol{{$}}
+; GCN:        s_endpgm
+define amdgpu_kernel void @system_acq_rel() {
+entry:
+  fence acq_rel
+  ret void
+}
+
+; FUNC-LABEL: {{^}}system_seq_cst
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_waitcnt vmcnt(0){{$}}
+; GFX6:       buffer_wbinvl1{{$}}
+; GFX8:       buffer_wbinvl1_vol{{$}}
+; GCN:        s_endpgm
+define amdgpu_kernel void @system_seq_cst() {
+entry:
+  fence seq_cst
+  ret void
+}
+
+; FUNC-LABEL: {{^}}singlethread_acquire
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_endpgm
+define amdgpu_kernel void @singlethread_acquire() {
+entry:
+  fence syncscope("singlethread") acquire
+  ret void
+}
+
+; FUNC-LABEL: {{^}}singlethread_release
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_endpgm
+define amdgpu_kernel void @singlethread_release() {
+entry:
+  fence syncscope("singlethread") release
+  ret void
+}
+
+; FUNC-LABEL: {{^}}singlethread_acq_rel
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_endpgm
+define amdgpu_kernel void @singlethread_acq_rel() {
+entry:
+  fence syncscope("singlethread") acq_rel
+  ret void
+}
+
+; FUNC-LABEL: {{^}}singlethread_seq_cst
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_endpgm
+define amdgpu_kernel void @singlethread_seq_cst() {
+entry:
+  fence syncscope("singlethread") seq_cst
+  ret void
+}
+
+; FUNC-LABEL: {{^}}agent_acquire
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GFX6:       s_waitcnt vmcnt(0){{$}}
+; GFX6-NEXT:  buffer_wbinvl1{{$}}
+; GFX8:       s_waitcnt vmcnt(0){{$}}
+; GFX8-NEXT:  buffer_wbinvl1_vol{{$}}
+; GCN:        s_endpgm
+define amdgpu_kernel void @agent_acquire() {
+entry:
+  fence syncscope("agent") acquire
+  ret void
+}
+
+; FUNC-LABEL: {{^}}agent_release
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_waitcnt vmcnt(0){{$}}
+; GCN:        s_endpgm
+define amdgpu_kernel void @agent_release() {
+entry:
+  fence syncscope("agent") release
+  ret void
+}
+
+; FUNC-LABEL: {{^}}agent_acq_rel
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_waitcnt vmcnt(0){{$}}
+; GFX6:       buffer_wbinvl1{{$}}
+; GFX8:       buffer_wbinvl1_vol{{$}}
+; GCN:        s_endpgm
+define amdgpu_kernel void @agent_acq_rel() {
+entry:
+  fence syncscope("agent") acq_rel
+  ret void
+}
+
+; FUNC-LABEL: {{^}}agent_seq_cst
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_waitcnt vmcnt(0){{$}}
+; GFX6:       buffer_wbinvl1{{$}}
+; GFX8:       buffer_wbinvl1_vol{{$}}
+; GCN:        s_endpgm
+define amdgpu_kernel void @agent_seq_cst() {
+entry:
+  fence syncscope("agent") seq_cst
+  ret void
+}
+
+; FUNC-LABEL: {{^}}workgroup_acquire
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_endpgm
+define amdgpu_kernel void @workgroup_acquire() {
+entry:
+  fence syncscope("workgroup") acquire
+  ret void
+}
+
+; FUNC-LABEL: {{^}}workgroup_release
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_endpgm
+define amdgpu_kernel void @workgroup_release() {
+entry:
+  fence syncscope("workgroup") release
+  ret void
+}
+
+; FUNC-LABEL: {{^}}workgroup_acq_rel
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_endpgm
+define amdgpu_kernel void @workgroup_acq_rel() {
+entry:
+  fence syncscope("workgroup") acq_rel
+  ret void
+}
+
+; FUNC-LABEL: {{^}}workgroup_seq_cst
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_endpgm
+define amdgpu_kernel void @workgroup_seq_cst() {
+entry:
+  fence syncscope("workgroup") seq_cst
+  ret void
+}
+
+; FUNC-LABEL: {{^}}wavefront_acquire
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_endpgm
+define amdgpu_kernel void @wavefront_acquire() {
+entry:
+  fence syncscope("wavefront") acquire
+  ret void
+}
+
+; FUNC-LABEL: {{^}}wavefront_release
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_endpgm
+define amdgpu_kernel void @wavefront_release() {
+entry:
+  fence syncscope("wavefront") release
+  ret void
+}
+
+; FUNC-LABEL: {{^}}wavefront_acq_rel
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_endpgm
+define amdgpu_kernel void @wavefront_acq_rel() {
+entry:
+  fence syncscope("wavefront") acq_rel
+  ret void
+}
+
+; FUNC-LABEL: {{^}}wavefront_seq_cst
+; GCN:        BB#0
+; GCN-NOT:    ATOMIC_FENCE
+; GCN:        s_endpgm
+define amdgpu_kernel void @wavefront_seq_cst() {
+entry:
+  fence syncscope("wavefront") seq_cst
+  ret void
+}

test/CodeGen/AMDGPU/memory-legalizer-atomic-load.ll

+; RUN: llc -mtriple=amdgcn-amd- -mcpu=gfx803 -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx803 -verify-machineinstrs < %s | FileCheck %s
+
+; CHECK-LABEL: {{^}}system_unordered
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @system_unordered(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in unordered, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @system_monotonic(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in monotonic, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}] glc{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @system_acquire(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in acquire, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_seq_cst
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}] glc{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @system_seq_cst(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in seq_cst, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_unordered
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @singlethread_unordered(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("singlethread") unordered, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @singlethread_monotonic(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("singlethread") monotonic, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @singlethread_acquire(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("singlethread") acquire, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_seq_cst
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @singlethread_seq_cst(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("singlethread") seq_cst, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_unordered
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @agent_unordered(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("agent") unordered, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @agent_monotonic(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("agent") monotonic, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}] glc{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @agent_acquire(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("agent") acquire, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_seq_cst
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}] glc{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @agent_seq_cst(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("agent") seq_cst, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_unordered
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @workgroup_unordered(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("workgroup") unordered, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @workgroup_monotonic(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("workgroup") monotonic, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @workgroup_acquire(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("workgroup") acquire, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_seq_cst
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @workgroup_seq_cst(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("workgroup") seq_cst, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_unordered
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @wavefront_unordered(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("wavefront") unordered, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @wavefront_monotonic(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("wavefront") monotonic, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @wavefront_acquire(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("wavefront") acquire, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_seq_cst
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_load_dword [[RET:v[0-9]+]], v[{{[0-9]+}}:{{[0-9]+}}]{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+; CHECK:       flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RET]]
+define amdgpu_kernel void @wavefront_seq_cst(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("wavefront") seq_cst, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}

test/CodeGen/AMDGPU/memory-legalizer-atomic-rmw.ll

+; RUN: llc -mtriple=amdgcn-amd- -mcpu=gfx803 -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx803 -verify-machineinstrs < %s | FileCheck %s
+
+; CHECK-LABEL: {{^}}system_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @system_monotonic(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @system_acquire(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_release
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @system_release(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in release
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_acq_rel
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @system_acq_rel(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in acq_rel
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_seq_cst
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @system_seq_cst(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in seq_cst
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_monotonic(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("singlethread") monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_acquire(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("singlethread") acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_release
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_release(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("singlethread") release
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_acq_rel
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_acq_rel(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("singlethread") acq_rel
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_seq_cst
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @singlethread_seq_cst(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("singlethread") seq_cst
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_monotonic(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("agent") monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_acquire(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("agent") acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_release
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_release(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("agent") release
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_acq_rel
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_acq_rel(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("agent") acq_rel
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_seq_cst
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NEXT:  s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  buffer_wbinvl1_vol
+define amdgpu_kernel void @agent_seq_cst(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("agent") seq_cst
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_monotonic(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("workgroup") monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_acquire(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("workgroup") acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_release
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_release(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("workgroup") release
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_acq_rel
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_acq_rel(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("workgroup") acq_rel
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_seq_cst
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @workgroup_seq_cst(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("workgroup") seq_cst
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_monotonic(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("wavefront") monotonic
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_acquire
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_acquire(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("wavefront") acquire
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_release
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_release(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("wavefront") release
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_acq_rel
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_acq_rel(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("wavefront") acq_rel
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_seq_cst
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_atomic_swap v[{{[0-9]+}}:{{[0-9]+}}], v{{[0-9]+}}{{$}}
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK-NOT:   buffer_wbinvl1_vol
+define amdgpu_kernel void @wavefront_seq_cst(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("wavefront") seq_cst
+  ret void
+}

test/CodeGen/AMDGPU/memory-legalizer-atomic-store.ll

+; RUN: llc -mtriple=amdgcn-amd- -mcpu=gfx803 -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx803 -verify-machineinstrs < %s | FileCheck %s
+
+; CHECK-LABEL: {{^}}system_unordered
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @system_unordered(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out unordered, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @system_monotonic(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out monotonic, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_release
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @system_release(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out release, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}system_seq_cst
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @system_seq_cst(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out seq_cst, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_unordered
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @singlethread_unordered(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("singlethread") unordered, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @singlethread_monotonic(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("singlethread") monotonic, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_release
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @singlethread_release(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("singlethread") release, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}singlethread_seq_cst
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @singlethread_seq_cst(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("singlethread") seq_cst, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_unordered
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @agent_unordered(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("agent") unordered, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @agent_monotonic(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("agent") monotonic, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_release
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @agent_release(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("agent") release, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}agent_seq_cst
+; CHECK:       s_waitcnt vmcnt(0){{$}}
+; CHECK-NEXT:  flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @agent_seq_cst(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("agent") seq_cst, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_unordered
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @workgroup_unordered(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("workgroup") unordered, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @workgroup_monotonic(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("workgroup") monotonic, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_release
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @workgroup_release(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("workgroup") release, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}workgroup_seq_cst
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @workgroup_seq_cst(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("workgroup") seq_cst, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_unordered
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @wavefront_unordered(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("wavefront") unordered, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_monotonic
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @wavefront_monotonic(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("wavefront") monotonic, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_release
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @wavefront_release(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("wavefront") release, align 4
+  ret void
+}
+
+; CHECK-LABEL: {{^}}wavefront_seq_cst
+; CHECK-NOT:   s_waitcnt vmcnt(0){{$}}
+; CHECK:       flat_store_dword v[{{[0-9]+}}:{{[0-9]+}}], {{v[0-9]+}}{{$}}
+define amdgpu_kernel void @wavefront_seq_cst(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("wavefront") seq_cst, align 4
+  ret void
+}

test/CodeGen/AMDGPU/memory-legalizer-invalid-syncscope.ll

+; RUN: not llc -mtriple=amdgcn-amd- -mcpu=gfx803 -verify-machineinstrs < %s 2>&1 | FileCheck %s
+; RUN: not llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx803 -verify-machineinstrs < %s 2>&1 | FileCheck %s
+
+; CHECK: error: <unknown>:0:0: in function invalid_fence void (): Unsupported synchronization scope
+define amdgpu_kernel void @invalid_fence() {
+entry:
+  fence syncscope("invalid") seq_cst
+  ret void
+}
+
+; CHECK: error: <unknown>:0:0: in function invalid_load void (i32 addrspace(4)*, i32 addrspace(4)*): Unsupported synchronization scope
+define amdgpu_kernel void @invalid_load(
+    i32 addrspace(4)* %in, i32 addrspace(4)* %out) {
+entry:
+  %val = load atomic i32, i32 addrspace(4)* %in syncscope("invalid") seq_cst, align 4
+  store i32 %val, i32 addrspace(4)* %out
+  ret void
+}
+
+; CHECK: error: <unknown>:0:0: in function invalid_store void (i32, i32 addrspace(4)*): Unsupported synchronization scope
+define amdgpu_kernel void @invalid_store(
+    i32 %in, i32 addrspace(4)* %out) {
+entry:
+  store atomic i32 %in, i32 addrspace(4)* %out syncscope("invalid") seq_cst, align 4
+  ret void
+}
+
+; CHECK: error: <unknown>:0:0: in function invalid_cmpxchg void (i32 addrspace(4)*, i32, i32): Unsupported synchronization scope
+define amdgpu_kernel void @invalid_cmpxchg(
+    i32 addrspace(4)* %out, i32 %in, i32 %old) {
+entry:
+  %gep = getelementptr i32, i32 addrspace(4)* %out, i32 4
+  %val = cmpxchg volatile i32 addrspace(4)* %gep, i32 %old, i32 %in syncscope("invalid") seq_cst seq_cst
+  ret void
+}
+
+; CHECK: error: <unknown>:0:0: in function invalid_rmw void (i32 addrspace(4)*, i32): Unsupported synchronization scope
+define amdgpu_kernel void @invalid_rmw(
+    i32 addrspace(4)* %out, i32 %in) {
+entry:
+  %val = atomicrmw volatile xchg i32 addrspace(4)* %out, i32 %in syncscope("invalid") seq_cst
+  ret void
+}

test/CodeGen/AMDGPU/syncscopes.ll

 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx803 -stop-before=si-debugger-insert-nops < %s | FileCheck --check-prefix=GCN %s
 
 ; GCN-LABEL: name: syncscopes
-; GCN: FLAT_STORE_DWORD killed %vgpr1_vgpr2, killed %vgpr0, 0, -1, 0, implicit %exec, implicit %flat_scr :: (volatile store syncscope("agent") seq_cst 4 into %ir.agent_out)
-; GCN: FLAT_STORE_DWORD killed %vgpr4_vgpr5, killed %vgpr3, 0, -1, 0, implicit %exec, implicit %flat_scr :: (volatile store syncscope("workgroup") seq_cst 4 into %ir.workgroup_out)
-; GCN: FLAT_STORE_DWORD killed %vgpr7_vgpr8, killed %vgpr6, 0, -1, 0, implicit %exec, implicit %flat_scr :: (volatile store syncscope("wavefront") seq_cst 4 into %ir.wavefront_out)
+; GCN: FLAT_STORE_DWORD killed %vgpr1_vgpr2, killed %vgpr0, 0, 0, 0, implicit %exec, implicit %flat_scr :: (volatile store syncscope("agent") seq_cst 4 into %ir.agent_out)
+; GCN: FLAT_STORE_DWORD killed %vgpr4_vgpr5, killed %vgpr3, 0, 0, 0, implicit %exec, implicit %flat_scr :: (volatile store syncscope("workgroup") seq_cst 4 into %ir.workgroup_out)
+; GCN: FLAT_STORE_DWORD killed %vgpr7_vgpr8, killed %vgpr6, 0, 0, 0, implicit %exec, implicit %flat_scr :: (volatile store syncscope("wavefront") seq_cst 4 into %ir.wavefront_out)
 define void @syncscopes(
     i32 %agent,
     i32 addrspace(4)* %agent_out,
     i32 %workgroup,
     i32 addrspace(4)* %workgroup_out,
     i32 %wavefront,
     i32 addrspace(4)* %wavefront_out) {
 entry:
   store atomic i32 %agent, i32 addrspace(4)* %agent_out syncscope("agent") seq_cst, align 4
   store atomic i32 %workgroup, i32 addrspace(4)* %workgroup_out syncscope("workgroup") seq_cst, align 4
   store atomic i32 %wavefront, i32 addrspace(4)* %wavefront_out syncscope("wavefront") seq_cst, align 4
   ret void
 }

test/CodeGen/MIR/AMDGPU/memory-legalizer-atomic-insert-end.mir

+# RUN: llc -march=amdgcn -mcpu=gfx803 -run-pass si-memory-legalizer  %s -o - | FileCheck %s
+
+--- |
+  ; ModuleID = '<stdin>'
+  source_filename = "<stdin>"
+  target datalayout = "e-p:32:32-p1:64:64-p2:64:64-p3:32:32-p4:64:64-p5:32:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64"
+
+  ; Function Attrs: nounwind readnone
+  declare i32 @llvm.amdgcn.workitem.id.x() #0
+
+  ; Function Attrs: nounwind
+  define amdgpu_kernel void @atomic_max_i32_noret(
+      i32 addrspace(1)* %out,
+      i32 addrspace(1)* addrspace(1)* %in,
+      i32 addrspace(1)* %x,
+      i32 %y) #1 {
+    %tid = call i32 @llvm.amdgcn.workitem.id.x()
+    %idxprom = sext i32 %tid to i64
+    %tid.gep = getelementptr i32 addrspace(1)*, i32 addrspace(1)* addrspace(1)* %in, i64 %idxprom
+    %ptr = load volatile i32 addrspace(1)*, i32 addrspace(1)* addrspace(1)* %tid.gep
+    %xor = xor i32 %tid, 1
+    %cmp = icmp ne i32 %xor, 0
+    %1 = call { i1, i64 } @llvm.amdgcn.if(i1 %cmp)
+    %2 = extractvalue { i1, i64 } %1, 0
+    %3 = extractvalue { i1, i64 } %1, 1
+    br i1 %2, label %atomic, label %exit
+
+  atomic:                                           ; preds = %0
+    %gep = getelementptr i32, i32 addrspace(1)* %ptr, i32 100
+    %ret = atomicrmw max i32 addrspace(1)* %gep, i32 %y seq_cst
+    br label %exit
+
+  exit:                                             ; preds = %atomic, %0
+    call void @llvm.amdgcn.end.cf(i64 %3)
+    ret void
+  }
+
+  declare { i1, i64 } @llvm.amdgcn.if(i1)
+
+  declare void @llvm.amdgcn.end.cf(i64)
+
+  ; Function Attrs: nounwind
+  declare void @llvm.stackprotector(i8*, i8**) #3
+
+  attributes #0 = { nounwind readnone "target-cpu"="tahiti" }
+  attributes #1 = { nounwind "target-cpu"="tahiti" }
+  attributes #2 = { readnone }
+  attributes #3 = { nounwind }
+
+...
+---
+
+# CHECK-LABEL: name: atomic_max_i32_noret
+
+# CHECK-LABEL: bb.1.atomic:
+# CHECK:       BUFFER_ATOMIC_SMAX_ADDR64
+# CHECK-NEXT:  S_WAITCNT 3952
+# CHECK-NEXT:  BUFFER_WBINVL1_VOL
+
+name:            atomic_max_i32_noret
+alignment:       0
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+tracksRegLiveness: true
+liveins:
+  - { reg: '%sgpr0_sgpr1' }
+  - { reg: '%vgpr0' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    0
+  adjustsStack:    false
+  hasCalls:        false
+  maxCallFrameSize: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+body:             |
+  bb.0 (%ir-block.0):
+    successors: %bb.1.atomic(0x40000000), %bb.2.exit(0x40000000)
+    liveins: %vgpr0, %sgpr0_sgpr1
+ 
+    %sgpr4_sgpr5 = S_LOAD_DWORDX2_IMM %sgpr0_sgpr1, 11, 0 :: (non-temporal dereferenceable invariant load 8 from `i64 addrspace(2)* undef`)
+    %vgpr1 = V_ASHRREV_I32_e32 31, %vgpr0, implicit %exec
+    %vgpr1_vgpr2 = V_LSHL_B64 %vgpr0_vgpr1, 3, implicit %exec
+    %sgpr7 = S_MOV_B32 61440
+    %sgpr6 = S_MOV_B32 0
+    S_WAITCNT 127
+    %vgpr1_vgpr2 = BUFFER_LOAD_DWORDX2_ADDR64 killed %vgpr1_vgpr2, %sgpr4_sgpr5_sgpr6_sgpr7, 0, 0, 0, 0, 0, implicit %exec :: (volatile load 8 from %ir.tid.gep)
+    %vgpr0 = V_XOR_B32_e32 1, killed %vgpr0, implicit %exec
+    V_CMP_NE_U32_e32 0, killed %vgpr0, implicit-def %vcc, implicit %exec
+    %sgpr2_sgpr3 = S_AND_SAVEEXEC_B64 killed %vcc, implicit-def %exec, implicit-def %scc, implicit %exec
+    %sgpr2_sgpr3 = S_XOR_B64 %exec, killed %sgpr2_sgpr3, implicit-def dead %scc
+    SI_MASK_BRANCH %bb.2.exit, implicit %exec
+ 
+  bb.1.atomic:
+    successors: %bb.2.exit(0x80000000)
+    liveins: %sgpr4_sgpr5_sgpr6_sgpr7:0x0000000C, %sgpr0_sgpr1, %sgpr2_sgpr3, %vgpr1_vgpr2_vgpr3_vgpr4:0x00000003
+ 
+    %sgpr0 = S_LOAD_DWORD_IMM killed %sgpr0_sgpr1, 15, 0 :: (non-temporal dereferenceable invariant load 4 from `i32 addrspace(2)* undef`)
+    dead %vgpr0 = V_MOV_B32_e32 -1, implicit %exec
+    dead %vgpr0 = V_MOV_B32_e32 61440, implicit %exec
+    %sgpr4_sgpr5 = S_MOV_B64 0
+    S_WAITCNT 127
+    %vgpr0 = V_MOV_B32_e32 killed %sgpr0, implicit %exec, implicit %exec
+    S_WAITCNT 3952
+    BUFFER_ATOMIC_SMAX_ADDR64 killed %vgpr0, killed %vgpr1_vgpr2, killed %sgpr4_sgpr5_sgpr6_sgpr7, 0, 400, 0, implicit %exec :: (volatile load seq_cst 4 from %ir.gep)
+ 
+  bb.2.exit:
+    liveins: %sgpr2_sgpr3
+
+    %exec = S_OR_B64 %exec, killed %sgpr2_sgpr3, implicit-def %scc
+    S_ENDPGM
+
+...
+