AMDGPU: Add support for amdgpu-unsafe-fp-atomics attribute

[kzhuravl]

Summary

If amdgpu-unsafe-fp-atomics is specified, allow {flat|global}_atomic_add_f32 even if atomic modes don't match.

[arsenm]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

11775–11777

Should sink this down to be the final check

[rampitec]

AFAIR the idea was to always expand fp atomics without this unsafe option.

[b-sumner]

That is the safest thing to do. However, I'm now hearing some thinking that memory likely to be targeted by f.p. atomics is likely to be cached. But again, better to be safe at least until we're clear this will always be the case.

[kzhuravl]

Address review feedback.

[arsenm]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

11775–11777

The idea was to hide the string attribute lookup as the final check. as in

!fpModeMatchesGlobalFPAtomicMode(RMW) && attributeStuff()

[rampitec]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

11776–11778

I believe condition should be "or", not "and". Expand if either is not satisfied.

[kzhuravl]

Address review feedback.

[rampitec]

LGTM

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

if (Ty->isHalfTy())
       return AtomicExpansionKind::None;
 
     if (!Ty->isFloatTy())
       return AtomicExpansionKind::CmpXChg;
 
     // TODO: Do have these for flat. Older targets also had them for buffers.
     unsigned AS = RMW->getPointerAddressSpace();
 
     if (AS == AMDGPUAS::GLOBAL_ADDRESS && Subtarget->hasAtomicFaddInsts()) {
-      if (!fpModeMatchesGlobalFPAtomicMode(RMW))
+      if (!fpModeMatchesGlobalFPAtomicMode(RMW) ||
+          RMW->getFunction()->getFnAttribute("amdgpu-unsafe-fp-atomics")
+              .getValueAsString() != "true")
         return AtomicExpansionKind::CmpXChg;
 
       return RMW->use_empty() ? AtomicExpansionKind::None :
                                 AtomicExpansionKind::CmpXChg;
     }
 
     // DS FP atomics do repect the denormal mode, but the rounding mode is fixed
     // to round-to-nearest-even.

llvm/test/CodeGen/AMDGPU/global-atomics-fp.ll

 }
 
 ; GCN-LABEL: {{^}}global_atomic_fadd_noret_f32_wrong_subtarget:
 define amdgpu_kernel void @global_atomic_fadd_noret_f32_wrong_subtarget(float addrspace(1)* %ptr) #1 {
   %result = atomicrmw fadd float addrspace(1)* %ptr, float 4.0 seq_cst
   ret void
 }
 
-attributes #0 = { "denormal-fp-math-f32"="preserve-sign,preserve-sign"}
+attributes #0 = { "denormal-fp-math-f32"="preserve-sign,preserve-sign" "amdgpu-unsafe-fp-atomics"="true" }
 attributes #1 = { "denormal-fp-math-f32"="preserve-sign,preserve-sign" "target-cpu"="gfx803" "target-features"="+atomic-fadd-insts" }

llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-rmw-fadd.ll

 ; GFX908-NEXT:    br i1 [[SUCCESS]], label [[ATOMICRMW_END:%.*]], label [[ATOMICRMW_START]]
 ; GFX908:       atomicrmw.end:
 ; GFX908-NEXT:    ret double [[TMP6]]
 ;
   %res = atomicrmw fadd double addrspace(3)* %ptr, double %value seq_cst
   ret double %res
 }
 
-attributes #0 = { "denormal-fp-math-f32"="preserve-sign,preserve-sign" }
+attributes #0 = { "denormal-fp-math-f32"="preserve-sign,preserve-sign" "amdgpu-unsafe-fp-atomics"="true" }