[LLVM] Add the support for fmax and fmin in atomicrmw instruction

[tianshilei1992]

Summary

This patch adds the support for fmax and fmin operations in atomicrmw
instruction. For now (at least in this patch), the instruction will be expanded
to CAS loop. There are already a couple of targets supporting the feature. I'll
create another patch(es) to enable them accordingly.

[arsenm]

llvm/docs/LangRef.rst

10377–10378

This is *not* what fmin/fmax do and ignores nan handling, and canonicalization

We plausibly need to have 3 different pairs:

• The libm behavior for fmin/fmax (which matches the misnamed llvm.minnum/maxnum intrinsics)
• The IEEE-754 2008 minnum/maxnum behavior which has different signaling nan behavior
• The IEEE-754 2019 minnum/maxnum behavior which inverts the nan behavior

This absolutely needs to state which one it is if it's just one

[tianshilei1992]

llvm/docs/LangRef.rst

10377–10378

Thanks for the information. My initial thought is we have a "unified" representation at the IR level, and leave the backend decide what should be the behavior. I'm not sure if one backend is gonna support more than one, but if that's the case, we would definitely need different representation. Based on https://en.cppreference.com/w/c/numeric/math/fmax, it looks like libm's behavior is pretty similar to what we expect to see here. Speaking of that, AMDGPU already has some support for atomic_fmax and atomic_fmin via builtins. Which standard does it follow?

[arsenm]

llvm/docs/LangRef.rst

10377–10378

Having backend interpretations of IR semantics is no good. We can have as many operations as needed and targets can request expansion as required.

The AMDGPU answer is complicated. For denormal handling it varies based on subtarget and address space. I’m not sure if they handle signaling nans correctly. For the ALU operations, it depends on the IEEE mode bit which I suspect is ignored by atomics

[tianshilei1992]

choose to let FMax and FMin match the behavior of llvm.maxnum.* and llvm.minnum.* respectively

[tianshilei1992]

update doc as well

[arsenm]

llvm/docs/LangRef.rst

1582–1585

Unrelated whitespace changes

llvm/lib/Target/X86/X86ISelLowering.cpp

44637

Unrelated whitespace change

llvm/lib/Transforms/InstCombine/InstCombineAtomicRMW.cpp

73–75

This is just wrong.

I think this is questionable for the fadd/fsub case since I guess it depends if the payload bits are preserved?

llvm/lib/Transforms/Utils/LowerAtomic.cpp

78–79

This is also wrong. Needs to insert minnum/maxnum intrinsics

[tianshilei1992]

update comments

[tianshilei1992]

fix typo

[tianshilei1992]

llvm/lib/Transforms/InstCombine/InstCombineAtomicRMW.cpp

73–75

I fixed the cases for fmax and fmin. That's the only case off the top of my head.

As for fadd/fsub, I think it will be fine because the operation will be changed to Xchg, and the result will be CF, which conforms with the standard that:

An operation that propagates a NaN operand to its result and has a single NaN as an input should produce a NaN with the payload of the input NaN if representable in the destination format.

[arsenm]

llvm/lib/Transforms/InstCombine/InstCombineAtomicRMW.cpp

73–75

This is missing test coverage

[tianshilei1992]

Add test for InstCombineAtomicRMW

[tianshilei1992]

better doc

[arsenm]

This is an IR change and should get a mention in the release notes

[tianshilei1992]

mention in release note

[tianshilei1992]

rebase and fix test failure

[tianshilei1992]

gentle ping

[tianshilei1992]

another ping

[tianshilei1992]

new week, new ping :-)

[arsenm]

llvm/lib/Transforms/InstCombine/InstCombineAtomicRMW.cpp

66–67

This isn't correct for signaling nans

llvm/test/Bitcode/compatibility.ll

870

Since you also touched dxil, is an additional test needed for dxil encoding?

[tianshilei1992]

llvm/lib/Transforms/InstCombine/InstCombineAtomicRMW.cpp

66–67

Like we mentioned above, fmax and fmin pair is expected to match the behavior of llvm.maxnum.* and llvm.minnum.*. Based on https://llvm.org/docs/LangRef.html#llvm-maxnum-intrinsic:

If either operand is a NaN, returns the other non-NaN operand. Returns NaN only if both operands are NaN. The returned NaN is always quiet.

So here, no matter what value the pointer operand would be, NaN or not, as long as we have +inf for maxnum, +inf should be returned.
As for the signaling NaNs, the manual says:

Unlike the IEEE-754 2008 behavior, this does not distinguish between signaling and quiet NaN inputs.

Therefore, I don't see how it is wrong. Could you expatiate it?

llvm/test/Bitcode/compatibility.ll

870

Thanks for catching that. Will do.

[tianshilei1992]

llvm/test/Bitcode/compatibility.ll

870

I took a quick look at tests for DXIL. There is no test for atomicrmw stuff. The test coverage is fairly minimum. I think at this moment, we could skip that.

[arsenm]

LGTM besides the test issue

llvm/lib/Transforms/InstCombine/InstCombineAtomicRMW.cpp

66–67

Right, ugh. These really should have been named fmin/fmax

llvm/test/Assembler/atomic.ll

50–55

Missing fmin test

llvm/test/Transforms/InstCombine/atomicrmw.ll

304–311

Could use some negative tests

[tianshilei1992]

fix comments

[tianshilei1992]

ping

[arsenm]

LGTM

[tianshilei1992]

In D127041#3632590, @arsenm wrote:

LGTM

Thanks for your review and all helpful comments!

[fdeazeve]

llvm/docs/GlobalISel/GenericOpcode.rst

733

I believe this patch broke the rendering of the docs.
All the intrinsics must be placed in a single line, otherwise the sequence of ^^^^ below won't work.

https://llvm.org/docs/GlobalISel/GenericOpcode.html#g-atomic-cmpxchg

llvm/docs/GlobalISel/GenericOpcode.rst

 MachineMemOperand in addition to explicit operands.
 
 G_ATOMIC_CMPXCHG
 ^^^^^^^^^^^^^^^^
 
 Generic atomic cmpxchg. Expects a MachineMemOperand in addition to explicit
 operands.
 
-G_ATOMICRMW_XCHG, G_ATOMICRMW_ADD, G_ATOMICRMW_SUB, G_ATOMICRMW_AND, G_ATOMICRMW_NAND, G_ATOMICRMW_OR, G_ATOMICRMW_XOR, G_ATOMICRMW_MAX, G_ATOMICRMW_MIN, G_ATOMICRMW_UMAX, G_ATOMICRMW_UMIN, G_ATOMICRMW_FADD, G_ATOMICRMW_FSUB
+G_ATOMICRMW_XCHG, G_ATOMICRMW_ADD, G_ATOMICRMW_SUB, G_ATOMICRMW_AND,
+G_ATOMICRMW_NAND, G_ATOMICRMW_OR, G_ATOMICRMW_XOR, G_ATOMICRMW_MAX,
+G_ATOMICRMW_MIN, G_ATOMICRMW_UMAX, G_ATOMICRMW_UMIN, G_ATOMICRMW_FADD,
+G_ATOMICRMW_FSUB, G_ATOMICRMW_FMAX, G_ATOMICRMW_FMIN
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Generic atomicrmw. Expects a MachineMemOperand in addition to explicit
 operands.
 
 G_FENCE
 ^^^^^^^
 

llvm/docs/LangRef.rst

.. code-block:: llvm
     define void @f() optsize { ... }
 
 ``alignstack(<n>)``
     This attribute indicates that, when emitting the prologue and
     epilogue, the backend should forcibly align the stack pointer.
     Specify the desired alignment, which must be a power of two, in
     parentheses.
 ``"alloc-family"="FAMILY"``
     This indicates which "family" an allocator function is part of. To avoid
     collisions, the family name should match the mangled name of the primary
     allocator function, that is "malloc" for malloc/calloc/realloc/free,
     "_Znwm" for ``::operator::new`` and ``::operator::delete``, and
     "_ZnwmSt11align_val_t" for aligned ``::operator::new`` and
     ``::operator::delete``. Matching malloc/realloc/free calls within a family
     can be optimized, but mismatched ones will be left alone.
 ``allockind("KIND")``
     Describes the behavior of an allocation function. The KIND string contains comma
     separated entries from the following options:
 
     * "alloc": the function returns a new block of memory or null.

 -  or
 -  xor
 -  max
 -  min
 -  umax
 -  umin
 -  fadd
 -  fsub
+-  fmax
+-  fmin
 
 For most of these operations, the type of '<value>' must be an integer
 type whose bit width is a power of two greater than or equal to eight
 and less than or equal to a target-specific size limit. For xchg, this
 may also be a floating point or a pointer type with the same size constraints
-as integers.  For fadd/fsub, this must be a floating point type.  The
+as integers.  For fadd/fsub/fmax/fmin, this must be a floating point type.  The
 type of the '``<pointer>``' operand must be a pointer to that type. If
 the ``atomicrmw`` is marked as ``volatile``, then the optimizer is not
 allowed to modify the number or order of execution of this
 ``atomicrmw`` with other :ref:`volatile operations <volatile>`.
 
 The instruction can take an optional ``align`` attribute.
 The alignment must be a power of two greater or equal to the size of the
 `<value>` type. If unspecified, the alignment is assumed to be equal to the

 -  or: ``*ptr = *ptr | val``
 -  xor: ``*ptr = *ptr ^ val``
 -  max: ``*ptr = *ptr > val ? *ptr : val`` (using a signed comparison)
 -  min: ``*ptr = *ptr < val ? *ptr : val`` (using a signed comparison)
 -  umax: ``*ptr = *ptr > val ? *ptr : val`` (using an unsigned comparison)
 -  umin: ``*ptr = *ptr < val ? *ptr : val`` (using an unsigned comparison)
 - fadd: ``*ptr = *ptr + val`` (using floating point arithmetic)
 - fsub: ``*ptr = *ptr - val`` (using floating point arithmetic)
+-  fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*`` intrinsic)
+-  fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*`` intrinsic)
 
 Example:
 """"""""
 
 .. code-block:: llvm
 
       %old = atomicrmw add i32* %ptr, i32 1 acquire                        ; yields i32
 

llvm/docs/ReleaseNotes.rst

 * The constant expression variants of the following instructions have been
   removed:
   * ``extractvalue``
   * ``insertvalue``
   * ``udiv``
   * ``sdiv``
   * ``urem``
   * ``srem``
+* Added the support for ``fmax`` and ``fmin`` in ``atomicrmw`` instruction. The
+  comparison is expected to match the behavior of ``llvm.maxnum.*`` and
+  ``llvm.minnum.*`` respectively.
 
 Changes to building LLVM
 ------------------------
 
 * Omitting ``CMAKE_BUILD_TYPE`` when using a single configuration generator is now
   an error. You now have to pass ``-DCMAKE_BUILD_TYPE=<type>`` in order to configure
   LLVM. This is done to help new users of LLVM select the correct type: since building
   LLVM in Debug mode is very resource intensive, we want to make sure that new users

llvm/include/llvm-c/Core.h

typedef enum {
     LLVMAtomicRMWBinOpUMax, /**< Sets the value if it's greater than the
                              original using an unsigned comparison and return
                              the old one */
     LLVMAtomicRMWBinOpUMin, /**< Sets the value if it's greater than the
                               original using an unsigned comparison and return
                               the old one */
     LLVMAtomicRMWBinOpFAdd, /**< Add a floating point value and return the
                               old one */
-    LLVMAtomicRMWBinOpFSub /**< Subtract a floating point value and return the
-                             old one */
+    LLVMAtomicRMWBinOpFSub, /**< Subtract a floating point value and return the
+    LLVMAtomicRMWBinOpFMax, /**< Sets the value if it's greater than the
+                             original using an floating point comparison and
+                             return the old one */
+    LLVMAtomicRMWBinOpFMin, /**< Sets the value if it's smaller than the
+                             original using an floating point comparison and
+                             return the old one */
 } LLVMAtomicRMWBinOp;
 
 typedef enum {
     LLVMDSError,
     LLVMDSWarning,
     LLVMDSRemark,
     LLVMDSNote
 } LLVMDiagnosticSeverity;

llvm/include/llvm/AsmParser/LLToken.h

#include "llvm/IR/Attributes.inc"
 
   // atomicrmw operations that aren't also instruction keywords.
   kw_xchg,
   kw_nand,
   kw_max,
   kw_min,
   kw_umax,
   kw_umin,
+  kw_fmax,
+  kw_fmin,
 
   // Instruction Opcodes (Opcode in UIntVal).
   kw_fneg,
   kw_add,
   kw_fadd,
   kw_sub,
   kw_fsub,
   kw_mul,

llvm/include/llvm/Bitcode/LLVMBitCodes.h

enum RMWOperations {
   RMW_NAND = 4,
   RMW_OR = 5,
   RMW_XOR = 6,
   RMW_MAX = 7,
   RMW_MIN = 8,
   RMW_UMAX = 9,
   RMW_UMIN = 10,
   RMW_FADD = 11,
-  RMW_FSUB = 12
+  RMW_FSUB = 12,
+  RMW_FMAX = 13,
+  RMW_FMIN = 14
 };
 
 /// OverflowingBinaryOperatorOptionalFlags - Flags for serializing
 /// OverflowingBinaryOperator's SubclassOptionalData contents.
 enum OverflowingBinaryOperatorOptionalFlags {
   OBO_NO_UNSIGNED_WRAP = 0,
   OBO_NO_SIGNED_WRAP = 1
 };

llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h

MachineInstrBuilder buildAtomicRMWFAdd(
     const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
     MachineMemOperand &MMO);
 
   /// Build and insert `OldValRes<def> = G_ATOMICRMW_FSUB Addr, Val, MMO`.
   MachineInstrBuilder buildAtomicRMWFSub(
         const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
         MachineMemOperand &MMO);
 
+  /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMAX Addr, Val, MMO`.
+  ///
+  /// Atomically replace the value at \p Addr with the floating point maximum of
+  /// \p Val and the original value. Puts the original value from \p Addr in \p
+  /// OldValRes.
+  ///
+  /// \pre setBasicBlock or setMI must have been called.
+  /// \pre \p OldValRes must be a generic virtual register.
+  /// \pre \p Addr must be a generic virtual register with pointer type.
+  /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+  ///      same type.
+  ///
+  /// \return a MachineInstrBuilder for the newly created instruction.
+  MachineInstrBuilder buildAtomicRMWFMax(
+        const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
+        MachineMemOperand &MMO);
+
+  /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMIN Addr, Val, MMO`.
+  ///
+  /// Atomically replace the value at \p Addr with the floating point minimum of
+  /// \p Val and the original value. Puts the original value from \p Addr in \p
+  /// OldValRes.
+  ///
+  /// \pre setBasicBlock or setMI must have been called.
+  /// \pre \p OldValRes must be a generic virtual register.
+  /// \pre \p Addr must be a generic virtual register with pointer type.
+  /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+  ///      same type.
+  ///
+  /// \return a MachineInstrBuilder for the newly created instruction.
+  MachineInstrBuilder buildAtomicRMWFMin(
+        const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
+        MachineMemOperand &MMO);
+
   /// Build and insert `G_FENCE Ordering, Scope`.
   MachineInstrBuilder buildFence(unsigned Ordering, unsigned Scope);
 
   /// Build and insert \p Dst = G_FREEZE \p Src
   MachineInstrBuilder buildFreeze(const DstOp &Dst, const SrcOp &Src) {
     return buildInstr(TargetOpcode::G_FREEZE, {Dst}, {Src});
   }
 

llvm/include/llvm/CodeGen/ISDOpcodes.h

   ATOMIC_LOAD_XOR,
   ATOMIC_LOAD_NAND,
   ATOMIC_LOAD_MIN,
   ATOMIC_LOAD_MAX,
   ATOMIC_LOAD_UMIN,
   ATOMIC_LOAD_UMAX,
   ATOMIC_LOAD_FADD,
   ATOMIC_LOAD_FSUB,
+  ATOMIC_LOAD_FMAX,
+  ATOMIC_LOAD_FMIN,
 
   // Masked load and store - consecutive vector load and store operations
   // with additional mask operand that prevents memory accesses to the
   // masked-off lanes.
   //
   // Val, OutChain = MLOAD(BasePtr, Mask, PassThru)
   // OutChain = MSTORE(Value, BasePtr, Mask)
   MLOAD,

llvm/include/llvm/CodeGen/SelectionDAGNodes.h

static bool classof(const SDNode *N) {
     case ISD::ATOMIC_LOAD_XOR:
     case ISD::ATOMIC_LOAD_NAND:
     case ISD::ATOMIC_LOAD_MIN:
     case ISD::ATOMIC_LOAD_MAX:
     case ISD::ATOMIC_LOAD_UMIN:
     case ISD::ATOMIC_LOAD_UMAX:
     case ISD::ATOMIC_LOAD_FADD:
     case ISD::ATOMIC_LOAD_FSUB:
+    case ISD::ATOMIC_LOAD_FMAX:
+    case ISD::ATOMIC_LOAD_FMIN:
     case ISD::ATOMIC_LOAD:
     case ISD::ATOMIC_STORE:
     case ISD::MLOAD:
     case ISD::MSTORE:
     case ISD::MGATHER:
     case ISD::MSCATTER:
     case ISD::VP_LOAD:
     case ISD::VP_STORE:
return N->getOpcode() == ISD::ATOMIC_CMP_SWAP     ||
            N->getOpcode() == ISD::ATOMIC_LOAD_XOR     ||
            N->getOpcode() == ISD::ATOMIC_LOAD_NAND    ||
            N->getOpcode() == ISD::ATOMIC_LOAD_MIN     ||
            N->getOpcode() == ISD::ATOMIC_LOAD_MAX     ||
            N->getOpcode() == ISD::ATOMIC_LOAD_UMIN    ||
            N->getOpcode() == ISD::ATOMIC_LOAD_UMAX    ||
            N->getOpcode() == ISD::ATOMIC_LOAD_FADD    ||
            N->getOpcode() == ISD::ATOMIC_LOAD_FSUB    ||
+           N->getOpcode() == ISD::ATOMIC_LOAD_FMAX    ||
+           N->getOpcode() == ISD::ATOMIC_LOAD_FMIN    ||
            N->getOpcode() == ISD::ATOMIC_LOAD         ||
            N->getOpcode() == ISD::ATOMIC_STORE;
   }
 };
 
 /// This SDNode is used for target intrinsics that touch
 /// memory and need an associated MachineMemOperand. Its opcode may be
 /// INTRINSIC_VOID, INTRINSIC_W_CHAIN, PREFETCH, or a target-specific opcode

llvm/include/llvm/IR/Instructions.h

enum BinOp : unsigned {
     UMin,
 
     /// *p = old + v
     FAdd,
 
     /// *p = old - v
     FSub,
 
+    /// *p = maxnum(old, v)
+    /// \p maxnum matches the behavior of \p llvm.maxnum.*.
+    FMax,
+
+    /// *p = minnum(old, v)
+    /// \p minnum matches the behavior of \p llvm.minnum.*.
+    FMin,
+
     FIRST_BINOP = Xchg,
-    LAST_BINOP = FSub,
+    LAST_BINOP = FMin,
     BAD_BINOP
   };
 
 private:
   template <unsigned Offset>
   using AtomicOrderingBitfieldElement =
       typename Bitfield::Element<AtomicOrdering, Offset, 3,
                                  AtomicOrdering::LAST>;
public:
   BinOp getOperation() const { return getSubclassData<OperationField>(); }
 
   static StringRef getOperationName(BinOp Op);
 
   static bool isFPOperation(BinOp Op) {
     switch (Op) {
     case AtomicRMWInst::FAdd:
     case AtomicRMWInst::FSub:
+    case AtomicRMWInst::FMax:
+    case AtomicRMWInst::FMin:
       return true;
     default:
       return false;
     }
   }
 
   void setOperation(BinOp Operation) {
     setSubclassData<OperationField>(Operation);

llvm/include/llvm/Support/TargetOpcodes.def

 HANDLE_TARGET_OPCODE(G_ATOMICRMW_OR)
 HANDLE_TARGET_OPCODE(G_ATOMICRMW_XOR)
 HANDLE_TARGET_OPCODE(G_ATOMICRMW_MAX)
 HANDLE_TARGET_OPCODE(G_ATOMICRMW_MIN)
 HANDLE_TARGET_OPCODE(G_ATOMICRMW_UMAX)
 HANDLE_TARGET_OPCODE(G_ATOMICRMW_UMIN)
 HANDLE_TARGET_OPCODE(G_ATOMICRMW_FADD)
 HANDLE_TARGET_OPCODE(G_ATOMICRMW_FSUB)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAX)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMIN)
 
 // Generic atomic fence
 HANDLE_TARGET_OPCODE(G_FENCE)
 
 /// Generic conditional branch instruction.
 HANDLE_TARGET_OPCODE(G_BRCOND)
 
 /// Generic indirect branch instruction.

llvm/include/llvm/Target/GenericOpcodes.td

 def G_ATOMICRMW_OR : G_ATOMICRMW_OP;
 def G_ATOMICRMW_XOR : G_ATOMICRMW_OP;
 def G_ATOMICRMW_MAX : G_ATOMICRMW_OP;
 def G_ATOMICRMW_MIN : G_ATOMICRMW_OP;
 def G_ATOMICRMW_UMAX : G_ATOMICRMW_OP;
 def G_ATOMICRMW_UMIN : G_ATOMICRMW_OP;
 def G_ATOMICRMW_FADD : G_ATOMICRMW_OP;
 def G_ATOMICRMW_FSUB : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMAX : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMIN : G_ATOMICRMW_OP;
 
 def G_FENCE : GenericInstruction {
   let OutOperandList = (outs);
   let InOperandList = (ins i32imm:$ordering, i32imm:$scope);
   let hasSideEffects = true;
 }
 
 //------------------------------------------------------------------------------

llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td

 def : GINodeEquiv<G_ATOMICRMW_OR, atomic_load_or>;
 def : GINodeEquiv<G_ATOMICRMW_XOR, atomic_load_xor>;
 def : GINodeEquiv<G_ATOMICRMW_MIN, atomic_load_min>;
 def : GINodeEquiv<G_ATOMICRMW_MAX, atomic_load_max>;
 def : GINodeEquiv<G_ATOMICRMW_UMIN, atomic_load_umin>;
 def : GINodeEquiv<G_ATOMICRMW_UMAX, atomic_load_umax>;
 def : GINodeEquiv<G_ATOMICRMW_FADD, atomic_load_fadd>;
 def : GINodeEquiv<G_ATOMICRMW_FSUB, atomic_load_fsub>;
+def : GINodeEquiv<G_ATOMICRMW_FMAX, atomic_load_fmax>;
+def : GINodeEquiv<G_ATOMICRMW_FMIN, atomic_load_fmin>;
 def : GINodeEquiv<G_FENCE, atomic_fence>;
 
 // Specifies the GlobalISel equivalents for SelectionDAG's ComplexPattern.
 // Should be used on defs that subclass GIComplexOperandMatcher<>.
 class GIComplexPatternEquiv<ComplexPattern seldag> {
   ComplexPattern SelDAGEquivalent = seldag;
 }
 
 // Specifies the GlobalISel equivalents for SelectionDAG's SDNodeXForm.
 // Should be used on defs that subclass GICustomOperandRenderer<>.
 class GISDNodeXFormEquiv<SDNodeXForm seldag> {
   SDNodeXForm SelDAGEquivalent = seldag;
 }

llvm/include/llvm/Target/TargetSelectionDAG.td

 def atomic_load_umin : SDNode<"ISD::ATOMIC_LOAD_UMIN", SDTAtomic2,
                     [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
 def atomic_load_umax : SDNode<"ISD::ATOMIC_LOAD_UMAX", SDTAtomic2,
                     [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
 def atomic_load_fadd : SDNode<"ISD::ATOMIC_LOAD_FADD" , SDTFPAtomic2,
                     [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
 def atomic_load_fsub : SDNode<"ISD::ATOMIC_LOAD_FSUB" , SDTFPAtomic2,
                     [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fmax : SDNode<"ISD::ATOMIC_LOAD_FMAX", SDTFPAtomic2,
+                    [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fmin : SDNode<"ISD::ATOMIC_LOAD_FMIN", SDTFPAtomic2,
+                    [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
 
 def atomic_load      : SDNode<"ISD::ATOMIC_LOAD", SDTAtomicLoad,
                     [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;
 def atomic_store     : SDNode<"ISD::ATOMIC_STORE", SDTAtomicStore,
                     [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;
 
 def masked_st    : SDNode<"ISD::MSTORE",  SDTMaskedStore,
                        [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;

llvm/lib/AsmParser/LLLexer.cpp

#include "llvm/IR/Attributes.inc"
   KEYWORD(samesize);
 
   KEYWORD(eq); KEYWORD(ne); KEYWORD(slt); KEYWORD(sgt); KEYWORD(sle);
   KEYWORD(sge); KEYWORD(ult); KEYWORD(ugt); KEYWORD(ule); KEYWORD(uge);
   KEYWORD(oeq); KEYWORD(one); KEYWORD(olt); KEYWORD(ogt); KEYWORD(ole);
   KEYWORD(oge); KEYWORD(ord); KEYWORD(uno); KEYWORD(ueq); KEYWORD(une);
 
   KEYWORD(xchg); KEYWORD(nand); KEYWORD(max); KEYWORD(min); KEYWORD(umax);
-  KEYWORD(umin);
+  KEYWORD(umin); KEYWORD(fmax); KEYWORD(fmin);
 
   KEYWORD(vscale);
   KEYWORD(x);
   KEYWORD(blockaddress);
   KEYWORD(dso_local_equivalent);
   KEYWORD(no_cfi);
 
   // Metadata types.

llvm/lib/AsmParser/LLParser.cpp

int LLParser::parseAtomicRMW(Instruction *&Inst, PerFunctionState &PFS) {
   case lltok::kw_fadd:
     Operation = AtomicRMWInst::FAdd;
     IsFP = true;
     break;
   case lltok::kw_fsub:
     Operation = AtomicRMWInst::FSub;
     IsFP = true;
     break;
+  case lltok::kw_fmax:
+    Operation = AtomicRMWInst::FMax;
+    IsFP = true;
+    break;
+  case lltok::kw_fmin:
+    Operation = AtomicRMWInst::FMin;
+    IsFP = true;
+    break;
   }
   Lex.Lex();  // Eat the operation.
 
   if (parseTypeAndValue(Ptr, PtrLoc, PFS) ||
       parseToken(lltok::comma, "expected ',' after atomicrmw address") ||
       parseTypeAndValue(Val, ValLoc, PFS) ||
       parseScopeAndOrdering(true /*Always atomic*/, SSID, Ordering) ||
       parseOptionalCommaAlign(Alignment, AteExtraComma))

llvm/lib/Bitcode/Reader/BitcodeReader.cpp

static AtomicRMWInst::BinOp getDecodedRMWOperation(unsigned Val) {
   case bitc::RMW_OR: return AtomicRMWInst::Or;
   case bitc::RMW_XOR: return AtomicRMWInst::Xor;
   case bitc::RMW_MAX: return AtomicRMWInst::Max;
   case bitc::RMW_MIN: return AtomicRMWInst::Min;
   case bitc::RMW_UMAX: return AtomicRMWInst::UMax;
   case bitc::RMW_UMIN: return AtomicRMWInst::UMin;
   case bitc::RMW_FADD: return AtomicRMWInst::FAdd;
   case bitc::RMW_FSUB: return AtomicRMWInst::FSub;
+  case bitc::RMW_FMAX: return AtomicRMWInst::FMax;
+  case bitc::RMW_FMIN: return AtomicRMWInst::FMin;
   }
 }
 
 static AtomicOrdering getDecodedOrdering(unsigned Val) {
   switch (Val) {
   case bitc::ORDERING_NOTATOMIC: return AtomicOrdering::NotAtomic;
   case bitc::ORDERING_UNORDERED: return AtomicOrdering::Unordered;
   case bitc::ORDERING_MONOTONIC: return AtomicOrdering::Monotonic;

llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

static unsigned getEncodedRMWOperation(AtomicRMWInst::BinOp Op) {
   case AtomicRMWInst::Or: return bitc::RMW_OR;
   case AtomicRMWInst::Xor: return bitc::RMW_XOR;
   case AtomicRMWInst::Max: return bitc::RMW_MAX;
   case AtomicRMWInst::Min: return bitc::RMW_MIN;
   case AtomicRMWInst::UMax: return bitc::RMW_UMAX;
   case AtomicRMWInst::UMin: return bitc::RMW_UMIN;
   case AtomicRMWInst::FAdd: return bitc::RMW_FADD;
   case AtomicRMWInst::FSub: return bitc::RMW_FSUB;
+  case AtomicRMWInst::FMax: return bitc::RMW_FMAX;
+  case AtomicRMWInst::FMin: return bitc::RMW_FMIN;
   }
 }
 
 static unsigned getEncodedOrdering(AtomicOrdering Ordering) {
   switch (Ordering) {
   case AtomicOrdering::NotAtomic: return bitc::ORDERING_NOTATOMIC;
   case AtomicOrdering::Unordered: return bitc::ORDERING_UNORDERED;
   case AtomicOrdering::Monotonic: return bitc::ORDERING_MONOTONIC;

llvm/lib/CodeGen/AtomicExpandPass.cpp

static ArrayRef<RTLIB::Libcall> GetRMWLibcall(AtomicRMWInst::BinOp Op) {
   case AtomicRMWInst::Xor:
     return makeArrayRef(LibcallsXor);
   case AtomicRMWInst::Nand:
     return makeArrayRef(LibcallsNand);
   case AtomicRMWInst::Max:
   case AtomicRMWInst::Min:
   case AtomicRMWInst::UMax:
   case AtomicRMWInst::UMin:
+  case AtomicRMWInst::FMax:
+  case AtomicRMWInst::FMin:
   case AtomicRMWInst::FAdd:
   case AtomicRMWInst::FSub:
     // No atomic libcalls are available for max/min/umax/umin.
     return {};
   }
   llvm_unreachable("Unexpected AtomicRMW operation.");
 }
 

llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp

case AtomicRMWInst::UMin:
     Opcode = TargetOpcode::G_ATOMICRMW_UMIN;
     break;
   case AtomicRMWInst::FAdd:
     Opcode = TargetOpcode::G_ATOMICRMW_FADD;
     break;
   case AtomicRMWInst::FSub:
     Opcode = TargetOpcode::G_ATOMICRMW_FSUB;
     break;
+  case AtomicRMWInst::FMax:
+    Opcode = TargetOpcode::G_ATOMICRMW_FMAX;
+    break;
+  case AtomicRMWInst::FMin:
+    Opcode = TargetOpcode::G_ATOMICRMW_FMIN;
+    break;
   }
 
   MIRBuilder.buildAtomicRMW(
       Opcode, Res, Addr, Val,
       *MF->getMachineMemOperand(MachinePointerInfo(I.getPointerOperand()),
                                 Flags, MRI->getType(Val), getMemOpAlign(I),
                                 I.getAAMetadata(), nullptr, I.getSyncScopeID(),
                                 I.getOrdering()));

llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp

 MachineInstrBuilder
 MachineIRBuilder::buildAtomicRMWFSub(const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
                                      MachineMemOperand &MMO) {
   return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FSUB, OldValRes, Addr, Val,
                         MMO);
 }
 
 MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMax(const DstOp &OldValRes, const SrcOp &Addr,
+                                     const SrcOp &Val, MachineMemOperand &MMO) {
+  return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMAX, OldValRes, Addr, Val,
+                        MMO);
+}
+
+MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMin(const DstOp &OldValRes, const SrcOp &Addr,
+                                     const SrcOp &Val, MachineMemOperand &MMO) {
+  return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMIN, OldValRes, Addr, Val,
+                        MMO);
+}
+
+MachineInstrBuilder
 MachineIRBuilder::buildFence(unsigned Ordering, unsigned Scope) {
   return buildInstr(TargetOpcode::G_FENCE)
     .addImm(Ordering)
     .addImm(Scope);
 }
 
 MachineInstrBuilder
 MachineIRBuilder::buildBlockAddress(Register Res, const BlockAddress *BA) {

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

assert((Opcode == ISD::ATOMIC_LOAD_ADD ||
           Opcode == ISD::ATOMIC_LOAD_XOR ||
           Opcode == ISD::ATOMIC_LOAD_NAND ||
           Opcode == ISD::ATOMIC_LOAD_MIN ||
           Opcode == ISD::ATOMIC_LOAD_MAX ||
           Opcode == ISD::ATOMIC_LOAD_UMIN ||
           Opcode == ISD::ATOMIC_LOAD_UMAX ||
           Opcode == ISD::ATOMIC_LOAD_FADD ||
           Opcode == ISD::ATOMIC_LOAD_FSUB ||
+          Opcode == ISD::ATOMIC_LOAD_FMAX ||
+          Opcode == ISD::ATOMIC_LOAD_FMIN ||
           Opcode == ISD::ATOMIC_SWAP ||
           Opcode == ISD::ATOMIC_STORE) &&
          "Invalid Atomic Op");
 
   EVT VT = Val.getValueType();
 
   SDVTList VTs = Opcode == ISD::ATOMIC_STORE ? getVTList(MVT::Other) :
                                                getVTList(VT, MVT::Other);

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

void SelectionDAGBuilder::visitAtomicRMW(const AtomicRMWInst &I) {
   case AtomicRMWInst::Or:   NT = ISD::ATOMIC_LOAD_OR; break;
   case AtomicRMWInst::Xor:  NT = ISD::ATOMIC_LOAD_XOR; break;
   case AtomicRMWInst::Max:  NT = ISD::ATOMIC_LOAD_MAX; break;
   case AtomicRMWInst::Min:  NT = ISD::ATOMIC_LOAD_MIN; break;
   case AtomicRMWInst::UMax: NT = ISD::ATOMIC_LOAD_UMAX; break;
   case AtomicRMWInst::UMin: NT = ISD::ATOMIC_LOAD_UMIN; break;
   case AtomicRMWInst::FAdd: NT = ISD::ATOMIC_LOAD_FADD; break;
   case AtomicRMWInst::FSub: NT = ISD::ATOMIC_LOAD_FSUB; break;
+  case AtomicRMWInst::FMax: NT = ISD::ATOMIC_LOAD_FMAX; break;
+  case AtomicRMWInst::FMin: NT = ISD::ATOMIC_LOAD_FMIN; break;
   }
   AtomicOrdering Ordering = I.getOrdering();
   SyncScope::ID SSID = I.getSyncScopeID();
 
   SDValue InChain = getRoot();
 
   auto MemVT = getValue(I.getValOperand()).getSimpleValueType();
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();

llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp

Value *OpenMPIRBuilder::emitRMWOpAsInstruction(Value *Src1, Value *Src2,
   case AtomicRMWInst::Xchg:
   case AtomicRMWInst::FAdd:
   case AtomicRMWInst::FSub:
   case AtomicRMWInst::BAD_BINOP:
   case AtomicRMWInst::Max:
   case AtomicRMWInst::Min:
   case AtomicRMWInst::UMax:
   case AtomicRMWInst::UMin:
+  case AtomicRMWInst::FMax:
+  case AtomicRMWInst::FMin:
     llvm_unreachable("Unsupported atomic update operation");
   }
   llvm_unreachable("Unsupported atomic update operation");
 }
 
 std::pair<Value *, Value *> OpenMPIRBuilder::emitAtomicUpdate(
     InsertPointTy AllocaIP, Value *X, Type *XElemTy, Value *Expr,
     AtomicOrdering AO, AtomicRMWInst::BinOp RMWOp,

llvm/lib/IR/Core.cpp

switch (BinOp) {
     case LLVMAtomicRMWBinOpOr: return AtomicRMWInst::Or;
     case LLVMAtomicRMWBinOpXor: return AtomicRMWInst::Xor;
     case LLVMAtomicRMWBinOpMax: return AtomicRMWInst::Max;
     case LLVMAtomicRMWBinOpMin: return AtomicRMWInst::Min;
     case LLVMAtomicRMWBinOpUMax: return AtomicRMWInst::UMax;
     case LLVMAtomicRMWBinOpUMin: return AtomicRMWInst::UMin;
     case LLVMAtomicRMWBinOpFAdd: return AtomicRMWInst::FAdd;
     case LLVMAtomicRMWBinOpFSub: return AtomicRMWInst::FSub;
+    case LLVMAtomicRMWBinOpFMax: return AtomicRMWInst::FMax;
+    case LLVMAtomicRMWBinOpFMin: return AtomicRMWInst::FMin;
   }
 
   llvm_unreachable("Invalid LLVMAtomicRMWBinOp value!");
 }
 
 static LLVMAtomicRMWBinOp mapToLLVMRMWBinOp(AtomicRMWInst::BinOp BinOp) {
   switch (BinOp) {
     case AtomicRMWInst::Xchg: return LLVMAtomicRMWBinOpXchg;
     case AtomicRMWInst::Add: return LLVMAtomicRMWBinOpAdd;
     case AtomicRMWInst::Sub: return LLVMAtomicRMWBinOpSub;
     case AtomicRMWInst::And: return LLVMAtomicRMWBinOpAnd;
     case AtomicRMWInst::Nand: return LLVMAtomicRMWBinOpNand;
     case AtomicRMWInst::Or: return LLVMAtomicRMWBinOpOr;
     case AtomicRMWInst::Xor: return LLVMAtomicRMWBinOpXor;
     case AtomicRMWInst::Max: return LLVMAtomicRMWBinOpMax;
     case AtomicRMWInst::Min: return LLVMAtomicRMWBinOpMin;
     case AtomicRMWInst::UMax: return LLVMAtomicRMWBinOpUMax;
     case AtomicRMWInst::UMin: return LLVMAtomicRMWBinOpUMin;
     case AtomicRMWInst::FAdd: return LLVMAtomicRMWBinOpFAdd;
     case AtomicRMWInst::FSub: return LLVMAtomicRMWBinOpFSub;
+    case AtomicRMWInst::FMax: return LLVMAtomicRMWBinOpFMax;
+    case AtomicRMWInst::FMin: return LLVMAtomicRMWBinOpFMin;
     default: break;
   }
 
   llvm_unreachable("Invalid AtomicRMWBinOp value!");
 }
 
 // TODO: Should this and other atomic instructions support building with
 // "syncscope"?

llvm/lib/IR/Instructions.cpp

StringRef AtomicRMWInst::getOperationName(BinOp Op) {
   case AtomicRMWInst::UMax:
     return "umax";
   case AtomicRMWInst::UMin:
     return "umin";
   case AtomicRMWInst::FAdd:
     return "fadd";
   case AtomicRMWInst::FSub:
     return "fsub";
+  case AtomicRMWInst::FMax:
+    return "fmax";
+  case AtomicRMWInst::FMin:
+    return "fmin";
   case AtomicRMWInst::BAD_BINOP:
     return "<invalid operation>";
   }
 
   llvm_unreachable("invalid atomicrmw operation");
 }
 
 //===----------------------------------------------------------------------===//

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp

 }
 
 TargetLowering::AtomicExpansionKind
 AMDGPUTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *RMW) const {
   switch (RMW->getOperation()) {
   case AtomicRMWInst::Nand:
   case AtomicRMWInst::FAdd:
   case AtomicRMWInst::FSub:
+  case AtomicRMWInst::FMax:
+  case AtomicRMWInst::FMin:
     return AtomicExpansionKind::CmpXChg;
   default:
     return AtomicExpansionKind::None;
   }
 }
 
 bool AMDGPUTargetLowering::isConstantUnsignedBitfieldExtractLegal(
     unsigned Opc, LLT Ty1, LLT Ty2) const {
   return (Ty1 == LLT::scalar(32) || Ty1 == LLT::scalar(64)) &&
          Ty2 == LLT::scalar(32);
 }

llvm/lib/Target/DirectX/DXILWriter/DXILBitcodeWriter.cpp

unsigned DXILBitcodeWriter::getEncodedRMWOperation(AtomicRMWInst::BinOp Op) {
   case AtomicRMWInst::UMax:
     return bitc::RMW_UMAX;
   case AtomicRMWInst::UMin:
     return bitc::RMW_UMIN;
   case AtomicRMWInst::FAdd:
     return bitc::RMW_FADD;
   case AtomicRMWInst::FSub:
     return bitc::RMW_FSUB;
+  case AtomicRMWInst::FMax:
+    return bitc::RMW_FMAX;
+  case AtomicRMWInst::FMin:
+    return bitc::RMW_FMIN;
   }
 }
 
 unsigned DXILBitcodeWriter::getEncodedOrdering(AtomicOrdering Ordering) {
   switch (Ordering) {
   case AtomicOrdering::NotAtomic:
     return bitc::ORDERING_NOTATOMIC;
   case AtomicOrdering::Unordered:

llvm/lib/Target/X86/X86ISelLowering.cpp

case AtomicRMWInst::Xor:
     return shouldExpandLogicAtomicRMWInIR(AI);
   case AtomicRMWInst::Nand:
   case AtomicRMWInst::Max:
   case AtomicRMWInst::Min:
   case AtomicRMWInst::UMax:
   case AtomicRMWInst::UMin:
   case AtomicRMWInst::FAdd:
   case AtomicRMWInst::FSub:
+  case AtomicRMWInst::FMax:
+  case AtomicRMWInst::FMin:
     // These always require a non-trivial set of data operations on x86. We must
     // use a cmpxchg loop.
     return AtomicExpansionKind::CmpXChg;
   }
 }
 
 LoadInst *
 X86TargetLowering::lowerIdempotentRMWIntoFencedLoad(AtomicRMWInst *AI) const {
if (N->getOpcode() == ISD::VSELECT && Cond.hasOneUse() &&
     SDValue CondNew = DAG.getNOT(DL, Cond, CondVT);
     // Vselect cond, op1, op2 = Vselect not(cond), op2, op1
     return DAG.getSelect(DL, VT, CondNew, RHS, LHS);
   }
 
   // Attempt to convert a (vXi1 bitcast(iX Cond)) selection mask before it might
   // get split by legalization.
   if (N->getOpcode() == ISD::VSELECT && Cond.getOpcode() == ISD::BITCAST &&
       CondVT.getVectorElementType() == MVT::i1 && Cond.hasOneUse() &&
       TLI.isTypeLegal(VT.getScalarType())) {
     EVT ExtCondVT = VT.changeVectorElementTypeToInteger();
     if (SDValue ExtCond = combineToExtendBoolVectorInReg(
             ISD::SIGN_EXTEND, DL, ExtCondVT, Cond, DAG, DCI, Subtarget)) {
       ExtCond = DAG.getNode(ISD::TRUNCATE, DL, CondVT, ExtCond);
       return DAG.getSelect(DL, VT, ExtCond, LHS, RHS);
     }
   }

llvm/lib/Transforms/InstCombine/InstCombineAtomicRMW.cpp

default:
       return false;
   }
 }
 
 /// Return true if the given instruction always produces a value in memory
 /// equivalent to its value operand.
 bool isSaturating(AtomicRMWInst& RMWI) {
   if (auto CF = dyn_cast<ConstantFP>(RMWI.getValOperand()))
-    switch(RMWI.getOperation()) {
+    switch (RMWI.getOperation()) {
+    case AtomicRMWInst::FMax:
+      // maxnum(x, +inf) -> +inf
+      return !CF->isNegative() && CF->isInfinity();
+    case AtomicRMWInst::FMin:
+      // minnum(x, -inf) -> +inf
+      return CF->isNegative() && CF->isInfinity();
     case AtomicRMWInst::FAdd:
     case AtomicRMWInst::FSub:
       return CF->isNaN();
     default:
       return false;
     };
 
   auto C = dyn_cast<ConstantInt>(RMWI.getValOperand());
   if(!C)
     return false;
 
   switch(RMWI.getOperation()) {
   default:

llvm/lib/Transforms/Utils/LowerAtomic.cpp

case AtomicRMWInst::UMax:
     return Builder.CreateSelect(NewVal, Loaded, Inc, "new");
   case AtomicRMWInst::UMin:
     NewVal = Builder.CreateICmpULE(Loaded, Inc);
     return Builder.CreateSelect(NewVal, Loaded, Inc, "new");
   case AtomicRMWInst::FAdd:
     return Builder.CreateFAdd(Loaded, Inc, "new");
   case AtomicRMWInst::FSub:
     return Builder.CreateFSub(Loaded, Inc, "new");
+  case AtomicRMWInst::FMax:
+    return Builder.CreateMaxNum(Loaded, Inc);
+  case AtomicRMWInst::FMin:
+    return Builder.CreateMinNum(Loaded, Inc);
   default:
     llvm_unreachable("Unknown atomic op");
   }
 }
 
 bool llvm::lowerAtomicRMWInst(AtomicRMWInst *RMWI) {
   IRBuilder<> Builder(RMWI);
   Value *Ptr = RMWI->getPointerOperand();

llvm/test/Assembler/atomic.ll

 
 define void @fp_atomics(float* %x) {
  ; CHECK: atomicrmw fadd float* %x, float 1.000000e+00 seq_cst
   atomicrmw fadd float* %x, float 1.0 seq_cst
 
   ; CHECK: atomicrmw volatile fadd float* %x, float 1.000000e+00 seq_cst
   atomicrmw volatile fadd float* %x, float 1.0 seq_cst
 
+  ; CHECK: atomicrmw fmax float* %x, float 1.000000e+00 seq_cst
+  atomicrmw fmax float* %x, float 1.0 seq_cst
+
+  ; CHECK: atomicrmw volatile fmax float* %x, float 1.000000e+00 seq_cst
+  atomicrmw volatile fmax float* %x, float 1.0 seq_cst
+
+  ; CHECK: atomicrmw fmin float* %x, float 1.000000e+00 seq_cst
+  atomicrmw fmin float* %x, float 1.0 seq_cst
+
+  ; CHECK: atomicrmw volatile fmin float* %x, float 1.000000e+00 seq_cst
+  atomicrmw volatile fmin float* %x, float 1.0 seq_cst
+
   ret void
 }

llvm/test/Bitcode/compatibility.ll

; CHECK: %atomicrmw.xchg = atomicrmw xchg float* %word, float 1.000000e+00 monotonic
   %atomicrmw.xchg = atomicrmw xchg float* %word, float 1.0 monotonic
 
 ; CHECK: %atomicrmw.fadd = atomicrmw fadd float* %word, float 1.000000e+00 monotonic
   %atomicrmw.fadd = atomicrmw fadd float* %word, float 1.0 monotonic
 
 ; CHECK: %atomicrmw.fsub = atomicrmw fsub float* %word, float 1.000000e+00 monotonic
   %atomicrmw.fsub = atomicrmw fsub float* %word, float 1.0 monotonic
 
+; CHECK: %atomicrmw.fmax = atomicrmw fmax float* %word, float 1.000000e+00 monotonic
+  %atomicrmw.fmax = atomicrmw fmax float* %word, float 1.0 monotonic
+
+; CHECK: %atomicrmw.fmin = atomicrmw fmin float* %word, float 1.000000e+00 monotonic
+  %atomicrmw.fmin = atomicrmw fmin float* %word, float 1.0 monotonic
+
   ret void
 }
 
 define void @pointer_atomics(i8** %word) {
 ; CHECK: %atomicrmw.xchg = atomicrmw xchg i8** %word, i8* null monotonic
   %atomicrmw.xchg = atomicrmw xchg i8** %word, i8* null monotonic
   ret void
 }

llvm/test/CodeGen/AArch64/GlobalISel/legalizer-info-validation.mir

 # DEBUG-NEXT: .. type index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: G_ATOMICRMW_FADD (opcode {{[0-9]+}}): 2 type indices
 # DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: G_ATOMICRMW_FSUB (opcode {{[0-9]+}}): 2 type indices
 # DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: G_ATOMICRMW_FMAX (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
+# DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: G_ATOMICRMW_FMIN (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
+# DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: G_FENCE (opcode {{[0-9]+}}): 0 type indices
 # DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: G_BRCOND (opcode {{[0-9]+}}): 1 type index, 0 imm indices
 # DEBUG-NEXT: .. the first uncovered type index: 1, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_BRINDIRECT (opcode {{[0-9]+}}): 1 type index, 0 imm indices
 # DEBUG-NEXT: .. the first uncovered type index: 1, OK

 # DEBUG-NEXT: G_VECREDUCE_MUL (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: G_VECREDUCE_AND (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. type index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: G_VECREDUCE_OR (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
-# DEBUG-NEXT: .. type index coverage check SKIPPED: user-defined predicate detected 
+# DEBUG-NEXT: .. type index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: G_VECREDUCE_XOR (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. type index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: G_VECREDUCE_SMAX (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined

llvm/test/Transforms/InstCombine/atomicrmw.ll

 ; CHECK-LABEL: undef_operand_used
 ; CHECK-NEXT: %res = atomicrmw or i32* %addr, i32 undef monotonic
 ; CHECK-NEXT: ret i32 %res
 define i32 @undef_operand_used(i32* %addr) {
   %res = atomicrmw or i32* %addr, i32 undef monotonic
   ret i32 %res
 }
 
+; CHECK-LABEL: sat_fmax_inf
+; CHECK-NEXT: %res = atomicrmw xchg double* %addr, double 0x7FF0000000000000 monotonic
+; CHECK-NEXT: ret double %res
+define double @sat_fmax_inf(double* %addr) {
+  %res = atomicrmw fmax double* %addr, double 0x7FF0000000000000 monotonic
+  ret double %res
+}
 
+; CHECK-LABEL: no_sat_fmax_inf
+; CHECK-NEXT: %res = atomicrmw fmax double* %addr, double 1.000000e-01 monotonic
+; CHECK-NEXT: ret double %res
+define double @no_sat_fmax_inf(double* %addr) {
+  %res = atomicrmw fmax double* %addr, double 1.000000e-01 monotonic
+  ret double %res
+}
 
+; CHECK-LABEL: sat_fmin_inf
+; CHECK-NEXT: %res = atomicrmw xchg double* %addr, double 0xFFF0000000000000 monotonic
+; CHECK-NEXT: ret double %res
+define double @sat_fmin_inf(double* %addr) {
+  %res = atomicrmw fmin double* %addr, double 0xFFF0000000000000 monotonic
+  ret double %res
+}
+
+; CHECK-LABEL: no_sat_fmin_inf
+; CHECK-NEXT: %res = atomicrmw fmin double* %addr, double 1.000000e-01 monotonic
+; CHECK-NEXT: ret double %res
+define double @no_sat_fmin_inf(double* %addr) {
+  %res = atomicrmw fmin double* %addr, double 1.000000e-01 monotonic
+  ret double %res
+}

llvm/test/Transforms/LowerAtomic/atomic-load.ll

; CHECK-LABEL: @fsub(
   %i = alloca float
   %j = atomicrmw fsub float* %i, float 42.0 monotonic
 ; CHECK: [[INST:%[a-z0-9]+]] = load
 ; CHECK-NEXT: fsub
 ; CHECK-NEXT: store
   ret float %j
 ; CHECK: ret float [[INST]]
 }
+
+define float @fmax() {
+; CHECK-LABEL: @fmax(
+  %i = alloca float
+  %j = atomicrmw fmax float* %i, float 42.0 monotonic
+; CHECK: [[INST:%[a-z0-9]+]] = load
+; CHECK-NEXT: call float @llvm.maxnum.f32
+; CHECK-NEXT: store
+  ret float %j
+; CHECK: ret float [[INST]]
+}
+
+define float @fmin() {
+; CHECK-LABEL: @fmin(
+  %i = alloca float
+  %j = atomicrmw fmin float* %i, float 42.0 monotonic
+; CHECK: [[INST:%[a-z0-9]+]] = load
+; CHECK-NEXT: call float @llvm.minnum.f32
+; CHECK-NEXT: store
+  ret float %j
+; CHECK: ret float [[INST]]
+}