[X86] Support llvm.{min,max}imum.f{16,32,64}

[e-kud]

Summary

Addresses https://github.com/llvm/llvm-project/issues/53353

[pengfei]

llvm/lib/Target/X86/X86ISelLowering.cpp

53226

The code doesn't combine anything, should be moved to LowerFMinimumFMaximum?

53267

Should we not do it for hasNoSignedZeros?

llvm/test/CodeGen/X86/fminimum-fmaximum.ll

182

The test does show anything interesting.

[pengfei]

llvm/test/CodeGen/X86/fminimum-fmaximum.ll

182

does -> doesn't

[e-kud]

Rebased.
Supported cases with nsz and nnan.
Updated tests.

[e-kud]

llvm/lib/Target/X86/X86ISelLowering.cpp

53226

I've tried to find a better place to handle ISD::FMAXIMUM but there is no place in chain Selecting->Combining->Legalization. So, I was inspired by ISD::FMAXNUM that is actually lowered during combining.

If you are about naming solely, it is not a big deal to change the name, but all callees in PerformDAGCombine are named as combine* even for ISD::FMAXNUM. Do we really want to have a single LowerFMinimumFMaximum?

llvm/test/CodeGen/X86/fminimum-fmaximum.ll

182

Probably, yes. I wanted to show that we are able to fold all these checks with constant arguments. But we've already tested folding of zero and nan checks. Original FMAX and FMIN are tested as well. Dropped it.

[pengfei]

Do you have plan to support minimumNumber?

llvm/lib/Target/X86/X86ISelLowering.cpp

53226

No, not the name. Did you try set action of FMAXIMUM to Custom? But I'm fine given it's similar to combineFMinNumFMaxNum.

53237

Subtarget.hasFP16() && VT == MVT::f16

53258

Better to give a table like above

//                   Op1                        Op1      
//               Num     xNaN                +0     -0  
//            -----------------           --------------
//       Num  |  Max  |  qNaN |       +0  |  +0  |  +0 |
// Op0        -----------------  Op0      --------------
//      xNaN  |  qNaN |  qNaN |       -0  |  +0  |  -0 |
//            -----------------           --------------

53284–53287

Can we check if it is 0 or NaN? We can put both in the second operand I think.

[pengfei]

llvm/lib/Target/X86/X86ISelLowering.cpp

53284–53287

We can use vfpclassps/d on AVX512DQ to optimize it.

[pengfei]

llvm/lib/Target/X86/X86ISelLowering.cpp

53226

Answer my previous question: combineFMinNumFMaxNum was intended here due to the reason described in D15294. I suppose the problem doesn't exist to combineFMinimumFMaximum. So I prefer to LowerFMinimumFMaximum.

[e-kud]

In D145634#4183847, @pengfei wrote:

Do you have plan to support minimumNumber?

Sorry, I didn't get it. Could you be more specific?

llvm/lib/Target/X86/X86ISelLowering.cpp

53226

Yes, thank you. I was unaware of this mechanism. It works but I've needed to include extra logic because with Custom lowering there is no more setcc combining.

53284–53287

It was my first attempt to compare with 0 and NaN at the same time. We have two problems. The first is that comparison with zero returns ZF regardless positive or negative zero is provided. The second is that we still need to know whether the second operand is NaN or not. We may have (0.0, NaN) as arguments. We checked that the first op is not NaN and is zero. Replaced it with the second when the second is NaN. It seems to me that we always need to check both operands on NaN and one check on zero.

We can use vfpclassps/d on AVX512DQ if one of operands is known never NaN. Working on it.

[pengfei]

In D145634#4187888, @e-kud wrote:

In D145634#4183847, @pengfei wrote:

Do you have plan to support minimumNumber?

Sorry, I didn't get it. Could you be more specific?

It's accompanying function of minimum in IEEE-754 2019, it will be introduced in new C/C++ standard too. I thought you are working for that.

[e-kud]

Rebased.
Moved from combine to lowering.
Supported f16 version.
Added optimization for avx512dq.
Added and updated tests.

[e-kud]

In D145634#4187914, @pengfei wrote:

In D145634#4187888, @e-kud wrote:

In D145634#4183847, @pengfei wrote:

Do you have plan to support minimumNumber?

Sorry, I didn't get it. Could you be more specific?

It's accompanying function of minimum in IEEE-754 2019, it will be introduced in new C/C++ standard too. I thought you are working for that.

I can't find any lib calls or specific intrinsics for them. I'd like to add them separately if we have any users or needs of them, do we?

[e-kud]

Broke formatting for premerge checks

[pengfei]

llvm/lib/Target/X86/X86ISelLowering.cpp

1004–1005

Make the format align with its context.

2127–2128

ditto.

29897

Put Max/Min together is a bit confusing. My first impression is it can return either +0 or -0 for a single comparison.

29926

Need hasFP16() for f16.

33600

ditto format.
Why do we need getNode?

[pengfei]

In D145634#4194990, @e-kud wrote:

In D145634#4187914, @pengfei wrote:

In D145634#4187888, @e-kud wrote:

In D145634#4183847, @pengfei wrote:

Do you have plan to support minimumNumber?

Sorry, I didn't get it. Could you be more specific?

It's accompanying function of minimum in IEEE-754 2019, it will be introduced in new C/C++ standard too. I thought you are working for that.

I can't find any lib calls or specific intrinsics for them. I'd like to add them separately if we have any users or needs of them, do we?

I heard glibc is supporting them. I'm fine with leaving it to the future. Do you have plan on the vector support?

llvm/test/CodeGen/X86/fminimum-fmaximum.ll

5

Guess you missed a AVX512F, i.e. AVX,AVX512,AVX512F

[e-kud]

Addressed formatting comments.
Check f16 explicitly even if avx512f16 implies avx512dq for now.

[pengfei]

llvm/lib/Target/X86/X86ISelLowering.cpp

29926

(VT == MVT::f16 && Subtarget.hasFP16()) || Subtarget.hasDQI()

[pengfei]

llvm/lib/Target/X86/X86ISelLowering.cpp

29926

Sorry, mistake. It should be:
(VT != MVT::f16 && Subtarget.hasDQI()) || Subtarget.hasFP16()

[e-kud]

In D145634#4195338, @pengfei wrote:

In D145634#4194990, @e-kud wrote:

In D145634#4187914, @pengfei wrote:

In D145634#4187888, @e-kud wrote:

In D145634#4183847, @pengfei wrote:

Do you have plan to support minimumNumber?

Sorry, I didn't get it. Could you be more specific?

It's accompanying function of minimum in IEEE-754 2019, it will be introduced in new C/C++ standard too. I thought you are working for that.

I can't find any lib calls or specific intrinsics for them. I'd like to add them separately if we have any users or needs of them, do we?

I heard glibc is supporting them. I'm fine with leaving it to the future. Do you have plan on the vector support?

Indeed, found this

• <math.h> functions for floating-point maximum and minimum, corresponding to new operations in IEEE 754-2019, and corresponding <tgmath.h> macros, are added from draft ISO C2X: fmaximum, fmaximum_num, fmaximum_mag, fmaximum_mag_num, fminimum, fminimum_num, fminimum_mag, fminimum_mag_num and corresponding functions for float, long double, _FloatN and _FloatNx.

About vector support. I want to try to implement vector support alternatively, transform floats into integers using shifts and compare them as integers preserving float semantics, even -0 < +0. I've tried this approach for scalars but current approach produces less code. Probably vectors can benefit more.

llvm/lib/Target/X86/X86ISelLowering.cpp

29926

Yes, thank you, missed it as fp16 implies dq. We actually can check only VT == MVT::f16 because above the predicate Subtarget.hasFP16() && VT == MVT::f16 has been checked already. Do we want to avoid such implicit implication? Alternative is VT != MVT::f16 && Subtarget.hasDQI() || VT == MVT::f16 && Subtarget.hasFP16().

33600

Format doesn't work, we have 80+ chars if it's aligned with return. We don't need getNode, I missed it when moved from combine to lowering.

[pengfei]

LGTM.

llvm/lib/Target/X86/X86ISelLowering.cpp

29926

Oh, we have checked hasFP16 in line 29876. So the way used here is correct. Sorry for the noise.

[e-kud]

@RKSimon @goldstein.w.n ping.

[e-kud]

Support i686 target: can't use integer representation of double -0.0

[e-kud]

Rebased

[e-kud]

@RKSimon @goldstein.w.n ping

[pengfei]

I think you can merge it. We don't need all reviewers sign off.

[RKSimon]

llvm/lib/Target/X86/X86ISelLowering.cpp

29864

You should be able to drop this early-out - we should never get here

29871

Again, you can drop this as the setOperationAction calls should ensure we never get here - replace it with an assertion if you're worried.

29909

auto *

29911

auto *

llvm/test/CodeGen/X86/avx512fp16-fminimum-fmaximum.ll

153 ↗

(On Diff #511826)

please can you add vector test coverage to ensure we scalarize?

llvm/test/CodeGen/X86/fminimum-fmaximum.ll

963

please can you add vector test coverage to ensure we scalarize?

[e-kud]

llvm/lib/Target/X86/X86ISelLowering.cpp

29871

Yes, you are right. Everything must be handled in X86TargetLowering. Dropped these checks.

llvm/test/CodeGen/X86/avx512fp16-fminimum-fmaximum.ll

153 ↗

(On Diff #511826)

Yes, I've added them. Also it reminded me about several commented tests with the intrinsics. Uncommented them as well.

[e-kud]

Rebased.
Uncommented existing tests for the intrinsics.
Addressed to comments.

[RKSimon]

llvm/test/CodeGen/X86/fminimum-fmaximum.ll

711

add nounwind attribute to get rid of the .cfi noise

[e-kud]

llvm/test/CodeGen/X86/fminimum-fmaximum.ll

711

I've allowed myself to add nounwind in half.ll since I've touched it. I think the attribute was missed.

[e-kud]

Rebased.
Added nounwind attribute to tests.

[RKSimon]

@pengfei @goldstein.w.n Any more comments?

llvm/test/CodeGen/X86/half.ll

957 ↗

(On Diff #514793)

pre-commit the nounwind change to keep it separate from this patch - you shouldn't really need dso_local either

[skatkov]

I have a side question (not delaying landing this one)

It looks like this change lowers vectorized form of intrinsic in a non-optimal way.
So I wonder whether there are some plans to improve it as follow-up?

[e-kud]

Rebased.
Excluded refactor of half.ll.

[e-kud]

In D145634#4291019, @skatkov wrote:

I have a side question (not delaying landing this one)

It looks like this change lowers vectorized form of intrinsic in a non-optimal way.
So I wonder whether there are some plans to improve it as follow-up?

Yes, for sure. I've tried to find a way to implement vectorized version for SSE, but nothing's come to my mind. We need at least SSE2 for PCMPEQ{W,D} because all fp comparison instructions treat -0.0 and 0.0 as equal. It seems that pentium3 will suffer from not optimal fmaximum/fminimum...

llvm/test/CodeGen/X86/half.ll

957 ↗

(On Diff #514793)

I haven't got commit access yet, here it is https://reviews.llvm.org/D149114

[skatkov]

In D145634#4294264, @e-kud wrote:

In D145634#4291019, @skatkov wrote:

I have a side question (not delaying landing this one)

It looks like this change lowers vectorized form of intrinsic in a non-optimal way.
So I wonder whether there are some plans to improve it as follow-up?

Yes, for sure. I've tried to find a way to implement vectorized version for SSE, but nothing's come to my mind. We need at least SSE2 for PCMPEQ{W,D} because all fp comparison instructions treat -0.0 and 0.0 as equal. It seems that pentium3 will suffer from not optimal fmaximum/fminimum...

Why we cannot do something like this https://godbolt.org/z/Yxfn3jTj1 ?
I mean at least for avx?

[skatkov]

In D145634#4294383, @skatkov wrote:

In D145634#4294264, @e-kud wrote:

In D145634#4291019, @skatkov wrote:

I have a side question (not delaying landing this one)

It looks like this change lowers vectorized form of intrinsic in a non-optimal way.
So I wonder whether there are some plans to improve it as follow-up?

Yes, for sure. I've tried to find a way to implement vectorized version for SSE, but nothing's come to my mind. We need at least SSE2 for PCMPEQ{W,D} because all fp comparison instructions treat -0.0 and 0.0 as equal. It seems that pentium3 will suffer from not optimal fmaximum/fminimum...

Why we cannot do something like this https://godbolt.org/z/Yxfn3jTj1 ?
I mean at least for avx?

BTW. my measurements on micro benchmarking (I know micro might cause that branch predictor works good), the version https://godbolt.org/z/rEj9GPfnY is the best one for scalar but it cannot be implemented in SelectionDAG as has a CFG.

[skatkov]

llvm/test/CodeGen/X86/fminimum-fmaximum.ll

121

Here is what I mentioned in terms of non-optimial vectorized version at least on AVX.

[RKSimon]

LGTM

llvm/lib/Target/X86/X86ISelLowering.cpp

29971

(style) remove braces from single line if()

llvm/test/CodeGen/X86/fminimum-fmaximum.ll

121

For now we just want x86 to support the intrinsics, vector optimization is better handled as a followup.

[skatkov]

llvm/test/CodeGen/X86/fminimum-fmaximum.ll

121

It is exactly what I said in the beginning of this discussion: side question not delaying landing this patch.

[RKSimon]

llvm/test/CodeGen/X86/fminimum-fmaximum.ll

121

yup - cheers

[e-kud]

Fix single line if style.

[e-kud]

In D145634#4294388, @skatkov wrote:

In D145634#4294383, @skatkov wrote:

In D145634#4294264, @e-kud wrote:

In D145634#4291019, @skatkov wrote:

I have a side question (not delaying landing this one)

It looks like this change lowers vectorized form of intrinsic in a non-optimal way.
So I wonder whether there are some plans to improve it as follow-up?

Yes, for sure. I've tried to find a way to implement vectorized version for SSE, but nothing's come to my mind. We need at least SSE2 for PCMPEQ{W,D} because all fp comparison instructions treat -0.0 and 0.0 as equal. It seems that pentium3 will suffer from not optimal fmaximum/fminimum...

Why we cannot do something like this https://godbolt.org/z/Yxfn3jTj1 ?
I mean at least for avx?

BTW. my measurements on micro benchmarking (I know micro might cause that branch predictor works good), the version https://godbolt.org/z/rEj9GPfnY is the best one for scalar but it cannot be implemented in SelectionDAG as has a CFG.

Both versions are incorrect. They don't work as expected in case of (-0.0, 0.0), (0.0, -0.0) inputs. Because comiss and max treat negative and positive zeros as equal.

[skatkov]

In D145634#4295568, @e-kud wrote:

In D145634#4294388, @skatkov wrote:

In D145634#4294383, @skatkov wrote:

In D145634#4294264, @e-kud wrote:

In D145634#4291019, @skatkov wrote:

I have a side question (not delaying landing this one)

It looks like this change lowers vectorized form of intrinsic in a non-optimal way.
So I wonder whether there are some plans to improve it as follow-up?

Yes, for sure. I've tried to find a way to implement vectorized version for SSE, but nothing's come to my mind. We need at least SSE2 for PCMPEQ{W,D} because all fp comparison instructions treat -0.0 and 0.0 as equal. It seems that pentium3 will suffer from not optimal fmaximum/fminimum...

Why we cannot do something like this https://godbolt.org/z/Yxfn3jTj1 ?
I mean at least for avx?

BTW. my measurements on micro benchmarking (I know micro might cause that branch predictor works good), the version https://godbolt.org/z/rEj9GPfnY is the best one for scalar but it cannot be implemented in SelectionDAG as has a CFG.

Both versions are incorrect. They don't work as expected in case of (-0.0, 0.0), (0.0, -0.0) inputs. Because comiss and max treat negative and positive zeros as equal.

Take a look carefully. In case of equality we check the sign of the first value. Thus comparison using ucommis is ok.

[e-kud]

In D145634#4295747, @skatkov wrote:

In D145634#4295568, @e-kud wrote:

In D145634#4294388, @skatkov wrote:

In D145634#4294383, @skatkov wrote:

In D145634#4294264, @e-kud wrote:

In D145634#4291019, @skatkov wrote:

I have a side question (not delaying landing this one)

It looks like this change lowers vectorized form of intrinsic in a non-optimal way.
So I wonder whether there are some plans to improve it as follow-up?

Yes, for sure. I've tried to find a way to implement vectorized version for SSE, but nothing's come to my mind. We need at least SSE2 for PCMPEQ{W,D} because all fp comparison instructions treat -0.0 and 0.0 as equal. It seems that pentium3 will suffer from not optimal fmaximum/fminimum...

Why we cannot do something like this https://godbolt.org/z/Yxfn3jTj1 ?
I mean at least for avx?

BTW. my measurements on micro benchmarking (I know micro might cause that branch predictor works good), the version https://godbolt.org/z/rEj9GPfnY is the best one for scalar but it cannot be implemented in SelectionDAG as has a CFG.

Both versions are incorrect. They don't work as expected in case of (-0.0, 0.0), (0.0, -0.0) inputs. Because comiss and max treat negative and positive zeros as equal.

Take a look carefully. In case of equality we check the sign of the first value. Thus comparison using ucommis is ok.

There is a need of %cmp = icmp slt i32 %bc, 0 -> %cmp = icmp sge i32 %bc, 0 to make valid.
Yeah, I got the idea that generic case may be more efficient implemented in such way.

[skatkov]

I wonder whether any problems with landing this patch?

[RKSimon]

llvm/lib/Target/X86/X86ISelLowering.cpp

1006

Do we have test coverage with SSE1 only?

[e-kud]

llvm/lib/Target/X86/X86ISelLowering.cpp

1006

Apparently, no. There is a fatal error: error in backend: Access past stack top! with doubles and +sse,-sse2

It seems I need to split float and double tests into two separate files to test SSE1 only. Are there better alternatives?

[RKSimon]

llvm/lib/Target/X86/X86ISelLowering.cpp

1006

I'd be very tempted to limit float maximum/minimum to SSE2 or later tbh

llvm/lib/Target/X86/X86ISelLowering.cpp

X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
   if (!Subtarget.useSoftFloat() && Subtarget.hasMMX()) {
     addRegisterClass(MVT::x86mmx, &X86::VR64RegClass);
     // No operations on x86mmx supported, everything uses intrinsics.
   }
 
   if (!Subtarget.useSoftFloat() && Subtarget.hasSSE1()) {
     addRegisterClass(MVT::v4f32, Subtarget.hasVLX() ? &X86::VR128XRegClass
                                                     : &X86::VR128RegClass);
 
     setOperationAction(ISD::FNEG,               MVT::v4f32, Custom);
     setOperationAction(ISD::FABS,               MVT::v4f32, Custom);
     setOperationAction(ISD::FCOPYSIGN,          MVT::v4f32, Custom);
     setOperationAction(ISD::BUILD_VECTOR,       MVT::v4f32, Custom);
     setOperationAction(ISD::VECTOR_SHUFFLE,     MVT::v4f32, Custom);
     setOperationAction(ISD::VSELECT,            MVT::v4f32, Custom);
     setOperationAction(ISD::EXTRACT_VECTOR_ELT, MVT::v4f32, Custom);
     setOperationAction(ISD::SELECT,             MVT::v4f32, Custom);
 
     setOperationAction(ISD::LOAD,               MVT::v2f32, Custom);
if (!Subtarget.useSoftFloat() && Subtarget.hasFP16()) {
     setOperationAction(ISD::SETCC,                MVT::f16, Custom);
     setOperationAction(ISD::STRICT_FSETCC,        MVT::f16, Custom);
     setOperationAction(ISD::STRICT_FSETCCS,       MVT::f16, Custom);
     setOperationAction(ISD::STRICT_FROUND,        MVT::f16, Promote);
     setOperationAction(ISD::FROUNDEVEN,           MVT::f16, Legal);
     setOperationAction(ISD::STRICT_FROUNDEVEN,    MVT::f16, Legal);
     setOperationAction(ISD::FP_ROUND,             MVT::f16, Custom);
     setOperationAction(ISD::STRICT_FP_ROUND,      MVT::f16, Custom);
     setOperationAction(ISD::FP_EXTEND,            MVT::f32, Legal);
     setOperationAction(ISD::STRICT_FP_EXTEND,     MVT::f32, Legal);
 
     setCondCodeAction(ISD::SETOEQ, MVT::f16, Expand);
     setCondCodeAction(ISD::SETUNE, MVT::f16, Expand);
 
     if (Subtarget.useAVX512Regs()) {
       setGroup(MVT::v32f16);
       setOperationAction(ISD::SCALAR_TO_VECTOR,       MVT::v32f16, Custom);
       setOperationAction(ISD::SINT_TO_FP,             MVT::v32i16, Legal);
setTargetDAGCombine({ISD::VECTOR_SHUFFLE,
                        ISD::ADD,
                        ISD::FADD,
                        ISD::FSUB,
                        ISD::FNEG,
                        ISD::FMA,
                        ISD::STRICT_FMA,
                        ISD::FMINNUM,
                        ISD::FMAXNUM,
+                       ISD::FMINIMUM,
+                       ISD::FMAXIMUM,
                        ISD::SUB,
                        ISD::LOAD,
                        ISD::MLOAD,
                        ISD::STORE,
                        ISD::MSTORE,
                        ISD::TRUNCATE,
                        ISD::ZERO_EXTEND,
                        ISD::ANY_EXTEND,
static SDValue LowerMINMAX(SDValue Op, const X86Subtarget &Subtarget,
   return SDValue();
 }
 
 static SDValue LowerABD(SDValue Op, const X86Subtarget &Subtarget,
                         SelectionDAG &DAG) {
   MVT VT = Op.getSimpleValueType();
 
   // For AVX1 cases, split to use legal ops.
   if (VT.is256BitVector() && !Subtarget.hasInt256())
     return splitVectorIntBinary(Op, DAG);
 
   if ((VT == MVT::v32i16 || VT == MVT::v64i8) && !Subtarget.useBWIRegs())
     return splitVectorIntBinary(Op, DAG);
 
   // TODO: Add TargetLowering expandABD() support.
   SDLoc dl(Op);
   bool IsSigned = Op.getOpcode() == ISD::ABDS;
   SDValue LHS = DAG.getFreeze(Op.getOperand(0));
   SDValue RHS = DAG.getFreeze(Op.getOperand(1));
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
 
   // abds(lhs, rhs) -> sub(smax(lhs,rhs), smin(lhs,rhs))
   // abdu(lhs, rhs) -> sub(umax(lhs,rhs), umin(lhs,rhs))
   unsigned MaxOpc = IsSigned ? ISD::SMAX : ISD::UMAX;
static SDValue LowerABD(SDValue Op, const X86Subtarget &Subtarget,
   EVT CCVT = TLI.getSetCCResultType(DAG.getDataLayout(), *DAG.getContext(), VT);
   ISD::CondCode CC = IsSigned ? ISD::CondCode::SETGT : ISD::CondCode::SETUGT;
   SDValue Cmp = DAG.getSetCC(dl, CCVT, LHS, RHS, CC);
   return DAG.getSelect(dl, VT, Cmp, DAG.getNode(ISD::SUB, dl, VT, LHS, RHS),
                        DAG.getNode(ISD::SUB, dl, VT, RHS, LHS));
 }
 
 static SDValue LowerMUL(SDValue Op, const X86Subtarget &Subtarget,
                         SelectionDAG &DAG) {
   SDLoc dl(Op);
   MVT VT = Op.getSimpleValueType();
 
   // Decompose 256-bit ops into 128-bit ops.
   if (VT.is256BitVector() && !Subtarget.hasInt256())
     return splitVectorIntBinary(Op, DAG);
 
   if ((VT == MVT::v32i16 || VT == MVT::v64i8) && !Subtarget.hasBWI())
     return splitVectorIntBinary(Op, DAG);
 
   SDValue A = Op.getOperand(0);
   SDValue B = Op.getOperand(1);
 
   // Lower v16i8/v32i8/v64i8 mul as sign-extension to v8i16/v16i16/v32i16
   // vector pairs, multiply and truncate.
   if (VT == MVT::v16i8 || VT == MVT::v32i8 || VT == MVT::v64i8) {
     unsigned NumElts = VT.getVectorNumElements();
 
     if ((VT == MVT::v16i8 && Subtarget.hasInt256()) ||
         (VT == MVT::v32i8 && Subtarget.canExtendTo512BW())) {
       MVT ExVT = MVT::getVectorVT(MVT::i16, VT.getVectorNumElements());
       return DAG.getNode(
           ISD::TRUNCATE, dl, VT,
           DAG.getNode(ISD::MUL, dl, ExVT,
                       DAG.getNode(ISD::ANY_EXTEND, dl, ExVT, A),
                       DAG.getNode(ISD::ANY_EXTEND, dl, ExVT, B)));
     }
 
     MVT ExVT = MVT::getVectorVT(MVT::i16, NumElts / 2);
 
     // Extract the lo/hi parts to any extend to i16.
     // We're going to mask off the low byte of each result element of the
     // pmullw, so it doesn't matter what's in the high byte of each 16-bit
     // element.
     SDValue Undef = DAG.getUNDEF(VT);
     SDValue ALo = DAG.getBitcast(ExVT, getUnpackl(DAG, dl, VT, A, Undef));
     SDValue AHi = DAG.getBitcast(ExVT, getUnpackh(DAG, dl, VT, A, Undef));
static SDValue LowerMUL(SDValue Op, const X86Subtarget &Subtarget,
   if (VT == MVT::v4i32) {
     assert(Subtarget.hasSSE2() && !Subtarget.hasSSE41() &&
            "Should not custom lower when pmulld is available!");
 
     // Extract the odd parts.
     static const int UnpackMask[] = { 1, -1, 3, -1 };
     SDValue Aodds = DAG.getVectorShuffle(VT, dl, A, A, UnpackMask);
     SDValue Bodds = DAG.getVectorShuffle(VT, dl, B, B, UnpackMask);
 
     // Multiply the even parts.
     SDValue Evens = DAG.getNode(X86ISD::PMULUDQ, dl, MVT::v2i64,
                                 DAG.getBitcast(MVT::v2i64, A),
                                 DAG.getBitcast(MVT::v2i64, B));
     // Now multiply odd parts.
     SDValue Odds = DAG.getNode(X86ISD::PMULUDQ, dl, MVT::v2i64,
                                DAG.getBitcast(MVT::v2i64, Aodds),
                                DAG.getBitcast(MVT::v2i64, Bodds));
SDValue X86TargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   case ISD::USUBSAT:
   case ISD::SSUBSAT:            return LowerADDSAT_SUBSAT(Op, DAG, Subtarget);
   case ISD::SMAX:
   case ISD::SMIN:
   case ISD::UMAX:
   case ISD::UMIN:               return LowerMINMAX(Op, Subtarget, DAG);
   case ISD::ABS:                return LowerABS(Op, Subtarget, DAG);
   case ISD::ABDS:
   case ISD::ABDU:               return LowerABD(Op, Subtarget, DAG);
   case ISD::AVGCEILU:           return LowerAVG(Op, Subtarget, DAG);
   case ISD::FSINCOS:            return LowerFSINCOS(Op, Subtarget, DAG);
   case ISD::MLOAD:              return LowerMLOAD(Op, Subtarget, DAG);
   case ISD::MSTORE:             return LowerMSTORE(Op, Subtarget, DAG);
   case ISD::MGATHER:            return LowerMGATHER(Op, Subtarget, DAG);
   case ISD::MSCATTER:           return LowerMSCATTER(Op, Subtarget, DAG);
   case ISD::GC_TRANSITION_START:
   case ISD::GC_TRANSITION_END:  return LowerGC_TRANSITION(Op, DAG);
static SDValue combineFMinNumFMaxNum(SDNode *N, SelectionDAG &DAG,
   SDValue MinOrMax = DAG.getNode(MinMaxOp, DL, VT, Op1, Op0);
   SDValue IsOp0Nan = DAG.getSetCC(DL, SetCCType, Op0, Op0, ISD::SETUO);
 
   // If Op0 is a NaN, select Op1. Otherwise, select the max. If both operands
   // are NaN, the NaN value of Op1 is the result.
   return DAG.getSelect(DL, VT, IsOp0Nan, Op1, MinOrMax);
 }
 
+static SDValue combineFMinimumFMaximum(SDNode *N, SelectionDAG &DAG,
+                                       const X86Subtarget &Subtarget) {
+  assert((N->getOpcode() == ISD::FMAXIMUM ||
+          N->getOpcode() == ISD::FMINIMUM) &&
+         "Expected FMAXIMUM or FMINIMUM opcode");
+  EVT VT = N->getValueType(0);
+  if (Subtarget.useSoftFloat())
+    return SDValue();
+
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+
+  if (!((Subtarget.hasSSE1() && VT == MVT::f32) ||
+        (Subtarget.hasSSE2() && VT == MVT::f64)))
+    return SDValue();
+
+  SDValue Op0 = N->getOperand(0);
+  SDValue Op1 = N->getOperand(1);
+  SDLoc DL(N);
+  uint64_t SizeInBits = VT.getFixedSizeInBits();
+  APInt PreferredZero;
+  EVT IVT = MVT::getIntegerVT(SizeInBits);
+  X86ISD::NodeType MinMaxOp;
+  if (N->getOpcode() == ISD::FMAXIMUM) {
+    PreferredZero = APInt::getZero(SizeInBits);
+    MinMaxOp = X86ISD::FMAX;
+  } else {
+    PreferredZero = APInt::getSignedMinValue(SizeInBits);
+    MinMaxOp = X86ISD::FMIN;
+  }
+  EVT SetCCType = TLI.getSetCCResultType(DAG.getDataLayout(), *DAG.getContext(),
+                                         VT);
+
+  // We reuse FMAX and FMIN operations which are not commutative. They return
+  // the second operand if at least one of operands is NaN or if both operands
+  // are equal (even if they are zeroes with different sign).
+  //
+  // Here we try to determine the correct order.
+  //
+  // We check if any of operands is NaN and return NaN. Then we check if any of
+  // operands is zero or negative zero (for fmaximum and fminimum respectively)
+  // to ensure the correct zero is returned.
+  auto IsPreferredZero = [PreferredZero](SDValue Op) {
+    Op = peekThroughBitcasts(Op);
+    if (ConstantFPSDNode *CstOp = dyn_cast<ConstantFPSDNode>(Op))
+      return CstOp->getValueAPF().bitcastToAPInt() == PreferredZero;
+    if (ConstantSDNode *CstOp = dyn_cast<ConstantSDNode>(Op))
+      return CstOp->getAPIntValue() == PreferredZero;
+    return false;
+  };
+
+  SDValue MinMax;
+  if (DAG.getTarget().Options.NoSignedZerosFPMath ||
+      N->getFlags().hasNoSignedZeros() ||
+      IsPreferredZero(Op1)) {
+    MinMax = DAG.getNode(MinMaxOp, DL, VT, Op0, Op1, N->getFlags());
+  } else if (IsPreferredZero(Op0)) {
+    MinMax = DAG.getNode(MinMaxOp, DL, VT, Op1, Op0, N->getFlags());
+  } else {
+    SDValue IsOp0Zero = DAG.getSetCC(DL, SetCCType,
+                                     DAG.getNode(ISD::BITCAST, DL, IVT, Op0),
+                                     DAG.getConstant(PreferredZero, DL, IVT),
+                                     ISD::SETEQ);
+    SDValue NewOp0 = DAG.getSelect(DL, VT, IsOp0Zero, Op1, Op0);
+    SDValue NewOp1 = DAG.getSelect(DL, VT, IsOp0Zero, Op0, Op1);
+    MinMax = DAG.getNode(MinMaxOp, DL, VT, NewOp0, NewOp1, N->getFlags());
+  }
+
+  if (DAG.isKnownNeverNaN(Op0) && DAG.isKnownNeverNaN(Op1)) {
+      return MinMax;
+  }
+
+  APFloat NaNValue = APFloat::getNaN(VT == MVT::f32 ? APFloat::IEEEsingle() : APFloat::IEEEdouble());
+  SDValue isNan = DAG.getSetCC(DL, SetCCType, Op0, Op1, ISD::SETUO);
+  return DAG.getSelect(DL, VT, isNan, DAG.getConstantFP(NaNValue, DL, VT), MinMax);
+}
+
 static SDValue combineX86INT_TO_FP(SDNode *N, SelectionDAG &DAG,
                                    TargetLowering::DAGCombinerInfo &DCI) {
   EVT VT = N->getValueType(0);
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
 
   APInt DemandedElts = APInt::getAllOnes(VT.getVectorNumElements());
   if (TLI.SimplifyDemandedVectorElts(SDValue(N, 0), DemandedElts, DCI))
     return SDValue(N, 0);
SDValue X86TargetLowering::PerformDAGCombine(SDNode *N,
   case X86ISD::FAND:        return combineFAnd(N, DAG, Subtarget);
   case X86ISD::FANDN:       return combineFAndn(N, DAG, Subtarget);
   case X86ISD::FXOR:
   case X86ISD::FOR:         return combineFOr(N, DAG, DCI, Subtarget);
   case X86ISD::FMIN:
   case X86ISD::FMAX:        return combineFMinFMax(N, DAG);
   case ISD::FMINNUM:
   case ISD::FMAXNUM:        return combineFMinNumFMaxNum(N, DAG, Subtarget);
+  case ISD::FMINIMUM:
+  case ISD::FMAXIMUM:       return combineFMinimumFMaximum(N, DAG, Subtarget);
   case X86ISD::CVTSI2P:
   case X86ISD::CVTUI2P:     return combineX86INT_TO_FP(N, DAG, DCI);
   case X86ISD::CVTP2SI:
   case X86ISD::CVTP2UI:
   case X86ISD::STRICT_CVTTP2SI:
   case X86ISD::CVTTP2SI:
   case X86ISD::STRICT_CVTTP2UI:
   case X86ISD::CVTTP2UI:

llvm/test/CodeGen/X86/fminimum-fmaximum.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse2    | FileCheck %s --check-prefixes=SSE2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx     | FileCheck %s --check-prefixes=AVX,AVX1
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512f | FileCheck %s --check-prefixes=AVX,AVX512
+
+declare float @llvm.maximum.f32(float, float)
+declare double @llvm.maximum.f64(double, double)
+declare float @llvm.minimum.f32(float, float)
+declare double @llvm.minimum.f64(double, double)
+
+;
+; fmaximum
+;
+
+define float @test_fmaximum(float %x, float %y) {
+; SSE2-LABEL: test_fmaximum:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movd %xmm0, %eax
+; SSE2-NEXT:    testl %eax, %eax
+; SSE2-NEXT:    movdqa %xmm0, %xmm3
+; SSE2-NEXT:    movdqa %xmm1, %xmm2
+; SSE2-NEXT:    je .LBB0_2
+; SSE2-NEXT:  # %bb.1:
+; SSE2-NEXT:    movdqa %xmm1, %xmm3
+; SSE2-NEXT:    movdqa %xmm0, %xmm2
+; SSE2-NEXT:  .LBB0_2:
+; SSE2-NEXT:    maxss %xmm3, %xmm2
+; SSE2-NEXT:    cmpunordss %xmm1, %xmm0
+; SSE2-NEXT:    movaps %xmm0, %xmm3
+; SSE2-NEXT:    andnps %xmm2, %xmm3
+; SSE2-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; SSE2-NEXT:    andps %xmm0, %xmm1
+; SSE2-NEXT:    orps %xmm3, %xmm1
+; SSE2-NEXT:    movaps %xmm1, %xmm0
+; SSE2-NEXT:    retq
+;
+; AVX1-LABEL: test_fmaximum:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vmovd %xmm0, %eax
+; AVX1-NEXT:    testl %eax, %eax
+; AVX1-NEXT:    vmovdqa %xmm0, %xmm2
+; AVX1-NEXT:    vmovdqa %xmm1, %xmm3
+; AVX1-NEXT:    je .LBB0_2
+; AVX1-NEXT:  # %bb.1:
+; AVX1-NEXT:    vmovdqa %xmm1, %xmm2
+; AVX1-NEXT:    vmovdqa %xmm0, %xmm3
+; AVX1-NEXT:  .LBB0_2:
+; AVX1-NEXT:    vmaxss %xmm2, %xmm3, %xmm2
+; AVX1-NEXT:    vcmpunordss %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vblendvps %xmm0, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm2, %xmm0
+; AVX1-NEXT:    retq
+;
+; AVX512-LABEL: test_fmaximum:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovd %xmm0, %eax
+; AVX512-NEXT:    testl %eax, %eax
+; AVX512-NEXT:    sete %al
+; AVX512-NEXT:    kmovw %eax, %k1
+; AVX512-NEXT:    vmovdqa %xmm0, %xmm2
+; AVX512-NEXT:    vmovss %xmm1, %xmm2, %xmm2 {%k1}
+; AVX512-NEXT:    vcmpunordss %xmm1, %xmm0, %k2
+; AVX512-NEXT:    vmovss %xmm0, %xmm1, %xmm1 {%k1}
+; AVX512-NEXT:    vmaxss %xmm1, %xmm2, %xmm0
+; AVX512-NEXT:    vmovss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0 {%k2}
+; AVX512-NEXT:    retq
+  %1 = tail call float @llvm.maximum.f32(float %x, float %y)
+  ret float %1
+}
+
+define float @test_fmaximum_nan0(float %x, float %y) {
+; SSE2-LABEL: test_fmaximum_nan0:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE2-NEXT:    retq
+;
+; AVX-LABEL: test_fmaximum_nan0:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX-NEXT:    retq
+  %1 = tail call float @llvm.maximum.f32(float 0x7fff000000000000, float %y)
+  ret float %1
+}
+
+define float @test_fmaximum_nan1(float %x, float %y) {
+; SSE2-LABEL: test_fmaximum_nan1:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE2-NEXT:    retq
+;
+; AVX-LABEL: test_fmaximum_nan1:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX-NEXT:    retq
+  %1 = tail call float @llvm.maximum.f32(float %x, float 0x7fff000000000000)
+  ret float %1
+}
+
+define float @test_fmaximum_nnan(float %x, float %y) {
+; SSE2-LABEL: test_fmaximum_nnan:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movaps %xmm0, %xmm2
+; SSE2-NEXT:    addss %xmm1, %xmm0
+; SSE2-NEXT:    subss %xmm1, %xmm2
+; SSE2-NEXT:    movd %xmm0, %eax
+; SSE2-NEXT:    testl %eax, %eax
+; SSE2-NEXT:    je .LBB3_1
+; SSE2-NEXT:  # %bb.2:
+; SSE2-NEXT:    maxss %xmm2, %xmm0
+; SSE2-NEXT:    retq
+; SSE2-NEXT:  .LBB3_1:
+; SSE2-NEXT:    movaps %xmm0, %xmm1
+; SSE2-NEXT:    movaps %xmm2, %xmm0
+; SSE2-NEXT:    maxss %xmm1, %xmm0
+; SSE2-NEXT:    retq
+;
+; AVX1-LABEL: test_fmaximum_nnan:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vaddss %xmm1, %xmm0, %xmm2
+; AVX1-NEXT:    vsubss %xmm1, %xmm0, %xmm1
+; AVX1-NEXT:    vmovd %xmm2, %eax
+; AVX1-NEXT:    testl %eax, %eax
+; AVX1-NEXT:    je .LBB3_1
+; AVX1-NEXT:  # %bb.2:
+; AVX1-NEXT:    vmaxss %xmm1, %xmm2, %xmm0
+; AVX1-NEXT:    retq
+; AVX1-NEXT:  .LBB3_1:
+; AVX1-NEXT:    vmovaps %xmm2, %xmm0
+; AVX1-NEXT:    vmaxss %xmm0, %xmm1, %xmm0
+; AVX1-NEXT:    retq
+;
+; AVX512-LABEL: test_fmaximum_nnan:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vaddss %xmm1, %xmm0, %xmm2
+; AVX512-NEXT:    vsubss %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vmovd %xmm2, %eax
+; AVX512-NEXT:    testl %eax, %eax
+; AVX512-NEXT:    sete %al
+; AVX512-NEXT:    kmovw %eax, %k1
+; AVX512-NEXT:    vmovaps %xmm0, %xmm1
+; AVX512-NEXT:    vmovss %xmm2, %xmm1, %xmm1 {%k1}
+; AVX512-NEXT:    vmovss %xmm0, %xmm2, %xmm2 {%k1}
+; AVX512-NEXT:    vmaxss %xmm1, %xmm2, %xmm0
+; AVX512-NEXT:    retq
+  %1 = fadd nnan float %x, %y
+  %2 = fsub nnan float %x, %y
+  %3 = tail call float @llvm.maximum.f32(float %1, float %2)
+  ret float %3
+}
+
+define double @test_fmaximum_zero0(double %x, double %y) {
+; SSE2-LABEL: test_fmaximum_zero0:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movapd %xmm1, %xmm0
+; SSE2-NEXT:    cmpunordsd %xmm1, %xmm0
+; SSE2-NEXT:    movsd {{.*#+}} xmm2 = mem[0],zero
+; SSE2-NEXT:    andpd %xmm0, %xmm2
+; SSE2-NEXT:    xorpd %xmm3, %xmm3
+; SSE2-NEXT:    maxsd %xmm3, %xmm1
+; SSE2-NEXT:    andnpd %xmm1, %xmm0
+; SSE2-NEXT:    orpd %xmm2, %xmm0
+; SSE2-NEXT:    retq
+;
+; AVX1-LABEL: test_fmaximum_zero0:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vxorpd %xmm0, %xmm0, %xmm0
+; AVX1-NEXT:    vmaxsd %xmm0, %xmm1, %xmm0
+; AVX1-NEXT:    vcmpunordsd %xmm1, %xmm1, %xmm1
+; AVX1-NEXT:    vblendvpd %xmm1, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
+; AVX1-NEXT:    retq
+;
+; AVX512-LABEL: test_fmaximum_zero0:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vxorpd %xmm0, %xmm0, %xmm0
+; AVX512-NEXT:    vmaxsd %xmm0, %xmm1, %xmm0
+; AVX512-NEXT:    vcmpunordsd %xmm1, %xmm1, %k1
+; AVX512-NEXT:    vmovsd {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0 {%k1}
+; AVX512-NEXT:    retq
+  %1 = tail call double @llvm.maximum.f64(double 0.0, double %y)
+  ret double %1
+}
+
+define double @test_fmaximum_zero1(double %x, double %y) {
+; SSE2-LABEL: test_fmaximum_zero1:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movapd %xmm0, %xmm1
+; SSE2-NEXT:    cmpunordsd %xmm0, %xmm1
+; SSE2-NEXT:    movsd {{.*#+}} xmm2 = mem[0],zero
+; SSE2-NEXT:    andpd %xmm1, %xmm2
+; SSE2-NEXT:    xorpd %xmm3, %xmm3
+; SSE2-NEXT:    maxsd %xmm3, %xmm0
+; SSE2-NEXT:    andnpd %xmm0, %xmm1
+; SSE2-NEXT:    orpd %xmm2, %xmm1
+; SSE2-NEXT:    movapd %xmm1, %xmm0
+; SSE2-NEXT:    retq
+;
+; AVX1-LABEL: test_fmaximum_zero1:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
+; AVX1-NEXT:    vmaxsd %xmm1, %xmm0, %xmm1
+; AVX1-NEXT:    vcmpunordsd %xmm0, %xmm0, %xmm0
+; AVX1-NEXT:    vblendvpd %xmm0, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1, %xmm0
+; AVX1-NEXT:    retq
+;
+; AVX512-LABEL: test_fmaximum_zero1:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
+; AVX512-NEXT:    vmaxsd %xmm1, %xmm0, %xmm1
+; AVX512-NEXT:    vcmpunordsd %xmm0, %xmm0, %k1
+; AVX512-NEXT:    vmovsd {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1 {%k1}
+; AVX512-NEXT:    vmovapd %xmm1, %xmm0
+; AVX512-NEXT:    retq
+  %1 = tail call double @llvm.maximum.f64(double %x, double 0.0)
+  ret double %1
+}
+
+define double @test_fmaximum_zero2(double %x, double %y) {
+; SSE2-LABEL: test_fmaximum_zero2:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    xorps %xmm0, %xmm0
+; SSE2-NEXT:    retq
+;
+; AVX-LABEL: test_fmaximum_zero2:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %1 = tail call double @llvm.maximum.f64(double 0.0, double -0.0)
+  ret double %1
+}
+
+define float @test_fmaximum_nsz(float %x, float %y) "no-signed-zeros-fp-math"="true" {
+; SSE2-LABEL: test_fmaximum_nsz:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movaps %xmm0, %xmm2
+; SSE2-NEXT:    maxss %xmm1, %xmm2
+; SSE2-NEXT:    cmpunordss %xmm1, %xmm0
+; SSE2-NEXT:    movaps %xmm0, %xmm1
+; SSE2-NEXT:    andnps %xmm2, %xmm1
+; SSE2-NEXT:    movss {{.*#+}} xmm2 = mem[0],zero,zero,zero
+; SSE2-NEXT:    andps %xmm2, %xmm0
+; SSE2-NEXT:    orps %xmm1, %xmm0
+; SSE2-NEXT:    retq
+;
+; AVX1-LABEL: test_fmaximum_nsz:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vcmpunordss %xmm1, %xmm0, %xmm2
+; AVX1-NEXT:    vmaxss %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vblendvps %xmm2, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
+; AVX1-NEXT:    retq
+;
+; AVX512-LABEL: test_fmaximum_nsz:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vcmpunordss %xmm1, %xmm0, %k1
+; AVX512-NEXT:    vmaxss %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vmovss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0 {%k1}
+; AVX512-NEXT:    retq
+  %1 = tail call float @llvm.maximum.f32(float %x, float %y)
+  ret float %1
+}
+
+;
+; fminimum
+;
+
+define float @test_fminimum(float %x, float %y) {
+; SSE2-LABEL: test_fminimum:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movd %xmm0, %eax
+; SSE2-NEXT:    cmpl $-2147483648, %eax # imm = 0x80000000
+; SSE2-NEXT:    movdqa %xmm0, %xmm3
+; SSE2-NEXT:    movdqa %xmm1, %xmm2
+; SSE2-NEXT:    je .LBB8_2
+; SSE2-NEXT:  # %bb.1:
+; SSE2-NEXT:    movdqa %xmm1, %xmm3
+; SSE2-NEXT:    movdqa %xmm0, %xmm2
+; SSE2-NEXT:  .LBB8_2:
+; SSE2-NEXT:    minss %xmm3, %xmm2
+; SSE2-NEXT:    cmpunordss %xmm1, %xmm0
+; SSE2-NEXT:    movaps %xmm0, %xmm3
+; SSE2-NEXT:    andnps %xmm2, %xmm3
+; SSE2-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; SSE2-NEXT:    andps %xmm0, %xmm1
+; SSE2-NEXT:    orps %xmm3, %xmm1
+; SSE2-NEXT:    movaps %xmm1, %xmm0
+; SSE2-NEXT:    retq
+;
+; AVX1-LABEL: test_fminimum:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vmovd %xmm0, %eax
+; AVX1-NEXT:    cmpl $-2147483648, %eax # imm = 0x80000000
+; AVX1-NEXT:    vmovdqa %xmm0, %xmm2
+; AVX1-NEXT:    vmovdqa %xmm1, %xmm3
+; AVX1-NEXT:    je .LBB8_2
+; AVX1-NEXT:  # %bb.1:
+; AVX1-NEXT:    vmovdqa %xmm1, %xmm2
+; AVX1-NEXT:    vmovdqa %xmm0, %xmm3
+; AVX1-NEXT:  .LBB8_2:
+; AVX1-NEXT:    vminss %xmm2, %xmm3, %xmm2
+; AVX1-NEXT:    vcmpunordss %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vblendvps %xmm0, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm2, %xmm0
+; AVX1-NEXT:    retq
+;
+; AVX512-LABEL: test_fminimum:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovd %xmm0, %eax
+; AVX512-NEXT:    cmpl $-2147483648, %eax # imm = 0x80000000
+; AVX512-NEXT:    sete %al
+; AVX512-NEXT:    kmovw %eax, %k1
+; AVX512-NEXT:    vmovdqa %xmm0, %xmm2
+; AVX512-NEXT:    vmovss %xmm1, %xmm2, %xmm2 {%k1}
+; AVX512-NEXT:    vcmpunordss %xmm1, %xmm0, %k2
+; AVX512-NEXT:    vmovss %xmm0, %xmm1, %xmm1 {%k1}
+; AVX512-NEXT:    vminss %xmm1, %xmm2, %xmm0
+; AVX512-NEXT:    vmovss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0 {%k2}
+; AVX512-NEXT:    retq
+  %1 = tail call float @llvm.minimum.f32(float %x, float %y)
+  ret float %1
+}
+
+define float @test_fminimum_nan0(float %x, float %y) {
+; SSE2-LABEL: test_fminimum_nan0:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE2-NEXT:    retq
+;
+; AVX-LABEL: test_fminimum_nan0:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX-NEXT:    retq
+  %1 = tail call float @llvm.minimum.f32(float 0x7fff000000000000, float %y)
+  ret float %1
+}
+
+define float @test_fminimum_nan1(float %x, float %y) {
+; SSE2-LABEL: test_fminimum_nan1:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE2-NEXT:    retq
+;
+; AVX-LABEL: test_fminimum_nan1:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX-NEXT:    retq
+  %1 = tail call float @llvm.minimum.f32(float %x, float 0x7fff000000000000)
+  ret float %1
+}
+
+define float @test_fminimum_nnan(float %x, float %y) "no-nans-fp-math"="true" {
+; SSE2-LABEL: test_fminimum_nnan:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movd %xmm0, %eax
+; SSE2-NEXT:    cmpl $-2147483648, %eax # imm = 0x80000000
+; SSE2-NEXT:    je .LBB11_1
+; SSE2-NEXT:  # %bb.2:
+; SSE2-NEXT:    minss %xmm1, %xmm0
+; SSE2-NEXT:    retq
+; SSE2-NEXT:  .LBB11_1:
+; SSE2-NEXT:    movdqa %xmm0, %xmm2
+; SSE2-NEXT:    movaps %xmm1, %xmm0
+; SSE2-NEXT:    minss %xmm2, %xmm0
+; SSE2-NEXT:    retq
+;
+; AVX1-LABEL: test_fminimum_nnan:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vmovd %xmm0, %eax
+; AVX1-NEXT:    cmpl $-2147483648, %eax # imm = 0x80000000
+; AVX1-NEXT:    je .LBB11_1
+; AVX1-NEXT:  # %bb.2:
+; AVX1-NEXT:    vminss %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    retq
+; AVX1-NEXT:  .LBB11_1:
+; AVX1-NEXT:    vmovdqa %xmm0, %xmm2
+; AVX1-NEXT:    vminss %xmm2, %xmm1, %xmm0
+; AVX1-NEXT:    retq
+;
+; AVX512-LABEL: test_fminimum_nnan:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovd %xmm0, %eax
+; AVX512-NEXT:    cmpl $-2147483648, %eax # imm = 0x80000000
+; AVX512-NEXT:    sete %al
+; AVX512-NEXT:    kmovw %eax, %k1
+; AVX512-NEXT:    vmovaps %xmm1, %xmm2
+; AVX512-NEXT:    vmovss %xmm0, %xmm2, %xmm2 {%k1}
+; AVX512-NEXT:    vmovss %xmm1, %xmm0, %xmm0 {%k1}
+; AVX512-NEXT:    vminss %xmm2, %xmm0, %xmm0
+; AVX512-NEXT:    retq
+  %1 = tail call float @llvm.minimum.f32(float %x, float %y)
+  ret float %1
+}
+
+define double @test_fminimum_zero0(double %x, double %y) {
+; SSE2-LABEL: test_fminimum_zero0:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movapd %xmm1, %xmm0
+; SSE2-NEXT:    cmpunordsd %xmm1, %xmm0
+; SSE2-NEXT:    movsd {{.*#+}} xmm2 = mem[0],zero
+; SSE2-NEXT:    andpd %xmm0, %xmm2
+; SSE2-NEXT:    minsd {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1
+; SSE2-NEXT:    andnpd %xmm1, %xmm0
+; SSE2-NEXT:    orpd %xmm2, %xmm0
+; SSE2-NEXT:    retq
+;
+; AVX1-LABEL: test_fminimum_zero0:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vcmpunordsd %xmm1, %xmm1, %xmm0
+; AVX1-NEXT:    vminsd {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1, %xmm1
+; AVX1-NEXT:    vblendvpd %xmm0, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1, %xmm0
+; AVX1-NEXT:    retq
+;
+; AVX512-LABEL: test_fminimum_zero0:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vcmpunordsd %xmm1, %xmm1, %k1
+; AVX512-NEXT:    vminsd {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1, %xmm0
+; AVX512-NEXT:    vmovsd {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0 {%k1}
+; AVX512-NEXT:    retq
+  %1 = tail call double @llvm.minimum.f64(double -0.0, double %y)
+  ret double %1
+}
+
+define double @test_fminimum_zero1(double %x, double %y) {
+; SSE2-LABEL: test_fminimum_zero1:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movapd %xmm0, %xmm1
+; SSE2-NEXT:    cmpunordsd %xmm0, %xmm1
+; SSE2-NEXT:    movsd {{.*#+}} xmm2 = mem[0],zero
+; SSE2-NEXT:    andpd %xmm1, %xmm2
+; SSE2-NEXT:    minsd {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; SSE2-NEXT:    andnpd %xmm0, %xmm1
+; SSE2-NEXT:    orpd %xmm2, %xmm1
+; SSE2-NEXT:    movapd %xmm1, %xmm0
+; SSE2-NEXT:    retq
+;
+; AVX1-LABEL: test_fminimum_zero1:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vcmpunordsd %xmm0, %xmm0, %xmm1
+; AVX1-NEXT:    vminsd {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
+; AVX1-NEXT:    vblendvpd %xmm1, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
+; AVX1-NEXT:    retq
+;
+; AVX512-LABEL: test_fminimum_zero1:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vcmpunordsd %xmm0, %xmm0, %k1
+; AVX512-NEXT:    vminsd {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
+; AVX512-NEXT:    vmovsd {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0 {%k1}
+; AVX512-NEXT:    retq
+  %1 = tail call double @llvm.minimum.f64(double %x, double -0.0)
+  ret double %1
+}
+
+define double @test_fminimum_zero2(double %x, double %y) {
+; SSE2-LABEL: test_fminimum_zero2:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
+; SSE2-NEXT:    retq
+;
+; AVX-LABEL: test_fminimum_zero2:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX-NEXT:    retq
+  %1 = tail call double @llvm.minimum.f64(double -0.0, double 0.0)
+  ret double %1
+}
+
+define float @test_fminimum_nsz(float %x, float %y) {
+; SSE2-LABEL: test_fminimum_nsz:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movaps %xmm0, %xmm2
+; SSE2-NEXT:    minss %xmm1, %xmm2
+; SSE2-NEXT:    cmpunordss %xmm1, %xmm0
+; SSE2-NEXT:    movaps %xmm0, %xmm1
+; SSE2-NEXT:    andnps %xmm2, %xmm1
+; SSE2-NEXT:    movss {{.*#+}} xmm2 = mem[0],zero,zero,zero
+; SSE2-NEXT:    andps %xmm2, %xmm0
+; SSE2-NEXT:    orps %xmm1, %xmm0
+; SSE2-NEXT:    retq
+;
+; AVX1-LABEL: test_fminimum_nsz:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vcmpunordss %xmm1, %xmm0, %xmm2
+; AVX1-NEXT:    vminss %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vblendvps %xmm2, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
+; AVX1-NEXT:    retq
+;
+; AVX512-LABEL: test_fminimum_nsz:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vcmpunordss %xmm1, %xmm0, %k1
+; AVX512-NEXT:    vminss %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vmovss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0 {%k1}
+; AVX512-NEXT:    retq
+  %1 = tail call nsz float @llvm.minimum.f32(float %x, float %y)
+  ret float %1
+}