[LV] Add support for minimum/maximum intrinsics

[anna]

Summary

{mini|maxi}mum intrinsics are different from {min|max}num intrinsics
in the propagation of NaN and signed zero. Also, the minnum/maxnum intrinsics require the presence of nsz flags to be valid reductions in vectorizer. In this regard, we introduce a new recurrence kind and also add support for identifying reduction patterns using these intrinsics.

There are tests added which show how this interacts across chains of
min/max patterns.

[anna]

The motivation here is we added support for lowering of minimum/maximum in X86:https://reviews.llvm.org/D149844
This is the last bit needed to support generating these vectorized fminimum/maximum through the optimizer, which should benefit other targets (apart from X86) that lower these intrinsics.

[dmgreen]

llvm/lib/Analysis/IVDescriptors.cpp

814

As far as I understand from the language ref, llvm.vector.reduce.fmin will be unspecified for 0.0/-0.0, and will return a value unless all the inputs are Nan. https://llvm.org/docs/LangRef.html#llvm-vector-reduce-fmin-intrinsic. This matches the behaviour of llvm.minnum. llvm.minimum has stronger semantics for signed zero and propagates nans, so would only be valid to convert with fast math flags.

I'm not 100% sure that llvm.minnum would need fast math flags, but llvm.minimum would seem to. Does it sound like I have that the right way around?

[anna]

llvm/lib/Analysis/IVDescriptors.cpp

814

As far as I understand from the language ref, llvm.vector.reduce.fmin will be unspecified for 0.0/-0.0, and will return a value unless all the inputs are Nan.

Ah, looks like vector.reduce.fmin was designed with only minnum in mind according to the langref: "This instruction has the same comparison semantics as the ‘llvm.minnum.*’ intrinsic. ". Adding the flags nnan and nsz for llvm.minimum so that we can feed it into vector.reduce.fmin unfortunately defeats the purpose of these minimum intrinsic.
What we need is something like:

%v = call <2 x float> @llvm.minimum.v2f32(...)
...
%F = llvm.vector.reduce.fminimum(%v)

where llvm.vector.reduce.fminimum follows the semantics of llvm.minimum.
I'm not sure if we can piggyback on the existing llvm.vector.reduce.fmin intrinsic or would need to create a new one.

[anna]

Looking at X86 lowering for the vector reduce intrinsics, we do not handle signed zeroes. So, we will need to introduce new intrinsics as llvm.vector.reduce.fminimum and llvm.vector.reduce.fmaximum, which is similar to existing vector reduce intrinsics but handles signed zeroes and NaN.

[anna]

The original support for fmax/fmin reduction was added here: https://reviews.llvm.org/D87391 and the idea was to rely on semantics of fminnum/fmaxnum since most targets would crash on fminimum/fmaximum. Now, that we do have the lowerings for these in more targets, we can introduce a separate intrinsic for propagating NaNs and handling signed zeroes (i.e. support fminimum and fmaximum).

[skatkov]

In D151482#4385109, @anna wrote:

Looking at X86 lowering for the vector reduce intrinsics, we do not handle signed zeroes. So, we will need to introduce new intrinsics as llvm.vector.reduce.fminimum and llvm.vector.reduce.fmaximum, which is similar to existing vector reduce intrinsics but handles signed zeroes and NaN.

My understanding is as follows:
You need to inroduce new reduce intrinsic for fminium and fmaximum as you wrote.
In ExpandReductionsPass replace this intrinsic with reduction where you delegates each comparison to existing fminimum/maximum intrinsics. You just need to propagate nnan and nsz flags to that inrinisics.

With that you will have a correct reduction supporting both NaNs and signed zeros.

[dmgreen]

I think that adding a new reduction intrinsic sounds like a good addition. We should even have native instructions we could use for it under AArch64. For Arm we have seemed to have never added support for llvm.minimum/maximum. (There isn't a good instruction for it, and there isn't any generic expansion, but adding a new type of reduction shouldn't make that any worse).

[anna]

In D151482#4386222, @skatkov wrote:

In D151482#4385109, @anna wrote:

Looking at X86 lowering for the vector reduce intrinsics, we do not handle signed zeroes. So, we will need to introduce new intrinsics as llvm.vector.reduce.fminimum and llvm.vector.reduce.fmaximum, which is similar to existing vector reduce intrinsics but handles signed zeroes and NaN.

My understanding is as follows:
You need to inroduce new reduce intrinsic for fminium and fmaximum as you wrote.
In ExpandReductionsPass replace this intrinsic with reduction where you delegates each comparison to existing fminimum/maximum intrinsics. You just need to propagate nnan and nsz flags to that inrinisics.

With that you will have a correct reduction supporting both NaNs and signed zeros.

This is a great idea to use the shuffle reductions as the generic lowering. Also, IIUC, propagating the nnan and nsz is an optimization, not necessary for correctness since fminimum and fmaximum supports NaNs and signed zeroes.

There is one comment in the shuffle reductions stating that we need nnan to perform a shuffle reduction (under the vector_reduce_fmax and vector_reduce_fmin categories), but I don't see anything in LangRef stating that the shuffles themselves do not support NaNs. So, we do not need this extra check of nnan flag when adding the fminimum and fmaximum intrinsic support here. In fact, I think if anyone's interested, one can even remove this restriction from vector_reduce_fmax/vector_reduce_fmin themselves once we add support for fminnum/fmaxnum in createMinMaxOp.

[skatkov]

In D151482#4387272, @dmgreen wrote:

I think that adding a new reduction intrinsic sounds like a good addition. We should even have native instructions we could use for it under AArch64. For Arm we have seemed to have never added support for llvm.minimum/maximum. (There isn't a good instruction for it, and there isn't any generic expansion, but adding a new type of reduction shouldn't make that any worse).

As I know AArch64 support minimum/maximum intrinsics and they are lowered to native fmin/fmax instructions.

Or you are talking about vector max/min on AArch64?

https://godbolt.org/z/Mb9K88scT

Ok, you told about Arm not AArch64 :)
I've nicely talked to myself :)

[anna]

The reduction intrinsic is introduced here: https://reviews.llvm.org/D152370
Once that and dependent patch lands, this patch will be rebased to use the new reduction intrinsic.

[anna]

rebased over landed fminimum/fmaximum.reduce intrinsics. The new recurrence (and supported operations) are introduced as well.

[anna]

llvm/lib/Analysis/IVDescriptors.cpp

814

this is handled now - we have the new llvm.vector.reduce.fminimum/fmaximum intrinsics introduced.

[dmgreen]

Hello. I've been looking at the AArch64 lowering lately, and some of the other little bits and pieces like cost modelling. From what I can tell this mostly LGTM, but can you add a test with interleaving? It appears to go via createMinMaxOp->getMinMaxReductionPredicate.

llvm/lib/Analysis/IVDescriptors.cpp

808

I'm not sure that minnum would require nnan and nsz to convert to a vector.reduce.fmin intrinsic. That might come from it matching select/fcmp, and sounds like a separate issue though.

[anna]

Thanks @dmgreen. Will add the test and update the missing intrinsics in the getMinMaxReductionPredicate API.

llvm/lib/Analysis/IVDescriptors.cpp

808

You might be right here. NaN is propagated with a different logic (compared to minimum) and signed zero is ignored. However, note that the recurKind used by select/fcmp and the fmin intrinsic is the same.

However, I've left the code as it was before for minnum/maxnum, so this can be addressed separately (along with tests added to make sure reduce.fmin lowers correctly without nnan flags?)

[anna]

added interleaving of 2. Better code gen for interleaving by using intrinsic itself instead of fcmp + select pattern.

[anna]

In D151482#4432974, @anna wrote:

Thanks @dmgreen. Will add the test and update the missing intrinsics in the getMinMaxReductionPredicate API.

Confirmed interleaving does infact go through this API and crashes since the intrinsic support was missing. Updated the API, but also made the createMinMaxOp use the intrinsic itself rather than the Fcmp + Select pattern. So the getMinMaxReductionPredicate API is unused at this point for fminimum/fmaximum, but support is left in for completeness.

[dmgreen]

Thanks. I think this LGTM, with some Nits

llvm/include/llvm/Analysis/IVDescriptors.h

51

Nit: Remove extra whitespace

llvm/lib/Transforms/Utils/LoopUtils.cpp

936

If this is unused, could it be removed? I think it goes via getMinMaxReductionIntrinsicOp?

I'm not sure if the expansion to cmp+select would be valid without nnan and nsz fmf flags, creating the intrinsic like you have it sounds much better.

llvm/test/Transforms/LoopVectorize/minmax_reduction.ll

1

Nit: Whitespace

[anna]

llvm/lib/Transforms/Utils/LoopUtils.cpp

936

There are two options:

1. Remove it and add a comment that usage directly goes through getMinMaxReductionIntrinsicOp.
2. Change the predicate to the correct one: FCMP_ULT. Instead of ordered, we need unordered variants as seen by the langref: https://llvm.org/docs/LangRef.html#id308 (it allows NaNs: QNaN or SNaN makes no difference I think).

I prefer the second option for completeness (if for any reason we choose getMinMaxReductionPredicate API for some other purpose in future, not necessarily within createMinMaxOp).

What do you think?

[dmgreen]

llvm/lib/Transforms/Utils/LoopUtils.cpp

936

Yeah I was wondering about the same thing.
minimum also handles -0.0 as less than 0.0, which means it isn't very obvious how to expand it out into fcmp+select. Something like this is needed: https://alive2.llvm.org/ce/z/iB2Dt7. (But that doesn't handle nan as Alive2 seems to think that SNaN turns into +inf, in a way I don't understand. So I wasn't able to test ULT vs OLT).

I think olt and ult will just pick differently between the first and second operand of the select if they are Nan though, not choose _which_ is a Nan. Ordered will chose the second operand if either operand is a nan. Unordered will pick the first. In neither case is that guaranteed to be the Nan.

So it would likely be valid to expand into fcmp+sel provoding it had nsz and nnan, but not without. It may be better to just remove it and rely on createMinMaxOp. We can hopefully remove getMinMaxReductionPredicate in the future if we get to the point where MinNum and MaxNum can be handled, with the necessary FMF.

[anna]

llvm/lib/Transforms/Utils/LoopUtils.cpp

936

agreed, thanks for checking it. I'll update.

[anna]

addressed review comments.

[dmgreen]

Thanks. LGTM

llvm/include/llvm/Analysis/IVDescriptors.h

enum class RecurKind {
   SMin,       ///< Signed integer min implemented in terms of select(cmp()).
   SMax,       ///< Signed integer max implemented in terms of select(cmp()).
   UMin,       ///< Unisgned integer min implemented in terms of select(cmp()).
   UMax,       ///< Unsigned integer max implemented in terms of select(cmp()).
   FAdd,       ///< Sum of floats.
   FMul,       ///< Product of floats.
   FMin,       ///< FP min implemented in terms of select(cmp()).
   FMax,       ///< FP max implemented in terms of select(cmp()).
+  FMinimum,   ///< FP min with llvm.minimum semantics
+  FMaximum,   ///< FP max with llvm.maximum semantics
   FMulAdd,    ///< Fused multiply-add of floats (a * b + c).
   SelectICmp, ///< Integer select(icmp(),x,y) where one of (x,y) is loop
               ///< invariant
   SelectFCmp  ///< Integer select(fcmp(),x,y) where one of (x,y) is loop
               ///< invariant
 };
 
 /// The RecurrenceDescriptor is used to identify recurrences variables in a
public:
   /// Returns true if the recurrence kind is an integer min/max kind.
   static bool isIntMinMaxRecurrenceKind(RecurKind Kind) {
     return Kind == RecurKind::UMin || Kind == RecurKind::UMax ||
            Kind == RecurKind::SMin || Kind == RecurKind::SMax;
   }
 
   /// Returns true if the recurrence kind is a floating-point min/max kind.
   static bool isFPMinMaxRecurrenceKind(RecurKind Kind) {
-    return Kind == RecurKind::FMin || Kind == RecurKind::FMax;
+    return Kind == RecurKind::FMin || Kind == RecurKind::FMax ||
+           Kind == RecurKind::FMinimum || Kind == RecurKind::FMaximum;
   }
 
   /// Returns true if the recurrence kind is any min/max kind.
   static bool isMinMaxRecurrenceKind(RecurKind Kind) {
     return isIntMinMaxRecurrenceKind(Kind) || isFPMinMaxRecurrenceKind(Kind);
   }
 
   /// Returns true if the recurrence kind is of the form

llvm/lib/Analysis/IVDescriptors.cpp

RecurrenceDescriptor::isMinMaxPattern(Instruction *I, RecurKind Kind,
   if (match(I, m_UnordFMin(m_Value(), m_Value())))
     return InstDesc(Kind == RecurKind::FMin, I);
   if (match(I, m_UnordFMax(m_Value(), m_Value())))
     return InstDesc(Kind == RecurKind::FMax, I);
   if (match(I, m_Intrinsic<Intrinsic::minnum>(m_Value(), m_Value())))
     return InstDesc(Kind == RecurKind::FMin, I);
   if (match(I, m_Intrinsic<Intrinsic::maxnum>(m_Value(), m_Value())))
     return InstDesc(Kind == RecurKind::FMax, I);
+  if (match(I, m_Intrinsic<Intrinsic::minimum>(m_Value(), m_Value())))
+    return InstDesc(Kind == RecurKind::FMinimum, I);
+  if (match(I, m_Intrinsic<Intrinsic::maximum>(m_Value(), m_Value())))
+    return InstDesc(Kind == RecurKind::FMaximum, I);
 
   return InstDesc(false, I);
 }
 
 /// Returns true if the select instruction has users in the compare-and-add
 /// reduction pattern below. The select instruction argument is the last one
 /// in the sequence.
 ///
if (Kind == RecurKind::FAdd || Kind == RecurKind::FMul ||
         Kind == RecurKind::Add || Kind == RecurKind::Mul)
       return isConditionalRdxPattern(Kind, I);
     [[fallthrough]];
   case Instruction::FCmp:
   case Instruction::ICmp:
   case Instruction::Call:
     if (isSelectCmpRecurrenceKind(Kind))
       return isSelectCmpPattern(L, OrigPhi, I, Prev);
+    auto HasRequiredFMF = [&]() {
+     if (FuncFMF.noNaNs() && FuncFMF.noSignedZeros())
+       return true;
+     if (isa<FPMathOperator>(I) && I->hasNoNaNs() && I->hasNoSignedZeros())
+       return true;
+     // minimum and maximum intrinsics do not require nsz and nnan flags since
+     // NaN and signed zeroes are propagated in the intrinsic implementation.
+     return match(I, m_Intrinsic<Intrinsic::minimum>(m_Value(), m_Value())) ||
+            match(I, m_Intrinsic<Intrinsic::maximum>(m_Value(), m_Value()));
+    };
     if (isIntMinMaxRecurrenceKind(Kind) ||
-        (((FuncFMF.noNaNs() && FuncFMF.noSignedZeros()) ||
-          (isa<FPMathOperator>(I) && I->hasNoNaNs() &&
-           I->hasNoSignedZeros())) &&
-         isFPMinMaxRecurrenceKind(Kind)))
+        (HasRequiredFMF() && isFPMinMaxRecurrenceKind(Kind)))
       return isMinMaxPattern(I, Kind, Prev);
     else if (isFMulAddIntrinsic(I))
       return InstDesc(Kind == RecurKind::FMulAdd, I,
                       I->hasAllowReassoc() ? nullptr : I);
     return InstDesc(false, I);
   }
 }
 
LLVM_DEBUG(dbgs() << "Found a float conditional select reduction PHI."
                       << " PHI." << *Phi << "\n");
     return true;
   }
   if (AddReductionVar(Phi, RecurKind::FMulAdd, TheLoop, FMF, RedDes, DB, AC, DT,
                       SE)) {
     LLVM_DEBUG(dbgs() << "Found an FMulAdd reduction PHI." << *Phi << "\n");
     return true;
   }
+  if (AddReductionVar(Phi, RecurKind::FMaximum, TheLoop, FMF, RedDes, DB, AC, DT,
+                      SE)) {
+    LLVM_DEBUG(dbgs() << "Found a float MAXIMUM reduction PHI." << *Phi << "\n");
+    return true;
+  }
+  if (AddReductionVar(Phi, RecurKind::FMinimum, TheLoop, FMF, RedDes, DB, AC, DT,
+                      SE)) {
+    LLVM_DEBUG(dbgs() << "Found a float MINIMUM reduction PHI." << *Phi << "\n");
+    return true;
+  }
   // Not a reduction of known type.
   return false;
 }
 
 bool RecurrenceDescriptor::isFixedOrderRecurrence(PHINode *Phi, Loop *TheLoop,
                                                   DominatorTree *DT) {
 
   // Ensure the phi node is in the loop header and has two incoming values.
Value *RecurrenceDescriptor::getRecurrenceIdentity(RecurKind K, Type *Tp,
   case RecurKind::FMin:
     assert((FMF.noNaNs() && FMF.noSignedZeros()) &&
            "nnan, nsz is expected to be set for FP min reduction.");
     return ConstantFP::getInfinity(Tp, false /*Negative*/);
   case RecurKind::FMax:
     assert((FMF.noNaNs() && FMF.noSignedZeros()) &&
            "nnan, nsz is expected to be set for FP max reduction.");
     return ConstantFP::getInfinity(Tp, true /*Negative*/);
+  case RecurKind::FMinimum:
+    return ConstantFP::getInfinity(Tp, false /*Negative*/);
+  case RecurKind::FMaximum:
+    return ConstantFP::getInfinity(Tp, true /*Negative*/);
   case RecurKind::SelectICmp:
   case RecurKind::SelectFCmp:
     return getRecurrenceStartValue();
     break;
   default:
     llvm_unreachable("Unknown recurrence kind");
   }
 }
unsigned RecurrenceDescriptor::getOpcode(RecurKind Kind) {
   case RecurKind::SMax:
   case RecurKind::SMin:
   case RecurKind::UMax:
   case RecurKind::UMin:
   case RecurKind::SelectICmp:
     return Instruction::ICmp;
   case RecurKind::FMax:
   case RecurKind::FMin:
+  case RecurKind::FMaximum:
+  case RecurKind::FMinimum:
   case RecurKind::SelectFCmp:
     return Instruction::FCmp;
   default:
     llvm_unreachable("Unknown recurrence operation");
   }
 }
 
 SmallVector<Instruction *, 4>

llvm/lib/Transforms/Utils/LoopUtils.cpp

Intrinsic::ID llvm::getMinMaxReductionIntrinsicOp(RecurKind RK) {
   case RecurKind::SMin:
     return Intrinsic::smin;
   case RecurKind::SMax:
     return Intrinsic::smax;
   case RecurKind::FMin:
     return Intrinsic::minnum;
   case RecurKind::FMax:
     return Intrinsic::maxnum;
+  case RecurKind::FMinimum:
+    return Intrinsic::minimum;
+  case RecurKind::FMaximum:
+    return Intrinsic::maximum;
   }
 }
 
 CmpInst::Predicate llvm::getMinMaxReductionPredicate(RecurKind RK) {
   switch (RK) {
   default:
     llvm_unreachable("Unknown min/max recurrence kind");
   case RecurKind::UMin:
     return CmpInst::ICMP_ULT;
   case RecurKind::UMax:
     return CmpInst::ICMP_UGT;
   case RecurKind::SMin:
     return CmpInst::ICMP_SLT;
   case RecurKind::SMax:
     return CmpInst::ICMP_SGT;
   case RecurKind::FMin:
     return CmpInst::FCMP_OLT;
   case RecurKind::FMax:
     return CmpInst::FCMP_OGT;
+  // We do not add FMinimum/FMaximum recurrence kind here since there is no
+  // equivalent predicate which compares signed zeroes according to the
+  // semantics of the intrinsics (llvm.minimum/maximum).
   }
 }
 
 Value *llvm::createSelectCmpOp(IRBuilderBase &Builder, Value *StartVal,
                                RecurKind RK, Value *Left, Value *Right) {
   if (auto VTy = dyn_cast<VectorType>(Left->getType()))
     StartVal = Builder.CreateVectorSplat(VTy->getElementCount(), StartVal);
   Value *Cmp =
       Builder.CreateCmp(CmpInst::ICMP_NE, Left, StartVal, "rdx.select.cmp");
   return Builder.CreateSelect(Cmp, Left, Right, "rdx.select");
 }
 
 Value *llvm::createMinMaxOp(IRBuilderBase &Builder, RecurKind RK, Value *Left,
                             Value *Right) {
   Type *Ty = Left->getType();
-  if (Ty->isIntOrIntVectorTy()) {
+  if (Ty->isIntOrIntVectorTy() ||
+      (RK == RecurKind::FMinimum || RK == RecurKind::FMaximum)) {
     // TODO: Add float minnum/maxnum support when FMF nnan is set.
     Intrinsic::ID Id = getMinMaxReductionIntrinsicOp(RK);
     return Builder.CreateIntrinsic(Ty, Id, {Left, Right}, nullptr,
                                    "rdx.minmax");
   }
   CmpInst::Predicate Pred = getMinMaxReductionPredicate(RK);
   Value *Cmp = Builder.CreateCmp(Pred, Left, Right, "rdx.minmax.cmp");
   Value *Select = Builder.CreateSelect(Cmp, Left, Right, "rdx.minmax.select");
Value *llvm::createSimpleTargetReduction(IRBuilderBase &Builder,
   case RecurKind::UMax:
     return Builder.CreateIntMaxReduce(Src, false);
   case RecurKind::UMin:
     return Builder.CreateIntMinReduce(Src, false);
   case RecurKind::FMax:
     return Builder.CreateFPMaxReduce(Src);
   case RecurKind::FMin:
     return Builder.CreateFPMinReduce(Src);
+  case RecurKind::FMinimum:
+    return Builder.CreateFPMinimumReduce(Src);
+  case RecurKind::FMaximum:
+    return Builder.CreateFPMaximumReduce(Src);
   default:
     llvm_unreachable("Unhandled opcode");
   }
 }
 
 Value *llvm::createTargetReduction(IRBuilderBase &B,
                                    const TargetTransformInfo *TTI,
                                    const RecurrenceDescriptor &Desc, Value *Src,

llvm/test/Transforms/LoopVectorize/minmax_reduction.ll

-; RUN: opt -S -passes=loop-vectorize,dce -force-vector-width=2 -force-vector-interleave=1  < %s | FileCheck %s
+; RUN: opt -S -passes=loop-vectorize,dce -force-vector-width=2 -force-vector-interleave=2  < %s | FileCheck %s
 
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 
 @A = common global [1024 x i32] zeroinitializer, align 16
 @fA = common global [1024 x float] zeroinitializer, align 16
 @dA = common global [1024 x double] zeroinitializer, align 16
 
 ; Signed tests.
for.body:                                         ; preds = %entry, %for.body
   %arrayidx3 = getelementptr inbounds i32, ptr %y, i32 %i.025
   %1 = load i32, ptr %arrayidx3, align 4
   %cond9 = tail call i32 @llvm.smin.i32(i32 %s.0., i32 %1)
   %inc = add nuw nsw i32 %i.025, 1
   %exitcond.not = icmp eq i32 %inc, 1024
   br i1 %exitcond.not, label %for.cond.cleanup, label %for.body
 }
 
+; CHECK-LABEL: fmaximum_intrinsic
+; CHECK-LABEL: vector.body:
+; CHECK: call <2 x float> @llvm.maximum.v2f32
+; CHECK: call <2 x float> @llvm.maximum.v2f32
+
+; CHECK-LABEL: middle.block:
+; CHECK: call <2 x float> @llvm.maximum.v2f32
+; CHECK: call float @llvm.vector.reduce.fmaximum.v2f32
+define float @fmaximum_intrinsic(ptr nocapture readonly %x) {
+entry:
+  br label %for.body
+
+for.cond.cleanup:                                 ; preds = %for.body
+  ret float %1
+
+for.body:                                         ; preds = %entry, %for.body
+  %i.012 = phi i32 [ 0, %entry ], [ %inc, %for.body ]
+  %s.011 = phi float [ 0.000000e+00, %entry ], [ %1, %for.body ]
+  %arrayidx = getelementptr inbounds float, ptr %x, i32 %i.012
+  %0 = load float, ptr %arrayidx, align 4
+  %1 = tail call float @llvm.maximum.f32(float %s.011, float %0)
+  %inc = add nuw nsw i32 %i.012, 1
+  %exitcond.not = icmp eq i32 %inc, 1024
+  br i1 %exitcond.not, label %for.cond.cleanup, label %for.body
+}
+
+; CHECK-LABEL: fminimum_intrinsic
+; CHECK-LABEL: vector.body:
+; CHECK: call <2 x float> @llvm.minimum.v2f32
+; CHECK: call <2 x float> @llvm.minimum.v2f32
+
+; CHECK-LABEL: middle.block:
+; CHECK: call <2 x float> @llvm.minimum.v2f32
+; CHECK: call float @llvm.vector.reduce.fminimum.v2f32
+define float @fminimum_intrinsic(ptr nocapture readonly %x) {
+entry:
+  br label %for.body
+
+for.cond.cleanup:                                 ; preds = %for.body
+  ret float %1
+
+for.body:                                         ; preds = %entry, %for.body
+  %i.012 = phi i32 [ 0, %entry ], [ %inc, %for.body ]
+  %s.011 = phi float [ 0.000000e+00, %entry ], [ %1, %for.body ]
+  %arrayidx = getelementptr inbounds float, ptr %x, i32 %i.012
+  %0 = load float, ptr %arrayidx, align 4
+  %1 = tail call float @llvm.minimum.f32(float %s.011, float %0)
+  %inc = add nuw nsw i32 %i.012, 1
+  %exitcond.not = icmp eq i32 %inc, 1024
+  br i1 %exitcond.not, label %for.cond.cleanup, label %for.body
+}
+
+; CHECK-LABEL: fminimum_fminimum
+; CHECK-LABEL: vector.body:
+; CHECK: call <2 x float> @llvm.minimum.v2f32
+; CHECK: call <2 x float> @llvm.minimum.v2f32
+; CHECK: call <2 x float> @llvm.minimum.v2f32
+; CHECK: call <2 x float> @llvm.minimum.v2f32
+
+; CHECK-LABEL: middle.block:
+; CHECK: call <2 x float> @llvm.minimum.v2f32
+; CHECK: call float @llvm.vector.reduce.fminimum.v2f32
+define float @fminimum_fminimum(ptr nocapture readonly %x, ptr nocapture readonly %y) {
+entry:
+  br label %for.body
+
+for.cond.cleanup:                                 ; preds = %for.body
+  ret float %cond9
+
+for.body:                                         ; preds = %entry, %for.body
+  %i.025 = phi i32 [ 0, %entry ], [ %inc, %for.body ]
+  %s.011 = phi float [ 0.000000e+00, %entry ], [ %cond9, %for.body ]
+  %arrayidx = getelementptr inbounds float, ptr %x, i32 %i.025
+  %0 = load float, ptr %arrayidx, align 4
+  %s.0. = tail call float @llvm.minimum.f32(float %s.011, float %0)
+  %arrayidx3 = getelementptr inbounds float, ptr %y, i32 %i.025
+  %1 = load float, ptr %arrayidx3, align 4
+  %cond9 = tail call float @llvm.minimum.f32(float %s.0., float %1)
+  %inc = add nuw nsw i32 %i.025, 1
+  %exitcond.not = icmp eq i32 %inc, 1024
+  br i1 %exitcond.not, label %for.cond.cleanup, label %for.body
+}
+
+; CHECK-LABEL: fminimum_fminimum_one_with_flags
+; CHECK-LABEL: vector.body:
+; CHECK: call nnan nsz <2 x float> @llvm.minimum.v2f32
+; CHECK: call nnan nsz <2 x float> @llvm.minimum.v2f32
+; CHECK: call <2 x float> @llvm.minimum.v2f32
+; CHECK: call <2 x float> @llvm.minimum.v2f32
+
+; CHECK-LABEL: middle.block:
+; CHECK: call <2 x float> @llvm.minimum.v2f32
+; CHECK: call float @llvm.vector.reduce.fminimum.v2f32
+define float @fminimum_fminimum_one_with_flags(ptr nocapture readonly %x, ptr nocapture readonly %y) {
+entry:
+  br label %for.body
+
+for.cond.cleanup:                                 ; preds = %for.body
+  ret float %cond9
+
+for.body:                                         ; preds = %entry, %for.body
+  %i.025 = phi i32 [ 0, %entry ], [ %inc, %for.body ]
+  %s.011 = phi float [ 0.000000e+00, %entry ], [ %cond9, %for.body ]
+  %arrayidx = getelementptr inbounds float, ptr %x, i32 %i.025
+  %0 = load float, ptr %arrayidx, align 4
+  %s.0. = tail call nnan nsz float @llvm.minimum.f32(float %s.011, float %0)
+  %arrayidx3 = getelementptr inbounds float, ptr %y, i32 %i.025
+  %1 = load float, ptr %arrayidx3, align 4
+  %cond9 = tail call float @llvm.minimum.f32(float %s.0., float %1)
+  %inc = add nuw nsw i32 %i.025, 1
+  %exitcond.not = icmp eq i32 %inc, 1024
+  br i1 %exitcond.not, label %for.cond.cleanup, label %for.body
+}
+
 ; Make sure any check-not directives are not triggered by function declarations.
 ; CHECK: declare
 
 declare i32 @llvm.smin.i32(i32, i32)
 declare i32 @llvm.smax.i32(i32, i32)
 declare i32 @llvm.umin.i32(i32, i32)
 declare i32 @llvm.umax.i32(i32, i32)
 declare float @llvm.minnum.f32(float, float)
 declare float @llvm.maxnum.f32(float, float)
+declare float @llvm.minimum.f32(float, float)
+declare float @llvm.maximum.f32(float, float)
 
 attributes #0 = { "no-nans-fp-math"="true" "no-signed-zeros-fp-math"="true" }
 attributes #1 = { "no-nans-fp-math"="true" }