[LoopVectorize] improve IR fast-math-flags propagation in reductions

[spatel]

Summary

This is another step (see D95452) towards correcting fast-math-flags bugs in vector reductions.

There are multiple bugs visible in the test diffs, and this is still not working as it should. We still use function attributes (rather than FMF) to drive part of the logic, but we are not checking for the correct FP function attributes.

Note that FMF may not be propagated optimally on selects (example in https://llvm.org/PR35607 ). That's why I'm proposing to union the FMF of a fcmp+select pair and avoid regressions on existing vectorizer tests.

[spatel]

llvm/test/Transforms/LoopVectorize/X86/reduction-fastmath.ll

302–303

This is a miscompile - we created fast ops from the more strict original code.

304

Yet we created a non-FMF reduction intrinsic...

345

The ninf is not actually necessary, but nsz should be required.

llvm/test/Transforms/LoopVectorize/float-minmax-instruction-flag.ll

107–108

This test demonstrates that we are not even relying on the instruction-level FMF, and it's another miscompile. The function only has the equivalent of nnan. I think we should require nsz too.

[grandinj]

random idea from a bystander:

Is there some way to do symbolic execution of IR that results in some kind of "output state" of FMF at the end of a block of IR?

If such a thing existed, then running it on the IR before and after a pass, and comparing the state of FMF at the end of that chunk of IR, would validate if the pass was correctly propagating FMF.

I only ask because I've seen other such "lets validate by doing a restricted symbolic execution of some state" tools floating around.

Feel free to ignore me if I'm talking nonsense, it's late here :-)

[spatel]

In D95690#2532922, @grandinj wrote:

Is there some way to do symbolic execution of IR that results in some kind of "output state" of FMF at the end of a block of IR?
If such a thing existed, then running it on the IR before and after a pass, and comparing the state of FMF at the end of that chunk of IR, would validate if the pass was correctly propagating FMF.

The closest that I know of is Alive2:
https://alive2.llvm.org/ce/z/2oNg9N
In that example, I added a nnan clause to the fadd, and Alive shows us how it is invalid (we produce poison in the transformed case where the original did not).
Alive2 is currently limited in its understanding of FMF, and I'm not sure if it can ever provide full FP analysis.

If we just want a weak/cheap solution, we could add a step to IR verification that checks if any FMF exist in the output that were not present in the input. For example, in the loop vectorizer tests shown here, we are obviously adding flags that are not anywhere in the original code. The verifier check would have to account for FP ops that have special semantics (eg, the intrinsic maxnum doesn't exactly specify how the sign of zero affects the result), and it would also need to account for FP function attributes since we still partly rely on those.

It's beyond the scope of anything I'm trying to do in this patch, but it's an interesting idea - file a bugzilla and/or llvm-dev note to get more attention?

[dmgreen]

It looks like we don't expand non-fast vecreduce_fmax in ExpandReductions:

// FIXME: We only expand 'fast' reductions here because the underlying
//        code in createMinMaxOp() assumes that comparisons use 'fast'
//        semantics.

And otherwise expand VECREDUCE_FMAX to (I think) FMAXNUM. Not using ExpandReductions sounds fine to me, but do we also need to fix those assumptions during lowering, if we are fixing the assumptions in the vectorizer? I guess we are still requiring NoNan so FMAXNUM should be fine? I didn't see anything normally requiring nsz for that.

[spatel]

In D95690#2533764, @dmgreen wrote:

It looks like we don't expand non-fast vecreduce_fmax in ExpandReductions:

// FIXME: We only expand 'fast' reductions here because the underlying
//        code in createMinMaxOp() assumes that comparisons use 'fast'
//        semantics.

And otherwise expand VECREDUCE_FMAX to (I think) FMAXNUM. Not using ExpandReductions sounds fine to me, but do we also need to fix those assumptions during lowering, if we are fixing the assumptions in the vectorizer? I guess we are still requiring NoNan so FMAXNUM should be fine? I didn't see anything normally requiring nsz for that.

There's still a bug in ExpandReductions, and we do need to fix that - it's preventing expected vectorization from SLP as noted here:
https://llvm.org/PR23116

But yes, since we are still requiring NoNan here, I think we are safe (this patch can't make things worse unless I've missed some loophole in lowering).

There was also this example:
https://llvm.org/PR43574
...but we either solved that one or made it invisible with the changes so far. I need to investigate the IR after each pass.

[dmgreen]

There's still a bug in ExpandReductions, and we do need to fix that - it's preventing expected vectorization from SLP as noted here:
https://llvm.org/PR23116

Why does ISel not lower it more optimally? ExpandReductions never seemed like it should be needed to me.

But yes, since we are still requiring NoNan here, I think we are safe (this patch can't make things worse unless I've missed some loophole in lowering).

Yeah, this seems more correct. LGTM :)

There was also this example:
https://llvm.org/PR43574
...but we either solved that one or made it invisible with the changes so far. I need to investigate the IR after each pass.

I thought that we haven't been able to vectorizer min/max reductions in the loop vectorizer for a while as the minnum/maxnum intrinsics do not get matched - it's still expecting a icmp/select in the reduction recognition code.

[spatel]

In D95690#2533980, @dmgreen wrote:

There's still a bug in ExpandReductions, and we do need to fix that - it's preventing expected vectorization from SLP as noted here:
https://llvm.org/PR23116

Why does ISel not lower it more optimally? ExpandReductions never seemed like it should be needed to me.

I think there was some problem dealing with the intrinsics in SDAG, but I don't remember the details. cc'ing @aemerson @RKSimon @craig.topper for a better answer.

But yes, since we are still requiring NoNan here, I think we are safe (this patch can't make things worse unless I've missed some loophole in lowering).

I thought that we haven't been able to vectorizer min/max reductions in the loop vectorizer for a while as the minnum/maxnum intrinsics do not get matched - it's still expecting a icmp/select in the reduction recognition code.

I think that's correct - I managed to get SLP to recognize the intrinsics, but LV still doesn't afaik. That's another step on this path. So the plan will be to get LV to deal with the FP minnum/maxnum, then handle the recent integer min/max intrinsics, then canonicalize to those intrinsics in instcombine.

[dmgreen]

I think there was some problem dealing with the intrinsics in SDAG, but I don't remember the details. cc'ing @aemerson @RKSimon @craig.topper for a better answer.

I think that might have been true originally, but I believe @nikic improved things significantly with the lowering in ISel. I'm not sure if there were other issues, but it seems like it should be viable - and possibly more efficient to do the lowering in ISel. It's only expanding an instruction after all. ISel is pretty good at that kind of thing!

[nikic]

In D95690#2534048, @dmgreen wrote:

I think there was some problem dealing with the intrinsics in SDAG, but I don't remember the details. cc'ing @aemerson @RKSimon @craig.topper for a better answer.

I think that might have been true originally, but I believe @nikic improved things significantly with the lowering in ISel. I'm not sure if there were other issues, but it seems like it should be viable - and possibly more efficient to do the lowering in ISel. It's only expanding an instruction after all. ISel is pretty good at that kind of thing!

Right. Historically we did not have full legalization support for reductions, but those have been implemented in the meantime, and the last holes were plugged in LLVM 12 (sequential reductions). I think that at this point, there isn't a good reason to use the IR expansion into shuffles anymore. Migrating X86 in particular would probably take some effort though, as I believe it has quite a bit of custom shuffle matching code.

[MaskRay]

llvm/lib/Analysis/IVDescriptors.cpp

312

I notice that the assert may fail. We will work on a reproduce.

[spatel]

llvm/lib/Analysis/IVDescriptors.cpp

312

Ok, thanks. I have missed some way that this loop works then, but it should be fine to use dyn_cast here to guard against that possibility.

[asbirlea]

Here's what the test reduced to. To reproduce run opt -passes='loop-vectorize' test.ll with the assert in place.

target datalayout = "e-m:e-p:32:32-i64:64-n32:64-S128"
target triple = "wasm32-unknown--wasm"

define void @f9() local_unnamed_addr {
entry:
  br label %"for f9.s0.v0"

"for f9.s0.v0":                                   ; preds = %"for f9.s0.v0", %entry
  %f9.s0.v0 = phi i32 [ 0, %entry ], [ %tmp3, %"for f9.s0.v0" ]
  %t14 = icmp eq i32 %f9.s0.v0, 5
  %t15 = select reassoc nnan ninf nsz contract afn i1 %t14, float 0x36A0000000000000, float 0.000000e+00
  %tmp3 = add nuw nsw i32 %f9.s0.v0, 1
  br i1 false, label %"end for f9.s0.v0", label %"for f9.s0.v0"

"end for f9.s0.v0":                               ; preds = %"for f9.s0.v0"
  ret void
}

[spatel]

In D95690#2538567, @asbirlea wrote:

Here's what the test reduced to. To reproduce run opt -passes='loop-vectorize' test.ll with the assert in place.

Thanks! I see what happened - we speculatively assume that we matched a reduction pattern based on the cmp only. So in this example, we are trying to match an integer smax recurrence before checking that the subsequent select is actually part of a valid integer min/max pattern. The logic is confusing, but I'm not sure how to improve it yet. Hopefully, the complexity all goes away when we canonicalize to min/max intrinsics. I added a slightly modified version of the test, so we don't fall into the same bug again:
916c4121c1

[aemerson]

In D95690#2534154, @nikic wrote:

In D95690#2534048, @dmgreen wrote:

I think there was some problem dealing with the intrinsics in SDAG, but I don't remember the details. cc'ing @aemerson @RKSimon @craig.topper for a better answer.

I think that might have been true originally, but I believe @nikic improved things significantly with the lowering in ISel. I'm not sure if there were other issues, but it seems like it should be viable - and possibly more efficient to do the lowering in ISel. It's only expanding an instruction after all. ISel is pretty good at that kind of thing!

Right. Historically we did not have full legalization support for reductions, but those have been implemented in the meantime, and the last holes were plugged in LLVM 12 (sequential reductions). I think that at this point, there isn't a good reason to use the IR expansion into shuffles anymore. Migrating X86 in particular would probably take some effort though, as I believe it has quite a bit of custom shuffle matching code.

Yes, ExpandReductions wasn't intended to be a permanent solution, but as a transitionary step for LLVM to generate the intrinsics in the vectorizer, while the legalization/target specific shuffle matching support was fleshed out.

llvm/include/llvm/IR/Operator.h

public:
   void setApproxFunc(bool B = true) {
     Flags = (Flags & ~ApproxFunc) | B * ApproxFunc;
   }
   void setFast(bool B = true) { B ? set() : clear(); }
 
   void operator&=(const FastMathFlags &OtherFlags) {
     Flags &= OtherFlags.Flags;
   }
+  void operator|=(const FastMathFlags &OtherFlags) {
+    Flags |= OtherFlags.Flags;
+  }
 };
 
 /// Utility class for floating point operations which can have
 /// information about relaxed accuracy requirements attached to them.
 class FPMathOperator : public Operator {
 private:
   friend class Instruction;
 

llvm/include/llvm/Transforms/Utils/LoopUtils.h

 /// If \p ORE is set use it to emit optimization remarks.
 bool canSinkOrHoistInst(Instruction &I, AAResults *AA, DominatorTree *DT,
                         Loop *CurLoop, AliasSetTracker *CurAST,
                         MemorySSAUpdater *MSSAU, bool TargetExecutesOncePerLoop,
                         SinkAndHoistLICMFlags *LICMFlags = nullptr,
                         OptimizationRemarkEmitter *ORE = nullptr);
 
 /// Returns a Min/Max operation corresponding to MinMaxRecurrenceKind.
+/// The Builder's fast-math-flags must be set to propagate the expected values.
 Value *createMinMaxOp(IRBuilderBase &Builder, RecurKind RK, Value *Left,
                       Value *Right);
 
 /// Generates an ordered vector reduction using extracts to reduce the value.
 Value *getOrderedReduction(IRBuilderBase &Builder, Value *Acc, Value *Src,
                            unsigned Op, RecurKind MinMaxKind = RecurKind::None,
                            ArrayRef<Value *> RedOps = None);
 

llvm/lib/Analysis/IVDescriptors.cpp

while (!Worklist.empty()) {
     // Any reduction instruction must be of one of the allowed kinds. We ignore
     // the starting value (the Phi or an AND instruction if the Phi has been
     // type-promoted).
     if (Cur != Start) {
       ReduxDesc = isRecurrenceInstr(Cur, Kind, ReduxDesc, HasFunNoNaNAttr);
       if (!ReduxDesc.isRecurrence())
         return false;
       // FIXME: FMF is allowed on phi, but propagation is not handled correctly.
-      if (isa<FPMathOperator>(ReduxDesc.getPatternInst()) && !IsAPhi)
-        FMF &= ReduxDesc.getPatternInst()->getFastMathFlags();
+      if (isa<FPMathOperator>(ReduxDesc.getPatternInst()) && !IsAPhi) {
+        FastMathFlags CurFMF = ReduxDesc.getPatternInst()->getFastMathFlags();
+        if (auto *Sel = dyn_cast<SelectInst>(ReduxDesc.getPatternInst())) {
+          // Accept FMF on either fcmp or select of a min/max idiom.
+          // TODO: This is a hack to work-around the fact that FMF may not be
+          //       assigned/propagated correctly. If that problem is fixed or we
+          //       standardize on fmin/fmax via intrinsics, this can be removed.
+          assert(isa<FCmpInst>(Sel->getCondition()) && "Expected fcmp min/max");
+          CurFMF |= cast<FCmpInst>(Sel->getCondition())->getFastMathFlags();
+        }
+        FMF &= CurFMF;
+      }
       // Update this reduction kind if we matched a new instruction.
       // TODO: Can we eliminate the need for a 2nd InstDesc by keeping 'Kind'
       //       state accurate while processing the worklist?
       if (ReduxDesc.getRecKind() != RecurKind::None)
         Kind = ReduxDesc.getRecKind();
     }
 
     bool IsASelect = isa<SelectInst>(Cur);

llvm/lib/Transforms/Utils/LoopUtils.cpp

Value *llvm::createMinMaxOp(IRBuilderBase &Builder, RecurKind RK, Value *Left,
   case RecurKind::FMin:
     Pred = CmpInst::FCMP_OLT;
     break;
   case RecurKind::FMax:
     Pred = CmpInst::FCMP_OGT;
     break;
   }
 
-  // We only match FP sequences that are 'fast', so we can unconditionally
-  // set it on any generated instructions.
-  IRBuilderBase::FastMathFlagGuard FMFG(Builder);
-  FastMathFlags FMF;
-  FMF.setFast();
-  Builder.setFastMathFlags(FMF);
   Value *Cmp = Builder.CreateCmp(Pred, Left, Right, "rdx.minmax.cmp");
   Value *Select = Builder.CreateSelect(Cmp, Left, Right, "rdx.minmax.select");
   return Select;
 }
 
 // Helper to generate an ordered reduction.
 Value *llvm::getOrderedReduction(IRBuilderBase &Builder, Value *Acc, Value *Src,
                                  unsigned Op, RecurKind RdxKind,

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

 
 /// A helper function that adds a 'fast' flag to floating-point operations.
 static Value *addFastMathFlag(Value *V) {
   if (isa<FPMathOperator>(V))
     cast<Instruction>(V)->setFastMathFlags(FastMathFlags::getFast());
   return V;
 }
 
-static Value *addFastMathFlag(Value *V, FastMathFlags FMF) {
-  if (isa<FPMathOperator>(V))
-    cast<Instruction>(V)->setFastMathFlags(FMF);
-  return V;
-}
-
 /// A helper function that returns an integer or floating-point constant with
 /// value C.
 static Constant *getSignedIntOrFpConstant(Type *Ty, int64_t C) {
   return Ty->isIntegerTy() ? ConstantInt::getSigned(Ty, C)
                            : ConstantFP::get(Ty, C);
 }
 
 /// Returns "best known" trip count for the specified loop \p L as defined by
void InnerLoopVectorizer::fixReduction(PHINode *Phi) {
   // The middle block terminator has already been assigned a DebugLoc here (the
   // OrigLoop's single latch terminator). We want the whole middle block to
   // appear to execute on this line because: (a) it is all compiler generated,
   // (b) these instructions are always executed after evaluating the latch
   // conditional branch, and (c) other passes may add new predecessors which
   // terminate on this line. This is the easiest way to ensure we don't
   // accidentally cause an extra step back into the loop while debugging.
   setDebugLocFromInst(Builder, LoopMiddleBlock->getTerminator());
+  {
+    // Floating-point operations should have some FMF to enable the reduction.
+    IRBuilderBase::FastMathFlagGuard FMFG(Builder);
+    Builder.setFastMathFlags(RdxDesc.getFastMathFlags());
     for (unsigned Part = 1; Part < UF; ++Part) {
       Value *RdxPart = VectorLoopValueMap.getVectorValue(LoopExitInst, Part);
-    if (Op != Instruction::ICmp && Op != Instruction::FCmp)
-      // Floating point operations had to be 'fast' to enable the reduction.
-      ReducedPartRdx = addFastMathFlag(
-          Builder.CreateBinOp((Instruction::BinaryOps)Op, RdxPart,
-                              ReducedPartRdx, "bin.rdx"),
-          RdxDesc.getFastMathFlags());
-    else
+      if (Op != Instruction::ICmp && Op != Instruction::FCmp) {
+        ReducedPartRdx = Builder.CreateBinOp(
+            (Instruction::BinaryOps)Op, RdxPart, ReducedPartRdx, "bin.rdx");
+      } else {
         ReducedPartRdx = createMinMaxOp(Builder, RK, ReducedPartRdx, RdxPart);
       }
+    }
+  }
 
   // Create the reduction after the loop. Note that inloop reductions create the
   // target reduction in the loop using a Reduction recipe.
   if (VF.isVector() && !IsInLoopReductionPhi) {
     ReducedPartRdx =
         createTargetReduction(Builder, TTI, RdxDesc, ReducedPartRdx);
     // If the reduction can be performed in a smaller type, we need to extend
     // the reduction to the wider type before we branch to the original loop.

llvm/test/Transforms/LoopVectorize/X86/reduction-fastmath.ll

loop:
   %be.cond = icmp ne i32 %idx.inc, 4096
   br i1 %be.cond, label %loop, label %loop.exit
 
 loop.exit:
   %sum.lcssa = phi float [ %sum.inc, %loop ], [ 0.000000e+00, %entry ]
   ret float %sum.lcssa
 }
 
-; FIXME: Some fcmp are 'nnan ninf', some are 'fast', but the reduction is sequential?
+; New instructions should have the same FMF as the original code.
+; Note that the select inherits FMF from its fcmp condition.
 
 define float @PR35538(float* nocapture readonly %a, i32 %N) #0 {
 ; CHECK-LABEL: @PR35538(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[CMP12:%.*]] = icmp sgt i32 [[N:%.*]], 0
 ; CHECK-NEXT:    br i1 [[CMP12]], label [[FOR_BODY_LR_PH:%.*]], label [[FOR_COND_CLEANUP:%.*]]
 ; CHECK:       for.body.lr.ph:
 ; CHECK-NEXT:    [[WIDE_TRIP_COUNT:%.*]] = zext i32 [[N]] to i64

 ; CHECK-NEXT:    [[TMP8:%.*]] = fcmp nnan ninf oge <4 x float> [[WIDE_LOAD]], [[VEC_PHI]]
 ; CHECK-NEXT:    [[TMP9:%.*]] = fcmp nnan ninf oge <4 x float> [[WIDE_LOAD2]], [[VEC_PHI1]]
 ; CHECK-NEXT:    [[TMP10]] = select <4 x i1> [[TMP8]], <4 x float> [[WIDE_LOAD]], <4 x float> [[VEC_PHI]]
 ; CHECK-NEXT:    [[TMP11]] = select <4 x i1> [[TMP9]], <4 x float> [[WIDE_LOAD2]], <4 x float> [[VEC_PHI1]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add i64 [[INDEX]], 8
 ; CHECK-NEXT:    [[TMP12:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; CHECK-NEXT:    br i1 [[TMP12]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP8:!llvm.loop !.*]]
 ; CHECK:       middle.block:
-; CHECK-NEXT:    [[RDX_MINMAX_CMP:%.*]] = fcmp fast ogt <4 x float> [[TMP10]], [[TMP11]]
-; CHECK-NEXT:    [[RDX_MINMAX_SELECT:%.*]] = select fast <4 x i1> [[RDX_MINMAX_CMP]], <4 x float> [[TMP10]], <4 x float> [[TMP11]]
-; CHECK-NEXT:    [[TMP13:%.*]] = call float @llvm.vector.reduce.fmax.v4f32(<4 x float> [[RDX_MINMAX_SELECT]])
+; CHECK-NEXT:    [[RDX_MINMAX_CMP:%.*]] = fcmp nnan ninf ogt <4 x float> [[TMP10]], [[TMP11]]
+; CHECK-NEXT:    [[RDX_MINMAX_SELECT:%.*]] = select nnan ninf <4 x i1> [[RDX_MINMAX_CMP]], <4 x float> [[TMP10]], <4 x float> [[TMP11]]
+; CHECK-NEXT:    [[TMP13:%.*]] = call nnan ninf float @llvm.vector.reduce.fmax.v4f32(<4 x float> [[RDX_MINMAX_SELECT]])
 ; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 [[WIDE_TRIP_COUNT]], [[N_VEC]]
 ; CHECK-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP_LOOPEXIT:%.*]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[FOR_BODY_LR_PH]] ]
 ; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] = phi float [ -1.000000e+00, [[FOR_BODY_LR_PH]] ], [ [[TMP13]], [[MIDDLE_BLOCK]] ]
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.cond.cleanup.loopexit:
 ; CHECK-NEXT:    [[MAX_0__LCSSA:%.*]] = phi float [ [[MAX_0_:%.*]], [[FOR_BODY]] ], [ [[TMP13]], [[MIDDLE_BLOCK]] ]
for.cond.cleanup:
   %max.0.lcssa = phi float [ -1.000000e+00, %entry ], [ %max.0., %for.body ]
   ret float %max.0.lcssa
 
 for.body:
   %indvars.iv = phi i64 [ 0, %for.body.lr.ph ], [ %indvars.iv.next, %for.body ]
   %max.013 = phi float [ -1.000000e+00, %for.body.lr.ph ], [ %max.0., %for.body ]
   %arrayidx = getelementptr inbounds float, float* %a, i64 %indvars.iv
   %0 = load float, float* %arrayidx, align 4
   %cmp1.inv = fcmp nnan ninf oge float %0, %max.013
   %max.0. = select i1 %cmp1.inv, float %0, float %max.013
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
   %exitcond = icmp eq i64 %indvars.iv.next, %wide.trip.count
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
+; Same as above, but this time the select already has matching FMF with its condition.
+
 define float @PR35538_more_FMF(float* nocapture readonly %a, i32 %N) #0 {
 ; CHECK-LABEL: @PR35538_more_FMF(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[CMP12:%.*]] = icmp sgt i32 [[N:%.*]], 0
 ; CHECK-NEXT:    br i1 [[CMP12]], label [[FOR_BODY_LR_PH:%.*]], label [[FOR_COND_CLEANUP:%.*]]
 ; CHECK:       for.body.lr.ph:
 ; CHECK-NEXT:    [[WIDE_TRIP_COUNT:%.*]] = zext i32 [[N]] to i64
 ; CHECK-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[WIDE_TRIP_COUNT]], 8

 ; CHECK-NEXT:    [[TMP8:%.*]] = fcmp nnan ninf oge <4 x float> [[WIDE_LOAD]], [[VEC_PHI]]
 ; CHECK-NEXT:    [[TMP9:%.*]] = fcmp nnan ninf oge <4 x float> [[WIDE_LOAD2]], [[VEC_PHI1]]
 ; CHECK-NEXT:    [[TMP10]] = select <4 x i1> [[TMP8]], <4 x float> [[WIDE_LOAD]], <4 x float> [[VEC_PHI]]
 ; CHECK-NEXT:    [[TMP11]] = select <4 x i1> [[TMP9]], <4 x float> [[WIDE_LOAD2]], <4 x float> [[VEC_PHI1]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add i64 [[INDEX]], 8
 ; CHECK-NEXT:    [[TMP12:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; CHECK-NEXT:    br i1 [[TMP12]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP10:!llvm.loop !.*]]
 ; CHECK:       middle.block:
-; CHECK-NEXT:    [[RDX_MINMAX_CMP:%.*]] = fcmp fast ogt <4 x float> [[TMP10]], [[TMP11]]
-; CHECK-NEXT:    [[RDX_MINMAX_SELECT:%.*]] = select fast <4 x i1> [[RDX_MINMAX_CMP]], <4 x float> [[TMP10]], <4 x float> [[TMP11]]
+; CHECK-NEXT:    [[RDX_MINMAX_CMP:%.*]] = fcmp nnan ninf ogt <4 x float> [[TMP10]], [[TMP11]]
+; CHECK-NEXT:    [[RDX_MINMAX_SELECT:%.*]] = select nnan ninf <4 x i1> [[RDX_MINMAX_CMP]], <4 x float> [[TMP10]], <4 x float> [[TMP11]]
 ; CHECK-NEXT:    [[TMP13:%.*]] = call nnan ninf float @llvm.vector.reduce.fmax.v4f32(<4 x float> [[RDX_MINMAX_SELECT]])
 ; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 [[WIDE_TRIP_COUNT]], [[N_VEC]]
 ; CHECK-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP_LOOPEXIT:%.*]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[FOR_BODY_LR_PH]] ]
 ; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] = phi float [ -1.000000e+00, [[FOR_BODY_LR_PH]] ], [ [[TMP13]], [[MIDDLE_BLOCK]] ]
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.cond.cleanup.loopexit:

llvm/test/Transforms/LoopVectorize/float-minmax-instruction-flag.ll

 ; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <4 x float>, <4 x float>* [[TMP3]], align 4
 ; CHECK-NEXT:    [[TMP4:%.*]] = fcmp olt <4 x float> [[VEC_PHI]], [[WIDE_LOAD]]
 ; CHECK-NEXT:    [[TMP5]] = select <4 x i1> [[TMP4]], <4 x float> [[VEC_PHI]], <4 x float> [[WIDE_LOAD]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add i64 [[INDEX]], 4
 ; CHECK-NEXT:    [[TMP6:%.*]] = icmp eq i64 [[INDEX_NEXT]], 65536
 ; CHECK-NEXT:    br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP0:!llvm.loop !.*]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    [[RDX_SHUF:%.*]] = shufflevector <4 x float> [[TMP5]], <4 x float> poison, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef>
-; CHECK-NEXT:    [[RDX_MINMAX_CMP:%.*]] = fcmp fast olt <4 x float> [[TMP5]], [[RDX_SHUF]]
-; CHECK-NEXT:    [[RDX_MINMAX_SELECT:%.*]] = select fast <4 x i1> [[RDX_MINMAX_CMP]], <4 x float> [[TMP5]], <4 x float> [[RDX_SHUF]]
+; CHECK-NEXT:    [[RDX_MINMAX_CMP:%.*]] = fcmp olt <4 x float> [[TMP5]], [[RDX_SHUF]]
+; CHECK-NEXT:    [[RDX_MINMAX_SELECT:%.*]] = select <4 x i1> [[RDX_MINMAX_CMP]], <4 x float> [[TMP5]], <4 x float> [[RDX_SHUF]]
 ; CHECK-NEXT:    [[RDX_SHUF1:%.*]] = shufflevector <4 x float> [[RDX_MINMAX_SELECT]], <4 x float> poison, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
-; CHECK-NEXT:    [[RDX_MINMAX_CMP2:%.*]] = fcmp fast olt <4 x float> [[RDX_MINMAX_SELECT]], [[RDX_SHUF1]]
-; CHECK-NEXT:    [[RDX_MINMAX_SELECT3:%.*]] = select fast <4 x i1> [[RDX_MINMAX_CMP2]], <4 x float> [[RDX_MINMAX_SELECT]], <4 x float> [[RDX_SHUF1]]
+; CHECK-NEXT:    [[RDX_MINMAX_CMP2:%.*]] = fcmp olt <4 x float> [[RDX_MINMAX_SELECT]], [[RDX_SHUF1]]
+; CHECK-NEXT:    [[RDX_MINMAX_SELECT3:%.*]] = select <4 x i1> [[RDX_MINMAX_CMP2]], <4 x float> [[RDX_MINMAX_SELECT]], <4 x float> [[RDX_SHUF1]]
 ; CHECK-NEXT:    [[TMP7:%.*]] = extractelement <4 x float> [[RDX_MINMAX_SELECT3]], i32 0
 ; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 65536, 65536
 ; CHECK-NEXT:    br i1 [[CMP_N]], label [[OUT:%.*]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 65537, [[MIDDLE_BLOCK]] ], [ 1, [[TOP:%.*]] ]
 ; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] = phi float [ [[T]], [[TOP]] ], [ [[TMP7]], [[MIDDLE_BLOCK]] ]
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
top:
   %t = load float, float* %arg
   br label %loop
 
 loop:                                             ; preds = %loop, %top
   %t1 = phi i64 [ %t7, %loop ], [ 1, %top ]
   %t2 = phi float [ %t6, %loop ], [ %t, %top ]
   %t3 = getelementptr float, float* %arg, i64 %t1
   %t4 = load float, float* %t3, align 4
   %t5 = fcmp olt float %t2, %t4
   %t6 = select i1 %t5, float %t2, float %t4
   %t7 = add i64 %t1, 1
   %t8 = icmp eq i64 %t7, 65537
   br i1 %t8, label %out, label %loop
 
 out:                                              ; preds = %loop
   ret float %t6
 }