[IR] allow fast-math-flags on select of FP values

[spatel]

Summary

This is a minimal start to correcting a problem most directly discussed in PR38086:
https://bugs.llvm.org/show_bug.cgi?id=38086

We have been hacking around a limitation for FP select patterns by using the fast-math-flags on the condition of the select rather than the select itself.
This patch just allows FMF to appear with the 'select' opcode. No changes are needed to "FPMathOperator" because it already includes select-of-FP because that definition is based on the return value type.

Once we have this ability, we can start correcting and adding IR transforms to use the FMF on a 'select' instruction. The instcombine and vectorizer test diffs only show that the IRBuilder change is behaving as expected by applying an FMF guard value to 'select'.

For reference:
rL241901 - allowed FMF with fcmp
rL255555 - allowed FMF with FP calls

[spatel]

Ping.

[hfinkel]

Just to make sure that I understand the general direction here: We want to make all fast-math flags apply only to inputs, not outputs, and so we're going to end up adding fast-math flags on all instructions that might take floating-point inputs?

[spatel]

In D61917#1510408, @hfinkel wrote:

Just to make sure that I understand the general direction here: We want to make all fast-math flags apply only to inputs, not outputs, and so we're going to end up adding fast-math flags on all instructions that might take floating-point inputs?

I think we want to have FMF apply to the value itself. That should allow removing FMF from fcmp by adjusting our analysis/pattern matching to detect the FMF on fcmp's operands. That would simplify the definition of FPMathOperator too - if it's an FP value (produces an FP value), it's an FPMO. To make that complete, I think we'd eventually allow adding FMF as an attribute to FP function arguments.

Some motivating examples:

define float @PR39535(float %x) {
  %cmp = fcmp nsz oeq float %x, 0.0           ; 'nsz' behavior is implicit in any fcmp, so this is strange
  %cond = select i1 %cmp, float %x, float 0.0 ; if this has 'nsz', we can simplify to '0.0' because both arms of the select are the same
  ret float %cond
}

define i1 @orderedLessZeroUIToFP_nnan(i32 %x) {
  %a = uitofp i32 %x to float                 ; if this has 'nnan', the result is a number and >=0.0 (the fcmp must be true)
  %uge = fcmp nnan oge float %a, 0.000000e+00
  ret i1 %uge
}

[hfinkel]

To make that complete, I think we'd eventually allow adding FMF as an attribute to FP function arguments.

Okay. So the flag will always be a property of the value, and so we'll add them to loads and function arguments, etc. This makes sense to me, and seems like it creates an unambiguous picture. We need to be a bit careful how we define this -- I think that we want a disallowed value in one of these arguments, etc. to be poison and not UB. We'll need to be more aggressive about propagating the flags when we inline the functions when they have the flags on their return values (because we don't otherwise have a way to get the information when the values are later used as inputs).

LGTM.

[foad]

llvm/trunk/lib/AsmParser/LLParser.cpp

5704

Why is the FMF handling done inline here, instead of in the body of ParseSelect ?

[spatel]

llvm/trunk/lib/AsmParser/LLParser.cpp

5704

Copied from the similar block for fcmp just above here. No objection if you want to change it.

llvm/trunk/docs/LangRef.rst

 '``select``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 ::
 
-      <result> = select selty <cond>, <ty> <val1>, <ty> <val2>             ; yields ty
+      <result> = select [fast-math flags] selty <cond>, <ty> <val1>, <ty> <val2>             ; yields ty
 
       selty is either i1 or {<N x i1>}
 
 Overview:
 """""""""
 
 The '``select``' instruction is used to choose one value based on a
 condition, without IR-level branching.
 
 Arguments:
 """"""""""
 
 The '``select``' instruction requires an 'i1' value or a vector of 'i1'
 values indicating the condition, and two values of the same :ref:`first
 class <t_firstclass>` type.
 
+#. The optional ``fast-math flags`` marker indicates that the select has one or more
+   :ref:`fast-math flags <fastmath>`. These are optimization hints to enable
+   otherwise unsafe floating-point optimizations. Fast-math flags are only valid
+   for selects that return a floating-point scalar or vector type.
+
 Semantics:
 """"""""""
 
 If the condition is an i1 and it evaluates to 1, the instruction returns
 the first value argument; otherwise, it returns the second value
 argument.
 
 If the condition is a vector of i1, then the value arguments must be

llvm/trunk/include/llvm/IR/IRBuilder.h

if (auto *CC = dyn_cast<Constant>(C))
           return Insert(Folder.CreateSelect(CC, TC, FC), Name);
 
     SelectInst *Sel = SelectInst::Create(C, True, False);
     if (MDFrom) {
       MDNode *Prof = MDFrom->getMetadata(LLVMContext::MD_prof);
       MDNode *Unpred = MDFrom->getMetadata(LLVMContext::MD_unpredictable);
       Sel = addBranchMetadata(Sel, Prof, Unpred);
     }
+    if (isa<FPMathOperator>(Sel))
+      Sel = cast<SelectInst>(setFPAttrs(Sel, nullptr /* MDNode* */, FMF));
     return Insert(Sel, Name);
   }
 
   VAArgInst *CreateVAArg(Value *List, Type *Ty, const Twine &Name = "") {
     return Insert(new VAArgInst(List, Ty), Name);
   }
 
   Value *CreateExtractElement(Value *Vec, Value *Idx,

llvm/trunk/lib/AsmParser/LLParser.cpp

int LLParser::ParseInstruction(Instruction *&Inst, BasicBlock *BB,
   case lltok::kw_addrspacecast:
   case lltok::kw_uitofp:
   case lltok::kw_sitofp:
   case lltok::kw_fptoui:
   case lltok::kw_fptosi:
   case lltok::kw_inttoptr:
   case lltok::kw_ptrtoint:       return ParseCast(Inst, PFS, KeywordVal);
   // Other.
-  case lltok::kw_select:         return ParseSelect(Inst, PFS);
+  case lltok::kw_select: {
+    FastMathFlags FMF = EatFastMathFlagsIfPresent();
+    int Res = ParseSelect(Inst, PFS);
+    if (Res != 0)
+      return Res;
+    if (FMF.any()) {
+      if (!Inst->getType()->isFPOrFPVectorTy())
+        return Error(Loc, "fast-math-flags specified for select without "
+                          "floating-point scalar or vector return type");
+      Inst->setFastMathFlags(FMF);
+    }
+    return 0;
+  }
   case lltok::kw_va_arg:         return ParseVA_Arg(Inst, PFS);
   case lltok::kw_extractelement: return ParseExtractElement(Inst, PFS);
   case lltok::kw_insertelement:  return ParseInsertElement(Inst, PFS);
   case lltok::kw_shufflevector:  return ParseShuffleVector(Inst, PFS);
   case lltok::kw_phi:            return ParsePHI(Inst, PFS);
   case lltok::kw_landingpad:     return ParseLandingPad(Inst, PFS);
   // Call.
   case lltok::kw_call:     return ParseCall(Inst, PFS, CallInst::TCK_None);

llvm/trunk/lib/Bitcode/Reader/BitcodeReader.cpp

case bitc::FUNC_CODE_INST_VSELECT: {// VSELECT: [ty,opval,opval,predty,pred]
       } else {
         // expect i1
         if (Cond->getType() != Type::getInt1Ty(Context))
           return error("Invalid type for value");
       }
 
       I = SelectInst::Create(Cond, TrueVal, FalseVal);
       InstructionList.push_back(I);
+      if (OpNum < Record.size() && isa<FPMathOperator>(I)) {
+        FastMathFlags FMF = getDecodedFastMathFlags(Record[OpNum]);
+        if (FMF.any())
+          I->setFastMathFlags(FMF);
+      }
       break;
     }
 
     case bitc::FUNC_CODE_INST_EXTRACTELT: { // EXTRACTELT: [opty, opval, opval]
       unsigned OpNum = 0;
       Value *Vec, *Idx;
       if (getValueTypePair(Record, OpNum, NextValueNo, Vec) ||
           getValueTypePair(Record, OpNum, NextValueNo, Idx))

llvm/trunk/lib/Bitcode/Writer/BitcodeWriter.cpp

void ModuleBitcodeWriter::writeInstruction(const Instruction &I,
   case Instruction::InsertValue: {
     Code = bitc::FUNC_CODE_INST_INSERTVAL;
     pushValueAndType(I.getOperand(0), InstID, Vals);
     pushValueAndType(I.getOperand(1), InstID, Vals);
     const InsertValueInst *IVI = cast<InsertValueInst>(&I);
     Vals.append(IVI->idx_begin(), IVI->idx_end());
     break;
   }
-  case Instruction::Select:
+  case Instruction::Select: {
     Code = bitc::FUNC_CODE_INST_VSELECT;
     pushValueAndType(I.getOperand(1), InstID, Vals);
     pushValue(I.getOperand(2), InstID, Vals);
     pushValueAndType(I.getOperand(0), InstID, Vals);
+    uint64_t Flags = getOptimizationFlags(&I);
+    if (Flags != 0)
+      Vals.push_back(Flags);
     break;
+  }
   case Instruction::ExtractElement:
     Code = bitc::FUNC_CODE_INST_EXTRACTELT;
     pushValueAndType(I.getOperand(0), InstID, Vals);
     pushValueAndType(I.getOperand(1), InstID, Vals);
     break;
   case Instruction::InsertElement:
     Code = bitc::FUNC_CODE_INST_INSERTELT;
     pushValueAndType(I.getOperand(0), InstID, Vals);

llvm/trunk/test/Bitcode/compatibility.ll

define void @fastmathflags_binops(float %op1, float %op2) {
   ; CHECK: %f.afn = fadd afn float %op1, %op2
   %f.reassoc = fadd reassoc float %op1, %op2
   ; CHECK: %f.reassoc = fadd reassoc float %op1, %op2
   %f.fast = fadd fast float %op1, %op2
   ; CHECK: %f.fast = fadd fast float %op1, %op2
   ret void
 }
 
+define void @fastmathflags_select(i1 %cond, float %op1, float %op2) {
+  %f.nnan = select nnan i1 %cond, float %op1, float %op2
+  ; CHECK: %f.nnan = select nnan i1 %cond, float %op1, float %op2
+  %f.ninf = select ninf i1 %cond, float %op1, float %op2
+  ; CHECK: %f.ninf = select ninf i1 %cond, float %op1, float %op2
+  %f.nsz = select nsz i1 %cond, float %op1, float %op2
+  ; CHECK: %f.nsz = select nsz i1 %cond, float %op1, float %op2
+  %f.arcp = select arcp i1 %cond, float %op1, float %op2
+  ; CHECK: %f.arcp = select arcp i1 %cond, float %op1, float %op2
+  %f.contract = select contract i1 %cond, float %op1, float %op2
+  ; CHECK: %f.contract = select contract i1 %cond, float %op1, float %op2
+  %f.afn = select afn i1 %cond, float %op1, float %op2
+  ; CHECK: %f.afn = select afn i1 %cond, float %op1, float %op2
+  %f.reassoc = select reassoc i1 %cond, float %op1, float %op2
+  ; CHECK: %f.reassoc = select reassoc i1 %cond, float %op1, float %op2
+  %f.fast = select fast i1 %cond, float %op1, float %op2
+  ; CHECK: %f.fast = select fast i1 %cond, float %op1, float %op2
+  ret void
+}
+
+define void @fastmathflags_vector_select(<2 x i1> %cond, <2 x double> %op1, <2 x double> %op2) {
+  %f.nnan.nsz = select nnan nsz <2 x i1> %cond, <2 x double> %op1, <2 x double> %op2
+  ; CHECK: %f.nnan.nsz = select nnan nsz <2 x i1> %cond, <2 x double> %op1, <2 x double> %op2
+  %f.fast = select fast <2 x i1> %cond, <2 x double> %op1, <2 x double> %op2
+  ; CHECK: %f.fast = select fast <2 x i1> %cond, <2 x double> %op1, <2 x double> %op2
+  ret void
+}
+
 ; Check various fast math flags and floating-point types on calls.
 
 declare float @fmf1()
 declare double @fmf2()
 declare <4 x double> @fmf3()
 
 ; CHECK-LABEL: fastMathFlagsForCalls(
 define void @fastMathFlagsForCalls(float %f, double %d1, <4 x double> %d2) {

llvm/trunk/test/CodeGen/Generic/expand-experimental-reductions.ll

entry:
   ret i64 %r
 }
 
 define double @fmax_f64(<2 x double> %vec) {
 ; CHECK-LABEL: @fmax_f64(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[RDX_SHUF:%.*]] = shufflevector <2 x double> [[VEC:%.*]], <2 x double> undef, <2 x i32> <i32 1, i32 undef>
 ; CHECK-NEXT:    [[RDX_MINMAX_CMP:%.*]] = fcmp fast ogt <2 x double> [[VEC]], [[RDX_SHUF]]
-; CHECK-NEXT:    [[RDX_MINMAX_SELECT:%.*]] = select <2 x i1> [[RDX_MINMAX_CMP]], <2 x double> [[VEC]], <2 x double> [[RDX_SHUF]]
+; CHECK-NEXT:    [[RDX_MINMAX_SELECT:%.*]] = select fast <2 x i1> [[RDX_MINMAX_CMP]], <2 x double> [[VEC]], <2 x double> [[RDX_SHUF]]
 ; CHECK-NEXT:    [[TMP0:%.*]] = extractelement <2 x double> [[RDX_MINMAX_SELECT]], i32 0
 ; CHECK-NEXT:    ret double [[TMP0]]
 ;
 entry:
   %r = call double @llvm.experimental.vector.reduce.fmax.f64.v2f64(<2 x double> %vec)
   ret double %r
 }
 
 define double @fmin_f64(<2 x double> %vec) {
 ; CHECK-LABEL: @fmin_f64(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[RDX_SHUF:%.*]] = shufflevector <2 x double> [[VEC:%.*]], <2 x double> undef, <2 x i32> <i32 1, i32 undef>
 ; CHECK-NEXT:    [[RDX_MINMAX_CMP:%.*]] = fcmp fast olt <2 x double> [[VEC]], [[RDX_SHUF]]
-; CHECK-NEXT:    [[RDX_MINMAX_SELECT:%.*]] = select <2 x i1> [[RDX_MINMAX_CMP]], <2 x double> [[VEC]], <2 x double> [[RDX_SHUF]]
+; CHECK-NEXT:    [[RDX_MINMAX_SELECT:%.*]] = select fast <2 x i1> [[RDX_MINMAX_CMP]], <2 x double> [[VEC]], <2 x double> [[RDX_SHUF]]
 ; CHECK-NEXT:    [[TMP0:%.*]] = extractelement <2 x double> [[RDX_MINMAX_SELECT]], i32 0
 ; CHECK-NEXT:    ret double [[TMP0]]
 ;
 entry:
   %r = call double @llvm.experimental.vector.reduce.fmin.f64.v2f64(<2 x double> %vec)
   ret double %r
 }

llvm/trunk/test/Transforms/InstCombine/fast-math.ll

 ; This should always be set when unsafe-fp-math is true, but
 ; alternate the attributes for additional test coverage.
 ; 'nsz' is implied by the definition of fmax or fmin itself.
 
 ; Shrink and remove the call.
 define float @max1(float %a, float %b) {
 ; CHECK-LABEL: @max1(
 ; CHECK-NEXT:    [[TMP1:%.*]] = fcmp fast ogt float [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[TMP2:%.*]] = select i1 [[TMP1]], float [[A]], float [[B]]
+; CHECK-NEXT:    [[TMP2:%.*]] = select fast i1 [[TMP1]], float [[A]], float [[B]]
 ; CHECK-NEXT:    ret float [[TMP2]]
 ;
   %c = fpext float %a to double
   %d = fpext float %b to double
   %e = call fast double @fmax(double %c, double %d)
   %f = fptrunc double %e to float
   ret float %f
 }
 
 define float @max2(float %a, float %b) {
 ; CHECK-LABEL: @max2(
 ; CHECK-NEXT:    [[TMP1:%.*]] = fcmp nnan nsz ogt float [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[TMP2:%.*]] = select i1 [[TMP1]], float [[A]], float [[B]]
+; CHECK-NEXT:    [[TMP2:%.*]] = select nnan nsz i1 [[TMP1]], float [[A]], float [[B]]
 ; CHECK-NEXT:    ret float [[TMP2]]
 ;
   %c = call nnan float @fmaxf(float %a, float %b)
   ret float %c
 }
 
 
 define double @max3(double %a, double %b) {
 ; CHECK-LABEL: @max3(
 ; CHECK-NEXT:    [[TMP1:%.*]] = fcmp fast ogt double [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[TMP2:%.*]] = select i1 [[TMP1]], double [[A]], double [[B]]
+; CHECK-NEXT:    [[TMP2:%.*]] = select fast i1 [[TMP1]], double [[A]], double [[B]]
 ; CHECK-NEXT:    ret double [[TMP2]]
 ;
   %c = call fast double @fmax(double %a, double %b)
   ret double %c
 }
 
 define fp128 @max4(fp128 %a, fp128 %b) {
 ; CHECK-LABEL: @max4(
 ; CHECK-NEXT:    [[TMP1:%.*]] = fcmp nnan nsz ogt fp128 [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[TMP2:%.*]] = select i1 [[TMP1]], fp128 [[A]], fp128 [[B]]
+; CHECK-NEXT:    [[TMP2:%.*]] = select nnan nsz i1 [[TMP1]], fp128 [[A]], fp128 [[B]]
 ; CHECK-NEXT:    ret fp128 [[TMP2]]
 ;
   %c = call nnan fp128 @fmaxl(fp128 %a, fp128 %b)
   ret fp128 %c
 }
 
 ; Shrink and remove the call.
 define float @min1(float %a, float %b) {
 ; CHECK-LABEL: @min1(
 ; CHECK-NEXT:    [[TMP1:%.*]] = fcmp nnan nsz olt float [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[TMP2:%.*]] = select i1 [[TMP1]], float [[A]], float [[B]]
+; CHECK-NEXT:    [[TMP2:%.*]] = select nnan nsz i1 [[TMP1]], float [[A]], float [[B]]
 ; CHECK-NEXT:    ret float [[TMP2]]
 ;
   %c = fpext float %a to double
   %d = fpext float %b to double
   %e = call nnan double @fmin(double %c, double %d)
   %f = fptrunc double %e to float
   ret float %f
 }
 
 define float @min2(float %a, float %b) {
 ; CHECK-LABEL: @min2(
 ; CHECK-NEXT:    [[TMP1:%.*]] = fcmp fast olt float [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[TMP2:%.*]] = select i1 [[TMP1]], float [[A]], float [[B]]
+; CHECK-NEXT:    [[TMP2:%.*]] = select fast i1 [[TMP1]], float [[A]], float [[B]]
 ; CHECK-NEXT:    ret float [[TMP2]]
 ;
   %c = call fast float @fminf(float %a, float %b)
   ret float %c
 }
 
 define double @min3(double %a, double %b) {
 ; CHECK-LABEL: @min3(
 ; CHECK-NEXT:    [[TMP1:%.*]] = fcmp nnan nsz olt double [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[TMP2:%.*]] = select i1 [[TMP1]], double [[A]], double [[B]]
+; CHECK-NEXT:    [[TMP2:%.*]] = select nnan nsz i1 [[TMP1]], double [[A]], double [[B]]
 ; CHECK-NEXT:    ret double [[TMP2]]
 ;
   %c = call nnan double @fmin(double %a, double %b)
   ret double %c
 }
 
 define fp128 @min4(fp128 %a, fp128 %b) {
 ; CHECK-LABEL: @min4(
 ; CHECK-NEXT:    [[TMP1:%.*]] = fcmp fast olt fp128 [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT:    [[TMP2:%.*]] = select i1 [[TMP1]], fp128 [[A]], fp128 [[B]]
+; CHECK-NEXT:    [[TMP2:%.*]] = select fast i1 [[TMP1]], fp128 [[A]], fp128 [[B]]
 ; CHECK-NEXT:    ret fp128 [[TMP2]]
 ;
   %c = call fast fp128 @fminl(fp128 %a, fp128 %b)
   ret fp128 %c
 }
 
 ; ((which ? 2.0 : a) + 1.0) => (which ? 3.0 : (a + 1.0))
 ; This is always safe.  No FMF required.

llvm/trunk/test/Transforms/InstCombine/pow-sqrt.ll

 
 ; This makes no difference, but FMF are propagated.
 
 define double @pow_libcall_half_approx(double %x) {
 ; CHECK-LABEL: @pow_libcall_half_approx(
 ; CHECK-NEXT:    [[SQRT:%.*]] = call afn double @sqrt(double [[X:%.*]])
 ; CHECK-NEXT:    [[ABS:%.*]] = call afn double @llvm.fabs.f64(double [[SQRT]])
 ; CHECK-NEXT:    [[ISINF:%.*]] = fcmp afn oeq double [[X]], 0xFFF0000000000000
-; CHECK-NEXT:    [[TMP1:%.*]] = select i1 [[ISINF]], double 0x7FF0000000000000, double [[ABS]]
+; CHECK-NEXT:    [[TMP1:%.*]] = select afn i1 [[ISINF]], double 0x7FF0000000000000, double [[ABS]]
 ; CHECK-NEXT:    ret double [[TMP1]]
 ;
   %pow = call afn double @pow(double %x, double 5.0e-01)
   ret double %pow
 }
 
 define <2 x double> @pow_intrinsic_half_approx(<2 x double> %x) {
 ; CHECK-LABEL: @pow_intrinsic_half_approx(
 ; CHECK-NEXT:    [[SQRT:%.*]] = call afn <2 x double> @llvm.sqrt.v2f64(<2 x double> [[X:%.*]])
 ; CHECK-NEXT:    [[ABS:%.*]] = call afn <2 x double> @llvm.fabs.v2f64(<2 x double> [[SQRT]])
 ; CHECK-NEXT:    [[ISINF:%.*]] = fcmp afn oeq <2 x double> [[X]], <double 0xFFF0000000000000, double 0xFFF0000000000000>
-; CHECK-NEXT:    [[TMP1:%.*]] = select <2 x i1> [[ISINF]], <2 x double> <double 0x7FF0000000000000, double 0x7FF0000000000000>, <2 x double> [[ABS]]
+; CHECK-NEXT:    [[TMP1:%.*]] = select afn <2 x i1> [[ISINF]], <2 x double> <double 0x7FF0000000000000, double 0x7FF0000000000000>, <2 x double> [[ABS]]
 ; CHECK-NEXT:    ret <2 x double> [[TMP1]]
 ;
   %pow = call afn <2 x double> @llvm.pow.v2f64(<2 x double> %x, <2 x double> <double 5.0e-01, double 5.0e-01>)
   ret <2 x double> %pow
 }
 
 define float @powf_intrinsic_half_fast(float %x) {
 ; CHECK-LABEL: @powf_intrinsic_half_fast(

 }
 
 ; If we can disregard -0.0, no need for fabs.
 
 define double @pow_libcall_half_nsz(double %x) {
 ; CHECK-LABEL: @pow_libcall_half_nsz(
 ; CHECK-NEXT:    [[SQRT:%.*]] = call nsz double @sqrt(double [[X:%.*]])
 ; CHECK-NEXT:    [[ISINF:%.*]] = fcmp nsz oeq double [[X]], 0xFFF0000000000000
-; CHECK-NEXT:    [[TMP1:%.*]] = select i1 [[ISINF]], double 0x7FF0000000000000, double [[SQRT]]
+; CHECK-NEXT:    [[TMP1:%.*]] = select nsz i1 [[ISINF]], double 0x7FF0000000000000, double [[SQRT]]
 ; CHECK-NEXT:    ret double [[TMP1]]
 ;
   %pow = call nsz double @pow(double %x, double 5.0e-01)
   ret double %pow
 }
 
 define double @pow_intrinsic_half_nsz(double %x) {
 ; CHECK-LABEL: @pow_intrinsic_half_nsz(
 ; CHECK-NEXT:    [[SQRT:%.*]] = call nsz double @llvm.sqrt.f64(double [[X:%.*]])
 ; CHECK-NEXT:    [[ISINF:%.*]] = fcmp nsz oeq double [[X]], 0xFFF0000000000000
-; CHECK-NEXT:    [[TMP1:%.*]] = select i1 [[ISINF]], double 0x7FF0000000000000, double [[SQRT]]
+; CHECK-NEXT:    [[TMP1:%.*]] = select nsz i1 [[ISINF]], double 0x7FF0000000000000, double [[SQRT]]
 ; CHECK-NEXT:    ret double [[TMP1]]
 ;
   %pow = call nsz double @llvm.pow.f64(double %x, double 5.0e-01)
   ret double %pow
 }
 
 ; This is just sqrt.
 

llvm/trunk/test/Transforms/LoopVectorize/float-minmax-instruction-flag.ll

 ; CHECK-NEXT:    [[TMP5:%.*]] = extractelement <4 x i1> [[TMP4]], i32 0
 ; CHECK-NEXT:    [[TMP6]] = select <4 x i1> [[TMP4]], <4 x float> [[VEC_PHI]], <4 x float> [[WIDE_LOAD]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add i64 [[INDEX]], 4
 ; CHECK-NEXT:    [[TMP7:%.*]] = icmp eq i64 [[INDEX_NEXT]], 65536
 ; CHECK-NEXT:    br i1 [[TMP7]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !0
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    [[RDX_SHUF:%.*]] = shufflevector <4 x float> [[TMP6]], <4 x float> undef, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef>
 ; CHECK-NEXT:    [[RDX_MINMAX_CMP:%.*]] = fcmp fast olt <4 x float> [[TMP6]], [[RDX_SHUF]]
-; CHECK-NEXT:    [[RDX_MINMAX_SELECT:%.*]] = select <4 x i1> [[RDX_MINMAX_CMP]], <4 x float> [[TMP6]], <4 x float> [[RDX_SHUF]]
+; CHECK-NEXT:    [[RDX_MINMAX_SELECT:%.*]] = select fast <4 x i1> [[RDX_MINMAX_CMP]], <4 x float> [[TMP6]], <4 x float> [[RDX_SHUF]]
 ; CHECK-NEXT:    [[RDX_SHUF1:%.*]] = shufflevector <4 x float> [[RDX_MINMAX_SELECT]], <4 x float> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
 ; CHECK-NEXT:    [[RDX_MINMAX_CMP2:%.*]] = fcmp fast olt <4 x float> [[RDX_MINMAX_SELECT]], [[RDX_SHUF1]]
-; CHECK-NEXT:    [[RDX_MINMAX_SELECT3:%.*]] = select <4 x i1> [[RDX_MINMAX_CMP2]], <4 x float> [[RDX_MINMAX_SELECT]], <4 x float> [[RDX_SHUF1]]
+; CHECK-NEXT:    [[RDX_MINMAX_SELECT3:%.*]] = select fast <4 x i1> [[RDX_MINMAX_CMP2]], <4 x float> [[RDX_MINMAX_SELECT]], <4 x float> [[RDX_SHUF1]]
 ; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <4 x float> [[RDX_MINMAX_SELECT3]], i32 0
 ; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 65536, 65536
 ; CHECK-NEXT:    br i1 [[CMP_N]], label [[OUT:%.*]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 65537, [[MIDDLE_BLOCK]] ], [ 1, [[TOP:%.*]] ]
 ; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] = phi float [ [[T]], [[TOP]] ], [ [[TMP8]], [[MIDDLE_BLOCK]] ]
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:

llvm/trunk/test/Transforms/LoopVectorize/minmax_reduction.ll

 ; Maximum.
 
 ; Turn this into a max reduction in the presence of a no-nans-fp-math attribute.
 ; CHECK-LABEL: @max_red_float(
 ; CHECK: fcmp fast ogt <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast ogt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @max_red_float(float %max) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %max.red.08 = phi float [ %max, %entry ], [ %max.red.0, %for.body ]
for.end:
   ret float %max.red.0
 }
 
 ; CHECK-LABEL: @max_red_float_ge(
 ; CHECK: fcmp fast oge <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast ogt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @max_red_float_ge(float %max) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %max.red.08 = phi float [ %max, %entry ], [ %max.red.0, %for.body ]
for.end:
   ret float %max.red.0
 }
 
 ; CHECK-LABEL: @inverted_max_red_float(
 ; CHECK: fcmp fast olt <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast ogt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @inverted_max_red_float(float %max) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %max.red.08 = phi float [ %max, %entry ], [ %max.red.0, %for.body ]
for.end:
   ret float %max.red.0
 }
 
 ; CHECK-LABEL: @inverted_max_red_float_le(
 ; CHECK: fcmp fast ole <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast ogt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @inverted_max_red_float_le(float %max) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %max.red.08 = phi float [ %max, %entry ], [ %max.red.0, %for.body ]
for.end:
   ret float %max.red.0
 }
 
 ; CHECK-LABEL: @unordered_max_red_float(
 ; CHECK: fcmp fast ole <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast ogt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @unordered_max_red_float(float %max) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %max.red.08 = phi float [ %max, %entry ], [ %max.red.0, %for.body ]
for.end:
   ret float %max.red.0
 }
 
 ; CHECK-LABEL: @unordered_max_red_float_ge(
 ; CHECK: fcmp fast olt <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast ogt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @unordered_max_red_float_ge(float %max) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %max.red.08 = phi float [ %max, %entry ], [ %max.red.0, %for.body ]
for.end:
   ret float %max.red.0
 }
 
 ; CHECK-LABEL: @inverted_unordered_max_red_float(
 ; CHECK: fcmp fast oge <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast ogt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @inverted_unordered_max_red_float(float %max) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %max.red.08 = phi float [ %max, %entry ], [ %max.red.0, %for.body ]
for.end:
   ret float %max.red.0
 }
 
 ; CHECK-LABEL: @inverted_unordered_max_red_float_le(
 ; CHECK: fcmp fast ogt <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast ogt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @inverted_unordered_max_red_float_le(float %max) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %max.red.08 = phi float [ %max, %entry ], [ %max.red.0, %for.body ]

 ; Minimum.
 
 ; Turn this into a min reduction in the presence of a no-nans-fp-math attribute.
 ; CHECK-LABEL: @min_red_float(
 ; CHECK: fcmp fast olt <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast olt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @min_red_float(float %min) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %min.red.08 = phi float [ %min, %entry ], [ %min.red.0, %for.body ]
for.end:
   ret float %min.red.0
 }
 
 ; CHECK-LABEL: @min_red_float_le(
 ; CHECK: fcmp fast ole <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast olt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @min_red_float_le(float %min) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %min.red.08 = phi float [ %min, %entry ], [ %min.red.0, %for.body ]
for.end:
   ret float %min.red.0
 }
 
 ; CHECK-LABEL: @inverted_min_red_float(
 ; CHECK: fcmp fast ogt <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast olt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @inverted_min_red_float(float %min) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %min.red.08 = phi float [ %min, %entry ], [ %min.red.0, %for.body ]
for.end:
   ret float %min.red.0
 }
 
 ; CHECK-LABEL: @inverted_min_red_float_ge(
 ; CHECK: fcmp fast oge <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast olt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @inverted_min_red_float_ge(float %min) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %min.red.08 = phi float [ %min, %entry ], [ %min.red.0, %for.body ]
for.end:
   ret float %min.red.0
 }
 
 ; CHECK-LABEL: @unordered_min_red_float(
 ; CHECK: fcmp fast oge <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast olt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @unordered_min_red_float(float %min) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %min.red.08 = phi float [ %min, %entry ], [ %min.red.0, %for.body ]
for.end:
   ret float %min.red.0
 }
 
 ; CHECK-LABEL: @unordered_min_red_float_le(
 ; CHECK: fcmp fast ogt <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast olt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @unordered_min_red_float_le(float %min) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %min.red.08 = phi float [ %min, %entry ], [ %min.red.0, %for.body ]
for.end:
   ret float %min.red.0
 }
 
 ; CHECK-LABEL: @inverted_unordered_min_red_float(
 ; CHECK: fcmp fast ole <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast olt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @inverted_unordered_min_red_float(float %min) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %min.red.08 = phi float [ %min, %entry ], [ %min.red.0, %for.body ]
for.end:
   ret float %min.red.0
 }
 
 ; CHECK-LABEL: @inverted_unordered_min_red_float_ge(
 ; CHECK: fcmp fast olt <2 x float>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast olt <2 x float>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define float @inverted_unordered_min_red_float_ge(float %min) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %min.red.08 = phi float [ %min, %entry ], [ %min.red.0, %for.body ]

 }
 
 ; Make sure we handle doubles, too.
 ; CHECK-LABEL: @min_red_double(
 ; CHECK: fcmp fast olt <2 x double>
 ; CHECK: select <2 x i1>
 ; CHECK: middle.block
 ; CHECK: fcmp fast olt <2 x double>
-; CHECK: select <2 x i1>
+; CHECK: select fast <2 x i1>
 
 define double @min_red_double(double %min) #0 {
 entry:
   br label %for.body
 
 for.body:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %min.red.08 = phi double [ %min, %entry ], [ %min.red.0, %for.body ]