[PatternMatch] Add new FP matchers. NFC.

[foad]

Summary

This adds matchers m_NonNaN, m_NonInf, m_Finite and m_NonZeroFP as well
as generic support for binding the matched value to an APFloat.

I tried to follow the existing convention of using an FP suffix for
predicates like zero and non-zero, which could be confused with the
integer versions, but not for predicates which are clearly already
FP-specific.

[RKSimon]

llvm/lib/Analysis/ValueTracking.cpp

3565–3566

Would it be better to simplify this entire loop to match(V, m_NonNaN()) ?

[foad]

Use matchers to handle fixed width vectors.

[RKSimon]

llvm/lib/Analysis/InstructionSimplify.cpp

4039

I'd expect this now to work for non-uniform vectors that contain different non-zero float values (and also undefs) - can we add test coverage for this please?

llvm/lib/Analysis/ValueTracking.cpp

5234

Ideally this function needs extending to support non-uniform vectors as well.

5247

m_APFloat didn't match constant vectors containing undefs but m_Finite will? Again, we need test coverage for this.

[foad]

Test simplifySelectWithFCmp on non-uniform vectors.

[foad]

llvm/lib/Analysis/ValueTracking.cpp

5234

Sure, but I'd prefer to do that in a separate patch. Currently it relies on checks like *FC1 < *FC2 below that won't work for vectors, and I'm not sure how to write that in a vector-friendly way.

5247

Since this whole function does support vectors yet, there's NFC here and nothing to test.

[RKSimon]

llvm/lib/Analysis/ValueTracking.cpp

5247

OK - in which case a TODO comment plus an assert that !getType->isVectorTy() at the top of the function would be fine.

[foad]

Add a vector test case for matchFastFloatClamp.

[foad]

llvm/lib/Analysis/ValueTracking.cpp

5247

I was mistaken. This function did already handle splat vectors with no undefs, because of the use of m_APFloat matchers. I've added an explicit test case for this. My patch does not change this behaviour.

[foad]

Ping!

[RKSimon]

@spatel @lebedev.ri any additional comments?

[lebedev.ri]

llvm/include/llvm/IR/PatternMatch.h

338–340

This should probably mention whether or not undef vector constants are allowed.

353

Note that this is implicitly /*AllowUndefs=*/false

587

I think i'd like to see some dedicated tests with undefs in llvm/unittest/IR/PatternMatch.cpp.

[foad]

Add comments.

[foad]

llvm/include/llvm/IR/PatternMatch.h

353

Right, so there is a pre-existing problem with matchers like this one, which claim to allow undef, but in fact the second (binding) form of the matcher does not:

/// Match an integer or vector of negative values.
/// For vectors, this includes constants with undefined elements.
inline cst_pred_ty<is_negative> m_Negative() {
  return cst_pred_ty<is_negative>();
}
inline api_pred_ty<is_negative> m_Negative(const APInt *&V) {
  return V;
}

[foad]

Add unit tests.

[lebedev.ri]

llvm/include/llvm/IR/PatternMatch.h

587

So does it or does it not allow undefs?

[RKSimon]

llvm/include/llvm/IR/PatternMatch.h

587

AFAICT cstfp_pred_ty supports vectors with undef elements but apf_pred_ty doesnot

[foad]

llvm/include/llvm/IR/PatternMatch.h

587

It allows undefs.

[lebedev.ri]

I'm lost.
If their code related to handling splats is identical (neither of them pass anything into getSplatValue()), why do they end up with different undef behaviour?

[foad]

In D89038#2347410, @lebedev.ri wrote:

I'm lost.
If their code related to handling splats is identical (neither of them pass anything into getSplatValue()), why do they end up with different undef behaviour?

cstfp_pred_ty has its own handling of non-splat vector constants. apf_pred_ty does not. I realise this is not great! It was copied directly from the integer equivalents, which have the same inconsistency.

[lebedev.ri]

llvm/include/llvm/IR/PatternMatch.h

614–618

So if this is using apf_pred_ty, then this doesn't support undef constants then?

[foad]

llvm/include/llvm/IR/PatternMatch.h

614–618

Correct. m_Finite() does, m_Finite(const APFloat *&) does not. This is the pre-existing problem I mentioned which has now been copied across from the integer matchers to the fp ones.

[lebedev.ri]

Anyways i think this looks fine to me now. Thanks.

llvm/include/llvm/IR/PatternMatch.h

614

But it says it does.

[foad]

Change apf_pred_ty to allow undefs and test that this works for m_Finite(const APFloat *&).

[foad]

llvm/include/llvm/IR/PatternMatch.h

614

Yes, that's the pre-existing bug! Anyway I have now fixed this for the new float matchers, but I have not touched the existing integer matchers.

[lebedev.ri]

llvm/include/llvm/IR/PatternMatch.h

614

Okay, you lost me there.
How it's a "pre-existing bug", if you add this matcher here?

[foad]

llvm/include/llvm/IR/PatternMatch.h

614

I should have said: a pre-existing bug in the integer matchers, which I copied across to the new fp matchers.

[lebedev.ri]

I would strongly recommend to split this into two separate commits - adding the matchers, and using them.
Because while the matchers themselves LG, i'm not at all confident with users.
When doing latter, please ensure that each separate change has appropriate undef test coverage, and ideally check that alive2 is happy about those transforms.

[foad]

In D89038#2347889, @lebedev.ri wrote:

I would strongly recommend to split this into two separate commits - adding the matchers, and using them.

Can do. But it feels like every time I try to separate things like that, some reviewer asks me to put it all into one patch again :( https://reviews.llvm.org/D87034#2254864

[foad]

In D89038#2347889, @lebedev.ri wrote:

I would strongly recommend to split this into two separate commits - adding the matchers, and using them.

OK, I split the "using them" part out into D90015.

llvm/include/llvm/IR/PatternMatch.h

if (V->getType()->isVectorTy())
             Res = &CI->getValue();
             return true;
           }
 
     return false;
   }
 };
 
+/// This helper class is used to match scalar and vector constants that
+/// satisfy a specified predicate, and bind them to an APFloat.
+template <typename Predicate> struct apf_pred_ty : public Predicate {
+  const APFloat *&Res;
+
+  apf_pred_ty(const APFloat *&R) : Res(R) {}
+
+  template <typename ITy> bool match(ITy *V) {
+    if (const auto *CI = dyn_cast<ConstantFP>(V))
+      if (this->isValue(CI->getValue())) {
+        Res = &CI->getValue();
+        return true;
+      }
+    if (V->getType()->isVectorTy())
+      if (const auto *C = dyn_cast<Constant>(V))
+        if (auto *CI = dyn_cast_or_null<ConstantFP>(C->getSplatValue()))
+          if (this->isValue(CI->getValue())) {
+            Res = &CI->getValue();
+            return true;
+          }
+
+    return false;
+  }
+};
+
 ///////////////////////////////////////////////////////////////////////////////
 //
 // Encapsulate constant value queries for use in templated predicate matchers.
 // This allows checking if constants match using compound predicates and works
 // with vector constants, possibly with relaxed constraints. For example, ignore
 // undef values.
 //
 ///////////////////////////////////////////////////////////////////////////////
struct is_nan {
   bool isValue(const APFloat &C) { return C.isNaN(); }
 };
 /// Match an arbitrary NaN constant. This includes quiet and signalling nans.
 /// For vectors, this includes constants with undefined elements.
 inline cstfp_pred_ty<is_nan> m_NaN() {
   return cstfp_pred_ty<is_nan>();
 }
 
+struct is_nonnan {
+  bool isValue(const APFloat &C) { return !C.isNaN(); }
+};
+/// Match a non-NaN FP constant.
+/// For vectors, this includes constants with undefined elements.
+inline cstfp_pred_ty<is_nonnan> m_NonNaN() {
+  return cstfp_pred_ty<is_nonnan>();
+}
+
 struct is_inf {
   bool isValue(const APFloat &C) { return C.isInfinity(); }
 };
 /// Match a positive or negative infinity FP constant.
 /// For vectors, this includes constants with undefined elements.
 inline cstfp_pred_ty<is_inf> m_Inf() {
   return cstfp_pred_ty<is_inf>();
 }
 
+struct is_noninf {
+  bool isValue(const APFloat &C) { return !C.isInfinity(); }
+};
+/// Match a non-infinity FP constant, i.e. finite or NaN.
+/// For vectors, this includes constants with undefined elements.
+inline cstfp_pred_ty<is_noninf> m_NonInf() {
+  return cstfp_pred_ty<is_noninf>();
+}
+
+struct is_finite {
+  bool isValue(const APFloat &C) { return C.isFinite(); }
+};
+/// Match a finite FP constant, i.e. not infinity or NaN.
+/// For vectors, this includes constants with undefined elements.
+inline cstfp_pred_ty<is_finite> m_Finite() {
+  return cstfp_pred_ty<is_finite>();
+}
+inline apf_pred_ty<is_finite> m_Finite(const APFloat *&V) { return V; }
+
 struct is_any_zero_fp {
   bool isValue(const APFloat &C) { return C.isZero(); }
 };
 /// Match a floating-point negative zero or positive zero.
 /// For vectors, this includes constants with undefined elements.
 inline cstfp_pred_ty<is_any_zero_fp> m_AnyZeroFP() {
   return cstfp_pred_ty<is_any_zero_fp>();
 }
struct is_neg_zero_fp {
   bool isValue(const APFloat &C) { return C.isNegZero(); }
 };
 /// Match a floating-point negative zero.
 /// For vectors, this includes constants with undefined elements.
 inline cstfp_pred_ty<is_neg_zero_fp> m_NegZeroFP() {
   return cstfp_pred_ty<is_neg_zero_fp>();
 }
 
+struct is_non_zero_fp {
+  bool isValue(const APFloat &C) { return C.isNonZero(); }
+};
+/// Match a floating-point non-zero.
+/// For vectors, this includes constants with undefined elements.
+inline cstfp_pred_ty<is_non_zero_fp> m_NonZeroFP() {
+  return cstfp_pred_ty<is_non_zero_fp>();
+}
+
 ///////////////////////////////////////////////////////////////////////////////
 
 template <typename Class> struct bind_ty {
   Class *&VR;
 
   bind_ty(Class *&V) : VR(V) {}
 
   template <typename ITy> bool match(ITy *V) {

llvm/lib/Analysis/InstructionSimplify.cpp

if (!match(Cond, m_FCmp(Pred, m_Specific(T), m_Specific(F))) &&
       !match(Cond, m_FCmp(Pred, m_Specific(F), m_Specific(T))))
     return nullptr;
 
   // This transform is safe if we do not have (do not care about) -0.0 or if
   // at least one operand is known to not be -0.0. Otherwise, the select can
   // change the sign of a zero operand.
   bool HasNoSignedZeros = Q.CxtI && isa<FPMathOperator>(Q.CxtI) &&
                           Q.CxtI->hasNoSignedZeros();
-  const APFloat *C;
-  if (HasNoSignedZeros || (match(T, m_APFloat(C)) && C->isNonZero()) ||
-                          (match(F, m_APFloat(C)) && C->isNonZero())) {
+  if (HasNoSignedZeros || match(T, m_NonZeroFP()) || match(F, m_NonZeroFP())) {
     // (T == F) ? T : F --> F
     // (F == T) ? T : F --> F
     if (Pred == FCmpInst::FCMP_OEQ)
       return F;
 
     // (T != F) ? T : F --> T
     // (F != T) ? T : F --> T
     if (Pred == FCmpInst::FCMP_UNE)

llvm/lib/Analysis/ValueTracking.cpp

bool llvm::isKnownNeverInfinity(const Value *V, const TargetLibraryInfo *TLI,
                                 unsigned Depth) {
   assert(V->getType()->isFPOrFPVectorTy() && "Querying for Inf on non-FP type");
 
   // If we're told that infinities won't happen, assume they won't.
   if (auto *FPMathOp = dyn_cast<FPMathOperator>(V))
     if (FPMathOp->hasNoInfs())
       return true;
 
-  // Handle scalar constants.
-  if (auto *CFP = dyn_cast<ConstantFP>(V))
-    return !CFP->isInfinity();
+  // Handle scalar and fixed width vector constants.
+  if (isa<Constant>(V))
+    return match(V, m_NonInf());
 
   if (Depth == MaxAnalysisRecursionDepth)
     return false;
 
   if (auto *Inst = dyn_cast<Instruction>(V)) {
     switch (Inst->getOpcode()) {
     case Instruction::Select: {
       return isKnownNeverInfinity(Inst->getOperand(1), TLI, Depth + 1) &&
case Instruction::UIToFP: {
       Type *FPTy = Inst->getType()->getScalarType();
       return ilogb(APFloat::getLargest(FPTy->getFltSemantics())) >= IntSize;
     }
     default:
       break;
     }
   }
 
-  // try to handle fixed width vector constants
-  auto *VFVTy = dyn_cast<FixedVectorType>(V->getType());
-  if (VFVTy && isa<Constant>(V)) {
-    // For vectors, verify that each element is not infinity.
-    unsigned NumElts = VFVTy->getNumElements();
-    for (unsigned i = 0; i != NumElts; ++i) {
-      Constant *Elt = cast<Constant>(V)->getAggregateElement(i);
-      if (!Elt)
-        return false;
-      if (isa<UndefValue>(Elt))
-        continue;
-      auto *CElt = dyn_cast<ConstantFP>(Elt);
-      if (!CElt || CElt->isInfinity())
-        return false;
-    }
-    // All elements were confirmed non-infinity or undefined.
-    return true;
-  }
-
   // was not able to prove that V never contains infinity
   return false;
 }
 
 bool llvm::isKnownNeverNaN(const Value *V, const TargetLibraryInfo *TLI,
                            unsigned Depth) {
   assert(V->getType()->isFPOrFPVectorTy() && "Querying for NaN on non-FP type");
 
   // If we're told that NaNs won't happen, assume they won't.
   if (auto *FPMathOp = dyn_cast<FPMathOperator>(V))
     if (FPMathOp->hasNoNaNs())
       return true;
 
-  // Handle scalar constants.
-  if (auto *CFP = dyn_cast<ConstantFP>(V))
-    return !CFP->isNaN();
+  // Handle scalar and fixed width vector constants.
+  if (isa<Constant>(V))
+    return match(V, m_NonNaN());
 
   if (Depth == MaxAnalysisRecursionDepth)
     return false;
 
   if (auto *Inst = dyn_cast<Instruction>(V)) {
     switch (Inst->getOpcode()) {
     case Instruction::FAdd:
     case Instruction::FSub:
if (const auto *II = dyn_cast<IntrinsicInst>(V)) {
     case Intrinsic::maxnum:
       // If either operand is not NaN, the result is not NaN.
       return isKnownNeverNaN(II->getArgOperand(0), TLI, Depth + 1) ||
              isKnownNeverNaN(II->getArgOperand(1), TLI, Depth + 1);
     default:
       return false;
     }
   }
 
-  // Try to handle fixed width vector constants
-  auto *VFVTy = dyn_cast<FixedVectorType>(V->getType());
-  if (VFVTy && isa<Constant>(V)) {
-    // For vectors, verify that each element is not NaN.
-    unsigned NumElts = VFVTy->getNumElements();
-    for (unsigned i = 0; i != NumElts; ++i) {
-      Constant *Elt = cast<Constant>(V)->getAggregateElement(i);
-      if (!Elt)
-        return false;
-      if (isa<UndefValue>(Elt))
-        continue;
-      auto *CElt = dyn_cast<ConstantFP>(Elt);
-      if (!CElt || CElt->isNaN())
-        return false;
-    }
-    // All elements were confirmed not-NaN or undefined.
-    return true;
-  }
-
   // Was not able to prove that V never contains NaN
   return false;
 }
 
 Value *llvm::isBytewiseValue(Value *V, const DataLayout &DL) {
 
   // All byte-wide stores are splatable, even of arbitrary variables.
   if (V->getType()->isIntegerTy(8))
     return V;

 /// pattern.
 static SelectPatternResult matchFastFloatClamp(CmpInst::Predicate Pred,
                                                Value *CmpLHS, Value *CmpRHS,
                                                Value *TrueVal, Value *FalseVal,
                                                Value *&LHS, Value *&RHS) {
   // Try to match
   //   X < C1 ? C1 : Min(X, C2) --> Max(C1, Min(X, C2))
   //   X > C1 ? C1 : Max(X, C2) --> Min(C1, Max(X, C2))
   // and return description of the outer Max/Min.
 
   // First, check if select has inverse order:
   if (CmpRHS == FalseVal) {
     std::swap(TrueVal, FalseVal);
     Pred = CmpInst::getInversePredicate(Pred);
   }
 
   // Assume success now. If there's no match, callers should not use these anyway.
   LHS = TrueVal;
   RHS = FalseVal;
 
   const APFloat *FC1;
-  if (CmpRHS != TrueVal || !match(CmpRHS, m_APFloat(FC1)) || !FC1->isFinite())
+  if (CmpRHS != TrueVal || !match(CmpRHS, m_Finite(FC1)))
     return {SPF_UNKNOWN, SPNB_NA, false};
 
   const APFloat *FC2;
   switch (Pred) {
   case CmpInst::FCMP_OLT:
   case CmpInst::FCMP_OLE:
   case CmpInst::FCMP_ULT:
   case CmpInst::FCMP_ULE:

llvm/test/Transforms/InstCombine/clamp-to-minmax.ll

 ;
   %cmp2 = fcmp fast olt float %x, 255.0
   %min = select i1 %cmp2, float %x, float 255.0
   %cmp1 = fcmp fast olt float %x, 1.0
   %r = select i1 %cmp1, float 1.0, float %min
   ret float %r
 }
 
+; (X < C1) ? C1 : MIN(X, C2)
+define <2 x float> @clamp_float_fast_ordered_strict_maxmin_vec(<2 x float> %x) {
+; CHECK-LABEL: define {{[^@]+}}@clamp_float_fast_ordered_strict_maxmin_vec(
+; CHECK-NEXT:    [[CMP2:%.*]] = fcmp fast olt <2 x float> [[X:%.*]], <float 2.550000e+02, float 2.550000e+02>
+; CHECK-NEXT:    [[MIN:%.*]] = select <2 x i1> [[CMP2]], <2 x float> [[X]], <2 x float> <float 2.550000e+02, float 2.550000e+02>
+; CHECK-NEXT:    [[TMP1:%.*]] = call fast <2 x float> @llvm.maxnum.v2f32(<2 x float> [[MIN]], <2 x float> <float 1.000000e+00, float 1.000000e+00>)
+; CHECK-NEXT:    ret <2 x float> [[TMP1]]
+;
+  %cmp2 = fcmp fast olt <2 x float> %x, <float 255.0, float 255.0>
+  %min = select <2 x i1> %cmp2, <2 x float> %x, <2 x float> <float 255.0, float 255.0>
+  %cmp1 = fcmp fast olt <2 x float> %x, <float 1.0, float 1.0>
+  %r = select <2 x i1> %cmp1, <2 x float> <float 1.0, float 1.0>, <2 x float> %min
+  ret <2 x float> %r
+}
+
 ; (X <= C1) ? C1 : MIN(X, C2)
 define float @clamp_float_fast_ordered_nonstrict_maxmin(float %x) {
 ; CHECK-LABEL: define {{[^@]+}}@clamp_float_fast_ordered_nonstrict_maxmin(
 ; CHECK-NEXT:    [[CMP2:%.*]] = fcmp fast olt float [[X:%.*]], 2.550000e+02
 ; CHECK-NEXT:    [[MIN:%.*]] = select i1 [[CMP2]], float [[X]], float 2.550000e+02
 ; CHECK-NEXT:    [[TMP1:%.*]] = call fast float @llvm.maxnum.f32(float [[MIN]], float 1.000000e+00)
 ; CHECK-NEXT:    ret float [[TMP1]]
 ;

llvm/test/Transforms/InstSimplify/fcmp-select.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -instsimplify -S | FileCheck %s
 
 ; X == 42.0 ? X : 42.0 --> 42.0
 
 define double @oeq(double %x) {
 ; CHECK-LABEL: @oeq(
 ; CHECK-NEXT:    ret double 4.200000e+01
 ;
   %cmp = fcmp oeq double %x, 42.0
   %cond = select i1 %cmp, double %x, double 42.0
   ret double %cond
 }
 
+; X == <4.0, 2.0, undef> ? X : <4.0, 2.0, undef> --> <4.0, 2.0, undef>
+
+define <3 x double> @oeq_vec(<3 x double> %x) {
+; CHECK-LABEL: @oeq_vec(
+; CHECK-NEXT:    ret <3 x double> <double 4.000000e+00, double 2.000000e+00, double undef>
+;
+  %cmp = fcmp oeq <3 x double> %x, <double 4.0, double 2.0, double undef>
+  %cond = select <3 x i1> %cmp, <3 x double> %x, <3 x double> <double 4.0, double 2.0, double undef>
+  ret <3 x double> %cond
+}
+
 ; X == 42.0 ? 42.0 : X --> X
 
 define float @oeq_swapped(float %x) {
 ; CHECK-LABEL: @oeq_swapped(
 ; CHECK-NEXT:    ret float [[X:%.*]]
 ;
   %cmp = fcmp oeq float %x, 42.0
   %cond = select i1 %cmp, float 42.0, float %x