Fix incorrect folding of an ordered fcmp with a vector of all NaN.

[andreadb]

Summary

This patch fixes a crash caused by an incorrect folding of an ordered comparison between a packed floating point vector and a splat vector of NaN.

An ordered comparison between a vector and a constant vector of NaN, should always be folded into a constant vector where each element is i1 false.
Since revision 266175, SimplifyFCmpInst folds the ordered fcmp into a scalar i1 'false'. Later on, this would cause an assertion failure, since the value type of the folded value doesn't match the expected value type of the uses of the original instruction:
"Assertion failed: New->getType() == getType() && "replaceAllUses of value with new value of different type!".

This patch fixes the issue and adds a test case to the already existing test InstSimplify/floating-point-compares.ll.

Please let me know if okay to commit.

Thanks,
Andrea

[spatel]

LGTM.

Given the number of times we duplicate this line in this function, we could make this parallel the code in SimplifyICmpInst:

Type *RetTy = GetCompareTy(LHS); // at the top of the function
getFalse(RetTy); // everywhere below

[andreadb]

In D24143#532881, @spatel wrote:

LGTM.

Given the number of times we duplicate this line in this function, we could make this parallel the code in SimplifyICmpInst:

Type *RetTy = GetCompareTy(LHS); // at the top of the function
getFalse(RetTy); // everywhere below

Thanks Sanjay!

I agree that we can avoid the code duplication (even for the case where 'true' is returned).
I commit this for now and then do the cleanup on a separate NFC patch.

Cheers,
Andrea

lib/Analysis/InstructionSimplify.cpp

if (RHS->getType()->isVectorTy())
       CFP = dyn_cast_or_null<ConstantFP>(RHSC->getSplatValue());
     else
       CFP = dyn_cast<ConstantFP>(RHSC);
   }
   if (CFP) {
     // If the constant is a nan, see if we can fold the comparison based on it.
     if (CFP->getValueAPF().isNaN()) {
       if (FCmpInst::isOrdered(Pred)) // True "if ordered and foo"
-        return ConstantInt::getFalse(CFP->getContext());
+        return ConstantInt::get(GetCompareTy(LHS), 0);
       assert(FCmpInst::isUnordered(Pred) &&
              "Comparison must be either ordered or unordered!");
       // True if unordered.
       return ConstantInt::get(GetCompareTy(LHS), 1);
     }
     // Check whether the constant is an infinity.
     if (CFP->getValueAPF().isInfinity()) {
       if (CFP->getValueAPF().isNegative()) {

test/Transforms/InstSimplify/floating-point-compare.ll

 
 define i1 @nonans2(double %in1, double %in2) {
 ; CHECK-LABEL: @nonans2(
 ; CHECK:         ret i1 true
 ;
   %cmp = fcmp nnan ord double %in1, %in2
   ret i1 %cmp
 }
+
+define <2 x i1> @orderedCompareWithNaNVector(<2 x double> %A) {
+; CHECK-LABEL: @orderedCompareWithNaNVector(
+; CHECK: ret <2 x i1> zeroinitializer
+  %cmp = fcmp olt <2 x double> %A, <double 0xFFFFFFFFFFFFFFFF, double 0xFFFFFFFFFFFFFFFF>
+  ret <2 x i1> %cmp
+}
+