[TargetLowering] expandFP_TO_UINT - avoid FPE due to out of range conversion (PR17686)

[RKSimon]

Summary

PR17686 demonstrates that for some targets FP exceptions can fire in cases where the FP_TO_UINT is expanded using a FP_TO_SINT instruction.

The existing code converts both the inrange and outofrange cases using FP_TO_SINT and then selects the result, this patch changes this for 'strict' cases to pre-select the FP_TO_SINT input and the offset adjustment.

The X87 cases doesn't need the strict flag but generates much nicer code with it....

[RKSimon]

Rebased on rL345473 which added vector legalization support

[uweigand]

So I had thought that normal LLVM floating-point IR operations are defined to assume that FP exceptions are disabled, and therefore transformations do not need to preserve the FP exception status.

If you rely on FP exceptions, you should instead use the new family of "strict" floating-point intrinsics.

In particular, this very issue came up in the context of correctly lowering the STRICT_FP_TO_UINT operation, where @kpn suggested to lower this directly to an if-then-else sequence at the IR level, before even getting to the DAG. This hasn't gone in the tree yet, but the discussion is ongoing here: https://reviews.llvm.org/D43515

The proposed new pass is here: https://reviews.llvm.org/D43515?id=144714#change-iPGnktlSiYV3

Now, the solution suggested in this patch is also interesting since it can be done without introducing new basic blocks (and therefore can be done in SelectionDAG, without needed a new pass). However, in order to generate efficient code we probably still need to get to the same assembler code we'd have gotten from that new pass -- but now instead the burden is on the back-end to recover the if-then-else block from a series of select statements. Not sure what is better here.

In any case, the current code I see e.g. in the SystemZ tests is definitely worse than what we have today.

[RKSimon]

Would you like me to keep this patch around for STRICT_FP_TO_UINT expansion?

[RKSimon]

Updated to only use the 'strict' FP_TO_UINT path when the TLI shouldUseStrictFP_TO_INT helper is true.

[RKSimon]

Fix signed/unsigned mismatch

[RKSimon]

ping?

[efriedma]

lib/CodeGen/SelectionDAG/TargetLowering.cpp

4187

Do we need to check that Cst is finite? For example, 0x80000000 doesn't fit into a half-precision float. I guess it doesn't matter for x86, but it might matter in the future.

4192

Maybe worth explaining why the extra rounding step here doesn't matter.

[RKSimon]

Added early-out for the case where the integer dst type is larger than the float src type maximum value - we should be able to just use FP_TO_SINT. Thanks to @efriedma for noticing that - it affects a MIPS test.

@atanasyan - should the f16 test should be using a smaller type than the i32 so the full uitofp half path is tested?

[kpn]

In D53794#1278834, @uweigand wrote:

Now, the solution suggested in this patch is also interesting since it can be done without introducing new basic blocks (and therefore can be done in SelectionDAG, without needed a new pass). However, in order to generate efficient code we probably still need to get to the same assembler code we'd have gotten from that new pass -- but now instead the burden is on the back-end to recover the if-then-else block from a series of select statements. Not sure what is better here.

But does this actually solve the issue with FP traps, or does it just mask it? What's to stop llvm from reordering the selects and bringing back the trap?

In any case, the current code I see e.g. in the SystemZ tests is definitely worse than what we have today.

Does this patch prevent llvm from introducing traps? If no, what is the benefit from a change that doesn't eliminate the problem and produces worse code?

[uweigand]

As I understand the approach, it can never introduce a new trap. Now of course, if the original source value is outside the range of the target integer type, some of the instructions introduced by this approach may trap, but in that case, it is expected for the fp_to_uint operation to trap somewhere.

If the original source value is in range however, then none of the instructions introduced here can ever trap:

	​    // Sel = Src < 0x8000000000000000
	​    // Val = select Sel, Src, Src - 0x8000000000000000
	​    // Ofs = select Sel, 0, 0x8000000000000000
	​    // Result = fp_to_sint(Val) ^ Ofs

The first is a compare that never traps. The select operations themselves also never trap. The first select in addition contains a subtraction, but if Src is in range for the fp_to_uint operation, that subtract cannot trap either. Finally, if Src is in range for fp_to_uint, then by construction the value Val will be in range for fp_to_sint, and therefore that operation cannot trap either.

[RKSimon]

Sorry @kpn I missed your post - thanks @uweigand for answering. I should add that I don't know of any combine that would move the select back again and reintroduce this.

@atanasyan The f16 conversion fix should probably be pulled out - do you want me to add 'in overflow range' and 'out of overflow range' fptoui tests to f16-llvm-ir.ll ?

[atanasyan]

In D53794#1302856, @RKSimon wrote:

@atanasyan The f16 conversion fix should probably be pulled out - do you want me to add 'in overflow range' and 'out of overflow range' fptoui tests to f16-llvm-ir.ll ?

Sorry @RKSimon, I missed your initial question. Yes, let's add such fptoui tests to f16-llvm-ir.ll.

[kpn]

In D53794#1302746, @uweigand wrote:

As I understand the approach, it can never introduce a new trap. Now of course, if the original source value is outside the range of the target integer type, some of the instructions introduced by this approach may trap, but in that case, it is expected for the fp_to_uint operation to trap somewhere.

If the original source value is in range however, then none of the instructions introduced here can ever trap:

	​    // Sel = Src < 0x8000000000000000
	​    // Val = select Sel, Src, Src - 0x8000000000000000
	​    // Ofs = select Sel, 0, 0x8000000000000000
	​    // Result = fp_to_sint(Val) ^ Ofs

The first is a compare that never traps. The select operations themselves also never trap. The first select in addition contains a subtraction, but if Src is in range for the fp_to_uint operation, that subtract cannot trap either. Finally, if Src is in range for fp_to_uint, then by construction the value Val will be in range for fp_to_sint, and therefore that operation cannot trap either.

OK, thanks for the clarification. That makes sense.

I like that this isn't a pass and therefore doesn't cost us any compilation time when it isn't needed.

[RKSimon]

Rebased

[RKSimon]

ping?

[efriedma]

lib/CodeGen/SelectionDAG/TargetLowering.cpp

4192

f32/f64 results should then be whole integers

I'm not sure what this means; are you saying the result of the subtraction a whole integer?

[RKSimon]

Cleaned up comment

[efriedma]

LGTM

include/llvm/CodeGen/TargetLowering.h

public:
   /// multiplication-by-constant into simpler operations like shifts and adds.
   /// This may be true if the target does not directly support the
   /// multiplication operation for the specified type or the sequence of simpler
   /// ops is faster than the multiply.
   virtual bool decomposeMulByConstant(EVT VT, SDValue C) const {
     return false;
   }
 
+  /// Return true if it is more correct/profitable to use strict FP_TO_INT
+  /// conversion operations - canonicalizing the FP source value instead of
+  /// converting all cases and then selecting based on value.
+  /// This may be true if the target throws exceptions for out of bounds
+  /// conversions or has fast FP CMOV.
+  virtual bool shouldUseStrictFP_TO_INT(EVT FpVT, EVT IntVT,
+                                        bool IsSigned) const {
+    return false;
+  }
+
   //===--------------------------------------------------------------------===//
   // TargetLowering Configuration Methods - These methods should be invoked by
   // the derived class constructor to configure this object for the target.
   //
 protected:
   /// Specify how the target extends the result of integer and floating point
   /// boolean values from i1 to a wider type.  See getBooleanContents.
   void setBooleanContents(BooleanContent Ty) {

lib/CodeGen/SelectionDAG/TargetLowering.cpp

bool TargetLowering::expandFP_TO_UINT(SDNode *Node, SDValue &Result,
   APFloat APF(APFSem, APInt::getNullValue(SrcVT.getScalarSizeInBits()));
   APInt SignMask = APInt::getSignMask(DstVT.getScalarSizeInBits());
   if (APFloat::opOverflow &
       APF.convertFromAPInt(SignMask, false, APFloat::rmNearestTiesToEven)) {
     Result = DAG.getNode(ISD::FP_TO_SINT, dl, DstVT, Src);
     return true;
   }
 
+  SDValue Cst = DAG.getConstantFP(APF, dl, SrcVT);
+  SDValue Sel = DAG.getSetCC(dl, SetCCVT, Src, Cst, ISD::SETLT);
+
+  bool Strict = shouldUseStrictFP_TO_INT(SrcVT, DstVT, false);
+  if (Strict) {
+    // Expand based on maximum range of FP_TO_SINT, if the value exceeds the
+    // signmask then offset (f32/f64 results should then be whole integers).
+    // Sel = Src < 0x8000000000000000
+    // Val = select Sel, Src, Src - 0x8000000000000000
+    // Ofs = select Sel, 0, 0x8000000000000000
+    // Result = fp_to_sint(Val) ^ Ofs
+
+    // TODO: Should any fast-math-flags be set for the FSUB?
+    SDValue Val = DAG.getSelect(dl, SrcVT, Sel, Src,
+                                DAG.getNode(ISD::FSUB, dl, SrcVT, Src, Cst));
+    SDValue Ofs = DAG.getSelect(dl, DstVT, Sel, DAG.getConstant(0, dl, DstVT),
+                                DAG.getConstant(SignMask, dl, DstVT));
+    Result = DAG.getNode(ISD::XOR, dl, DstVT,
+                         DAG.getNode(ISD::FP_TO_SINT, dl, DstVT, Val), Ofs);
+  } else {
     // Expand based on maximum range of FP_TO_SINT:
     // True = fp_to_sint(Src)
     // False = 0x8000000000000000 + fp_to_sint(Src - 0x8000000000000000)
     // Result = select (Src < 0x8000000000000000), True, False
-  SDValue Cst = DAG.getConstantFP(APF, dl, SrcVT);
-  SDValue Sel = DAG.getSetCC(dl, SetCCVT, Src, Cst, ISD::SETLT);
 
     SDValue True = DAG.getNode(ISD::FP_TO_SINT, dl, DstVT, Src);
     // TODO: Should any fast-math-flags be set for the FSUB?
     SDValue False = DAG.getNode(ISD::FP_TO_SINT, dl, DstVT,
                                 DAG.getNode(ISD::FSUB, dl, SrcVT, Src, Cst));
     False = DAG.getNode(ISD::XOR, dl, DstVT, False,
                         DAG.getConstant(SignMask, dl, DstVT));
     Result = DAG.getSelect(dl, DstVT, Sel, True, False);
+  }
   return true;
 }
 
 bool TargetLowering::expandUINT_TO_FP(SDNode *Node, SDValue &Result,
                                       SelectionDAG &DAG) const {
   SDValue Src = Node->getOperand(0);
   EVT SrcVT = Src.getValueType();
   EVT DstVT = Node->getValueType(0);

lib/Target/X86/X86ISelLowering.h

public:
     /// to just the constant itself.
     bool shouldConvertConstantLoadToIntImm(const APInt &Imm,
                                            Type *Ty) const override;
 
     bool convertSelectOfConstantsToMath(EVT VT) const override;
 
     bool decomposeMulByConstant(EVT VT, SDValue C) const override;
 
+    bool shouldUseStrictFP_TO_INT(EVT FpVT, EVT IntVT,
+                                  bool IsSigned) const override;
+
     /// Return true if EXTRACT_SUBVECTOR is cheap for this result type
     /// with this index.
     bool isExtractSubvectorCheap(EVT ResVT, EVT SrcVT,
                                  unsigned Index) const override;
 
     bool storeOfVectorConstantIsCheap(EVT MemVT, unsigned NumElem,
                                       unsigned AddrSpace) const override {
       // If we can replace more than 2 scalar stores, there will be a reduction

lib/Target/X86/X86ISelLowering.cpp

bool X86TargetLowering::decomposeMulByConstant(EVT VT, SDValue C) const {
   if (isOperationLegal(ISD::MUL, VT))
     return false;
 
   // shl+add, shl+sub, shl+add+neg
   return (MulC + 1).isPowerOf2() || (MulC - 1).isPowerOf2() ||
          (1 - MulC).isPowerOf2() || (-(MulC + 1)).isPowerOf2();
 }
 
+bool X86TargetLowering::shouldUseStrictFP_TO_INT(EVT FpVT, EVT IntVT,
+                                                 bool IsSigned) const {
+  // f80 UINT_TO_FP is more efficient using Strict code if FCMOV is available.
+  return !IsSigned && FpVT == MVT::f80 && Subtarget.hasCMov();
+}
+
 bool X86TargetLowering::isExtractSubvectorCheap(EVT ResVT, EVT SrcVT,
                                                 unsigned Index) const {
   if (!isOperationLegalOrCustom(ISD::EXTRACT_SUBVECTOR, ResVT))
     return false;
 
   // Mask vectors support all subregister combinations and operations that
   // extract half of vector.
   if (ResVT.getVectorElementType() == MVT::i1)

test/CodeGen/X86/fp-cvt.ll

 ; X86-NEXT:    retl
 ;
 ; X64-X87-LABEL: fptoui_i64_fp80:
 ; X64-X87:       # %bb.0:
 ; X64-X87-NEXT:    fldt {{[0-9]+}}(%rsp)
 ; X64-X87-NEXT:    flds {{.*}}(%rip)
 ; X64-X87-NEXT:    fld %st(1)
 ; X64-X87-NEXT:    fsub %st(1)
+; X64-X87-NEXT:    xorl %eax, %eax
+; X64-X87-NEXT:    fxch %st(1)
+; X64-X87-NEXT:    fucompi %st(2)
+; X64-X87-NEXT:    fcmovnbe %st(1), %st(0)
+; X64-X87-NEXT:    fstp %st(1)
 ; X64-X87-NEXT:    fnstcw -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    movzwl -{{[0-9]+}}(%rsp), %eax
-; X64-X87-NEXT:    movw $3199, -{{[0-9]+}}(%rsp) # imm = 0xC7F
-; X64-X87-NEXT:    fldcw -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    movw %ax, -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    fistpll -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    fldcw -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    fnstcw -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    movzwl -{{[0-9]+}}(%rsp), %eax
+; X64-X87-NEXT:    movzwl -{{[0-9]+}}(%rsp), %ecx
 ; X64-X87-NEXT:    movw $3199, -{{[0-9]+}}(%rsp) # imm = 0xC7F
 ; X64-X87-NEXT:    fldcw -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    movw %ax, -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    fld %st(1)
+; X64-X87-NEXT:    movw %cx, -{{[0-9]+}}(%rsp)
 ; X64-X87-NEXT:    fistpll -{{[0-9]+}}(%rsp)
 ; X64-X87-NEXT:    fldcw -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    fucompi %st(1)
-; X64-X87-NEXT:    fstp %st(0)
-; X64-X87-NEXT:    jbe .LBB10_1
-; X64-X87-NEXT:  # %bb.2:
-; X64-X87-NEXT:    movq -{{[0-9]+}}(%rsp), %rax
-; X64-X87-NEXT:    retq
-; X64-X87-NEXT:  .LBB10_1:
-; X64-X87-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
+; X64-X87-NEXT:    setbe %al
+; X64-X87-NEXT:    shlq $63, %rax
 ; X64-X87-NEXT:    xorq -{{[0-9]+}}(%rsp), %rax
 ; X64-X87-NEXT:    retq
 ;
 ; X64-SSSE3-LABEL: fptoui_i64_fp80:
 ; X64-SSSE3:       # %bb.0:
 ; X64-SSSE3-NEXT:    fldt {{[0-9]+}}(%rsp)
 ; X64-SSSE3-NEXT:    flds {{.*}}(%rip)
 ; X64-SSSE3-NEXT:    fld %st(1)
 ; X64-SSSE3-NEXT:    fsub %st(1)
+; X64-SSSE3-NEXT:    xorl %eax, %eax
+; X64-SSSE3-NEXT:    fxch %st(1)
+; X64-SSSE3-NEXT:    fucompi %st(2)
+; X64-SSSE3-NEXT:    fcmovnbe %st(1), %st(0)
+; X64-SSSE3-NEXT:    fstp %st(1)
 ; X64-SSSE3-NEXT:    fisttpll -{{[0-9]+}}(%rsp)
-; X64-SSSE3-NEXT:    fld %st(1)
-; X64-SSSE3-NEXT:    fisttpll -{{[0-9]+}}(%rsp)
-; X64-SSSE3-NEXT:    fucompi %st(1)
-; X64-SSSE3-NEXT:    fstp %st(0)
-; X64-SSSE3-NEXT:    jbe .LBB10_1
-; X64-SSSE3-NEXT:  # %bb.2:
-; X64-SSSE3-NEXT:    movq -{{[0-9]+}}(%rsp), %rax
-; X64-SSSE3-NEXT:    retq
-; X64-SSSE3-NEXT:  .LBB10_1:
-; X64-SSSE3-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
+; X64-SSSE3-NEXT:    setbe %al
+; X64-SSSE3-NEXT:    shlq $63, %rax
 ; X64-SSSE3-NEXT:    xorq -{{[0-9]+}}(%rsp), %rax
 ; X64-SSSE3-NEXT:    retq
   %1 = fptoui x86_fp80 %a0 to i64
   ret i64  %1
 }
 
 define i64 @fptoui_i64_fp80_ld(x86_fp80 *%a0) nounwind {
 ; X86-LABEL: fptoui_i64_fp80_ld:

 ; X86-NEXT:    retl
 ;
 ; X64-X87-LABEL: fptoui_i64_fp80_ld:
 ; X64-X87:       # %bb.0:
 ; X64-X87-NEXT:    fldt (%rdi)
 ; X64-X87-NEXT:    flds {{.*}}(%rip)
 ; X64-X87-NEXT:    fld %st(1)
 ; X64-X87-NEXT:    fsub %st(1)
+; X64-X87-NEXT:    xorl %eax, %eax
+; X64-X87-NEXT:    fxch %st(1)
+; X64-X87-NEXT:    fucompi %st(2)
+; X64-X87-NEXT:    fcmovnbe %st(1), %st(0)
+; X64-X87-NEXT:    fstp %st(1)
 ; X64-X87-NEXT:    fnstcw -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    movzwl -{{[0-9]+}}(%rsp), %eax
-; X64-X87-NEXT:    movw $3199, -{{[0-9]+}}(%rsp) # imm = 0xC7F
-; X64-X87-NEXT:    fldcw -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    movw %ax, -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    fistpll -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    fldcw -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    fnstcw -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    movzwl -{{[0-9]+}}(%rsp), %eax
+; X64-X87-NEXT:    movzwl -{{[0-9]+}}(%rsp), %ecx
 ; X64-X87-NEXT:    movw $3199, -{{[0-9]+}}(%rsp) # imm = 0xC7F
 ; X64-X87-NEXT:    fldcw -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    movw %ax, -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    fld %st(1)
+; X64-X87-NEXT:    movw %cx, -{{[0-9]+}}(%rsp)
 ; X64-X87-NEXT:    fistpll -{{[0-9]+}}(%rsp)
 ; X64-X87-NEXT:    fldcw -{{[0-9]+}}(%rsp)
-; X64-X87-NEXT:    fucompi %st(1)
-; X64-X87-NEXT:    fstp %st(0)
-; X64-X87-NEXT:    jbe .LBB11_1
-; X64-X87-NEXT:  # %bb.2:
-; X64-X87-NEXT:    movq -{{[0-9]+}}(%rsp), %rax
-; X64-X87-NEXT:    retq
-; X64-X87-NEXT:  .LBB11_1:
-; X64-X87-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
+; X64-X87-NEXT:    setbe %al
+; X64-X87-NEXT:    shlq $63, %rax
 ; X64-X87-NEXT:    xorq -{{[0-9]+}}(%rsp), %rax
 ; X64-X87-NEXT:    retq
 ;
 ; X64-SSSE3-LABEL: fptoui_i64_fp80_ld:
 ; X64-SSSE3:       # %bb.0:
 ; X64-SSSE3-NEXT:    fldt (%rdi)
 ; X64-SSSE3-NEXT:    flds {{.*}}(%rip)
 ; X64-SSSE3-NEXT:    fld %st(1)
 ; X64-SSSE3-NEXT:    fsub %st(1)
+; X64-SSSE3-NEXT:    xorl %eax, %eax
+; X64-SSSE3-NEXT:    fxch %st(1)
+; X64-SSSE3-NEXT:    fucompi %st(2)
+; X64-SSSE3-NEXT:    fcmovnbe %st(1), %st(0)
+; X64-SSSE3-NEXT:    fstp %st(1)
 ; X64-SSSE3-NEXT:    fisttpll -{{[0-9]+}}(%rsp)
-; X64-SSSE3-NEXT:    fld %st(1)
-; X64-SSSE3-NEXT:    fisttpll -{{[0-9]+}}(%rsp)
-; X64-SSSE3-NEXT:    fucompi %st(1)
-; X64-SSSE3-NEXT:    fstp %st(0)
-; X64-SSSE3-NEXT:    jbe .LBB11_1
-; X64-SSSE3-NEXT:  # %bb.2:
-; X64-SSSE3-NEXT:    movq -{{[0-9]+}}(%rsp), %rax
-; X64-SSSE3-NEXT:    retq
-; X64-SSSE3-NEXT:  .LBB11_1:
-; X64-SSSE3-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
+; X64-SSSE3-NEXT:    setbe %al
+; X64-SSSE3-NEXT:    shlq $63, %rax
 ; X64-SSSE3-NEXT:    xorq -{{[0-9]+}}(%rsp), %rax
 ; X64-SSSE3-NEXT:    retq
   %1 = load x86_fp80, x86_fp80 *%a0
   %2 = fptoui x86_fp80 %1 to i64
   ret i64  %2
 }
 
 ;

test/CodeGen/X86/scalar-fp-to-i64.ll

 ; SSE3_32_LIN-NEXT:    shll $31, %edx
 ; SSE3_32_LIN-NEXT:    xorl {{[0-9]+}}(%esp), %edx
 ; SSE3_32_LIN-NEXT:    movl (%esp), %eax
 ; SSE3_32_LIN-NEXT:    addl $12, %esp
 ; SSE3_32_LIN-NEXT:    retl
 ;
 ; SSE3_64_WIN-LABEL: x_to_u64:
 ; SSE3_64_WIN:       # %bb.0:
-; SSE3_64_WIN-NEXT:    subq $16, %rsp
+; SSE3_64_WIN-NEXT:    pushq %rax
 ; SSE3_64_WIN-NEXT:    fldt (%rcx)
 ; SSE3_64_WIN-NEXT:    flds __real@{{.*}}(%rip)
 ; SSE3_64_WIN-NEXT:    fld %st(1)
 ; SSE3_64_WIN-NEXT:    fsub %st(1)
-; SSE3_64_WIN-NEXT:    fisttpll {{[0-9]+}}(%rsp)
-; SSE3_64_WIN-NEXT:    fld %st(1)
+; SSE3_64_WIN-NEXT:    xorl %eax, %eax
+; SSE3_64_WIN-NEXT:    fxch %st(1)
+; SSE3_64_WIN-NEXT:    fucompi %st(2)
+; SSE3_64_WIN-NEXT:    fcmovnbe %st(1), %st(0)
+; SSE3_64_WIN-NEXT:    fstp %st(1)
 ; SSE3_64_WIN-NEXT:    fisttpll (%rsp)
-; SSE3_64_WIN-NEXT:    fucompi %st(1)
-; SSE3_64_WIN-NEXT:    fstp %st(0)
-; SSE3_64_WIN-NEXT:    jbe .LBB4_1
-; SSE3_64_WIN-NEXT:  # %bb.2:
-; SSE3_64_WIN-NEXT:    movq (%rsp), %rax
-; SSE3_64_WIN-NEXT:    addq $16, %rsp
-; SSE3_64_WIN-NEXT:    retq
-; SSE3_64_WIN-NEXT:  .LBB4_1:
-; SSE3_64_WIN-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
-; SSE3_64_WIN-NEXT:    xorq {{[0-9]+}}(%rsp), %rax
-; SSE3_64_WIN-NEXT:    addq $16, %rsp
+; SSE3_64_WIN-NEXT:    setbe %al
+; SSE3_64_WIN-NEXT:    shlq $63, %rax
+; SSE3_64_WIN-NEXT:    xorq (%rsp), %rax
+; SSE3_64_WIN-NEXT:    popq %rcx
 ; SSE3_64_WIN-NEXT:    retq
 ;
 ; SSE3_64_LIN-LABEL: x_to_u64:
 ; SSE3_64_LIN:       # %bb.0:
 ; SSE3_64_LIN-NEXT:    fldt {{[0-9]+}}(%rsp)
 ; SSE3_64_LIN-NEXT:    flds {{.*}}(%rip)
 ; SSE3_64_LIN-NEXT:    fld %st(1)
 ; SSE3_64_LIN-NEXT:    fsub %st(1)
+; SSE3_64_LIN-NEXT:    xorl %eax, %eax
+; SSE3_64_LIN-NEXT:    fxch %st(1)
+; SSE3_64_LIN-NEXT:    fucompi %st(2)
+; SSE3_64_LIN-NEXT:    fcmovnbe %st(1), %st(0)
+; SSE3_64_LIN-NEXT:    fstp %st(1)
 ; SSE3_64_LIN-NEXT:    fisttpll -{{[0-9]+}}(%rsp)
-; SSE3_64_LIN-NEXT:    fld %st(1)
-; SSE3_64_LIN-NEXT:    fisttpll -{{[0-9]+}}(%rsp)
-; SSE3_64_LIN-NEXT:    fucompi %st(1)
-; SSE3_64_LIN-NEXT:    fstp %st(0)
-; SSE3_64_LIN-NEXT:    jbe .LBB4_1
-; SSE3_64_LIN-NEXT:  # %bb.2:
-; SSE3_64_LIN-NEXT:    movq -{{[0-9]+}}(%rsp), %rax
-; SSE3_64_LIN-NEXT:    retq
-; SSE3_64_LIN-NEXT:  .LBB4_1:
-; SSE3_64_LIN-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
+; SSE3_64_LIN-NEXT:    setbe %al
+; SSE3_64_LIN-NEXT:    shlq $63, %rax
 ; SSE3_64_LIN-NEXT:    xorq -{{[0-9]+}}(%rsp), %rax
 ; SSE3_64_LIN-NEXT:    retq
 ;
 ; SSE2_32_WIN-LABEL: x_to_u64:
 ; SSE2_32_WIN:       # %bb.0:
 ; SSE2_32_WIN-NEXT:    pushl %ebp
 ; SSE2_32_WIN-NEXT:    movl %esp, %ebp
 ; SSE2_32_WIN-NEXT:    andl $-8, %esp

 ; SSE2_32_LIN-NEXT:    shll $31, %edx
 ; SSE2_32_LIN-NEXT:    xorl {{[0-9]+}}(%esp), %edx
 ; SSE2_32_LIN-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; SSE2_32_LIN-NEXT:    addl $20, %esp
 ; SSE2_32_LIN-NEXT:    retl
 ;
 ; SSE2_64_WIN-LABEL: x_to_u64:
 ; SSE2_64_WIN:       # %bb.0:
-; SSE2_64_WIN-NEXT:    subq $24, %rsp
+; SSE2_64_WIN-NEXT:    subq $16, %rsp
 ; SSE2_64_WIN-NEXT:    fldt (%rcx)
 ; SSE2_64_WIN-NEXT:    flds __real@{{.*}}(%rip)
 ; SSE2_64_WIN-NEXT:    fld %st(1)
 ; SSE2_64_WIN-NEXT:    fsub %st(1)
+; SSE2_64_WIN-NEXT:    xorl %eax, %eax
+; SSE2_64_WIN-NEXT:    fxch %st(1)
+; SSE2_64_WIN-NEXT:    fucompi %st(2)
+; SSE2_64_WIN-NEXT:    fcmovnbe %st(1), %st(0)
+; SSE2_64_WIN-NEXT:    fstp %st(1)
 ; SSE2_64_WIN-NEXT:    fnstcw {{[0-9]+}}(%rsp)
-; SSE2_64_WIN-NEXT:    movzwl {{[0-9]+}}(%rsp), %eax
+; SSE2_64_WIN-NEXT:    movzwl {{[0-9]+}}(%rsp), %ecx
 ; SSE2_64_WIN-NEXT:    movw $3199, {{[0-9]+}}(%rsp) # imm = 0xC7F
 ; SSE2_64_WIN-NEXT:    fldcw {{[0-9]+}}(%rsp)
-; SSE2_64_WIN-NEXT:    movw %ax, {{[0-9]+}}(%rsp)
+; SSE2_64_WIN-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
 ; SSE2_64_WIN-NEXT:    fistpll {{[0-9]+}}(%rsp)
 ; SSE2_64_WIN-NEXT:    fldcw {{[0-9]+}}(%rsp)
-; SSE2_64_WIN-NEXT:    fnstcw {{[0-9]+}}(%rsp)
-; SSE2_64_WIN-NEXT:    movzwl {{[0-9]+}}(%rsp), %eax
-; SSE2_64_WIN-NEXT:    movw $3199, {{[0-9]+}}(%rsp) # imm = 0xC7F
-; SSE2_64_WIN-NEXT:    fldcw {{[0-9]+}}(%rsp)
-; SSE2_64_WIN-NEXT:    movw %ax, {{[0-9]+}}(%rsp)
-; SSE2_64_WIN-NEXT:    fld %st(1)
-; SSE2_64_WIN-NEXT:    fistpll {{[0-9]+}}(%rsp)
-; SSE2_64_WIN-NEXT:    fldcw {{[0-9]+}}(%rsp)
-; SSE2_64_WIN-NEXT:    fucompi %st(1)
-; SSE2_64_WIN-NEXT:    fstp %st(0)
-; SSE2_64_WIN-NEXT:    jbe .LBB4_1
-; SSE2_64_WIN-NEXT:  # %bb.2:
-; SSE2_64_WIN-NEXT:    movq {{[0-9]+}}(%rsp), %rax
-; SSE2_64_WIN-NEXT:    addq $24, %rsp
-; SSE2_64_WIN-NEXT:    retq
-; SSE2_64_WIN-NEXT:  .LBB4_1:
-; SSE2_64_WIN-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
+; SSE2_64_WIN-NEXT:    setbe %al
+; SSE2_64_WIN-NEXT:    shlq $63, %rax
 ; SSE2_64_WIN-NEXT:    xorq {{[0-9]+}}(%rsp), %rax
-; SSE2_64_WIN-NEXT:    addq $24, %rsp
+; SSE2_64_WIN-NEXT:    addq $16, %rsp
 ; SSE2_64_WIN-NEXT:    retq
 ;
 ; SSE2_64_LIN-LABEL: x_to_u64:
 ; SSE2_64_LIN:       # %bb.0:
 ; SSE2_64_LIN-NEXT:    fldt {{[0-9]+}}(%rsp)
 ; SSE2_64_LIN-NEXT:    flds {{.*}}(%rip)
 ; SSE2_64_LIN-NEXT:    fld %st(1)
 ; SSE2_64_LIN-NEXT:    fsub %st(1)
+; SSE2_64_LIN-NEXT:    xorl %eax, %eax
+; SSE2_64_LIN-NEXT:    fxch %st(1)
+; SSE2_64_LIN-NEXT:    fucompi %st(2)
+; SSE2_64_LIN-NEXT:    fcmovnbe %st(1), %st(0)
+; SSE2_64_LIN-NEXT:    fstp %st(1)
 ; SSE2_64_LIN-NEXT:    fnstcw -{{[0-9]+}}(%rsp)
-; SSE2_64_LIN-NEXT:    movzwl -{{[0-9]+}}(%rsp), %eax
+; SSE2_64_LIN-NEXT:    movzwl -{{[0-9]+}}(%rsp), %ecx
 ; SSE2_64_LIN-NEXT:    movw $3199, -{{[0-9]+}}(%rsp) # imm = 0xC7F
 ; SSE2_64_LIN-NEXT:    fldcw -{{[0-9]+}}(%rsp)
-; SSE2_64_LIN-NEXT:    movw %ax, -{{[0-9]+}}(%rsp)
+; SSE2_64_LIN-NEXT:    movw %cx, -{{[0-9]+}}(%rsp)
 ; SSE2_64_LIN-NEXT:    fistpll -{{[0-9]+}}(%rsp)
 ; SSE2_64_LIN-NEXT:    fldcw -{{[0-9]+}}(%rsp)
-; SSE2_64_LIN-NEXT:    fnstcw -{{[0-9]+}}(%rsp)
-; SSE2_64_LIN-NEXT:    movzwl -{{[0-9]+}}(%rsp), %eax
-; SSE2_64_LIN-NEXT:    movw $3199, -{{[0-9]+}}(%rsp) # imm = 0xC7F
-; SSE2_64_LIN-NEXT:    fldcw -{{[0-9]+}}(%rsp)
-; SSE2_64_LIN-NEXT:    movw %ax, -{{[0-9]+}}(%rsp)
-; SSE2_64_LIN-NEXT:    fld %st(1)
-; SSE2_64_LIN-NEXT:    fistpll -{{[0-9]+}}(%rsp)
-; SSE2_64_LIN-NEXT:    fldcw -{{[0-9]+}}(%rsp)
-; SSE2_64_LIN-NEXT:    fucompi %st(1)
-; SSE2_64_LIN-NEXT:    fstp %st(0)
-; SSE2_64_LIN-NEXT:    jbe .LBB4_1
-; SSE2_64_LIN-NEXT:  # %bb.2:
-; SSE2_64_LIN-NEXT:    movq -{{[0-9]+}}(%rsp), %rax
-; SSE2_64_LIN-NEXT:    retq
-; SSE2_64_LIN-NEXT:  .LBB4_1:
-; SSE2_64_LIN-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
+; SSE2_64_LIN-NEXT:    setbe %al
+; SSE2_64_LIN-NEXT:    shlq $63, %rax
 ; SSE2_64_LIN-NEXT:    xorq -{{[0-9]+}}(%rsp), %rax
 ; SSE2_64_LIN-NEXT:    retq
 ;
 ; X87_WIN-LABEL: x_to_u64:
 ; X87_WIN:       # %bb.0:
 ; X87_WIN-NEXT:    pushl %ebp
 ; X87_WIN-NEXT:    movl %esp, %ebp
 ; X87_WIN-NEXT:    andl $-8, %esp