[builtins] Fix signed integer overflows in fp_fixint_impl.inc

[Ka-Ka]

Summary

When compiling the builtins with the undefined behavior sanitizer and running
testcases you end up with the following warning:

UBSan: fp_fixint_impl.inc:39:42: left shift of 8388608 by 40 places cannot be represented in type 'fixint_t' (aka 'long long')
UBSan: fp_fixint_impl.inc:39:17: signed integer overflow: -1 * -9223372036854775808 cannot be represented in type 'fixint_t' (aka 'long long')

This can be avoided by doing the shift and the multiplication in a matching
unsigned variant of the type.

This was found in an out of tree target.

[Ka-Ka]

compiler-rt/test/builtins/Unit/fixsfdi_test.c

73–76

The added test pass when compiling with gcc -lgcc.

[MaskRay]

compiler-rt/lib/builtins/fp_fixint_impl.inc

39

Looks ok.

compiler-rt/test/builtins/Unit/fixsfdi_test.c

73–76

This checks the __fixunssfdi code path, not fp_fixint_impl.inc.

[Ka-Ka]

compiler-rt/test/builtins/Unit/fixsfdi_test.c

73–76

It turn out that the added test only trigger the intended case if compiler-rt is compiled with -D__SOFTFP__ (which we do in our out of tree target). Maybe it is impossible to trigger the signed overflow in fp_fixint_impl.inc in a in tree target (ARM seems to have a feature for softfp but probably not enabled when compiling compiler-rt?).
Should I simply remove the added test from the patch as it don't trigger the intended case?

[MaskRay]

compiler-rt/test/builtins/Unit/fixsfdi_test.c

73–76

should be fine to keep it...

[Ka-Ka]

compiler-rt/test/builtins/Unit/fixsfdi_test.c

73–76

Sure, then I keep it. I will write something in the commit message about it.

Thanks for reviewing.

compiler-rt/lib/builtins/fp_fixint_impl.inc

static __inline fixint_t __fixint(fp_t a) {
   if ((unsigned)exponent >= sizeof(fixint_t) * CHAR_BIT)
     return sign == 1 ? fixint_max : fixint_min;
 
   // If 0 <= exponent < significandBits, right shift to get the result.
   // Otherwise, shift left.
   if (exponent < significandBits)
     return sign * (significand >> (significandBits - exponent));
   else
-    return sign * ((fixint_t)significand << (exponent - significandBits));
+    return sign * ((fixuint_t)significand << (exponent - significandBits));
 }

compiler-rt/test/builtins/Unit/fixsfdi_test.c

int main()
     if (test__fixsfdi(0x1.FFFFFCp+62F, 0x7FFFFF0000000000LL))
         return 1;
 
     if (test__fixsfdi(-0x1.FFFFFEp+62F, 0x8000008000000000LL))
         return 1;
     if (test__fixsfdi(-0x1.FFFFFCp+62F, 0x8000010000000000LL))
         return 1;
 
+    if (test__fixsfdi(-0x8000000000000000.0p+0F, 0x8000000000000000LL))
+        return 1;
+
     return 0;
 }