[builtins] Rounding mode support for addxf3/subxf3

[kongyi]

Summary

Implement rounding mode support for addxf3/subxf3.

On architectures that implemented the support, this will access the corresponding floating point environment register to apply the correct rounding. For other architectures, it will keep the current behaviour and use IEEE-754 default rounding mode (to nearest, ties to even).

ARM32/AArch64 support implemented in this change. i386 and AMD64 will be added in a follow up change.

[efriedma]

compiler-rt/lib/builtins/arm/fp_mode.c

25 ↗

(On Diff #183262)

This needs to be guarded; vmrs is only valid on targets which support VFP. Not sure if we need to try to check that at runtime...

[MaskRay]

compiler-rt/lib/builtins/arm/fp_mode.c

25 ↗

(On Diff #183262)

On ARM, perhaps we can assume only FE_TONEAREST is available.

https://github.com/gcc-mirror/gcc/blob/master/libgcc/config/arm/ieee754-sf.S

Only the default rounding mode is intended for best performances.

[MaskRay]

compiler-rt/lib/builtins/arm/fp_mode.c

25 ↗

(On Diff #183262)

For aarch64, implementing __fegetround in assembly also looks good to me. In musl http://git.musl-libc.org/cgit/musl/tree/src/fenv/aarch64/fenv.s

.global fegetround
.type fegetround,%function
fegetround:
	mrs x0, fpcr
	and w0, w0, #0xc00000
	ret

[joerg]

This seems to imply hard-float support though? Accessing floating point registers when using soft-float is not going to get very far.

[kongyi]

compiler-rt/lib/builtins/arm/fp_mode.c

25 ↗

(On Diff #183262)

Added check for __ARM_FP. Fallback to default rounding if VFP is not available.

[efriedma]

compiler-rt/lib/builtins/arm/fp_mode.c

25 ↗

(On Diff #183262)

I'm a little concerned this will do something unexpected on soft-float ARM Linux (which is usually built for a target without VFP support, but often runs on a chip that actually supports it). I guess always disabling other rounding modes is more intuitive than enabling them conditionally, though; should be fine.

[peter.smith]

The mix of soft-floating point and hard-floating point (with soft-float interface) can happen on embedded systems code as well. The solution Arm's proprietary toolchain had to this problem is for the linker to select libraries based on build-attibutes. It would detect the rounding mode build attribute and select the appropriate library variant, but that isn't available to us here. A colleague suggested an easily available customisation point that someone that had a local need for a different rounding mode could implement.

compiler-rt/lib/builtins/arm/fp_mode.c

25 ↗

(On Diff #183262)

In this situation I'm not sure we can do much better in generic compiler-rt. If we thought it would be necessary to allow change from FE_TONEAREST one possibility would be to replace the return FE_TONEAREST in

#else
  return FE_TONEAREST;
#endif

With a call to a weak definition of another function returning FE_TONEAREST . A user that cared sufficiently about the rounding mode could then reimplement this function with the rounding mode of their choice. This would be a bit easier than adding definitions of both fe_raise_inexact and fe_getround but not much.

[kongyi]

ping...

[craig.topper]

Can you upload the patch with full context? https://llvm.org/docs/Phabricator.html#requesting-a-review-via-the-web-interface

[peter.smith]

In D57143#1378752, @peter.smith wrote:

The mix of soft-floating point and hard-floating point (with soft-float interface) can happen on embedded systems code as well. The solution Arm's proprietary toolchain had to this problem is for the linker to select libraries based on build-attibutes. It would detect the rounding mode build attribute and select the appropriate library variant, but that isn't available to us here. A colleague suggested an easily available customisation point that someone that had a local need for a different rounding mode could implement.

I'm happy with Yi's latest update. Do we have anyone left with any objections? If there aren't any I'm inclined to approve.

[efriedma]

My comments have been addressed, I think. Looks fine.

compiler-rt/trunk/lib/builtins/CMakeLists.txt

set(GENERIC_SOURCES
   floattidf.c
   floattisf.c
   floatundidf.c
   floatundisf.c
   floatunsidf.c
   floatunsisf.c
   floatuntidf.c
   floatuntisf.c
+  fp_mode.c
   int_util.c
   lshrdi3.c
   lshrti3.c
   moddi3.c
   modsi3.c
   modti3.c
   muldc3.c
   muldf3.c

 endif () # if (NOT MSVC)
 
 set(x86_64h_SOURCES ${x86_64h_SOURCES} ${x86_ARCH_SOURCES})
 set(x86_64_SOURCES ${x86_64_SOURCES} ${x86_ARCH_SOURCES})
 set(i386_SOURCES ${i386_SOURCES} ${x86_ARCH_SOURCES})
 set(i686_SOURCES ${i686_SOURCES} ${x86_ARCH_SOURCES})
 
 set(arm_SOURCES
+  arm/fp_mode.c
   arm/bswapdi2.S
   arm/bswapsi2.S
   arm/clzdi2.S
   arm/clzsi2.S
   arm/comparesf2.S
   arm/divmodsi4.S
   arm/divsi3.S
   arm/modsi3.S
elseif(NOT WIN32)
 
   set(thumb1_SOURCES
     ${thumb1_SOURCES}
     ${arm_EABI_SOURCES})
 endif()
 
 set(aarch64_SOURCES
   ${GENERIC_TF_SOURCES}
-  ${GENERIC_SOURCES})
+  ${GENERIC_SOURCES}
+  aarch64/fp_mode.c)
 
 if (MINGW)
   set(aarch64_SOURCES
       ${aarch64_SOURCES}
       aarch64/chkstk.S)
 endif()
 
 set(armhf_SOURCES ${arm_SOURCES})

compiler-rt/trunk/lib/builtins/aarch64/fp_mode.c

+//===----- lib/aarch64/fp_mode.c - Floaing-point mode utilities ---*- C -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include <stdint.h>
+
+#include "../fp_mode.h"
+
+#define AARCH64_TONEAREST  0x0
+#define AARCH64_UPWARD     0x1
+#define AARCH64_DOWNWARD   0x2
+#define AARCH64_TOWARDZERO 0x3
+#define AARCH64_RMODE_MASK (AARCH64_TONEAREST | AARCH64_UPWARD | \
+                            AARCH64_DOWNWARD | AARCH64_TOWARDZERO)
+#define AARCH64_RMODE_SHIFT 22
+
+#define AARCH64_INEXACT     0x10
+
+#ifndef __ARM_FP
+// For soft float targets, allow changing rounding mode by overriding the weak
+// __aarch64_fe_default_rmode symbol.
+FE_ROUND_MODE __attribute__((weak)) __aarch64_fe_default_rmode = FE_TONEAREST;
+#endif
+
+FE_ROUND_MODE __fe_getround() {
+#ifdef __ARM_FP
+  uint64_t fpcr;
+  __asm__ __volatile__("mrs  %0, fpcr" : "=r" (fpcr));
+  fpcr = fpcr >> AARCH64_RMODE_SHIFT & AARCH64_RMODE_MASK;
+  switch (fpcr) {
+    case AARCH64_UPWARD:
+      return FE_UPWARD;
+    case AARCH64_DOWNWARD:
+      return FE_DOWNWARD;
+    case AARCH64_TOWARDZERO:
+      return FE_TOWARDZERO;
+    case AARCH64_TONEAREST:
+    default:
+      return FE_TONEAREST;
+  }
+#else
+  return __aarch64_fe_default_rmode;
+#endif
+}
+
+int __fe_raise_inexact() {
+#ifdef __ARM_FP
+  uint64_t fpsr;
+  __asm__ __volatile__("mrs  %0, fpsr" : "=r" (fpsr));
+  __asm__ __volatile__("msr  fpsr, %0" : : "ri" (fpsr | AARCH64_INEXACT));
+  return 0;
+#else
+  return 0;
+#endif
+}

compiler-rt/trunk/lib/builtins/adddf3.c

 //===-- lib/adddf3.c - Double-precision addition ------------------*- C -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// This file implements double-precision soft-float addition with the IEEE-754
-// default rounding (to nearest, ties to even).
+// This file implements double-precision soft-float addition.
 //
 //===----------------------------------------------------------------------===//
 
 #define DOUBLE_PRECISION
 #include "fp_add_impl.inc"
 
 COMPILER_RT_ABI double __adddf3(double a, double b){
     return __addXf3__(a, b);

compiler-rt/trunk/lib/builtins/addsf3.c

 //===-- lib/addsf3.c - Single-precision addition ------------------*- C -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// This file implements single-precision soft-float addition with the IEEE-754
-// default rounding (to nearest, ties to even).
+// This file implements single-precision soft-float addition.
 //
 //===----------------------------------------------------------------------===//
 
 #define SINGLE_PRECISION
 #include "fp_add_impl.inc"
 
 COMPILER_RT_ABI float __addsf3(float a, float b) {
     return __addXf3__(a, b);

compiler-rt/trunk/lib/builtins/addtf3.c

 //===-- lib/addtf3.c - Quad-precision addition --------------------*- C -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// This file implements quad-precision soft-float addition with the IEEE-754
-// default rounding (to nearest, ties to even).
+// This file implements quad-precision soft-float addition.
 //
 //===----------------------------------------------------------------------===//
 
 #define QUAD_PRECISION
 #include "fp_lib.h"
 
 #if defined(CRT_HAS_128BIT) && defined(CRT_LDBL_128BIT)
 #include "fp_add_impl.inc"
 
 COMPILER_RT_ABI long double __addtf3(long double a, long double b){
     return __addXf3__(a, b);
 }
 
 #endif

compiler-rt/trunk/lib/builtins/arm/fp_mode.c

+//===----- lib/arm/fp_mode.c - Floaing-point mode utilities -------*- C -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include <stdint.h>
+
+#include "../fp_mode.h"
+
+#define ARM_TONEAREST  0x0
+#define ARM_UPWARD     0x1
+#define ARM_DOWNWARD   0x2
+#define ARM_TOWARDZERO 0x3
+#define ARM_RMODE_MASK (ARM_TONEAREST | ARM_UPWARD | \
+                        ARM_DOWNWARD | ARM_TOWARDZERO)
+#define ARM_RMODE_SHIFT 22
+
+#define ARM_INEXACT     0x1000
+
+#ifndef __ARM_FP
+// For soft float targets, allow changing rounding mode by overriding the weak
+// __arm_fe_default_rmode symbol.
+FE_ROUND_MODE __attribute__((weak)) __arm_fe_default_rmode = FE_TONEAREST;
+#endif
+
+FE_ROUND_MODE __fe_getround() {
+#ifdef __ARM_FP
+  uint32_t fpscr;
+  __asm__ __volatile__("vmrs  %0, fpscr" : "=r" (fpscr));
+  fpscr = fpscr >> ARM_RMODE_SHIFT & ARM_RMODE_MASK;
+  switch (fpscr) {
+    case ARM_UPWARD:
+      return FE_UPWARD;
+    case ARM_DOWNWARD:
+      return FE_DOWNWARD;
+    case ARM_TOWARDZERO:
+      return FE_TOWARDZERO;
+    case ARM_TONEAREST:
+    default:
+      return FE_TONEAREST;
+  }
+#else
+  return __arm_fe_default_rmode;
+#endif
+}
+
+int __fe_raise_inexact() {
+#ifdef __ARM_FP
+  uint32_t fpscr;
+  __asm__ __volatile__("vmrs  %0, fpscr" : "=r" (fpscr));
+  __asm__ __volatile__("vmsr  fpscr, %0" : : "ri" (fpscr | ARM_INEXACT));
+  return 0;
+#else
+  return 0;
+#endif
+}

compiler-rt/trunk/lib/builtins/fp_add_impl.inc

 //===----- lib/fp_add_impl.inc - floaing point addition -----------*- C -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements soft-float addition with the IEEE-754 default rounding
 // (to nearest, ties to even).
 //
 //===----------------------------------------------------------------------===//
 
 #include "fp_lib.h"
+#include "fp_mode.h"
 
 static __inline fp_t __addXf3__(fp_t a, fp_t b) {
     rep_t aRep = toRep(a);
     rep_t bRep = toRep(b);
     const rep_t aAbs = aRep & absMask;
     const rep_t bAbs = bRep & absMask;
 
     // Detect if a or b is zero, infinity, or NaN.
static __inline fp_t __addXf3__(fp_t a, fp_t b) {
     rep_t result = aSignificand >> 3 & significandMask;
 
     // Insert the exponent and sign.
     result |= (rep_t)aExponent << significandBits;
     result |= resultSign;
 
     // Final rounding.  The result may overflow to infinity, but that is the
     // correct result in that case.
+    switch (__fe_getround()){
+    case FE_TONEAREST:
       if (roundGuardSticky > 0x4) result++;
       if (roundGuardSticky == 0x4) result += result & 1;
+      break;
+    case FE_DOWNWARD:
+      if (resultSign && roundGuardSticky) result++;
+      break;
+    case FE_UPWARD:
+      if (!resultSign && roundGuardSticky) result++;
+      break;
+    case FE_TOWARDZERO:
+      break;
+    }
+    if (roundGuardSticky)
+      __fe_raise_inexact();
     return fromRep(result);
 }

compiler-rt/trunk/lib/builtins/fp_mode.h

+//===----- lib/fp_mode.h - Floaing-point environment mode utilities --C -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is not part of the interface of this library.
+//
+// This file defines an interface for accessing hardware floating point
+// environment mode.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef FP_MODE
+#define FP_MODE
+
+typedef enum {
+  FE_TONEAREST,
+  FE_DOWNWARD,
+  FE_UPWARD,
+  FE_TOWARDZERO
+} FE_ROUND_MODE;
+
+FE_ROUND_MODE __fe_getround();
+int __fe_raise_inexact();
+
+#endif // FP_MODE_H

compiler-rt/trunk/lib/builtins/fp_mode.c

+//===----- lib/fp_mode.c - Floaing-point environment mode utilities --C -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file provides a default implementation of fp_mode.h for architectures
+// that does not support or does not have an implementation of floating point
+// environment mode.
+//
+//===----------------------------------------------------------------------===//
+
+#include "fp_mode.h"
+
+// IEEE-754 default rounding (to nearest, ties to even).
+FE_ROUND_MODE __fe_getround() {
+  return FE_TONEAREST;
+}
+
+int __fe_raise_inexact() {
+  return 0;
+}

compiler-rt/trunk/lib/builtins/subdf3.c

 //===-- lib/adddf3.c - Double-precision subtraction ---------------*- C -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// This file implements double-precision soft-float subtraction with the
-// IEEE-754 default rounding (to nearest, ties to even).
+// This file implements double-precision soft-float subtraction.
 //
 //===----------------------------------------------------------------------===//
 
 #define DOUBLE_PRECISION
 #include "fp_lib.h"
 
 // Subtraction; flip the sign bit of b and add.
 COMPILER_RT_ABI fp_t

compiler-rt/trunk/lib/builtins/subsf3.c

 //===-- lib/subsf3.c - Single-precision subtraction ---------------*- C -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// This file implements single-precision soft-float subtraction with the
-// IEEE-754 default rounding (to nearest, ties to even).
+// This file implements single-precision soft-float subtraction.
 //
 //===----------------------------------------------------------------------===//
 
 #define SINGLE_PRECISION
 #include "fp_lib.h"
 
 // Subtraction; flip the sign bit of b and add.
 COMPILER_RT_ABI fp_t

compiler-rt/trunk/lib/builtins/subtf3.c

 //===-- lib/subtf3.c - Quad-precision subtraction -----------------*- C -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// This file implements quad-precision soft-float subtraction with the
-// IEEE-754 default rounding (to nearest, ties to even).
+// This file implements quad-precision soft-float subtraction.
 //
 //===----------------------------------------------------------------------===//
 
 #define QUAD_PRECISION
 #include "fp_lib.h"
 
 #if defined(CRT_HAS_128BIT) && defined(CRT_LDBL_128BIT)
 COMPILER_RT_ABI fp_t __addtf3(fp_t a, fp_t b);
 
 // Subtraction; flip the sign bit of b and add.
 COMPILER_RT_ABI fp_t
 __subtf3(fp_t a, fp_t b) {
     return __addtf3(a, fromRep(toRep(b) ^ signBit));
 }
 
 #endif

compiler-rt/trunk/test/builtins/Unit/addtf3_test.c

 // RUN: %clang_builtins %s %librt -o %t && %run %t
 //===--------------- addtf3_test.c - Test __addtf3 ------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file tests __addtf3 for the compiler_rt library.
 //
 //===----------------------------------------------------------------------===//
 
-#include "int_lib.h"
+#include <fenv.h>
 #include <stdio.h>
 
 #if __LDBL_MANT_DIG__ == 113
 
+#include "int_lib.h"
 #include "fp_test.h"
 
 // Returns: a + b
 COMPILER_RT_ABI long double __addtf3(long double a, long double b);
 
 int test__addtf3(long double a, long double b,
                  uint64_t expectedHi, uint64_t expectedLo)
 {
if (test__addtf3(makeInf128(),
         return 1;
     // any + any
     if (test__addtf3(0x1.23456734245345543849abcdefp+5L,
                      0x1.edcba52449872455634654321fp-1L,
                      UINT64_C(0x40042afc95c8b579),
                      UINT64_C(0x61e58dd6c51eb77c)))
         return 1;
 
+#if (defined(__arm__) || defined(__aarch64__)) && defined(__ARM_FP)
+    // Rounding mode tests on supported architectures
+    long double m = 1234.0L, n = 0.01L;
+    fesetround(FE_UPWARD);
+    if (__addtf3(m, n) != 1235.0L)
+        return 1;
+
+    fesetround(FE_DOWNWARD);
+    if (__addtf3(m, n) != 1234.0L)
+        return 1;
+
+    fesetround(FE_TOWARDZERO);
+    if (__addtf3(m, n) != 1234.0L)
+        return 1;
+
+    fesetround(FE_TONEAREST);
+    if (__addtf3(m, n) != 1234.0L)
+        return 1;
+#endif
+
 #else
     printf("skipped\n");
 
 #endif
     return 0;
 }

compiler-rt/trunk/test/builtins/Unit/subtf3_test.c

 // RUN: %clang_builtins %s %librt -o %t && %run %t
 //===--------------- subtf3_test.c - Test __subtf3 ------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file tests __subtf3 for the compiler_rt library.
 //
 //===----------------------------------------------------------------------===//
 
+#include <fenv.h>
 #include <stdio.h>
 
 #if __LDBL_MANT_DIG__ == 113
 
 #include "int_lib.h"
 #include "fp_test.h"
 
 // Returns: a - b
if (test__subtf3(makeInf128(),
         return 1;
     // any - any
     if (test__subtf3(0x1.234567829a3bcdef5678ade36734p+5L,
                      0x1.ee9d7c52354a6936ab8d7654321fp-1L,
                      UINT64_C(0x40041b8af1915166),
                      UINT64_C(0xa44a7bca780a166c)))
         return 1;
 
+#if (defined(__arm__) || defined(__aarch64__)) && defined(__ARM_FP)
+    // Rounding mode tests on supported architectures
+    long double m = 1234.0L, n = 0.01L;
+    fesetround(FE_UPWARD);
+    if (__subtf3(m, n) != 1234.0L)
+        return 1;
+
+    fesetround(FE_DOWNWARD);
+    if (__subtf3(m, n) != 1233.0L)
+        return 1;
+
+    fesetround(FE_TOWARDZERO);
+    if (__subtf3(m, n) != 1233.0L)
+        return 1;
+
+    fesetround(FE_TONEAREST);
+    if (__subtf3(m, n) != 1234.0L)
+        return 1;
+#endif
+
 #else
     printf("skipped\n");
 
 #endif
     return 0;
 }