[compiler-rt][SelectionDAG] Add extendbfsf2 libcall and use it for bf16 extends with soft FP

[asb]

Summary

Previously this resulted in an assert (reproducible on RISC-V with soft FP). The existing code path assumes a libcall is present, and adding the libcall seems like the easiest fix. This libcall _is_ provided by libgcc, which perhaps providing its own motivation for adding it here.

The legalisation code in LegalizeDAG lowers to an anyext and shift which might be an alternative. This would however be more invasive to support vs just adding an extra case to the existing libcall lowering logic, and these soft targets are likely not a target we care strongly about BF16 support beyond wanting some basic support for completeness.

I'm not able to convince myself that the anyext+shift lowering is always identical to the more elaborate extension performed by the libcall in all cases (and if so, why do the trunc and extend libcalls even exist?). though I'm not sure I can convince myself. I know @craig.topper was involved in a previous discussion on this so I'd appreciate your view.

[craig.topper]

I'm not able to convince myself that the anyext+shift lowering is always identical to the more elaborate extension performed by the libcall in all cases (and if so, why do the trunc and extend libcalls even exist?). though I'm not sure I can convince myself. I know @craig.topper was involved in a previous discussion on this so I'd appreciate your view.

fp32 has more bits of mantissa than bfloat16 but they have the same number of exponent bits.

The trunc libcall exists because the extra bits of mantissa that exist in fp32 need to be rounded to convert to bfloat16. Also some f32 subnormal values can't be represented in bfloat16. So it can't be done as an integer truncate.

For extend, we should just need to add 0s to the end of the mantissa. The +0.0, -0.0 are encoded as all 0s in the mantissa and exponent in both encodings. infinity is encoded with a special exponent and all 0 mantissa in both formats. nan uses the same exponent as infinity but a non-zero mantissa. If the mantissa is already non-zero, adding more zeros doesn't change that. Adding zeros to the end of the mantissa for normals and denormals shouldn't change their value.

[asb]

In D151436#4372633, @craig.topper wrote:

For extend, we should just need to add 0s to the end of the mantissa. The +0.0, -0.0 are encoded as all 0s in the mantissa and exponent in both encodings. infinity is encoded with a special exponent and all 0 mantissa in both formats. nan uses the same exponent as infinity but a non-zero mantissa. If the mantissa is already non-zero, adding more zeros doesn't change that. Adding zeros to the end of the mantissa for normals and denormals shouldn't change their value.

And then we'd just lose out on FE_INVALID being set if the input is a signalling NaN - it seems libgcc does have some support for setting these exception bits (on some platforms at least, with the right support hooks implemented) while compiler-rt has none. So I think that justifies the libcall for them. Thanks for helping clear that up.

[arsenm]

You would only need to worry about snans with the constrained fptrunc

[asb]

Retired in favour of D151563.

compiler-rt/lib/builtins/CMakeLists.txt

set(GENERIC_SOURCES
   udivti3.c
   umoddi3.c
   umodsi3.c
   umodti3.c
 )
 
 # We only build BF16 files when "__bf16" is available.
 set(BF16_SOURCES
+    extendbfsf2.c
     truncdfbf2.c
     truncsfbf2.c
 )
 
 # TODO: Several "tf" files (and divtc3.c, but not multc3.c) are in
 # GENERIC_SOURCES instead of here.
 set(GENERIC_TF_SOURCES
   addtf3.c

compiler-rt/lib/builtins/extendbfsf2.c

+//===-- lib/extendbfsf2.c - bfloat -> single conversion -----------*- C -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#define SRC_BFLOAT
+#define DST_SINGLE
+#include "fp_extend_impl.inc"
+
+COMPILER_RT_ABI float __extendbfsf2(src_t a) { return __extendXfYf2__(a); }

compiler-rt/lib/builtins/fp_extend.h

 #else
 typedef uint16_t src_t;
 #endif
 typedef uint16_t src_rep_t;
 #define SRC_REP_C UINT16_C
 static const int srcSigBits = 10;
 #define src_rep_t_clz __builtin_clz
 
+#elif defined SRC_BFLOAT
+typedef __bf16 src_t;
+typedef uint16_t src_rep_t;
+#define SRC_REP_C UINT16_C
+static const int srcSigBits = 7;
+#define src_rep_t_clz __builtin_clz
+
 #else
-#error Source should be half, single, or double precision!
+#error Source should be bfloat, half, single, or double precision!
 #endif // end source precision
 
 #if defined DST_SINGLE
 typedef float dst_t;
 typedef uint32_t dst_rep_t;
 #define DST_REP_C UINT32_C
 static const int dstSigBits = 23;
 

llvm/include/llvm/IR/RuntimeLibcalls.def

 HANDLE_LIBCALL(FPEXT_F80_F128, "__extendxftf2")
 HANDLE_LIBCALL(FPEXT_F64_F128, "__extenddftf2")
 HANDLE_LIBCALL(FPEXT_F32_F128, "__extendsftf2")
 HANDLE_LIBCALL(FPEXT_F16_F128, "__extendhftf2")
 HANDLE_LIBCALL(FPEXT_F16_F80, "__extendhfxf2")
 HANDLE_LIBCALL(FPEXT_F32_F64, "__extendsfdf2")
 HANDLE_LIBCALL(FPEXT_F16_F64, "__extendhfdf2")
 HANDLE_LIBCALL(FPEXT_F16_F32, "__gnu_h2f_ieee")
+HANDLE_LIBCALL(FPEXT_BF16_F32, "__extendbfsf2")
 HANDLE_LIBCALL(FPROUND_F32_F16, "__gnu_f2h_ieee")
 HANDLE_LIBCALL(FPROUND_F64_F16, "__truncdfhf2")
 HANDLE_LIBCALL(FPROUND_F80_F16, "__truncxfhf2")
 HANDLE_LIBCALL(FPROUND_F128_F16, "__trunctfhf2")
 HANDLE_LIBCALL(FPROUND_PPCF128_F16, "__trunctfhf2")
 HANDLE_LIBCALL(FPROUND_F32_BF16, "__truncsfbf2")
 HANDLE_LIBCALL(FPROUND_F64_BF16, "__truncdfbf2")
 HANDLE_LIBCALL(FPROUND_F64_F32, "__truncdfsf2")

llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

SDValue DAGTypeLegalizer::SoftenFloatRes_FP_EXTEND(SDNode *N) {
   if (getTypeAction(Op.getValueType()) == TargetLowering::TypePromoteFloat) {
     Op = GetPromotedFloat(Op);
     // If the promotion did the FP_EXTEND to the destination type for us,
     // there's nothing left to do here.
     if (Op.getValueType() == N->getValueType(0))
       return BitConvertToInteger(Op);
   }
 
-  // There's only a libcall for f16 -> f32, so proceed in two stages. Also, it's
-  // entirely possible for both f16 and f32 to be legal, so use the fully
-  // hard-float FP_EXTEND rather than FP16_TO_FP.
-  if (Op.getValueType() == MVT::f16 && N->getValueType(0) != MVT::f32) {
+  // There's only a libcall for [b]f16 -> f32, so proceed in two stages. Also,
+  // it's entirely possible for both [b]f16 and f32 to be legal, so use the
+  // fully hard-float FP_EXTEND rather than {FP16,BF16}_TO_FP.
+  if ((Op.getValueType() == MVT::f16 || Op.getValueType() == MVT::bf16) &&
+      N->getValueType(0) != MVT::f32) {
     if (IsStrict) {
       Op = DAG.getNode(ISD::STRICT_FP_EXTEND, SDLoc(N),
                        { MVT::f32, MVT::Other }, { Chain, Op });
       Chain = Op.getValue(1);
     } else {
       Op = DAG.getNode(ISD::FP_EXTEND, SDLoc(N), MVT::f32, Op);
     }
   }

llvm/lib/CodeGen/TargetLoweringBase.cpp

if (OpVT == MVT::f16) {
     if (RetVT == MVT::f32)
       return FPEXT_F16_F32;
     if (RetVT == MVT::f64)
       return FPEXT_F16_F64;
     if (RetVT == MVT::f80)
       return FPEXT_F16_F80;
     if (RetVT == MVT::f128)
       return FPEXT_F16_F128;
+  } else if (OpVT == MVT::bf16) {
+    if (RetVT == MVT::f32)
+      return FPEXT_BF16_F32;
   } else if (OpVT == MVT::f32) {
     if (RetVT == MVT::f64)
       return FPEXT_F32_F64;
     if (RetVT == MVT::f128)
       return FPEXT_F32_F128;
     if (RetVT == MVT::ppcf128)
       return FPEXT_F32_PPCF128;
   } else if (OpVT == MVT::f64) {

llvm/test/CodeGen/RISCV/bfloat.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -verify-machineinstrs < %s | FileCheck %s -check-prefix=RV32I-ILP32
+; RUN: llc -mtriple=riscv64 -verify-machineinstrs < %s | FileCheck %s -check-prefix=RV64I-LP64
+
+; TODO: Enable codegen for hard float.
+
+define bfloat @float_to_bfloat(float %a) nounwind {
+; RV32I-ILP32-LABEL: float_to_bfloat:
+; RV32I-ILP32:       # %bb.0:
+; RV32I-ILP32-NEXT:    addi sp, sp, -16
+; RV32I-ILP32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; RV32I-ILP32-NEXT:    call __truncsfbf2@plt
+; RV32I-ILP32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; RV32I-ILP32-NEXT:    addi sp, sp, 16
+; RV32I-ILP32-NEXT:    ret
+;
+; RV64I-LP64-LABEL: float_to_bfloat:
+; RV64I-LP64:       # %bb.0:
+; RV64I-LP64-NEXT:    addi sp, sp, -16
+; RV64I-LP64-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
+; RV64I-LP64-NEXT:    call __truncsfbf2@plt
+; RV64I-LP64-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
+; RV64I-LP64-NEXT:    addi sp, sp, 16
+; RV64I-LP64-NEXT:    ret
+  %1 = fptrunc float %a to bfloat
+  ret bfloat %1
+}
+
+define bfloat @double_to_bfloat(double %a) nounwind {
+; RV32I-ILP32-LABEL: double_to_bfloat:
+; RV32I-ILP32:       # %bb.0:
+; RV32I-ILP32-NEXT:    addi sp, sp, -16
+; RV32I-ILP32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; RV32I-ILP32-NEXT:    call __truncdfbf2@plt
+; RV32I-ILP32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; RV32I-ILP32-NEXT:    addi sp, sp, 16
+; RV32I-ILP32-NEXT:    ret
+;
+; RV64I-LP64-LABEL: double_to_bfloat:
+; RV64I-LP64:       # %bb.0:
+; RV64I-LP64-NEXT:    addi sp, sp, -16
+; RV64I-LP64-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
+; RV64I-LP64-NEXT:    call __truncdfbf2@plt
+; RV64I-LP64-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
+; RV64I-LP64-NEXT:    addi sp, sp, 16
+; RV64I-LP64-NEXT:    ret
+  %1 = fptrunc double %a to bfloat
+  ret bfloat %1
+}
+
+define float @bfloat_to_float(bfloat %a) nounwind {
+; RV32I-ILP32-LABEL: bfloat_to_float:
+; RV32I-ILP32:       # %bb.0:
+; RV32I-ILP32-NEXT:    addi sp, sp, -16
+; RV32I-ILP32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; RV32I-ILP32-NEXT:    call __extendbfsf2@plt
+; RV32I-ILP32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; RV32I-ILP32-NEXT:    addi sp, sp, 16
+; RV32I-ILP32-NEXT:    ret
+;
+; RV64I-LP64-LABEL: bfloat_to_float:
+; RV64I-LP64:       # %bb.0:
+; RV64I-LP64-NEXT:    addi sp, sp, -16
+; RV64I-LP64-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
+; RV64I-LP64-NEXT:    call __extendbfsf2@plt
+; RV64I-LP64-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
+; RV64I-LP64-NEXT:    addi sp, sp, 16
+; RV64I-LP64-NEXT:    ret
+  %1 = fpext bfloat %a to float
+  ret float %1
+}
+
+define double @bfloat_to_double(bfloat %a) nounwind {
+; RV32I-ILP32-LABEL: bfloat_to_double:
+; RV32I-ILP32:       # %bb.0:
+; RV32I-ILP32-NEXT:    addi sp, sp, -16
+; RV32I-ILP32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; RV32I-ILP32-NEXT:    call __extendbfsf2@plt
+; RV32I-ILP32-NEXT:    call __extendsfdf2@plt
+; RV32I-ILP32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; RV32I-ILP32-NEXT:    addi sp, sp, 16
+; RV32I-ILP32-NEXT:    ret
+;
+; RV64I-LP64-LABEL: bfloat_to_double:
+; RV64I-LP64:       # %bb.0:
+; RV64I-LP64-NEXT:    addi sp, sp, -16
+; RV64I-LP64-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
+; RV64I-LP64-NEXT:    call __extendbfsf2@plt
+; RV64I-LP64-NEXT:    call __extendsfdf2@plt
+; RV64I-LP64-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
+; RV64I-LP64-NEXT:    addi sp, sp, 16
+; RV64I-LP64-NEXT:    ret
+  %1 = fpext bfloat %a to double
+  ret double %1
+}