[RISCV][BF16] Make backend type bf16 to follow the psABI

[joshua-arch1]

Summary

Recently, bf16 type has been added in RISCV psABI. Now we need to support this datatype in LLVM backend. Right now, according to psABI, we basically address __bf16 like FP16.

[asb]

Sorry to not have specifics on this - I'm working on a patch in this same area and as you've found as well, it's quite fiddly. Using the __truncsfhf2 and __extendhfsf2 libcalls isn't correct - they expect 16-bit floats in a different format to bfloat16.

[joshua-arch1]

This comment was removed by joshua-arch1.

[joshua-arch1]

In D151313#4367945, @asb wrote:

Sorry to not have specifics on this - I'm working on a patch in this same area and as you've found as well, it's quite fiddly. Using the __truncsfhf2 and __extendhfsf2 libcalls isn't correct - they expect 16-bit floats in a different format to bfloat16.

I have checked compiler-rt/lib/builtins. There is truncsfbf2 for single-bfloat conversion, but I cannot find any library for bfloat extend.

[joshua-arch1]

Modify libcall for single-bfloat conversion.

[kito-cheng]

libgcc has defined truncsfbf2, truncdfbf2, truncxfbf2, trunctfbf2, trunchfbf2, truncbfhf2 and extendbfsf2

I guess this might made me seems like a GNUism, but following the same rule should be easier for our life :P

Ref: https://github.com/gcc-mirror/gcc/commit/c2565a31c1622ab0926aeef4a6579413e121b9f9

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

2458

If the libcall doesn't exist, you can cast it to i16, any_extend to i32, and shift it left by 16 and bitcast it to f32.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

2458

Which is what the default lowering in LegalizeDAG.cpp does.

[joshua-arch1]

Address bf16_to_fp/fp_to_bf16 via bitcast and shift.

[joshua-arch1]

Modify implementation for f64 input.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

2438

This is an incorrect conversion. We can't truncate the mantissa. It would turn some nan encodings into infinity. Probably other issues I haven't thought of yet.

2438

We need to use truncsfbf2

2469

No need for else after an if that returns

2486

Doesn't this need to be i64 on 64 bit target?

2487

This creates an MVT::i16 after type legalization which is illegal.

2493

You share the shift code between both paths by using XLenVT.

2495

Use getShiftAmountConstant

2509

No need for else after if returns

[joshua-arch1]

Update FP_TO_BF16 lowering by __truncsfbf2 libcall.

[joshua-arch1]

This comment was removed by joshua-arch1.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

2458

We cannot directly use the default lowering in LegalizeDAG.cpp. It uses a bitcast from i32 to f32. However, i32 is not a legal type in RV64 backend.

2487

In fact, without Zfbfmin enabled, bf16 type isn't legal and this condition will never happen. If Zfbfmin is enabled, maybe we can use FCVT.S.BF16 to directly convert BF16 value to an FP32 value.

[joshua-arch1]

This comment was removed by joshua-arch1.

[joshua-arch1]

This comment was removed by joshua-arch1.

[joshua-arch1]

The first version of this patch came in May 24th, earlier than a similar patch implemented by @asb https://reviews.llvm.org/D151663. Why could @asb's patch be accepted first?

[craig.topper]

In D151313#4431368, @joshua-arch1 wrote:

The first version of this patch came in May 24th, earlier than a similar patch implemented by @asb https://reviews.llvm.org/D151663. Why could @asb's patch be accepted first?

Some parts of @asb’s patch look more correct. Like the use isSoftFloatABI().

[majin2020]

I think in this case, it should be to provide review suggestions for modification, instead of immediately accepting other people's improvement patch, which has a great impact on the enthusiasm of contributors :)

[craig.topper]

llvm/test/CodeGen/RISCV/bfloat.ll

1

This file already exists in trunk. Not sure if is the same or not?

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

428

This wasn't needed by D151663? Is there a test case missing from D151663 that requires this?

432

D151663 uses

setOperationAction(ISD::FP_TO_BF16, MVT::f32,
                   Subtarget.isSoftFPABI() ? LibCall : Custom);

Is that difference important?

2461

Why wasn't this part needed in D151663?

[joshua-arch1]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

428

Maybe I can remove this part. In my first version of this patch, the expand is needed for LoadExt and TruncStore.

432

Since we use also libcall in the lowerFP_TO_BF16 function, I don't think we need to use "Libcall" here for SoftABI.

2461

Although bf16 type isn't legal, that does not mean this condition will never happen.

llvm/lib/Target/RISCV/RISCVISelLowering.h

SDValue lowerVPFPIntConvOp(SDValue Op, SelectionDAG &DAG,
                              unsigned RISCVISDOpc) const;
   SDValue lowerVPStridedLoad(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerVPStridedStore(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerFixedLengthVectorExtendToRVV(SDValue Op, SelectionDAG &DAG,
                                             unsigned ExtendOpc) const;
   SDValue lowerGET_ROUNDING(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerSET_ROUNDING(SDValue Op, SelectionDAG &DAG) const;
 
+  SDValue lowerFP_TO_BF16(SDValue Op, SelectionDAG &DAG) const;
+  SDValue lowerBF16_TO_FP(SDValue Op, SelectionDAG &DAG) const;
+
   SDValue lowerEH_DWARF_CFA(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerCTLZ_CTTZ_ZERO_UNDEF(SDValue Op, SelectionDAG &DAG) const;
 
   SDValue lowerStrictFPExtendOrRoundLike(SDValue Op, SelectionDAG &DAG) const;
 
   SDValue lowerVectorStrictFSetcc(SDValue Op, SelectionDAG &DAG) const;
 
   SDValue expandUnalignedRVVLoad(SDValue Op, SelectionDAG &DAG) const;

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

if (Subtarget.hasStdExtFOrZfinx()) {
     setOperationAction(FPRndMode, MVT::f32,
                        Subtarget.hasStdExtZfa() ? Legal : Custom);
     setCondCodeAction(FPCCToExpand, MVT::f32, Expand);
     setOperationAction(ISD::SELECT_CC, MVT::f32, Expand);
     setOperationAction(ISD::SELECT, MVT::f32, Custom);
     setOperationAction(ISD::BR_CC, MVT::f32, Expand);
     setOperationAction(FPOpToExpand, MVT::f32, Expand);
     setLoadExtAction(ISD::EXTLOAD, MVT::f32, MVT::f16, Expand);
+    setLoadExtAction(ISD::EXTLOAD, MVT::f32, MVT::bf16, Expand);
     setTruncStoreAction(MVT::f32, MVT::f16, Expand);
+    setTruncStoreAction(MVT::f32, MVT::bf16, Expand);
+    setOperationAction(ISD::FP_TO_BF16, MVT::f32, Custom);
+    setOperationAction(ISD::BF16_TO_FP, MVT::f32, Custom);
     setOperationAction(ISD::IS_FPCLASS, MVT::f32, Custom);
 
     if (Subtarget.hasStdExtZfa())
       setOperationAction(ISD::FNEARBYINT, MVT::f32, Legal);
   }
 
   if (Subtarget.hasStdExtFOrZfinx() && Subtarget.is64Bit())
     setOperationAction(ISD::BITCAST, MVT::i32, Custom);
if (Subtarget.hasStdExtDOrZdinx()) {
     setOperationAction(ISD::SELECT_CC, MVT::f64, Expand);
     setOperationAction(ISD::SELECT, MVT::f64, Custom);
     setOperationAction(ISD::BR_CC, MVT::f64, Expand);
     setLoadExtAction(ISD::EXTLOAD, MVT::f64, MVT::f32, Expand);
     setTruncStoreAction(MVT::f64, MVT::f32, Expand);
     setOperationAction(FPOpToExpand, MVT::f64, Expand);
     setLoadExtAction(ISD::EXTLOAD, MVT::f64, MVT::f16, Expand);
     setTruncStoreAction(MVT::f64, MVT::f16, Expand);
+    setLoadExtAction(ISD::EXTLOAD, MVT::f64, MVT::bf16, Expand);
+    setTruncStoreAction(MVT::f64, MVT::bf16, Expand);
+    setOperationAction(ISD::FP_TO_BF16, MVT::f64, Custom);
+    setOperationAction(ISD::BF16_TO_FP, MVT::f64, Custom);
     setOperationAction(ISD::IS_FPCLASS, MVT::f64, Custom);
   }
 
   if (Subtarget.is64Bit()) {
     setOperationAction({ISD::FP_TO_UINT, ISD::FP_TO_SINT,
                         ISD::STRICT_FP_TO_UINT, ISD::STRICT_FP_TO_SINT},
                        MVT::i32, Custom);
     setOperationAction(ISD::LROUND, MVT::i32, Custom);
Res = DAG.getNode(RISCVISD::VSELECT_VL, DL, DstContainerVT, IsNan, SplatZero,
                     Res, VL);
 
   if (DstVT.isFixedLengthVector())
     Res = convertFromScalableVector(DstVT, Res, DAG, Subtarget);
 
   return Res;
 }
 
+SDValue RISCVTargetLowering::lowerFP_TO_BF16(SDValue Op,
+                                             SelectionDAG &DAG) const {
+  SDLoc DL(Op);
+  MakeLibCallOptions CallOptions;
+  RTLIB::Libcall LC =
+      RTLIB::getFPROUND(Op.getOperand(0).getValueType(), MVT::bf16);
+  SDValue Res =
+      makeLibCall(DAG, LC, MVT::f32, Op.getOperand(0), CallOptions, DL).first;
+  if (Subtarget.is64Bit())
+    return DAG.getNode(RISCVISD::FMV_X_ANYEXTW_RV64, DL, MVT::i64, Res);
+
+  return DAG.getBitcast(MVT::i32, Res);
+}
+
+SDValue RISCVTargetLowering::lowerBF16_TO_FP(SDValue Op,
+                                             SelectionDAG &DAG) const {
+  // Always expand bf16 to f32 casts, they lower to ext + shift.
+  //
+  // Note that the operand of this code can be bf16 or an integer type in case
+  // bf16 is not supported on the target and was softened.
+  SDLoc DL(Op);
+  SDValue Op0;
+  SDValue Res;
+  MVT XLenVT = Subtarget.getXLenVT();
+  if (Op.getOperand(0).getValueType() == MVT::bf16) {
+    Op0 =
+        DAG.getNode(ISD::ANY_EXTEND, DL, XLenVT,
+                    DAG.getNode(ISD::BITCAST, DL, MVT::i16, Op.getOperand(0)));
+  } else {
+    Op0 = Op.getOperand(0);
+  }
+
+  SDValue Shift = DAG.getNode(ISD::SHL, DL, XLenVT, Op0,
+                              DAG.getShiftAmountConstant(16, XLenVT, DL));
+
+  if (Subtarget.is64Bit()) {
+    Res = DAG.getNode(RISCVISD::FMV_W_X_RV64, DL, MVT::f32, Shift);
+  } else {
+    Res = DAG.getNode(ISD::BITCAST, DL, MVT::f32, Shift);
+  }
+
+  // Add fp_extend in case the output is bigger than f32.
+  if (Op.getValueType() != MVT::f32)
+    return DAG.getNode(ISD::FP_EXTEND, DL, Op.getValueType(), Res);
+
+  return Res;
+}
+
 static RISCVFPRndMode::RoundingMode matchRoundingOp(unsigned Opc) {
   switch (Opc) {
   case ISD::FROUNDEVEN:
   case ISD::STRICT_FROUNDEVEN:
   case ISD::VP_FROUNDEVEN:
     return RISCVFPRndMode::RNE;
   case ISD::FTRUNC:
   case ISD::STRICT_FTRUNC:
   case ISD::VP_FROUNDTOZERO:
     return RISCVFPRndMode::RTZ;
   case ISD::FFLOOR:
   case ISD::STRICT_FFLOOR:
   case ISD::VP_FFLOOR:
     return RISCVFPRndMode::RDN;
   case ISD::FCEIL:
   case ISD::STRICT_FCEIL:
   case ISD::VP_FCEIL:
     return RISCVFPRndMode::RUP;
   case ISD::FROUND:
   case ISD::STRICT_FROUND:
   case ISD::VP_FROUND:
     return RISCVFPRndMode::RMM;
   case ISD::FRINT:
     return RISCVFPRndMode::DYN;
   }
 
   return RISCVFPRndMode::Invalid;
 }
 
 // Expand vector FTRUNC, FCEIL, FFLOOR, FROUND, VP_FCEIL, VP_FFLOOR, VP_FROUND
 // VP_FROUNDEVEN, VP_FROUNDTOZERO, VP_FRINT and VP_FNEARBYINT by converting to
 // the integer domain and back. Taking care to avoid converting values that are
 // nan or already correct.
 static SDValue
if (IsStrict) {
       return DAG.getMergeValues({SubVec, Src.getValue(1)}, DL);
     }
     Src = DAG.getNode(RVVOpc, DL, ContainerVT, Src, Mask, VL);
     return convertFromScalableVector(VT, Src, DAG, Subtarget);
   }
   case ISD::FP_TO_SINT_SAT:
   case ISD::FP_TO_UINT_SAT:
     return lowerFP_TO_INT_SAT(Op, DAG, Subtarget);
+  case ISD::FP_TO_BF16:
+    return lowerFP_TO_BF16(Op, DAG);
+  case ISD::BF16_TO_FP:
+    return lowerBF16_TO_FP(Op, DAG);
   case ISD::FTRUNC:
   case ISD::FCEIL:
   case ISD::FFLOOR:
   case ISD::FNEARBYINT:
   case ISD::FRINT:
   case ISD::FROUND:
   case ISD::FROUNDEVEN:
     return lowerFTRUNC_FCEIL_FFLOOR_FROUND(Op, DAG, Subtarget);
bool RISCVTargetLowering::allowsMisalignedMemoryAccesses(
   return Subtarget.enableUnalignedVectorMem();
 }
 
 bool RISCVTargetLowering::splitValueIntoRegisterParts(
     SelectionDAG &DAG, const SDLoc &DL, SDValue Val, SDValue *Parts,
     unsigned NumParts, MVT PartVT, std::optional<CallingConv::ID> CC) const {
   bool IsABIRegCopy = CC.has_value();
   EVT ValueVT = Val.getValueType();
-  if (IsABIRegCopy && ValueVT == MVT::f16 && PartVT == MVT::f32) {
-    // Cast the f16 to i16, extend to i32, pad with ones to make a float nan,
+  if (IsABIRegCopy && (ValueVT == MVT::f16 || ValueVT == MVT::bf16) && 
+      PartVT == MVT::f32) {
+    // Cast the f16/bf16 to i16, extend to i32, pad with ones to make a float nan,
     // and cast to f32.
     Val = DAG.getNode(ISD::BITCAST, DL, MVT::i16, Val);
     Val = DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i32, Val);
     Val = DAG.getNode(ISD::OR, DL, MVT::i32, Val,
                       DAG.getConstant(0xFFFF0000, DL, MVT::i32));
     Val = DAG.getNode(ISD::BITCAST, DL, MVT::f32, Val);
     Parts[0] = Val;
     return true;
bool RISCVTargetLowering::splitValueIntoRegisterParts(
   }
   return false;
 }
 
 SDValue RISCVTargetLowering::joinRegisterPartsIntoValue(
     SelectionDAG &DAG, const SDLoc &DL, const SDValue *Parts, unsigned NumParts,
     MVT PartVT, EVT ValueVT, std::optional<CallingConv::ID> CC) const {
   bool IsABIRegCopy = CC.has_value();
-  if (IsABIRegCopy && ValueVT == MVT::f16 && PartVT == MVT::f32) {
+  if (IsABIRegCopy && (ValueVT == MVT::f16 || ValueVT == MVT::bf16) && 
+      PartVT == MVT::f32) {
     SDValue Val = Parts[0];
 
-    // Cast the f32 to i32, truncate to i16, and cast back to f16.
+    // Cast the f32 to i32, truncate to i16, and cast back to f16/bf16.
     Val = DAG.getNode(ISD::BITCAST, DL, MVT::i32, Val);
     Val = DAG.getNode(ISD::TRUNCATE, DL, MVT::i16, Val);
-    Val = DAG.getNode(ISD::BITCAST, DL, MVT::f16, Val);
+    Val = DAG.getNode(ISD::BITCAST, DL, ValueVT, Val);
     return Val;
   }
 
   if (ValueVT.isScalableVector() && PartVT.isScalableVector()) {
     LLVMContext &Context = *DAG.getContext();
     SDValue Val = Parts[0];
     EVT ValueEltVT = ValueVT.getVectorElementType();
     EVT PartEltVT = PartVT.getVectorElementType();

llvm/test/CodeGen/RISCV/bfloat.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=riscv64 -mattr=+f,+d -verify-machineinstrs| FileCheck %s --check-prefixes=RV64
+; RUN: llc < %s -mtriple=riscv32 -mattr=+f,+d -verify-machineinstrs| FileCheck %s --check-prefixes=RV32
+
+define void @add(ptr %pa, ptr %pb, ptr %pc) nounwind {
+; RV64-LABEL: add:
+; RV64:       # %bb.0:
+; RV64-NEXT:    addi sp, sp, -16
+; RV64-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
+; RV64-NEXT:    sd s0, 0(sp) # 8-byte Folded Spill
+; RV64-NEXT:    lhu a1, 0(a1)
+; RV64-NEXT:    lhu a0, 0(a0)
+; RV64-NEXT:    mv s0, a2
+; RV64-NEXT:    slli a1, a1, 16
+; RV64-NEXT:    fmv.w.x fa5, a1
+; RV64-NEXT:    slli a0, a0, 16
+; RV64-NEXT:    fmv.w.x fa4, a0
+; RV64-NEXT:    fadd.s fa0, fa4, fa5
+; RV64-NEXT:    call __truncsfbf2@plt
+; RV64-NEXT:    fmv.x.w a0, fa0
+; RV64-NEXT:    sh a0, 0(s0)
+; RV64-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
+; RV64-NEXT:    ld s0, 0(sp) # 8-byte Folded Reload
+; RV64-NEXT:    addi sp, sp, 16
+; RV64-NEXT:    ret
+;
+; RV32-LABEL: add:
+; RV32:       # %bb.0:
+; RV32-NEXT:    addi sp, sp, -16
+; RV32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; RV32-NEXT:    sw s0, 8(sp) # 4-byte Folded Spill
+; RV32-NEXT:    lhu a1, 0(a1)
+; RV32-NEXT:    lhu a0, 0(a0)
+; RV32-NEXT:    mv s0, a2
+; RV32-NEXT:    slli a1, a1, 16
+; RV32-NEXT:    fmv.w.x fa5, a1
+; RV32-NEXT:    slli a0, a0, 16
+; RV32-NEXT:    fmv.w.x fa4, a0
+; RV32-NEXT:    fadd.s fa0, fa4, fa5
+; RV32-NEXT:    call __truncsfbf2@plt
+; RV32-NEXT:    fmv.x.w a0, fa0
+; RV32-NEXT:    sh a0, 0(s0)
+; RV32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; RV32-NEXT:    lw s0, 8(sp) # 4-byte Folded Reload
+; RV32-NEXT:    addi sp, sp, 16
+; RV32-NEXT:    ret
+  %a = load bfloat, ptr %pa
+  %b = load bfloat, ptr %pb
+  %add = fadd bfloat %a, %b
+  store bfloat %add, ptr %pc
+  ret void
+}
+
+define bfloat @add2(bfloat %a, bfloat %b) nounwind {
+; RV64-LABEL: add2:
+; RV64:       # %bb.0:
+; RV64-NEXT:    addi sp, sp, -16
+; RV64-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
+; RV64-NEXT:    fmv.x.w a0, fa0
+; RV64-NEXT:    lui a1, 16
+; RV64-NEXT:    addiw a1, a1, -1
+; RV64-NEXT:    and a0, a0, a1
+; RV64-NEXT:    fmv.x.w a2, fa1
+; RV64-NEXT:    and a1, a2, a1
+; RV64-NEXT:    slli a1, a1, 16
+; RV64-NEXT:    fmv.w.x fa5, a1
+; RV64-NEXT:    slli a0, a0, 16
+; RV64-NEXT:    fmv.w.x fa4, a0
+; RV64-NEXT:    fadd.s fa0, fa4, fa5
+; RV64-NEXT:    call __truncsfbf2@plt
+; RV64-NEXT:    fmv.x.w a0, fa0
+; RV64-NEXT:    lui a1, 1048560
+; RV64-NEXT:    or a0, a0, a1
+; RV64-NEXT:    fmv.w.x fa0, a0
+; RV64-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
+; RV64-NEXT:    addi sp, sp, 16
+; RV64-NEXT:    ret
+;
+; RV32-LABEL: add2:
+; RV32:       # %bb.0:
+; RV32-NEXT:    addi sp, sp, -16
+; RV32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; RV32-NEXT:    fmv.x.w a0, fa0
+; RV32-NEXT:    fmv.x.w a1, fa1
+; RV32-NEXT:    slli a1, a1, 16
+; RV32-NEXT:    fmv.w.x fa5, a1
+; RV32-NEXT:    slli a0, a0, 16
+; RV32-NEXT:    fmv.w.x fa4, a0
+; RV32-NEXT:    fadd.s fa0, fa4, fa5
+; RV32-NEXT:    call __truncsfbf2@plt
+; RV32-NEXT:    fmv.x.w a0, fa0
+; RV32-NEXT:    lui a1, 1048560
+; RV32-NEXT:    or a0, a0, a1
+; RV32-NEXT:    fmv.w.x fa0, a0
+; RV32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; RV32-NEXT:    addi sp, sp, 16
+; RV32-NEXT:    ret
+  %add = fadd bfloat %a, %b
+  ret bfloat %add
+}
+
+define void @add_constant(ptr %pa, ptr %pc) nounwind {
+; RV64-LABEL: add_constant:
+; RV64:       # %bb.0:
+; RV64-NEXT:    addi sp, sp, -16
+; RV64-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
+; RV64-NEXT:    sd s0, 0(sp) # 8-byte Folded Spill
+; RV64-NEXT:    lhu a0, 0(a0)
+; RV64-NEXT:    mv s0, a1
+; RV64-NEXT:    slli a0, a0, 16
+; RV64-NEXT:    fmv.w.x fa5, a0
+; RV64-NEXT:    lui a0, 260096
+; RV64-NEXT:    fmv.w.x fa4, a0
+; RV64-NEXT:    fadd.s fa0, fa5, fa4
+; RV64-NEXT:    call __truncsfbf2@plt
+; RV64-NEXT:    fmv.x.w a0, fa0
+; RV64-NEXT:    sh a0, 0(s0)
+; RV64-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
+; RV64-NEXT:    ld s0, 0(sp) # 8-byte Folded Reload
+; RV64-NEXT:    addi sp, sp, 16
+; RV64-NEXT:    ret
+;
+; RV32-LABEL: add_constant:
+; RV32:       # %bb.0:
+; RV32-NEXT:    addi sp, sp, -16
+; RV32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; RV32-NEXT:    sw s0, 8(sp) # 4-byte Folded Spill
+; RV32-NEXT:    lhu a0, 0(a0)
+; RV32-NEXT:    mv s0, a1
+; RV32-NEXT:    slli a0, a0, 16
+; RV32-NEXT:    fmv.w.x fa5, a0
+; RV32-NEXT:    lui a0, 260096
+; RV32-NEXT:    fmv.w.x fa4, a0
+; RV32-NEXT:    fadd.s fa0, fa5, fa4
+; RV32-NEXT:    call __truncsfbf2@plt
+; RV32-NEXT:    fmv.x.w a0, fa0
+; RV32-NEXT:    sh a0, 0(s0)
+; RV32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; RV32-NEXT:    lw s0, 8(sp) # 4-byte Folded Reload
+; RV32-NEXT:    addi sp, sp, 16
+; RV32-NEXT:    ret
+  %a = load bfloat, ptr %pa
+  %add = fadd bfloat %a, 1.0
+  store bfloat %add, ptr %pc
+  ret void
+}
+
+define bfloat @add_constant2(bfloat %a) nounwind {
+; RV64-LABEL: add_constant2:
+; RV64:       # %bb.0:
+; RV64-NEXT:    addi sp, sp, -16
+; RV64-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
+; RV64-NEXT:    fmv.x.w a0, fa0
+; RV64-NEXT:    slli a0, a0, 48
+; RV64-NEXT:    srli a0, a0, 48
+; RV64-NEXT:    slli a0, a0, 16
+; RV64-NEXT:    fmv.w.x fa5, a0
+; RV64-NEXT:    lui a0, 260096
+; RV64-NEXT:    fmv.w.x fa4, a0
+; RV64-NEXT:    fadd.s fa0, fa5, fa4
+; RV64-NEXT:    call __truncsfbf2@plt
+; RV64-NEXT:    fmv.x.w a0, fa0
+; RV64-NEXT:    lui a1, 1048560
+; RV64-NEXT:    or a0, a0, a1
+; RV64-NEXT:    fmv.w.x fa0, a0
+; RV64-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
+; RV64-NEXT:    addi sp, sp, 16
+; RV64-NEXT:    ret
+;
+; RV32-LABEL: add_constant2:
+; RV32:       # %bb.0:
+; RV32-NEXT:    addi sp, sp, -16
+; RV32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; RV32-NEXT:    fmv.x.w a0, fa0
+; RV32-NEXT:    slli a0, a0, 16
+; RV32-NEXT:    fmv.w.x fa5, a0
+; RV32-NEXT:    lui a0, 260096
+; RV32-NEXT:    fmv.w.x fa4, a0
+; RV32-NEXT:    fadd.s fa0, fa5, fa4
+; RV32-NEXT:    call __truncsfbf2@plt
+; RV32-NEXT:    fmv.x.w a0, fa0
+; RV32-NEXT:    lui a1, 1048560
+; RV32-NEXT:    or a0, a0, a1
+; RV32-NEXT:    fmv.w.x fa0, a0
+; RV32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; RV32-NEXT:    addi sp, sp, 16
+; RV32-NEXT:    ret
+  %add = fadd bfloat %a, 1.0
+  ret bfloat %add
+}
+
+define void @store_constant(ptr %pc) nounwind {
+; RV64-LABEL: store_constant:
+; RV64:       # %bb.0:
+; RV64-NEXT:    lui a1, 4
+; RV64-NEXT:    addiw a1, a1, -128
+; RV64-NEXT:    sh a1, 0(a0)
+; RV64-NEXT:    ret
+;
+; RV32-LABEL: store_constant:
+; RV32:       # %bb.0:
+; RV32-NEXT:    lui a1, 4
+; RV32-NEXT:    addi a1, a1, -128
+; RV32-NEXT:    sh a1, 0(a0)
+; RV32-NEXT:    ret
+  store bfloat 1.0, ptr %pc
+  ret void
+}
+
+define void @fold_ext_trunc(ptr %pa, ptr %pc) nounwind {
+; RV64-LABEL: fold_ext_trunc:
+; RV64:       # %bb.0:
+; RV64-NEXT:    lh a0, 0(a0)
+; RV64-NEXT:    sh a0, 0(a1)
+; RV64-NEXT:    ret
+;
+; RV32-LABEL: fold_ext_trunc:
+; RV32:       # %bb.0:
+; RV32-NEXT:    lh a0, 0(a0)
+; RV32-NEXT:    sh a0, 0(a1)
+; RV32-NEXT:    ret
+  %a = load bfloat, ptr %pa
+  %ext = fpext bfloat %a to float
+  %trunc = fptrunc float %ext to bfloat
+  store bfloat %trunc, ptr %pc
+  ret void
+}
+
+define bfloat @fold_ext_trunc2(bfloat %a) nounwind {
+; RV64-LABEL: fold_ext_trunc2:
+; RV64:       # %bb.0:
+; RV64-NEXT:    fmv.x.w a0, fa0
+; RV64-NEXT:    lui a1, 1048560
+; RV64-NEXT:    or a0, a0, a1
+; RV64-NEXT:    fmv.w.x fa0, a0
+; RV64-NEXT:    ret
+;
+; RV32-LABEL: fold_ext_trunc2:
+; RV32:       # %bb.0:
+; RV32-NEXT:    fmv.x.w a0, fa0
+; RV32-NEXT:    lui a1, 1048560
+; RV32-NEXT:    or a0, a0, a1
+; RV32-NEXT:    fmv.w.x fa0, a0
+; RV32-NEXT:    ret
+  %ext = fpext bfloat %a to float
+  %trunc = fptrunc float %ext to bfloat
+  ret bfloat %trunc
+}
+
+define void @add_double(ptr %pa, ptr %pb, ptr %pc) nounwind {
+; RV64-LABEL: add_double:
+; RV64:       # %bb.0:
+; RV64-NEXT:    addi sp, sp, -32
+; RV64-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; RV64-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; RV64-NEXT:    sd s1, 8(sp) # 8-byte Folded Spill
+; RV64-NEXT:    sd s2, 0(sp) # 8-byte Folded Spill
+; RV64-NEXT:    fld fa0, 0(a0)
+; RV64-NEXT:    mv s0, a2
+; RV64-NEXT:    mv s1, a1
+; RV64-NEXT:    call __truncdfbf2@plt
+; RV64-NEXT:    fmv.x.w a0, fa0
+; RV64-NEXT:    fld fa0, 0(s1)
+; RV64-NEXT:    lui a1, 16
+; RV64-NEXT:    addiw s1, a1, -1
+; RV64-NEXT:    and s2, a0, s1
+; RV64-NEXT:    call __truncdfbf2@plt
+; RV64-NEXT:    fmv.x.w a0, fa0
+; RV64-NEXT:    and a0, a0, s1
+; RV64-NEXT:    slli a0, a0, 16
+; RV64-NEXT:    fmv.w.x fa5, a0
+; RV64-NEXT:    slli s2, s2, 16
+; RV64-NEXT:    fmv.w.x fa4, s2
+; RV64-NEXT:    fadd.s fa0, fa4, fa5
+; RV64-NEXT:    call __truncsfbf2@plt
+; RV64-NEXT:    fmv.x.w a0, fa0
+; RV64-NEXT:    and a0, a0, s1
+; RV64-NEXT:    slli a0, a0, 16
+; RV64-NEXT:    fmv.w.x fa5, a0
+; RV64-NEXT:    fcvt.d.s fa5, fa5
+; RV64-NEXT:    fsd fa5, 0(s0)
+; RV64-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; RV64-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; RV64-NEXT:    ld s1, 8(sp) # 8-byte Folded Reload
+; RV64-NEXT:    ld s2, 0(sp) # 8-byte Folded Reload
+; RV64-NEXT:    addi sp, sp, 32
+; RV64-NEXT:    ret
+;
+; RV32-LABEL: add_double:
+; RV32:       # %bb.0:
+; RV32-NEXT:    addi sp, sp, -16
+; RV32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; RV32-NEXT:    sw s0, 8(sp) # 4-byte Folded Spill
+; RV32-NEXT:    sw s1, 4(sp) # 4-byte Folded Spill
+; RV32-NEXT:    fld fa0, 0(a0)
+; RV32-NEXT:    mv s0, a2
+; RV32-NEXT:    mv s1, a1
+; RV32-NEXT:    call __truncdfbf2@plt
+; RV32-NEXT:    fld fa5, 0(s1)
+; RV32-NEXT:    fmv.x.w s1, fa0
+; RV32-NEXT:    fmv.d fa0, fa5
+; RV32-NEXT:    call __truncdfbf2@plt
+; RV32-NEXT:    fmv.x.w a0, fa0
+; RV32-NEXT:    slli a0, a0, 16
+; RV32-NEXT:    fmv.w.x fa5, a0
+; RV32-NEXT:    slli s1, s1, 16
+; RV32-NEXT:    fmv.w.x fa4, s1
+; RV32-NEXT:    fadd.s fa0, fa4, fa5
+; RV32-NEXT:    call __truncsfbf2@plt
+; RV32-NEXT:    fmv.x.w a0, fa0
+; RV32-NEXT:    slli a0, a0, 16
+; RV32-NEXT:    fmv.w.x fa5, a0
+; RV32-NEXT:    fcvt.d.s fa5, fa5
+; RV32-NEXT:    fsd fa5, 0(s0)
+; RV32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; RV32-NEXT:    lw s0, 8(sp) # 4-byte Folded Reload
+; RV32-NEXT:    lw s1, 4(sp) # 4-byte Folded Reload
+; RV32-NEXT:    addi sp, sp, 16
+; RV32-NEXT:    ret
+  %la = load double, ptr %pa
+  %a = fptrunc double %la to bfloat
+  %lb = load double, ptr %pb
+  %b = fptrunc double %lb to bfloat
+  %add = fadd bfloat %a, %b
+  %dadd = fpext bfloat %add to double
+  store double %dadd, ptr %pc
+  ret void
+}
+
+define double @add_double2(double %da, double %db) nounwind {
+; RV64-LABEL: add_double2:
+; RV64:       # %bb.0:
+; RV64-NEXT:    addi sp, sp, -32
+; RV64-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
+; RV64-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
+; RV64-NEXT:    sd s1, 8(sp) # 8-byte Folded Spill
+; RV64-NEXT:    fsd fs0, 0(sp) # 8-byte Folded Spill
+; RV64-NEXT:    fmv.d fs0, fa1
+; RV64-NEXT:    call __truncdfbf2@plt
+; RV64-NEXT:    fmv.x.w a0, fa0
+; RV64-NEXT:    lui a1, 16
+; RV64-NEXT:    addiw s0, a1, -1
+; RV64-NEXT:    and s1, a0, s0
+; RV64-NEXT:    fmv.d fa0, fs0
+; RV64-NEXT:    call __truncdfbf2@plt
+; RV64-NEXT:    fmv.x.w a0, fa0
+; RV64-NEXT:    and a0, a0, s0
+; RV64-NEXT:    slli a0, a0, 16
+; RV64-NEXT:    fmv.w.x fa5, a0
+; RV64-NEXT:    slli s1, s1, 16
+; RV64-NEXT:    fmv.w.x fa4, s1
+; RV64-NEXT:    fadd.s fa0, fa4, fa5
+; RV64-NEXT:    call __truncsfbf2@plt
+; RV64-NEXT:    fmv.x.w a0, fa0
+; RV64-NEXT:    and a0, a0, s0
+; RV64-NEXT:    slli a0, a0, 16
+; RV64-NEXT:    fmv.w.x fa5, a0
+; RV64-NEXT:    fcvt.d.s fa0, fa5
+; RV64-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
+; RV64-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
+; RV64-NEXT:    ld s1, 8(sp) # 8-byte Folded Reload
+; RV64-NEXT:    fld fs0, 0(sp) # 8-byte Folded Reload
+; RV64-NEXT:    addi sp, sp, 32
+; RV64-NEXT:    ret
+;
+; RV32-LABEL: add_double2:
+; RV32:       # %bb.0:
+; RV32-NEXT:    addi sp, sp, -16
+; RV32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
+; RV32-NEXT:    sw s0, 8(sp) # 4-byte Folded Spill
+; RV32-NEXT:    fsd fs0, 0(sp) # 8-byte Folded Spill
+; RV32-NEXT:    fmv.d fs0, fa1
+; RV32-NEXT:    call __truncdfbf2@plt
+; RV32-NEXT:    fmv.x.w s0, fa0
+; RV32-NEXT:    fmv.d fa0, fs0
+; RV32-NEXT:    call __truncdfbf2@plt
+; RV32-NEXT:    fmv.x.w a0, fa0
+; RV32-NEXT:    slli a0, a0, 16
+; RV32-NEXT:    fmv.w.x fa5, a0
+; RV32-NEXT:    slli s0, s0, 16
+; RV32-NEXT:    fmv.w.x fa4, s0
+; RV32-NEXT:    fadd.s fa0, fa4, fa5
+; RV32-NEXT:    call __truncsfbf2@plt
+; RV32-NEXT:    fmv.x.w a0, fa0
+; RV32-NEXT:    slli a0, a0, 16
+; RV32-NEXT:    fmv.w.x fa5, a0
+; RV32-NEXT:    fcvt.d.s fa0, fa5
+; RV32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
+; RV32-NEXT:    lw s0, 8(sp) # 4-byte Folded Reload
+; RV32-NEXT:    fld fs0, 0(sp) # 8-byte Folded Reload
+; RV32-NEXT:    addi sp, sp, 16
+; RV32-NEXT:    ret
+  %a = fptrunc double %da to bfloat
+  %b = fptrunc double %db to bfloat
+  %add = fadd bfloat %a, %b
+  %dadd = fpext bfloat %add to double
+  ret double %dadd
+}