[RISCV] Add codegen for Zfbfmin instructions

[joshua-arch1]

Summary

The implementation in https://reviews.llvm.org/D151313 is done for the circumstance without Zfbfmin. This patch adds codegen support for the 6 instructions provided in Zfbfmin extension.

[joshua-arch1]

To avoid crash in AArch64, ConstantFP nodes default to expand only in RISCV.

[joshua-arch1]

Rebase, as well as change the operand type in riscv_fpround_bf16/riscv_fpextend_bf16 from f16 to bf16.

[joshua-arch1]

Ping.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1880

Is this change tested?

llvm/lib/Target/RISCV/RISCVInstrInfoZfh.td

457

These patterns duplicate patterns from HasStdExtZfhOrZfhmin, can we share them by using a new predicate?

llvm/lib/Target/RISCV/RISCVRegisterInfo.td

245 ↗

(On Diff #532776)

Can we just add bf16 to the type list for FPR16? If the only reason to have a separate register class is for tablegen type inference, I don't thinks that a good reason.

[asb]

@joshua-arch1 are you still planning to work through the review feedback from Craig? Happy to help if useful.

[joshua-arch1]

In D153234#4493739, @asb wrote:

@joshua-arch1 are you still planning to work through the review feedback from Craig? Happy to help if useful.

@craig.topper Ping.
@asb Sorry for the delay! I have replied to Craig, but there is no feedback again.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1880

define bfloat @bfloat_imm() nounwind {
; CHECKIZFBFMIN-LABEL: bfloat_imm:
; CHECKIZFBFMIN: # %bb.0:
; CHECKIZFBFMIN-NEXT: lui a0, %hi(.LCPI6_0)
; CHECKIZFBFMIN-NEXT: flh fa0, %lo(.LCPI6_0)(a0)
; CHECKIZFBFMIN-NEXT: ret

ret bfloat 3.0

}

In this case, we need this change to address bf16 constant.

llvm/lib/Target/RISCV/RISCVRegisterInfo.td

245 ↗

(On Diff #532776)

I have tried to add bf16 to the type list for FPR16, but that needs to modify almost all the patterns with f16 for type inference, which will be heavy work. Also, bf16 is used only in some special cases. I think addressing this type in an individual part will be better. Mixing bf16 and f16 together will make fp16 instructions hard to maintain. I'm wondering why you don't think this is a good reason.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1880

I thought this function is supposed to return true for constants that don't need a constant pool. Your test for 3.0 is using a constant pool, which I think means this function is returning false for 3.0. Wouldn't that be the behavior you got without this change?

llvm/lib/Target/RISCV/RISCVRegisterInfo.td

245 ↗

(On Diff #532776)

This ends up creating two register classes that are circularly subclasses of each other. I think RISCVInstrInfo::loadRegFromStackSlot for BFPR16 only works because RISCV::FPR16RegClass.hasSubClassEq(RISCV::BFPR16) returns true?

I'm not sure RISCVInstrInfo::copyPhysReg works at all unless Zfh or Zfhmin is also enabled? Is that tested?

[joshua-arch1]

Use predicate HasScalarHalfFPLoadStoreMove for patterns that f16 and bf16 share

[joshua-arch1]

Use the FPR16 register class for bf16 instructions

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfoZfh.td

441

The patterns need to be duplicated for bf16. I think it accidentally works now because the type type profile says they only take f16 types so tablegen optimized out the checks.

We need to relax the types in the type constraints and add new patterns.

def SDT_RISCVFMV_H_X                                                             
    : SDTypeProfile<1, 1, [SDTCisVT<0, f16>, SDTCisVT<1, XLenVT>]>;              
def SDT_RISCVFMV_X_EXTH                                                          
    : SDTypeProfile<1, 1, [SDTCisVT<0, XLenVT>, SDTCisVT<1, f16>]>;

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfoZfh.td

424

Can we put these patterns in the RISCVInstrInfoZfbfmin.td? If the include order doesn't work, I hope we can just fix the include order so that Zfh is included first.

[joshua-arch1]

Move bf16 patterns into RISCVInstrInfoZfbfmin.td

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfoZfbfmin.td

34

There's only supposed to be one SDNode declaration per RISCVISD opcode name. The SDTypeProfile for the existing version needs to be changed to SDTCisFP instead of SDTCisVT<1, f16>. The semantics of the opcode changed, so the type description needs to change accordingly.

[joshua-arch1]

Relax the types in the type constraints for SDT_RISCVFMV_H_X and SDT_RISCV_FMV_X_EXTH

[craig.topper]

LGTM

llvm/lib/Target/RISCV/RISCVInstrInfoZfh.td

456

space after f16

llvm/lib/Target/RISCV/RISCVISelLowering.h

enum NodeType : unsigned {
   FCVT_XU,
   // FP to 32 bit int conversions for RV64. These are used to keep track of the
   // result being sign extended to 64 bit. These saturate out of range inputs.
   // Used for FP_TO_S/UINT and FP_TO_S/UINT_SAT lowering. Rounding mode
   // is passed as a TargetConstant operand using the RISCVFPRndMode enum.
   FCVT_W_RV64,
   FCVT_WU_RV64,
 
+  FP_ROUND_BF16,
+  FP_EXTEND_BF16,
+
   // Rounds an FP value to its corresponding integer in the same FP format.
   // First operand is the value to round, the second operand is the largest
   // integer that can be represented exactly in the FP format. This will be
   // expanded into multiple instructions and basic blocks with a custom
   // inserter.
   FROUND,
 
   FPCLASS,

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

RISCVTargetLowering::RISCVTargetLowering(const TargetMachine &TM,
 
   MVT XLenVT = Subtarget.getXLenVT();
 
   // Set up the register classes.
   addRegisterClass(XLenVT, &RISCV::GPRRegClass);
 
   if (Subtarget.hasStdExtZfhOrZfhmin())
     addRegisterClass(MVT::f16, &RISCV::FPR16RegClass);
+  if (Subtarget.hasStdExtZfbfmin())
+    addRegisterClass(MVT::bf16, &RISCV::FPR16RegClass);
   if (Subtarget.hasStdExtF())
     addRegisterClass(MVT::f32, &RISCV::FPR32RegClass);
   if (Subtarget.hasStdExtD())
     addRegisterClass(MVT::f64, &RISCV::FPR64RegClass);
   if (Subtarget.hasStdExtZhinxOrZhinxmin())
     addRegisterClass(MVT::f16, &RISCV::GPRF16RegClass);
   if (Subtarget.hasStdExtZfinx())
     addRegisterClass(MVT::f32, &RISCV::GPRF32RegClass);
static const unsigned FPOpToExpand[] = {
       ISD::FREM};
 
   static const unsigned FPRndMode[] = {
       ISD::FCEIL, ISD::FFLOOR, ISD::FTRUNC, ISD::FRINT, ISD::FROUND,
       ISD::FROUNDEVEN};
 
   if (Subtarget.hasStdExtZfhOrZfhminOrZhinxOrZhinxmin())
     setOperationAction(ISD::BITCAST, MVT::i16, Custom);
   
+  if (Subtarget.hasStdExtZfbfmin()) {
+    setOperationAction(ISD::BITCAST, MVT::i16, Custom);
+    setOperationAction(ISD::BITCAST, MVT::bf16, Custom);
+    setOperationAction(ISD::FP_ROUND, MVT::bf16, Custom);
+    setOperationAction(ISD::FP_EXTEND, MVT::f32, Custom);
+    setOperationAction(ISD::FP_EXTEND, MVT::f64, Custom);
+    setOperationAction(ISD::ConstantFP, MVT::bf16, Expand);
+  }
+
   if (Subtarget.hasStdExtZfhOrZfhminOrZhinxOrZhinxmin()) {
     if (Subtarget.hasStdExtZfhOrZhinx()) {
       setOperationAction(FPLegalNodeTypes, MVT::f16, Legal);
       setOperationAction(FPRndMode, MVT::f16,
                          Subtarget.hasStdExtZfa() ? Legal : Custom);
       setOperationAction(ISD::SELECT, MVT::f16, Custom);
       setOperationAction(ISD::IS_FPCLASS, MVT::f16, Custom);
     } else {
int RISCVTargetLowering::getLegalZfaFPImm(const APFloat &Imm, EVT VT) const {
   return RISCVLoadFPImm::getLoadFPImm(Imm);
 }
 
 bool RISCVTargetLowering::isFPImmLegal(const APFloat &Imm, EVT VT,
                                        bool ForCodeSize) const {
   bool IsLegalVT = false;
   if (VT == MVT::f16)
     IsLegalVT = Subtarget.hasStdExtZfhOrZfhminOrZhinxOrZhinxmin();
   else if (VT == MVT::f32)
     IsLegalVT = Subtarget.hasStdExtFOrZfinx();
   else if (VT == MVT::f64)
     IsLegalVT = Subtarget.hasStdExtDOrZdinx();
 
   if (!IsLegalVT)
     return false;
 
   if (getLegalZfaFPImm(Imm, VT) >= 0)
case ISD::BITCAST: {
     EVT Op0VT = Op0.getValueType();
     MVT XLenVT = Subtarget.getXLenVT();
     if (VT == MVT::f16 && Op0VT == MVT::i16 &&
         Subtarget.hasStdExtZfhOrZfhminOrZhinxOrZhinxmin()) {
       SDValue NewOp0 = DAG.getNode(ISD::ANY_EXTEND, DL, XLenVT, Op0);
       SDValue FPConv = DAG.getNode(RISCVISD::FMV_H_X, DL, MVT::f16, NewOp0);
       return FPConv;
     }
+    if (VT == MVT::bf16 && Op0VT == MVT::i16 &&
+        Subtarget.hasStdExtZfbfmin()) {
+      SDValue NewOp0 = DAG.getNode(ISD::ANY_EXTEND, DL, XLenVT, Op0);
+      SDValue FPConv = DAG.getNode(RISCVISD::FMV_H_X, DL, MVT::bf16, NewOp0);
+      return FPConv;
+    }
     if (VT == MVT::f32 && Op0VT == MVT::i32 && Subtarget.is64Bit() &&
         Subtarget.hasStdExtFOrZfinx()) {
       SDValue NewOp0 = DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i64, Op0);
       SDValue FPConv =
           DAG.getNode(RISCVISD::FMV_W_X_RV64, DL, MVT::f32, NewOp0);
       return FPConv;
     }
     if (VT == MVT::f64 && Op0VT == MVT::i64 && XLenVT == MVT::i32 &&
if (Op.getValueType() == MVT::f16 && Subtarget.is64Bit() &&
       SDValue Op0 = DAG.getNode(ISD::FP_EXTEND, DL, MVT::f32, Op.getOperand(0));
       SDValue Powi =
           DAG.getNode(ISD::FPOWI, DL, MVT::f32, Op0, Op.getOperand(1));
       return DAG.getNode(ISD::FP_ROUND, DL, MVT::f16, Powi,
                          DAG.getIntPtrConstant(0, DL, /*isTarget=*/true));
     }
     return SDValue();
   }
-  case ISD::FP_EXTEND:
-  case ISD::FP_ROUND:
+  case ISD::FP_EXTEND: {
+    SDLoc DL(Op);
+    EVT VT = Op.getValueType();
+    SDValue Op0 = Op.getOperand(0);
+    EVT Op0VT = Op0.getValueType();
+    if (VT == MVT::f32 && Op0VT == MVT::bf16 && Subtarget.hasStdExtZfbfmin())
+      return DAG.getNode(RISCVISD::FP_EXTEND_BF16, DL, MVT::f32, Op0);
+    if (VT == MVT::f64 && Op0VT == MVT::bf16 && Subtarget.hasStdExtZfbfmin()) {
+      SDValue FloatVal =
+          DAG.getNode(RISCVISD::FP_EXTEND_BF16, DL, MVT::f32, Op0);
+      return DAG.getNode(ISD::FP_EXTEND, DL, MVT::f64, FloatVal);
+    }
+
+    if (!Op.getValueType().isVector())
+      return Op;
+    return lowerVectorFPExtendOrRoundLike(Op, DAG);
+  }
+  case ISD::FP_ROUND: {
+    SDLoc DL(Op);
+    EVT VT = Op.getValueType();
+    SDValue Op0 = Op.getOperand(0);
+    EVT Op0VT = Op0.getValueType();
+    if (VT == MVT::bf16 && Op0VT == MVT::f32 && Subtarget.hasStdExtZfbfmin())
+      return DAG.getNode(RISCVISD::FP_ROUND_BF16, DL, MVT::bf16, Op0);
+    if (VT == MVT::bf16 && Op0VT == MVT::f64 && Subtarget.hasStdExtZfbfmin() &&
+        Subtarget.hasStdExtDOrZdinx()) {
+      SDValue FloatVal =
+          DAG.getNode(ISD::FP_ROUND, DL, MVT::f32, Op0,
+                      DAG.getIntPtrConstant(0, DL, /*isTarget=*/true));
+      return DAG.getNode(RISCVISD::FP_ROUND_BF16, DL, MVT::bf16, FloatVal);
+    }
+
     if (!Op.getValueType().isVector())
       return Op;
     return lowerVectorFPExtendOrRoundLike(Op, DAG);
+  }
   case ISD::STRICT_FP_ROUND:
   case ISD::STRICT_FP_EXTEND:
     return lowerStrictFPExtendOrRoundLike(Op, DAG);
   case ISD::FP_TO_SINT:
   case ISD::FP_TO_UINT:
   case ISD::SINT_TO_FP:
   case ISD::UINT_TO_FP:
   case ISD::STRICT_FP_TO_SINT:
case ISD::BITCAST: {
     assert(VT.isInteger() && !VT.isVector() && "Unexpected VT!");
     SDValue Op0 = N->getOperand(0);
     EVT Op0VT = Op0.getValueType();
     MVT XLenVT = Subtarget.getXLenVT();
     if (VT == MVT::i16 && Op0VT == MVT::f16 &&
         Subtarget.hasStdExtZfhOrZfhminOrZhinxOrZhinxmin()) {
       SDValue FPConv = DAG.getNode(RISCVISD::FMV_X_ANYEXTH, DL, XLenVT, Op0);
       Results.push_back(DAG.getNode(ISD::TRUNCATE, DL, MVT::i16, FPConv));
+    } else if (VT == MVT::i16 && Op0VT == MVT::bf16 &&
+        Subtarget.hasStdExtZfbfmin()) {
+      SDValue FPConv = DAG.getNode(RISCVISD::FMV_X_ANYEXTH, DL, XLenVT, Op0);
+      Results.push_back(DAG.getNode(ISD::TRUNCATE, DL, MVT::i16, FPConv));
     } else if (VT == MVT::i32 && Op0VT == MVT::f32 && Subtarget.is64Bit() &&
                Subtarget.hasStdExtFOrZfinx()) {
       SDValue FPConv =
           DAG.getNode(RISCVISD::FMV_X_ANYEXTW_RV64, DL, MVT::i64, Op0);
       Results.push_back(DAG.getNode(ISD::TRUNCATE, DL, MVT::i32, FPConv));
     } else if (VT == MVT::i64 && Op0VT == MVT::f64 && XLenVT == MVT::i32 &&
                Subtarget.hasStdExtZfa()) {
       SDValue NewReg = DAG.getNode(RISCVISD::SplitF64, DL,
if (State.getFirstUnallocated(ArgFPR32s) == std::size(ArgFPR32s)) {
     UseGPRForF16_F32 = true;
     UseGPRForF64 = true;
   }
 
   // From this point on, rely on UseGPRForF16_F32, UseGPRForF64 and
   // similar local variables rather than directly checking against the target
   // ABI.
 
-  if (UseGPRForF16_F32 && (ValVT == MVT::f16 || ValVT == MVT::f32)) {
+  if (UseGPRForF16_F32 &&
+      (ValVT == MVT::f16 || ValVT == MVT::bf16 || ValVT == MVT::f32)) {
     LocVT = XLenVT;
     LocInfo = CCValAssign::BCvt;
   } else if (UseGPRForF64 && XLen == 64 && ValVT == MVT::f64) {
     LocVT = MVT::i64;
     LocInfo = CCValAssign::BCvt;
   }
 
   // If this is a variadic argument, the RISC-V calling convention requires
return CC_RISCVAssign2XLen(XLen, State, VA, AF, ValNo, ValVT, LocVT,
                                ArgFlags);
   }
 
   // Allocate to a register if possible, or else a stack slot.
   Register Reg;
   unsigned StoreSizeBytes = XLen / 8;
   Align StackAlign = Align(XLen / 8);
 
-  if (ValVT == MVT::f16 && !UseGPRForF16_F32)
+  if ((ValVT == MVT::f16 || ValVT == MVT::bf16) && !UseGPRForF16_F32)
     Reg = State.AllocateReg(ArgFPR16s);
   else if (ValVT == MVT::f32 && !UseGPRForF16_F32)
     Reg = State.AllocateReg(ArgFPR32s);
   else if (ValVT == MVT::f64 && !UseGPRForF64)
     Reg = State.AllocateReg(ArgFPR64s);
   else if (ValVT.isVector()) {
     Reg = allocateRVVReg(ValVT, ValNo, FirstMaskArgument, State, TLI);
     if (!Reg) {
static SDValue convertLocVTToValVT(SelectionDAG &DAG, SDValue Val,
   switch (VA.getLocInfo()) {
   default:
     llvm_unreachable("Unexpected CCValAssign::LocInfo");
   case CCValAssign::Full:
     if (VA.getValVT().isFixedLengthVector() && VA.getLocVT().isScalableVector())
       Val = convertFromScalableVector(VA.getValVT(), Val, DAG, Subtarget);
     break;
   case CCValAssign::BCvt:
-    if (VA.getLocVT().isInteger() && VA.getValVT() == MVT::f16)
-      Val = DAG.getNode(RISCVISD::FMV_H_X, DL, MVT::f16, Val);
+    if (VA.getLocVT().isInteger() &&
+        (VA.getValVT() == MVT::f16 || VA.getValVT() == MVT::bf16))
+      Val = DAG.getNode(RISCVISD::FMV_H_X, DL, VA.getValVT(), Val);
     else if (VA.getLocVT() == MVT::i64 && VA.getValVT() == MVT::f32)
       Val = DAG.getNode(RISCVISD::FMV_W_X_RV64, DL, MVT::f32, Val);
     else
       Val = DAG.getNode(ISD::BITCAST, DL, VA.getValVT(), Val);
     break;
   }
   return Val;
 }
static SDValue convertValVTToLocVT(SelectionDAG &DAG, SDValue Val,
   switch (VA.getLocInfo()) {
   default:
     llvm_unreachable("Unexpected CCValAssign::LocInfo");
   case CCValAssign::Full:
     if (VA.getValVT().isFixedLengthVector() && LocVT.isScalableVector())
       Val = convertToScalableVector(LocVT, Val, DAG, Subtarget);
     break;
   case CCValAssign::BCvt:
-    if (VA.getLocVT().isInteger() && VA.getValVT() == MVT::f16)
+    if (VA.getLocVT().isInteger() &&
+        (VA.getValVT() == MVT::f16 || VA.getValVT() == MVT::bf16))
       Val = DAG.getNode(RISCVISD::FMV_X_ANYEXTH, DL, VA.getLocVT(), Val);
     else if (VA.getLocVT() == MVT::i64 && VA.getValVT() == MVT::f32)
       Val = DAG.getNode(RISCVISD::FMV_X_ANYEXTW_RV64, DL, MVT::i64, Val);
     else
       Val = DAG.getNode(ISD::BITCAST, DL, LocVT, Val);
     break;
   }
   return Val;
#define NODE_NAME_CASE(NODE)                                                   \
   NODE_NAME_CASE(FMV_W_X_RV64)
   NODE_NAME_CASE(FMV_X_ANYEXTW_RV64)
   NODE_NAME_CASE(FCVT_X)
   NODE_NAME_CASE(FCVT_XU)
   NODE_NAME_CASE(FCVT_W_RV64)
   NODE_NAME_CASE(FCVT_WU_RV64)
   NODE_NAME_CASE(STRICT_FCVT_W_RV64)
   NODE_NAME_CASE(STRICT_FCVT_WU_RV64)
+  NODE_NAME_CASE(FP_ROUND_BF16)
+  NODE_NAME_CASE(FP_EXTEND_BF16)
   NODE_NAME_CASE(FROUND)
   NODE_NAME_CASE(FPCLASS)
   NODE_NAME_CASE(READ_CYCLE_WIDE)
   NODE_NAME_CASE(BREV8)
   NODE_NAME_CASE(ORC_B)
   NODE_NAME_CASE(ZIP)
   NODE_NAME_CASE(UNZIP)
   NODE_NAME_CASE(CLMUL)

llvm/lib/Target/RISCV/RISCVInstrInfoZfbfmin.td

 //===-- RISCVInstrInfoZfbfmin.td - 'Zfbfmin' instructions --*- tablegen -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file describes the RISC-V instructions from the standard 'Zfbfmin'
 // extension, providing scalar conversion instructions for BFloat16.
 // This version is still experimental as the 'Zfbfmin' extension hasn't been
 // ratified yet.
 //
 //===----------------------------------------------------------------------===//
 
 //===----------------------------------------------------------------------===//
+// RISC-V specific DAG Nodes.
+//===----------------------------------------------------------------------===//
+ 
+def SDT_RISCVFP_ROUND_BF16
+    : SDTypeProfile<1, 1, [SDTCisVT<0, bf16>, SDTCisVT<1, f32>]>;
+def SDT_RISCVFP_EXTEND_BF16
+    : SDTypeProfile<1, 1, [SDTCisVT<0, f32>, SDTCisVT<1, bf16>]>;
+ 
+def riscv_fpround_bf16
+    : SDNode<"RISCVISD::FP_ROUND_BF16", SDT_RISCVFP_ROUND_BF16>;
+def riscv_fpextend_bf16
+    : SDNode<"RISCVISD::FP_EXTEND_BF16", SDT_RISCVFP_EXTEND_BF16>;
+
+//===----------------------------------------------------------------------===//
 // Instructions
 //===----------------------------------------------------------------------===//
 
 let Predicates = [HasStdExtZfbfmin] in {
 def FCVT_BF16_S : FPUnaryOp_r_frm<0b0100010, 0b01000, FPR16, FPR32, "fcvt.bf16.s">,
                   Sched<[WriteFCvtF32ToF16, ReadFCvtF32ToF16]>;
 def FCVT_S_BF16 : FPUnaryOp_r_frm<0b0100000, 0b00110, FPR32, FPR16, "fcvt.s.bf16">,
                   Sched<[WriteFCvtF32ToF16, ReadFCvtF32ToF16]>;
 } // Predicates = [HasStdExtZfbfmin]
+
+//===----------------------------------------------------------------------===//
+// Pseudo-instructions and codegen patterns
+//===----------------------------------------------------------------------===//
+ 
+let Predicates = [HasStdExtZfbfmin] in {
+/// Loads
+def : LdPat<load, FLH, bf16>;
+
+/// Stores
+def : StPat<store, FSH, FPR16, bf16>;
+
+/// Float conversion operations
+// f32 -> bf16, bf16 -> f32
+def : Pat<(bf16 (riscv_fpround_bf16 FPR32:$rs1)), 
+          (FCVT_BF16_S FPR32:$rs1, FRM_DYN)>;
+def : Pat<(riscv_fpextend_bf16 (bf16 FPR16:$rs1)), 
+          (FCVT_S_BF16 FPR16:$rs1, FRM_DYN)>;
+
+// Moves (no conversion)
+def : Pat<(bf16 (riscv_fmv_h_x GPR:$src)), (FMV_H_X GPR:$src)>;
+def : Pat<(riscv_fmv_x_anyexth (bf16 FPR16:$src)), (FMV_X_H FPR16:$src)>;
+def : Pat<(riscv_fmv_x_signexth (bf16 FPR16:$src)), (FMV_X_H FPR16:$src)>;
+} // Predicates = [HasStdExtZfbfmin]

llvm/lib/Target/RISCV/RISCVInstrInfoZfh.td

 //
 //===----------------------------------------------------------------------===//
 
 //===----------------------------------------------------------------------===//
 // RISC-V specific DAG Nodes.
 //===----------------------------------------------------------------------===//
 
 def SDT_RISCVFMV_H_X
-    : SDTypeProfile<1, 1, [SDTCisVT<0, f16>, SDTCisVT<1, XLenVT>]>;
+    : SDTypeProfile<1, 1, [SDTCisFP<0>, SDTCisVT<1, XLenVT>]>;
 def SDT_RISCVFMV_X_EXTH
-    : SDTypeProfile<1, 1, [SDTCisVT<0, XLenVT>, SDTCisVT<1, f16>]>;
+    : SDTypeProfile<1, 1, [SDTCisVT<0, XLenVT>, SDTCisFP<1>]>;
 
 def riscv_fmv_h_x
     : SDNode<"RISCVISD::FMV_H_X", SDT_RISCVFMV_H_X>;
 def riscv_fmv_x_anyexth
     : SDNode<"RISCVISD::FMV_X_ANYEXTH", SDT_RISCVFMV_X_EXTH>;
 def riscv_fmv_x_signexth
     : SDNode<"RISCVISD::FMV_X_SIGNEXTH", SDT_RISCVFMV_X_EXTH>;
 

 let Predicates = [HasStdExtZfhOrZfhmin] in {
 /// Loads
 def : LdPat<load, FLH, f16>;
 
 /// Stores
 def : StPat<store, FSH, FPR16, f16>;
 } // Predicates = [HasStdExtZfhOrZfhmin]
 
 let Predicates = [HasStdExtZhinxOrZhinxmin] in {
 /// Loads
 def : Pat<(f16 (load GPR:$rs1)), (COPY_TO_REGCLASS (LH GPR:$rs1, 0), GPRF16)>;
 
 /// Stores
 def : Pat<(store (f16 FPR16INX:$rs2), GPR:$rs1),
           (SH (COPY_TO_REGCLASS FPR16INX:$rs2, GPR), GPR:$rs1, 0)>;
 } // Predicates = [HasStdExtZhinxOrZhinxmin]
 
 let Predicates = [HasStdExtZfhOrZfhmin] in {
 /// Float conversion operations
 
 // f32 -> f16, f16 -> f32
 def : Pat<(f16 (any_fpround FPR32:$rs1)), (FCVT_H_S FPR32:$rs1, FRM_DYN)>;
 def : Pat<(any_fpextend (f16 FPR16:$rs1)), (FCVT_S_H FPR16:$rs1)>;
 
 // Moves (no conversion)
-def : Pat<(riscv_fmv_h_x GPR:$src), (FMV_H_X GPR:$src)>;
+def : Pat<(f16 (riscv_fmv_h_x GPR:$src)), (FMV_H_X GPR:$src)>;
 def : Pat<(riscv_fmv_x_anyexth (f16 FPR16:$src)), (FMV_X_H FPR16:$src)>;
 def : Pat<(riscv_fmv_x_signexth (f16 FPR16:$src)), (FMV_X_H FPR16:$src)>;
 
 def : Pat<(fcopysign FPR32:$rs1, (f16 FPR16:$rs2)), (FSGNJ_S $rs1, (FCVT_S_H $rs2))>;
 } // Predicates = [HasStdExtZfhOrZfhmin]
 
 let Predicates = [HasStdExtZhinxOrZhinxmin] in {
 /// Float conversion operations
 
 // f32 -> f16, f16 -> f32
 def : Pat<(any_fpround FPR32INX:$rs1), (FCVT_H_S_INX FPR32INX:$rs1, FRM_DYN)>;
 def : Pat<(any_fpextend FPR16INX:$rs1), (FCVT_S_H_INX FPR16INX:$rs1)>;
 
 // Moves (no conversion)
-def : Pat<(riscv_fmv_h_x GPR:$src), (COPY_TO_REGCLASS GPR:$src, GPR)>;
+def : Pat<(f16 (riscv_fmv_h_x GPR:$src)), (COPY_TO_REGCLASS GPR:$src, GPR)>;
 def : Pat<(riscv_fmv_x_anyexth FPR16INX:$src), (COPY_TO_REGCLASS FPR16INX:$src, GPR)>;
 def : Pat<(riscv_fmv_x_signexth FPR16INX:$src), (COPY_TO_REGCLASS FPR16INX:$src, GPR)>;
 
 def : Pat<(fcopysign FPR32INX:$rs1, FPR16INX:$rs2), (FSGNJ_S_INX $rs1, (FCVT_S_H_INX $rs2))>;
 } // Predicates = [HasStdExtZhinxOrZhinxmin]
 
 let Predicates = [HasStdExtZfh, IsRV32] in {
 // half->[u]int. Round-to-zero must be used.
 def : Pat<(i32 (any_fp_to_sint (f16 FPR16:$rs1))), (FCVT_W_H $rs1, 0b001)>;

llvm/test/CodeGen/RISCV/zfbfmin.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+d,+zfh,+experimental-zfbfmin -verify-machineinstrs \
+; RUN:   -target-abi ilp32d < %s | FileCheck -check-prefix=CHECKIZFBFMIN %s
+; RUN: llc -mtriple=riscv64 -mattr=+d,+zfh,+experimental-zfbfmin -verify-machineinstrs \
+; RUN:   -target-abi lp64d < %s | FileCheck -check-prefix=CHECKIZFBFMIN %s
+
+define bfloat @bitcast_bf16_i16(i16 %a) nounwind {
+; CHECKIZFBFMIN-LABEL: bitcast_bf16_i16:
+; CHECKIZFBFMIN:       # %bb.0:
+; CHECKIZFBFMIN-NEXT:    fmv.h.x fa0, a0
+; CHECKIZFBFMIN-NEXT:    ret
+  %1 = bitcast i16 %a to bfloat
+  ret bfloat %1
+}
+
+define i16 @bitcast_i16_bf16(bfloat %a) nounwind {
+; CHECKIZFBFMIN-LABEL: bitcast_i16_bf16:
+; CHECKIZFBFMIN:       # %bb.0:
+; CHECKIZFBFMIN-NEXT:    fmv.x.h a0, fa0
+; CHECKIZFBFMIN-NEXT:    ret
+  %1 = bitcast bfloat %a to i16
+  ret i16 %1
+}
+
+define bfloat @fcvt_bf16_s(float %a) nounwind {
+; CHECKIZFBFMIN-LABEL: fcvt_bf16_s:
+; CHECKIZFBFMIN:       # %bb.0:
+; CHECKIZFBFMIN-NEXT:    fcvt.bf16.s fa0, fa0
+; CHECKIZFBFMIN-NEXT:    ret
+  %1 = fptrunc float %a to bfloat
+  ret bfloat %1
+}
+
+define float @fcvt_s_bf16(bfloat %a) nounwind {
+; CHECKIZFBFMIN-LABEL: fcvt_s_bf16:
+; CHECKIZFBFMIN:       # %bb.0:
+; CHECKIZFBFMIN-NEXT:    fcvt.s.bf16 fa0, fa0
+; CHECKIZFBFMIN-NEXT:    ret
+  %1 = fpext bfloat %a to float
+  ret float %1
+}
+
+define bfloat @fcvt_bf16_d(double %a) nounwind {
+; CHECKIZFBFMIN-LABEL: fcvt_bf16_d:
+; CHECKIZFBFMIN:       # %bb.0:
+; CHECKIZFBFMIN-NEXT:    fcvt.s.d fa5, fa0
+; CHECKIZFBFMIN-NEXT:    fcvt.bf16.s fa0, fa5
+; CHECKIZFBFMIN-NEXT:    ret
+  %1 = fptrunc double %a to bfloat
+  ret bfloat %1
+}
+
+define double @fcvt_d_bf16(bfloat %a) nounwind {
+; CHECKIZFBFMIN-LABEL: fcvt_d_bf16:
+; CHECKIZFBFMIN:       # %bb.0:
+; CHECKIZFBFMIN-NEXT:    fcvt.s.bf16 fa5, fa0
+; CHECKIZFBFMIN-NEXT:    fcvt.d.s fa0, fa5
+; CHECKIZFBFMIN-NEXT:    ret
+  %1 = fpext bfloat %a to double
+  ret double %1
+}
+
+define bfloat @bfloat_load(ptr %a) nounwind {
+; CHECKIZFBFMIN-LABEL: bfloat_load:
+; CHECKIZFBFMIN:       # %bb.0:
+; CHECKIZFBFMIN-NEXT:    flh fa0, 6(a0)
+; CHECKIZFBFMIN-NEXT:    ret
+  %1 = getelementptr bfloat, ptr %a, i32 3
+  %2 = load bfloat, ptr %1
+  ret bfloat %2
+}
+
+define bfloat @bfloat_imm() nounwind {
+; CHECKIZFBFMIN-LABEL: bfloat_imm:
+; CHECKIZFBFMIN:       # %bb.0:
+; CHECKIZFBFMIN-NEXT:    lui a0, %hi(.LCPI7_0)
+; CHECKIZFBFMIN-NEXT:    flh fa0, %lo(.LCPI7_0)(a0)
+; CHECKIZFBFMIN-NEXT:    ret
+  ret bfloat 3.0
+}
+
+define dso_local void @bfloat_store(ptr %a, bfloat %b) nounwind {
+; CHECKIZFBFMIN-LABEL: bfloat_store:
+; CHECKIZFBFMIN:       # %bb.0:
+; CHECKIZFBFMIN-NEXT:    fsh fa0, 0(a0)
+; CHECKIZFBFMIN-NEXT:    fsh fa0, 16(a0)
+; CHECKIZFBFMIN-NEXT:    ret
+  store bfloat %b, ptr %a
+  %1 = getelementptr bfloat, ptr %a, i32 8
+  store bfloat %b, ptr %1
+  ret void
+}