[RISCV][BF16] Enable __bf16 for riscv targets

[joshua-arch1]

Summary

The RISC-V psABI recently added __bf16 in https://github.com/riscv-non-isa/riscv-elf-psabi-doc/pull/367.
Now we can enable this new type in clang.

[craig.topper]

This test seems to crash in the backend with -march=rv64gc -mabi=lp64d

 __bf16 h1(__bf16 a) {
  return a;
}

clang/docs/LanguageExtensions.rst

807

RISC-V

[joshua-arch1]

In D150929#4355303, @craig.topper wrote:

This test seems to crash in the backend with -march=rv64gc -mabi=lp64d

 __bf16 h1(__bf16 a) {
  return a;
}

I haven't done any backend support fo bf16 yet. What I want to do in this patch is only to enable __bf16 for clang.

[craig.topper]

In D150929#4355353, @joshua-arch1 wrote:

In D150929#4355303, @craig.topper wrote:

This test seems to crash in the backend with -march=rv64gc -mabi=lp64d

 __bf16 h1(__bf16 a) {
  return a;
}

I haven't done any backend support fo bf16 yet. What I want to do in this patch is only to enable __bf16 for clang.

We can’t have the compiler crashing if you use it. The backend has to go first.

[joshua-arch1]

In D150929#4355360, @craig.topper wrote:

In D150929#4355353, @joshua-arch1 wrote:

In D150929#4355303, @craig.topper wrote:

This test seems to crash in the backend with -march=rv64gc -mabi=lp64d

 __bf16 h1(__bf16 a) {
  return a;
}

I haven't done any backend support fo bf16 yet. What I want to do in this patch is only to enable __bf16 for clang.

We can’t have the compiler crashing if you use it. The backend has to go first.

But we do not have any intrinsics to generate bf16 instructions right now.

[craig.topper]

In D150929#4355368, @joshua-arch1 wrote:

In D150929#4355360, @craig.topper wrote:

In D150929#4355353, @joshua-arch1 wrote:

In D150929#4355303, @craig.topper wrote:

This test seems to crash in the backend with -march=rv64gc -mabi=lp64d

 __bf16 h1(__bf16 a) {
  return a;
}

I haven't done any backend support fo bf16 yet. What I want to do in this patch is only to enable __bf16 for clang.

We can’t have the compiler crashing if you use it. The backend has to go first.

But we do not have any intrinsics to generate bf16 instructions right now.

That doesn’t matter. The test I sent crashes, thats not acceptable.

Probably also means the ABI is not being implemented correctly.

[joshua-arch1]

That doesn’t matter. The test I sent crashes, thats not acceptable.

Probably also means the ABI is not being implemented correctly.

I have checked the log where it crashes. That's just because there are no corresponding instructions to address bf16 type in instruction selection. Everything is correct before instruction selection.

[craig.topper]

In D150929#4355443, @joshua-arch1 wrote:

That doesn’t matter. The test I sent crashes, thats not acceptable.

Probably also means the ABI is not being implemented correctly.

I have checked the log where it crashes. That's just because there are no corresponding instructions to address bf16 type in instruction selection. Everything is correct before instruction selection.

It created an extending load from bfloat because it tried to return as an f32. That’s not the ABI. It should be in the lower 16 bits of an FP register.

[kito-cheng]

clang/lib/Basic/Targets/RISCV.h

118

Should be DF16b since we intend to use that for std::bfloat16_t.

clang/test/Sema/vector-decl-crash.c

0

What??? we should just remove this testcase IF we really support __bf16 IMO.

[joshua-arch1]

Just like AArch64 and X86, I have updated my implementation by limiting bfloat16 only when Zfbfmin/Zvfbfmin/Zvfbfwma are enabled. Now the compiler can work for bfloat16 programs without crashing.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

435

This needs an llvm/test/CodeGen/RISCV test.

[asb]

I feel (IMHO) this is jumping to the endpoint a bit - the usual route for something like this is:

• MC layer support
• LLVM codegen support (and tests!)
• Any needed Clang support

I was hoping to push out some patches related to zfbfmin tests+codegen today but might time-out (if so, I'll return to it tomorrow).

I think the tentative conclusion of https://discourse.llvm.org/t/rfc-c-23-p1467r9-extended-floating-point-types-and-standard-names/70033 is to make __bf16_t an arithmetic type on all targets - I'd recommend keeping an eye on https://reviews.llvm.org/D150913 (and ideally helping to review it), as that's bound to interact with this patch.

[joshua-arch1]

This comment was removed by joshua-arch1.

clang/test/Sema/vector-decl-crash.c

0

Actually, I didn't want to change riscv64 into another target. However, I see someone directly changed x86 into riscv64 when he supported __bf16.

[joshua-arch1]

In D150929#4361151, @asb wrote:

I feel (IMHO) this is jumping to the endpoint a bit - the usual route for something like this is:

• MC layer support
• LLVM codegen support (and tests!)
• Any needed Clang support

I was hoping to push out some patches related to zfbfmin tests+codegen today but might time-out (if so, I'll return to it tomorrow).

I think the tentative conclusion of https://discourse.llvm.org/t/rfc-c-23-p1467r9-extended-floating-point-types-and-standard-names/70033 is to make __bf16_t an arithmetic type on all targets - I'd recommend keeping an eye on https://reviews.llvm.org/D150913 (and ideally helping to review it), as that's bound to interact with this patch.

Have you started the codegen part yet? I have basically tried but it may be a little troublesome since we use FPR16 for both fp16 and bf16. During instruction selection, it will be difficult to match those instructions fp16 and bf16 share.

[MaskRay]

The ABI information for __bf16 has been approved in RISCV psABI and is to be merged. Now we can enable this new type in clang.
See https://github.com/riscv-non-isa/riscv-elf-psabi-doc/pull/367

The description seems verbose and inaccurate now. You can say:

The RISC-V psABI recently added __bf16 in https://github.com/riscv-non-isa/riscv-elf-psabi-doc/pull/367

[joshua-arch1]

Since bf16 has been supported in backend, we can now come back to this clang patch. @craig.topper

[joshua-arch1]

Ping.

[craig.topper]

This test crashes

float foo(__bf16 a, __bf16 b)  {
  return a + b;
}

error in backend: Cannot select: t35: f32,ch = load<(dereferenceable load (s16) from %ir.b.addr), anyext from bf16> t49, FrameIndex:i32<1>, undef:i32

clang/docs/LanguageExtensions.rst

776–777

Put this above X86 so that the other X86 comment below stays together.

[joshua-arch1]

In D150929#4545521, @craig.topper wrote:

This test crashes

float foo(__bf16 a, __bf16 b)  {
  return a + b;
}

error in backend: Cannot select: t35: f32,ch = load<(dereferenceable load (s16) from %ir.b.addr), anyext from bf16> t49, FrameIndex:i32<1>, undef:i32

I have fixed this backend error in the patch https://reviews.llvm.org/D156646.

[craig.topper]

LGTM

clang/docs/LanguageExtensions.rst

void f2() {
     matrix_5_5<double> d;
     matrix_5_5<int> i;
     i = (matrix_5_5<int>)d;
     i = static_cast<matrix_5_5<int>>(d);
   }
 
 Half-Precision Floating Point
 =============================
 
 Clang supports three half-precision (16-bit) floating point types: ``__fp16``,
 ``_Float16`` and ``__bf16``.  These types are supported in all language modes.
 
 ``__fp16`` is supported on every target, as it is purely a storage format; see below.
 ``_Float16`` is currently only supported on the following targets, with further
 targets pending ABI standardization:
 
 * 32-bit ARM
 * 64-bit ARM (AArch64)

 
 ``_Float16`` will be supported on more targets as they define ABIs for it.
 
 ``__bf16`` is purely a storage format; it is currently only supported on the following targets:
 
 * 32-bit ARM
 * 64-bit ARM (AArch64)
 * X86 (see below)
+* RISC-V
 
 On X86 targets, ``__bf16`` is supported as long as SSE2 is available, which
 includes all 64-bit and all recent 32-bit processors.
 
 ``__fp16`` is a storage and interchange format only.  This means that values of
 ``__fp16`` are immediately promoted to (at least) ``float`` when used in arithmetic
 operations, so that e.g. the result of adding two ``__fp16`` values has type ``float``.
 The behavior of ``__fp16`` is specified by the Arm C Language Extensions (`ACLE <https://github.com/ARM-software/acle/releases>`_).

clang/lib/Basic/Targets/RISCV.h

 class RISCVTargetInfo : public TargetInfo {
 protected:
   std::string ABI, CPU;
   std::unique_ptr<llvm::RISCVISAInfo> ISAInfo;
 
 public:
   RISCVTargetInfo(const llvm::Triple &Triple, const TargetOptions &)
       : TargetInfo(Triple) {
+    BFloat16Width = 16;
+    BFloat16Align = 16;
+    BFloat16Format = &llvm::APFloat::BFloat();
     LongDoubleWidth = 128;
     LongDoubleAlign = 128;
     LongDoubleFormat = &llvm::APFloat::IEEEquad();
     SuitableAlign = 128;
     WCharType = SignedInt;
     WIntType = UnsignedInt;
     HasRISCVVTypes = true;
     MCountName = "_mcount";
public:
 
   bool hasFeature(StringRef Feature) const override;
 
   bool handleTargetFeatures(std::vector<std::string> &Features,
                             DiagnosticsEngine &Diags) override;
 
   bool hasBitIntType() const override { return true; }
 
+  bool hasBFloat16Type() const override;
+
   bool useFP16ConversionIntrinsics() const override {
     return false;
   }
 
   bool isValidCPUName(StringRef Name) const override;
   void fillValidCPUList(SmallVectorImpl<StringRef> &Values) const override;
   bool isValidTuneCPUName(StringRef Name) const override;
   void fillValidTuneCPUList(SmallVectorImpl<StringRef> &Values) const override;
+
+  const char *getBFloat16Mangling() const override { return "DF16b"; };
 };
 class LLVM_LIBRARY_VISIBILITY RISCV32TargetInfo : public RISCVTargetInfo {
 public:
   RISCV32TargetInfo(const llvm::Triple &Triple, const TargetOptions &Opts)
       : RISCVTargetInfo(Triple, Opts) {
     IntPtrType = SignedInt;
     PtrDiffType = SignedInt;
     SizeType = UnsignedInt;

clang/lib/Basic/Targets/RISCV.cpp

if (Result)
     return *Result;
 
   if (ISAInfo->isSupportedExtensionFeature(Feature))
     return ISAInfo->hasExtension(Feature);
 
   return false;
 }
 
+bool RISCVTargetInfo::hasBFloat16Type() const { return HasBFloat16; }
+
 /// Perform initialization based on the user configured set of features.
 bool RISCVTargetInfo::handleTargetFeatures(std::vector<std::string> &Features,
                                            DiagnosticsEngine &Diags) {
   unsigned XLen = getTriple().isArch64Bit() ? 64 : 32;
   auto ParseResult = llvm::RISCVISAInfo::parseFeatures(XLen, Features);
   if (!ParseResult) {
     std::string Buffer;
     llvm::raw_string_ostream OutputErrMsg(Buffer);
     handleAllErrors(ParseResult.takeError(), [&](llvm::StringError &ErrMsg) {
       OutputErrMsg << ErrMsg.getMessage();
     });
     Diags.Report(diag::err_invalid_feature_combination) << OutputErrMsg.str();
     return false;
   } else {
     ISAInfo = std::move(*ParseResult);
   }
 
   if (ABI.empty())
     ABI = ISAInfo->computeDefaultABI().str();
 
   if (ISAInfo->hasExtension("zfh") || ISAInfo->hasExtension("zhinx"))
     HasLegalHalfType = true;
 
+  if (ISAInfo->hasExtension("experimental-zfbfmin") ||
+      ISAInfo->hasExtension("experimental-zvfbfmin") ||
+      ISAInfo->hasExtension("experimental-zvfbfwma"))
+    HasBFloat16 = true;
+
   return true;
 }
 
 bool RISCVTargetInfo::isValidCPUName(StringRef Name) const {
   bool Is64Bit = getTriple().isArch64Bit();
   return llvm::RISCV::parseCPU(Name, Is64Bit);
 }
 

clang/test/CodeGen/RISCV/bfloat-abi.c

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --version 2
+// RUN: %clang_cc1 -triple riscv64 -target-feature +experimental-zfbfmin -emit-llvm %s -o - | FileCheck %s --check-prefixes=CHECK
+// RUN: %clang_cc1 -triple riscv64 -target-feature +experimental-zvfbfmin -emit-llvm %s -o - | FileCheck %s --check-prefixes=CHECK-V
+// RUN: %clang_cc1 -triple riscv64 -target-feature +experimental-zvfbfwma -emit-llvm %s -o - | FileCheck %s --check-prefixes=CHECK-MA
+
+struct bfloat1 {
+  __bf16 a;
+};
+
+// CHECK-LABEL: define dso_local i64 @h1
+// CHECK-SAME: (bfloat noundef [[A:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT1:%.*]], align 2
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[COERCE_DIVE_COERCE:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT1]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-NEXT:    [[COERCE_DIVE:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT1]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[COERCE_DIVE_COERCE]], ptr align 2 [[COERCE_DIVE]], i64 2, i1 false)
+// CHECK-NEXT:    [[TMP1:%.*]] = load i64, ptr [[COERCE_DIVE_COERCE]], align 8
+// CHECK-NEXT:    ret i64 [[TMP1]]
+//
+// CHECK-V-LABEL: define dso_local i64 @h1
+// CHECK-V-SAME: (bfloat noundef [[A:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-V-NEXT:  entry:
+// CHECK-V-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT1:%.*]], align 2
+// CHECK-V-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[COERCE_DIVE_COERCE:%.*]] = alloca i64, align 8
+// CHECK-V-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT1]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-V-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-V-NEXT:    [[COERCE_DIVE:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT1]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-V-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[COERCE_DIVE_COERCE]], ptr align 2 [[COERCE_DIVE]], i64 2, i1 false)
+// CHECK-V-NEXT:    [[TMP1:%.*]] = load i64, ptr [[COERCE_DIVE_COERCE]], align 8
+// CHECK-V-NEXT:    ret i64 [[TMP1]]
+//
+// CHECK-MA-LABEL: define dso_local i64 @h1
+// CHECK-MA-SAME: (bfloat noundef [[A:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-MA-NEXT:  entry:
+// CHECK-MA-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT1:%.*]], align 2
+// CHECK-MA-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[COERCE_DIVE_COERCE:%.*]] = alloca i64, align 8
+// CHECK-MA-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT1]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-MA-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-MA-NEXT:    [[COERCE_DIVE:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT1]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-MA-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[COERCE_DIVE_COERCE]], ptr align 2 [[COERCE_DIVE]], i64 2, i1 false)
+// CHECK-MA-NEXT:    [[TMP1:%.*]] = load i64, ptr [[COERCE_DIVE_COERCE]], align 8
+// CHECK-MA-NEXT:    ret i64 [[TMP1]]
+//
+struct bfloat1 h1(__bf16 a) {
+  struct bfloat1 x;
+  x.a = a;
+  return x;
+}
+
+struct bfloat2 {
+  __bf16 a;
+  __bf16 b;
+};
+
+// CHECK-LABEL: define dso_local i64 @h2
+// CHECK-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT2:%.*]], align 2
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[RETVAL_COERCE:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 2 [[RETVAL]], i64 4, i1 false)
+// CHECK-NEXT:    [[TMP2:%.*]] = load i64, ptr [[RETVAL_COERCE]], align 8
+// CHECK-NEXT:    ret i64 [[TMP2]]
+//
+// CHECK-V-LABEL: define dso_local i64 @h2
+// CHECK-V-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-V-NEXT:  entry:
+// CHECK-V-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT2:%.*]], align 2
+// CHECK-V-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[RETVAL_COERCE:%.*]] = alloca i64, align 8
+// CHECK-V-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-V-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-V-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-V-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-V-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 2 [[RETVAL]], i64 4, i1 false)
+// CHECK-V-NEXT:    [[TMP2:%.*]] = load i64, ptr [[RETVAL_COERCE]], align 8
+// CHECK-V-NEXT:    ret i64 [[TMP2]]
+//
+// CHECK-MA-LABEL: define dso_local i64 @h2
+// CHECK-MA-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-MA-NEXT:  entry:
+// CHECK-MA-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT2:%.*]], align 2
+// CHECK-MA-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[RETVAL_COERCE:%.*]] = alloca i64, align 8
+// CHECK-MA-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-MA-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-MA-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-MA-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-MA-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 2 [[RETVAL]], i64 4, i1 false)
+// CHECK-MA-NEXT:    [[TMP2:%.*]] = load i64, ptr [[RETVAL_COERCE]], align 8
+// CHECK-MA-NEXT:    ret i64 [[TMP2]]
+//
+struct bfloat2 h2(__bf16 a, __bf16 b) {
+  struct bfloat2 x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct bfloat3 {
+  __bf16 a;
+  __bf16 b;
+  __bf16 c;
+};
+
+// CHECK-LABEL: define dso_local i64 @h3
+// CHECK-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT3:%.*]], align 2
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[RETVAL_COERCE:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 2 [[RETVAL]], i64 6, i1 false)
+// CHECK-NEXT:    [[TMP3:%.*]] = load i64, ptr [[RETVAL_COERCE]], align 8
+// CHECK-NEXT:    ret i64 [[TMP3]]
+//
+// CHECK-V-LABEL: define dso_local i64 @h3
+// CHECK-V-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-V-NEXT:  entry:
+// CHECK-V-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT3:%.*]], align 2
+// CHECK-V-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[RETVAL_COERCE:%.*]] = alloca i64, align 8
+// CHECK-V-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-V-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-V-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-V-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-V-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-V-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-V-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-V-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-V-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 2 [[RETVAL]], i64 6, i1 false)
+// CHECK-V-NEXT:    [[TMP3:%.*]] = load i64, ptr [[RETVAL_COERCE]], align 8
+// CHECK-V-NEXT:    ret i64 [[TMP3]]
+//
+// CHECK-MA-LABEL: define dso_local i64 @h3
+// CHECK-MA-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-MA-NEXT:  entry:
+// CHECK-MA-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT3:%.*]], align 2
+// CHECK-MA-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[RETVAL_COERCE:%.*]] = alloca i64, align 8
+// CHECK-MA-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-MA-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-MA-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-MA-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-MA-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-MA-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-MA-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-MA-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-MA-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 2 [[RETVAL]], i64 6, i1 false)
+// CHECK-MA-NEXT:    [[TMP3:%.*]] = load i64, ptr [[RETVAL_COERCE]], align 8
+// CHECK-MA-NEXT:    ret i64 [[TMP3]]
+//
+struct bfloat3 h3(__bf16 a, __bf16 b, __bf16 c) {
+  struct bfloat3 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct bfloat4 {
+  __bf16 a;
+  __bf16 b;
+  __bf16 c;
+  __bf16 d;
+};
+
+// CHECK-LABEL: define dso_local i64 @h4
+// CHECK-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]]) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT4:%.*]], align 2
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 3
+// CHECK-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 2
+// CHECK-NEXT:    [[TMP4:%.*]] = load i64, ptr [[RETVAL]], align 2
+// CHECK-NEXT:    ret i64 [[TMP4]]
+//
+// CHECK-V-LABEL: define dso_local i64 @h4
+// CHECK-V-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]]) #[[ATTR0]] {
+// CHECK-V-NEXT:  entry:
+// CHECK-V-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT4:%.*]], align 2
+// CHECK-V-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-V-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-V-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-V-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-V-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-V-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-V-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-V-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-V-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-V-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 3
+// CHECK-V-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 2
+// CHECK-V-NEXT:    [[TMP4:%.*]] = load i64, ptr [[RETVAL]], align 2
+// CHECK-V-NEXT:    ret i64 [[TMP4]]
+//
+// CHECK-MA-LABEL: define dso_local i64 @h4
+// CHECK-MA-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]]) #[[ATTR0]] {
+// CHECK-MA-NEXT:  entry:
+// CHECK-MA-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT4:%.*]], align 2
+// CHECK-MA-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-MA-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-MA-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-MA-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-MA-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-MA-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-MA-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-MA-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-MA-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-MA-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 3
+// CHECK-MA-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 2
+// CHECK-MA-NEXT:    [[TMP4:%.*]] = load i64, ptr [[RETVAL]], align 2
+// CHECK-MA-NEXT:    ret i64 [[TMP4]]
+//
+struct bfloat4 h4(__bf16 a, __bf16 b, __bf16 c, __bf16 d) {
+  struct bfloat4 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  x.d = d;
+  return x;
+}
+
+struct floatbfloat {
+  float a;
+  __bf16 b;
+};
+
+// CHECK-LABEL: define dso_local i64 @fh
+// CHECK-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT:%.*]], align 4
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-NEXT:    ret i64 [[TMP2]]
+//
+// CHECK-V-LABEL: define dso_local i64 @fh
+// CHECK-V-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-V-NEXT:  entry:
+// CHECK-V-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT:%.*]], align 4
+// CHECK-V-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-V-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-V-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-V-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-V-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-V-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-V-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-V-NEXT:    [[TMP2:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-V-NEXT:    ret i64 [[TMP2]]
+//
+// CHECK-MA-LABEL: define dso_local i64 @fh
+// CHECK-MA-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-MA-NEXT:  entry:
+// CHECK-MA-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT:%.*]], align 4
+// CHECK-MA-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-MA-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-MA-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-MA-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-MA-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-MA-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-MA-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-MA-NEXT:    [[TMP2:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-MA-NEXT:    ret i64 [[TMP2]]
+//
+struct floatbfloat fh(float a, __bf16 b) {
+  struct floatbfloat x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct floatbfloat2 {
+  float a;
+  __bf16 b;
+  __bf16 c;
+};
+
+// CHECK-LABEL: define dso_local i64 @fh2
+// CHECK-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT2:%.*]], align 4
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-NEXT:    [[TMP3:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-NEXT:    ret i64 [[TMP3]]
+//
+// CHECK-V-LABEL: define dso_local i64 @fh2
+// CHECK-V-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-V-NEXT:  entry:
+// CHECK-V-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT2:%.*]], align 4
+// CHECK-V-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-V-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-V-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-V-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-V-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-V-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-V-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-V-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-V-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-V-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-V-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-V-NEXT:    [[TMP3:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-V-NEXT:    ret i64 [[TMP3]]
+//
+// CHECK-MA-LABEL: define dso_local i64 @fh2
+// CHECK-MA-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-MA-NEXT:  entry:
+// CHECK-MA-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT2:%.*]], align 4
+// CHECK-MA-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-MA-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-MA-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-MA-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-MA-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-MA-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-MA-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-MA-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-MA-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-MA-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-MA-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-MA-NEXT:    [[TMP3:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-MA-NEXT:    ret i64 [[TMP3]]
+//
+struct floatbfloat2 fh2(float a, __bf16 b, __bf16 c) {
+  struct floatbfloat2 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct bfloatfloat {
+  __bf16 a;
+  float b;
+};
+
+// CHECK-LABEL: define dso_local i64 @hf
+// CHECK-SAME: (bfloat noundef [[A:%.*]], float noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOATFLOAT:%.*]], align 4
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca float, align 4
+// CHECK-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    store float [[B]], ptr [[B_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOATFLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load float, ptr [[B_ADDR]], align 4
+// CHECK-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOATFLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-NEXT:    store float [[TMP1]], ptr [[B2]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-NEXT:    ret i64 [[TMP2]]
+//
+// CHECK-V-LABEL: define dso_local i64 @hf
+// CHECK-V-SAME: (bfloat noundef [[A:%.*]], float noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-V-NEXT:  entry:
+// CHECK-V-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOATFLOAT:%.*]], align 4
+// CHECK-V-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[B_ADDR:%.*]] = alloca float, align 4
+// CHECK-V-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    store float [[B]], ptr [[B_ADDR]], align 4
+// CHECK-V-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOATFLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-V-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 4
+// CHECK-V-NEXT:    [[TMP1:%.*]] = load float, ptr [[B_ADDR]], align 4
+// CHECK-V-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOATFLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-V-NEXT:    store float [[TMP1]], ptr [[B2]], align 4
+// CHECK-V-NEXT:    [[TMP2:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-V-NEXT:    ret i64 [[TMP2]]
+//
+// CHECK-MA-LABEL: define dso_local i64 @hf
+// CHECK-MA-SAME: (bfloat noundef [[A:%.*]], float noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-MA-NEXT:  entry:
+// CHECK-MA-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOATFLOAT:%.*]], align 4
+// CHECK-MA-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[B_ADDR:%.*]] = alloca float, align 4
+// CHECK-MA-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    store float [[B]], ptr [[B_ADDR]], align 4
+// CHECK-MA-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOATFLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-MA-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 4
+// CHECK-MA-NEXT:    [[TMP1:%.*]] = load float, ptr [[B_ADDR]], align 4
+// CHECK-MA-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOATFLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-MA-NEXT:    store float [[TMP1]], ptr [[B2]], align 4
+// CHECK-MA-NEXT:    [[TMP2:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-MA-NEXT:    ret i64 [[TMP2]]
+//
+struct bfloatfloat hf(__bf16 a, float b) {
+  struct bfloatfloat x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct bfloat2float {
+  __bf16 a;
+  __bf16 b;
+  float c;
+};
+
+// CHECK-LABEL: define dso_local i64 @h2f
+// CHECK-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], float noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT2FLOAT:%.*]], align 4
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[C_ADDR:%.*]] = alloca float, align 4
+// CHECK-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    store float [[C]], ptr [[C_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-NEXT:    [[TMP2:%.*]] = load float, ptr [[C_ADDR]], align 4
+// CHECK-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-NEXT:    store float [[TMP2]], ptr [[C3]], align 4
+// CHECK-NEXT:    [[TMP3:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-NEXT:    ret i64 [[TMP3]]
+//
+// CHECK-V-LABEL: define dso_local i64 @h2f
+// CHECK-V-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], float noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-V-NEXT:  entry:
+// CHECK-V-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT2FLOAT:%.*]], align 4
+// CHECK-V-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[C_ADDR:%.*]] = alloca float, align 4
+// CHECK-V-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    store float [[C]], ptr [[C_ADDR]], align 4
+// CHECK-V-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-V-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 4
+// CHECK-V-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-V-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-V-NEXT:    [[TMP2:%.*]] = load float, ptr [[C_ADDR]], align 4
+// CHECK-V-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-V-NEXT:    store float [[TMP2]], ptr [[C3]], align 4
+// CHECK-V-NEXT:    [[TMP3:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-V-NEXT:    ret i64 [[TMP3]]
+//
+// CHECK-MA-LABEL: define dso_local i64 @h2f
+// CHECK-MA-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], float noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-MA-NEXT:  entry:
+// CHECK-MA-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT2FLOAT:%.*]], align 4
+// CHECK-MA-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[C_ADDR:%.*]] = alloca float, align 4
+// CHECK-MA-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    store float [[C]], ptr [[C_ADDR]], align 4
+// CHECK-MA-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-MA-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 4
+// CHECK-MA-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-MA-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-MA-NEXT:    [[TMP2:%.*]] = load float, ptr [[C_ADDR]], align 4
+// CHECK-MA-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-MA-NEXT:    store float [[TMP2]], ptr [[C3]], align 4
+// CHECK-MA-NEXT:    [[TMP3:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-MA-NEXT:    ret i64 [[TMP3]]
+//
+struct bfloat2float h2f(__bf16 a, __bf16 b, float c) {
+  struct bfloat2float x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct floatbfloat3 {
+  float a;
+  __bf16 b;
+  __bf16 c;
+  __bf16 d;
+};
+
+// CHECK-LABEL: define dso_local [2 x i64] @fh3
+// CHECK-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]]) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT3:%.*]], align 4
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[RETVAL_COERCE:%.*]] = alloca [2 x i64], align 8
+// CHECK-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 3
+// CHECK-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 4
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 4 [[RETVAL]], i64 12, i1 false)
+// CHECK-NEXT:    [[TMP4:%.*]] = load [2 x i64], ptr [[RETVAL_COERCE]], align 8
+// CHECK-NEXT:    ret [2 x i64] [[TMP4]]
+//
+// CHECK-V-LABEL: define dso_local [2 x i64] @fh3
+// CHECK-V-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]]) #[[ATTR0]] {
+// CHECK-V-NEXT:  entry:
+// CHECK-V-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT3:%.*]], align 4
+// CHECK-V-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-V-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[RETVAL_COERCE:%.*]] = alloca [2 x i64], align 8
+// CHECK-V-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-V-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-V-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-V-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-V-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-V-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-V-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-V-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-V-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-V-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-V-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-V-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 3
+// CHECK-V-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 4
+// CHECK-V-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 4 [[RETVAL]], i64 12, i1 false)
+// CHECK-V-NEXT:    [[TMP4:%.*]] = load [2 x i64], ptr [[RETVAL_COERCE]], align 8
+// CHECK-V-NEXT:    ret [2 x i64] [[TMP4]]
+//
+// CHECK-MA-LABEL: define dso_local [2 x i64] @fh3
+// CHECK-MA-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]]) #[[ATTR0]] {
+// CHECK-MA-NEXT:  entry:
+// CHECK-MA-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT3:%.*]], align 4
+// CHECK-MA-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-MA-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[RETVAL_COERCE:%.*]] = alloca [2 x i64], align 8
+// CHECK-MA-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-MA-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-MA-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-MA-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-MA-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-MA-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-MA-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-MA-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-MA-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-MA-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-MA-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-MA-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 3
+// CHECK-MA-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 4
+// CHECK-MA-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 4 [[RETVAL]], i64 12, i1 false)
+// CHECK-MA-NEXT:    [[TMP4:%.*]] = load [2 x i64], ptr [[RETVAL_COERCE]], align 8
+// CHECK-MA-NEXT:    ret [2 x i64] [[TMP4]]
+//
+struct floatbfloat3 fh3(float a, __bf16 b, __bf16 c, __bf16 d) {
+  struct floatbfloat3 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  x.d = d;
+  return x;
+}
+
+struct bfloat5 {
+  __bf16 a;
+  __bf16 b;
+  __bf16 c;
+  __bf16 d;
+  __bf16 e;
+};
+
+// CHECK-LABEL: define dso_local [2 x i64] @h5
+// CHECK-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]], bfloat noundef [[E:%.*]]) #[[ATTR0]] {
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT5:%.*]], align 2
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[E_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-NEXT:    [[RETVAL_COERCE:%.*]] = alloca [2 x i64], align 8
+// CHECK-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-NEXT:    store bfloat [[E]], ptr [[E_ADDR]], align 2
+// CHECK-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 3
+// CHECK-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 2
+// CHECK-NEXT:    [[TMP4:%.*]] = load bfloat, ptr [[E_ADDR]], align 2
+// CHECK-NEXT:    [[E5:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 4
+// CHECK-NEXT:    store bfloat [[TMP4]], ptr [[E5]], align 2
+// CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 2 [[RETVAL]], i64 10, i1 false)
+// CHECK-NEXT:    [[TMP5:%.*]] = load [2 x i64], ptr [[RETVAL_COERCE]], align 8
+// CHECK-NEXT:    ret [2 x i64] [[TMP5]]
+//
+// CHECK-V-LABEL: define dso_local [2 x i64] @h5
+// CHECK-V-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]], bfloat noundef [[E:%.*]]) #[[ATTR0]] {
+// CHECK-V-NEXT:  entry:
+// CHECK-V-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT5:%.*]], align 2
+// CHECK-V-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[E_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-V-NEXT:    [[RETVAL_COERCE:%.*]] = alloca [2 x i64], align 8
+// CHECK-V-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-V-NEXT:    store bfloat [[E]], ptr [[E_ADDR]], align 2
+// CHECK-V-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-V-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-V-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-V-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-V-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-V-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-V-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-V-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-V-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-V-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-V-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 3
+// CHECK-V-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 2
+// CHECK-V-NEXT:    [[TMP4:%.*]] = load bfloat, ptr [[E_ADDR]], align 2
+// CHECK-V-NEXT:    [[E5:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 4
+// CHECK-V-NEXT:    store bfloat [[TMP4]], ptr [[E5]], align 2
+// CHECK-V-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 2 [[RETVAL]], i64 10, i1 false)
+// CHECK-V-NEXT:    [[TMP5:%.*]] = load [2 x i64], ptr [[RETVAL_COERCE]], align 8
+// CHECK-V-NEXT:    ret [2 x i64] [[TMP5]]
+//
+// CHECK-MA-LABEL: define dso_local [2 x i64] @h5
+// CHECK-MA-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]], bfloat noundef [[E:%.*]]) #[[ATTR0]] {
+// CHECK-MA-NEXT:  entry:
+// CHECK-MA-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT5:%.*]], align 2
+// CHECK-MA-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[E_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-MA-NEXT:    [[RETVAL_COERCE:%.*]] = alloca [2 x i64], align 8
+// CHECK-MA-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-MA-NEXT:    store bfloat [[E]], ptr [[E_ADDR]], align 2
+// CHECK-MA-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-MA-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-MA-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-MA-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-MA-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-MA-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-MA-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-MA-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-MA-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-MA-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-MA-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 3
+// CHECK-MA-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 2
+// CHECK-MA-NEXT:    [[TMP4:%.*]] = load bfloat, ptr [[E_ADDR]], align 2
+// CHECK-MA-NEXT:    [[E5:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 4
+// CHECK-MA-NEXT:    store bfloat [[TMP4]], ptr [[E5]], align 2
+// CHECK-MA-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 2 [[RETVAL]], i64 10, i1 false)
+// CHECK-MA-NEXT:    [[TMP5:%.*]] = load [2 x i64], ptr [[RETVAL_COERCE]], align 8
+// CHECK-MA-NEXT:    ret [2 x i64] [[TMP5]]
+//
+struct bfloat5 h5(__bf16 a, __bf16 b, __bf16 c, __bf16 d, __bf16 e) {
+  struct bfloat5 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  x.d = d;
+  x.e = e;
+  return x;
+}

clang/test/CodeGen/RISCV/bfloat-mangle.cpp

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --version 2
+// RUN: %clang_cc1 -triple riscv64 -target-feature +experimental-zfbfmin -emit-llvm -o - %s | FileCheck %s --check-prefixes=CHECK-RV64
+// RUN: %clang_cc1 -triple riscv32 -target-feature +experimental-zfbfmin -emit-llvm -o - %s | FileCheck %s --check-prefixes=CHECK-RV32
+// RUN: %clang_cc1 -triple riscv64 -target-feature +experimental-zvfbfmin -emit-llvm -o - %s | FileCheck %s --check-prefixes=CHECK-RV64-V
+// RUN: %clang_cc1 -triple riscv32 -target-feature +experimental-zvfbfmin -emit-llvm -o - %s | FileCheck %s --check-prefixes=CHECK-RV32-V
+// RUN: %clang_cc1 -triple riscv64 -target-feature +experimental-zvfbfwma -emit-llvm -o - %s | FileCheck %s --check-prefixes=CHECK-RV64-MA
+// RUN: %clang_cc1 -triple riscv32 -target-feature +experimental-zvfbfwma -emit-llvm -o - %s | FileCheck %s --check-prefixes=CHECK-RV32-MA
+
+// CHECK-RV64-LABEL: define dso_local void @_Z3fooDF16b
+// CHECK-RV64-SAME: (bfloat noundef [[B:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-RV64-NEXT:  entry:
+// CHECK-RV64-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    ret void
+//
+// CHECK-RV32-LABEL: define dso_local void @_Z3fooDF16b
+// CHECK-RV32-SAME: (bfloat noundef [[B:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-RV32-NEXT:  entry:
+// CHECK-RV32-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    ret void
+//
+// CHECK-RV64-V-LABEL: define dso_local void @_Z3fooDF16b
+// CHECK-RV64-V-SAME: (bfloat noundef [[B:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-RV64-V-NEXT:  entry:
+// CHECK-RV64-V-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-V-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV64-V-NEXT:    ret void
+//
+// CHECK-RV32-V-LABEL: define dso_local void @_Z3fooDF16b
+// CHECK-RV32-V-SAME: (bfloat noundef [[B:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-RV32-V-NEXT:  entry:
+// CHECK-RV32-V-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-V-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV32-V-NEXT:    ret void
+//
+// CHECK-RV64-MA-LABEL: define dso_local void @_Z3fooDF16b
+// CHECK-RV64-MA-SAME: (bfloat noundef [[B:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-RV64-MA-NEXT:  entry:
+// CHECK-RV64-MA-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-MA-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV64-MA-NEXT:    ret void
+//
+// CHECK-RV32-MA-LABEL: define dso_local void @_Z3fooDF16b
+// CHECK-RV32-MA-SAME: (bfloat noundef [[B:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-RV32-MA-NEXT:  entry:
+// CHECK-RV32-MA-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-MA-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV32-MA-NEXT:    ret void
+//
+void foo(__bf16 b) {}

clang/test/Sema/vector-decl-crash.c

-// RUN: %clang_cc1 %s -fsyntax-only -verify -triple riscv64-unknown-unknown
-
-// GH50171
-// This would previously crash when __bf16 was not a supported type.
-__bf16 v64bf __attribute__((vector_size(128))); // expected-error {{__bf16 is not supported on this target}} \
-                                                   expected-error {{vector size not an integral multiple of component size}}
-

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

RISCVTargetLowering::RISCVTargetLowering(const TargetMachine &TM,
   addRegisterClass(XLenVT, &RISCV::GPRRegClass);
 
   if (Subtarget.hasStdExtZfhOrZfhmin())
     addRegisterClass(MVT::f16, &RISCV::FPR16RegClass);
   if (Subtarget.hasStdExtF())
     addRegisterClass(MVT::f32, &RISCV::FPR32RegClass);
   if (Subtarget.hasStdExtD())
     addRegisterClass(MVT::f64, &RISCV::FPR64RegClass);
+  if (Subtarget.hasStdExtZfbfmin() || Subtarget.hasStdExtZvfbfmin() ||
+      Subtarget.hasStdExtZvfbfwma())
+    addRegisterClass(MVT::bf16, &RISCV::FPR16RegClass);
   if (Subtarget.hasStdExtZhinxOrZhinxmin())
     addRegisterClass(MVT::f16, &RISCV::GPRF16RegClass);
   if (Subtarget.hasStdExtZfinx())
     addRegisterClass(MVT::f32, &RISCV::GPRF32RegClass);
   if (Subtarget.hasStdExtZdinx()) {
     if (Subtarget.is64Bit())
       addRegisterClass(MVT::f64, &RISCV::GPRF64RegClass);
   }
if (Subtarget.hasStdExtFOrZfinx()) {
     setOperationAction(ISD::SELECT_CC, MVT::f32, Expand);
     setOperationAction(ISD::SELECT, MVT::f32, Custom);
     setOperationAction(ISD::BR_CC, MVT::f32, Expand);
     setOperationAction(FPOpToExpand, MVT::f32, Expand);
     setLoadExtAction(ISD::EXTLOAD, MVT::f32, MVT::f16, Expand);
     setTruncStoreAction(MVT::f32, MVT::f16, Expand);
     setOperationAction(ISD::IS_FPCLASS, MVT::f32, Custom);
 
     if (Subtarget.hasStdExtZfa())
       setOperationAction(ISD::FNEARBYINT, MVT::f32, Legal);
   }
 
   if (Subtarget.hasStdExtFOrZfinx() && Subtarget.is64Bit())
     setOperationAction(ISD::BITCAST, MVT::i32, Custom);
 
   if (Subtarget.hasStdExtDOrZdinx()) {
     setOperationAction(FPLegalNodeTypes, MVT::f64, Legal);