[RISCV][BF16] Enable __bf16 for riscv targets

[joshua-arch1]

Summary

The RISC-V psABI recently added __bf16 in https://github.com/riscv-non-isa/riscv-elf-psabi-doc/pull/367.
Now we can enable this new type in clang.

[craig.topper]

This test seems to crash in the backend with -march=rv64gc -mabi=lp64d

 __bf16 h1(__bf16 a) {
  return a;
}

clang/docs/LanguageExtensions.rst

796

RISC-V

[joshua-arch1]

In D150929#4355303, @craig.topper wrote:

This test seems to crash in the backend with -march=rv64gc -mabi=lp64d

 __bf16 h1(__bf16 a) {
  return a;
}

I haven't done any backend support fo bf16 yet. What I want to do in this patch is only to enable __bf16 for clang.

[craig.topper]

In D150929#4355353, @joshua-arch1 wrote:

In D150929#4355303, @craig.topper wrote:

This test seems to crash in the backend with -march=rv64gc -mabi=lp64d

 __bf16 h1(__bf16 a) {
  return a;
}

I haven't done any backend support fo bf16 yet. What I want to do in this patch is only to enable __bf16 for clang.

We can’t have the compiler crashing if you use it. The backend has to go first.

[joshua-arch1]

In D150929#4355360, @craig.topper wrote:

In D150929#4355353, @joshua-arch1 wrote:

In D150929#4355303, @craig.topper wrote:

This test seems to crash in the backend with -march=rv64gc -mabi=lp64d

 __bf16 h1(__bf16 a) {
  return a;
}

I haven't done any backend support fo bf16 yet. What I want to do in this patch is only to enable __bf16 for clang.

We can’t have the compiler crashing if you use it. The backend has to go first.

But we do not have any intrinsics to generate bf16 instructions right now.

[craig.topper]

In D150929#4355368, @joshua-arch1 wrote:

In D150929#4355360, @craig.topper wrote:

In D150929#4355353, @joshua-arch1 wrote:

In D150929#4355303, @craig.topper wrote:

This test seems to crash in the backend with -march=rv64gc -mabi=lp64d

 __bf16 h1(__bf16 a) {
  return a;
}

I haven't done any backend support fo bf16 yet. What I want to do in this patch is only to enable __bf16 for clang.

We can’t have the compiler crashing if you use it. The backend has to go first.

But we do not have any intrinsics to generate bf16 instructions right now.

That doesn’t matter. The test I sent crashes, thats not acceptable.

Probably also means the ABI is not being implemented correctly.

[joshua-arch1]

That doesn’t matter. The test I sent crashes, thats not acceptable.

Probably also means the ABI is not being implemented correctly.

I have checked the log where it crashes. That's just because there are no corresponding instructions to address bf16 type in instruction selection. Everything is correct before instruction selection.

[craig.topper]

In D150929#4355443, @joshua-arch1 wrote:

That doesn’t matter. The test I sent crashes, thats not acceptable.

Probably also means the ABI is not being implemented correctly.

I have checked the log where it crashes. That's just because there are no corresponding instructions to address bf16 type in instruction selection. Everything is correct before instruction selection.

It created an extending load from bfloat because it tried to return as an f32. That’s not the ABI. It should be in the lower 16 bits of an FP register.

[kito-cheng]

clang/lib/Basic/Targets/RISCV.h

118

Should be DF16b since we intend to use that for std::bfloat16_t.

clang/test/Sema/vector-decl-crash.c

0

What??? we should just remove this testcase IF we really support __bf16 IMO.

[joshua-arch1]

Just like AArch64 and X86, I have updated my implementation by limiting bfloat16 only when Zfbfmin/Zvfbfmin/Zvfbfwma are enabled. Now the compiler can work for bfloat16 programs without crashing.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

435 ↗

(On Diff #524148)

This needs an llvm/test/CodeGen/RISCV test.

[asb]

I feel (IMHO) this is jumping to the endpoint a bit - the usual route for something like this is:

• MC layer support
• LLVM codegen support (and tests!)
• Any needed Clang support

I was hoping to push out some patches related to zfbfmin tests+codegen today but might time-out (if so, I'll return to it tomorrow).

I think the tentative conclusion of https://discourse.llvm.org/t/rfc-c-23-p1467r9-extended-floating-point-types-and-standard-names/70033 is to make __bf16_t an arithmetic type on all targets - I'd recommend keeping an eye on https://reviews.llvm.org/D150913 (and ideally helping to review it), as that's bound to interact with this patch.

[joshua-arch1]

This comment was removed by joshua-arch1.

clang/test/Sema/vector-decl-crash.c

0

Actually, I didn't want to change riscv64 into another target. However, I see someone directly changed x86 into riscv64 when he supported __bf16.

[joshua-arch1]

In D150929#4361151, @asb wrote:

I feel (IMHO) this is jumping to the endpoint a bit - the usual route for something like this is:

• MC layer support
• LLVM codegen support (and tests!)
• Any needed Clang support

I was hoping to push out some patches related to zfbfmin tests+codegen today but might time-out (if so, I'll return to it tomorrow).

I think the tentative conclusion of https://discourse.llvm.org/t/rfc-c-23-p1467r9-extended-floating-point-types-and-standard-names/70033 is to make __bf16_t an arithmetic type on all targets - I'd recommend keeping an eye on https://reviews.llvm.org/D150913 (and ideally helping to review it), as that's bound to interact with this patch.

Have you started the codegen part yet? I have basically tried but it may be a little troublesome since we use FPR16 for both fp16 and bf16. During instruction selection, it will be difficult to match those instructions fp16 and bf16 share.

[MaskRay]

The ABI information for __bf16 has been approved in RISCV psABI and is to be merged. Now we can enable this new type in clang.
See https://github.com/riscv-non-isa/riscv-elf-psabi-doc/pull/367

The description seems verbose and inaccurate now. You can say:

The RISC-V psABI recently added __bf16 in https://github.com/riscv-non-isa/riscv-elf-psabi-doc/pull/367

[joshua-arch1]

Since bf16 has been supported in backend, we can now come back to this clang patch. @craig.topper

[joshua-arch1]

Ping.

[craig.topper]

This test crashes

float foo(__bf16 a, __bf16 b)  {
  return a + b;
}

error in backend: Cannot select: t35: f32,ch = load<(dereferenceable load (s16) from %ir.b.addr), anyext from bf16> t49, FrameIndex:i32<1>, undef:i32

clang/docs/LanguageExtensions.rst

821

Put this above X86 so that the other X86 comment below stays together.

[joshua-arch1]

In D150929#4545521, @craig.topper wrote:

This test crashes

float foo(__bf16 a, __bf16 b)  {
  return a + b;
}

error in backend: Cannot select: t35: f32,ch = load<(dereferenceable load (s16) from %ir.b.addr), anyext from bf16> t49, FrameIndex:i32<1>, undef:i32

I have fixed this backend error in the patch https://reviews.llvm.org/D156646.

[craig.topper]

LGTM

clang/docs/LanguageExtensions.rst

void f2() {
     matrix_5_5<int> i;
     i = (matrix_5_5<int>)d;
     i = static_cast<matrix_5_5<int>>(d);
   }
 
 Half-Precision Floating Point
 =============================
 
 Clang supports three half-precision (16-bit) floating point types:
 ``__fp16``, ``_Float16`` and ``__bf16``. These types are supported
 in all language modes, but their support differs between targets.
 A target is said to have "native support" for a type if the target
 processor offers instructions for directly performing basic arithmetic
 on that type.  In the absence of native support, a type can still be
 supported if the compiler can emulate arithmetic on the type by promoting
 to ``float``; see below for more information on this emulation.
 
 * ``__fp16`` is supported on all targets. The special semantics of this
   type mean that no arithmetic is ever performed directly on ``__fp16`` values;
   see below.
 
 * ``_Float16`` is supported on the following targets:
   * 32-bit ARM (natively on some architecture versions)
   * 64-bit ARM (AArch64) (natively on ARMv8.2a and above)
   * AMDGPU (natively)
   * SPIR (natively)
   * X86 (if SSE2 is available; natively if AVX512-FP16 is also available)
 
 * ``__bf16`` is supported on the following targets (currently never natively):
   * 32-bit ARM
   * 64-bit ARM (AArch64)
+  * RISC-V
   * X86 (when SSE2 is available)
 
 (For X86, SSE2 is available on 64-bit and all recent 32-bit processors.)
 
 ``__fp16`` and ``_Float16`` both use the binary16 format from IEEE
 754-2008, which provides a 5-bit exponent and an 11-bit significand
 (counting the implicit leading 1). ``__bf16`` uses the `bfloat16
 <https://en.wikipedia.org/wiki/Bfloat16_floating-point_format>`_ format,
 which provides an 8-bit exponent and an 8-bit significand; this is the same
 exponent range as `float`, just with greatly reduced precision.

clang/lib/Basic/Targets/RISCV.h

 class RISCVTargetInfo : public TargetInfo {
 protected:
   std::string ABI, CPU;
   std::unique_ptr<llvm::RISCVISAInfo> ISAInfo;
 
 public:
   RISCVTargetInfo(const llvm::Triple &Triple, const TargetOptions &)
       : TargetInfo(Triple) {
+    BFloat16Width = 16;
+    BFloat16Align = 16;
+    BFloat16Format = &llvm::APFloat::BFloat();
     LongDoubleWidth = 128;
     LongDoubleAlign = 128;
     LongDoubleFormat = &llvm::APFloat::IEEEquad();
     SuitableAlign = 128;
     WCharType = SignedInt;
     WIntType = UnsignedInt;
     HasRISCVVTypes = true;
     MCountName = "_mcount";
public:
 
   bool hasFeature(StringRef Feature) const override;
 
   bool handleTargetFeatures(std::vector<std::string> &Features,
                             DiagnosticsEngine &Diags) override;
 
   bool hasBitIntType() const override { return true; }
 
+  bool hasBFloat16Type() const override { return true; }
+
   bool useFP16ConversionIntrinsics() const override {
     return false;
   }
 
   bool isValidCPUName(StringRef Name) const override;
   void fillValidCPUList(SmallVectorImpl<StringRef> &Values) const override;
   bool isValidTuneCPUName(StringRef Name) const override;
   void fillValidTuneCPUList(SmallVectorImpl<StringRef> &Values) const override;
 };
 class LLVM_LIBRARY_VISIBILITY RISCV32TargetInfo : public RISCVTargetInfo {
 public:
   RISCV32TargetInfo(const llvm::Triple &Triple, const TargetOptions &Opts)
       : RISCVTargetInfo(Triple, Opts) {
     IntPtrType = SignedInt;
     PtrDiffType = SignedInt;
     SizeType = UnsignedInt;
     resetDataLayout("e-m:e-p:32:32-i64:64-n32-S128");
   }

clang/test/CodeGen/RISCV/bfloat-abi.c

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --version 2
+// RUN: %clang_cc1 -triple riscv64 -emit-llvm %s -o - | FileCheck %s --check-prefixes=CHECK-RV64
+// RUN: %clang_cc1 -triple riscv32 -emit-llvm %s -o - | FileCheck %s --check-prefixes=CHECK-RV32
+
+struct bfloat1 {
+  __bf16 a;
+};
+
+// CHECK-RV64-LABEL: define dso_local i64 @h1
+// CHECK-RV64-SAME: (bfloat noundef [[A:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-RV64-NEXT:  entry:
+// CHECK-RV64-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT1:%.*]], align 2
+// CHECK-RV64-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[COERCE_DIVE_COERCE:%.*]] = alloca i64, align 8
+// CHECK-RV64-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT1]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV64-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-RV64-NEXT:    [[COERCE_DIVE:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT1]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV64-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[COERCE_DIVE_COERCE]], ptr align 2 [[COERCE_DIVE]], i64 2, i1 false)
+// CHECK-RV64-NEXT:    [[TMP1:%.*]] = load i64, ptr [[COERCE_DIVE_COERCE]], align 8
+// CHECK-RV64-NEXT:    ret i64 [[TMP1]]
+//
+// CHECK-RV32-LABEL: define dso_local i32 @h1
+// CHECK-RV32-SAME: (bfloat noundef [[A:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-RV32-NEXT:  entry:
+// CHECK-RV32-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT1:%.*]], align 2
+// CHECK-RV32-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[COERCE_DIVE_COERCE:%.*]] = alloca i32, align 4
+// CHECK-RV32-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT1]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV32-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-RV32-NEXT:    [[COERCE_DIVE:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT1]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV32-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[COERCE_DIVE_COERCE]], ptr align 2 [[COERCE_DIVE]], i32 2, i1 false)
+// CHECK-RV32-NEXT:    [[TMP1:%.*]] = load i32, ptr [[COERCE_DIVE_COERCE]], align 4
+// CHECK-RV32-NEXT:    ret i32 [[TMP1]]
+//
+struct bfloat1 h1(__bf16 a) {
+  struct bfloat1 x;
+  x.a = a;
+  return x;
+}
+
+struct bfloat2 {
+  __bf16 a;
+  __bf16 b;
+};
+
+// CHECK-RV64-LABEL: define dso_local i64 @h2
+// CHECK-RV64-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-RV64-NEXT:  entry:
+// CHECK-RV64-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT2:%.*]], align 2
+// CHECK-RV64-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[RETVAL_COERCE:%.*]] = alloca i64, align 8
+// CHECK-RV64-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV64-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-RV64-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV64-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-RV64-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 2 [[RETVAL]], i64 4, i1 false)
+// CHECK-RV64-NEXT:    [[TMP2:%.*]] = load i64, ptr [[RETVAL_COERCE]], align 8
+// CHECK-RV64-NEXT:    ret i64 [[TMP2]]
+//
+// CHECK-RV32-LABEL: define dso_local i32 @h2
+// CHECK-RV32-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-RV32-NEXT:  entry:
+// CHECK-RV32-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT2:%.*]], align 2
+// CHECK-RV32-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV32-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-RV32-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV32-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-RV32-NEXT:    [[TMP2:%.*]] = load i32, ptr [[RETVAL]], align 2
+// CHECK-RV32-NEXT:    ret i32 [[TMP2]]
+//
+struct bfloat2 h2(__bf16 a, __bf16 b) {
+  struct bfloat2 x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct bfloat3 {
+  __bf16 a;
+  __bf16 b;
+  __bf16 c;
+};
+
+// CHECK-RV64-LABEL: define dso_local i64 @h3
+// CHECK-RV64-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-RV64-NEXT:  entry:
+// CHECK-RV64-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT3:%.*]], align 2
+// CHECK-RV64-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[RETVAL_COERCE:%.*]] = alloca i64, align 8
+// CHECK-RV64-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV64-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-RV64-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV64-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-RV64-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-RV64-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-RV64-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 2 [[RETVAL]], i64 6, i1 false)
+// CHECK-RV64-NEXT:    [[TMP3:%.*]] = load i64, ptr [[RETVAL_COERCE]], align 8
+// CHECK-RV64-NEXT:    ret i64 [[TMP3]]
+//
+// CHECK-RV32-LABEL: define dso_local [2 x i32] @h3
+// CHECK-RV32-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-RV32-NEXT:  entry:
+// CHECK-RV32-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT3:%.*]], align 2
+// CHECK-RV32-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[RETVAL_COERCE:%.*]] = alloca [2 x i32], align 4
+// CHECK-RV32-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV32-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-RV32-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV32-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-RV32-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT3]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-RV32-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-RV32-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[RETVAL_COERCE]], ptr align 2 [[RETVAL]], i32 6, i1 false)
+// CHECK-RV32-NEXT:    [[TMP3:%.*]] = load [2 x i32], ptr [[RETVAL_COERCE]], align 4
+// CHECK-RV32-NEXT:    ret [2 x i32] [[TMP3]]
+//
+struct bfloat3 h3(__bf16 a, __bf16 b, __bf16 c) {
+  struct bfloat3 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct bfloat4 {
+  __bf16 a;
+  __bf16 b;
+  __bf16 c;
+  __bf16 d;
+};
+
+// CHECK-RV64-LABEL: define dso_local i64 @h4
+// CHECK-RV64-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]]) #[[ATTR0]] {
+// CHECK-RV64-NEXT:  entry:
+// CHECK-RV64-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT4:%.*]], align 2
+// CHECK-RV64-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV64-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-RV64-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV64-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-RV64-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-RV64-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-RV64-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 3
+// CHECK-RV64-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 2
+// CHECK-RV64-NEXT:    [[TMP4:%.*]] = load i64, ptr [[RETVAL]], align 2
+// CHECK-RV64-NEXT:    ret i64 [[TMP4]]
+//
+// CHECK-RV32-LABEL: define dso_local [2 x i32] @h4
+// CHECK-RV32-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]]) #[[ATTR0]] {
+// CHECK-RV32-NEXT:  entry:
+// CHECK-RV32-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT4:%.*]], align 2
+// CHECK-RV32-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV32-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-RV32-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV32-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-RV32-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-RV32-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-RV32-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT4]], ptr [[RETVAL]], i32 0, i32 3
+// CHECK-RV32-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 2
+// CHECK-RV32-NEXT:    [[TMP4:%.*]] = load [2 x i32], ptr [[RETVAL]], align 2
+// CHECK-RV32-NEXT:    ret [2 x i32] [[TMP4]]
+//
+struct bfloat4 h4(__bf16 a, __bf16 b, __bf16 c, __bf16 d) {
+  struct bfloat4 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  x.d = d;
+  return x;
+}
+
+struct floatbfloat {
+  float a;
+  __bf16 b;
+};
+
+// CHECK-RV64-LABEL: define dso_local i64 @fh
+// CHECK-RV64-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-RV64-NEXT:  entry:
+// CHECK-RV64-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT:%.*]], align 4
+// CHECK-RV64-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-RV64-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-RV64-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-RV64-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV64-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-RV64-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV64-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-RV64-NEXT:    [[TMP2:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-RV64-NEXT:    ret i64 [[TMP2]]
+//
+// CHECK-RV32-LABEL: define dso_local [2 x i32] @fh
+// CHECK-RV32-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-RV32-NEXT:  entry:
+// CHECK-RV32-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT:%.*]], align 4
+// CHECK-RV32-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-RV32-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-RV32-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-RV32-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV32-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-RV32-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV32-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-RV32-NEXT:    [[TMP2:%.*]] = load [2 x i32], ptr [[RETVAL]], align 4
+// CHECK-RV32-NEXT:    ret [2 x i32] [[TMP2]]
+//
+struct floatbfloat fh(float a, __bf16 b) {
+  struct floatbfloat x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct floatbfloat2 {
+  float a;
+  __bf16 b;
+  __bf16 c;
+};
+
+// CHECK-RV64-LABEL: define dso_local i64 @fh2
+// CHECK-RV64-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-RV64-NEXT:  entry:
+// CHECK-RV64-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT2:%.*]], align 4
+// CHECK-RV64-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-RV64-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-RV64-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-RV64-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV64-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-RV64-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV64-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-RV64-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-RV64-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-RV64-NEXT:    [[TMP3:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-RV64-NEXT:    ret i64 [[TMP3]]
+//
+// CHECK-RV32-LABEL: define dso_local [2 x i32] @fh2
+// CHECK-RV32-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-RV32-NEXT:  entry:
+// CHECK-RV32-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT2:%.*]], align 4
+// CHECK-RV32-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-RV32-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-RV32-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-RV32-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV32-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-RV32-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV32-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-RV32-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT2]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-RV32-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-RV32-NEXT:    [[TMP3:%.*]] = load [2 x i32], ptr [[RETVAL]], align 4
+// CHECK-RV32-NEXT:    ret [2 x i32] [[TMP3]]
+//
+struct floatbfloat2 fh2(float a, __bf16 b, __bf16 c) {
+  struct floatbfloat2 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct bfloatfloat {
+  __bf16 a;
+  float b;
+};
+
+// CHECK-RV64-LABEL: define dso_local i64 @hf
+// CHECK-RV64-SAME: (bfloat noundef [[A:%.*]], float noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-RV64-NEXT:  entry:
+// CHECK-RV64-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOATFLOAT:%.*]], align 4
+// CHECK-RV64-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[B_ADDR:%.*]] = alloca float, align 4
+// CHECK-RV64-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    store float [[B]], ptr [[B_ADDR]], align 4
+// CHECK-RV64-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOATFLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV64-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 4
+// CHECK-RV64-NEXT:    [[TMP1:%.*]] = load float, ptr [[B_ADDR]], align 4
+// CHECK-RV64-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOATFLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV64-NEXT:    store float [[TMP1]], ptr [[B2]], align 4
+// CHECK-RV64-NEXT:    [[TMP2:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-RV64-NEXT:    ret i64 [[TMP2]]
+//
+// CHECK-RV32-LABEL: define dso_local [2 x i32] @hf
+// CHECK-RV32-SAME: (bfloat noundef [[A:%.*]], float noundef [[B:%.*]]) #[[ATTR0]] {
+// CHECK-RV32-NEXT:  entry:
+// CHECK-RV32-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOATFLOAT:%.*]], align 4
+// CHECK-RV32-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[B_ADDR:%.*]] = alloca float, align 4
+// CHECK-RV32-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    store float [[B]], ptr [[B_ADDR]], align 4
+// CHECK-RV32-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOATFLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV32-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 4
+// CHECK-RV32-NEXT:    [[TMP1:%.*]] = load float, ptr [[B_ADDR]], align 4
+// CHECK-RV32-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOATFLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV32-NEXT:    store float [[TMP1]], ptr [[B2]], align 4
+// CHECK-RV32-NEXT:    [[TMP2:%.*]] = load [2 x i32], ptr [[RETVAL]], align 4
+// CHECK-RV32-NEXT:    ret [2 x i32] [[TMP2]]
+//
+struct bfloatfloat hf(__bf16 a, float b) {
+  struct bfloatfloat x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct bfloat2float {
+  __bf16 a;
+  __bf16 b;
+  float c;
+};
+
+// CHECK-RV64-LABEL: define dso_local i64 @h2f
+// CHECK-RV64-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], float noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-RV64-NEXT:  entry:
+// CHECK-RV64-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT2FLOAT:%.*]], align 4
+// CHECK-RV64-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[C_ADDR:%.*]] = alloca float, align 4
+// CHECK-RV64-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    store float [[C]], ptr [[C_ADDR]], align 4
+// CHECK-RV64-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV64-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 4
+// CHECK-RV64-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV64-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-RV64-NEXT:    [[TMP2:%.*]] = load float, ptr [[C_ADDR]], align 4
+// CHECK-RV64-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-RV64-NEXT:    store float [[TMP2]], ptr [[C3]], align 4
+// CHECK-RV64-NEXT:    [[TMP3:%.*]] = load i64, ptr [[RETVAL]], align 4
+// CHECK-RV64-NEXT:    ret i64 [[TMP3]]
+//
+// CHECK-RV32-LABEL: define dso_local [2 x i32] @h2f
+// CHECK-RV32-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], float noundef [[C:%.*]]) #[[ATTR0]] {
+// CHECK-RV32-NEXT:  entry:
+// CHECK-RV32-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT2FLOAT:%.*]], align 4
+// CHECK-RV32-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[C_ADDR:%.*]] = alloca float, align 4
+// CHECK-RV32-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    store float [[C]], ptr [[C_ADDR]], align 4
+// CHECK-RV32-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV32-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 4
+// CHECK-RV32-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV32-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-RV32-NEXT:    [[TMP2:%.*]] = load float, ptr [[C_ADDR]], align 4
+// CHECK-RV32-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT2FLOAT]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-RV32-NEXT:    store float [[TMP2]], ptr [[C3]], align 4
+// CHECK-RV32-NEXT:    [[TMP3:%.*]] = load [2 x i32], ptr [[RETVAL]], align 4
+// CHECK-RV32-NEXT:    ret [2 x i32] [[TMP3]]
+//
+struct bfloat2float h2f(__bf16 a, __bf16 b, float c) {
+  struct bfloat2float x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct floatbfloat3 {
+  float a;
+  __bf16 b;
+  __bf16 c;
+  __bf16 d;
+};
+
+// CHECK-RV64-LABEL: define dso_local [2 x i64] @fh3
+// CHECK-RV64-SAME: (float noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]]) #[[ATTR0]] {
+// CHECK-RV64-NEXT:  entry:
+// CHECK-RV64-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_FLOATBFLOAT3:%.*]], align 4
+// CHECK-RV64-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-RV64-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[RETVAL_COERCE:%.*]] = alloca [2 x i64], align 8
+// CHECK-RV64-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-RV64-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-RV64-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV64-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-RV64-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV64-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-RV64-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-RV64-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-RV64-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[RETVAL]], i32 0, i32 3
+// CHECK-RV64-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 4
+// CHECK-RV64-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 4 [[RETVAL]], i64 12, i1 false)
+// CHECK-RV64-NEXT:    [[TMP4:%.*]] = load [2 x i64], ptr [[RETVAL_COERCE]], align 8
+// CHECK-RV64-NEXT:    ret [2 x i64] [[TMP4]]
+//
+// CHECK-RV32-LABEL: define dso_local void @fh3
+// CHECK-RV32-SAME: (ptr noalias sret([[STRUCT_FLOATBFLOAT3:%.*]]) align 4 [[AGG_RESULT:%.*]], float noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]]) #[[ATTR0]] {
+// CHECK-RV32-NEXT:  entry:
+// CHECK-RV32-NEXT:    [[RESULT_PTR:%.*]] = alloca ptr, align 4
+// CHECK-RV32-NEXT:    [[A_ADDR:%.*]] = alloca float, align 4
+// CHECK-RV32-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    store ptr [[AGG_RESULT]], ptr [[RESULT_PTR]], align 4
+// CHECK-RV32-NEXT:    store float [[A]], ptr [[A_ADDR]], align 4
+// CHECK-RV32-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[TMP0:%.*]] = load float, ptr [[A_ADDR]], align 4
+// CHECK-RV32-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[AGG_RESULT]], i32 0, i32 0
+// CHECK-RV32-NEXT:    store float [[TMP0]], ptr [[A1]], align 4
+// CHECK-RV32-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[AGG_RESULT]], i32 0, i32 1
+// CHECK-RV32-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 4
+// CHECK-RV32-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[AGG_RESULT]], i32 0, i32 2
+// CHECK-RV32-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-RV32-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_FLOATBFLOAT3]], ptr [[AGG_RESULT]], i32 0, i32 3
+// CHECK-RV32-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 4
+// CHECK-RV32-NEXT:    ret void
+//
+struct floatbfloat3 fh3(float a, __bf16 b, __bf16 c, __bf16 d) {
+  struct floatbfloat3 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  x.d = d;
+  return x;
+}
+
+struct bfloat5 {
+  __bf16 a;
+  __bf16 b;
+  __bf16 c;
+  __bf16 d;
+  __bf16 e;
+};
+
+// CHECK-RV64-LABEL: define dso_local [2 x i64] @h5
+// CHECK-RV64-SAME: (bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]], bfloat noundef [[E:%.*]]) #[[ATTR0]] {
+// CHECK-RV64-NEXT:  entry:
+// CHECK-RV64-NEXT:    [[RETVAL:%.*]] = alloca [[STRUCT_BFLOAT5:%.*]], align 2
+// CHECK-RV64-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[E_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    [[RETVAL_COERCE:%.*]] = alloca [2 x i64], align 8
+// CHECK-RV64-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-RV64-NEXT:    store bfloat [[E]], ptr [[E_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 0
+// CHECK-RV64-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-RV64-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 1
+// CHECK-RV64-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-RV64-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 2
+// CHECK-RV64-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-RV64-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 3
+// CHECK-RV64-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 2
+// CHECK-RV64-NEXT:    [[TMP4:%.*]] = load bfloat, ptr [[E_ADDR]], align 2
+// CHECK-RV64-NEXT:    [[E5:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[RETVAL]], i32 0, i32 4
+// CHECK-RV64-NEXT:    store bfloat [[TMP4]], ptr [[E5]], align 2
+// CHECK-RV64-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[RETVAL_COERCE]], ptr align 2 [[RETVAL]], i64 10, i1 false)
+// CHECK-RV64-NEXT:    [[TMP5:%.*]] = load [2 x i64], ptr [[RETVAL_COERCE]], align 8
+// CHECK-RV64-NEXT:    ret [2 x i64] [[TMP5]]
+//
+// CHECK-RV32-LABEL: define dso_local void @h5
+// CHECK-RV32-SAME: (ptr noalias sret([[STRUCT_BFLOAT5:%.*]]) align 2 [[AGG_RESULT:%.*]], bfloat noundef [[A:%.*]], bfloat noundef [[B:%.*]], bfloat noundef [[C:%.*]], bfloat noundef [[D:%.*]], bfloat noundef [[E:%.*]]) #[[ATTR0]] {
+// CHECK-RV32-NEXT:  entry:
+// CHECK-RV32-NEXT:    [[RESULT_PTR:%.*]] = alloca ptr, align 4
+// CHECK-RV32-NEXT:    [[A_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[C_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[D_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    [[E_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    store ptr [[AGG_RESULT]], ptr [[RESULT_PTR]], align 4
+// CHECK-RV32-NEXT:    store bfloat [[A]], ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[C]], ptr [[C_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[D]], ptr [[D_ADDR]], align 2
+// CHECK-RV32-NEXT:    store bfloat [[E]], ptr [[E_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[TMP0:%.*]] = load bfloat, ptr [[A_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[A1:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[AGG_RESULT]], i32 0, i32 0
+// CHECK-RV32-NEXT:    store bfloat [[TMP0]], ptr [[A1]], align 2
+// CHECK-RV32-NEXT:    [[TMP1:%.*]] = load bfloat, ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[B2:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[AGG_RESULT]], i32 0, i32 1
+// CHECK-RV32-NEXT:    store bfloat [[TMP1]], ptr [[B2]], align 2
+// CHECK-RV32-NEXT:    [[TMP2:%.*]] = load bfloat, ptr [[C_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[C3:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[AGG_RESULT]], i32 0, i32 2
+// CHECK-RV32-NEXT:    store bfloat [[TMP2]], ptr [[C3]], align 2
+// CHECK-RV32-NEXT:    [[TMP3:%.*]] = load bfloat, ptr [[D_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[D4:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[AGG_RESULT]], i32 0, i32 3
+// CHECK-RV32-NEXT:    store bfloat [[TMP3]], ptr [[D4]], align 2
+// CHECK-RV32-NEXT:    [[TMP4:%.*]] = load bfloat, ptr [[E_ADDR]], align 2
+// CHECK-RV32-NEXT:    [[E5:%.*]] = getelementptr inbounds [[STRUCT_BFLOAT5]], ptr [[AGG_RESULT]], i32 0, i32 4
+// CHECK-RV32-NEXT:    store bfloat [[TMP4]], ptr [[E5]], align 2
+// CHECK-RV32-NEXT:    ret void
+//
+struct bfloat5 h5(__bf16 a, __bf16 b, __bf16 c, __bf16 d, __bf16 e) {
+  struct bfloat5 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  x.d = d;
+  x.e = e;
+  return x;
+}

clang/test/CodeGen/RISCV/bfloat-mangle.cpp

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --version 2
+// RUN: %clang_cc1 -triple riscv64 -emit-llvm -o - %s | FileCheck %s --check-prefixes=CHECK-RV64
+// RUN: %clang_cc1 -triple riscv32 -emit-llvm -o - %s | FileCheck %s --check-prefixes=CHECK-RV32
+
+// CHECK-RV64-LABEL: define dso_local void @_Z3fooDF16b
+// CHECK-RV64-SAME: (bfloat noundef [[B:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-RV64-NEXT:  entry:
+// CHECK-RV64-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV64-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV64-NEXT:    ret void
+//
+// CHECK-RV32-LABEL: define dso_local void @_Z3fooDF16b
+// CHECK-RV32-SAME: (bfloat noundef [[B:%.*]]) #[[ATTR0:[0-9]+]] {
+// CHECK-RV32-NEXT:  entry:
+// CHECK-RV32-NEXT:    [[B_ADDR:%.*]] = alloca bfloat, align 2
+// CHECK-RV32-NEXT:    store bfloat [[B]], ptr [[B_ADDR]], align 2
+// CHECK-RV32-NEXT:    ret void
+//
+void foo(__bf16 b) {}

clang/test/Sema/vector-decl-crash.c

-// RUN: %clang_cc1 %s -fsyntax-only -verify -triple riscv64-unknown-unknown
-
-// GH50171
-// This would previously crash when __bf16 was not a supported type.
-__bf16 v64bf __attribute__((vector_size(128))); // expected-error {{__bf16 is not supported on this target}} \
-                                                   expected-error {{vector size not an integral multiple of component size}}
-