[clang][BFloat] Add create/set/get/dup intrinsics

[miyuki]

Summary

This patch is part of a series that adds support for the Bfloat16 extension of
the Armv8.6-a architecture, as detailed here:

https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/arm-architecture-developments-armv8-6-a

The bfloat type and its properties are specified in the Arm Architecture
Reference Manual:

https://developer.arm.com/docs/ddi0487/latest/arm-architecture-reference-manual-armv8-for-armv8-a-architecture-profile

The following people contributed to this patch:

• Luke Cheeseman
• Momchil Velikov
• Luke Geeson
• Ties Stuij
• Mikhail Maltsev

[fpetrogalli]

clang/test/CodeGen/aarch64-bf16-getset-intrinsics.c

120–121

This seems to be the only place where you need to differentiate between check32 and check64, and I am not 100% sure the extra q in the name of the variable is relevant in terms of codegen testing.

Maybe you can just test both aarch32 and aarch64 with the same CHECK prefix?

[LukeGeeson]

I was an author for part of this patch. Please add all authors as a list of authors to this commit message. Thanks!

As an aside, it would be worth doing this for all the patches in this series

[stuij]

adhere to patch attribution conventions: change author to Ties, add all the contributors

[LukeGeeson]

Can you update the commit message in this differential as well please? Same for the other commits :)

[labrinea]

clang/include/clang/Basic/arm_neon.td

1859

v8.6-A ?

[stuij]

clang/include/clang/Basic/arm_neon.td

1859

Yes V8.2-A, the feature has been added to the 8.2 architecture, but in a later release.

For example, the section on BFloat in the Arm ARM reads:
ARMv8.2-BF16, Armv8.2 AArch64 BFloat16 Extension

[stuij]

In D79710#2041418, @LukeGeeson wrote:

Can you update the commit message in this differential as well please? Same for the other commits :)

done

[stuij]

moving 'CartesianProductWith' to more apt patch

[dmgreen]

clang/include/clang/Basic/arm_neon.td

1874

Do you know what InstName = "vmov" does, and is it needed here?

I'm pretty sure it's a vestige of an earlier implementation of the neon emitter.

1881

Does this need let isLaneQ = 1, like the other vdup_laneq's?

clang/include/clang/Basic/arm_neon_incl.td

293 ↗

(On Diff #265764)

Is this needed in this patch?

clang/lib/CodeGen/CGBuiltin.cpp

6313

How come these are needed for vduph_lane_bf16 if they were not needed for vduph_lane_f16?

clang/test/CodeGen/aarch64-bf16-getset-intrinsics.c

3

It's best to have auto generated tests if we can, and ideally not rely on running the entire -O2 pipeline. I think a lot of the other tests use clang ... -disable-O0-optnone | opt -S -mem2reg | Filecheck ...

Also this aarch64 tests is running arm codegen too? Not sure if that matters, but I don't immediately see it being done elsewhere.

20–21

A lot of these 32 and 64 bit lines look the same.

[labrinea]

clang/include/clang/Basic/arm_neon.td

1868

My local build points here with:
arm_neon.td:1926:3: error: No compatible intrinsic found - looking up intrinsic 'splat_laneq(bfloat16x8_t, int32_t)'

[stuij]

clang/include/clang/Basic/arm_neon.td

1868

Thanks, yes we need to upstream another patch to match the upstreamed work already done for other types.

[miyuki]

clang/include/clang/Basic/arm_neon.td

1874

It looks like InstName was removed from NeonEmitter in

commit dee4ab08ba956efd76aa10da46510dcddecceacf
Author: James Molloy <[e-mail]>
Date:   Tue Jun 17 13:11:27 2014 +0000

    Rewrite ARM NEON intrinsic emission completely.

[miyuki]

Addressed reviewers' comments.

[miyuki]

Fixed the "RUN:" line in the A32 test, implemented vduph_lane for A32.

clang/include/clang/Basic/arm_neon.td

 def OP_SCALAR_QDMULH_LN : ScalarMulOp<"vqdmulh">;
 def OP_SCALAR_QRDMULH_LN : ScalarMulOp<"vqrdmulh">;
 
 def OP_SCALAR_QRDMLAH_LN : Op<(call "vqadd", $p0, (call "vqrdmulh", $p1,
                               (call "vget_lane", $p2, $p3)))>;
 def OP_SCALAR_QRDMLSH_LN : Op<(call "vqsub", $p0, (call "vqrdmulh", $p1,
                               (call "vget_lane", $p2, $p3)))>;
 
-def OP_SCALAR_HALF_GET_LN : Op<(bitcast "float16_t",
-                                   (call "vget_lane",
-                                         (bitcast "int16x4_t", $p0), $p1))>;
-def OP_SCALAR_HALF_GET_LNQ : Op<(bitcast "float16_t",
-                                    (call "vget_lane",
-                                          (bitcast "int16x8_t", $p0), $p1))>;
-def OP_SCALAR_HALF_SET_LN : Op<(bitcast "float16x4_t",
-                                   (call "vset_lane",
-                                         (bitcast "int16_t", $p0),
-                                         (bitcast "int16x4_t", $p1), $p2))>;
-def OP_SCALAR_HALF_SET_LNQ : Op<(bitcast "float16x8_t",
-                                    (call "vset_lane",
-                                          (bitcast "int16_t", $p0),
-                                          (bitcast "int16x8_t", $p1), $p2))>;
+multiclass ScalarGetSetLaneOpsF16<string scalarTy,
+                                  string vectorTy4, string vectorTy8> {
+  def _GET_LN  : Op<(bitcast scalarTy,
+  def _GET_LNQ : Op<(bitcast scalarTy,
+  def _SET_LN  : Op<(bitcast vectorTy4,
+  def _SET_LNQ : Op<(bitcast vectorTy8,
+}
+
+defm OP_SCALAR_HALF: ScalarGetSetLaneOpsF16<"float16_t",
+                                            "float16x4_t", "float16x8_t">;
+defm OP_SCALAR_BF16: ScalarGetSetLaneOpsF16<"bfloat16_t",
+                                            "bfloat16x4_t", "bfloat16x8_t">;
 
 def OP_DOT_LN
     : Op<(call "vdot", $p0, $p1,
           (bitcast $p1, (call_mangled "splat_lane", (bitcast "32", $p2), $p3)))>;
 def OP_DOT_LNQ
     : Op<(call "vdot", $p0, $p1,
           (bitcast $p1, (call_mangled "splat_lane", (bitcast "32", $p2), $p3)))>;
 

 
 // Splat operation - performs a range-checked splat over a vector
 def SPLAT  : WInst<"splat_lane", ".(!q)I",
                    "UcUsUicsilPcPsfQUcQUsQUiQcQsQiQPcQPsQflUlQlQUlhdQhQdPlQPl">;
 def SPLATQ : WInst<"splat_laneq", ".(!Q)I",
                    "UcUsUicsilPcPsfQUcQUsQUiQcQsQiQPcQPsQflUlQlQUlhdQhQdPlQPl"> {
   let isLaneQ = 1;
 }
+let ArchGuard = "defined(__ARM_FEATURE_BF16_VECTOR_ARITHMETIC)" in {
+  def SPLAT_BF  : WInst<"splat_lane", ".(!q)I", "bQb">;
+  def SPLATQ_BF : WInst<"splat_laneq", ".(!Q)I", "bQb"> {
+    let isLaneQ = 1;
+  }
+}
 
 //===----------------------------------------------------------------------===//
 // Intrinsics
 //===----------------------------------------------------------------------===//
 
 ////////////////////////////////////////////////////////////////////////////////
 // E.3.1 Addition
 def VADD    : IOpInst<"vadd", "...",
let ArchGuard = "defined(__ARM_FEATURE_COMPLEX)" in {
   def VCADD_ROT270  : SInst<"vcadd_rot270", "...", "f">;
   def VCADDQ_ROT90  : SInst<"vcaddq_rot90", "QQQ", "f">;
   def VCADDQ_ROT270 : SInst<"vcaddq_rot270", "QQQ", "f">;
 }
 let ArchGuard = "defined(__ARM_FEATURE_COMPLEX) && defined(__aarch64__)" in {
   def VCADDQ_ROT90_FP64  : SInst<"vcaddq_rot90", "QQQ", "d">;
   def VCADDQ_ROT270_FP64 : SInst<"vcaddq_rot270", "QQQ", "d">;
 }
+
+// V8.2-A BFloat intrinsics
+let ArchGuard = "defined(__ARM_FEATURE_BF16_VECTOR_ARITHMETIC)" in {
+  def VCREATE_BF : NoTestOpInst<"vcreate", ".(IU>)", "b", OP_CAST> {
+    let BigEndianSafe = 1;
+  }
+
+  def VDUP_N_BF    : WOpInst<"vdup_n", ".1", "bQb", OP_DUP>;
+
+  def VDUP_LANE_BF : WOpInst<"vdup_lane", ".qI", "bQb", OP_DUP_LN>;
+  def VDUP_LANEQ_BF: WOpInst<"vdup_laneq", ".QI", "bQb", OP_DUP_LN> {
+    let isLaneQ = 1;
+  }
+
+  def VCOMBINE_BF  : NoTestOpInst<"vcombine", "Q..", "b", OP_CONC>;
+
+  def VGET_HIGH_BF : NoTestOpInst<"vget_high", ".Q", "b", OP_HI>;
+  def VGET_LOW_BF  : NoTestOpInst<"vget_low", ".Q", "b", OP_LO>;
+
+  def VGET_LANE_BF : IOpInst<"vget_lane", "1.I", "b", OP_SCALAR_BF16_GET_LN>;
+  def VSET_LANE_BF : IOpInst<"vset_lane", ".1.I", "b", OP_SCALAR_BF16_SET_LN>;
+  def VGET_LANEQ_BF : IOpInst<"vget_lane", "1.I", "Qb", OP_SCALAR_BF16_GET_LNQ>;
+  def VSET_LANEQ_BF : IOpInst<"vset_lane", ".1.I", "Qb", OP_SCALAR_BF16_SET_LNQ>;
+}
+
+let ArchGuard = "defined(__ARM_FEATURE_BF16_VECTOR_ARITHMETIC) && !defined(__aarch64__)" in {
+  def SCALAR_VDUP_LANE_BF_A32 : IOpInst<"vduph_lane", "1.I", "b", OP_SCALAR_BF16_GET_LN>;
+  def SCALAR_VDUP_LANEQ_BF_A32 : IOpInst<"vduph_laneq", "1.I", "Hb", OP_SCALAR_BF16_GET_LNQ>;
+}
+
+let ArchGuard = "defined(__ARM_FEATURE_BF16_VECTOR_ARITHMETIC) && defined(__aarch64__)" in {
+  def SCALAR_VDUP_LANE_BF_A64 : IInst<"vdup_lane", "1.I", "Sb">;
+  def SCALAR_VDUP_LANEQ_BF_A64 : IInst<"vdup_laneq", "1QI", "Sb"> {
+    let isLaneQ = 1;
+  }
+}

clang/lib/CodeGen/CGBuiltin.cpp

static bool HasExtraNeonArgument(unsigned BuiltinID) {
   switch (BuiltinID) {
   default: break;
   case NEON::BI__builtin_neon_vget_lane_i8:
   case NEON::BI__builtin_neon_vget_lane_i16:
   case NEON::BI__builtin_neon_vget_lane_i32:
   case NEON::BI__builtin_neon_vget_lane_i64:
   case NEON::BI__builtin_neon_vget_lane_f32:
   case NEON::BI__builtin_neon_vgetq_lane_i8:
   case NEON::BI__builtin_neon_vgetq_lane_i16:
   case NEON::BI__builtin_neon_vgetq_lane_i32:
   case NEON::BI__builtin_neon_vgetq_lane_i64:
   case NEON::BI__builtin_neon_vgetq_lane_f32:
   case NEON::BI__builtin_neon_vset_lane_i8:
   case NEON::BI__builtin_neon_vset_lane_i16:
   case NEON::BI__builtin_neon_vset_lane_i32:
   case NEON::BI__builtin_neon_vset_lane_i64:
   case NEON::BI__builtin_neon_vset_lane_f32:
case NEON::BI__builtin_neon_vqdmlsls_laneq_s32: {
     Ops.pop_back();
 
     unsigned AccInt = (BuiltinID == NEON::BI__builtin_neon_vqdmlals_lane_s32 ||
                        BuiltinID == NEON::BI__builtin_neon_vqdmlals_laneq_s32)
                           ? Intrinsic::aarch64_neon_sqadd
                           : Intrinsic::aarch64_neon_sqsub;
     return EmitNeonCall(CGM.getIntrinsic(AccInt, Int64Ty), Ops, "vqdmlXl");
   }
+  case NEON::BI__builtin_neon_vduph_lane_bf16:
   case NEON::BI__builtin_neon_vduph_lane_f16: {
     return Builder.CreateExtractElement(Ops[0], EmitScalarExpr(E->getArg(1)),
                                         "vget_lane");
   }
+  case NEON::BI__builtin_neon_vduph_laneq_bf16:
   case NEON::BI__builtin_neon_vduph_laneq_f16: {
     return Builder.CreateExtractElement(Ops[0], EmitScalarExpr(E->getArg(1)),
                                         "vgetq_lane");
   }
   case AArch64::BI_BitScanForward:
   case AArch64::BI_BitScanForward64:
     return EmitMSVCBuiltinExpr(MSVCIntrin::_BitScanForward, E);
   case AArch64::BI_BitScanReverse:

clang/test/CodeGen/aarch64-bf16-getset-intrinsics.c

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// RUN: %clang_cc1 -triple aarch64-arm-none-eabi -target-feature +neon -target-feature +bf16 \
+// RUN:  -disable-O0-optnone -emit-llvm %s -o - | opt -S -mem2reg -instcombine | FileCheck %s
+
+#include <arm_neon.h>
+
+// CHECK-LABEL: @test_vcreate_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i64 [[A:%.*]] to <4 x bfloat>
+// CHECK-NEXT:    ret <4 x bfloat> [[TMP0]]
+//
+bfloat16x4_t test_vcreate_bf16(uint64_t a) {
+  return vcreate_bf16(a);
+}
+
+// CHECK-LABEL: @test_vdup_n_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[VECINIT_I:%.*]] = insertelement <4 x bfloat> undef, bfloat [[V:%.*]], i32 0
+// CHECK-NEXT:    [[VECINIT3_I:%.*]] = shufflevector <4 x bfloat> [[VECINIT_I]], <4 x bfloat> undef, <4 x i32> zeroinitializer
+// CHECK-NEXT:    ret <4 x bfloat> [[VECINIT3_I]]
+//
+bfloat16x4_t test_vdup_n_bf16(bfloat16_t v) {
+  return vdup_n_bf16(v);
+}
+
+// CHECK-LABEL: @test_vdupq_n_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[VECINIT_I:%.*]] = insertelement <8 x bfloat> undef, bfloat [[V:%.*]], i32 0
+// CHECK-NEXT:    [[VECINIT7_I:%.*]] = shufflevector <8 x bfloat> [[VECINIT_I]], <8 x bfloat> undef, <8 x i32> zeroinitializer
+// CHECK-NEXT:    ret <8 x bfloat> [[VECINIT7_I]]
+//
+bfloat16x8_t test_vdupq_n_bf16(bfloat16_t v) {
+  return vdupq_n_bf16(v);
+}
+
+// CHECK-LABEL: @test_vdup_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[LANE:%.*]] = shufflevector <4 x bfloat> [[V:%.*]], <4 x bfloat> undef, <4 x i32> <i32 1, i32 1, i32 1, i32 1>
+// CHECK-NEXT:    ret <4 x bfloat> [[LANE]]
+//
+bfloat16x4_t test_vdup_lane_bf16(bfloat16x4_t v) {
+  return vdup_lane_bf16(v, 1);
+}
+
+// CHECK-LABEL: @test_vdupq_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[LANE:%.*]] = shufflevector <4 x bfloat> [[V:%.*]], <4 x bfloat> undef, <8 x i32> <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
+// CHECK-NEXT:    ret <8 x bfloat> [[LANE]]
+//
+bfloat16x8_t test_vdupq_lane_bf16(bfloat16x4_t v) {
+  return vdupq_lane_bf16(v, 1);
+}
+
+// CHECK-LABEL: @test_vdup_laneq_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[LANE:%.*]] = shufflevector <8 x bfloat> [[V:%.*]], <8 x bfloat> undef, <4 x i32> <i32 7, i32 7, i32 7, i32 7>
+// CHECK-NEXT:    ret <4 x bfloat> [[LANE]]
+//
+bfloat16x4_t test_vdup_laneq_bf16(bfloat16x8_t v) {
+  return vdup_laneq_bf16(v, 7);
+}
+
+// CHECK-LABEL: @test_vdupq_laneq_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[LANE:%.*]] = shufflevector <8 x bfloat> [[V:%.*]], <8 x bfloat> undef, <8 x i32> <i32 7, i32 7, i32 7, i32 7, i32 7, i32 7, i32 7, i32 7>
+// CHECK-NEXT:    ret <8 x bfloat> [[LANE]]
+//
+bfloat16x8_t test_vdupq_laneq_bf16(bfloat16x8_t v) {
+  return vdupq_laneq_bf16(v, 7);
+}
+
+// CHECK-LABEL: @test_vcombine_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[SHUFFLE_I:%.*]] = shufflevector <4 x bfloat> [[LOW:%.*]], <4 x bfloat> [[HIGH:%.*]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+// CHECK-NEXT:    ret <8 x bfloat> [[SHUFFLE_I]]
+//
+bfloat16x8_t test_vcombine_bf16(bfloat16x4_t low, bfloat16x4_t high) {
+  return vcombine_bf16(low, high);
+}
+
+// CHECK-LABEL: @test_vget_high_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[SHUFFLE_I:%.*]] = shufflevector <8 x bfloat> [[A:%.*]], <8 x bfloat> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+// CHECK-NEXT:    ret <4 x bfloat> [[SHUFFLE_I]]
+//
+bfloat16x4_t test_vget_high_bf16(bfloat16x8_t a) {
+  return vget_high_bf16(a);
+}
+
+// CHECK-LABEL: @test_vget_low_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[SHUFFLE_I:%.*]] = shufflevector <8 x bfloat> [[A:%.*]], <8 x bfloat> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+// CHECK-NEXT:    ret <4 x bfloat> [[SHUFFLE_I]]
+//
+bfloat16x4_t test_vget_low_bf16(bfloat16x8_t a) {
+  return vget_low_bf16(a);
+}
+
+// CHECK-LABEL: @test_vget_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DOTCAST1:%.*]] = extractelement <4 x bfloat> [[V:%.*]], i32 1
+// CHECK-NEXT:    ret bfloat [[DOTCAST1]]
+//
+bfloat16_t test_vget_lane_bf16(bfloat16x4_t v) {
+  return vget_lane_bf16(v, 1);
+}
+
+// CHECK-LABEL: @test_vgetq_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DOTCAST1:%.*]] = extractelement <8 x bfloat> [[V:%.*]], i32 7
+// CHECK-NEXT:    ret bfloat [[DOTCAST1]]
+//
+bfloat16_t test_vgetq_lane_bf16(bfloat16x8_t v) {
+  return vgetq_lane_bf16(v, 7);
+}
+
+// CHECK-LABEL: @test_vset_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = insertelement <4 x bfloat> [[V:%.*]], bfloat [[A:%.*]], i32 1
+// CHECK-NEXT:    ret <4 x bfloat> [[TMP0]]
+//
+bfloat16x4_t test_vset_lane_bf16(bfloat16_t a, bfloat16x4_t v) {
+  return vset_lane_bf16(a, v, 1);
+}
+
+// CHECK-LABEL: @test_vsetq_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = insertelement <8 x bfloat> [[V:%.*]], bfloat [[A:%.*]], i32 7
+// CHECK-NEXT:    ret <8 x bfloat> [[TMP0]]
+//
+bfloat16x8_t test_vsetq_lane_bf16(bfloat16_t a, bfloat16x8_t v) {
+  return vsetq_lane_bf16(a, v, 7);
+}
+
+// CHECK-LABEL: @test_vduph_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[VGET_LANE:%.*]] = extractelement <4 x bfloat> [[V:%.*]], i32 1
+// CHECK-NEXT:    ret bfloat [[VGET_LANE]]
+//
+bfloat16_t test_vduph_lane_bf16(bfloat16x4_t v) {
+  return vduph_lane_bf16(v, 1);
+}
+
+// CHECK-LABEL: @test_vduph_laneq_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[VGETQ_LANE:%.*]] = extractelement <8 x bfloat> [[V:%.*]], i32 7
+// CHECK-NEXT:    ret bfloat [[VGETQ_LANE]]
+//
+bfloat16_t test_vduph_laneq_bf16(bfloat16x8_t v) {
+  return vduph_laneq_bf16(v, 7);
+}

clang/test/CodeGen/arm-bf16-getset-intrinsics.c

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// RUN: %clang_cc1 -triple armv8.6a-arm-none-eabi -target-feature +neon -target-feature +bf16 -mfloat-abi hard \
+// RUN:  -disable-O0-optnone -emit-llvm %s -o - | opt -S -mem2reg -instcombine | FileCheck %s
+
+#include <arm_neon.h>
+
+// CHECK-LABEL: @test_vcreate_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i64 [[A:%.*]] to <4 x bfloat>
+// CHECK-NEXT:    ret <4 x bfloat> [[TMP0]]
+//
+bfloat16x4_t test_vcreate_bf16(uint64_t a) {
+  return vcreate_bf16(a);
+}
+
+// CHECK-LABEL: @test_vdup_n_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[VECINIT_I:%.*]] = insertelement <4 x bfloat> undef, bfloat [[V:%.*]], i32 0
+// CHECK-NEXT:    [[VECINIT3_I:%.*]] = shufflevector <4 x bfloat> [[VECINIT_I]], <4 x bfloat> undef, <4 x i32> zeroinitializer
+// CHECK-NEXT:    ret <4 x bfloat> [[VECINIT3_I]]
+//
+bfloat16x4_t test_vdup_n_bf16(bfloat16_t v) {
+  return vdup_n_bf16(v);
+}
+
+// CHECK-LABEL: @test_vdupq_n_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[VECINIT_I:%.*]] = insertelement <8 x bfloat> undef, bfloat [[V:%.*]], i32 0
+// CHECK-NEXT:    [[VECINIT7_I:%.*]] = shufflevector <8 x bfloat> [[VECINIT_I]], <8 x bfloat> undef, <8 x i32> zeroinitializer
+// CHECK-NEXT:    ret <8 x bfloat> [[VECINIT7_I]]
+//
+bfloat16x8_t test_vdupq_n_bf16(bfloat16_t v) {
+  return vdupq_n_bf16(v);
+}
+
+// CHECK-LABEL: @test_vdup_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[LANE:%.*]] = shufflevector <4 x bfloat> [[V:%.*]], <4 x bfloat> undef, <4 x i32> <i32 1, i32 1, i32 1, i32 1>
+// CHECK-NEXT:    ret <4 x bfloat> [[LANE]]
+//
+bfloat16x4_t test_vdup_lane_bf16(bfloat16x4_t v) {
+  return vdup_lane_bf16(v, 1);
+}
+
+// CHECK-LABEL: @test_vdupq_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[LANE:%.*]] = shufflevector <4 x bfloat> [[V:%.*]], <4 x bfloat> undef, <8 x i32> <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
+// CHECK-NEXT:    ret <8 x bfloat> [[LANE]]
+//
+bfloat16x8_t test_vdupq_lane_bf16(bfloat16x4_t v) {
+  return vdupq_lane_bf16(v, 1);
+}
+
+// CHECK-LABEL: @test_vdup_laneq_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[LANE:%.*]] = shufflevector <8 x bfloat> [[V:%.*]], <8 x bfloat> undef, <4 x i32> <i32 7, i32 7, i32 7, i32 7>
+// CHECK-NEXT:    ret <4 x bfloat> [[LANE]]
+//
+bfloat16x4_t test_vdup_laneq_bf16(bfloat16x8_t v) {
+  return vdup_laneq_bf16(v, 7);
+}
+
+// CHECK-LABEL: @test_vdupq_laneq_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[LANE:%.*]] = shufflevector <8 x bfloat> [[V:%.*]], <8 x bfloat> undef, <8 x i32> <i32 7, i32 7, i32 7, i32 7, i32 7, i32 7, i32 7, i32 7>
+// CHECK-NEXT:    ret <8 x bfloat> [[LANE]]
+//
+bfloat16x8_t test_vdupq_laneq_bf16(bfloat16x8_t v) {
+  return vdupq_laneq_bf16(v, 7);
+}
+
+// CHECK-LABEL: @test_vcombine_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[SHUFFLE_I:%.*]] = shufflevector <4 x bfloat> [[LOW:%.*]], <4 x bfloat> [[HIGH:%.*]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+// CHECK-NEXT:    ret <8 x bfloat> [[SHUFFLE_I]]
+//
+bfloat16x8_t test_vcombine_bf16(bfloat16x4_t low, bfloat16x4_t high) {
+  return vcombine_bf16(low, high);
+}
+
+// CHECK-LABEL: @test_vget_high_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[SHUFFLE_I:%.*]] = shufflevector <8 x bfloat> [[A:%.*]], <8 x bfloat> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+// CHECK-NEXT:    ret <4 x bfloat> [[SHUFFLE_I]]
+//
+bfloat16x4_t test_vget_high_bf16(bfloat16x8_t a) {
+  return vget_high_bf16(a);
+}
+
+// CHECK-LABEL: @test_vget_low_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[SHUFFLE_I:%.*]] = shufflevector <8 x bfloat> [[A:%.*]], <8 x bfloat> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+// CHECK-NEXT:    ret <4 x bfloat> [[SHUFFLE_I]]
+//
+bfloat16x4_t test_vget_low_bf16(bfloat16x8_t a) {
+  return vget_low_bf16(a);
+}
+
+// CHECK-LABEL: @test_vget_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DOTCAST1:%.*]] = extractelement <4 x bfloat> [[V:%.*]], i32 1
+// CHECK-NEXT:    ret bfloat [[DOTCAST1]]
+//
+bfloat16_t test_vget_lane_bf16(bfloat16x4_t v) {
+  return vget_lane_bf16(v, 1);
+}
+
+// CHECK-LABEL: @test_vgetq_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DOTCAST1:%.*]] = extractelement <8 x bfloat> [[V:%.*]], i32 7
+// CHECK-NEXT:    ret bfloat [[DOTCAST1]]
+//
+bfloat16_t test_vgetq_lane_bf16(bfloat16x8_t v) {
+  return vgetq_lane_bf16(v, 7);
+}
+
+// CHECK-LABEL: @test_vset_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = insertelement <4 x bfloat> [[V:%.*]], bfloat [[A:%.*]], i32 1
+// CHECK-NEXT:    ret <4 x bfloat> [[TMP0]]
+//
+bfloat16x4_t test_vset_lane_bf16(bfloat16_t a, bfloat16x4_t v) {
+  return vset_lane_bf16(a, v, 1);
+}
+
+// CHECK-LABEL: @test_vsetq_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = insertelement <8 x bfloat> [[V:%.*]], bfloat [[A:%.*]], i32 7
+// CHECK-NEXT:    ret <8 x bfloat> [[TMP0]]
+//
+bfloat16x8_t test_vsetq_lane_bf16(bfloat16_t a, bfloat16x8_t v) {
+  return vsetq_lane_bf16(a, v, 7);
+}
+
+// CHECK-LABEL: @test_vduph_lane_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[VGET_LANE:%.*]] = extractelement <4 x bfloat> [[V:%.*]], i32 1
+// CHECK-NEXT:    ret bfloat [[VGET_LANE]]
+//
+bfloat16_t test_vduph_lane_bf16(bfloat16x4_t v) {
+  return vduph_lane_bf16(v, 1);
+}
+
+// CHECK-LABEL: @test_vduph_laneq_bf16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[VGETQ_LANE:%.*]] = extractelement <8 x bfloat> [[V:%.*]], i32 7
+// CHECK-NEXT:    ret bfloat [[VGETQ_LANE]]
+//
+bfloat16_t test_vduph_laneq_bf16(bfloat16x8_t v) {
+  return vduph_laneq_bf16(v, 7);
+}