[llvm][SveEmitter] Emit the bfloat version of `svld1ro`.

[fpetrogalli]

Summary

The new SVE builtin type __SVBFloat16_t` is used to represent scalable
vectors of bfloat elements.

[sdesmalen]

Thanks @fpetrogalli, the changes look good to me. Just added some minor comments.

clang/include/clang/Basic/AArch64SVEACLETypes.def

53–54

nit: I'd move this below __SVFloat64_t separated by a newline.

clang/utils/TableGen/SveEmitter.cpp

367

Maybe it's worth adding a

bool isFloatingPoint() const { return Float || BFloat; }

to avoid having to write both isFloat and isBFloat everywhere.

386

It's easier to just assert that ElementBitwidth == 16.

941

It's easier to just assert that ElementBitwidth == 16

[stuij]

clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_ld1ro-bfloat.c

2

There should be no dependency on -fallow-half-arguments-and-returns. For bfloat we should use -mfloat-abi hard. Does this work for -mfloat-abi softfp?

[fpetrogalli]

Thank you for the review @sdesmalen.

Francesco

[fpetrogalli]

[sdesmalen]

clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_ld1ro-bfloat.c

2

-fallow-half-arguments-and-returns isn't strictly needed for this test, we just use the same RUN line for all the ACLE tests and we needed this for __fp16 in some of the tests.

I don't believe that -mfloat-abi softfp is supported for AArch64.

[sdesmalen]

LGTM

clang/include/clang/Basic/AArch64SVEACLETypes.def

43

nit: odd formatting (of the last \), did you use clang-format?

[fpetrogalli]

Fix formatting issues, remove an unnecessary empty line.

clang/include/clang/Basic/AArch64SVEACLETypes.def

43

Yes, but def files are left untouched. I fixed it.

clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_ld1ro-bfloat.c

2

@stuij - the following lines work, one with softfp and one with hard:

// RUN: %clang_cc1 -D__ARM_FEATURE_SVE_MATMUL_FP64 -D__ARM_FEATURE_BF16_SCALAR_ARITHMETIC -D__ARM_FEATURE_SVE -DSVE_OVERLOADED_FORMS -triple aarch64-none-linux-gnu -target-feature +sve -target\
-feature +bf16 -mfloat-abi softfp -fallow-half-arguments-and-returns -S -O1 -Werror -Wall -emit-llvm -o - %s | FileCheck %s
// RUN: %clang_cc1 -D__ARM_FEATURE_SVE_MATMUL_FP64 -D__ARM_FEATURE_BF16_SCALAR_ARITHMETIC -D__ARM_FEATURE_SVE -DSVE_OVERLOADED_FORMS -triple aarch64-none-linux-gnu -target-feature +sve -target\
-feature +bf16 -mfloat-abi hard -fallow-half-arguments-and-returns -S -O1 -Werror -Wall -emit-llvm -o - %s | FileCheck %s

@sdesmalen I am not an experer here, but there is a test which targets aarch64 that uses softfp (see clang/test/CodeGen/arm-bf16-params-returns.c). The following line in that test clearly targets aarch64:

// RUN: %clang_cc1 -triple aarch64-arm-none-eabi -target-abi aapcs -mfloat-abi softfp -target-feature +bf16 -target-feature +neon -emit-llvm -O2 -o - %s | opt -S -mem2reg -sroa | FileCheck %s \
--check-prefix=CHECK64-SOFTFP

@both - should I update the test with the two extra RUN lines mentioned up in the message?

[sdesmalen]

clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_ld1ro-bfloat.c

2

@sdesmalen I am not an experer here, but there is a test which targets aarch64 that uses softfp (see clang/test/CodeGen/arm-bf16-params-returns.c). The following line in that test clearly targets aarch64:

clang/test/CodeGen/arm-bf16-params-returns.c also shows that setting softfp has no effect for AArch64.

@both - should I update the test with the two extra RUN lines mentioned up in the message?

No, I think the extra RUN lines aren't necessary.

[stuij]

clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_ld1ro-bfloat.c

2

@fpetrogalli: @sdesmalen is totally right. Softfp doesn't make sense on AArch64 as fp isn't optional. I think the original intent of that AArch64 line in arm-bf16-params-returns.c was to make sure AArch64 indeed doesn't change, but then the option should of course never be passed in the first place. I guess this is a bit of over-defensive coding against the way clang isn't stellar at argument passing.

[stuij]

LGTM

clang/include/clang/Basic/AArch64SVEACLETypes.def

 //
 //  - ElBits is the size of one element in bits.
 //
 //  - IsSigned is true for vectors of signed integer elements and
 //    for vectors of floating-point elements.
 //
 //  - IsFP is true for vectors of floating-point elements.
 //
+//  - IsBF true for vector of brain float elements.
 //===----------------------------------------------------------------------===//
 
 #ifndef SVE_VECTOR_TYPE
 #define SVE_VECTOR_TYPE(Name, MangledName, Id, SingletonId, NumEls, ElBits,    \
-                        IsSigned, IsFP)                                        \
+                        IsSigned, IsFP, IsBF)                                  \
   SVE_TYPE(Name, Id, SingletonId)
 #endif
 
 #ifndef SVE_PREDICATE_TYPE
 #define SVE_PREDICATE_TYPE(Name, MangledName, Id, SingletonId, NumEls)         \
   SVE_TYPE(Name, Id, SingletonId)
 #endif
 
 //===- Vector point types -----------------------------------------------===//
 
-SVE_VECTOR_TYPE("__SVInt8_t", "__SVInt8_t",  SveInt8, SveInt8Ty, 16, 8, true, false)
-SVE_VECTOR_TYPE("__SVInt16_t", "__SVInt16_t", SveInt16, SveInt16Ty, 8, 16, true, false)
-SVE_VECTOR_TYPE("__SVInt32_t", "__SVInt32_t", SveInt32, SveInt32Ty, 4, 32, true, false)
-SVE_VECTOR_TYPE("__SVInt64_t", "__SVInt64_t", SveInt64, SveInt64Ty, 2, 64, true, false)
-
-SVE_VECTOR_TYPE("__SVUint8_t", "__SVUint8_t",  SveUint8, SveUint8Ty, 16, 8, false, false)
-SVE_VECTOR_TYPE("__SVUint16_t", "__SVUint16_t", SveUint16, SveUint16Ty, 8, 16, false, false)
-SVE_VECTOR_TYPE("__SVUint32_t", "__SVUint32_t", SveUint32, SveUint32Ty, 4, 32, false, false)
-SVE_VECTOR_TYPE("__SVUint64_t", "__SVUint64_t", SveUint64, SveUint64Ty, 2, 64, false, false)
-
-SVE_VECTOR_TYPE("__SVFloat16_t", "__SVFloat16_t", SveFloat16, SveFloat16Ty, 8, 16, true, true)
-SVE_VECTOR_TYPE("__SVFloat32_t", "__SVFloat32_t", SveFloat32, SveFloat32Ty, 4, 32, true, true)
-SVE_VECTOR_TYPE("__SVFloat64_t", "__SVFloat64_t", SveFloat64, SveFloat64Ty, 2, 64, true, true)
+
+SVE_VECTOR_TYPE("__SVInt8_t", "__SVInt8_t",  SveInt8, SveInt8Ty, 16, 8, true, false, false)
+SVE_VECTOR_TYPE("__SVInt16_t", "__SVInt16_t", SveInt16, SveInt16Ty, 8, 16, true, false, false)
+SVE_VECTOR_TYPE("__SVInt32_t", "__SVInt32_t", SveInt32, SveInt32Ty, 4, 32, true, false, false)
+SVE_VECTOR_TYPE("__SVInt64_t", "__SVInt64_t", SveInt64, SveInt64Ty, 2, 64, true, false, false)
+
+SVE_VECTOR_TYPE("__SVUint8_t", "__SVUint8_t",  SveUint8, SveUint8Ty, 16, 8, false, false, false)
+SVE_VECTOR_TYPE("__SVUint16_t", "__SVUint16_t", SveUint16, SveUint16Ty, 8, 16, false, false, false)
+SVE_VECTOR_TYPE("__SVUint32_t", "__SVUint32_t", SveUint32, SveUint32Ty, 4, 32, false, false, false)
+SVE_VECTOR_TYPE("__SVUint64_t", "__SVUint64_t", SveUint64, SveUint64Ty, 2, 64, false, false, false)
+
+SVE_VECTOR_TYPE("__SVFloat16_t", "__SVFloat16_t", SveFloat16, SveFloat16Ty, 8, 16, true, true, false)
+SVE_VECTOR_TYPE("__SVFloat32_t", "__SVFloat32_t", SveFloat32, SveFloat32Ty, 4, 32, true, true, false)
+SVE_VECTOR_TYPE("__SVFloat64_t", "__SVFloat64_t", SveFloat64, SveFloat64Ty, 2, 64, true, true, false)
+
+SVE_VECTOR_TYPE("__SVBFloat16_t", "__SVBFloat16_t", SveBFloat16, SveBFloat16Ty, 8, 16, false, false, true)
 
 //
 // x2
 //
-SVE_VECTOR_TYPE("__clang_svint8x2_t", "svint8x2_t",  SveInt8x2, SveInt8x2Ty, 32, 8, true, false)
-SVE_VECTOR_TYPE("__clang_svint16x2_t", "svint16x2_t", SveInt16x2, SveInt16x2Ty, 16, 16, true, false)
-SVE_VECTOR_TYPE("__clang_svint32x2_t", "svint32x2_t", SveInt32x2, SveInt32x2Ty, 8, 32, true, false)
-SVE_VECTOR_TYPE("__clang_svint64x2_t", "svint64x2_t", SveInt64x2, SveInt64x2Ty, 4, 64, true, false)
-
-SVE_VECTOR_TYPE("__clang_svuint8x2_t", "svuint8x2_t",  SveUint8x2, SveUint8x2Ty, 32, 8, false, false)
-SVE_VECTOR_TYPE("__clang_svuint16x2_t", "svuint16x2_t", SveUint16x2, SveUint16x2Ty, 16, 16, false, false)
-SVE_VECTOR_TYPE("__clang_svuint32x2_t", "svuint32x2_t", SveUint32x2, SveUint32x2Ty, 8, 32, false, false)
-SVE_VECTOR_TYPE("__clang_svuint64x2_t", "svuint64x2_t", SveUint64x2, SveUint64x2Ty, 4, 64, false, false)
-
-SVE_VECTOR_TYPE("__clang_svfloat16x2_t", "svfloat16x2_t", SveFloat16x2, SveFloat16x2Ty, 16, 16, true, true)
-SVE_VECTOR_TYPE("__clang_svfloat32x2_t", "svfloat32x2_t", SveFloat32x2, SveFloat32x2Ty, 8, 32, true, true)
-SVE_VECTOR_TYPE("__clang_svfloat64x2_t", "svfloat64x2_t", SveFloat64x2, SveFloat64x2Ty, 4, 64, true, true)
+SVE_VECTOR_TYPE("__clang_svint8x2_t", "svint8x2_t",  SveInt8x2, SveInt8x2Ty, 32, 8, true, false, false)
+SVE_VECTOR_TYPE("__clang_svint16x2_t", "svint16x2_t", SveInt16x2, SveInt16x2Ty, 16, 16, true, false, false)
+SVE_VECTOR_TYPE("__clang_svint32x2_t", "svint32x2_t", SveInt32x2, SveInt32x2Ty, 8, 32, true, false, false)
+SVE_VECTOR_TYPE("__clang_svint64x2_t", "svint64x2_t", SveInt64x2, SveInt64x2Ty, 4, 64, true, false, false)
+
+SVE_VECTOR_TYPE("__clang_svuint8x2_t", "svuint8x2_t",  SveUint8x2, SveUint8x2Ty, 32, 8, false, false, false)
+SVE_VECTOR_TYPE("__clang_svuint16x2_t", "svuint16x2_t", SveUint16x2, SveUint16x2Ty, 16, 16, false, false, false)
+SVE_VECTOR_TYPE("__clang_svuint32x2_t", "svuint32x2_t", SveUint32x2, SveUint32x2Ty, 8, 32, false, false, false)
+SVE_VECTOR_TYPE("__clang_svuint64x2_t", "svuint64x2_t", SveUint64x2, SveUint64x2Ty, 4, 64, false, false, false)
+
+SVE_VECTOR_TYPE("__clang_svfloat16x2_t", "svfloat16x2_t", SveFloat16x2, SveFloat16x2Ty, 16, 16, true, true, false)
+SVE_VECTOR_TYPE("__clang_svfloat32x2_t", "svfloat32x2_t", SveFloat32x2, SveFloat32x2Ty, 8, 32, true, true, false)
+SVE_VECTOR_TYPE("__clang_svfloat64x2_t", "svfloat64x2_t", SveFloat64x2, SveFloat64x2Ty, 4, 64, true, true, false)
 
 //
 // x3
 //
-SVE_VECTOR_TYPE("__clang_svint8x3_t", "svint8x3_t",  SveInt8x3, SveInt8x3Ty, 48, 8, true, false)
-SVE_VECTOR_TYPE("__clang_svint16x3_t", "svint16x3_t", SveInt16x3, SveInt16x3Ty, 24, 16, true, false)
-SVE_VECTOR_TYPE("__clang_svint32x3_t", "svint32x3_t", SveInt32x3, SveInt32x3Ty, 12, 32, true, false)
-SVE_VECTOR_TYPE("__clang_svint64x3_t", "svint64x3_t", SveInt64x3, SveInt64x3Ty, 6, 64, true, false)
-
-SVE_VECTOR_TYPE("__clang_svuint8x3_t", "svuint8x3_t",  SveUint8x3, SveUint8x3Ty, 48, 8, false, false)
-SVE_VECTOR_TYPE("__clang_svuint16x3_t", "svuint16x3_t", SveUint16x3, SveUint16x3Ty, 24, 16, false, false)
-SVE_VECTOR_TYPE("__clang_svuint32x3_t", "svuint32x3_t", SveUint32x3, SveUint32x3Ty, 12, 32, false, false)
-SVE_VECTOR_TYPE("__clang_svuint64x3_t", "svuint64x3_t", SveUint64x3, SveUint64x3Ty, 6, 64, false, false)
-
-SVE_VECTOR_TYPE("__clang_svfloat16x3_t", "svfloat16x3_t", SveFloat16x3, SveFloat16x3Ty, 24, 16, true, true)
-SVE_VECTOR_TYPE("__clang_svfloat32x3_t", "svfloat32x3_t", SveFloat32x3, SveFloat32x3Ty, 12, 32, true, true)
-SVE_VECTOR_TYPE("__clang_svfloat64x3_t", "svfloat64x3_t", SveFloat64x3, SveFloat64x3Ty, 6, 64, true, true)
+SVE_VECTOR_TYPE("__clang_svint8x3_t", "svint8x3_t",  SveInt8x3, SveInt8x3Ty, 48, 8, true, false, false)
+SVE_VECTOR_TYPE("__clang_svint16x3_t", "svint16x3_t", SveInt16x3, SveInt16x3Ty, 24, 16, true, false, false)
+SVE_VECTOR_TYPE("__clang_svint32x3_t", "svint32x3_t", SveInt32x3, SveInt32x3Ty, 12, 32, true, false, false)
+SVE_VECTOR_TYPE("__clang_svint64x3_t", "svint64x3_t", SveInt64x3, SveInt64x3Ty, 6, 64, true, false, false)
+
+SVE_VECTOR_TYPE("__clang_svuint8x3_t", "svuint8x3_t",  SveUint8x3, SveUint8x3Ty, 48, 8, false, false, false)
+SVE_VECTOR_TYPE("__clang_svuint16x3_t", "svuint16x3_t", SveUint16x3, SveUint16x3Ty, 24, 16, false, false, false)
+SVE_VECTOR_TYPE("__clang_svuint32x3_t", "svuint32x3_t", SveUint32x3, SveUint32x3Ty, 12, 32, false, false, false)
+SVE_VECTOR_TYPE("__clang_svuint64x3_t", "svuint64x3_t", SveUint64x3, SveUint64x3Ty, 6, 64, false, false, false)
+
+SVE_VECTOR_TYPE("__clang_svfloat16x3_t", "svfloat16x3_t", SveFloat16x3, SveFloat16x3Ty, 24, 16, true, true, false)
+SVE_VECTOR_TYPE("__clang_svfloat32x3_t", "svfloat32x3_t", SveFloat32x3, SveFloat32x3Ty, 12, 32, true, true, false)
+SVE_VECTOR_TYPE("__clang_svfloat64x3_t", "svfloat64x3_t", SveFloat64x3, SveFloat64x3Ty, 6, 64, true, true, false)
 
 //
 // x4
 //
-SVE_VECTOR_TYPE("__clang_svint8x4_t", "svint8x4_t",  SveInt8x4, SveInt8x4Ty, 64, 8, true, false)
-SVE_VECTOR_TYPE("__clang_svint16x4_t", "svint16x4_t", SveInt16x4, SveInt16x4Ty, 32, 16, true, false)
-SVE_VECTOR_TYPE("__clang_svint32x4_t", "svint32x4_t", SveInt32x4, SveInt32x4Ty, 16, 32, true, false)
-SVE_VECTOR_TYPE("__clang_svint64x4_t", "svint64x4_t", SveInt64x4, SveInt64x4Ty, 8, 64, true, false)
-
-SVE_VECTOR_TYPE("__clang_svuint8x4_t", "svuint8x4_t",  SveUint8x4, SveUint8x4Ty, 64, 8, false, false)
-SVE_VECTOR_TYPE("__clang_svuint16x4_t", "svuint16x4_t", SveUint16x4, SveUint16x4Ty, 32, 16, false, false)
-SVE_VECTOR_TYPE("__clang_svuint32x4_t", "svuint32x4_t", SveUint32x4, SveUint32x4Ty, 16, 32, false, false)
-SVE_VECTOR_TYPE("__clang_svuint64x4_t", "svuint64x4_t", SveUint64x4, SveUint64x4Ty, 8, 64, false, false)
-
-SVE_VECTOR_TYPE("__clang_svfloat16x4_t", "svfloat16x4_t", SveFloat16x4, SveFloat16x4Ty, 32, 16, true, true)
-SVE_VECTOR_TYPE("__clang_svfloat32x4_t", "svfloat32x4_t", SveFloat32x4, SveFloat32x4Ty, 16, 32, true, true)
-SVE_VECTOR_TYPE("__clang_svfloat64x4_t", "svfloat64x4_t", SveFloat64x4, SveFloat64x4Ty, 8, 64, true, true)
+SVE_VECTOR_TYPE("__clang_svint8x4_t", "svint8x4_t",  SveInt8x4, SveInt8x4Ty, 64, 8, true, false, false)
+SVE_VECTOR_TYPE("__clang_svint16x4_t", "svint16x4_t", SveInt16x4, SveInt16x4Ty, 32, 16, true, false, false)
+SVE_VECTOR_TYPE("__clang_svint32x4_t", "svint32x4_t", SveInt32x4, SveInt32x4Ty, 16, 32, true, false, false)
+SVE_VECTOR_TYPE("__clang_svint64x4_t", "svint64x4_t", SveInt64x4, SveInt64x4Ty, 8, 64, true, false, false)
+
+SVE_VECTOR_TYPE("__clang_svuint8x4_t", "svuint8x4_t",  SveUint8x4, SveUint8x4Ty, 64, 8, false, false, false)
+SVE_VECTOR_TYPE("__clang_svuint16x4_t", "svuint16x4_t", SveUint16x4, SveUint16x4Ty, 32, 16, false, false, false)
+SVE_VECTOR_TYPE("__clang_svuint32x4_t", "svuint32x4_t", SveUint32x4, SveUint32x4Ty, 16, 32, false, false, false)
+SVE_VECTOR_TYPE("__clang_svuint64x4_t", "svuint64x4_t", SveUint64x4, SveUint64x4Ty, 8, 64, false, false, false)
+
+SVE_VECTOR_TYPE("__clang_svfloat16x4_t", "svfloat16x4_t", SveFloat16x4, SveFloat16x4Ty, 32, 16, true, true, false)
+SVE_VECTOR_TYPE("__clang_svfloat32x4_t", "svfloat32x4_t", SveFloat32x4, SveFloat32x4Ty, 16, 32, true, true, false)
+SVE_VECTOR_TYPE("__clang_svfloat64x4_t", "svfloat64x4_t", SveFloat64x4, SveFloat64x4Ty, 8, 64, true, true, false)
 
 SVE_PREDICATE_TYPE("__SVBool_t", "__SVBool_t", SveBool, SveBoolTy, 16)
 
 #undef SVE_VECTOR_TYPE
 #undef SVE_PREDICATE_TYPE
 #undef SVE_TYPE

clang/include/clang/Basic/arm_sve.td

 // ---------
 // c: char
 // s: short
 // i: int
 // l: long
 // f: float
 // h: half-float
 // d: double
+// b: bfloat
 
 // Typespec modifiers
 // ------------------
 // P: boolean
 // U: unsigned
 
 // Prototype modifiers
 // -------------------

 def EltTyInt64   : EltType<4>;
 def EltTyFloat16 : EltType<5>;
 def EltTyFloat32 : EltType<6>;
 def EltTyFloat64 : EltType<7>;
 def EltTyBool8   : EltType<8>;
 def EltTyBool16  : EltType<9>;
 def EltTyBool32  : EltType<10>;
 def EltTyBool64  : EltType<11>;
+def EltTyBFloat16 : EltType<12>;
 
 class MemEltType<int val> {
   int Value = val;
 }
 def MemEltTyDefault   : MemEltType<0>;
 def MemEltTyInt8      : MemEltType<1>;
 def MemEltTyInt16     : MemEltType<2>;
 def MemEltTyInt32     : MemEltType<3>;

 
 // Load one quadword and replicate (scalar base)
 def SVLD1RQ : SInst<"svld1rq[_{2}]", "dPc", "csilUcUsUiUlhfd", MergeNone, "aarch64_sve_ld1rq">;
 
 // Load one octoword and replicate (scalar base)
 let ArchGuard = "defined(__ARM_FEATURE_SVE_MATMUL_FP64)" in {
   def SVLD1RO : SInst<"svld1ro[_{2}]", "dPc", "csilUcUsUiUlhfd", MergeNone, "aarch64_sve_ld1ro">;
 }
+let ArchGuard = "defined(__ARM_FEATURE_SVE_MATMUL_FP64) && defined(__ARM_FEATURE_BF16_SCALAR_ARITHMETIC)" in {
+  def SVLD1RO_BF : SInst<"svld1ro[_{2}]", "dPc", "b", MergeNone, "aarch64_sve_ld1ro">;
+}
 ////////////////////////////////////////////////////////////////////////////////
 // Stores
 
 // Store one vector (scalar base)
 def SVST1    : MInst<"svst1[_{d}]",  "vPpd", "csilUcUsUiUlhfd", [IsStore], MemEltTyDefault, "aarch64_sve_st1">;
 def SVST1B_S : MInst<"svst1b[_{d}]", "vPAd", "sil",             [IsStore], MemEltTyInt8,    "aarch64_sve_st1">;
 def SVST1B_U : MInst<"svst1b[_{d}]", "vPEd", "UsUiUl",          [IsStore], MemEltTyInt8,    "aarch64_sve_st1">;
 def SVST1H_S : MInst<"svst1h[_{d}]", "vPBd", "il",              [IsStore], MemEltTyInt16,   "aarch64_sve_st1">;

clang/lib/AST/ASTContext.cpp

#include "clang/Basic/OpenCLExtensionTypes.def"
     // SVE_VECTOR_TYPE is only known at runtime, but it is always a multiple
     // of 128 bits.  There is one predicate bit for each vector byte, so the
     // length of an SVE_PREDICATE_TYPE is always a multiple of 16 bits.
     //
     // Because the length is only known at runtime, we use a dummy value
     // of 0 for the static length.  The alignment values are those defined
     // by the Procedure Call Standard for the Arm Architecture.
 #define SVE_VECTOR_TYPE(Name, MangledName, Id, SingletonId, NumEls, ElBits,    \
-                        IsSigned, IsFP)                                        \
+                        IsSigned, IsFP, IsBF)                                  \
   case BuiltinType::Id:                                                        \
     Width = 0;                                                                 \
     Align = 128;                                                               \
     break;
 #define SVE_PREDICATE_TYPE(Name, MangledName, Id, SingletonId, NumEls)         \
   case BuiltinType::Id:                                                        \
     Width = 0;                                                                 \
     Align = 16;                                                                \

 /// getScalableVectorType - Return the unique reference to a scalable vector
 /// type of the specified element type and size. VectorType must be a built-in
 /// type.
 QualType ASTContext::getScalableVectorType(QualType EltTy,
                                            unsigned NumElts) const {
   if (Target->hasAArch64SVETypes()) {
     uint64_t EltTySize = getTypeSize(EltTy);
 #define SVE_VECTOR_TYPE(Name, MangledName, Id, SingletonId, NumEls, ElBits,    \
-                        IsSigned, IsFP)                                        \
+                        IsSigned, IsFP, IsBF)                                  \
   if (!EltTy->isBooleanType() &&                                               \
       ((EltTy->hasIntegerRepresentation() &&                                   \
         EltTy->hasSignedIntegerRepresentation() == IsSigned) ||                \
-       (EltTy->hasFloatingRepresentation() && IsFP)) &&                        \
-      EltTySize == ElBits && NumElts == NumEls)                                \
-    return SingletonId;
+       (EltTy->hasFloatingRepresentation() && !EltTy->isBFloat16Type() &&      \
+        IsFP && !IsBF) ||                                                      \
+       (EltTy->hasFloatingRepresentation() && EltTy->isBFloat16Type() &&       \
+        IsBF && !IsFP)) &&                                                     \
+      EltTySize == ElBits && NumElts == NumEls) {                              \
+    return SingletonId;                                                        \
+  }
 #define SVE_PREDICATE_TYPE(Name, MangledName, Id, SingletonId, NumEls)         \
   if (EltTy->isBooleanType() && NumElts == NumEls)                             \
     return SingletonId;
 #include "clang/Basic/AArch64SVEACLETypes.def"
   }
   return QualType();
 }
 

clang/lib/AST/ItaniumMangle.cpp

case BuiltinType::Id: \
     type_name = "ocl_" #ExtType; \
     Out << type_name.size() << type_name; \
     break;
 #include "clang/Basic/OpenCLExtensionTypes.def"
   // The SVE types are effectively target-specific.  The mangling scheme
   // is defined in the appendices to the Procedure Call Standard for the
   // Arm Architecture.
 #define SVE_VECTOR_TYPE(InternalName, MangledName, Id, SingletonId, NumEls,    \
-                        ElBits, IsSigned, IsFP)                                \
+                        ElBits, IsSigned, IsFP, IsBF)                          \
   case BuiltinType::Id:                                                        \
     type_name = MangledName;                                                   \
     Out << (type_name == InternalName ? "u" : "") << type_name.size()          \
         << type_name;                                                          \
     break;
 #define SVE_PREDICATE_TYPE(InternalName, MangledName, Id, SingletonId, NumEls) \
   case BuiltinType::Id:                                                        \
     type_name = MangledName;                                                   \

clang/lib/CodeGen/CGBuiltin.cpp

llvm::Type *CodeGenFunction::getEltType(SVETypeFlags TypeFlags) {
 
   case SVETypeFlags::EltTyFloat16:
     return Builder.getHalfTy();
   case SVETypeFlags::EltTyFloat32:
     return Builder.getFloatTy();
   case SVETypeFlags::EltTyFloat64:
     return Builder.getDoubleTy();
 
+  case SVETypeFlags::EltTyBFloat16:
+    return Builder.getBFloatTy();
+
   case SVETypeFlags::EltTyBool8:
   case SVETypeFlags::EltTyBool16:
   case SVETypeFlags::EltTyBool32:
   case SVETypeFlags::EltTyBool64:
     return Builder.getInt1Ty();
   }
 }
 
case SVETypeFlags::EltTyInt16:
     return llvm::ScalableVectorType::get(Builder.getInt16Ty(), 8);
   case SVETypeFlags::EltTyInt32:
     return llvm::ScalableVectorType::get(Builder.getInt32Ty(), 4);
   case SVETypeFlags::EltTyInt64:
     return llvm::ScalableVectorType::get(Builder.getInt64Ty(), 2);
 
   case SVETypeFlags::EltTyFloat16:
     return llvm::ScalableVectorType::get(Builder.getHalfTy(), 8);
+  case SVETypeFlags::EltTyBFloat16:
+    return llvm::ScalableVectorType::get(Builder.getBFloatTy(), 8);
   case SVETypeFlags::EltTyFloat32:
     return llvm::ScalableVectorType::get(Builder.getFloatTy(), 4);
   case SVETypeFlags::EltTyFloat64:
     return llvm::ScalableVectorType::get(Builder.getDoubleTy(), 2);
 
   case SVETypeFlags::EltTyBool8:
     return llvm::ScalableVectorType::get(Builder.getInt1Ty(), 16);
   case SVETypeFlags::EltTyBool16:

clang/lib/CodeGen/CodeGenTypes.cpp

llvm::ScalableVectorType::get(                                               \
     case BuiltinType::SveFloat64:
       return GET_SVE_FP_VEC(DoubleTy, false, 2);
     case BuiltinType::SveFloat64x2:
       return GET_SVE_FP_VEC(DoubleTy, false, 4);
     case BuiltinType::SveFloat64x3:
       return GET_SVE_FP_VEC(DoubleTy, false, 6);
     case BuiltinType::SveFloat64x4:
       return GET_SVE_FP_VEC(DoubleTy, false, 8);
+    case BuiltinType::SveBFloat16:
+      return GET_SVE_FP_VEC(BFloat16Ty, false, 8);
 #undef GET_SVE_FP_VEC
     case BuiltinType::Dependent:
 #define BUILTIN_TYPE(Id, SingletonId)
 #define PLACEHOLDER_TYPE(Id, SingletonId) \
     case BuiltinType::Id:
 #include "clang/AST/BuiltinTypes.def"
       llvm_unreachable("Unexpected placeholder builtin type!");
     }

clang/test/AST/ast-dump-aarch64-sve-types.c

 // CHECK-NEXT: -BuiltinType {{.*}} '__SVFloat16_t'
 
 // CHECK: TypedefDecl {{.*}} implicit __SVFloat32_t '__SVFloat32_t'
 // CHECK-NEXT: -BuiltinType {{.*}} '__SVFloat32_t'
 
 // CHECK: TypedefDecl {{.*}} implicit __SVFloat64_t '__SVFloat64_t'
 // CHECK-NEXT: -BuiltinType {{.*}} '__SVFloat64_t'
 
+// CHECK: TypedefDecl {{.*}} implicit __SVBFloat16_t '__SVBFloat16_t'
+// CHECK-NEXT: -BuiltinType {{.*}} '__SVBFloat16_t'
+
 // CHECK: TypedefDecl {{.*}} implicit __SVBool_t '__SVBool_t'
 // CHECK-NEXT: -BuiltinType {{.*}} '__SVBool_t'

clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_ld1ro-bfloat.c

+// RUN: %clang_cc1 -D__ARM_FEATURE_SVE_MATMUL_FP64 -D__ARM_FEATURE_BF16_SCALAR_ARITHMETIC -D__ARM_FEATURE_SVE -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -fallow-half-arguments-and-returns -S -O1 -Werror -Wall -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -D__ARM_FEATURE_SVE_MATMUL_FP64 -D__ARM_FEATURE_BF16_SCALAR_ARITHMETIC -D__ARM_FEATURE_SVE -DSVE_OVERLOADED_FORMS -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -fallow-half-arguments-and-returns -S -O1 -Werror -Wall -emit-llvm -o - %s | FileCheck %s
+
+#include <arm_sve.h>
+
+#ifdef SVE_OVERLOADED_FORMS
+// A simple used,unused... macro, long enough to represent any SVE builtin.
+#define SVE_ACLE_FUNC(A1, A2_UNUSED, A3, A4_UNUSED) A1##A3
+#else
+#define SVE_ACLE_FUNC(A1, A2, A3, A4) A1##A2##A3##A4
+#endif
+
+svbfloat16_t test_svld1ro_bf16(svbool_t pg, const bfloat16_t *base) {
+  // CHECK-LABEL: test_svld1ro_bf16
+  // CHECK: %[[PG:.*]] = call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> %pg)
+  // CHECK: %[[INTRINSIC:.*]] = call <vscale x 8 x bfloat> @llvm.aarch64.sve.ld1ro.nxv8bf16(<vscale x 8 x i1> %[[PG]], bfloat* %base)
+  // CHECK: ret <vscale x 8 x bfloat> %[[INTRINSIC]]
+  return SVE_ACLE_FUNC(svld1ro, _bf16, , )(pg, base);
+}

clang/unittests/AST/ASTImporterTest.cpp

static const char *const TypeNames[] = {
     "__SVInt16_t",
     "__SVInt32_t",
     "__SVInt64_t",
     "__SVUint8_t",
     "__SVUint16_t",
     "__SVUint32_t",
     "__SVUint64_t",
     "__SVFloat16_t",
+    "__SVBFloat16_t",
     "__SVFloat32_t",
     "__SVFloat64_t",
     "__SVBool_t"
   };
 
   TranslationUnitDecl *ToTU = getToTuDecl("", Lang_CXX03);
   TranslationUnitDecl *FromTU = getTuDecl("", Lang_CXX03, "input.cc");
   for (auto *TypeName : TypeNames) {

clang/unittests/AST/SizelessTypesTest.cpp

TEST_F(SizelessTypeTester, TestSizelessBuiltin) {
   ASSERT_TRUE(Ctx.SveUint16Ty->isSizelessBuiltinType());
   ASSERT_TRUE(Ctx.SveUint32Ty->isSizelessBuiltinType());
   ASSERT_TRUE(Ctx.SveUint64Ty->isSizelessBuiltinType());
 
   ASSERT_TRUE(Ctx.SveFloat16Ty->isSizelessBuiltinType());
   ASSERT_TRUE(Ctx.SveFloat32Ty->isSizelessBuiltinType());
   ASSERT_TRUE(Ctx.SveFloat64Ty->isSizelessBuiltinType());
 
+  ASSERT_TRUE(Ctx.SveBFloat16Ty->isSizelessBuiltinType());
+
   ASSERT_TRUE(Ctx.SveBoolTy->isSizelessBuiltinType());
 
   ASSERT_FALSE(Ctx.VoidTy->isSizelessBuiltinType());
   ASSERT_FALSE(Ctx.PseudoObjectTy->isSizelessBuiltinType());
   ASSERT_FALSE(FooTy->isSizelessBuiltinType());
 
   ASSERT_FALSE(Ctx.getPointerType(Ctx.SveBoolTy)->isSizelessBuiltinType());
   ASSERT_FALSE(
TEST_F(SizelessTypeTester, TestSizeless) {
   ASSERT_TRUE(Ctx.SveUint16Ty->isSizelessType());
   ASSERT_TRUE(Ctx.SveUint32Ty->isSizelessType());
   ASSERT_TRUE(Ctx.SveUint64Ty->isSizelessType());
 
   ASSERT_TRUE(Ctx.SveFloat16Ty->isSizelessType());
   ASSERT_TRUE(Ctx.SveFloat32Ty->isSizelessType());
   ASSERT_TRUE(Ctx.SveFloat64Ty->isSizelessType());
 
+  ASSERT_TRUE(Ctx.SveBFloat16Ty->isSizelessType());
+
   ASSERT_TRUE(Ctx.SveBoolTy->isSizelessType());
 
   ASSERT_FALSE(Ctx.VoidTy->isSizelessType());
   ASSERT_FALSE(Ctx.PseudoObjectTy->isSizelessType());
   ASSERT_FALSE(FooTy->isSizelessType());
 
   ASSERT_FALSE(Ctx.getPointerType(Ctx.SveBoolTy)->isSizelessType());
   ASSERT_FALSE(Ctx.getLValueReferenceType(Ctx.SveBoolTy)->isSizelessType());
   ASSERT_FALSE(Ctx.getRValueReferenceType(Ctx.SveBoolTy)->isSizelessType());
 }

clang/utils/TableGen/SveEmitter.cpp

public:
 
   unsigned getArg() const { return Arg; }
   unsigned getKind() const { return Kind; }
   unsigned getElementSizeInBits() const { return ElementSizeInBits; }
 };
 
 class SVEType {
   TypeSpec TS;
-  bool Float, Signed, Immediate, Void, Constant, Pointer;
+  bool Float, Signed, Immediate, Void, Constant, Pointer, BFloat;
   bool DefaultType, IsScalable, Predicate, PredicatePattern, PrefetchOp;
   unsigned Bitwidth, ElementBitwidth, NumVectors;
 
 public:
   SVEType() : SVEType(TypeSpec(), 'v') {}
 
   SVEType(TypeSpec TS, char CharMod)
       : TS(TS), Float(false), Signed(true), Immediate(false), Void(false),
-        Constant(false), Pointer(false), DefaultType(false), IsScalable(true),
-        Predicate(false), PredicatePattern(false), PrefetchOp(false),
-        Bitwidth(128), ElementBitwidth(~0U), NumVectors(1) {
+        Constant(false), Pointer(false), BFloat(false), DefaultType(false),
+        IsScalable(true), Predicate(false), PredicatePattern(false),
+        PrefetchOp(false), Bitwidth(128), ElementBitwidth(~0U), NumVectors(1) {
     if (!TS.empty())
       applyTypespec();
     applyModifier(CharMod);
   }
 
   bool isPointer() const { return Pointer; }
   bool isVoidPointer() const { return Pointer && Void; }
   bool isSigned() const { return Signed; }
   bool isImmediate() const { return Immediate; }
   bool isScalar() const { return NumVectors == 0; }
   bool isVector() const { return NumVectors > 0; }
   bool isScalableVector() const { return isVector() && IsScalable; }
   bool isChar() const { return ElementBitwidth == 8; }
   bool isVoid() const { return Void & !Pointer; }
   bool isDefault() const { return DefaultType; }
   bool isFloat() const { return Float; }
-  bool isInteger() const { return !Float && !Predicate; }
+  bool isBFloat() const { return BFloat; }
+  bool isFloatingPoint() const { return Float || BFloat; }
+  bool isInteger() const { return !isFloatingPoint() && !Predicate; }
   bool isScalarPredicate() const {
-    return !Float && Predicate && NumVectors == 0;
+    return !isFloatingPoint() && Predicate && NumVectors == 0;
   }
   bool isPredicateVector() const { return Predicate; }
   bool isPredicatePattern() const { return PredicatePattern; }
   bool isPrefetchOp() const { return PrefetchOp; }
   bool isConstant() const { return Constant; }
   unsigned getElementSizeInBits() const { return ElementBitwidth; }
   unsigned getNumVectors() const { return NumVectors; }
 

 
 std::string SVEType::builtin_str() const {
   std::string S;
   if (isVoid())
     return "v";
 
   if (isVoidPointer())
     S += "v";
-  else if (!Float)
+  else if (!isFloatingPoint())
     switch (ElementBitwidth) {
     case 1: S += "b"; break;
     case 8: S += "c"; break;
     case 16: S += "s"; break;
     case 32: S += "i"; break;
     case 64: S += "Wi"; break;
     case 128: S += "LLLi"; break;
     default: llvm_unreachable("Unhandled case!");
     }
-  else
+  else if (isFloat())
     switch (ElementBitwidth) {
     case 16: S += "h"; break;
     case 32: S += "f"; break;
     case 64: S += "d"; break;
     default: llvm_unreachable("Unhandled case!");
     }
+  else if (isBFloat()) {
+    assert(ElementBitwidth == 16 && "Not a valid BFloat.");
+    S += "y";
+  }
 
-  if (!isFloat()) {
+  if (!isFloatingPoint()) {
     if ((isChar() || isPointer()) && !isVoidPointer()) {
       // Make chars and typed pointers explicitly signed.
       if (Signed)
         S = "S" + S;
       else if (!Signed)
         S = "U" + S;
     } else if (!isVoidPointer() && !Signed) {
       S = "U" + S;
if (isPrefetchOp())
     return "sv_prfop";
 
   std::string S;
   if (Void)
     S += "void";
   else {
     if (isScalableVector())
       S += "sv";
-    if (!Signed && !Float)
+    if (!Signed && !isFloatingPoint())
       S += "u";
 
     if (Float)
       S += "float";
     else if (isScalarPredicate() || isPredicateVector())
       S += "bool";
+    else if (isBFloat())
+      S += "bfloat";
     else
       S += "int";
 
     if (!isScalarPredicate() && !isPredicateVector())
       S += utostr(ElementBitwidth);
     if (!isScalableVector() && isVector())
       S += "x" + utostr(getNumElements());
     if (NumVectors > 1)
for (char I : TS) {
     case 'f':
       Float = true;
       ElementBitwidth = 32;
       break;
     case 'd':
       Float = true;
       ElementBitwidth = 64;
       break;
+    case 'b':
+      BFloat = true;
+      ElementBitwidth = 16;
+      break;
     default:
       llvm_unreachable("Unhandled type code!");
     }
   }
   assert(ElementBitwidth != ~0U && "Bad element bitwidth!");
 }
 
 void SVEType::applyModifier(char Mod) {
case 'b':
     ElementBitwidth /= 4;
     break;
   case 'o':
     ElementBitwidth *= 4;
     break;
   case 'P':
     Signed = true;
     Float = false;
+    BFloat = false;
     Predicate = true;
     Bitwidth = 16;
     ElementBitwidth = 1;
     break;
   case 's':
   case 'a':
     Bitwidth = ElementBitwidth;
     NumVectors = 0;
Intrinsic::Intrinsic(StringRef Name, StringRef Proto, uint64_t MergeTy,
                      StringRef MergeSuffix, uint64_t MemoryElementTy,
                      StringRef LLVMName, uint64_t Flags,
                      ArrayRef<ImmCheck> Checks, TypeSpec BT, ClassKind Class,
                      SVEEmitter &Emitter, StringRef Guard)
     : Name(Name.str()), LLVMName(LLVMName), Proto(Proto.str()),
       BaseTypeSpec(BT), Class(Class), Guard(Guard.str()),
       MergeSuffix(MergeSuffix.str()), BaseType(BT, 'd'), Flags(Flags),
       ImmChecks(Checks.begin(), Checks.end()) {
-
   // Types[0] is the return value.
   for (unsigned I = 0; I < Proto.size(); ++I) {
     SVEType T(BaseTypeSpec, Proto[I]);
     Types.push_back(T);
 
     // Add range checks for immediates
     if (I > 0) {
       if (T.isPredicatePattern())
while (Ret.find('{') != std::string::npos) {
     }
 
     // Replace templated arg with the right suffix (e.g. u32)
     std::string TypeCode;
     if (T.isInteger())
       TypeCode = T.isSigned() ? 's' : 'u';
     else if (T.isPredicateVector())
       TypeCode = 'b';
+    else if (T.isBFloat())
+      TypeCode = "bf";
     else
       TypeCode = 'f';
     Ret.replace(Pos, NumChars, TypeCode + utostr(T.getElementSizeInBits()));
   }
 
   return Ret;
 }
 
case 32:
       return encodeEltType("EltTyFloat32");
     case 64:
       return encodeEltType("EltTyFloat64");
     default:
       llvm_unreachable("Unhandled float element bitwidth!");
     }
   }
 
+  if (T.isBFloat()) {
+    assert(T.getElementSizeInBits() == 16 && "Not a valid BFloat.");
+    return encodeEltType("EltTyBFloat16");
+  }
+
   if (T.isPredicateVector()) {
     switch (T.getElementSizeInBits()) {
     case 8:
       return encodeEltType("EltTyBool8");
     case 16:
       return encodeEltType("EltTyBool16");
     case 32:
       return encodeEltType("EltTyBool32");
void SVEEmitter::createHeader(raw_ostream &OS) {
   OS << "typedef __SVInt16_t svint16_t;\n";
   OS << "typedef __SVInt32_t svint32_t;\n";
   OS << "typedef __SVInt64_t svint64_t;\n";
   OS << "typedef __SVUint8_t svuint8_t;\n";
   OS << "typedef __SVUint16_t svuint16_t;\n";
   OS << "typedef __SVUint32_t svuint32_t;\n";
   OS << "typedef __SVUint64_t svuint64_t;\n";
   OS << "typedef __SVFloat16_t svfloat16_t;\n";
+  OS << "typedef __SVBFloat16_t svbfloat16_t;\n\n";
+
+  OS << "#ifdef __ARM_FEATURE_BF16_SCALAR_ARITHMETIC\n";
+  OS << "typedef __bf16 bfloat16_t;\n";
+  OS << "#endif\n\n";
+
   OS << "typedef __SVFloat32_t svfloat32_t;\n";
   OS << "typedef __SVFloat64_t svfloat64_t;\n";
   OS << "typedef __clang_svint8x2_t svint8x2_t;\n";
   OS << "typedef __clang_svint16x2_t svint16x2_t;\n";
   OS << "typedef __clang_svint32x2_t svint32x2_t;\n";
   OS << "typedef __clang_svint64x2_t svint64x2_t;\n";
   OS << "typedef __clang_svuint8x2_t svuint8x2_t;\n";
   OS << "typedef __clang_svuint16x2_t svuint16x2_t;\n";